\documentclass{standalone}

\begin{document}

\section*{Mathematical background}\addcontentsline{toc}{section}{Mathematical background}
\markboth{Appendix A}{Mathematical background}

The exact knowledge of prior probabilities and conditional probabilities are generally hard to evaluate, thus a parametric approach is often needed.
A parametric approach aims to create reasonable hypotheses about the data distribution and its fundamental parameters (e.g mean, variance, $\cdots$).
In the following discussion, we are going to focus only on normal distributions for mathematical convenience, but the results could be easily generalized.

Given the multi-dimensional form of Gauss distribution:

\begin{equation}
G(\mathbf{x}|\mu, \Sigma) = \frac{1}{(2\pi)^{d/2}\cdot\left|\Sigma\right|^{1/2}}\cdot exp\left[-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^T\Sigma^{-1}(\mathbf{x}-\mathbf{\mu})\right]
\end{equation}
\\
where $\mathbf{x}$ is a $d$-dimensional column vector, $\mathbf{\mu}$ is the mean vector of the distribution, $\Sigma$ is the covariance matrix ($d\times d$) and $|\Sigma|$ and $\Sigma^{-1}$ the determinant and its inverse, respectively.
We can notice the quadratic dependence of $G$ by $\mathbf{x}$,

\begin{equation}
\Delta^2 = (\mathbf{x}-\mu)^T\Sigma^{-1}(\mathbf{x}-\mu)
\end{equation}
\\
where the exponent ($\Delta^2$) is called \emph{Mahalanobis distance} of vector $\mathbf{x}$ from its mean.
This distance can be reduced to the Euclidean one when the covariance matrix is the identity matrix ($\mathbf{I}$).

The covariance matrix is always symmetric and positive semi-definite by definition (useful information for the next algorithmic strategies) so it is invertible.
If the covariance matrix has only diagonal terms the multidimensional distribution can be expressed as the simple product of $d$ mono-dimensional normal distributions.
In this case the main axes are parallel to the Cartesian axes.

Starting from a multi-variate Gaussian distribution\footnote{
  In Machine Learning it will correspond to the conditional probability density.
}, the Bayesian rule for classification problems can be rewritten as:

\begin{equation}
g_i(\mathbf{x}) = P(w_i|\mathbf{x}) = \frac{p(\mathbf{x}|w_i)P(w_i)}{p(\mathbf{x})} = \frac{1}{(2\pi)^{d/2}\cdot\left|\Sigma_i\right|^{1/2}}\cdot exp\left[-\frac{1}{2}(\mathbf{x}-\mathbf{\mu_i})^T{\Sigma_i}^{-1}(\mathbf{x}-\mathbf{\mu_i})\right] \frac{P(w_i)}{p(\mathbf{x})}
\end{equation}
\\
where, removing constant terms ($\pi$ factors and the absolute probability density $p(\mathbf{x}) = \sum_{i=1}^s p(\mathbf{x}|w_i)\cdot P(w_i)$) and using the monotonicity of the function, we can extract the logarithmic relation:

\begin{equation}
g_i(\mathbf{x}) = -\frac{1}{2}(\mathbf{x}-\mu_i)^T{\Sigma_i}^{-1}(\mathbf{x}-\mu_i) -\frac{1}{2}\log\left|\Sigma_i\right|+\log P(w_i)
\end{equation}
\\
which is called \emph{Quadratic Discriminant function}.

The dependency by the covariance matrix allows 5 different cases:

\begin{itemize}

\item \textbf{$\Sigma_i=\sigma^2I$ - DiagLinear Classifier}

\begin{minipage}{.30\textwidth}
\hspace{-.5cm}
\includegraphics[width=1\textwidth]{case1.png}
\end{minipage}%
\begin{minipage}{.70\textwidth}
This is the case in which features are completely independent, i.e they have equal variances for each class.
This hypothesis allows us to simplify the discriminant function as:
\end{minipage}\\

\begin{equation}
g_i(\mathbf{x})=-\frac{1}{2\sigma^2}(\mathbf{x^Tx}-2{\mu_i}^T\mathbf{x} + {\mu_i}^T\mu_i) + \log P(w_i)
\end{equation}
\\
and removing all the $\mathbf{x^Tx}$ constant terms for each class

\begin{equation}
g_i(\mathbf{x}) = -\frac{1}{2\sigma^2}(-2{\mu_i}^T\mathbf{x}+{\mu_i}^T\mu_i)+\log P(w_i) = \mathbf{{w_i}^Tx}+\mathbf{w_0}
\end{equation}
\\
These simplifications create a linear discriminant function and the separation surfaces between classes are hyper-planes ($g_i(\mathbf{x})=g_j(\mathbf{x})$).

With equal prior probability the function can be rewritten as

\begin{equation}
g_i(\mathbf{x}) = -\frac{1}{2\sigma^2}(\mathbf{x}-\mu_i)^T(\mathbf{x}-\mu_i)
\end{equation}
\\
which is called \emph{nearest mean classifier} and the equal-probability surfaces are hyper-spheres.

\item \textbf{$\Sigma_i = \Sigma$ (diagonal matrix) - Linear Classifier}

\begin{minipage}{.30\textwidth}
\hspace{-.5cm}
\includegraphics[width=1\textwidth]{case2.png}
\end{minipage}%
\begin{minipage}{.70\textwidth}
In this case the classes have same covariances but each feature has its own different variance.
After the substitution of $\Sigma$ in the equation, we obtain
\end{minipage}\\

\begin{equation}
g_i(\mathbf{x}) = -\frac{1}{2}\sum_{k=1}^{s}\frac{(\mathbf{x_k}-\mu_{i,k})^2}{{\sigma_k}^2}-\frac{1}{2}\log\prod_{k=1}^{s}{\sigma_k}^2+\log P(w_i)
\end{equation}
\\
where we can remove constant $\mathbf{x_k}^2$ terms (equal for each class) and obtain another time a linear discriminant function and discriminant surfaces given by hyper-planes and equal-probability boundaries given by hyper-ellipsoids.
We remark that the only difference from the previous case is the normalization factor of each axis that in this case is given by its variance.

\item \textbf{$\Sigma_i = \Sigma$ (non-diagonal matrix) - Mahalanobis Classifier}

\begin{minipage}{.30\textwidth}
\hspace{-.5cm}
\includegraphics[width=1\textwidth]{case3.png}
\end{minipage}%
\begin{minipage}{.70\textwidth}
In this case we assume that each class has the same covariance matrix, but they are non-diagonal ones.
The discriminant function becomes
\end{minipage}\\

\begin{equation}
g_i(\mathbf{x}) = -\frac{1}{2}(\mathbf{x}-\mu_i)^T{\Sigma}^{-1}(\mathbf{x}-\mu_i) -\frac{1}{2}\log\left|\Sigma\right|+\log P(w_i)
\end{equation}
\\
where we can remove the $\log\left|\Sigma\right|$ term because it is constant for all the classes and we can assume equal prior probabilities.
In this case we obtain

\begin{equation}
g_i(\mathbf{x}) = -\frac{1}{2}(\mathbf{x}-\mu_i)^T{\Sigma}^{-1}(\mathbf{x}-\mu_i)
\end{equation}
\\
where the quadratic term is the above told \emph{Mahalanobis distance}, i.e a normalization of the distance according to the inverse of the covariance matrix.
We can prove that expanding the scalar product and removing the constant $\mathbf{x^T\Sigma^{-1}x}$ term, we still obtain a linear discriminant function with the same properties of the previous case.
In this case the hyper-ellipsoids have axes aligned to the eigenvectors of the $\Sigma$ matrix.

\item \textbf{$\Sigma_i = {\sigma_i}^2I$ - DiagQuadratic Classifier}

\begin{minipage}{.30\textwidth}
\hspace{-.5cm}
\includegraphics[width=1\textwidth]{case4.png}
\end{minipage}%
\begin{minipage}{.70\textwidth}
In this case we have a different covariance matrix for each class but they are all proportional to the identity matrix, i.e diagonal matrix.
The discriminant function in this case becomes
\end{minipage}\\

\begin{equation}
g_i(\mathbf{x}) = -\frac{1}{2}(\mathbf{x}-\mu_i)^T{\sigma_i}^{-2}(\mathbf{x}-\mu_i) -\frac{1}{2}s\log\left|{\sigma_i}^2\right|+\log P(w_i)
\end{equation}
\\
where this expression can be further reduced obtaining a quadratic discriminant function.
In this case the equal-probability boundaries are hyper-spheres aligned to the feature axes.

\item \textbf{$\Sigma_i \neq\Sigma_j$ (general case) - Quadratic Classifier}

\begin{minipage}{.30\textwidth}
\hspace{-.5cm}
\includegraphics[width=1\textwidth]{case5.png}
\end{minipage}%
\begin{minipage}{.70\textwidth}
Starting from the more general discriminant function we can relabel the variables and highlight its quadratic form as
\end{minipage}\\

\begin{equation}
g_i(\mathbf{x}) = \mathbf{x^TW_{2,i}x}+\mathbf{w_{1,i}^Tx} + \mathbf{w_{0,i}} \quad \mbox{with}\quad \left\{\begin{array}{l} \mathbf{W_{2,i}}=-\frac{1}{2}{\Sigma_i}^{-1}\\ \mathbf{w_{1,i}}={\Sigma_i}^{-1}\mu_i \\ \mathbf{w_{0,i}}=-\frac{1}{2}{\mu_i}^T{\Sigma_i}^{-1}\mu_i-\frac{1}{2}\log\left|\Sigma_i\right|+\log P(w_i) \\ \end{array}\right.
\end{equation}
\\
In this case each class has its own covariance matrix $\Sigma_i$ and the equal-probability boundaries are hyper-ellipsoids oriented to the eigenvectors of the covariance matrix of each class.

\end{itemize}

The Gaussian distribution hypothesis of data should be tested before using these classifiers.
It can be evaluated using statistical tests as \href{https://www.jstor.org/stable/2284163?seq=1#page_scan_tab_contents}{\emph{Malkovich-Afifi}} based on \href{https://en.wikipedia.org/wiki/Kolmogorov–Smirnov_test}{\emph{Kolmogorov-Smirnov}} index or using the empirical visualization of the data points.

\end{document}


%
% Chapter 3.1
%

\section*{3.1 Max and Min Values}

\subsection*{Absolute Extrema}

Let \(c\) be a number in the domain \(D\) of the function \(f\). Then \(f(x)\) is the
\begin{itemize}
    \item \textbf{absolute maximum} value of \(f\) on \(D\) if \(f(c) \geq f(x)\) for all \(x\) in \(D\).
    \item \textbf{absolute minimum} value of \(f\) on \(D\) if \(f(c) \leq f(x)\) for all \(x\) in \(D\).
\end{itemize}

\subsection*{Relative Extrema}

The number \(f(c)\) is a
\begin{itemize}
    \item \textbf{local maximum} value of \(f\) if \(f(c) \geq f(x)\) when \(x\) is near \(c\).
    \item \textbf{local minimum} value of \(f\) if \(f(c) \leq f(x)\) when \(x\) is near \(c\).
\end{itemize}

\subsection*{Extreme Value Theorem}

If \(f\) is continuous on a closed interval \([a, b]\), then \(f\) attains an absolute maximum value \(f(x)\) and an absolute minimum value \(f(d)\) at some numbers \(c\) and \(d\) in \([a, b]\).

\subsection*{Fermat's Theorem}

If \(f\) has a local maximum or minimum at \(c\), and if \(f'(x)\) exists, then \(f'(c)=0\).
\\\\\\
A \textbf{critical number} of a function \(f\) is a number \(c\) in the domain of \(f\) such that either \(f'(c)=0\) or \(f'(c)\) does not exist. If \(f\) has a local maximum or a minimum at \(c\), then \(c\) is a critical number of \(f\).

\subsection*{Closed Interval Method (Championship Method)}

To find the absolute maximum and minimum values of a continuous function \(f\) on a closed interval \([a, b]\):
\begin{enumerate}
    \item Find the values of \(f\) at the critical values of \(f\) in \((a, b)\).
    \item Find the values of \(f\) at the endpoints of the interval.
    \item The largest of the values is the absolute maximum; the smallest of these values is the absolute minimum.
\end{enumerate}


\section{Definition of vector spaces}
\label{sec:definition-vector-spaces}

\begin{outcome}
  \begin{enumerate}
  \item Develop the concept of a vector space through axioms.
  \item Use the vector space axioms to determine if a set and its
    operations constitute a vector space.
  \item Encounter several examples of vector spaces.
  \end{enumerate}
\end{outcome}

\begin{definition}{Vector space}{vector-space}
  Let $K$ be a field. A \textbf{vector space} over $K$%
  \index{vector space} is a set $V$ equipped with two operations of
  \textbf{addition}%
  \index{addition!in a vector space}%
  \index{vector!addition}%
  \index{addition!of vectors} and \textbf{scalar multiplication}%
  \index{scalar multiplication!in a vector space}%
  \index{vector!scalar multiplication}%
  \index{scalar multiplication!of a vector}%
  \index{multiplication!scalar multiplication|see{scalar multiplication}},
  such that the following properties hold:
  \begin{itemize}\setlength\itemsep{0em}
  \item[(A1)] Commutative law of addition%
    \index{commutative law!of addition}%
    \index{vector!commutative law of addition}:
    $\vect{u} + \vect{v}=\vect{v} + \vect{u}$.
  \item[(A2)] Associative law of addition%
    \index{associative law!of addition}%
    \index{vector!associative law of addition}:
    $(\vect{u} + \vect{v}) + \vect{w}=\vect{u} + (\vect{v} + \vect{w})$.
  \item[(A3)] The existence of an additive unit%
    \index{additive unit}%
    \index{unit!of addition|see{additive unit}}%
    \index{vector!additive unit}: there exists an element
    $\vect{0}\in V$ such that for all $\vect{u}$,
    $\vect{u} + \vect{0}=\vect{u}$.
  \item[(A4)] The law of additive inverses%
    \index{additive inverse}%
    \index{vector!additive inverse}%
    \index{inverse!additive}:
    $\vect{u} + (-\vect{u}) =\vect{0}$.
  \item[(SM1)] The distributive law over vector addition%
    \index{distributive law!over vector addition}%
    \index{vector!distributive law}:
    $k(\vect{u} + \vect{v}) = k\vect{u} + k\vect{v}$.
  \item[(SM2)] The distributive law over scalar addition%
    \index{distributive law!over scalar addition}:
    $(k + \ell) \vect{u} = k \vect{u} + \ell\vect{u}$.
  \item[(SM3)] The associative law for scalar multiplication%
    \index{associative law!of scalar multiplication}%
    \index{vector!associative law of scalar multiplication}:
    $k(\ell\vect{u}) = (k \ell)\vect{u}$.
  \item[(SM4)] The rule for multiplication by one%
    \index{rule for multiplication by 1}%
    \index{vector!rule for multiplication by 1}:
    $1\vect{u}=\vect{u}$.
  \end{itemize}
\end{definition}

The above definition is concerned about two operations: vector
addition, denoted by $\vect{v} + \vect{w}$, and scalar multiplication,
denoted by $k\vect{v}$ or sometimes $k\cdot\vect{v}$. In the law of
additive inverses, we have written $-\vect{u}$ for $(-1)\vect{u}$.
Often, the scalars will be real numbers, but it is also possible to
use scalars from a different field $K$. We also use the term
\textbf{$K$-vector space}%
\index{K-vector space@$K$-vector space} to refer to a vector space
over a field $K$.  When $K=\R$, we also speak of a \textbf{real vector
  space}%
\index{vector space!real}%
\index{real vector space}, and when $K=\C$, we speak of a
\textbf{complex vector space}%
\index{vector space!complex}%
\index{complex vector space}. If the field is clear from the context,
we often don't mention it at all, and just speak of a ``vector
space''. The elements of a vector space are called \textbf{vectors}%
\index{vector!in a vector space}.

Our first example of a vector space is of course $\R^n$.

\begin{example}{$\R^n$ is a vector space}{Rn-vector-space}
  The set $\R^n$ of $n$-dimensional real column vectors, with the
  usual operations of vector addition and scalar multiplication, is a
  vector space.

  More generally, if $K$ is a field, the set $K^n$ of $n$-dimensional
  column vectors with components in $K$ is a $K$-vector space.
\end{example}

\begin{proof}
  Properties (A1)--(A4) hold by
  Proposition~\ref{prop:properties-vector-addition}, and properties
  (SM1)--(SM4) hold by Proposition~\ref{prop:vector-scalar-multiplication}.
\end{proof}

We now consider some other examples of vector spaces.

\begin{example}{Vector space of polynomials of degree 2}{vector-space-polynomials2}
  Let $\Poly_2$%
  \index{P2@$\Poly_2$}%
  \index{vector space!of polynomials} be the set of all polynomials%
  \index{polynomial} of degree at most $2$ with coefficients from a
  field $K$, i.e., expressions of the form
  \begin{equation*}
    p(x) = ax^2 + bx + c,
  \end{equation*}
  where $a,b,c\in K$. Define addition%
  \index{polynomial!addition}%
  \index{addition!of polynomials} and scalar multiplication%
  \index{polynomial!scalar multiplication}%
  \index{scalar multiplication!of polynomials} of polynomials in the
  usual way, i.e.,
  \begin{eqnarray*}
    (ax^2 + bx + c) + (a'x^2 + b'x + c') &=& (a + a')x^2 + (b + b')x + (c + c') \\
    k(ax^2 + bx + c) &=& ka\,x^2 + kb\,x + kc.
  \end{eqnarray*}
  Then $\Poly_2$ is a vector space.
\end{example}

\begin{proof}
  To show that $\Poly_2$ is a vector space, we verify the $8$ vector
  space axioms. Let
  \begin{eqnarray*}
    p(x) &=& a_2x^2 + a_1x + a_0, \\
    q(x) &=& b_2x^2 + b_1x + b_0, \\
    r(x) &=& c_2x^2 + c_1x + c_0
  \end{eqnarray*}
  be polynomials in $\Poly_2$ and let $k,\ell$ be scalars.

  \begin{itemize}
  \item[(A1)] We prove the commutative law of addition.
    \begin{eqnarray*}
      p(x) + q(x)
      &=& (a_2x^2 + a_1x + a_0) + (b_2x^2 + b_1x + b_0) \\
      &=& (a_2 + b_2)x^2 + (a_1 + b_1)x + (a_0 + b_0) \\
      &=& (b_2 + a_2)x^2 + (b_1 + a_1)x + (b_0 + a_0) \\
      &=& (b_2x^2 + b_1x + b_0) + (a_2x^2 + a_1x + a_0) \\
      &=& q(x) + p(x).
    \end{eqnarray*}
  \item[(A2)] We prove the associative law of addition.
    \begin{eqnarray*}
      (p(x) + q(x)) + r(x)
      &=& ((a_2x^2 + a_1x + a_0) + (b_2x^2 + b_1x + b_0)) + (c_2x^2 + c_1x + c_0) \\
      &=& ((a_2 + b_2)x^2 + (a_1 + b_1)x + (a_0 + b_0)) + (c_2x^2 + c_1x + c_0) \\
      &=& ((a_2 + b_2) + c_2)x^2 + ((a_1 + b_1) + c_1)x + ((a_0 + b_0) + c_0) \\
      &=& (a_2 + (b_2 + c_2))x^2 + (a_1 + (b_1 + c_1))x + (a_0 + (b_0 + c_0)) \\
      &=& (a_2x^2 + a_1x + a_0) + ((b_2 + c_2)x^2 + (b_1 + c_1)x + (b_0 + c_0)) \\
      &=& (a_2x^2 + a_1x + a_0) + ((b_2x^2 + b_1x + b_0) + (c_2x^2 + c_1x + c_0)) \\
      &=& p(x) + (q(x) + r(x)).
    \end{eqnarray*}
  \item[(A3)] To prove the existence of an additive unit, let $0(x) =
    0x^2 + 0x + 0$, the so-called \textbf{zero polynomial}%
    \index{polynomial!zero}%
    \index{zero polynomial}. Then
    \begin{eqnarray*}
      p(x) + 0(x)  &=&  (a_2x^2 + a_1x + a_0) + (0x^2 + 0x + 0) \\
                   &=&  (a_2 + 0)x^2 + (a_1 + 0)x + (a_0 + 0) \\
                   &=&  a_2x^2 + a_1x + a_0 \\
                   &=&  p(x).
    \end{eqnarray*}
  \item[(A4)] We prove the law of additive inverses.
    \begin{eqnarray*}
      p(x) + (-p(x)) &=& (a_2x^2 + a_1x + a_0) + (- a_2x^2  - a_1x - a_0) \\
                     &=& (a_2 - a_2)x^2 + (a_1 - a_1)x + (a_0 - a_0) \\
                     &=& 0x^2 + 0x + 0 \\
                     &=& 0(x).
    \end{eqnarray*}
  \item[(SM1)] We prove the distributive law over vector addition.
    \begin{eqnarray*}
      k(p(x) + q(x)) &=& k ((a_2x^2 + a_1x + a_0) + (b_2x^2 + b_1x + b_0)) \\
                     &=& k ((a_2 + b_2)x^2 + (a_1 + b_1)x + (a_0 + b_0)) \\
                     &=& k(a_2 + b_2)x^2 + k(a_1 + b_1)x + k(a_0 + b_0) \\
                     &=& (ka_2 + kb_2)x^2 + (ka_1 + kb_1)x + (ka_0 + kb_0) \\
                     &=& (ka_2x^2 + ka_1x + ka_0) + (kb_2x^2 + kb_1x + kb_0) \\
                     &=& kp(x) + kq(x).
    \end{eqnarray*}
  \item[(SM2)] We prove the distributive law over scalar addition.
    \begin{eqnarray*}
      (k + \ell) p(x) &=& (k + \ell) (a_2x^2 + a_1x + a_0) \\
                 &=& (k + \ell)a_2x^2 + (k + \ell)a_1x + (k + \ell)a_0   \\
                 &=& (ka_2x^2 + ka_1x + ka_0) + (\ell a_2x^2 + \ell a_1x + \ell a_0) \\
                 &=& kp(x) + \ell p(x).
    \end{eqnarray*}
  \item[(SM3)] We prove the associative law for scalar multiplication.
    \begin{eqnarray*}
      k(\ell p(x)) &=& k(\ell(a_2x^2 + a_1x + a_0)) \\
               &=& k(\ell a_2x^2 + \ell a_1x + \ell a_0) \\
               &=&  k\ell a_2x^2 + k\ell a_1x + k\ell a_0 \\
               &=& (k\ell) (a_2x^2 + a_1x + a_0) \\
               &=& (k\ell) p(x).
    \end{eqnarray*}
  \item[(SM4)] Finally, we prove the rule for multiplication by one.
    \begin{eqnarray*}
      1p(x) &=& 1 (a_2x^2 + a_1x + a_0) \\
            &=& 1a_2x^2 + 1a_1x + 1a_0 \\
            &=& a_2x^2 + a_1x + a_0 \\
            &=& p(x).
    \end{eqnarray*}
  \end{itemize}
  Since the operations of addition and scalar multiplication on
  $\Poly_2$ satisfy the $8$ vector space axioms, $\Poly_2$ is a vector
  space.
\end{proof}

Our next example of a vector space is the set of all
$n\times m$-matrices.

\begin{example}{Vector space of matrices}{vector-space-matrices}
  Let $\Mat_{m,n}$%
  \index{Mmn@$\Mat_{m,n}$} be the set of all $m\times n$-matrices
  with entries in a field $K$%
  \index{vector space!of matrices}%
  \index{matrix!vector space of}, together with the usual operations
  of matrix addition and scalar multiplication. Then $\Mat_{m,n}$ is a
  vector space.
\end{example}

\begin{proof}
  The properties (A1)--(A4) hold by
  Proposition~\ref{prop:properties-of-addition}, and the properties
  (SM1)--(SM4) hold by Proposition~\ref{prop:properties-scalar-multiplication}.
\end{proof}

We now examine an example of a set that does not satisfy all of the
above axioms, and is therefore \textit{not} a vector space.

\begin{example}{Not a vector space}{not-vector-space}
  Let $V$ denote the set of $2 \times 3$-matrices. Let us define a
  non-standard addition in $V$ by $A \oplus B = A$ for all matrices
  $A,B\in V$. Let scalar multiplication in $V$ be the usual scalar
  multiplication of matrices. Show that $V$ is not a vector space.
\end{example}

\begin{solution}
  In order to show that $V$ is not a vector space, it suffices to find
  one of the 8 axioms that is not satisfied. We will begin by examining
  the axioms for addition until one is found which does not hold. In
  fact, for this example, the very first axiom fails. Let
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      1 & 0 & 0 \\
      0 & 0 & 0
    \end{mymatrix},
    \quad
    B = \begin{mymatrix}{rrr}
      0 & 0 & 0 \\
      1 & 0 & 0
    \end{mymatrix}.
  \end{equation*}
  Then $A\oplus B=A$ and $B\oplus A=B$. Since $A\neq B$, we have
  $A\oplus B\neq B\oplus A$ for these two matrices, so property (A1)
  is false.
\end{solution}

Our next example looks a little different.

\begin{example}{Vector space of functions}{vector-space-function}
  Let $X$ be a nonempty set, $K$ a field, and define $\Func_{X,K}$%
  \index{FuncXK@$\Func_{X,K}$} to be the set of functions
  \index{function!vector space of} defined on $X$ and valued in
  $K$. In other words, the elements of $\Func_{X,K}$ are functions
  $f:X\to K$. The sum of two functions is defined by
  \begin{equation*}
    (f + g)(x) = f(x) + g(x),
  \end{equation*}
  and the scalar multiplication is defined by
  \begin{equation*}
    (kf) (x) = k(f(x)).
  \end{equation*}
  Then $\Func_{X,K}$ is a vector space.
\end{example}

\begin{proof}
  To verify that $\Func_{X,K}$ is a vector space, we must prove the 8
  axioms of vector spaces. Let $f, g, h$ be functions in $\Func_{X,K}$,
  and let $k,\ell$ be scalars. Recall that two functions $f,g$
  are \textbf{equal}%
  \index{function!equality of}%
  \index{equality of functions} if for all $x\in X$, we have
  $f(x)=g(x)$.

  \begin{itemize}
  \item[(A1)] We prove the commutative law of addition. For all
    $x\in X$, we have
    \begin{equation*}
      (f + g) (x)
      ~=~ f(x) + g(x)
      ~=~ g(x) + f(x)
      ~=~ (g + f) (x).
    \end{equation*}
    Therefore, $f + g = g + f$.
  \item[(A2)] We prove the associative law of addition. For all
    $x\in X$, we have
    \begin{equation*}
      ((f + g) + h) (x)
      ~=~ (f + g) (x) + h(x)
      ~=~ (f(x) + g(x)) + h(x)
    \end{equation*}
    \begin{equation*}
      ~=~ f(x) + (g(x) + h(x))
      ~=~ (f(x) + (g + h) (x))
      ~=~ (f + (g + h)) (x).
    \end{equation*}
    Therefore, $(f + g) + h = f + (g + h)$.
  \item[(A3)] To prove the existence of an additive unit, let $0$
    denote the function that is given by $0(x)=0$. This is called the
    \textbf{zero function}%
    \index{function!zero function}%
    \index{zero function}. It is an additive unit because for all $x$,
    \begin{equation*}
      (f + 0) (x)
      ~=~ f(x) + 0(x)
      ~=~ f(x),
    \end{equation*}
    and so $f+0 = f$.
  \item[(A4)] We prove the law of additive inverses. Let $-f = (-1)f$
    be the function that satisfies $(-f) (x) = -f(x)$. Then for all $x$,
    \begin{equation*}
      (f + (-f)) (x)
      ~=~ f(x) + (-f) (x)
      ~=~ f(x) + -f(x)
      ~=~ 0.
    \end{equation*}
    Therefore $f + (-f) = 0$.
  \item[(SM1)] We prove the distributive law over vector addition. For
    all $x$, we have
    \begin{equation*}
      (k(f + g)) (x)
      ~=~ k(f + g) (x)
      ~=~ k(f(x) + g(x))
    \end{equation*}
    \begin{equation*}
      ~=~ kf(x) + k g(x)
      ~=~ (kf + kg) (x),
    \end{equation*}
    and so $k(f + g) = kf + kg$.
  \item[(SM2)] We prove the distributive law over scalar addition.
    \begin{equation*}
      ((k + \ell) f) (x)
      ~=~ (k + \ell) f(x)
      ~=~ kf(x) + \ell f(x)
      ~=~ (kf + \ell f) (x),
    \end{equation*}
    and so $(k + \ell) f = kf + \ell f$.
  \item[(SM3)] We prove the associative law for scalar multiplication.
    \begin{equation*}
      ((k\ell ) f) (x)
      ~=~ (k\ell) f(x)
      ~=~ k(\ell f(x))
      ~=~ (k(\ell f)) (x),
    \end{equation*}
    so $(k\ell f) =k(\ell f)$.
  \item[(SM4)] Finally, we prove the rule for multiplication by one.
    For all $x\in X$, we have
    \begin{equation*}
      (1f) ( x) ~=~ 1f(x) ~=~f(x),
    \end{equation*}
    and therefore $1f=f$.
  \end{itemize}

  It follows that $\Func_{X,K}$ satisfies all the required axioms and is a
  vector space.
\end{proof}

For the next two examples of vector spaces, we leave the proofs as an
exercise.

\begin{example}{Infinite sequences}{vector-space-sequences}
  Let $K$ be a field. A \textbf{sequence}%
  \index{sequence} of elements of $K$ is an infinite list
  \begin{equation*}
    (a_0,\,a_1,\,a_2,\,a_3\, \ldots),
  \end{equation*}
  where $a_i\in K$ for all $i$. We also use the notation
  $(a_i)_{i\in\N}$, or occasionally $(a_i)$, to denote such a
  sequence. Let $\Seq_K$ be the set of sequences%
  \index{vector space!of sequences}%
  \index{SeqK@$\Seq_K$} of elements of $K$. We add two
  sequences by adding their $i\th$ elements:
  \begin{equation*}
    (a_i)_{i\in\N} + (b_i)_{i\in\N} = (a_i+b_i)_{i\in\N}.
  \end{equation*}
  We scale a sequence by scaling each of its elements:
  \begin{equation*}
    k(a_i)_{i\in\N} = (ka_i)_{i\in\N}.
  \end{equation*}
  Then $\Seq_K$ is a vector space.
\end{example}

\begin{example}{Vector space of polynomials of unbounded degree}{vector-space-polynomials}
  Let $K$ be a field, and let $\Poly$%
  \index{P@$\Poly$}%
  \index{vector space!of polynomials} be the set of all polynomials%
  \index{polynomial} (of any degree) with coefficients from $K$, i.e.,
  expressions of the form
  \begin{equation*}
    p(x) = a_nx^n + a_{n-1}x^{n-1} + \ldots + a_1x + a_0,
  \end{equation*}
  where $n\geq 0$ and $a_0,\ldots,a_n\in K$. Addition and scalar
  multiplication of polynomials are defined in the usual way
  \index{polynomial!addition}%
  \index{addition!of polynomials}%
  \index{polynomial!scalar multiplication}%
  \index{scalar multiplication!of polynomials}.
  Then $\Poly$ is a vector space.
\end{example}

We conclude this section by deriving some initial consequences of the
vector space axioms.

\begin{proposition}{Elementary consequences of the vector space axioms}{vector-space-elementary}
  In any vector space, the following are true:
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item The additive unit is unique. In other words, whenever
      $\vect{u}+\vect{v}=\vect{u}$, then $\vect{v}=\vect{0}$.
    \item Additive inverses are unique. In other words, whenever
      $\vect{u}+\vect{v}=\vect{0}$, then $\vect{v}=-\vect{u}$.
    \item\label{vector-space-elementary-c} $0\vect{u}=\vect{0}$ for
      all vectors $\vect{u}$.
    \item The following \textbf{cancellation law}%
      \index{cancellation law!of addition}%
      \index{vector!cancellation law}
      holds: if $\vect{u} + \vect{w} = \vect{v} + \vect{w}$, then
      $\vect{u} = \vect{v}$.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{proposition}

\begin{proof}
  We prove the first three properties, and leave the last one as an
  exercise. Assume $V$ is any vector space over a field $K$.
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item Consider arbitrary vectors $\vect{u},\vect{v}\in V$ and assume
      \begin{equation*}
        \vect{u}+\vect{v}=\vect{u}.
      \end{equation*}
      Applying the law (A1) (commutative law) to the left-hand side, we have
      \begin{equation*}
        \vect{v}+\vect{u}=\vect{u}.
      \end{equation*}
      Adding $-\vect{u}$ to both sides of the equation, we have
      \begin{equation*}
        (\vect{v}+\vect{u})+(-\vect{u}) = \vect{u}+(-\vect{u}).
      \end{equation*}
      Applying the law (A2) (associative law) to the left-hand side, we
      have
      \begin{equation*}
        \vect{v}+(\vect{u}+(-\vect{u})) = \vect{u}+(-\vect{u}).
      \end{equation*}
      Applying the law (A4) (additive inverse law) to both sides of the
      equation, we have
      \begin{equation*}
        \vect{v}+\vect{0} = \vect{0}.
      \end{equation*}
      Applying the law (A3) (additive unit law) to the left-hand side,
      we have
      \begin{equation*}
        \vect{v} = \vect{0}.
      \end{equation*}
      This proves that whenever $\vect{u}+\vect{v}=\vect{u}$, then
      $\vect{v} = \vect{0}$, or in other words, $\vect{v} = \vect{0}$ is
      the only element acting as an additive unit.

    \item Consider arbitrary vectors $\vect{u},\vect{v}\in V$ and assume
      \begin{equation*}
        \vect{u}+\vect{v}=\vect{0}.
      \end{equation*}
      Applying the law (A1) (commutative law) to the left-hand side, we
      have
      \begin{equation*}
        \vect{v}+\vect{u}=\vect{0}.
      \end{equation*}
      Adding $-\vect{u}$ to both sides of the equation, we have
      \begin{equation*}
        (\vect{v}+\vect{u})+(-\vect{u}) = \vect{0}+(-\vect{u}).
      \end{equation*}
      Applying the law (A2) (associative law) to the left-hand side, we
      have
      \begin{equation*}
        \vect{v}+(\vect{u}+(-\vect{u})) = \vect{0}+(-\vect{u}).
      \end{equation*}
      Applying the law (A4) (additive inverse law) to the left-hand
      side, we have
      \begin{equation*}
        \vect{v}+\vect{0} = \vect{0}+(-\vect{u}).
      \end{equation*}
      Applying the law (A1) (commutative law) to the right-hand side, we
      have
      \begin{equation*}
        \vect{v}+\vect{0} = -\vect{u} + \vect{0}.
      \end{equation*}
      Applying the law (A3) (additive unit law) to both sides of the
      equation, we have
      \begin{equation*}
        \vect{v} = -\vect{u}.
      \end{equation*}
      This proves that whenever $\vect{u}+\vect{v}=\vect{0}$, then
      $\vect{v} = -\vect{u}$, or in other words, $\vect{v} = -\vect{u}$
      is the only element acting as an additive inverse of $\vect{u}$.

    \item First, note that the scalar $0\in K$ satisfies the property
      $0+0=0$, by property (A3) of the definition of a field. Now let
      $\vect{u}\in V$ be any vector. Using the vector space law (SM2)
      (distributive law over scalar addition) and $0+0=0$, we have
      \begin{equation*}
        0\vect{u}+0\vect{u} = (0+0) \vect{u} = 0\vect{u}.
      \end{equation*}
      Next, we use a small trick: add $-(0\vect{u})$ to both sides of
      the equation. This gives
      \begin{eqnarray*}
        (0\vect{u}+0\vect{u})+(-(0\vect{u})) &=& 0\vect{u} + (-(0\vect{u})).
      \end{eqnarray*}
      Applying the additional laws (A2), (A4), and (A3), we have
      \begin{eqnarray*}
        0\vect{u}+(0\vect{u}+(-(0\vect{u}))) &=& 0\vect{u} + (-(0\vect{u})), \\
        0\vect{u} + \vect{0} &=& \vect{0}, \\
        0\vect{u} &=& \vect{0}.
      \end{eqnarray*}
      This proves that $0\vect{u} = \vect{0}$ holds for all vectors
      $\vect{u}$, as desired.

    \item This is left as an exercise.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{proof}


\section{Sums and intersections}

\begin{outcome}
  \begin{enumerate}
  \item Show that the sum of two subspaces is a subspace.
  \item Show that the intersection of two subspaces is a subspace.
  \end{enumerate}
\end{outcome}

We begin this section with a definition.

\begin{definition}{Sum and intersection}{sum-intersection}
Let $V$ be a vector space, and let $U$ and $W$ be subspaces of
$V$.
Then
\begin{enumerate}
\item $U+W = \set{\vect{u}+\vect{w} \mid \vect{u}\in U\mbox{ and } \vect{w}\in W}$ and is
called the sum of $U$ and $W$.

\item $U\cap W = \set{\vect{v} \mid \vect{v}\in U\mbox{ and } \vect{v}\in W}$ and is
called the intersection of $U$ and $W$.
\end{enumerate}
\end{definition}

Therefore the intersection of two subspaces is all the vectors shared by both. If there are no vectors shared by both subspaces, meaning that $U \cap W = \set{\vect{0} }$, the sum $U+W$ takes on a special name.

\begin{definition}{Direct sum}{direct-sum}
Let $V$ be a vector space and suppose $U$ and $W$ are subspaces of $V$ such that  $U \cap W = \set{\vect{0} }$. Then the sum of $U$ and $W$ is called the direct sum and is denoted $U \oplus W$.
\end{definition}

An interesting result is that both the sum $U + W$ and the intersection $U \cap W$ are subspaces of $V$.

\begin{example}{Intersection is a subspace}{intersection-subspace}
Let $V$ be a vector space and suppose $U$ and $W$ are subspaces. Then the intersection $U \cap W$ is a subspace of $V$.
\end{example}

\begin{solution}
By the subspace test, we must show three things:
\begin{enumerate}
\item $\vect{0} \in U \cap W$
\item For vectors $\vect{v}_1, \vect{v}_2 \in U \cap W, \vect{v}_1+\vect{v}_2 \in U \cap W$
\item For scalar $a$ and vector $\vect{v} \in U \cap W, a\vect{v} \in U \cap W$
\end{enumerate}

We proceed to show each of these three conditions hold.
\begin{enumerate}
\item
Since $U$ and $W$ are subspaces of $V$, they each contain $\vect{0}$. By definition of the intersection, $\vect{0} \in U \cap W$.

\item
Let  $\vect{v}_1, \vect{v}_2 \in U \cap W$,. Then in particular,  $\vect{v}_1, \vect{v}_2 \in U$. Since $U$ is a subspace, it follows that $ \vect{v}_1+\vect{v}_2 \in U$. The same argument holds for $W$. Therefore $\vect{v}_1+\vect{v}_2$ is in both $U$ and $W$ and by definition is also in $U \cap W$.

\item
Let $a$ be a scalar and $\vect{v} \in U \cap W$. Then in particular, $\vect{v} \in U$. Since $U$ is a subspace, it follows that $a \vect{v} \in U$. The same argument holds for $W$ so $a\vect{v}$ is in both $U$ and $W$. By definition, it is in $U \cap W$.
\end{enumerate}

Therefore $U \cap W$ is a subspace of $V$.
\end{solution}

It can also be shown that $U + W$ is a subspace of $V$.

We conclude this section with an important theorem on dimension.

\begin{theorem}{Dimension of sum}{dimension-sum}
Let $V$ be a vector space with subspaces $U$ and $W$. Suppose $U$ and $W$ each have finite dimension. Then $U + W$ also has finite dimension which is given by\[
\dim (U+W) = \dim(U) + \dim(W) - \dim (U \cap W)
\]
\end{theorem}

Notice that when $U \cap W = \set{\vect{0} }$, the sum becomes the direct sum and the above equation becomes
\[
\dim (U \oplus W) = \dim(U) + \dim(W)
\]


\subsection{Ordering}

\subsubsection{Ordering of the natural numbers}

For natural numbers we can say that number \(n\) preceeds number \(s(n)\). That is:

\(n\le s(n)\)

Similarly:

\(s(n)\le s(s(n))\)

From the transitive property we know that:

\(n\le s(s(n))\)

We can continue this to get:

\(n\le s(s(...s(n)..))\)

What can we say about an arbitary comparison?

\(a\le b\)

We know that either:

\begin{itemize}
\item \(a=b\)
\item \(b=s(s(...s(a)...))\)
\item \(a=s(s(...s(b)...))\)
\end{itemize}

In the first case the relation holds.

In the second case the relation holds.

In the third case the relation does not hold, but antisymmetry holds.

As this is is then defined on any pair, the order on natural numbers is total.

As there is a minimum, \(0\), the relation is also well-ordered.

However if this does not hold then the following instead holds:


\subsection{Enumerative combinatorics}\label{subsec:enumerative_combinatorics}

This subsection lists several results of various importance that don't really belong to any more consistent theory.

\begin{theorem}[Dirichlet's pigeonhole principle]\label{def:pigeonhole_principle}
  If we have more pigeons than pigeonholes, then at least one pigeonhole must contain multiple pigeons in it.

  More formally, if \( \card(A) > \card(B) \), then there exists no injective function from \( A \) to \( B \).
\end{theorem}
\begin{proof}
  This is a corollary of \fullref{thm:set_domination_relation_trichotomy}.
\end{proof}

\begin{definition}\label{def:binomial_coefficient}
  The \term{binomial coefficient} of the \hyperref[rem:peano_arithmetic_zero/positive]{positive integers} \( n \) and \( k \) is
  \begin{equation*}
    \binom n k \coloneqq \frac {n!} {k!(n-k)!}
  \end{equation*}

  They are motivated by \fullref{thm:binomial_theorem}.
\end{definition}

\begin{theorem}[Pascal's identity]\label{thm:pascals_identity}
  \begin{equation*}
    \binom n k = \binom {n - 1} k + \binom {n - 1} {k - 1}.
  \end{equation*}
\end{theorem}
\begin{proof}
  \begin{balign*}
    \binom {n - 1} k + \binom {n - 1} {k - 1}
    &=
    \frac {(n - 1)!} {k! (n - 1 - k)!} + \frac {(n - 1)!} {(k - 1)! (n - k)!}
    = \\ &=
    \frac {(n - 1)!} {(k - 1)! (n - 1 - k)!} \bracks*{ \frac 1 k + \frac 1 {n - k} }
    = \\ &=
    \frac {(n - 1)!} {(k - 1)! (n - 1 - k)!} \frac n {k(n - k)}
    = \\ &=
    \frac {n!} {k! (n - k)!}
    = \\ &=
    \binom n k.
  \end{balign*}
\end{proof}

\begin{definition}\label{def:factorial}
  The \term{factorial} of a \hyperref[rem:peano_arithmetic_zero/nonnegative]{nonnegative integer} \( n \) is defined recursively as
  \begin{equation*}
    n! \coloneqq \begin{cases}
      1,          &n = 0 \\
      (n - 1)! n, &n > 0.
    \end{cases}
  \end{equation*}
\end{definition}

\begin{proposition}\label{thm:gamma_function_interpolates_factorial}
  For every \hyperref[rem:peano_arithmetic_zero/nonnegative]{nonnegative integer} \( n \) we have
  \begin{equation*}
    \Gamma(n + 1) \coloneqq n!,
  \end{equation*}
  where \( \Gamma \) is the Gamma function defined in \fullref{def:gamma_function}.
\end{proposition}
\begin{proof}
  We use induction on \( n \).
  \begin{itemize}
    \item If \( n = 0 \), then
    \begin{equation*}
      \Gamma(1)
      =
      \int_0^\infty x^0 e^{-x} \dl x
      =
      -e^{-x}\restr_{x=0}^\infty
      =
      -\underbrace{\lim_{x \to \infty} e^{-x}}_{0} + 1
      =
      1
      =
      0!
    \end{equation*}

    \item If \( n > 0 \) and \( \Gamma(n) = (n - 1)! \), then
    \begin{balign*}
      \Gamma(n + 1)
      &=
      \int_0^\infty x^n \cdot e^{-x} \dl x
      = \\ &=
      \underbrace{(- x^n e^{-x})\restr_{x=0}^\infty}_{-(0 - 0)} + n \int_0^\infty e^{-x} x^{n-1} \dl x
      = \\ &=
      n \Gamma(n)
      = \\ &=
      n (n - 1)!
      = \\ &=
      n!
    \end{balign*}
  \end{itemize}
\end{proof}

\begin{theorem}[Stirling's factorial approximation]\label{thm:stirlings_factorial_approximation}
  For every \hyperref[rem:peano_arithmetic_zero/nonnegative]{nonnegative integer} \( n \) there exists some constant \( \theta \in (0, 1) \) such that
  \begin{equation*}
    n! = \sqrt{2 \pi n} \cdot \parens*{ \frac n e }^n \cdot e^{\frac 1 {12n + \theta}}.
  \end{equation*}
\end{theorem}
\begin{proof}
  Follows from \fullref{thm:gamma_function_interpolates_factorial} and \fullref{thm:stirlings_gamma_approximation}.
\end{proof}

\begin{remark}\label{rem:double_index_maps}
  We want to be able to map single indices to double indices and vice versa, for example for the purpose of \fullref{thm:matrix_spaces_are_tuple_spaces}. As an example, we want to be able to \enquote{linearize} an \( m \times n \) matrix such as the \( 2 \times 3 \) matrix
  \begin{equation}\label{eq:rem:double_index_maps/example/matrix}
    \begin{pmatrix}
      1 & 2 & 3 \\
      4 & 5 & 6
    \end{pmatrix}
  \end{equation}
  into the tuple
  \begin{equation}\label{eq:rem:double_index_maps/example/row_major}
    (1, 2, 3, 4, 5, 6)
  \end{equation}
  and vice versa. This is called \term{row-major order} of the elements of a matrix. The \term{column-major order} would instead be
  \begin{equation}\label{eq:rem:double_index_maps/example/column_major}
    (1, 4, 2, 5, 3, 6).
  \end{equation}

  Let \( m \) and \( n \) be \hyperref[rem:peano_arithmetic_zero/positive]{positive integers}. We will explicitly define functions for linearizing a matrix like \eqref{eq:rem:double_index_maps/example/matrix} into its row-major order \eqref{eq:rem:double_index_maps/example/row_major}. Define the sets
  \begin{align*}
    S &\coloneqq \overbrace{ \set{ 1, \ldots, mn - 1, mn } }^{\T{single indices}}
    \\
    D &\coloneqq \underbrace{ \set{ 1, \ldots, m } \times \set{ 1, \ldots, n } }_{\T{double indices}}
  \end{align*}
  and the mutually inverse operations
  \begin{align}
    &\begin{aligned}\label{eq:rem:double_index_maps/sharp}
      &\sharp: S \to D \\
      &\sharp(k) \coloneqq \parens[\Big]{ \quot(k - 1, m) + 1, \rem(k - 1, m) + 1 } \\
    \end{aligned}
    \\[0.5\baselineskip]
    &\begin{aligned}\label{eq:rem:double_index_maps/flat}
      &\flat: D \to S \\
      &\flat(i, j) \coloneqq (i - 1) \cdot m + (j - 1) + 1.
    \end{aligned}
  \end{align}

  The operation \( \sharp \) encodes the matrix \eqref{eq:rem:double_index_maps/example/matrix} into its row-major order \eqref{eq:rem:double_index_maps/example/row_major} and \( \flat \) does the opposite. Both operations are trivial except for the shifting needed in to allow us to use \hyperref[def:euclidean_domain]{remainders and quotients}.

  We can easily verify that \( \sharp \) is a \hyperref[def:morphism_invertibility/left_invertible]{left inverse} of \( \flat \) (note that \( j < m \)):
  \begin{align*}
    \sharp(\flat(i, j))
    &=
    \sharp\parens[\Big]{ (i - 1) \cdot m + (j - 1) + 1 }
    = \\ &=
    \parens[\Big]{ \quot(\cdots, m) + 1, \rem(\cdots, m) + 1 }
    = \\ &=
    \parens[\Big]{ (i - 1) + 1, (j - 1) + 1 }
    = \\ &=
    (i, j).
  \end{align*}

  We can just as easily verify that \( \flat \) is a \hyperref[def:morphism_invertibility/right_invertible]{right inverse} of \( \sharp \):
  \begin{align*}
    \flat(\sharp(k))
    &=
    \flat\parens[\Big]{ \quot(k, m) + 1, \rem(k, m) + 1 }
    = \\ &=
    \quot(k, m) \cdot m + \rem(k, m)
    = \\ &=
    k.
  \end{align*}

  Hence, \( \sharp \) is fully invertible with inverse \( \flat \). By \fullref{thm:function_invertibility_categorical/fully_invertible}, it is bijective.
\end{remark}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage[pdfborder={0 0 0}]{hyperref}
\newcommand\set[1]{\left\{#1\right\}}

\begin{document}
\section{Exercise 9}

\subsection{Straight}

This is five cards in a sequence (e.g., 4,5,6,7,8), with aces allowed to be either 1 or 13 (low or high) and with the cards allowed to be of the same suit (e.g., all hearts) or from some different suits. The number of such hands is $10 \cdot {4 \choose 1}^5$. The probability is 0.003940

\subsection{3 of a kind}

This hand has the pattern AAABC where A, B, and C are from distinct kinds. The number of such hands is ${13 \choose 1}{4 \choose 3}{12 \choose 2}{4 \choose 1}^2 = 54 912$. After dividing by ${52 \choose 5}$, the probability is $0.021128$.

\subsection{2 pair}

This hand has the pattern AABBC where A, B, and C are from distinct kinds. The number of such hands is ${13 \choose 2}{4 \choose 2}{4 \choose 2}{11 \choose 1}{4 \choose 1}$. After dividing by ${52 \choose 5}$, the probability is $0.047539$.

\subsection{Pair}

 This the hand with the pattern AABCD, where A, B, C and D are from the distinct "kinds" of cards: aces, twos, threes, tens, jacks, queens, and kings (there are 13 kinds, and four of each kind, in the standard 52 card deck). The number of such hands is ${13 \choose 1}\cdot{4 \choose 2}\cdot{12 \choose 3}\cdot {4 \choose 1}^3$. If all hands are equally likely, the probability of a single pair is obtained by dividing by ${52 \choose 5}$. This probability is 0.422569.

\subsection{High card}

We have to choose 5 distinct kinds ${13 \choose 5}$ but exclude any straights (subtract 10). We can have any pattern of suits except the 4 patterns where all 5 cards have the same suit: $4^5-4$. The total number of such hands is $[{13 \choose 5}-10] \cdot (4^5-4)$. The probability is $0.501177$.

\section{Whiteboard solution}

\[ \Omega = \set{\omega \subseteq \set{1, \dots, 52}\,|\, |\omega| = 5} \]

\begin{description}
  \item[Straight]
    \[ \mathbb P(\text{straight}) = \frac{10 \cdot 4^5}{{52 \choose 5}} - \frac{36}{{52 \choose 5}} - \frac{4}{{52 \choose 5}} = \frac{10200}{{52 \choose 5}} = 0.003925 \]
  \item[3 with same rank] $13$ different ranks.
    \[ {4 \choose 3} \text{ for color combinations} \]
    \[ {12 \choose 2} \text{ possibilities for remainder} \]
    \[ 4^2 \text{ color combinations} \]
    \[ \mathbb P(\text{3 of a kind}) = \frac{13 {4 \choose 3} {12 \choose 2} 4^2}{{52 \choose 5}} - \frac{54912}{{52 \choose 5}} \]
  \item[2-pair] $2\times 2$ cards of same rank
    \[ \mathbb P(\text{2-pair}) = \frac{{13 \choose 2} {4 \choose 2} {4 \choose 2} {11 \choose 1} \cdot 4}{{52 \choose 5}} \]
  \item[pair]
    13 for the rank, ${4 \choose 2}$ for colors ensures that we avoid duplicates
    \[ \mathbb P(\text{pair}) = \frac{13 {4 \choose 2} {12 \choose 3} 4^3}{|\Omega|} \]
  \item[high card]
    We can simply compute $1 - \text{(what we had so far)}$, but we will be exhaustive here:

    Possibilities for $5$ cards: ${13 \choose 5}$ \\
    -10 straights: $[{13 \choose 5} - 10]$ \\
    $4^5$ color combinations \\
    $-4$: $[4^5 - 4]$

    \[ \mathbb P(\text{high card}) = \frac{[{13 \choose 5} - 10] [4^5 - 4]}{{52 \choose 5}} \]
\end{description}

\end{document}


\chapter{Pseudo Random Number Generation}
\section{Producing binomial random variables}

The exercise is about different methods to produce pseudo random numbers with different distributions. One can do it in several ways to get pseudo random numbers. Most of methods use uniform distributed variables in some way or other. We don't consider the process of simulating uniformly distributed variables here. There are different methods, like Whichmann-Hill or the LCG, that are provided by R.

\noindent
One easy way to get random simulations of a specific distribution is by using uniformly distributed variables and the quantile transformation of the desired distribution. In the first part of the exercise we simulate binomial random variables using this idea. The second part is about the accept-reject method to simulate standard normal random variables.

The inversion method is based on the generalised inverse of the distribution function (cdf) $F$ which is considered to be defined on $\RR$ here and its inverse defined on $t \in (0, 1)$ as
$$F^{-1}(t) = inf\{x \in \RR | F(x) \geq t\}.$$
This, requires that the cdf is in this way analytical invertible. Additionally, if $U \sim unif[0, 1]$ holds, then $F^{-1}(U) \sim F$, i.e., it is possible to generate $F$-distributed variables using uniform variables.

In this exercise we were asked to simulate binomial random variables with parameters $n = 10$ and $p = 0.4.$ The inverse is easily computed as a sum of indicator functions over the intervals $[0,B(n,p)(\{k\})]$ for $k = 0, \dots 10.$ So we produced $1000$ binomials using the inverse of simulated uniforms. Another way to get binomial distributed variables is the summation of independent Bernoulli random variables with success
probability $p.$ Those can also be simulated using the inverse method with $F^{-1}(t) = \one_{[0,p)}(t).$ Therefore we used in another trial $10000$ uniform distributed variables that we inverted to Bernoulli variables and added up in the end. Finally we compared those
two approaches with the provided method \texttt{rbinom} which is based on a special accept-
reject-method. In the visualisation we can see very clear that all approaches lead to
a similar distribution (see Figure \ref{fig:emp_all}). Although the binomial distribution is discrete, I have added a smooth density estimate to compare the overall shape of the distributions.

\begin{figure}[th]
\centering
\includegraphics[scale = 0.7]{ex2/emp_all.png}
\caption{Empirical distribution of simulated binomial variables ($n=10, p=0.4$). Method 1: inverse method with the binomial quantile function; \\
Method 2: Summation of Bernoulli variables (simulated with uniform variables and inverse method);\\
Method 3: simulated with \texttt{rbinom}.}
\label{fig:emp_all}
\end{figure}

So we see that the first two method, i.e. those using inverse method just differ in a slightly more centred distribution for method $1$. The blue bins show a small shift of the distribution to the right most dominant at $5.$

Drawing a histogram of three groups into one plot is sometimes problematic because the bins are drawn shifted in a graphical way. This shift may suggest a shift in distribution that is not existent in the data. Therefore I have added another graphic (Figure \ref{fig:emp_all_col}) where I put the three histograms beneath each other. As the three distributions are very close, in general differences are hard to see, but one sees that method $1$ generates a more centred distribution and that the first row shows a slightly more right skewed distribution, as one would expect from a binomial with $p=0.4$. All in all the differences seem somehow negligible and we can assume that every method produced reasonable simulations.

\begin{figure}[thbp]
\centering
\includegraphics[scale = 0.8, keepaspectratio]{ex2/emp_all_col.png}
\caption{Histograms of simulated binomial variables with the three methods.}
\label{fig:emp_all_col}
\end{figure}

\newpage
\section{Accept-reject method for normal distribution}

The next task is to produce standard normal random variables with the accept-reject method. Therefore one has to be able to simulate variables from another distribution with known density. Here we use the standard Cauchy distribution with the density $g(x)=[\pi (1+x^2)]^{-1}$.\\

The general approach for generating variables with a density function $f$ by variables with a distribution given by a density $g$ demands a constant $c\geq 1$, s.t. $f(x)\leq cg(x)$ for all $x$ and is given by following algorithm:
\begin{enumerate}
\item generate random variable $X$ from density $g$
\item generate uniformly distributed $U\sim unif[0,1]$ (independent from $X$)
\item Accept $X$ as drawn from $f$, if $Ucg(X)\leq f(X)$, else reject.
\end{enumerate}

The resulting variables $X$ are then distributed with density $f$. In this exercise we produce the variables $X$ with the inversion method. The cdf of the standard Cauchy distribution is given by $$G(x)= (\arctan (x) - 0.5)/\pi$$ and therefore we get the inverse $$G^{-1}(t)=\tan \{\pi (y-0.5)\}.$$

To determine the best $c$ for the rejection procedure, we search for the smallest possible constant - as we don't want to reject unnecessary many $X$ so we have to draw less candidates and thus keep the algorithm fast - such that $c\geq \frac{f}{g}(x)$ for all $x$, i.e. $$c:= \sup_{x\in \mathbb{R}} \frac{f(x)}{g(x)}= \sup_{x\in \mathbb{R}} \frac{1}{\sqrt{2\pi}}\exp\{-x^2/2\} \pi (1+x^2).$$ By differentiation we get three local extrema and the maxima at $x\in \{-1,1\}$ resulting in $c=\sqrt{2\pi}e^{-1/2}$.

We simulated $10,000$ standard normals using this method after setting the random number generator back to default, i.e. using Mersenne-Twister. The result is shown in Figure \ref{2normhist}. There we can see a very good fit of the data.
\begin{figure}[!th]
\centering
\includegraphics[width=0.8\textwidth, keepaspectratio]{ex2/normal_hist.png}
\caption{Histogram of simulated standard normals. The orange line shows the density of a standard normal variable}
\label{2normhist}
\end{figure}

Additionally one can examine the normality of a sample for example with QQ- and PP- plots. The QQ-plot for the sample is shown in Figure \ref{2normqq}. The points lie pretty much on the reference line. Just the few highest quantiles differ. Since on the tails are just a few data points this may happen by accident. I omit other plots since a very good fit of the data is already obvious in the histogram and the QQ- plot.
\begin{figure}[bh]
\centering
\includegraphics[width=0.45\textwidth, keepaspectratio]{ex2/normal_qq.png}
\caption{QQ-plot of a simulated sample of standard normals.}
\label{2normqq}
\end{figure}

Because in this method we reject some samples of the distribution $g$, it is interesting to consider the acceptance probability. The theoretical overall acceptance probability should be $1/c =0.658$. In this simulation we used 15 168 Cauchy samples resulting giving a similar empirical acceptance probability of $0.659$. One can also look in more detail and visualise more local acceptance probabilities. The probability that the uniform variable drawn in the second step of the algorithm accepts the sample $X$ as drawn from $f$ is $$P(U\leq \frac{f(X)}{cg(X)})=\frac{f(X)}{cg(X)}$$ and can be seen as a function in $X$. I estimated the local empirical acceptance probabilities by binning the data with the \texttt{hist} function in R (see Figure \ref{2normacc}). Since the Cauchy distribution has very heavy tails compared to the standard normal, there are of course very high and low values in the Cauchy sample. I restricted the data for the visualisation to the area, where the empirical acceptance probability is positive (and not 0), i.e. where we have indeed data in the final normal sample. As expected we see in the plot that the empirical probabilities fluctuate around the theoretical ones over the whole range of values.
\begin{figure}[hbt]
\includegraphics[width=0.7\linewidth, keepaspectratio]{ex2/acc_probs.png}
\centering
\caption{Theoretical and empirical acceptance probabilities of the normal sample generated with the accept-reject method.}
\label{2normacc}
\end{figure}

The last task of the exercise is to check, whether it is possible to simulate Cauchy distributed samples from a standard normal one. As mentioned the Cauchy distribution hat heavy tails compared to the normal. This can be also seen in the quotient of densities. $$\frac{g(x)}{f(x)}=\sqrt{2\pi}exp\{x^2/2\}[\pi (1+x^2)]^{-1}$$ This is obviously unbounded as $x$ tends to infinity, i.e. we can't find a suitable constant $c$ for the algorithm. This can also be seen in Figure \ref{2normacc}. As the limit of the function is 0 the inverse has to go to infinity.


\subsection{Estimator error and bias}

\subsubsection{Error of an estimator}

The error of an estimator is the difference between it and the actual parameter.

\(Error_{\theta }[\hat \theta ]=\hat \theta - \theta \)

\subsubsection{Bias of an estimator}

The bias of an estimator is the expected error.

\(Bias_\theta [\hat \theta ]:=E_\theta [\hat \theta -\theta ]\)

\(Bias_\theta [\hat \theta ]:=E_\theta [\hat \theta] -\theta\)


\subsubsection{The Laplacian}
The Laplacian is the higher dimension version of concavity. Let $f$ be a scalar function, We say that $\nabla^2 f = \nabla \cdot (\nabla f) = \text{div} (\text{grad}(f))$. Written out more fully,
\begin{equation*}
	\nabla^2 f = f_{xx} + f_{yy} + \ldots
\end{equation*}


\chapter{Sentential Logic}

\section{Grammar}
\label{sec:Gramma}

\subsection{Symbols}
\label{sub:Symbols}
\begin{itemize}
    \item Logical Symbols
    \begin{itemize}
        \item Sentential Connectives
        \begin{itemize}
            \item $\neg$
            \item $\wedge$
            \item $\vee$
            \item $\rightarrow$
            \item $\leftrightarrow$
        \end{itemize}
        \item Parentheses
    \end{itemize}
    \item Non-logical Symbols: An enumerable set of elements
\end{itemize}

\subsection{Expressions}
\label{sub:Expressions}

\begin{definition}[Expression]
    \label{def:Expression}
    An expression is a finite sequence of symbols.
\end{definition}
\begin{remark}
    The set of all expressions is enumerable.
\end{remark}

We often use Greek alphabets $\alpha,\beta,\dots$ to represent expressions.

\subsection{Well-Formed Formulas}
\label{sub:WellFormedFormulas}

\begin{definition}[Well-Formed Formula]
    \label{def:WFF}
    A \textbf{well-formed formula} (or formula or wff) is an expression built up from sentence symbols by applying some finite times of \emph{formula building operations}
\end{definition}

\begin{definition}[Formula Building Operations]~{}
    \begin{itemize}
        \item $\xi_{\neg}(\alpha) = (\neg\alpha)$
        \item $\xi_{\wedge}(\alpha,\beta) = (\alpha \wedge \beta)$
        \item $\xi_{\vee}(\alpha,\beta) = (\alpha\vee\beta)$
        \item $\xi_{\rightarrow}(\alpha,\beta) = (\alpha\rightarrow\beta)$
        \item $\xi_{\leftrightarrow}(\alpha,\beta) = (\alpha\leftrightarrow\beta)$
    \end{itemize}
\end{definition}
\begin{remark}
    Do NOT omit the parentheses.
\end{remark}

\begin{definition}[Well-Formed Sequences of Expressions]
    \label{def:WellFormedSeqOfExpr}
    A \textbf{well-formed sequence of expressions} is a finite sequence $\alpha_1,\alpha_2,\dots,\alpha_n$ of expressions such that each $\alpha_i$ is either
    \begin{itemize}
        \item A sentence symbol
        \item $(\neg\alpha_j)$ for some $j < i$
        \item $(\alpha_j \wedge \beta_k)$ for some $j,k<i$
        \item $(\alpha_j \vee \beta_k)$ for some $j,k<i$
        \item $(\alpha_j \leftarrow \beta_k)$ for some $j,k<i$
        \item $(\alpha_j \leftrightarrow \beta_k)$ for some $j,k<i$
    \end{itemize}
\end{definition}

\begin{proposition}
    An expression $\alpha$ is a well-formed formula iff there is a well-formed sequence $(\alpha_1,\dots,\alpha_n)$ s.t. $\alpha = \alpha_n$
\end{proposition}

\subsection{The Induction Principle}
\label{sub:Induction}

Well-formed formulas are a form of inductive definitions with
\begin{itemize}
    \item Basic building blocks
    \item Closing operations
\end{itemize}

\begin{theorem}[The Induction Principle]
    \label{thm:InductionPrinciple}
    Let $S$ be a set of wffs ($S \subseteq W$), if
    \begin{enumerate}
        \item Every sentence symbol is in $S$
        \item For each wff $\alpha$ and $\beta$, if $\alpha$ and $\beta$ are in $S$ then each of the following are in $S$
        \begin{itemize}
            \item $(\neg \alpha)$
            \item $(\alpha \wedge \beta)$
            \item $(\alpha \vee \beta)$
            \item $(\alpha \rightarrow \beta)$
            \item $(\alpha \leftrightarrow \beta)$
        \end{itemize}
    \end{enumerate}
    Then $S$ is the set of \emph{all wffs} ($S = W$).
\end{theorem}

We can see an example of Indunction.

\begin{proposition}
    Every wff has the same number of left parentheses as right parenthesis
\end{proposition}
\begin{proof}
    Let $S\triangleq \{\alpha|\alpha\text{has equal number of left and right parentheses}\}$.
    \begin{itemize}
        \item[Base] $\alpha = A$. Straightforward. Sentense symbols do not have parenthesis
        \item[Step]
        \begin{enumerate}
            \item Let $\beta \in S$, $\alpha = (\neg \beta) \in S$.
            \item Let $\alpha_1, \alpha_2 \in S$, $\alpha = (\alpha_1 \wedge \alpha_2) \in S$.
            \item $\cdots$
        \end{enumerate}
    \end{itemize}
\end{proof}

\subsection{Parsing Formulas}
\label{sub:ParsingFormulas}

The induction principle actually gives an algorithm for parsing formulas.

On input expression $\alpha$

\begin{enumerate}
    \item If is leaf node, we are done. Return.
    \item The first symbol must be `('.
    \item If the second symbol is `$\neg$', then expect an non-empty expression $\beta$ and parse $\beta$.
    \item If the second symbol is not `$\neg$', then expect a non-empty expression $\beta_1$, an operator and another expression $\beta_2$.
\end{enumerate}

\subsection{Abbreviations}
\label{sub:Abbreviations}

\begin{itemize}
    \item The outermost parentheses can be omitted.
    \item $\neg$ appplies to as little as possible, with the highest precedence.
    \item $\wedge$ and $\vee$ apply to as little as possible, subject to $\neg$.
    \item $\rightarrow$ and $\leftrightarrow$ apply to as little as possible, subject to other operators.
    \item When handling operators with the same precedence, grouping is always to the right. $A \rightarrow B \rightarrow C= (A \rightarrow (B\rightarrow C))$.
\end{itemize}

\section{Semantics}
\label{sec:Semantics}

\subsection{Truth Assignments}

Consider a math domain $\{T,F\}$ of truth values

\begin{itemize}
    \item T is called truth
    \item F is called falsity
\end{itemize}

\begin{definition}[Truth Assignment]
    A truth assignment for a set $\mathcal{S}$ of sentence symbols is a function
    \[ v:\mathcal{S}\mapsto\{T,F\} \]
\end{definition}

\begin{definition}[Extended Truth Assignment]
    Let $\bar{\mathcal{S}}$ be the set of wffs that can be built up from $\mathcal{S}$ by formula-building operations. Let $v$ be a truth assignment for $\mathcal{S}$. An \textbf{extension} $\bar{v}$ of $v$
    \[ \bar{v}:\bar{\mathcal{S}}\mapsto \{T,F\} \]
    assigns truth values to every wff in $\mathcal{S}$ s.t.
    \begin{itemize}
        \item $\bar{v}(\alpha) = v(\alpha)$ if $\alpha \in \mathcal{S}$
        \item $\bar{v}(\neg(\alpha))$ is T if $\bar{v}(\alpha)$ is F and F otherwise.
        \item $\bar{v}((\alpha\wedge\beta))$ is T if $\bar{v}(\alpha)$ is T and $\bar{v}(\beta)$ is T and F otherwise.
        \item $\bar{v}((\alpha\vee\beta))$ is T if $\bar{v}(\alpha)$ is T or $\bar{v}(\beta)$ is T and F otherwise.
        \item $\bar{v}((\alpha\to\beta))$ is F if $\bar{v}(\alpha)$ is T and $\bar{v}(\beta)$ is F and T otherwise.\footnote{Emphasizes the promise of a condition implying a consequence. If the condition is falsy then no guarantee for the consequence. \emph{“骗你是小狗”}}
        \item $\bar{v}((\alpha\leftrightarrow\beta))$ is T if $\bar{v}(\alpha) = \bar{v}(\beta)$ and is F otherwise.
    \end{itemize}
\end{definition}

\begin{theorem}[Determinacy of Truth Assignments]
    \label{thm:DeterminacyofTruthAssignments}
    For every $v_1$ and $v_2$ and wff $\alpha$, if
    \[ v_1(A) = v_2(A) \]
    for every sentence symbol that occurs in $\alpha$, then
    \[ \bar{v}_1(\alpha) = \bar{v}_2(\alpha) \]
\end{theorem}

\begin{remark}
    To determine the value of $\bar{v}(\alpha)$, we only need to know the value of $v$ on the sentence symbols that occur in $\alpha$. This leads to the method of \textbf{truth tables}.
\end{remark}

\subsection{Satisfiability}
\label{sub:Satisfiability}

We first introduce some new notations. We use captial Greek letters, $\Delta$, $\Sigma$, etc. to represent sets of wffs. And we use $\Sigma;\alpha$ to represent $\Sigma \cup \{\alpha\}$.

\begin{definition}~{}
    \begin{itemize}
        \item $v$ satisfies $\alpha$ if $\bar{v}(\alpha) = T$
        \item $v$ satisfies $\Sigma$ if $\bar{v}(\alpha) = T$ for every $\alpha \in \Sigma$.
    \end{itemize}
\end{definition}

\begin{definition}[Satisfiability]~{}
    \label{def:Satisfiability}
    \begin{itemize}
        \item $\alpha$ is satisfiable if there exists some $v$ that satisfies $\alpha$
        \item $\Sigma$ is satisfiable if there exists some $v$ that satisfies $\Sigma$
    \end{itemize}
\end{definition}

\begin{remark}
    Every $v$ satisfies $\emptyset$. Because $v$ satisfies $\emptyset$ iff
    \[ \forall \alpha, \alpha\in\emptyset \Longrightarrow \bar{v}(\alpha) = T \]
    The assumption itself is false, and therefore the consequence is always true.
\end{remark}

\subsection{Semantic Implications}
\label{sub:SemanticImplications}

\begin{definition}
    A set of wffs $\Sigma$ semantically implies $\alpha$ when every truth assignment satisfying $\Sigma$ also satisfies $\alpha$.
    \begin{itemize}
        \item $\Sigma \vDash \alpha$ denotes that $\Sigma$ implies $\alpha$
        \item $\alpha \vDash \beta$ denotes that $\{\alpha\} \vDash \beta$
    \end{itemize}
    If $\Sigma \vDash \alpha$, we call $\alpha$ a semantic consiquence of $\Sigma$.
\end{definition}

Semantic implication is also referred to as tautological implication.

\begin{remark}
    Note that $\{\alpha,\neg\alpha\} \vDash \beta$ also holds, because the assumption does not hold, so the consequence trivially holds.
\end{remark}

\subsection{Tautologies}
\label{sub:Tautologies}

\begin{definition}[Tautologies]
    \label{def:Tautology}
    $\alpha$ is a tautology if $\emptyset \vDash \alpha$, denoted by $\vDash \alpha$.
\end{definition}
\begin{remark}
    ~{}
    \begin{itemize}
        \item $\alpha$ is a tautology iff $\forall v$, $\bar{v}(\alpha)=T$.
        \item $\alpha$ is a tautology iff $\neg \alpha$ is \emph{not satisfiable}
        \item $\alpha$ is satisfiable iff $\neg \alpha$ is not a tautology.
    \end{itemize}
\end{remark}

\subsection{Semantic Equivalence}
\label{sub:SemanticEquivalence}

\begin{definition}[Semantic Equivalence]
    Two wffs $\alpha$ and $\beta$ are semantically equivalent if both $\alpha\vDash\beta$ and $\beta\vDash\alpha$ hold. We use $\alpha\vDash\Dashv\beta$
\end{definition}

\begin{proposition}
    The following are equivalent
    \begin{itemize}
        \item $\alpha$ and $\beta$ are semantically equivalent
        \item For every $v$, $\bar{v}(\alpha)=\bar{v}(\beta)$
        \item $\alpha$ and $\beta$ have the same truth table
    \end{itemize}
\end{proposition}

We can use semantic equivalence to derive truthfulness of wffs. If $\alpha\vDash\models\beta$, we can freely exchange one for the other in deriving the truth of some formula $\sigma$ where $\alpha$ and/or $\beta$ occur.

Remember not to mix syntax and semantics

\begin{itemize}
    \item $\alpha=T$ is incorrect. Use $\bar{v}(\alpha)=T$.
    \item $v(\Sigma)=T$ is incorrect. Use $v$ satisfies $\Sigma$.
\end{itemize}

\subsection{Properties of Satisfaction and Implication}

\begin{itemize}
    \item If $\alpha$ is a tautology, then $\Sigma\vDash\alpha$ for every $\Sigma$
    \item If $\alpha\in\Sigma$ then $\Sigma\vDash\alpha$
    \item If $\Sigma\vDash\alpha$ and $\Sigma\vDash\alpha\to\beta$ then $\Sigma\vDash\beta$
    \item If $\Sigma\vDash\alpha$ and $\alpha\vDash\beta$ then $\Sigma\vDash\beta$.
    \item If $\Sigma\vDash\alpha$ then for all $\beta$, $\Sigma\vDash\beta\to\alpha$
    \item If $\Sigma\vDash\alpha$ and $\Sigma\vDash\beta$ then $\Sigma\vDash \alpha\wedge\beta$
    \item If $\Sigma\vDash\alpha$ or $\Sigma\vDash\beta$ then $\Sigma\vDash \alpha\vee\beta$
    \item $\Sigma\nvDash\alpha$ iff $\Sigma\cup\{\neg\alpha\}$ is satisfiable
    \item $\Sigma \vDash \alpha$ iff $\Sigma \cup \{\neg \alpha\}$ is not satisfiable
    \item $\Sigma \vDash \alpha \to \beta$ iff $\Sigma;\alpha \vDash \beta$
    \item If $\Sigma$ is not satisfiable, then for every $\alpha$, $\Sigma\vDash\alpha$
    \item If $\Sigma\vDash\alpha$ and $\Sigma\subseteq\Delta$ then $\Delta\vDash\alpha$
    \item If $\Sigma$ is satisfiable then every subset of $\Sigma$ is satisfiable
    \item If every subset of $\Sigma$ is satisfiable then $\Sigma$ is satisfiable.
    \item If every finite subset of $\Sigma$ is satisfiable then $\Sigma$ is satisfiable
    \item If $\Sigma\vDash\alpha$ then there is a finite subset $\Delta$ of $\Sigma$ such that $\Delta\vDash\alpha$
\end{itemize}

\section{Normal Forms}
\label{sec:NormalForms}

\subsection{Disjunctive Normal Forms}
\label{sub:DisjunctiveNormalForms}

\begin{definition}[Disjunctive Normal Form]
    \label{def:DisjunctiveNormalForm}
    The wff $\alpha$ is in \textbf{disjunctive normal form} if $\alpha=\gamma_1\vee\gamma_2\vee\cdots\vee\gamma_k$ where each $\gamma_i$ is a conjunction
    \[ \gamma_i = \beta_{i1}\wedge\beta_{i2}\wedge\cdots\wedge\beta_{in_i} \]
    where each $\beta_{ij}$ is either a sentence symbol or the negation of a sentence symbol
\end{definition}

\subsection{Conjunctive Normal Forms}
\label{sub:ConjunctiveNormalForms}

\begin{definition}[Conjunctive Normal Form]
    \label{def:ConjunctiveNormalForm}
    The wff $\alpha$ is in \textbf{conjunctive normal form} if $\alpha=\gamma_1\wedge\gamma_2\wedge\cdots\wedge\gamma_k$ where each $\gamma_i$ is a disjunction
    \[ \gamma_i = \beta_{i1}\vee\beta_{i2}\vee\cdots\vee\beta_{in_i} \]
    where each $\beta_{ij}$ is either a sentence symbol or the negation of a sentence symbol
\end{definition}

\subsection{Completeness of Normal Forms}

\begin{theorem}[Completeness of DNFs]
    \label{thm:CompletenessOfDNF}
    Every wff is semantically equivalent to a wff in disjunctive normal form
\end{theorem}
\begin{proof}
    \begin{enumerate}
        \item Construct the disjunctive normal form truth tables
        \item Select all assignments $v$ s.t. $\bar{v}(\alpha)=T$
        \item Construct $\gamma_i$ where $\beta_{ij} = A_j$ if $A_j$ is assigned $T$ and $\beta_{ij} = \neg A_j$ otherwise
    \end{enumerate}
\end{proof}

\begin{theorem}[Completeness of CNFs]
    \label{thm:CompletenessOfCNF}
    Every wff is semantically equivalent to a wff in conjunctive normal form
\end{theorem}
\begin{proof}
    Given $\alpha=\gamma_1\vee\cdots\vee\gamma_n$ in DNF. If every $\gamma_i$ is a sentence symbol or the negation of a sentence symbol, we are done. Otherwise, there is some $\gamma_i = \beta_{i1}\wedge\beta_{i2}$. Then
    \[ \alpha \equiv (\beta_{i1} \wedge \beta_{i2}) \vee \alpha' \equiv (\beta_{i1}\vee\alpha') \wedge (\beta_{i2}\vee\alpha') \]
    where $\alpha'$ is the disjunction of $\{\gamma_k|k \neq i\}$

    And we recursively repeat the steps
\end{proof}

\section{Finite Satisfiability}
\label{sec:FiniteSatisfiability}

\begin{definition}[Finite Satisfiability]
    \label{def:FiniteSatisfiability}
    $\Sigma$ is \textbf{finitely satisfiable} if every finite subset of $\Sigma$ is satisfiable.
\end{definition}
\begin{remark}
    Suppose $\Delta$ is finitely satisfiable, and for every $\alpha$, $\alpha\in\Delta$ or $\neg\alpha\in\Delta$

    Then $\alpha\in\Delta$ iff $\neg\alpha\notin\Delta$
\end{remark}

\subsection{Compactness Theorem}

\begin{theorem}[Compactness Theorem]
    \label{thm:CompactnessTheorem}
    If $\Sigma$ is finitely satisfiable, then $\Sigma$ is satisfiable
\end{theorem}
\begin{sketchproof}
    We break down the proof into the following steps
    \begin{enumerate}
        \item From $\Sigma$, construct its superset $\Delta$ s.t.
        \begin{enumerate}
            \item $\Delta$ is finitely satisfiable
            \item For every wff $\alpha$, $\alpha\in\Delta$ or $\neg\alpha\in\Delta$
        \end{enumerate}
        \item Show that $\Delta$ is satisfiable, so that $\Sigma$ is also satisfiable
    \end{enumerate}
\end{sketchproof}

\begin{lemma}
    \label{lem:CompactnessLemma1}
    If $\Delta$ is finitely satisfiable, then for every wff $\alpha$,
    \begin{itemize}
        \item either $\Delta\cup\{\alpha\}$ is finitely satisfiable
        \item or $\Delta\cup\{\neg\alpha\}$ is finitely satisfiable
    \end{itemize}
\end{lemma}
\begin{proof}
    Prove by contradiction. Suppose neither $\Delta\cup\{\alpha\}$ nor $\Delta\cup\{\neg\alpha\}$ is finitely satisfiable. Then there are some finite subsets $\Delta_1 \subseteq \Delta\cup\{\alpha\}$ and $\Delta_2\subseteq\Delta\cup\{\neg\alpha\}$ which are not satisfiable. Notice that $\alpha$ must be in $\Delta_1$ and $\Delta_2$, or otherwise $\Delta_1$ and $\Delta_2$ would be satisfiable by finite satisfiability of $\Delta$. Therefore $\Delta_1 = \Delta'_1\cup\{\alpha\}$ and $\Delta_2'\cup\{\neg\alpha\}$.

    Then we construct $\Delta' = \Delta_1\cup\Delta_2 = \Delta_1'\cup\Delta_2'\cup\{\alpha,\neg\alpha\}$. Then there exists an assignment $v$ such that $v$ satisfies $\Delta_1'\cup\Delta_2'$, and that $v$ satisfies either $\alpha$ or $\neg\alpha$. Suppose $\bar{v}(\alpha)=T$, then $\Delta_1$ is satisfiable, which causes contradiction. Conversely, if $\bar{v}(\alpha) = F$ then $\bar{v}(\neg\alpha)=T$, then $\Delta_2$ is satisfiable, which is also a contradiction.
\end{proof}
\begin{remark}
    This lemma implies that we can expand $\Sigma$ for one step, by including $\alpha$ or $\neg\alpha$
\end{remark}

\begin{lemma}
    \label{lem:CompactnessLemma2}
    If $\Sigma$ is finitely satisfiable, then there is a $\Delta \supseteq \Sigma$ such that
    \begin{itemize}
        \item $\Delta$ is finitely satisfiable
        \item For each wff $\alpha$, $\alpha\in\Delta$ or $\neg\alpha\in\Delta$
    \end{itemize}
\end{lemma}
\begin{proof}
    The set of all wffs is enumerable, so we can write all wffs in a sequence
    \[ \alpha_1, \alpha_2,\dots,\alpha_n,\dots \]
    Then we can scan through the sequence and check if $\alpha_i$ can be added to $\Sigma$. Starting from $\Delta_0 = \Sigma$,
    \[\Delta_{i+1}\begin{cases}
        \Delta_i \cup \{\alpha_i\} &\quad\text{if it is finitely satisfiable}\\
        \Delta_i \cup \{\neg \alpha_i\} &\quad \text{otherwise}
    \end{cases}\]

    It can be proved by induction and Lemma~\ref{lem:CompactnessLemma1} that $\forall i,\Delta_i$ is finitely satisfiable.

    And $\Delta$ can be constructed by the union of all $\Delta_i$'s

    \[ \Delta = \bigcup_{i\in\mathbb{N}} \Delta_i \]

    $\Delta$ is finitely satisifiable because for each $\Delta'\subseteq\Delta$, $\Delta'\subseteq\Delta_i$ for some $\Delta_i$. Since $\Delta_i$ is finitely sat, $\Delta'$ is also satisfiable, and therefore $\Delta$ is finitely sat.

    For each wff $\alpha$, either $\alpha\in\Delta$ or $\neg\alpha\in\Delta$. Because $\alpha$ must exist as $\alpha_i$ in the sequence of all wffs, then either $\alpha_i\in\Delta_{i+1}$ or $\neg\alpha_i\in\Delta_{i+1}$.
\end{proof}

\begin{lemma}
    \label{lem:CompactnessLemma3}
    Let $\Delta$ be a set of wffs such that
    \begin{itemize}
        \item $\Delta$ is finitely satisifiable
        \item For every wff $\alpha$, $\alpha\in\Delta$ or $\neg\alpha\in\Delta$
    \end{itemize}
    Then $\Delta$ is satisfiable
\end{lemma}
\begin{proof}
    Consider the sentence symbols. All sentence symbols (or their negations) must be in $\Delta$ because they are also wffs. Therefore if there is an assignment $v$ that satisfies $\Delta$, its values is already determined by the sentence symbols in $\Delta$.

    \[v(A)=\begin{cases}
        T &\quad A\in\Delta\\
        F &\quad \neg A\in\Delta
    \end{cases}\]

    Then proving Lemma~\ref{lem:CompactnessLemma3} is equivalent to proving
    \[ \forall \alpha, \alpha\in\Delta\Leftrightarrow\bar{v}(\alpha) = T \]
    This can be proved by induction
    \begin{itemize}
        \item[base] Consider $\alpha=A$. Obviously it holds.
        \item[induction] \begin{enumerate}[(a)]
            \item $\alpha=\neg\beta$. The hypothesis is $\beta\in\Delta \Leftrightarrow \bar{v}(\beta)=T$. If $\neg\beta\in\Delta$, then $\beta$ cannot be in $\Delta$ due to the finite satisfiability of $\Delta$, and therefore $\bar{v}(\beta) = F$, and therefore $\bar{v}(\neg\beta) = T$. Conversely, If $\bar{v}(\neg\beta) = T$, then $\bar{v}(\beta) =F$, and therefore $\beta\notin\Delta$ and thus $\neg\beta$ must be in $\Delta$.
        \end{enumerate}
    \end{itemize}
\end{proof}

\begin{corollary}
    \label{coroll:CorollaryOfTheCompactnessTheorem}
    If $\Sigma\vDash\tau$, then there is a finite subset $\Delta$ of $\Sigma$ such that $\Delta\vDash\tau$
\end{corollary}
\begin{proof}
    Assume for every subset $\Delta$ of $\Sigma$, $\Delta \nvDash \tau$. Then $\Delta;\tau$ is satisfiable. So $\Sigma;\tau$ is finitely satisfiable, and by compactness theorem we have $\Sigma;\tau$ is satisfiable, which implies $\Sigma\nvDash\tau$, and this leads to a contradiction.
\end{proof}

\subsection{Decidability Results for Semantic Implications}

\begin{theorem}
    Given any finite set $\Sigma$ of wffs and any wff $\alpha$, there is an algorithm for deciding whether or not $\Sigma\vDash\alpha$.
\end{theorem}
\begin{enumerate}
    \item Collect all sentence symbols in $\Sigma$ and $\alpha$
    \item Use truth table
\end{enumerate}

\begin{corollary}
    Given a finite set of wffs $\Sigma$, the set of its semantic consequences is effectively decidable. In particular, the set of tautologies is effectively decidable.
\end{corollary}

\subsection{Enumerability Results for Semantic Implications}

\begin{theorem}
    If $\Sigma$ is an effectively enumerable set of wffs, then the set of semantic consequences of $\Sigma$ is effectively enumerable.
\end{theorem}
\begin{proof}
    Let $\beta_1,\dots,\beta_n,\dots$ be an effective enumeration of $\Sigma$. Let $\Delta_n=\beta_1,\dots,\beta_n$. Let $\alpha_1,\dots,\alpha_m,\dots$ be an effective enumeration of all wffs. We construct a table $T$ where $T_{ij} = \Delta_i \vDash \alpha_j$. Due to Corollary~\ref{coroll:CorollaryOfTheCompactnessTheorem}, if some $T_{ij}$ holds, then $\alpha_j$ is a semantic consequence of $\Sigma$.
\end{proof}


\subsection{Comparing functions}
\textit{CLRS section 3.1 page 51}
For the following, assume that $ f(n) $ and $ g(n) $ are asymptotically positive.
\subsubsection{Transitivity}
\begin{alignat*}{11}
	&f(n) &&= \Theta(g(n)) &\quad\text{and}\quad g(n) &&= \Theta(h(n)) &\quad\text{imply}\quad f(n) &&= \Theta(h(n))\\
	&f(n) &&= O(g(n)) 	&\quad\text{and}\quad g(n) &&= O(h(n)) &\quad\text{imply}\quad f(n) &&= O(h(n))\\
	&f(n) &&= \Omega(g(n)) 	&\quad\text{and}\quad g(n) &&= \Omega(h(n)) &\quad\text{imply}\quad f(n) &&= \Omega(h(n))\\
	&f(n) &&= o(g(n)) 	&\quad\text{and}\quad g(n) &&= o(h(n)) &\quad\text{imply}\quad f(n) &&= o(h(n))\\
	&f(n) &&= \omega(g(n)) 	&\quad\text{and}\quad g(n) &&= \omega(h(n)) &\quad\text{imply}\quad f(n) &&= \omega(h(n))
\end{alignat*}

\subsubsection{Reflexivity}
\begin{align*}
	f(n) &= \Theta(f(n))\\
	f(n) &= O(f(n))\\
	f(n) &= \Omega(f(n))
\end{align*}

\subsubsection{Symmetry}
$$ f(n)=\Theta(g(n)) \text{ if and only if } g(n)=\Theta(f(n)) $$

\subsubsection{Transpose symmetry}
\begin{alignat*}{3}
	&f(n) &&= O(g(n)) \quad\text{if and only if}\quad  g(n) &&= \Omega(f(n))\\
	&f(n) &&= o(g(n)) \quad\text{if and only if}\quad  g(n) &&= \omega(f(n))
\end{alignat*}

\subsection{The engineering way}
Ignore its leading constants: $ T(n)=1000n^5 = \Theta(n^5) $.\\
Drop its low order terms: $ T(n)=n^5+n^3+\lg n = \Theta(n^5) $\\
\textbf{Identifying lower order terms}
\[
\begin{array}{ccccccc}
	\text{Constant} &<& \text{Poly-logarithm} &<& \text{Polynomial} &<& \text{Exponential}\\
	c &<& \lg^kn &<& n^a &<& b^n
\end{array}
\]

Algorithms and Data Structures - Reference/asymptotic.tex


\section{The \innerproduct algorithm}
\Label{sec:innerproduct}

The \innerproduct algorithm in the \cxx Standard Library \cite[\S 29.8.4]{cxx-17-draft} computes
the \emph{inner product}\footnote{
  Also referred to as \emph{dot product}, see \url{http://en.wikipedia.org/wiki/Dot_product}
}
of two ranges.
%
Our version of the original signature
reads:

\begin{lstlisting}[style=acsl-block]

  value_type
  inner_product(const value_type* a, const value_type* b,
                size_type n, value_type init);
\end{lstlisting}

The result of \innerproduct equals the value
\begin{gather*}
\mathtt{init} + \sum_{i = 0}^{\mathtt{n}-1} \mathtt{a}[i] \cdot \mathtt{b}[i]
\end{gather*}
thus, \innerproduct will return \inl{init} for empty ranges.

%\clearpage

\subsection{The logic function \InnerProduct}

As in the case of \specref{accumulate} we specify \innerproduct
by defining in the following listing the logic function \InnerProduct
that formally expresses the summation of the element-wise product of two arrays.

Predicate \logicref{ProductBounds} expresses that for $0 \leq i < n$ the products
%
\begin{align}
\Label{eq:innerproduct1}
\mathtt{a}[i] \cdot \mathtt{b}[i]
\end{align}
%
do not overflow.
Predicate \logicref{InnerProductBounds}, on the other hand, states that for $0 \leq i < n$
the following sums do not overflow.
cc%
\begin{align}
\Label{eq:innerproduct2}
\mathtt{init} + \sum_{k = 0}^{\mathtt{i}} \mathtt{a}[k] \cdot \mathtt{b}[k]
\end{align}

Otherwise, one cannot guarantee that the result of our implementation
of \implref{innerproduct} equals the mathematical description of \InnerProduct.
Finally, Lemma \logicref{InnerProductUnchanged} states that the result of the \InnerProduct only
depends on the values of \inl{a[0..n-1]} and \inl{b[0..n-1]}.

\input{Listings/InnerProduct.acsl.tex}

\subsection{Formal specification of \innerproduct}

Using the logic function \logicref{InnerProduct}, we specify \innerproduct as shown
in the following listing.
Note that we needn't require that \inl{a} and \inl{b} are separated.

\input{Listings/inner_product.h.tex}

\clearpage

\subsection{Implementation of \innerproduct}

The following listing shows an implementation of \innerproduct
with corresponding loop annotations.

\input{Listings/inner_product.c.tex}

Note that the loop invariant \inl{inner} claims
that in the $i$-th iteration step the current value of \inl{init}
equals the accumulated value of Equation~\eqref{eq:innerproduct2}.
This depends of course on the properties \inl{bounds} in the contract
of \specref{innerproduct}, which express that there is no arithmetic overflow
when computing the updates of the variable \inl{init}.

\clearpage


\section{The Discrete Logarithm problem}
The \textbf{discrete logarithm} cryptosystem was proposed in the mid-1970s, and to this day there are no deterministically successful known attacks against it if the parameters are chosen carefully. It is the base of several public key cryptosystems taking advantage of its computational difficulty.

It is a public-key cryptography algorithm which can be used for encryption, integrity check and digital signatures, binding the key to an identity. It works taking advantage of the properties of mathematical operations within groups.

\textit{If $G$ is a finite group, $b$ is an element of $G$, and $y$ is an element of $G$ which is a power of $b$, then the discrete logarithm of $y$ to the base $b$ is any integer $x$ such that $b^x = y$.}

Applying this statement to the finite cyclic group $\mathbb{Z}^*_p$ of order $p - 1$, taking a primitive element $\alpha \in\mathbb{Z}^*_p$ and another element $\beta \in \mathbb{Z}^*_p$, the discrete logarithm problem aims to determine the integer $1 \leq x \leq p - 1$ such that:
$$\alpha^x \equiv \beta \mod p \qquad \rightarrow \qquad x = \log_\alpha\beta \mod p$$

The discrete logarithm cryptosystem, as previously stated, relies on the fundamental fact that raising a number $b$ to a power $x$ in a large finite field is an one-way function, i.e.\ it is far more difficult to apply the inverse operation and finding $\log_bx$.

Since the domain is a finite group, such as $\mathbb{Z}^*_n$, the repeated-squaring method can be used to compute $b^x$ with time polynomial in $\log x$, but given an element $y = b^x$, finding $y$ is an open problem.

\subsection{Example}
This example considers a discrete logarithm in the group $\mathbb{Z}^*_{47}$, in which $\alpha = 5$ is a primitive element. The problem consist in finding the positive integer $x$ such that $5^x \equiv 41 \mod 47$.

Even for small numbers, obtaining this value is not immediate. A brute-force attack, i.e.\ systematically trying all possible values for $x$, reveals that the result is $x = 15$.

$\mathbb{Z}^*_{47}$ has order 46, which implies that its subgroups have cardinality of 23, 2 and 1. $\alpha = 2$ is an element with 23 elements, and since 23 is prime then $\alpha$ is primitive. Using prime cardinality is important in order to avoid potential attacks.

When a number $a$ is not a primitive element but another generator for reasonably small $n$, computing the discrete logarithm is still doable solving a system of linear congruences.

\subsection{Generalized discrete logarithm}
Despite prime multiplicative groups increment security, the discrete logarithm applications are not limited to them and can be defined over \textit{any cyclic group}.

Given a finite cyclic group $G$ with an operation $\circ$ and cardinality $n$, having a primitive element $\alpha \in G$ and another element $\beta \in G$, the generalized discrete logarithm problem is finding the integer $1 \leq x \leq n$ such that:
$$\beta = \alpha \circ \alpha \circ \dots \circ \ alpha = \alpha^x$$

$x$ always exists in both cases with $p$ and $n$, since $\alpha$ is a primitive element and can generate every other component.

However, there are cyclic groups in which cracking discrete logarithm is not difficult, since the function is not a one-way. For instance, in $G = (\mathbb{Z}_{11}, +)$ with primitive element $\alpha = 2$, the term $x$ for $\beta = 3$ can be computed with the following steps:
\begin{enumerate}
	\item Expressing the operation as multiplication, $x \cdot 2 \equiv 3 \mod 11$;
	\item Inverting the primitive element $\alpha$, resulting in $x \equiv 2^{-1}3 \mod 11$;
	\item Computing $2^{-1} \equiv 6 \mod 11$;
	\item Obtaining $x \equiv 2^{-1}3 \equiv 7 \mod 11$.
\end{enumerate}
This procedure can be generalized to any additive group for arbitrary $n$ and $\alpha, \beta \in \mathbb{Z}_n$, making the \textbf{generalized discrete logarithm} computationally easy over $\mathbb{Z}_n$ since there are mathematical operations which are not in the additive group (inversion, multiplication).

Strong groups in which discrete logarithm can be used in practice to encrypt data are instead $\mathbb{Z}_p$ or the cyclic group formed by an elliptic curve.

\subsection{Computational aspects}
Despite the discrete logarithm being a strong encryption system, it requires strict constraints to be met regarding fields, length of involved factors, as well as a relevant number of exponentiation, an operation known to be expensive.

An undesired consequence of the long operands of discrete logarithm is that computation is extremely arithmetically intensive: it is not uncommon that one public key operation is by 2-3 orders of magnitude slower than using a private key system.

Since the numbers involved are large, it is more efficient to reduce to modulo multiple times during the computation, with a process called modular exponentiation which relies on the Euclidean algorithm, usually a built-in function of modern programming languages.

On modern computers, such execution procedures are common, and do not consist in a problem, yet performance can be a serious bottleneck in constrained devices with small CPUs.

\subsection{Security and NP-completeness}
Since public key algorithms are based on number-theoretic functions, one distinguish feature of them is that they require arithmetic with very long operands and keys. An algorithm is said to have a ``security level of $n$ bits'' if the best known attack requires $2^n$ steps, since the key is $n$ bits long.

Different security levels for discrete logarithms are:
\begin{table}[h]
	\centering
	\begin{tabular}{c|c}
		\textit{Bit lengths} & \textit{Security level} \\
		\hline
		1024 bit & 80 bit \\
		\hline
		3072 bit & 128 bit \\
		\hline
		7680 bit & 192 bit \\
		\hline
		15360 bit & 256 bit
	\end{tabular}
\end{table}

In order to provide long-term security, i.e.\ ensure that the schema will not be cracked even with the computational power obtained with several decades of hardware development, a security level of at least 128 bit should be chosen.

Despite the existence of sophisticated algorithms running faster than a brute-force approach, and even a quantum algorithm, none of them is able to run in polynomial time.

Furthermore, efficient classical algorithms also exist in certain cases, while on the other hand there exist groups for which computing the discrete algorithm is much more difficult, therefore the domain must be chosen carefully.

\subsection{Real-life applications}
Usage of the discrete logarithm is standardized by The Internet Key Exchange (IKE), which defines cyclic groups and their order, despite some of them being of less than 1024 bit and therefore potentially breakable.

The European Institute for System Security has furthermore established a family of practical cryptography protocols based on discrete logarithm, which were originally intended for LAN security but quickly extended their scope: nowadays, discrete logarithm is used within implementations of access control, authentication, confidentiality protection, key exchange, digital signature and distributed network security management.

The most popular practical application of discrete logarithm is, however, the \textit{Diffie-Hellman key exchange}: since discrete logarithm needs extensive computational power and longer keys than symmetric encryption, it is mostly used for exchanging a private key in a secure way.

Seminar - Algebraic Methods and Algorithms in Cryptology  /chapters/discrete-logarithm.tex