For the remainder, we will focus on distributions of the form
\[ \pi(x)=\mathcal{Z}^{-1}\e^{-U(x)},\]
where \(\mathcal{Z}\) is a normalising constant. Borrowing some terminology from physics, we call the function \(U:\R^d\to\R\) the potential function. In statistical mechanics, this distribution is known as the Boltzmann distribution. The aim here is to use machinery from physics to solve the problem of sampling accurately. To do this we first must motivate the distribution's origin.

Consider a particle in a potential well of shape \(U\). The equation of motion for such a particle is \cite{Langevin1908},
\begin{equation} \dif X_t = -\nabla U(X_t)\dif t +\sqrt{2}\dif W_t. \label{eq:ODLang}.\end{equation}
Here, \(X_t\) is the displacement of the particle from the origin at time \(t\), \(W_t\) is a \(d\)-dimensional Wiener process (Brownian motion) and \(U:\R^d \to \R\) is the potential function. The form of interest here is the \emph{overdamped} Langevin equation, in which the particle experiences no average acceleration, obtained as the high friction limit of full Langevin dynamics.

From the form of the equation, it can be seen that the particle moves down the slope of the well, ``sliding'' towards the minimum energy state. As each particle moves randomly, it is natural to ask what is the average position of many particles in such a well? One would expect the average position to be the one that has lowest energy. To recast the question in statistical language, we ask what is the expected value of the random variable \(X_t\)? This is exactly the goal of MCMC given in Section \ref{sec:intro}. In fact for the Langevin equation, we can give the probability of a particle being at any given displacement. This probability is given exactly by the measure \(\pi\), the Boltzmann distribution. For a diffusion process such as Equation \ref{eq:ODLang} this is called the \emph{stationary distribution}\footnote{Another common term is \emph{invariant measure}}. To show that \(\pi\) is indeed the stationary distribution we use the following lemma.

\begin{lemma}[\cite{OttobreDiffProc}]
	For a one-dimensional It\^o diffusion\footnote{That is \(X_t\) solves \(\dif X_t = \mu(X_t)\dif t +\sigma^2(X_t)\dif W_t\)}, suppose \(\sigma^2(t,x)\) is bounded. Suppose that the transition probabilities of \(X_t\) admit a density for every \(t>0\). Further assume that \(\mu(X_t),\sigma^2(X_t) \in C^2\). Define the Fokker-Planck operator, \(\L^*\), as
	\[\L^*:= -\partial_x(\mu(x)\cdot)+\frac{1}{2}\partial^2_x(\sigma^2(x)\cdot).\]
	Then a measure \(\pi\) is invariant for the diffusion if and only if
	\[\L^*\pi = 0\]
\end{lemma}
The proof of this is omitted however it can be seen by forming the Fokker-Planck equation for the probability density of the diffusion. If a diffusion admits a unique stationary measure, then it is ergodic. This fact means one can apply Theorem \eqref{thm:ergodic} and calculate expectations using Langevin dynamics. The proof that \(\pi\) is the stationary measure of Equation \eqref{eq:ODLang} is given only in the one dimensional case, however it is extendable to higher dimensions. For the Langevin equation, the Fokker-Planck operator is

\[\L^* = \partial_x(U'(x)\cdot)+\partial_{xx}\cdot . \]
So it remains to calculate \(\L^*\pi\).
\begin{align*}
\L^*\pi &= \pd{}{x}\bigg\lbrack U'(x)\pi(x) + \pd{}{x}\pi(x)\bigg\rbrack\\
		&= \pd{}{x}\bigg\lbrack U'(x)\mathcal{Z}\e^{-U(x)}+ \left(-U'(x)\mathcal{Z}\e^{-U(x)}\right)\bigg\rbrack\\
		&= \pd{}{x}\lbrack 0 \rbrack\\
		&= 0
\end{align*}
Hence \(\pi\) is indeed the invariant measure of \eqref{eq:ODLang}. \qed
\\
\\
Although this shows that the Langevin equation has an invariant measure, the question of convergence to this measure remains unanswered. Roberts and Tweedie give the following restriction \cite{RT96}.
\begin{theorem}[Theorem 2.1, \cite{RT96}]
	Let \(P^t_X(x,A) = \P(X_t\in A | X_0 =x_0)\) and suppose that \(\grad U(x)\) is continuously differentiable and that, for some \(N,a,b < \infty\),
	\[\grad U(x)\cdot x \leq a|x|^2 + b, \qquad |x|>N. \]
	Then the measure \(\pi\) is invariant for the Langevin diffusion \(X\). Moreover, for all \(x \in \R^d \) and Borel sets \(A\),
	\[\|P^t_X(x,\cdot) - \pi \| = \frac{1}{2}\sup_A \big|P^t_X(x,A)-\pi(A)\big| \to 0\]

\end{theorem}

\begin{figure}[ht]
	\centering
		\includegraphics[width=\linewidth]{quadraticLD.pdf}
	\caption{Simulating Langevin dynamics in one dimension with a quadratic potential \(U(x)=x^2/2\)}
	\label{fig:quadLD}
\end{figure}

The problem of sampling from the high dimensional distribution has been reduced to being able to accurately simulate Langevin dynamics. This is not as simple as it sounds. To simulate the continuous process  \eqref{eq:ODLang}, it must first be discretised. Doing so may not preserve the convergence to the invariant measure. The discretised process may not have the same stationary measure or it may not even exist. This means that the method used to discretise must be chosen carefully to ensure good convergence properties. The most natural way to discretise an SDE is to use the stochastic analogue of the (forward) Euler method used on ordinary differential equations, known as the Euler-Maruyama (EM) method. Doing so leads to the Unadjusted Langevin Algorithm (\texttt{ULA}).

\subsection{The Unadjusted Langevin Algorithm}
Applying the Euler-Maruyama method to Equation \eqref{eq:ODLang} gives the following iterative scheme.

\[X_{n+1} = X_n -h \nabla U(X_n) +\sqrt{2h} Z_{n+1},\qquad X_0= x_0 \]
Here the \(Z_n \) are i.i.d. standard normal random variables and \(h\) is the step size. This is equivalent to \(X_{n+1} \sim N(X_n - h\grad U(X_n), 2h I_d )\).\footnote{\(I_d\) denotes the \(d \times d\) identity matrix.} A simple example shows that this discretisation does not converge to \(\pi\). Let \(\pi\) be a standard Gaussian distribution, that is \(U(x) = |x|^2/2 \) and choose \(h = 1\). Then the update is given by

\begin{align*}
	X_{n+1} &\sim N(X_n - \grad U(X_n), 2)\\
	& \sim  N(X_n - X_n, 2)\\
	& \sim N(0,2) \nsim \pi .
\end{align*}
So the chain converges immediately, but to the wrong distribution. Let \(\pi^{\text{ULA}}_{h} \) denote the stationary distribution of \texttt{ULA} with a stepsize \(h\). This is not the only issue that can occur. As well as not converging to the correct distribution, the discretised chain may not be  ergodic, even when the continuous diffusion is exponentially ergodic \cite{RT96}. In particular, the algorithm misbehaves when the gradient of the potential is superlinear. That is,
\[\liminf_{\|x\|\to \infty} \frac{\|\grad U(x)\|}{\|x\|} = +\infty. \]
Figure \ref{fig:ULAdiverge} illustrates this problem for a double well potential. To mitigate these issues there are two main approaches: taming the gradient and Metropolisation. A further third method involves using a different discretisation scheme.  Our main focus will be the former, although all three approaches will be discussed.
\begin{figure}
    \centering
    \includegraphics[width=0.75\textwidth]{Figures/ula_doublewell_divergence_step0_2_trace.png}
    \caption{Trace plot of the \texttt{ULA} applied to a double well potential with \(h=0.2\) started from \((0,0)\). The algorithm gets progressively worse until it diverges to infinity.}
    \label{fig:ULAdiverge}
\end{figure}
\subsection{Metropolis Adjustment}
Before describing the Metropolis-adjusted Langevin algorithm \texttt{MALA}, it is pertinent at this point to recall the random walk Metropolis-Hastings algorithm \texttt{RWM }\cite{Hastings70, Metropolis53}. This popular variant of the Metropolis-Hastings algorithm \emph{proposes} values and then accepts/rejects them according to some probability \(\alpha\).  So given \(X_n\), propose a candidate \(Y_{n+1}\) as

\[Y_{n+1} = X_n  + \sqrt{2h} Z_{n+1}.\]
Once again, \(h\) is the step size and \(Z\) is a normal random variable. Then, accept or reject this proposal using Metropolis rejection, that is with some probability
\[\alpha(X_n,Y_{n+1}) = 1\wedge \frac{\pi(Y_{n+1})q(Y_{n+1},X_n)}{\pi(X_n)q(X_n,Y_{n+1})}.\footnote{Here \(t\wedge s = \min\lbrace t,s\rbrace.\) }\]
Here \(q(x,y)\) is the transition probability, \(\P(Y_{n+1}=y | X_{n}=x)\sim N(X_n, h^2)\). This rejection step is key in creating a kernel that is reversible and thus invariant for the measure \(\pi\). \\

\texttt{MALA} can be seen as another variant of the Metropolis-Hastings algorithm, using Langevin dynamics to propose new states as follows.
\[Y_{n+1} = X_n -h\grad U(X_n) + \sqrt{2h} Z_{n+1}\]
It is perhaps better understood as \texttt{ULA} but with an added Metropolis rejection step \cite{RT96}. Adding this rejection step means the algorithm always has the correct invariant distribution, although convergence is still not guaranteed as the following theorem shows.

\begin{theorem}[Theroem 4.2, \cite{RT96}]
	If \(\pi\) is bounded, and
		\[\liminf_{\|x\|\to \infty} \frac{\|\grad U(x)\|}{\|x\|} > \frac{4}{h}\]
	then the \texttt{MALA} chain is not exponentially ergodic. That is, there exists \(R_x <\infty, \rho<1\) such that for all \(x\in \R^d\),
	\[\|P^t_{\tiny \sc MALA}(x,\cdot) - \pi \| \leq R_x\rho^t,\]
	where \(P^t_{\tiny \sc MALA}\) is the transition kernel of the \texttt{MALA} chain.
\end{theorem}
So it can be seen that \texttt{MALA} is not without its issues, and does not solve all the problems of \texttt{ULA}. The concept of taming was introduced to try and reduce the magnitude of these problems.
\begin{figure}[H]
\centering
  \begin{minipage}[b]{0.49\textwidth}
  \centering
    \includegraphics[width=\textwidth]{Figures/tula_tmala_step_1.png}
  \end{minipage} %
  \begin{minipage}[b]{0.49\textwidth}
  \centering
    \includegraphics[width=\textwidth]{Figures/tula_tmala_step_10.png}
  \end{minipage}
   \caption{\textbf{Trade-off between rejection-based algorithms and \texttt{tULA} for large step-sizes} ($h = 1$ on the left and $h = 10$ on the right, distribution is Gaussian with covariance matrix $\text{diag}(1.0, 0.1)$). With increasing step size, even \texttt{tULA} starts to suffer. Rejection-based algorithms resolve the issue, however, their acceptance rate drops very low ($\approx 0.2$ on the left and $\approx 0.03$ on the right). }
\end{figure}

\subsection{Taming the Gradient}
We have seen that both \texttt{ULA} and \texttt{MALA} run into issues when the gradient of the potential is superlinear. Given an SDE such as \eqref{eq:ODLang}, taming adjusts the drift coefficient in such a way that preserves the invariant measure and improves speed of convergence \cite{Brosse18tULA,RT96,Sabanis13}. To do this, a family of drift functions \((G_h)_{h>0}, \ G_h:\R^d \to \R^d\) are introduced. The SDE to be discretised is thus
	\begin{equation*} \dif X_t = -G_h(X_t)\dif t +\sqrt{2}\dif W_t. \end{equation*}
Applying the Euler-Maruyama method gives the following Markov chain
	\[X_{k+1} =X_k-hG_h(X_k)+\sqrt{2h}Z_{k+1},\qquad  X_0=x_0.\]
To preserve the invariant measure, some restrictions must be placed on \((G_h)_{h>0}\), namely that they are `close' to \(\grad U\) ({\bf A1}) while {\bf A2} ensures ergodicity is preserved and improves stability \cite{Brosse18tULA}.

\begin{enumerate}[label={\bf A{\arabic*}}]
	\item  For all \(h>0, G_h\) is continuous. There exist \(\alpha\geq 0, C_{\alpha}<+\infty\) such that for all \(h >0 \) and \(x \in \R^d\),
		\[\|G_h(x)-\grad U(x)\| \leq hC_{\alpha}(1+\|x\|^{\alpha}).\]\label{A1}
	\item For all \(h>0\),
		\[ \liminf_{\|x\|\to \infty} \bigg\lbrack \bigg\langle \frac{x}{\|x\|}, G_h(x)\bigg\rangle - \frac{h}{2\|x\|}\|G_h(x)\|^2\bigg\rbrack >0\]\label{A2}
\end{enumerate}
Here we consider two specific taming functions,
 \begin{align*}
 T_h(x) = \frac{\grad U(x)}{1+h\|\grad U(x)\|}, &&  T^{\text{\sc \tiny RT}}_h = \frac{\grad U(x)}{1\vee h\|\grad U(x)\|}.
 \end{align*}
Brosse et al. introduced and studied \(T_h\) whilst Roberts \& Tweedie suggested \(T^{\text{\sc \tiny RT}}_h\), later analysed by Bou-Rabee \& Vanden-Eijnden \cite{BV10MALTA,Brosse18tULA,RT96}. Both taming functions retain the direction of the gradient, only reducing the magnitude of its effect. The latter is the usual \texttt{ULA} until the gradient gets large enough \((\|\grad U(x)\|> 1/h)\), at which point it begins normalising. In contrast, the first will always tame, regardless of size of the gradient. However for the scaling to have noticeable effect, the gradient must be \(\mathcal{O}(h^{-1})\).
When \(T_h\) is the taming function, the algorithm will be referred to as \texttt{tULA}, the tamed unadjusted Langevin algorithm. When the second is applied, it will be called \texttt{MALTA}, the Metropolis adjusted Langevin truncated algorithm after \cite{RT96}. Any tamed algorithm using \(T_h\) will be prefixed with a lowercase \texttt{t}. For a proof that \(T_h\) satisfies \ref{A1} and \ref{A2}, see \cite[Lemma~2]{Brosse18tULA}.
\\
When the problem is ill-conditioned, taming the gradient does not help.
\subsubsection{Coordinatewise Taming}
So far, the gradient has only been tamed globally. This means that the information the gradient gives is reduced in dimensions where it is not causing divergence. A solution to this is to use coordinatewise taming with the following drift.
  \[T^c_{h}(x) =\left(\frac{\partial_i U(x)}{1+h|\partial_i U(x)|}\right)_{i=\lbrace 1, \dots, d\rbrace} \]
This allows each dimension to be scaled individually. Any algorithm with coordinate-wise taming will be suffixed with a lowercase \texttt{c}.

\subsubsection{Stiff Problems}\label{sec:stiff}
To illustrate the effectiveness of coordinatewise taming, consider an ill-conditioned Gaussian distribution with mean \(\mu,\) covariance matrix \(\Sigma\) as follows.
\begin{align*}
    \mu &= \mathbf{0}, && \Sigma = \begin{bmatrix}  1 & 0 \\ 0 & 0.0001 \end{bmatrix}
\end{align*}
Figure \ref{fig:stiffULA} shows the samples generated by the tamed unadjusted and coordinatewise tamed unadjusted Langevin algorithms (\texttt{tULA} and \texttt{tULAc}, respectively) applied to this problem. It can be seen that \texttt{tULA} overestimates the width of the distribution in both dimensions by a large margin. This is because the slope fo the potential well is relatively shallow in this direction, meaning that the strong taming generated by the gradient in the other direction reduces this shallow well to almost flat. This means that the particle does not `slide down' as quickly back to the minimum. Contrast this with \texttt{tULAc} which provides a range of samples much closer to the true distribution in one dimension. The problem still remains in the other dimension, however this is almost unavoidable with an explicit scheme.
\begin{figure}[H]
\centering
  \begin{minipage}[b]{0.49\textwidth}
  \centering
    \includegraphics[width=\textwidth]{Figures/ic_gaussian_tula.png}
  \end{minipage} %
  \begin{minipage}[b]{0.49\textwidth}
  \centering
    \includegraphics[width=\textwidth]{Figures/ic_gaussian_tulac.png}
  \end{minipage}
   \caption{\textbf{Overestimation of Global Taming:} \texttt{tULA} (left) generates samples that are greatly overdispersed in both dimensions. Contrast with \texttt{tULAc} (right) which gives a much tighter spread over one dimension. In both plots, orange shows samples drawn from the \texttt{RWM} algorithm as reference for `true' samples.}
   \label{fig:stiffULA}
\end{figure}

\begin{figure}[H]
\centering
  \begin{minipage}[b]{0.49\textwidth}
  \centering
    \includegraphics[width=\textwidth]{Figures/transparentBoth.png}
  \end{minipage} %
   \caption{\textbf{Coordinatewise Improvement:} A more visible comparison of the improvement coordinatewise taming offers over global taming.}
   \label{fig:stifftULAc}
\end{figure}

\subsection{Discretise Differently}
An alternative approach is to use a different discretisation of the SDE \eqref{eq:ODLang}, which we consider in this section. The first is an extension of the Euler method \cite{Sabanis18tHOLA}, while the latter uses a non-Markovian scheme developed for use in molecular dynamics \cite{LM12}.
\subsubsection{Higher Order Langevin Algorithm}
As in the ordinary case, the Euler-Maruyama method is not the only way of discretising an SDE. One can also take a higher order expansion, analogous to the Runge-Kutta method in ODE theory, known as the order 1.5 Wagner-Platen expansion\footnote{Or the stochastic Runge-Kutta method \cite{Schaffter10numericalintegration}}. For a one dimensional Langevin diffusion \eqref{eq:ODLang}, this is
\[X_{n+1} = X_n -hU'(X_n)+\sqrt{2h}Z_n -\sqrt{2} U''(X_n) \tilde{Z}_n +\frac{ h^2 }{2}\bigg\lbrack U'(X_n)U''(X_n)-U'''(X_n)\bigg\rbrack.  \]
Here, \(\tilde{Z}_n\) is defined as
\[  \tilde{Z}_n = \int_{t_n}^{t_{n+1}} \int_{t_n}^s \dif W_r \dif s. \]
This is a Gaussian random variable with mean \(0\) and variance \(\frac{1}{3}h^3 \). Extending this to \(d\)-dimensions and applying to the Langevin SDE \eqref{eq:ODLang} and taming as above gives the following iterative scheme. The untamed version is the same, but with all subscripts removed.
\[X_{n+1} = X_n + \mu_{h}(X_n)h +\sigma_{h}(X_n)\sqrt{h}Z_{n+1},\]
where
\[\mu_{h}(x) = -\grad U_{h}(x) +\frac{h}{2}\left( \left( \grad^2U\grad U\right)_{h}(x) - \vec{\Delta}(\grad U)_{h}(x)\right) ,\]
and \(\sigma_{h}(x) = \text{diag}\left(\left( \sigma_{h}^{(k)}(x)\right)_{k\in \lbrace 1,\dots,d\rbrace}\right)\) with,
\[\sigma_{h}^{(k)}(x) = \sqrt{2+\frac{2h^2}{3}\sum_{j=1}^d |\grad^2 U_{h}^{(k,j)}(x)|^2 - 2h \grad^2 U_{h}^{(k,k)}(x)}.\]
The subscript \(h\) indicates a taming of the variable has occurred as follows. For any \(x\in \R^d\),
\begin{align*}
    \grad U_h(x) &=\frac{\grad U(x)}{(1+h^{3/2}|\grad U(x)|^{3/2})^{2/3}}\, , && \grad^2U_h(x) = \frac{\grad^2 U(x)}{1+h|\grad^2 U(x)|} ,\\
    (\grad U\grad^2U)_h(x) =&\frac{\grad^2U(x)\grad U(x)}{1+h|x||\grad^2U(x)||\grad U(x)|}\, , && \vec{\Delta}(\grad U)_h(x) = \frac{\vec{\Delta}(\grad U)(x)}{1+h^{1/2}|x||\vec{\Delta}(\grad U)(x)|}.
\end{align*}
Like in the classical case, the aim here is to improve the rate of convergence by using a more accurate discretisation of the underlying diffusion.

\subsubsection{Leimkuhler-Matthews Method}
The Leimkuhler-Matthews method was developed in \cite{LM12} and cleverly exploits the link between the sampling problem and molecular dynamics. The scheme they developed is as follows:
\begin{equation} X_{n+1} = X_n - h \grad U (X_n) +\sqrt{\frac{h}{2}} (Z_n+Z_{n+1}) \label{eq:LM} \end{equation}

Note that this method is non-Markovian as it incorporates the noise term from the previous iteration. At first glance it appears that this is almost identical to the standard EM scheme, however the slight modification drastically improves convergence.  It is derived by considering both positions and momenta in Langevin dynamics (as opposed to the overdamped equation we have considered thus far which neglects momenta).

\begin{align} \dif X_t = P_t \dif t, && \dif P_t = \lbrack -\grad U(X_t) - \gamma P_t \rbrack \dif t + \sqrt{2} \dif W_t \label{eq:LMLang} \end{align}

Here, \(\gamma\) is the friction coefficient. It does no appear in the overdamped Langevin equation as it is the high friction limit of the above system.  Equation \eqref{eq:LMLang} can be seen as a combination of an Ornstein-Uhlenbeck process and Hamiltonian dynamics. Splitting the system in this way allows the exact solution of each part in each time step.

\[ \dif \begin{bmatrix} X_t\\P_t \end{bmatrix} = \underbrace{\begin{bmatrix} P_t \\ 0 \end{bmatrix}}_{\text{A}} \dif t + \underbrace{\begin{bmatrix}  0\\-\grad U \end{bmatrix}}_{\text{B}} \dif t + \underbrace{\begin{bmatrix} 0\\ -\gamma P_t \dif t + \dif W_t \end{bmatrix}}_{\text{O}} \]

The Hamiltonian dynamics have been split further in to A and B.  As well as solving these sections separately at each time step, it is also possible to do fractions of a time step for each term. If we solve in the order BAOAB, that is, half a time step each of B and A before a full time step of the OU process before solving the rest of A and B, we recover Equation \eqref{eq:LM} upon taking the high friction limit. Despite the dependence of iterates, the noise quickly decorrelates. Let \(R_n = (Z_n+Z_{n+1})/\sqrt{2}\). Then,

\begin{align*}
    \langle R_n,R_n\rangle  &= \frac{1}{2} \left( \langle Z_{n+1},Z_{n+1}\rangle +\langle Z_{n},Z_{n}\rangle\right) =1\\
    \langle R_n,R_n\rangle = \langle Z_n,Z_n\rangle = \frac{1}{2}\\
    \langle R_n,R_{n-k} \rangle &= 0,   \quad k=2,3,\dots
\end{align*}

\subsection{Visualization}
A demonstration of the above methods has been implemented using the visualization library of \cite{rogozhnikov2016hmc}\footnote{With kind permission of Alex Rogozhnikov, \url{https://arogozhnikov.github.io/about/}.}. The visualization dynamically follows the trace of a chosen method applied to a chosen two-dimensional distribution. Distributions of various qualitative properties are available. This can be found at the following \textsc{url}: \\
\\
   \centerline{ \url{http://goatleaps.xyz/assets/ULA/ULA.html}}
\\
\begin{figure}[H]
\centering
  \begin{minipage}[b]{0.8\textwidth}
  \centering
    \includegraphics[width=0.8\textwidth]{Figures/ulavis.PNG}
    \caption{Screenshot from the visualization;  \texttt{tULAc} applied to a Gaussian mixture distribution.}
  \end{minipage}
\end{figure}


\subsection{Higher-order tensors}

We can create higher order tensors products. For example

\(V\otimes V \otimes V\otimes V^* \otimes V^*\)

We write elements of these as:

\(T_{j_1,...,j_q}^{i_1,...,i_p}\)

We can map from matrix to matrix etc higher dimensional

Matrix has \(A\): \(a_{ij}\).

Tensor can have \(T\): \(t_{ijk}\) for example

\(0\) rank tensor: scalar

\(1\) rank tensor: vector

\(2\) rank tensor: matrix

page on covariance and contravariance and type \((p,q)\)


\section{Similarity}

We begin this section by recalling the definition  of similar matrices.
Recall that if $A,B$ are two $n\times n$-matrices, then they are \textbf{similar}\index{similar matrices}
if and only if there exists an invertible matrix $P$
such that
\begin{equation*}
A=P^{-1}BP
\end{equation*}

In this case we write $A\similar B$. The concept of similarity is an example of an \textbf{equivalence relation}\index{equivalence relation}.

\begin{lemma}{Similarity is an equivalence relation}{similarity-equivalence}
Similarity is an equivalence relation, i.e. for $n \times n$-matrices $A,B$, and $C$,
\begin{enumerate}
\item $A \similar A$ (reflexive)
\item If $A \similar B$, then $B \similar A$ (symmetric)
\item If $A \similar B$ and $B \similar C$, then $A \similar C$ (transitive)
\end{enumerate}
\end{lemma}

\begin{proof}
It is clear that $A\similar A$, taking $P=I$.

Now, if $A\similar B$, then for some $P$ invertible,
\begin{equation*}
A=P^{-1}BP
\end{equation*}
and so
\begin{equation*}
PAP^{-1}=B
\end{equation*}
But then
\begin{equation*}
(P^{-1}) ^{-1}AP^{-1}=B
\end{equation*}
which shows that $B\similar A$.

Now suppose $A\similar B$ and $B\similar C$. Then there exist invertible matrices
$P,Q$ such that
\begin{equation*}
A=P^{-1}BP,\ B=Q^{-1}CQ
\end{equation*}
Then,
\begin{equation*}
A=P^{-1} (Q^{-1}CQ)P=(QP) ^{-1}C(QP)
\end{equation*}
showing that $A$ is similar to $C$.
\end{proof}

Another important concept necessary to this section is the trace of a matrix. Consider the definition.

\begin{definition}{Trace of a matrix}{matrix-trace}
\index{trace of a matrix}%
If $A=[a_{ij}]$ is an $n\times n$-matrix, then the
trace of $A$ is
\[ \tr(A) = \sum_{i=1}^n a_{ii}.\]
\end{definition}

In words, the trace of a matrix is the sum of the entries on the main diagonal.

\begin{lemma}{Properties of trace}{trace-properties}
For $n\times n$-matrices $A$ and $B$, and any $k\in\R$,
\begin{enumerate}
\item $\tr(A+B)=\tr(A) + \tr(B)$
\item $\tr(kA)=k\cdot\tr(A)$
\item $\tr(AB)=\tr(BA)$
\end{enumerate}
\end{lemma}

The following theorem includes a reference to the characteristic polynomial of a matrix. Recall that for any $n \times n$-matrix $A$, the characteristic polynomial of $A$ is $c_A(x)=\det(xI-A)$.

\begin{theorem}{Properties of similar matrices}{properties-similar}
If $A$ and $B$ are $n\times n$-matrices and $A\similar B$, then
\begin{enumerate}
\item $\det(A) = \det(B)$
\item $\rank(A) = \rank(B)$
\item $\tr(A)= \tr(B)$
\item $c_A(x)=c_B(x)$
\item $A$ and $B$ have the same eigenvalues
\end{enumerate}
\end{theorem}


% !TEX root = Main.tex
\section{K-Means}
\textbf{Target:} $\min_{\mathbf{U}, \mathbf{Z}} J(\mathbf{U}, \mathbf{Z}) = \|\mathbf{X} - \mathbf{U} \mathbf{Z}\|_F^2$\\
$= \sum_{n=1}^N \sum_{k=1}^K \mathbf{z}_{k,n} \|\mathbf{x}_n - \mathbf{u}_k\|_2^2$\\
1. \textbf{Initiate:} choose $K$ centroids $\mathbf{U} = [\mathbf{u}_1, \ldots, \mathbf{u}_K]$\\
2. \textbf{Assign:} data points to clusters. $k^\star(\mathbf{x}_n) = \argmin_k \{ \|\mathbf{x}_n - \mathbf{u}_k\|_2 \}$ returns cluster $k^\star$, whose centroid $\mathbf{u}_{k^\star}$ is closest to data point $\mathbf{x}_n$. Set $\mathbf{z}_{k^\star,n} = 1$, and for $ l \neq k^\star~ \mathbf{z}_{l,n}=0$.\\
3. \textbf{Update} centroids: $\mathbf{u}_k = \frac{\sum_{n=1}^N z_{k,n} \mathbf{x}_n}{\sum_{n=1}^N z_{k,n}}$.\\
4. Repeat from step 2, stops if $\|\mathbf{Z} - \mathbf{Z}^\text{new}\|_0 = \|\mathbf{Z} - \mathbf{Z}^\text{new}\|^2_F = 0$.\\
Computational cost: $O(k\cdot n \cdot d)$

\subsection{K-Means++}
Initialize cluster centers ``most far apart'' based on data points. $D(x)$ shortest distance from data point to closest, already chosen center.\\
1. Set center $c_1$, chosen uniformly at random from $\mathcal{X}$.\\
2. Assign new center $c_i$, choosing $x \in \mathcal{X}$ with $p = \tfrac{D(x)^2}{\sum_{x \in \mathcal{X}} D(x)^2}$.\\
3. Repeat 2 until $i = k$ for $k$ centers.\\
4. Continue with K-Means.


If you haven't downloaded and unzipped \href{https://libaoj.in/courses/2021f/MATH3341/zip/Math.3341.zip}{\texttt{Math.3341.zip}}. Download and unzip it under \verb|H:| (H Drive if you are working on the Remote Lab). Change the current working directory by typing \verb|cd H:\Math.3341\Math.3341.Lab.06| in the Command Window, and type \verb|edit lab_06_script| in the Command Window to edit \verb|lab_06_script.m|.

%---------------------------------------------
\section{Solve a System with LU Decomposition}
%---------------------------------------------
\label{sec:lu}
\begin{enumerate}[(a)]
\item Define matrix \verb|A| and vector \verb|b| as \eqref{eq:ls}.
  \begin{equation}
  \label{eq:ls}
  \underbrace{
  \begin{bmatrix}
    7 & -26 &  45 & -47 \\
    1 &   2 &   3 &   4 \\
    2 & -11 & -12 & -13 \\
    4 & -17 &  30 &  35 \\
  \end{bmatrix}}_{A}
  \underbrace{
  \begin{bmatrix}
    x_1 \\
    x_2 \\
    x_3 \\
    x_4 \\
  \end{bmatrix}}_{\mathbf{x}}
  =
  \underbrace{
  \begin{bmatrix}
     -98 \\
      30 \\
    -108 \\
     200 \\
  \end{bmatrix}}_{\mathbf{b}}
  \end{equation}
\item Calculate the LU decomposition \verb|L|, \verb|U| of the matrix \verb|A|.
\item \label{enu:lz} Solve the following system \eqref{eq:lz} and store the solution to \verb|z|.
  \begin{equation}
  \label{eq:lz}
  L \mathbf{z} = \mathbf{b}.
  \end{equation}
\item \label{enu:ux} Then solve the following system \eqref{eq:ux} and store the solution to \verb|x|.
    \begin{equation}
    \label{eq:ux}
    U \mathbf{x} = \mathbf{z}.
    \end{equation}
\item Check your solution by calculating the norm of the residual $\|A\mathbf{x} - \mathbf{b}\|_2$ and store the result to \verb|res|.
\end{enumerate}
%---------------------------------------------
\section{Varying the Vector $\mathbf{b}$}
%---------------------------------------------
Suppose we want to solve the system for each integer value of $m$ in between $m = 0$ and $m = 20$. This time use the LU decomposition of the system matrix; perform the decomposition only once and use the lower and upper triangular factors repeatedly to find each successive solution. Then generate a table (Table \ref{tab:solution}) and a plot (Figure \ref{fig:solution}) of the solution versus the integer $m$.
\begin{equation}
  \label{eq:sys}
  \begin{cases}
    3 x + y + z = m \\
    x - 5 y + 2 z = 5 \\
    2 x + y + 5 z = 10 \\
  \end{cases}
\end{equation}
To do this you'll follow the steps below:
\begin{enumerate}[(a)]
\item Define coefficient matrix \verb|A| given in \eqref{eq:sys}, and get the LU decomposition \verb|L|, \verb|U| of the matrix \verb|A|.
\item Define a vector \verb|m| which ranges from $0$ to $20$ with step size $1$.
\item Then create a for-loop, of which the loop iterator \verb|i| starts from \verb|1| to \verb|length(m)|. In the body of the loop, define a column vector \verb|b| as the right-hand side of \eqref{eq:sys}, where $m$ should be the \verb|i|th component of \verb|m|. Then repeat \eqref{enu:lz} and \eqref{enu:ux} in Part \ref{sec:lu}. Store the solution \verb|x| to the \verb|i|th row of \verb|X|.
\item Format the output of \verb|m| and \verb|X| to a file called \verb|solution.tex| as you did in Part 3 of Lab 05:
  \begin{enumerate}[(i)]
    \item Use \verb|fprintf| to print out the setup for the \emph{table} and \emph{tabular} environments. The first column of the table is centered while the rest three columns are right-justified in \LaTeX{}.
    \item Between \verb|\toprule| and \verb|\midrule|, use \verb|fprintf| to print out the heading of the table. The column widths are $4, 11, 11, 11$, respectively.
    \item Between \verb|\midrule| and \verb|\bottomrule|, use a for-loop to print each row of the table. Note that the $i$th row of the table consists of the \verb|i|th component of \verb|m| and the \verb|i|th row of matrix \verb|X|. The column widths are $2, 9, 9, 9$, respectively. For floating point numbers, output 6 digits after the decimal point.
    \item Call \verb|type('solution.tex')| to print the content of \verb|solution.tex|.
  \end{enumerate}
\item Plot the solution versus $m$ using a for-loop as you did in Part 4 of Lab 05:
  \begin{enumerate}[(i)]
    \item Get the size of \verb|X| and assign it to \verb|XSize|. Define a cell array \verb|styles|, of which the entries are dashed line with hexagram, dotted line with pentagram, solid line with diamond.
    \item The use a for-loop to plot each column of \verb|X| versus \verb|m| in the same figure window with the above styles.
    \item Add labels, title, grid, legend as shown in Figure \ref{fig:solution}.
    \item Save the plot to a file named \verb|lab_06_plot.pdf|.
  \end{enumerate}
\end{enumerate}
%---------------------------------------------
Type \verb|diary('lab_06_output.txt')| in the Command Window, run the script file \verb|lab_06_script.m|, and type \verb|diary off| in the Command Window. Upload \verb|lab_06_output.txt|, \verb|lab_06_script.m|, \verb|solution.tex|, and \verb|lab_06_plot.pdf| to the folder \verb|src| on Overleaf.

On Overleaf, open \verb|body.tex| under the folder \verb|LaTeX|. In the last section of the report, you will reproduce Section \ref{sec:bol} using \LaTeX{}. You may find the following helpful:

\begin{itemize}
  \item You may use enviroments such as  \verb|equation|, \verb|cases|, \verb|figure|, and \verb|table|.
  \item You may use \verb|\includegraphics[width=amount unit]{/path/to/figure.pdf}| to specify the width of a figure. In our case, the width of the figure is \verb|0.85\textwidth|.
  \item You may use \verb|\ref{labelName}| to refer to figures, tables; use \verb|\eqref{labelName}| to refer to equations.
  \item For special symbols, you may look them up in \href{https://libaoj.in/files/LaTeX.Mathematical.Symbols.pdf}{\LaTeX{}.Mathematics.Symbols.pdf}.
  \item You may use \verb|\input{/path/to/solution.tex}| to include the table you got from MATLAB.
\end{itemize}

Recompile and submit the PDF file generated by Overleaf to WyoCourses.

\newpage
%---------------------------------------------
\section{Basics of \LaTeX{}}
\label{sec:bol}
%---------------------------------------------
\subsection{LU Decomposition}

Given the linear system \eqref{eq:varyRHS}
\begin{equation}
  \label{eq:varyRHS}
  \begin{cases}
    3x + y + z = m \\
    x - 5y + 2z = 5 \\
    2x + y + 5z = 10
  \end{cases}
\end{equation}
where $m = 0, 1, 2, \ldots, 20$. Using LU Decomposition we can obtain the solution to the linear system \eqref{eq:varyRHS} for corresponding $m$ (see Table \ref{tab:solution} and Figure \ref{fig:solution}).

\input{../Math.3341.Lab.06.ans/solution.tex}

\begin{figure}[!hbtp]
  \centering
  \includegraphics[width=0.85\textwidth]{../Math.3341.Lab.06.ans/lab_06_plot.pdf}
  \caption{Solution to the linear system vs. $m$}
  \label{fig:solution}
\end{figure}


\section{Complex inner product spaces}

\begin{outcome}
  \begin{enumerate}
  \item Compute dot products in $\C^n$.
  \item Use properties of the complex dot product to prove equalities
    and inequalities.
  \item Compute the adjoint of a matrix.
  \item Check whether an operation is a complex inner product.
  \item Determine whether vectors in a complex inner product space are
    orthogonal.
  \item Calculate the complex Fourier coefficients of a vector.
  \item Use the Gram-Schmidt procedure to find an orthogonal basis of
    a subspace of a complex inner product space.
  \item Compute the orthogonal projection of a complex vector onto a
    subspace.
  \end{enumerate}
\end{outcome}

So far, in this chapter, the field $K$ was always $\R$, the set of
real numbers.  The reason we have not considered inner products over
other fields $K$ is that the positive definite property requires
$\iprod{\vect{u},\vect{u}}\geq 0$, and the requirement that a scalar
is ``greater or equal to 0'' does not make sense if $K$ is, say, the
field of integers modulo $p$.

In this section, we will consider inner product spaces over the
complex numbers. It turns out that the theory of complex inner product
spaces is similar, but not completely identical, to that of real inner
product spaces. To explain the difference, consider the definition of
the dot product. In $\R^n$, the dot product of two vectors
\begin{equation*}
  \vect{v}=\begin{mymatrix}{c} x_1 \\ \vdots \\ x_n \end{mymatrix}
  \quad\mbox{and}\quad
  \vect{w}=\begin{mymatrix}{c} y_1 \\ \vdots \\ y_n \end{mymatrix}
\end{equation*}
is defined to be
\begin{equation*}
  \vect{v}\dotprod \vect{w} = x_1y_1 + \ldots + x_ny_n.
\end{equation*}
One of the most important properties of the dot product is positivity:
for all $\vect{v}$, we have
\begin{equation*}
  \vect{v}\dotprod\vect{v} = x_1^2 + \ldots + x_n^2\geq 0.
\end{equation*}
The reason positivity holds is that the square of a real number is
always greater or equal to 0. If we blindly replaced $x_1,\ldots,x_n$
by complex numbers and kept the same definition of dot product,
positivity would no longer hold. This is because for a complex number
$z$, it is not in general true that $z^2\geq 0$. In fact, $z^2$ may
not be a real number, and even in cases where $z^2$ is real, it may
not be positive. For example, if $z=i$, then $z^2=-1$.

Fortunately, all is not lost: the complex numbers actually do have a
useful positivity property. Namely, if $z=a+bi$ is a complex number
and $\conjugate{z}=a-bi$ is its complex conjugate, then
\begin{equation*}
  \conjugate{z}z = (a-bi)(a+bi) = a^2 + b^2 \geq 0.
\end{equation*}
So instead of squaring a complex number, we should multiply it by its
conjugate. With this in mind, we arrive at the following definition of
dot product on $\C^n$:

\begin{definition}{The dot product on $\C^n$}{complex-dot-product}
  Let
  \begin{equation*}
    \vect{v}=\begin{mymatrix}{c} v_1 \\ \vdots \\ v_n \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{w}=\begin{mymatrix}{c} w_1 \\ \vdots \\ w_n \end{mymatrix}
  \end{equation*}
  be vectors in $\C^n$. Their \textbf{(complex) dot product}%
  \index{dot product!complex}%
  \index{vector!dot product!complex}%
  \index{complex dot product}
  is defined to be
  \begin{equation*}
    \vect{v}\dotprod \vect{w} = \conjugate{v_1}w_1 + \ldots + \conjugate{v_n}w_n.
  \end{equation*}
\end{definition}

The complex dot product satisfies properties that are similar to, but
not exactly the same as, the properties satisfied by the real dot
product.

\begin{proposition}{Properties of the complex dot product}{properties-complex-dot-product}
  \index{properties of dot product!complex}%
  \index{vector!properties of dot product!complex}%
  \index{complex dot product!properties}%
  The dot product satisfies the following properties, where
  $\vect{u},\vect{v},\vect{w}\in\C^n$ and $k,\ell\in\C$.
  \begin{itemize}
  \item Conjugate symmetry: $\vect{u}\dotprod\vect{v}=\conjugate{\vect{v}\dotprod\vect{u}}$.
  \item Linearity on the right: $\vect{u}\dotprod(k\vect{v}+\ell\vect{w})
    =k(\vect{u}\dotprod \vect{v})+\ell(\vect{u}\dotprod\vect{w})$.
  \item Antilinearity on the left: $(k\vect{u}+\ell\vect{v})\dotprod\vect{w}=\conjugate{k}(\vect{u}\dotprod\vect{w})+\conjugate{\ell}(\vect{v}\dotprod\vect{w})$.
  \item The positive definite property: $\vect{u}\dotprod\vect{u}\geq 0$, and $\vect{u}\dotprod\vect{u}=0$ if and only if $\vect{u}=\vect{0}$.
  \end{itemize}
\end{proposition}

We note that the complex dot product can be equivalently expressed as
a matrix product, namely
\begin{equation*}
  \vect{v}\dotprod\vect{w}
  ~=~ \begin{mymatrix}{ccc} \conjugate{v_1} & \cdots & \conjugate{v_n} \end{mymatrix}
  \begin{mymatrix}{c} w_1 \\ \vdots \\ w_n \end{mymatrix}
  ~=~ \conjugate{\vect{v}}^T \vect{w}.
\end{equation*}
Here, $\conjugate{\vect{v}}$ denotes the complex conjugate of a vector
(i.e., taking the complex conjugate of each component of a vector),
and $(-)^T$ denotes the transpose as usual.  As a matter of fact, when
working with complex vectors and matrices, it turns out that we should
almost {\em always} take the complex conjugate at the same time as
taking the transpose. For this reason, we introduce a special name and
notation for the conjugate transpose of a vector or matrix.

\begin{definition}{Adjoint of a matrix}{adjoint}
  Let $A$ be a complex $n\times m$-matrix. The \textbf{adjoint} of
  $A$, denoted $A^{\adjoint}$, is the transpose of the complex conjugate of
  $A$. In symbols:
  \begin{equation*}
    A^{\adjoint} = \conjugate{A}{}^T.
  \end{equation*}
\end{definition}

With this definition, we can also write the dot product as
\begin{equation*}
  \vect{v}\dotprod\vect{w} = \vect{v}^{\adjoint}\vect{w}.
\end{equation*}

We are now ready to state the definition of a complex inner product,
which is a generalization of the complex dot product.

\begin{definition}{Complex inner product space}{complex-inner-product-space}
  A \textbf{complex inner product space}%
  \index{inner product space!complex}%
  \index{complex inner product space|see{inner product space}} is a
  complex vector space $V$ equipped with an operation that assigns to
  any pair of vectors $\vect{u},\vect{v}\in V$ a complex number
  $\iprod{\vect{u},\vect{v}}$, called the \textbf{inner product}%
  \index{inner product!complex}%
  \index{multiplication!complex inner product} of $\vect{u}$ and $\vect{v}$,
  satisfying the following properties:
  \begin{enumerate}
  \item Conjugate symmetry:%
    \index{conjugate symmetry!of inner product}%
    \index{symmetry!conjugate symmetry}
    $\iprod{\vect{u},\vect{v}} =
    \conjugate{\iprod{\vect{v},\vect{u}}}$.
  \item Linearity on the right:%
    \index{linearity!of inner product}
    $\iprod{\vect{u},k\vect{v}+\ell\vect{w}}
    =k\iprod{\vect{u}, \vect{v}}+\ell\iprod{\vect{u},\vect{w}}$.
  \item Antilinearity on the left:%
    \index{antilinearity!of inner product}
    $\iprod{k\vect{u}+\ell\vect{v},\vect{w}}=\conjugate{k}\iprod{\vect{u},\vect{w}}+\conjugate{\ell}\iprod{\vect{v},\vect{w}}$.
  \item The positive definite property:%
    \index{positive definite property}
    $\iprod{\vect{u},\vect{u}} \geq 0$, and
    $\iprod{\vect{u},\vect{u}}=0$ if and only if
    $\vect{u} = \vect{0}$.
  \end{enumerate}
\end{definition}

Note that conjugate symmetry implies that $\iprod{\vect{u},\vect{u}}$
is a real number for every vector $\vect{u}$. Namely, let
$z=\iprod{\vect{u},\vect{u}}$. Then by conjugate symmetry, we have
\begin{equation*}
  z
  = \iprod{\vect{u},\vect{u}}
  = \conjugate{\iprod{\vect{u},\vect{u}}}
  = \conjugate{z}.
\end{equation*}
Since $z$ is equal to its own conjugate, it must be a real
number. Therefore, the positive definite property makes sense: when we
require that $\iprod{\vect{u},\vect{u}} \geq 0$, we are talking about
a real number that is greater than or equal to $0$. (It would not in
general make sense to ask whether a complex number is  greater than or
equal to $0$).

The space $\C^n$ with the complex dot product is evidently an example
of a complex inner product space. Here is another example:

\begin{example}{Complex-valued continuous functions}{continuous-complex}
  Let $a<b$ be real numbers, and let $C[a,b]$ be the space of
  continuous, complex-valued functions $f : [a,b]\to\C$. Given two such
  functions $f,g\in C[a,b]$, we define their inner product as
  \begin{equation*}
    \iprod{f,g} = \int_{a}^{b} \conjugate{f(x)}g(x)\,dx.
  \end{equation*}
  With this operation, $C[a,b]$ is a complex inner product space.
\end{example}

Armed with this definition of complex inner products, we can now
pretty much redo everything we did for real inner products in the
complex case. The only thing we have to be careful about is to put the
complex conjugate operation in the correct places.

\begin{itemize}
\item The \textbf{norm}%
  \index{norm!in Cn@in $\C^n$}%
  \index{vector!norm}%
  \index{inner product space!norm} of a vector in a complex inner
  product space is defined to be
  $\norm{\vect{u}} = \sqrt{\iprod{\vect{u},\vect{u}}}$.  This
  definition makes sense because $\iprod{\vect{u},\vect{u}}\geq 0$.
\item The \textbf{Cauchy-Schwarz inequality}%
  \index{Cauchy-Schwarz inequality!complex}%
  \index{inner product space!Cauchy-Schwarz inequality}
  $\abs{\iprod{\vect{u},\vect{v}}}\leq \norm{\vect{u}}\norm{\vect{v}}$
  and the \textbf{triangle inequality}%
  \index{triangle inequality!complex}%
  \index{inner product space!triangle inequality}
  $\norm{\vect{u}+\vect{v}}\leq \norm{\vect{u}} + \norm{\vect{v}}$
  hold in complex inner product spaces.
\item Two vectors $\vect{u},\vect{v}$ in a complex inner product space
  are called \textbf{orthogonal}%
  \index{orthogonal vectors}%
  \index{vector!orthogonal}, in symbols $\vect{u}\orth\vect{v}$, if
  $\iprod{\vect{u},\vect{v}}=0$.
\item A vector $\vect{u}$ in a complex inner product space is called
  \textbf{normalized}%
  \index{normalized vector}%
  \index{vector!normalized} if $\norm{\vect{u}} = 1$.
\item A set of vectors $\set{\vect{u}_1,\ldots,\vect{u}_n}$ is called
  an \textbf{orthogonal set}%
  \index{orthogonal set} if the vectors are non-zero and pairwise
  orthogonal, and an \textbf{orthonormal set}%
  \index{orthonormal set} if the vectors are moreover normalized.
\item If $\vect{u}_1,\ldots,\vect{u}_k$ are orthogonal, then
  $\norm{a_1\vect{u}_1+\ldots+a_k\vect{u}_k}^2 =
  \abs{a_1}^2\norm{\vect{u}_1}^2 + \ldots + \abs{a_k}^2\norm{\vect{u}_k}^2$.
  The absolute value signs are necessary because $\conjugate{a_i}a_i=|a_i|^2$.
\end{itemize}

\begin{example}{Orthogonal vectors}{orthogonal-vectors-complex}
  Consider $\C^2$ with the complex dot product, and the following
  vectors:
  \begin{equation*}
    \vect{u}=\begin{mymatrix}{c} 1 \\ i \end{mymatrix},
    \quad
    \vect{v}=\begin{mymatrix}{c} -i \\ 1 \end{mymatrix},
    \quad\mbox{and}\quad
    \vect{w}=\begin{mymatrix}{c} i \\ 1 \end{mymatrix}.
  \end{equation*}
  Are $\vect{u}$ and $\vect{v}$ orthogonal? Are $\vect{u}$ and
  $\vect{w}$ orthogonal?
\end{example}

\begin{solution}
  We have
  \begin{equation*}
    \iprod{\vect{u},\vect{v}}
    = \vect{u}^{\adjoint}\vect{v}
    = \begin{mymatrix}{cc} 1 & -i \end{mymatrix}
    \begin{mymatrix}{c} -i \\ 1 \end{mymatrix}
    = -i-i
    = -2i
    \neq 0,
  \end{equation*}
  so $\vect{u}$ and $\vect{v}$ are not orthogonal. We have
  \begin{equation*}
    \iprod{\vect{u},\vect{w}}
    = \vect{u}^{\adjoint}\vect{w}
    = \begin{mymatrix}{cc} 1 & -i \end{mymatrix}
    \begin{mymatrix}{c} i \\ 1 \end{mymatrix}
    = i-i
    = 0,
  \end{equation*}
  so $\vect{u}$ and $\vect{w}$ are orthogonal. Note that it is crucial
  here that we did not forget to take the complex conjugate of
  $\vect{u}$, or else we would have gotten a different answer.
\end{solution}

In some formulas, we must be careful about whether we use
$\iprod{\vect{v},\vect{w}}$ or $\iprod{\vect{w},\vect{v}}$. Although
this did not make any difference in the case of real inner products,
it does make a difference for complex inner products, because in
general, $\iprod{\vect{v},\vect{w}}\neq \iprod{\vect{w},\vect{v}}$. In
particular, we have to be careful about this in the formulas for
Fourier coefficients, projections, and the Gram-Schmidt procedure.

\begin{example}{Complex Fourier coefficients}{complex-fourier-coefficients}
  Let $\set{\vect{u}_1,\ldots,\vect{u}_n}$ be an orthogonal set of
  vectors in a complex inner product space, and let
  \begin{equation*}
    \vect{v} = a_1\vect{u}_1 + \ldots + a_n\vect{u}_n.
  \end{equation*}
  Which of the following formulas is correct?
  \begin{equation*}
    a_i = \frac{\iprod{\vect{v},\vect{u}_i}}{\iprod{\vect{u}_i,\vect{u}_i}}
    \quad\mbox{or}\quad
    a_i = \frac{\iprod{\vect{u}_i,\vect{v}}}{\iprod{\vect{u}_i,\vect{u}_i}}
    \quad?
  \end{equation*}
\end{example}

\begin{solution}
  To check whether the first formula is correct, we calculate
  \begin{eqnarray*}
    \frac{\iprod{\vect{v},\vect{u}_i}}{\iprod{\vect{u}_i,\vect{u}_i}}
    &=&
        \frac{\iprod{a_1\vect{u}_1 + \ldots + a_i\vect{u}_i + \ldots + a_n\vect{u}_n,\vect{u}_i}}{\iprod{\vect{u}_i,\vect{u}_i}} \\
    &=&
        \conjugate{a_1}\frac{\iprod{\vect{u}_1,\vect{u}_i}}{\iprod{\vect{u}_i,\vect{u}_i}}
        + \ldots
        + \conjugate{a_i}\frac{\iprod{\vect{u}_i,\vect{u}_i}}{\iprod{\vect{u}_i,\vect{u}_i}}
        + \ldots
        + \conjugate{a_n}\frac{\iprod{\vect{u}_n,\vect{u}_i}}{\iprod{\vect{u}_i,\vect{u}_i}} \\
    &=&
        \conjugate{a_1}\cdot 0
        + \ldots
        + \conjugate{a_i}\cdot 1
        + \ldots
        + \conjugate{a_n}\cdot 0 \\
    &=&
        \conjugate{a_i}.
  \end{eqnarray*}
  Note that because of antilinearity, this formula came out to be
  $\conjugate{a_i}$, and not $a_i$. Therefore, the first formula is
  not correct.  The check the second formula, we calculate
  \begin{eqnarray*}
    \frac{\iprod{\vect{u}_i,\vect{v}}}{\iprod{\vect{u}_i,\vect{u}_i}}
    &=&
        \frac{\iprod{\vect{u}_i,a_1\vect{u}_1 + \ldots + a_i\vect{u}_i + \ldots + a_n\vect{u}_n}}{\iprod{\vect{u}_i,\vect{u}_i}} \\
    &=&
        a_1\frac{\iprod{\vect{u}_i,\vect{u}_1}}{\iprod{\vect{u}_i}}
        + \ldots
        + a_i\frac{\iprod{\vect{u}_i,\vect{u}_i}}{\iprod{\vect{u}_i,\vect{u}_i}}
        + \ldots
        + a_n\frac{\iprod{\vect{u}_i,\vect{u}_n}}{\iprod{\vect{u}_i,\vect{u}_i}} \\
    &=&
        a_1\cdot 0
        + \ldots
        + a_i 1
        + \ldots
        + a_n\cdot 0 \\
    &=&
        a_i.
  \end{eqnarray*}
  Therefore, the second formula is correct.
\end{solution}

Since this is an important result, we state it as a proposition.

\begin{proposition}{Complex Fourier coefficients}{complex-fourier-coefficients}
  If $\set{\vect{u}_1,\ldots,\vect{u}_n}$ is an orthogonal set of
  vectors in a complex inner product space and
  \begin{equation*}
    \vect{v} = a_1\vect{u}_1 + \ldots + a_n\vect{u}_n,
  \end{equation*}
  then
  \begin{equation*}
    a_i = \frac{\iprod{\vect{u}_i,\vect{v}}}{\iprod{\vect{u}_i,\vect{u}_i}}
    \quad\mbox{and}\quad
    \conjugate{a_i} = \frac{\iprod{\vect{v},\vect{u}_i}}{\iprod{\vect{u}_i,\vect{u}_i}}.
  \end{equation*}
\end{proposition}

\begin{example}{Calculating complex Fourier coefficients}{calculating-complex-fourier-coefficients}
  Suppose that $B=\set{\vect{u}_1,\vect{u}_2,\vect{u}_3}$ is an
  orthogonal basis for a complex inner product space $V$, such that
  $\norm{\vect{u}_1}=1$, $\norm{\vect{u}_2}=\sqrt{5}$, and
  $\norm{\vect{u}_3}=2$. Moreover, suppose that $\vect{v}\in V$ is a
  vector such that $\iprod{\vect{u}_1,\vect{v}} = i$,
  $\iprod{\vect{u}_2,\vect{v}} = -2$, and
  $\iprod{\vect{u}_3,\vect{v}} = 1-2i$. Find the coordinates of
  $\vect{v}$ with respect to $B$.
\end{example}

\begin{solution}
  We need to find $a_1, a_2, a_3$ such that
  $\vect{v} = a_1\vect{u}_1 + a_2\vect{u}_2 + a_3\vect{u}_3$.  By
  Proposition~\ref{prop:complex-fourier-coefficients}, we have
  \begin{eqnarray*}
    a_1
    &=& \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}
        ~~=~~ \frac{\iprod{\vect{u}_1,\vect{v}}}{\norm{\vect{u}_1}^2}
        ~~=~~ \frac{i}{1}
        ~~=~~ i, \\
    a_2
    &=& \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}
        ~~=~~ \frac{\iprod{\vect{u}_2,\vect{v}}}{\norm{\vect{u}_2}^2}
        ~~=~~ \frac{-2}{5}
        ~~=~~ -\frac{2}{5}, \\
    a_3
    &=& \frac{\iprod{\vect{u}_3,\vect{v}}}{\iprod{\vect{u}_3,\vect{u}_3}}
        ~~=~~ \frac{\iprod{\vect{u}_3,\vect{v}}}{\norm{\vect{u}_3}^2}
        ~~=~~ \frac{1-2i}{4}
        ~~=~~ \frac{1}{4} - \frac{1}{2}i.
  \end{eqnarray*}
\end{solution}

The Gram-Schmidt orthogonalization procedure works without changes in
complex inner product spaces, as long as we are careful not to confuse
$\iprod{\vect{u}_i,\vect{v}_j}$ with $\iprod{\vect{v}_j,\vect{u}_i}$.

\begin{proposition}{Complex Gram-Schmidt orthogonalization procedure}{gram-schmidt-complex}
  Let $\set{\vect{v}_1,\ldots,\vect{v}_k}$ be a basis for some subspace $W$
  of a complex inner product space $V$.%
  \index{Gram-Schmidt procedure!complex}%
  \index{orthogonalization!complex}%
  \index{orthogonal basis!Gram-Schmidt procedure!complex}
  Define vectors
  $\vect{u}_1,\ldots,\vect{u}_k$ as follows:
  \begin{eqnarray*}
    \vect{u}_1
    &=& \vect{v}_1,
    \\
    \vect{u}_2
    &=& \vect{v}_2
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_2}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1,
    \\
    \vect{u}_3
    &=& \vect{v}_3
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_3}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~-~ \frac{\iprod{\vect{u}_2,\vect{v}_3}}{\iprod{\vect{u}_2,\vect{u}_2}}\vect{u}_2,
    \\
    &\vdots&
    \\
    \vect{u}_k
    &=& \vect{v}_k
        ~-~ \frac{\iprod{\vect{u}_1,\vect{v}_k}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
        ~-~ \frac{\iprod{\vect{u}_2,\vect{v}_k}}{\iprod{\vect{u}_2,\vect{u}_2}}\vect{u}_2
        ~-~ \ldots
        ~-~ \frac{\iprod{\vect{u}_{k-1},\vect{v}_k}}{\iprod{\vect{u}_{k-1},\vect{u}_{k-1}}}\vect{u}_{k-1}.
  \end{eqnarray*}
  Then $\set{\vect{u}_1,\ldots,\vect{u}_k}$ is an orthogonal basis of $W$.
\end{proposition}

\begin{example}{Complex Gram-Schmidt orthogonalization procedure}{gram-schmidt-complex}
  Consider $\C^3$ with the complex dot product. Let
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{c} 1+i \\ 1 \\ i \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{v}_2 = \begin{mymatrix}{c} i \\ -1 \\ -3  \end{mymatrix}
  \end{equation*}
  Use the Gram-Schmidt procedure to find an orthogonal basis for
  $\sspan\set{\vect{v}_1,\vect{v}_2}$.
\end{example}

\begin{solution}
  We start with
  \begin{equation*}
    \vect{u}_1
    ~=~ \vect{v}_1
    ~=~ \begin{mymatrix}{c} 1+i \\ 1 \\ i \end{mymatrix}.
  \end{equation*}
  Next, we calculate
  \begin{equation*}
    \iprod{\vect{u}_1,\vect{v}_2}
    ~=~ \vect{u}_1^{\adjoint} \vect{v}_2
    ~=~ \begin{mymatrix}{ccc} 1-i & 1 & -i \end{mymatrix}
    \begin{mymatrix}{c} i \\ -1 \\ -3  \end{mymatrix}
    ~=~ (1-i)i + 1(-1) + (-i)(-3)
    ~=~ 4i
  \end{equation*}
  and
  \begin{equation*}
    \iprod{\vect{u}_1,\vect{u}_1}
    ~=~ \vect{u}_1^{\adjoint} \vect{u}_1
    ~=~ \begin{mymatrix}{ccc} 1-i & 1 & -i \end{mymatrix}
    \begin{mymatrix}{c} 1+i \\ 1 \\ i  \end{mymatrix}
    ~=~ (1-i)(1+i) + 1\cdot 1 + (-i)(i)
    ~=~ 4.
  \end{equation*}
  Therefore
  \begin{equation*}
    \vect{u}_2
    ~=~ \vect{v}_2
    - \frac{\iprod{\vect{u}_1,\vect{v}_2}}{\iprod{\vect{u}_1,\vect{u}_1}}\vect{u}_1
    ~=~ \vect{v}_2
    - \frac{4i}{4} \vect{u}_1
    ~=~ \vect{v}_2
    - i \vect{u}_1
    ~=~ \begin{mymatrix}{c} i \\ -1 \\ -3  \end{mymatrix}
    -i \begin{mymatrix}{c} 1+i \\ 1 \\ i \end{mymatrix}
    ~=~ \begin{mymatrix}{c} 1 \\ -1-i \\ -2  \end{mymatrix}.
  \end{equation*}
  The desired orthogonal basis is $\set{\vect{u}_1,\vect{u}_2}$. We
  double-check that $\vect{u}_1$ and $\vect{u}_2$ are indeed
  orthogonal:
  \begin{equation*}
    \iprod{\vect{u}_1,\vect{u}_2}
    ~=~ \vect{u}_1^{\adjoint}\vect{u}_2
    ~=~ \begin{mymatrix}{ccc} 1-i & 1 & -i \end{mymatrix}
    \begin{mymatrix}{c} 1 \\ -1-i \\ -2  \end{mymatrix}
    ~=~ (1-i)1 + 1(-1-i) + (-i)(-2)
    ~=~ 0.
  \end{equation*}
\end{solution}

Orthogonal projections also work in complex inner product spaces.

\begin{proposition}{Orthogonal projection onto a subspace}{projection-subspace-complex}
  Let $V$ be a complex inner product space, and let $W$ be a subspace
  of\/ $V$. Assume $\set{\vect{u}_1,\ldots,\vect{u}_k}$ is an
  orthogonal basis of\/ $W$, and $\vect{v}\in V$ is any vector.
  Let
  \begin{equation*}
    \vect{v}' =
    \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
    + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2
    + \ldots
    + \frac{\iprod{\vect{u}_k,\vect{v}}}{\iprod{\vect{u}_k,\vect{u}_k}}\,\vect{u}_k.
  \end{equation*}
  Then $\vect{v}'$ is the best approximation%
  \index{approximation} of $\vect{v}$ in $W$, i.e., it is the element
  of $W$ such that $\norm{\vect{v}-\vect{v}'}$ is as small as
  possible. Moreover, the vector $\vect{v}-\vect{v}'$ is orthogonal to
  $W$. We say that $\vect{v}'$ is the \textbf{orthogonal projection of
    $\vect{v}$ onto $W$}%
  \index{orthogonal projection!onto subspace!complex}%
  \index{projection!onto subspace!complex}.
\end{proposition}

\begin{example}{Orthogonal projection onto a subspace}{projection-subspace-complex}
  Consider the subspace of $\C^3$ spanned by
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{c} 1 \\ i \\ 0 \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{u}_2 = \begin{mymatrix}{c} i \\ 1 \\ 1 \end{mymatrix}.
  \end{equation*}
  With respect to the complex dot product, find the best approximation
  of $\vect{v}=\begin{mymatrix}{c} 0 \\ 2 \\ 4\end{mymatrix}$ in this
  subspace.
\end{example}

\begin{solution}
  First notice that $\iprod{\vect{u}_1,\vect{u}_2}=0$, so that
  $\vect{u}_1$ and $\vect{u}_2$ are orthogonal. Therefore, the best
  approximation is given by
  \begin{equation*}
    \vect{v}' =
    \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
    + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2.
  \end{equation*}
  We calculate the relevant inner products:
  \begin{eqnarray*}
    \iprod{\vect{u}_1,\vect{v}}
    &=& \begin{mymatrix}{ccc} 1 & -i & 0 \end{mymatrix}
        \begin{mymatrix}{c} 0 \\ 2 \\ 4\end{mymatrix}
    ~~=~~ -2i, \\
    \iprod{\vect{u}_2,\vect{v}}
    &=& \begin{mymatrix}{ccc} -i & 1 & 1 \end{mymatrix}
        \begin{mymatrix}{c} 0 \\ 2 \\ 4\end{mymatrix}
    ~~=~~ 6, \\
    \iprod{\vect{u}_1,\vect{u}_1}
    &=& \begin{mymatrix}{ccc} 1 & -i & 0 \end{mymatrix}
        \begin{mymatrix}{c} 1 \\ i \\ 0 \end{mymatrix}
    ~~=~~ 2, \\
    \iprod{\vect{u}_2,\vect{u}_2}
    &=& \begin{mymatrix}{ccc} -i & 1 & 1 \end{mymatrix}
        \begin{mymatrix}{c} i \\ 1 \\ 1\end{mymatrix}
    ~~=~~ 3.
  \end{eqnarray*}
  Therefore,
  \begin{equation*}
    \vect{v}'
    ~~=~~
    \frac{\iprod{\vect{u}_1,\vect{v}}}{\iprod{\vect{u}_1,\vect{u}_1}}\,\vect{u}_1
    + \frac{\iprod{\vect{u}_2,\vect{v}}}{\iprod{\vect{u}_2,\vect{u}_2}}\,\vect{u}_2.
    ~~=~~
    \frac{-2i}{2}\,\vect{u}_1 + \frac{6}{3}\,\vect{u}_2
    ~~=~~
    - i \begin{mymatrix}{c} 1 \\ i \\ 0 \end{mymatrix}
    + 2 \begin{mymatrix}{c} i \\ 1 \\ 1 \end{mymatrix}
    ~~=~~
    \begin{mymatrix}{c} i \\ 3 \\ 2 \end{mymatrix}.
  \end{equation*}
  To double-check the answer, we can check that $\vect{v}-\vect{v}'$
  is indeed orthogonal to $\vect{u}_1$ and $\vect{u}_2$. We have
  \begin{equation*}
    \vect{v}-\vect{v}' = \begin{mymatrix}{c} -i \\ -1 \\ 2 \end{mymatrix}
  \end{equation*}
  and
  \begin{eqnarray*}
    \iprod{\vect{u}_1,\vect{v}-\vect{v}'}
    &=&
    \begin{mymatrix}{ccc} 1 & -i & 0 \end{mymatrix}
    \begin{mymatrix}{c} -i \\ -1 \\ 2 \end{mymatrix}
    ~~=~~ 0, \\
    \iprod{\vect{u}_2,\vect{v}-\vect{v}'}
    &=&
    \begin{mymatrix}{ccc} -i & 1 & 1 \end{mymatrix}
    \begin{mymatrix}{c} -i \\ -1 \\ 2 \end{mymatrix}
    ~~=~~ 0.
  \end{eqnarray*}
\end{solution}

We finish this section with some remarks on the differences between
the notations used in mathematics and in physics. Complex inner
product spaces are very important in physics because they are the
foundation of quantum mechanics. The adjoint of a matrix $A$ is
usually denoted $A^{\adjoint}$ in mathematics and $A^{\dagger}$ in
physics. In quantum mechanics, a column vector $\vect{v}$ is often
written $\ket{v}$, and the corresponding row vector
$\vect{v}^{\adjoint}$ is then written as $\bra{v}$.  This is the
so-called \textbf{Dirac notation}%
\index{Dirac notation}. With this convention, an inner product
$\vect{v}^{\adjoint}\vect{w}$ is $\bra{v}\ket{w}$, which is usually
written as $\braket{v}{w}$. Also, the matrix
$\vect{v}\vect{w}^{\adjoint}$ is written $\ket{v}\bra{w}$ and is
called an \textbf{outer product}%
\index{outer product}%
\index{multiplication!outer product}. In mathematics, it is customary
for inner products to be linear in the left component and antilinear
in the right component. In physics, it is customary to use the
opposite convention, i.e., inner products are antilinear in the left
component and linear in the right component. In this book, we have
used the physics convention of antilinearity in the left component,
because it is the better convention.


\section{Linear Sample of Dimension One}

\begin{defi}[valid convolution]
A valid convolution is the linear map
\begin{align*}
    x[n] \circ y[k] \to z[m]
\end{align*}
where
\begin{align*}
    z[i] &= \sum_{j=0}^{k - 1} x[i + j] \cdot y[j] \quad (0 \leq i < m) \\
    n &= m + k - 1
\end{align*}
\end{defi}

\begin{defi}[strided valid convolution]
A strided valid convolution is the linear map
\begin{align*}
    x[n] \circ y[k] \to z[m]
\end{align*}
where
\begin{align*}
    z[i] &= \sum_{j=0}^{k - 1} x[i \cdot s + j] \cdot y[j] \quad (0 \leq i < m) \\
    n &= s (m - 1) + k
\end{align*}
\end{defi}

\begin{defi}[linear sample]
A linear sample is a map that maps a sequence of length $n$ to a sequence of
length $m$.
\end{defi}

The sampling takes the following steps:
\begin{itemize}
\item The input sequence $x[n]$ is padded to $\hat x[\hat n]$ by
prepending $p_l$ zero elements and appending $p_r$ zero elements, where
\begin{align*}
    \hat n = p_l + n + p_r
\end{align*}
In most case $p_r = p_l$ or $p_r = p_l + 1$.

\item The kernel (filter) $y[k]$ is extended to $\hat y[\hat k]$
by inserting $r - 1$ zero elements to adjacent elements, where
\begin{align*}
    \hat k = r * (k - 1) + 1
\end{align*}

\item The padded input $\hat x$ and the extended kernel $\hat y$ takes a
valid convolution with stride $s$, resulting the output sequence $z[m]$,
where
\begin{align*}
    m = {\hat n - \hat k \over s} + 1
\end{align*}
and $s \mid \hat n - \hat k$ should be granted.
\end{itemize}

$(p_l, p_r, k, s, r, n, m)$, $p_l, p_r \in \mathbb N$, $k, s, r, n, m \in \mathbb N^\ast$
is called a valid configuration of linear sample if
\begin{align*}
    s (m - 1) = (n + p_l + p_r) - (r (k - 1) + 1)
\end{align*}


\chapter{Functions}
\label{chapter:functions}
\marginurl{%
  Functions and Quantifiers:\\\noindent
  Introduction to Mathematical Reasoning \#7
}{youtu.be/VHJeUrCedTU}

Another important type of objects in mathematics are functions. Function $f$
from a set $X$ to a set $Y$ (written as $f : X \to Y$) is a unique
assignment of elements of $Y$ to the elements of $X$ (note that it is not
necessary that all the elements of $Y$ are used). In other words, for each
element $x \in X$ there is one assigned element $f(x) \in Y$. We call such an
element the \emph{value} of $f$ at $x$, we also say that $f(x)$ is an
\emph{image} of $x$.

Unfortunately, the definition is not formal. Through this chapter we will
provide a more formal definition.

\section{Quantifiers.}
The first ingredient is called quantifiers. Very often we use phrases like ``all
the people in the class have smartphones.'' However, we still do not know how to
write it using symbols.

\paragraph{The Universal Quantifier.}
In order to say ``all'' or ``every'' we use the symbol $\forall$\footnote{%
  The symbol is a turned ``A'' symbol, the first letter of the word ``all''.
}: if $P(a)$ is a predicate about $a \in A$, then $\forall a \in A\ P(a)$ is a
statement saying that all the elements of $A$ satisfy the predicate $P$. In
other words it is the same as the statement
$\set[P(a)]{a \in A} = A$. For example, $\forall x \in \R\ x \cdot 0 = 0$ says
that product of every real number and zero is equal to zero.
\nomenclature[L]{$\forall x \in X~P(x)$}{denotes the statement saying that
$P$ is true for all elements of $X$}

\paragraph{The Existential Quantifier.}
The second quantifier means ``there is'' and is denoted by the symbol
$\exists$\footnote{%
  The symbol is a turned ``E'' symbol, the first letter of the word
  ``exists''. It is also interesting that the symbol for the universal
  quantifier was introduced by Gerhard Gentzen in 1935 but the symbol for the
  existential quantifier was introduced, 38 years earlier, by Giuseppe Peano
  in 1897.
}:
if $P(a)$ is a predicate about an element of $A$, then $\exists a \in A\ P(a)$
says that there is an element of $A$ satisfying the predicate $P$ i.e.
$\set[P(a)]{a \in A} \neq \emptyset$. For example,
$\exists x \in \R\ x^2 - 1 = 0$ states that there is a real solution of the
equation $x^2 - 1 = 0$.
\nomenclature[L]{$\exists x \in X \  P(x)$}{denotes the statement saying that
  $P$ is true for some element of $X$}

\begin{warning}
  Note that the word ``any'' sometimes indicates a universal statement and
  sometimes an existential statement.

  Standard meaning of ``any'' is ``every'' like in the statement ``$a^2 \ge 0$
  for any real number'', therefore this statement can be rewritten as
  $\forall a \in \R \ a^2 \ge 0$. Nonetheless, in the negative and interrogative
  statements ``any'' is used to mean ``some''. For example, ``There is not any
  real number $a$ such that $a^2 < 0$'' is asserting that the statement
  $\exists a \in \R \ a^2 < 0$ is false. And ``Is there any real number $a$ such
  that $a^2 = 1$?'' is asking whether the existential statement
  $\exists a \in \R \ a^2 = 1$ is true.

  Real care is required with questions involving ``any'': ``Is there any
  integer $a$ such that $a \ge 1$?'' clearly is asking whether $\exists a \in
  \R \ a^2 \ge 1$ is true; however, ``Is $a \ge 1$ for any integer $a$'' is less
  clear and might be taken to asking about the same question as the first
  question, $\exists a \in \Z \ a \ge 1$ (which is true) but might also be
  taken to be asking about $\forall a \in \Z \ a \ge 1$ (which is false).
\end{warning}

\subsection{Proving Statements Involving Quantifiers}
Most of the statements in mathematics involve quantifiers. This is one of the
factors distinguishing advanced from elementary mathematics. In this section we
give an overview of the main methods of proof. Though the whole book is about
proving such results.

\paragraph{Proving statements of the form $\forall a \in A \ P(a)$.}
Such statements can be rewritten in the form $a \in A \implies P(a)$. For
example, we proved earlier that $a^2 \ge 0$ for all real numbers $a$ using this
approach.

\paragraph{Proving statements of the form $\exists a \in A \ P(a)$.}
The easiest way to prove such a statement is by simply exhibiting an element $a$
of $A$ such that $P(a)$ is true. This method is called \emph{proof by
example}.

Let us prove the statement $\exists x \in \N \ x^2 = 4$ using this method.
Observe that $2 \in N$ and $2^2 = 4$ so $x = 2$ provides an example proving
this statement. There are, however, less direct methods such as use of the
counting arguments.

\paragraph{Proving statements involving both quantifiers.}
To illustrate problems of this type let us prove that for any integer $n$, if
$n$ is even, then $n^2$ is also even.

This statement is a universal statement $\forall n \in \Z \ (n \text{ is even}
\implies n^2 \text{ is even})$. However, the hypothesis that $n$ is even is an
existential statement $\exists q \in \Z \ n = 2q$. So we begin the proof as
follows:
\begin{center}
  Suppose that $n$ is an even integer. Then $n = 2q$ for some integer $q$.
\end{center}
The conclusion we wish to prove is that $n^2$ is even, which may be written as
$\exists q \in \Z \ n^2 = 2q$. Note that $q$ here is a dummy variable used to
express the statement $n^2$ is a doubled integer. We may replace it by any
other letter not already in use, for example $\exists p \in \Z \ n^2 = 2p$.
Hence, if we present $p$ such that $n^2 = 2p$, we finish the proof. As a result,
we can complete the proof as follows.
\begin{center}
  Therefore, $n^2 = (2q)^2 = 4q^2$ and so, since $2q^2$ is an integer $n^2$ is
  even.
\end{center}

\subsection{Disproving Statements Involving Quantifiers}
Disproving something seems a bit off from the first glance, but to some extent
it is the same as proving the negation.

\paragraph{Disproving statements of the form $\forall a \in A \ P(a)$.}
We may note that the negation of such a statement is the statement $\exists a
\in A \ \lnot P(a)$. So we can disprove it by giving a single example for which
it is false. This is called \emph{disproof by counterexample} to $P(a)$.

For example, we may disprove the statement $\forall x \in \R \ x^2 > 2$ by
giving a counterexample $x = 1$ since $1^2 = 1 < 2$.

\paragraph{Disproving statements of the form $\exists a \in A \ P(a)$.}
The negation of this statement is the statement $\forall a \in A \ \lnot P(a)$.
Which gives one way of disproving the statement.

Let us prove that there does not exist a real number $x$ such that $x^2 = -1$.
We know that, for all $x \in \R$, we have the inequality $x^2 \ge 0$ and so $x^2
\neq -1$. Hence, there does not exist $x \in \R$ such that $x^2 = -1$.

\section{Cartesian product}
Another ingredient is the notion of Cartesian product. If $X$ and $Y$ are two
sets, then $X \times Y = \set[x \in X \text{ and } y \in Y]{(x, y)}$. We also
denote $\underbrace{X \times X \times \dots \times X}_{k \text{ times}}$ by
$X^k$.

Consider the following example. If $X = \set{a, b, c}$ and $Y = \set{a, b}$,
then
\[
  X \times Y = \set{(a, a), (a, b), (b, a), (b, b), (c, a), (c, b)}.
\]
Additionally, $\R^2 = \R \times \R$ is the familiar $2$-dimensional Euclidean
plane.
\nomenclature[S]{$A \times B$}{denotes the set of all ordered pairs of elements
of $A$ and $B$}

\begin{exercise}
  Find the set $\set{a, b} \times \set{a, b} \setminus
    \set[x \in \set{a, b}]{(x, x)}$
\end{exercise}

\begin{theorem}
\label{theorem:cartesian-product-distributivity}
  For all sets $A$, $B$, $C$, and $D$ the following hold:
  \begin{itemize}
    \item $A \times (B \cup C) = (A \times B) \cup (A \times C)$;
    \item $A \times (B \cap C) = (A \times B) \cap (A \times C)$;
    \item $(A \times B) \cup (C \times D) \subseteq
        (A \cup C) \times (B \cup D)$;
    \item $(A \times B) \cap (C \times D) =
        (A \cap C) \times (B \cap D)$.
  \end{itemize}
\end{theorem}
\begin{proof}
  It is easy to prove this statement by the definitions. Let us prove only the
  second equality, the rest is
  Exercise~\ref{exercise:cartesian-product-distributivity}.

  Note that $(x, y) \in A \times (B \cap C)$ iff $x \in A$ and
  $y \in (B \cap C)$.
  Hence, $(x, y) \in A \times (B \cap C)$ iff $x \in A$, $y \in B$, and
  $y \in C$. Thus $(x, y) \in A \times (B \cap C)$ iff
  $(x, y) \in (A \times B)$ and $(x, y) \in (A \times C)$. As a result,
  $(x, y) \in A \times (B \cap C)$ iff
  $(x, y) \in (A \times B) \cap (A \times C)$ as required.
\end{proof}

\begin{exercise}
\label{exercise:cartesian-product-distributivity}
  Prove the rest of the equalities in
  Theorem~\ref{theorem:cartesian-product-distributivity}.
\end{exercise}

\section{Graphs of Functions}
\label{section:graph}
Now we have all the components to define a function. Mathematicians think about
the functions in the way we defined them at the beginning of the chapter,
however formally in order to define a function $f : X \to Y$ one need to define
a set $D \subseteq X \times Y$ (such a set is called the \emph{graph of the
function} $f$) such that
\begin{itemize}
  \item $\forall x \in X~\exists y \in Y\ (x, y) \in D$ and
  \item $\forall x \in X, y_1, y_2 \in Y\
    ((x, y_1) \in D \land (x, y_2) \in D \implies y_1 = y_2)$.
\end{itemize}
We say that $y \in Y$ is the value $f(x)$ of the function described by $D$ at
$x \in X$ iff $(x, y) \in D$.

The simplest way to think about the functions is in the terms of tables.
Let us use this idea to list all the functions $\set{a, b, c}$ to $\set{d, e}$.
\begin{center}
  \begin{tabular}{l l l l l l l l l}
    \toprule
    $x$ & $f_1(x)$ & $f_2(x)$ & $f_3(x)$ & $f_4(x)$ &
    $f_5(x)$ & $f_6(x)$ & $f_7(x)$ & $f_8(x)$ \\
    \midrule
    a & d & d & d & d & e & e & e & e \\
    b & d & d & e & e & d & d & e & e \\
    c & d & e & d & e & d & e & d & e \\
    \bottomrule
  \end{tabular}
\end{center}

\begin{exercise}
  List all the functions from $\set{a, b}$ to $\set{a, b}$.
\end{exercise}

However, listing all the values of a function is only possible when the domain
of the function is finite. Thus the most common way to describe a function is
using a formula which provides a way to find the value of a function. When the
function is defined as a formula it is important to be clear which sets are the
domain and the codomain of the function.

Let $\R_+ = \set[x \ge 0]{x \in \R}$. Consider the following functions.
\begin{itemize}
  \item $g_1 : \R \to \R$ such that $g_1(x) = x^2$;
  \item $g_2 : \R_+ \to \R$ such that $g_2(x) = x^2$;
  \item $g_3 : \R \to \R_+$ such that $g_3(x) = x^2$;
  \item $g_4 : \R_+ \to \R_+$ such that $g_4(x) = x^2$;
\end{itemize}
Nonetheless that all these functions are defined using the same formula $x^2$,
we will see in the next chapters that these four functions have different
properties.

\begin{exercise}
  Find the graph of the function $f : \Z \to \Z$ such that $f(x) = 3x$.
\end{exercise}

Note that when you define the function you need to define it such that the
definition makes sense for all the elements of the domain. For example, the
formula $g(x) = \frac{x^2 - 3x + 2}{x - 1}$ does not define a function from
$\R$ to $\R$ since it is not defined for $x = 1$. It is typical to define a
function from real numbers to real numbers by a formula and the convention is
that the domain is the set of all numbers for which the formula makes sense
(unless the domain is specified explicitly).
Using this convention the formula $g$ defines a function from $\R \setminus
\set{1}$ to $\R$.

If we really need a function from $\R$ there are two possible approaches for
extending $g$.

\paragraph{Rewriting the formula.}
We can rewrite the formula such that it makes sense for all the real numbers.
Note that for all $x \in \R \setminus \set{1}$,
\[
  \frac{x^2 - 3x + 2}{x - 1} = \frac{(x - 2)(x - 1)}{x - 1} = x - 2.
\]
Then $g_1(x) = x - 2$ defines a function on $\R$ extending the function $g$.

\paragraph{Explicit definition.} Alternatively we can explicitly specify the
value of $g$ at $1$. So
\[
  g_2(x) =
  \begin{cases}
    \frac{x^2 - 3x + 2}{x - 1} & \text{if } x \neq 1 \\
    -1 & \text{if } x = 1
  \end{cases}
\]
defines a function from $\R$ to $\R$. Note that we can specify the values at
individual points any way we want.

Similarly to sets we may define the equality between functions. We say that two
functions $f, g : X \to Y$ are equal ($f = g$) iff $f(x) = g(x)$ for all
$x \in X$ i.e. their graphs are equal. Note that two functions are equal only
if they have the same domains and codomains. For example, $g_1$ and $g_2$ we
just defined are equal to each other nonetheless that we defined them in two
different ways.

We defined $g_1$ and $g_2$ to extend $g$ to a bigger domain, similarly we can
make a domain smaller.
\begin{definition}
  Let $f : X \to Y$ and $A \subseteq X$. Then $f|_{A} : A \to Y$ is a function
  such that $\forall x \in A\ f|_{A}(x) = f(x)$ (we say that $f|_A$ is the
  restriction of $f$ to the set $A$).
\end{definition}

\nomenclature[F]{$f\big\rvert_A$}{denotes the restriction of $f$ to the set $A$}

\section{Composition of Functions}
\begin{figure}
  \begin{center}
    \begin{tikzpicture}[
      >=stealth,
      bullet/.style={
        fill=black,
        circle,
        minimum width=1pt,
        inner sep=1pt
      },
      projection/.style={
        ->,
        thick,
        shorten <=2pt,
        shorten >=2pt
      },
      every fit/.style={
        ellipse,
        draw,
        inner sep=0pt
      }
    ]

      \node at (2,4.7) {$f$};
      \draw[projection] (1,4.5) -- (3,4.5);
      \node at (0,5) {$X$};
      \node[bullet,label=below:$x$] (START)   at (0,2.5){};
      \node at (4,5) {$Y$};
      \node[bullet,label=below:$f(x)$] at (4,2.5){};
      \node at (6,4.7) {$g$};
      \draw[projection] (5,4.5) -- (7,4.5);
      \node at (8,5) {$Z$};
      \node[bullet,label=below:$g\big(f(x)\big)$] (END) at (8,2.5){};

      \draw [line width=8pt, yellow, shorten <=0.25cm,, shorten >=0.6cm, ->]
        (START.south)
        to[out=-50, in=-120, distance=4cm, ]
        node[black, midway, below, align = center, yshift = -0.25cm]
        {$g \circ f$} (END);

      \draw (0,2.5) ellipse (1.02cm and 2.2cm);
      \draw (4,2.5) ellipse (1.02cm and 2.2cm);
      \draw (8,2.5) ellipse (1.02cm and 2.2cm);

      \draw[projection] (0.3,2.5) -- (3.7,2.5);
      \draw[projection] (4.3,2.5) -- (7.7,2.5);
    \end{tikzpicture}
    \newline
  \end{center}
  \caption{Composition of functions}
\end{figure}

Suppose $f : X \to Y$ and $g : Y \to Z$ be some function. Then, given an
element $x \in X$, the function $f$ assigns $y = f(x) \in Y$, and the function
$g$ assigns $z = g(y) = g(f(x)) \in Z$. Thus using $f$ and $g$ an element of
$Z$ can be assigned to $x$. This operation defines a function from $X$ to $Z$
and the result of this operation is called the \emph{composition} of $f$ and
$g$.
\nomenclature[F]{$f \circ g$}{denotes the composition of functions $f$ and $g$;
i.e, it denotes the function $h(x) = f(g(x))$.}

\begin{definition}
    If $f : X \to Y$ and $g : Y \to Z$, then $h = g \circ f$ is a
    function from $X$ to $Z$ such that $g(f(x)) = h(x)$ for all $x \in X$.
\end{definition}

Let us consider an example. Let $f : \R \to \R$ such that $f(x) = x + 1$ and
$g : \R \to \R$ such that $g(x) = x^2$. Then $(g \circ f) : \R \to \R$ and
$(g \circ f)(x) = (x + 1)^2$ for all $x \in \R$. Note that the order of $f$ and
$g$ is important since $(f \circ g)(x) = x^2 + 1$. Thus composition is not
\emph{commutative}.

There are two special type functions.
\begin{itemize}
    \item Let $A \subseteq X$, then $i : A \to X$ such that $i(a) = a$ for all
        $a \in A$ is called the \emph{inclusion} function of $A$ into $X$. Observe
        that $(f \circ i) : A \to Y$ and $(f \circ i) = f|_A$
        for any function $f : X \to Y$.
    \item Another important function is called the \emph{identity} function.
        Let $X$ be some set. Then $\identity{X} : X \to X$ is the identity
        function on $X$ iff $\identity{X}(x) = x$.
\end{itemize}
\nomenclature[F]{$\identity{A}$}{denotes the identity function on the set $A$}

\begin{theorem}
  Let $f : X \to Y$, $g : Y \to Z$, and $h : Z \to W$. Then
  \begin{itemize}
    \item $h \circ (g \circ f) = (h \circ g) \circ f$.
    \item $f \circ I_X = f = I_Y \circ f$.
  \end{itemize}
\end{theorem}
\begin{proof}
  These results can be proven simply by evaluating the functions. For example,
  both functions in the first equality assign $h(g(f(x)))$ for any $x \in X$
  and so functions are equal.
\end{proof}
Notice that this theorem states that we may write $f \circ g \circ h$ without
ambiguity.

\section{The Image of a Function}
Given a function $f : X \to Y$, it is not necessary that every element of $Y$
is an image of some $x \in X$. For example, the function $\R \to \R$ defined by
the formula $x^2$ does not have $-1$ as a value.

Thus we may give the following definition.
\begin{definition}
  The image of the function $f$ is defined as follows
  \[
    \Im f = \set[\exists x \in X\ f(x) = y]{y \in Y}
          = \set[x \in X]{f(x)}
  \]
  (in other words it is the projection of the graph $D$ of $f$ on the second
  coordinate: $\Im f = \set[(x, y) \in D]{y}$).
\end{definition}
\nomenclature[F]{$\Im f$}{denotes the image of $f$}

\begin{chapterendexercises}
  \exercise Is there $x, y, z \in \N$ such that $29x + 30y + 31z = 366$.
  \exercise Find then image of the function $f : \Z \to \Z$ such that
    $f(x) = 3x$.
  \exercise[recommended] Determine the following sets:
    \begin{itemize}
      \item $\set[\exists n \in \N \ m \le n]{m \in \N}$;
      \item $\set[\forall n \in \N \ m \le n]{m \in \N}$;
      \item $\set[\exists m \in \N \ m \le n]{n \in \N}$;
      \item $\set[\forall m \in \N \ m \le n]{n \in \N}$.
    \end{itemize}
  \exercise Prove or disprove the following statements.
    \begin{itemize}
      \item $\forall m, n \in \N \  m \le n$.
      \item $\exists m, n \in \N \ m \le n$.
      \item $\exists m \in \N \  \forall n \in \N \  m \le n$.
      \item $\forall m \in \N \  \exists n \in \N \  m \le n$.
      \item $\exists n \in \N \  \forall m \in \N \  m \le n$.
      \item $\forall n \in \N \  \exists m \in \N \  m \le n$.
    \end{itemize}
  \exercise[recommended]
    We call elements of the set $\set{0, 1}^n$ Binary strings of length
    $n$. Moreover, instead of $(c_1, \dots, c_n)$ we write $c_1, \dots, c_n$
    and we call $c_i$s characters.
    Show that all Binary strings of length $n$
    may be ordered such that every successive strings in this order are
    different only in one character.
    (For example, for $n = 2$ the order may be $00$, $01$, $11$, $10$.)
\end{chapterendexercises}


\section*{Chapter 7: Deriving a Generic Algorithm}

\paragraph{Exercise 7.1}
How many additions are needed to compute \texttt{fib0(n)}?

\begin{proof}[Answer]
Let $\alpha(n)$ be the number of additions needed to compute
\texttt{fib0(}$n$\texttt{)}. $\alpha(n)$ can be characterized by
the following recurrence relation:
$$
\alpha(n) =
\begin{cases}
    0 & \textrm{if } n \leq 1 \\
    1 + \alpha(n-1) + \alpha(n-2) & \textrm{if } n \geq 2
\end{cases}
$$

It can be shown by induction on $n$ that $\alpha(n) = F_{n+1} - 1$.
In fact, if $n \leq 1$, $\alpha(n) = 0 = F_{n+1} - 1$, since
by definition $F_1 = F_2 = 1$. For $n \geq 2$,
\begin{eqnarray*}
\alpha(n)  &=& 1 + \alpha(n-1) + \alpha(n-2) \\
           &=& 1 + (F_n - 1) + (F_{n-1} - 1) \\
           &=& (F_n + F_{n-1}) - 1 \\
           &=& F_{n+1} - 1
\end{eqnarray*}
Thus, the number of additions we seek is
$\alpha(n) = F_{n+1} -1 \in \Theta(\varphi^n)$, where $\varphi$ is the
golden ratio.
\end{proof}


\chapter{The Pigeonhole Principle}
\label{chapter:pigeonhole}
\marginurl{%
  The Pigeonhole Principle:\\\noindent
  Introduction to Combinatorics \#3
}{youtu.be/1D1Fa7WIUO8}

The principle we are going to discuss in this chapter is very simple, it states
that if you have more objects than boxes, then you cannot put all the objects into
boxes without putting two objects into the same box.

More formally the principle can be formulated as follows: if $n > m$, then any
function from $\range{n}$ to $\range{m}$ is not an injection. This simple
statement is famous in mathematics and called
\emph{the pigeonhole principle}\footnote{%
  The pigeonhole principle is also called the Dirichlet principle, after the
  German mathematician G. Lejeune Dirichlet, who demonstrated, using this
  principle, that there were at least two Parisians with the same number of
  hairs on their heads.
}.

\begin{theorem}[the pigeonhole principle]
  Let $X$ and $Y$ be some sets such that $\cardinality{X} > \cardinality{Y}$.
  Then for any function $f : X \to Y$ there are $x_0 \neq x_1 \in X$ such that
  $f(x_0) = f(x_1)$.
\end{theorem}
\begin{proof}
  The statement follows from
  Theorem~\ref{theorem:injections-surjections-inequalities}.
\end{proof}

This simple statement is very handy in combinatorics. For example, using this
statement one may prove that in any group of more than $12$ people there are
two people who were born in the same month.

Assume that there are $n$ people in the group and $n > 12$.
Consider the following function $f : \range{n} \to \range{12}$ such that $f(i) = j$ if the
$i$th person was born in $j$th month. Note that $f$ is not an injection since
$n > 12$ i.e. there are $i_0 \neq i_1$ such that $i_0$th and $i_1$th person are
born in the same month.

\begin{exercise}
  Show that among any group of five (not necessarily consecutive)
  integers, there are two with the same remainder when divided by $4$.
\end{exercise}

We may also prove that in any group of people there are two people who are
friends with the same number of people in the group.

Assume the number of people is $n$. It is easy to see that every person may
have at most $n - 1$ friends. Hence, we may define a function $f: \range{n} \to
\set{0, \dots, n - 1}$ such that $f(i)$ is equal to the number of friends in
this group of the $i$th person in this group.
We need to consider two cases.
\begin{itemize}
  \item If $\Im f \subseteq \range{n - 1}$, then
    $\cardinality{\range{n}} > \cardinality{\Im f}$ and $f$ is not an injection.
  \item Otherwise, note that it is not possible that $(n - 1) \in \Im f$
    because if there is a friend with no friends it is not possible that there
    is a friend who is friends with everyone. Hence,
    $f : \range{n} \to \set{0, 1, \dots, n - 2}$ and $f$ is not an injection.
\end{itemize}

\begin{theorem}[Erd\H{o}s–-Szekeres]
  Every sequence of $(r - 1)(s - 1) + 1$ distinct real numbers contains a
  subsequence of length $r$ that is increasing or a
  subsequence of length $s$ that is decreasing.
\end{theorem}
\begin{proof}
  Given a sequence of length $(r - 1)(s - 1) + 1$, label each number $x_i$ in
  the sequence with the pair $(a_i, b_i)$, where $a_i$ is the length of the
  longest increasing subsequence ending with $x_i$ and $b_i$ is
  the length of the longest decreasing subsequence ending with $x_i$.
  Each two numbers in the sequence are labeled with a different pair: if $i < j$
  and $x_i < x_j$ then $a_i < a_j$, and on the other hand if $x_i > x_j$ then
  $b_i < b_j$. But there are only $(r - 1)(s - 1)$ possible labels if $a_i$ is
  at most $r - 1$ and $b_i$ is at most $s - 1$, so by the pigeonhole principle
  there must exist a value of $i$ for which $a_i$ or $bi$ is outside this
  range. If $a_i$ is out of range then $x_i$ is part of an increasing sequence of
  length at least $r$, and if $b_i$ is out of range then $x_i$ is part of a
  decreasing sequence of length at least $s$.
\end{proof}

We can also use the pigeonhole principle to show that the lower bound from
\Cref{theorem:guess-one-out-of-many} is precise.
\begin{theorem}
  There is a $B$-decision tree $T$ such that $h(T) \le 9$ and
  $\mathrm{val}(T, S) \in S$ for all $S \in \binom{\range{1000}}{500}$.
\end{theorem}
\begin{proof}
  Let us fix some set $S \in \binom{\range{1000}}{500}$. Note that $S \cap
  \range{501} \neq \emptyset$. Therefore, the minimal element of $S$ belongs to
  $\range{501}$. Hence, using an algorithm similar to the algorithm from
  \Cref{chapter:structural-induciton}, we can find the minimal element of $S$
  using at most $\ceil{\log{501}} = 9$ questions.
\end{proof}
\nomenclature[U]{$\ceil{\alpha}$}{denotes the smallest integer greater than or
equal to $\alpha$}

\section{The Generalized Pigeonhole Principle}
One may generalize the pigeonhole principle in the following way.
If $N$ objects are placed into $k$ boxes, then there is at least one box
containing at least $\ceil{N / k}$ objects.
\begin{theorem}[the generalized pigeonhole principle]
\label{theorem:generalized-pigeonhole-principle}
  Let $X$ and $Y$ be some sets. Then for any function $f : \cardinality{X} \to
  \cardinality{Y}$ there are $x_1, \dots, x_\ell \in X$ such that
  \begin{itemize}
    \item $f(x_i) = f(x_j)$,
    \item $x_i \neq x_j$ for any $i \neq j \in \range{\ell}$, and
    \item $\ell \ge \ceil{\cardinality{X} / \cardinality{Y}}$, where
      $\ceil{\alpha}$ denotes the least integer greater than or equal to
      $\alpha$.
  \end{itemize}
\end{theorem}

Now we illustrate applications of this principle on some examples and prove the
statement in the next section.

Using this theorem we can prove that if we draw $9$ cards out of a deck of
cards, we are guaranteed that at least three of them are of the same suit.
Given that, there are $4$ suits in the deck, by pigeonhole principle if we put each card into
one of the four boxes according to their suits, one of the boxes should have
at least $\ceil{9 / 4} = 3$ cards.

Another example shows how the generalized pigeonhole principle can be applied
to an important part of combinatorics called Ramsey theory.

Assume that in a group of six people, each pair of individuals consists of two
friends or two enemies. One may prove that there are either three mutual
friends or three mutual enemies in the group.

Let $A$ be one of the six people; of the five other people in the group, there
are either three or more who are friends of $A$, or three or more who are
his enemies $A$. This statements follows from the generalized pigeonhole
principle since when five objects are divided into two sets, one of the sets
has at least $\ceil{5 / 2} = 3$ elements. Without loss of generality we may
suppose that $B$, $C$, and $D$ are friends of $A$. If any two of these three
individuals are friends, then these two and $A$ form a group of three mutual
friends. Otherwise, $B$, $C$, and $D$ form a set of three mutual enemies.

\section{The Averaging Principle}
Assume that we have a collection of $m$ objects, the $i$th of which has
``size'' $l_i$. We wish to show that at least one of the objects is large.
In this situation we can argue that at least one of the objects has size
greater or equal to the average size ($\sum l_i / m$).
\begin{theorem}[the averaging principle]
\label{theorem:averaging-principle}
  Every sequence of numbers has a number at least as large as the average and a
  number at least as small as the average; i.e. for any sequence $a_1$, \dots,
  $a_m$ there are $i$ and $j$ such that
  \begin{gather*}
    a_i \ge \frac{1}{m} \sum_{i = 1}^m a_i \\
    \text{and} \\
    a_j \le \frac{1}{m} \sum_{i = 1}^m a_i.
  \end{gather*}
\end{theorem}
\begin{proof}
  We prove only the existence of $i$, proof of the existence of $j$ is almost
  the same.

  Assume the opposite, i.e. that $a_i < \sum_{i = 1}^n a_i / m$
  for any $i \in \range{n}$. Note that this implies that
  $\sum_{i = 1}^n a_i \le m \cdot \sum_{i = 1}^n a_i / m = \sum_{i = 1}^n a_i$.
  Which is a contradiction.
\end{proof}

\begin{exercise}
  Finish the proof of Theorem~\ref{theorem:averaging-principle}
\end{exercise}

Like the pigeonhole principle, this principle is very simple but the
applications of it are surprisingly interesting.

First, it allows to prove the generalized pigeonhole principle.
\begin{proof}[Proof of Theorem~\ref{theorem:generalized-pigeonhole-principle}]
  Let $Y = \range{m}$ (it is easy to see that the proof works for any other finite
  $Y$). Define the sequence $a_i = \cardinality{f^{-1}(i)}$.
  Note that we need to prove that $a_i \ge \ceil{\cardinality{X} / m}$ for some
  $i \in \range{m}$

  It is clear that $\bigcup_{i = 1}^m f^{-1}(i) = X$ and that $f^{-1}(i) \cap
  f^{-1}(j) = \emptyset$ for any $i \neq j \in \range{m}$. Thus, by the additive
  principle, $\sum_{i = 1}^m a_i = \cardinality{X}$. Hence, by the averaging
  principle, $a_i \ge \cardinality{X} / m$ for some $i \in \range{m}$. However,
  $a_i$ is an integer, thus $a_i \ge \ceil{\cardinality{X} / m}$.
\end{proof}

Another nice application of the averaging principle allows us to prove that if
in some group (with more than one person) the number of pairs of people who
know each other is less than $n - 1$, then we can split this group into two
subgroups such that people from different subgroups do not know each other.

Let us assume that there are $n$ people in the group. We prove the statement
using the induction by $n$.
\begin{description}
  \item [(the base case)] If $n = 2$, there are less than $n - 1 = 1$ pairs
    of people who know each other, in other words, these two people in the
    group do not know each other. Thus we can put each of them into a separate
    subgroup.
  \item [(the induction step)] Let $p_i$ ($i \in \range{n}$) be the number of
    acquaintances of the $i$th person. Note that
    $\sum_{i = 1}^n p_i \le 2(n - 2)$ since we count each pair twice.
    By the averaging principle, $p_i \le 2(n - 2) / n = 2 - 2 / n$ for some
    $i \in \range{n}$.  Thus $p_i$ is either $0$ or $1$.
    \begin{itemize}
      \item If $p_i = 0$, we can put the $i$th person into the first subgroup
        and everyone else into another.
      \item If $p_i = 1$ we consider the group of $n - 1$ people without the
        $i$th person, by the induction hypothesis, we can split everyone but
        $i$th person into two subgroups and since the $i$th person has only one
        acquaintance we can put them in the same subgroup.
      \end{itemize}
\end{description}

\begin{chapterendexercises}
  \exercise Let $ABC$ be an equilateral triangle such that the length of $AB$ is
    equal to $1$, and let $p_1$, \dots, $p_5$ be points inside of $ABC$.
    Show that there are $i \neq j \in \range{5}$ such that the distance between
    $p_i$ and $p_j$ is at most $0.5$.
  \exercise We are given $17$ points inside a regular triangle of side of
    length $1$. Prove that two of these points have distance at most $1 / 4$.
    \begin{solution}
      In order to do it let us consider the following partition of the triangle.

      \begin{center}
        \begin{tikzpicture}[thick, scale=0.3]
          \draw (-5,0)
            -- (5,0)
            -- (0, 8.660)
            -- cycle;
          \draw (0,0)
            -- (-2.5,4.330)
            -- (2.5,4.330)
            -- cycle;
          \draw (-2.5,0)
             -- (-3.75,2.165)
             -- (-1.25,2.165)
             -- cycle;
          \draw (2.5,0)
             -- (3.75,2.165)
             -- (1.25,2.165)
             -- cycle;
          \draw (0,4.330)
             -- (-1.25,6.495)
             -- (1.25,6.495)
             -- cycle;
          \draw (0,4.330)
             -- (-1.25,2.165)
             -- (1.25,2.165)
             -- cycle;
        \end{tikzpicture}
      \end{center}

      Note that there are 16 small triangles inside, hence by pigeonhole
      principle there are at least two points in the same small triangle.
      Additionally, distance between any two points inside the regular triangle
      with side $1 / 4$ is at most $1 / 4$.
    \end{solution}
  \exercise Show that if there are 30 students in a class, then at least
    two have last names that begin with the same letter.
  \exercise[recommended] Let $n$ be a positive integer. Show that in any set of
    $n$ consecutive integers there is exactly one divisible by $n$.
    \begin{solution}
      Let the numbers be $k$, $k + 1$, \dots, $k + n - 1$. Assume that all the
      numbers are not divisible by $n$. Consider the function
      $f: \set{k, k + 1, \dots, k + n - 1} \to \set{1, \dots, n - 1}$ such that
      $f(k + i) \equiv k + i \pmod{n}$. Note that the set on the left has more
      elements than the set on the right, so there are $i_1 < i_2$ such that
      $k + i_1$ has the same reminder as $k + i_2$.
      Thus $n$ divides $0 < i_2 - i_1 < n$, which is a contradiction.
    \end{solution}
  \exercise[recommended] Prove that for every sequence of integers $a_1$, \dots,
    $a_n$ there are $k > 0$ and $\ell \ge 0$ such that $k + \ell \le n$ and
    $\sum_{i = k}^{k + \ell} a_i$ is divisible by $n$.
    \begin{solution}
      We consider the $n$ sums modulo $n$ of the form $a_1$, $a_1 + a_2$,
      \dots, $a_1 + a_2 + \dots + a_n$. First, we note that if any of these sums
      $a_1 + \cdots + a_\ell \equiv 0 \pmod{n}$, then we are done by picking $k
      = 1$ and $\ell$ accordingly since being equivalent to $0$ modulo $n$ is
      the same as being divisible by $n$. As a result, it suffices to show the
      result when none of the sums are equivalent to $0$ modulo $n$.

      In this case, each of these sums modulo $n$ necessarily must be one of
      $1$, $2$, \dots, $(n - 1)$. That is, there are $n - 1$ possible values for
      each of these $n$ sums. Therefore, by the pigeonhole principle, we must
      have that two of these sums are equivalent modulo $n$. Thus there exists
      $m > 0 $ and $j > 0$ (and without loss of generality may assume that $j >
      m$ ) so that
      \[
        a_1 + a_2 + \dots + a_m \equiv a_1 + a_2 + \cdots + a_j \pmod{n}.
      \]
      Note that subtracting $ a_1 + a_2 + \cdots + a_m$ from both sides yields
      that
      \[
        0 \equiv
        a_{m + 1} + \cdots + a_{j} \pmod{n},
      \]
      and hence by taking $k = m + 1$ and $\ell = j - m - 1 = j - k $ we prove that
      \[
        0 \equiv a_k + a_{k + 1} + \dots + a_{k + \ell} \pmod{n};
      \]
      in other words, $\sum\limits_{i = 0}^\ell a_{k + i}$ is divisible by $n$.
    \end{solution}
  \exercise[recommended] Let $S \subseteq \range{20}$ be a set. Show that if
    $\cardinality{S} \ge 13$, then there are $a, b \in S$ such that $a - b = 6$.
  \exercise Let $S \subseteq \range{20}$ be a set. Show that if $\cardinality{S}
    \ge 11$, then there are $a \neq b \in S$ such that $a + b = 21$.
    \begin{solution}
      Note that there are $10$ pairs $(1, 20)$, \dots, $(10, 11)$ such that their
      sum is equal to $21$. Hence, by the pigeonhole principle, there are two
      elements of $S$ such that they belong to the same pair; i.e. their sum is
      equal to $21$.
    \end{solution}
  \exercise How many numbers must be selected from the set $\range{6}$ to
    guarantee that at least one pair of these numbers add up to $7$?
    \begin{solution}
      Let us split the numbers from $\range{6}$ into $3$ pairs:
      $(1, 6)$, $(2, 5)$, and $(3, 4)$. Note that the sum within the pair is
      $7$. Is is also easy to see that, by the pigeonhole principle, if we pick
      $4$ numbers out of $[6]$, two of them are in the same pair. Therefore
      whenever we choose $4$ numbers two of them sum up to $7$.

      It is also easy to see that we may select $3$ numbers so that the sum of
      any two of them is not equal to $7$. For example, we may select $1$, $2$,
      and $3$.
    \end{solution}
  \exercise Sasha is training for a triathlon. Over a $30$ day period, he
    pledges to train at least once per day, and $45$ times in all. Then there
    will be a period of consecutive days where he trains exactly $14$ times.
  \exercise Show that among any $n + 1$ positive integers not exceeding $2n$
    there must be an integer that divides one of the other integers.
    \hint{Consider the set of holes equal to the set of odd numbers
    from $1$ to $2n$.}
    \begin{solution}
      Let $R = \set{1, 3, \dots, 2n - 1}$ (note that $\cardinality{R} = n$).
      Assume that there is a set $L$ that contradicts to the statement of the
      problem ($\cardinality{L} = n + 1$). It is easy to see that any number
      $x \in \range{2n}$ is equal to $2^k \ell$, where $\ell$ is odd. We define
      the function $f : L \to R$ such that $f(2^k \ell) = \ell$. By the
      pigeonhole principle, there are $x < y$ such that $f(x) = f(y)$. But it
      implies that $x$ divides $y$.
    \end{solution}
  \exercise Let $A_1, \dots, A_\ell \subseteq \range{n}$ be some sets such that
    $A_i \cap A_j \neq \emptyset$. Show that $\ell \le 2^{n - 1}$.
    \begin{solution}
      We are going to prove this statement using the pigeonhole principle. The
      main component of the proof is the fact that $A \cap \bar{A} = \emptyset$
      for any set $A \subseteq \range{n}$, where $\bar{A} = \range{n} \setminus A$.

      Let $L = \{A_1, \dots, A_\ell\}$ and
      $R = \set[A \subseteq \range{n}]{\set{A, \bar{A}}}$. It is easy to see
      that $\cardinality{R} = 2^{n - 1}$. Define the function $f : L \to R$ such that
      $f(A_i) = \set{A_i, \bar{A_i}}$. Let us prove that $f$ is an injection;
      assume the opposite i.e. that there are $i \neq j$ such that
      $f(A_i) = f(A_j)$. Since $f(A_i) = f(A_j)$ it implies that $A_i =
      \bar{A_j}$ but it contradicts to the statement that $A_i \cap A_j =
      \bar{A_j} \cap A_j = \emptyset$.

      Hence, there is an injection from $L$ to $R$ which implies that
      $\cardinality{L} \le \cardinality{R}$.
      As a result, we proved that $\ell \le 2^{n - 1}$.
    \end{solution}
  \exercise[recommended] Let $a_1$, $a_2$, \dots, $a_t$ be positive integers. Show that
    if $a_1 + a_2 + \dots + a_t - t + 1$ objects are placed into $t$ boxes,
    then for some $i \in \range{t}$, the $i$th box contains at least $a_i$ objects.
    \hint{It is important in this question that $a_1$, \dots, $a_t$ are
    integers.}
  \exercise Let $\set{(x_1,y_1), \dots, (x_5, y_5)} \subseteq \Z^2$ be a
    set of five distinct points with integer coordinates in the xy plane. Show
    that the midpoint of the line joining at least one pair of these points has
    integer coordinates.
  \exercise Let $S \subseteq \range{2n}$ be a set such that
    $\cardinality{S} = n + 1$. Show
    that there are $x \neq y \in S$ such that $x$ and $y$ are coprime.
  \exercise[recommended] In some school there are three clubs. For each two
    students there is a club such that these students are  members of this club.
    Show that there is a club that at least $2 / 3$ of all students are members
    of this club.
  \exercise Show that the set $\set{7, 77, 777, \dots}$ contains a number
    divisible by $2019$.
  \exercise What is the maximal number of chess knights one may put on a
    chessboard so that they would not attack each other?
  \exercise[recommended] Let us assume that we are given $\ell$ lines that are
    not parallel to each other. Prove that there are at least two of them such
    that angle between them is at most $\pi / \ell$.
    \begin{solution}
      Proof of this statement is very similar to the proof of the averaging
      principle.
      Let us move all the lines (using parallel shift) such that all of them are
      going through $(0, 0)$. Let us denote angles between lines (in clockwise
      order) $\alpha_1$, \dots, $\alpha_{2\ell}$ respectively and assume that
      all of them are greater than $\pi / \ell$. In this case we may note that
      $\sum\limits_{i = 1}^{2\ell} \alpha_i > 2\ell \cdot \pi / \ell = 2\pi$,
      but we know that $\sum\limits_{i = 1}^{2\ell} \alpha_i  = 2\pi$.
    \end{solution}
\end{chapterendexercises}