\chapter{Gaussian processes}

\section{Parametric and nonparametric regression}
Consider the standard regression problem: given $\mathcal{D} = \{(x_i,y_i)\}_{i=1}^N$, with $x_i \in \mathcal{X}$ and $y_i \in \mathbb{R}$, one wants, for a $x_* \in \mathcal{X}$, find $p(y_*|x_*)$. If we assume a model $M$ for $y|x$, for all $x \in \mathcal{X}$, parameterized by $\theta$, then by the Bayesian view one should marginalize over $\theta$ resulting in
\begin{equation}
 p(y_*|x_*,M,\mathcal{D}) = \int_{\Theta} p(y_*|x_*,\theta,M,\mathcal{D}) p(\theta|\mathcal{D}) d\theta,
\end{equation}
reducing the problem to one of posterior inference in $\theta$, discussed in the previous chapter (equation \eqref{marginalizationpred}).

Since the model $M$ is parameterized, in a sense the model will be always limited, since there cannot be a mapping from finite parameters to all distributions. Nonparametric models by contrast are models that cannot be parameterized by a finite set of parameters, and Bayesian nonparametric regression is when a nonparametric model is used in the Bayesian setting \cite{Ghahramani_2013,Hjort_2010}.

\section{Gaussian process regression}
Bayesian nonparametric regression seems to be an impossible task, since it requires working with distributions in infinite-dimensional spaces. However, Gaussian process regression \cite{Rasmussen06} does this, by choosing a suitable model, given by a Gaussian process
\begin{Definition}
	A \textit{Gaussian process} (GP) is a distribution over the space of functions from $\mathcal{X}$ to $\mathbb{R}$ such that, for each $\mathbf{x} = (x_1,...,x_N) \in \mathcal{X}^N$, $\mathbf{f} := (f(x_1),...,f(x_N))$ follows a multivariate normal distribution \cite{Rasmussen06}.
\end{Definition}

A GP is completely specified by a \textit{mean function}
\begin{displaymath}
m : \mathcal{X} \to \mathbb{R}
\end{displaymath}
and a \textit{covariance function}
\begin{displaymath}
k : \mathcal{X} \times \mathcal{X} \to \mathbb{R},
\end{displaymath}
in such a way that, for $\mathbf{x} = (x_1,\ldots,x_N)$, letting
\begin{equation}
\mathbf{m}(\mathbf{x}) := (m(x_1),...,m(x_N))^T \quad K(\mathbf{x},\mathbf{x}') = \bigg(k(x_i,x_j')\bigg)_{i,j},
\end{equation}
then $f(\mathbf{x}) \sim \mathcal{N}(\mathbf{m}(\mathbf{x}),K(\x,\x))$.

Notice that this requires the function $k$ to be a \textit{positive-semidefinite} (PSD) function, that is, for any $N \in \mathbb{N}$, for any $\mathbf{x} = (x_1,...,x_N) \in \mathcal{X}^N$, the matrix $K(\mathbf{x},\mathbf{x})$ defined by $(K(\mathbf{x},\mathbf{x}))_{i,j} = k(x_i,x_j)$ is PSD. Conversely, any pair $(m,k)$, with $k$ PSD defines a GP \cite{Dudley_2002}, thus ensuring a correspondence between GPs and $(m,k)$ pairs as above. In the context of GPs, $k$ is also called an \textit{kernel}. \footnote{PSD functions have the property that, for $x,x' \in \mathcal{X}$, $k(x,x') = \langle \Phi(x), \Phi(x') \rangle_{\mathcal{H}}$, where $\Phi : \mathcal{X} \to \mathcal{H}$ is a map from $\mathcal{X}$ to a Hilbert space $\mathcal{H}$ \cite{Shawe_Taylor_2004}. In this context, $k$ is called a \textit{kernel function}, and machine learning techniques that uses them are called kernel methods \cite{Shawe_Taylor_2004}. Hence, in the context of GPs, the terms \textit{kernel function} and {covariance function} are used interchangeably.}

A \textit{Gaussian process regression} is done, with $\mathcal{D} = \{(x_i,y_i)\}_{i=1}^N$, by using the model $M$ that says, for each $x$, $p(y|x,M) = p(y|f(x),M)$, with the prior for $f$ being distributed as $GP(m,k)$. To see that this prior is attractive, assume for now $y = f(x)$. By letting $\mathbf{x} = (x_1,\ldots,x_N)$ \footnote{Usually, $x_i \in \mathbb{R}^D$, and $\mathbf{x}$ is written as a matrix whose $i$-th row is $x_i$. In this case, we may use $\mathbf{X}$ to denote this matrix instead, and reserve $\mathbf{x}$ to denote each point in $\mathbb{R}^D$. We will however use a more general notation.} and $\mathbf{y} = (y_1,\ldots,y_N)^T$, and assuming $K(\mathbf{x},\mathbf{x})$ to be non-singular, one can find the posterior distribution for $y_*|x_*,\mathcal{D},M$ without resorting to Bayes' rule, which is important since, in infinite-dimensional spaces, Bayes' rule is more involved, and may not result in a computable expression by itself \cite{Kanagawa_2018}. \footnote{For a derivation of Bayes' rule for infinite-dimensional spaces (which involves measure theory), see \cite{Stuart_2010}, Section 6.6.}

Too see why is this, notice that, by the definition of a GP, for any other $\mathbf{x}^\star \in \mathcal{X}^M$,
\begin{equation} \label{jointGP}
\left[ \begin{array}{c}
f(\mathbf{x}) \\
f(\mathbf{x}^*) \end{array} \right] \sim \mathcal{N}
\left( \left[ \begin{array}{c}
\mathbf{m}(\mathbf{x}) \\
\mathbf{m}(\mathbf{x}^*) \end{array} \right] ,
\left[ \begin{array}{c c}
K(\mathbf{x},\mathbf{x}) & K(\mathbf{x},\mathbf{x}^*) \\
K(\mathbf{x}^*,\mathbf{x}) & K(\mathbf{x}^*,\mathbf{x}^*) \end{array} \right]
\right).
\end{equation}
Hence, by conditioning on $f(\mathbf{x})$, by Appendix \ref{appendixconditional}:
\begin{equation}\label{meancovGPRpure}
\begin{split}
& f(\mathbf{x}^*) | \mathbf{x}^\star, \mathcal{D}, M = \mathbf{f}^*|\mathbf{f},M \sim \mathcal{N}(\mu^\star,\Sigma^*) \\
& \mu^\star = \mathbf{m}(\mathbf{x}^\star) + K(\mathbf{x}^\star,\mathbf{x}) K(\mathbf{x},\mathbf{x})^{-1} (f(\mathbf{x}) - m(\mathbf{x})) \\
& \Sigma^\star = K(\mathbf{x}^\star,\mathbf{x}^\star) - K(\mathbf{x}^\star,\mathbf{x}) K(\mathbf{x},\mathbf{x})^{-1} K(\mathbf{x},\mathbf{x}^\star).
\end{split}
\end{equation}
Since $\x^*$ was chosen arbitrarily, this implies that $f|\mathcal{D},M$ itself follows a GP, with mean function and covariance functions given by:
\begin{equation}\label{GPRasGP}
\begin{split}
& m_{\mathcal{D}}(x) = \mu^*(x) = m(x) + K(x,\mathbf{x}) K(\mathbf{x},\mathbf{x})^{-1} (f(\mathbf{x}) - m(x)) \\
& k_{\mathcal{D}}(x,x') = k(x,x') - K(x,\mathbf{x}) K(\mathbf{x},\mathbf{x})^{-1} K(\mathbf{x},x').
\end{split}
\end{equation}
Since $y_* = f(x_*)$ and $\mathbf{y} = f(\mathbf{x}^*)$, $y_*|x_*,\mathcal{D},M \sim \mathcal{N}(m_\mathcal{D}(x_*),k_\mathcal{D}(x_*,x_*))$. An illustration of GP regression is shown if Figure \ref{gprfig}.

\begin{figure}
	\centering
	\subfloat[GP prior]{\label{gprex1a}\includegraphics[width=0.45\textwidth]
		{figs/gprex1a.png}}
	\subfloat[GP posterior]{\label{gprex1b}\includegraphics[width=0.45\textwidth]
		{figs/gprex1b.png}}

	\caption[Gaussian process regression of $f(x) = \sin(\pi x)$]{\label{gprfig}Gaussian process regression of $f(x) = \sin(\pi x)$. In (a), it is shown the prior space distribution, with samples path in blue and confidence interval in red. In (b), the posterior space distribution, after 4 measurements (in green) of $f(x)$ (black). Generating code can be found in \url{https://github.com/DFNaiff/Dissertation/blob/master/illustrations_dissertation/gp_prior_posterior}.}
\end{figure}

\subsection{Gaussian noise}
This equation can be generalized by assuming $p(y|x,M) = \mathcal{N}(y|f(x),\sigma^2_n)$. Then, letting $\epsilon \sim \mathcal{N}(0,\sigma^2_n)$, $\mathbf{y} = f(\mathbf{x})$, $\mathbf{y}^* = f(\mathbf{x}^*)$ and
\begin{equation} \label{jointGPnoise}
\left[ \begin{array}{c}
\mathbf{y} \\
\mathbf{y}^* \end{array} \right] \sim \mathcal{N}
\left( \left[ \begin{array}{c}
\mathbf{m}(\mathbf{x}) \\
\mathbf{m}(\mathbf{x}^*) \end{array} \right] ,
\left[ \begin{array}{c c}
K(\mathbf{x},\mathbf{x}) + \sigma_n^2 I & K(\mathbf{x},\mathbf{x}^*) \\
K(\mathbf{x}^*,\mathbf{x}) & K(\mathbf{x}^*,\mathbf{x}^*) + \sigma^2_n I \end{array} \right]
\right).
\end{equation}
Conditioning $\mathbf{y}^*$ on $\mathbf{y}$ and, letting $K_\sigma(\mathbf{x},\mathbf{x}) := K(\mathbf{x},\mathbf{x}) + \sigma_n I$, we have
\begin{equation}\label{meancovGPR}
\begin{split}
& \mathbf{y}^* | \mathbf{x}^\star, \mathcal{D}, M = \mathbf{y}^*|\mathbf{y},M \sim \mathcal{N}(\mu^\star,\Sigma^*) \\
& \mu^\star = \mathbf{m}(\mathbf{x}^\star) + K(\mathbf{x}^\star,\mathbf{x}) K_\sigma(\mathbf{x},\mathbf{x}){-1} (\mathbf{y} - m(\mathbf{x})) \\
& \Sigma^\star = K(\mathbf{x}^\star,\mathbf{x}^\star) - K(\mathbf{x}^\star,\mathbf{x}) K_\sigma(\mathbf{x},\mathbf{x})^{-1} K(\mathbf{x}^\star,\mathbf{x}) + \sigma_n I.
\end{split}
\end{equation}
Notice \eqref{meancovGPR} reduces to \eqref{meancovGPRpure} when $\sigma^2_n = 0$.
\subsection{General noise}
In the general case where $p(y|x,M)$ = $p(y|f(x))$, there is not a closed form solution and one must resort to explicit marginalization and Bayes' rule:
\begin{equation}\label{generalnoise}
\begin{split}
 & p(y_*|x_*,\mathcal{D}) = \\  & =
  \int p(y_*|f(x_*)) p(f(x_*)|x_*,\mathbf{x},\mathbf{y}) df(x_*)\\ & =
  \int p(y_*|f(x_*)) \int p(f(x_*)|x_*,f(\mathbf{x}), \mathbf{x}) p(f(\mathbf{x}) | \mathbf{x},\mathbf{y}) df(\mathbf{x}) df(x_*) \\ & \propto
  \int p(y_*|f(x_*)) \int p(f(x_*)|x_*,f(\mathbf{x}), \mathbf{x}) p(\mathbf{y}|f(\mathbf{x})) p(f(\mathbf{x})|\mathbf{x}) df(\mathbf{x})df(x_*).
 \end{split}
\end{equation}
Since $p(f(\mathbf{x})|\x) = \mathcal{N}(f(\mathbf{x})|m(\mathbf{x}),K(\mathbf{x},\mathbf{x}))$ and $p(f(x_*)|x_*,f(\mathbf{x}), \mathbf{x})$ is given by \eqref{meancovGPRpure}, these two terms may be joined together by \eqref{productgaussians}. However, this still leaves a double integral, that must be treated by approximate inference methods.

\subsection{Mean function}
Usually, the mean function $m$ is set to zero, letting the covariance function determine the whole structure of the regression. This is a reasonable assumption since, for any $f \sim GP(m,k)$, due to the sum of a Gaussian distribution and a constant being itself Gaussian with the constant added to its mean, we have $f-m \sim GP(0,k)$. Thus, fixed the model $(m,k)$, one can then do the GP regression on $f-m$ and then later add $m$. This is particularly useful if it is assumed that $f$ is modeled by some function $m$ that is known to be an incomplete model, thus complementing the regression by modeling this incompleteness by a zero mean GP.

\section{Covariance functions}

As said in the previous section, covariance functions $k$ must be PSD. This raises the question on which kind of functions are PSD, thus able to define a GP. A few functions can be easily shown to be PSD directly by their definitions, such as:
\begin{itemize}
	\item The constant function $k(x,x') = c \geq 0$, since the matrix $K_{i,j} = c$ is PSD, for all $c \geq 0$
	\item $k(x,x') = \mathbb{I}_{x = x'}$, since the corresponding matrix $K$ is the identity matrix
	\item If $\mathcal{X} = \{x_1,...,x_N\}$  is a finite set of size $N$, and $k$ is such that, for $\mathbf{x} = (x_1,...,x_N)$,
	$K(\mathbf{x},\mathbf{x})$ is PSD, then $k$ is PSD, since for any other subset of $\mathcal{X}$, the corresponding kernel matrix will be a subset of $K$, thus also PSD \cite{hogben14}.
	\item If we have an explicit feature map $\Phi : \mathcal{X} \to \mathbb{R}^N$, then $k(x,x') = \langle \Phi(x),\Phi(x') \rangle$ is PSD \cite{Shawe_Taylor_2004}. For instance, if $\mathcal{X} = \mathbb{R}$ and $\Phi(x) = (x,x^2,\ldots,x^N)$, then $k(x,x') = \sum_{i=1}^N (x x')^i$ is PSD.
\end{itemize}
However, for many covariance functions, direct proof of being PSD is infeasible. However, there are some covariance functions that can be shown to be PSD in a indirect way, as shown below.
\subsubsection{Stationary covariance functions}
\begin{Definition}
	Let $\mathcal{X} = \mathbb{R}^d$. An covariance function is \textit{stationary} if $k(x,x') = k(x - x')$ ,
	for $k : \mathbb{R}^d \to \mathbb{R}$ \footnote{Here we overload the notation, letting the reader infer whether $k$ refers to a covariance function or an autocavariance function by the number of its arguments.}. Conversely, $k$ is a \textit{autocovariance function} if $k(x,x') = k(x - x')$ is a covariance function.
\end{Definition}
For this class of functions, we can reduce the analysis of $k$ to that of $k$. In particular, the next theorem says that one can analyze the Fourier transform of $k$ to check if it is an autocovariance function, thus $k$ being a covariance function (here, it is conveniente to consider $k$ as a function into $\mathbb{C}$).

\begin{Theorem}[Bochner's Theorem]
	A function $k : \mathbb{R}^d \to \mathbb{C}$, continuous at $0$, is an autocovariance function if and only if
	\begin{equation}
	k(\tau) = \int_{\mathbb{R}^D} e^{2 \pi i \mathbf{s}^T \tau} d \mu (\mathbf{s}),
	\end{equation}
	where $\mu$ is a positive finite measure \cite{Stein_1999,Rasmussen06}.
	If $\mu$ has a density $S$, then $S$ an $k$ are Fourier duals of each other \cite{Chatfield_2004}.
	\begin{displaymath}
	\begin{split}
	k(\tau) & = \int_{\mathbb{R}^D} e^{2 \pi i \mathbf{s}^T \tau} S(\mathbf{s}) d\mathbf{s} \\
	S(\mathbf{s}) & = \int_{\mathbb{R}^D} e^{-2 \pi i \mathbf{s}^T \tau} k(\tau) d \tau.
	\end{split}
	\end{displaymath}
	In this case, $S$ is called the \text{spectral density} corresponding to $k$.
\end{Theorem}

One particular case of stationary functions are \textit{isotropic} functions, in which $k(\tau)$ is a function of $r = ||\tau||_2$. In this case, $S(\mathbf{s})$ is a function of $s = ||\mathbf{s}||_2$ \cite{CIS-4647}. For simplicity, those will also be referred as $k$ ans $S$. We show here some examples of isotropic covariance functions, along with their spectral densities (many others can be found in \cite{Rasmussen06}):
\begin{itemize}
	\item The squared exponential (SQE) kernel, also called RBF kernel
	\begin{equation}\label{sqekernel}
	k_{SQE}(r;l) = \exp \left(-\frac{r^2}{l^2}\right),
	\end{equation}
	whose spectral density is given by a normal distribution
	\begin{equation}
	S(s;l) = (2 \pi l^2)^{D/2} \exp(-2 \pi^2 l^2 s^2)
	\end{equation}
	The squared exponential kernel is the most widely used in the field of Gaussian process, and kernel methods in general. However, the squared exponential kernel generates functions that are infinitely differentiable, thus being far too smooth for some applications. Moreover, the resulting kernel matrix tends to be very ill-conditioned, which results in numerical issues in applications with low noise.
	\item The \text{Matérn} class of kernels, parameterized by $\nu > 0$, given by
	\begin{equation}\label{maternkernel}
	k_{Matern,\nu}(r;l) = \frac{2^{1-\nu}}{\Gamma(\nu)} \left(\frac{\sqrt{2 \nu} r}{l} \right)^\nu
	K_\nu \left( \frac{\sqrt{2 \nu} r}{l} \right),
	\end{equation}
	where $K_\nu$ is the modified Bessel function of second kind. The corresponding spectral density
	is a $2 \nu$-degreed multivariate t-distribution
	\begin{equation}
	S_\nu(s;l) = 2^D \pi^{D/2} \frac{\Gamma(\nu + D/2)(2 \nu)^\nu}{\Gamma(\nu)l^{2 \nu}}\left(\frac{2 \nu}{l^2} +
	4 \pi^2 s^2\right)^{-(2 \nu+D)/2}.
	\end{equation}
	If $\nu$ is a half-integer, the kernel formula is simplified to a product of a polynomial of order $\nu - 1/2$ and an exponential. The most commonly used values of $\nu$ are:
	\begin{itemize}
		\item $\nu = 1/2$, giving $k_{Matern,1/2}(s;l) = \exp(-r/l)$.
		\item $\nu = 3/2$, giving $k_{Matern,3/2}(s;l) = \left(1 + \sqrt{3} r/l \right) \exp(-\sqrt{3}r/l)$.
		\item $\nu = 5/2$, giving $k_{Matern,5/2}(s;l) = \left(1 + \sqrt{5} r/l + 5 r^2/l^2 \right) \exp(-\sqrt{5}r/l)$.
	\end{itemize}
	In the limit $\nu \to \infty$, the Matern kernel converges to the squared exponential kernel \cite{Stein_1999}. In practice, for values of $\nu \geq 7/2$, the Matern kernel is similar enough to the squared exponential kernel to be of use, thus in practice only the three values of $\nu$ shown above are used.
	\item The spectral mixture kernel \cite{wilson2013gaussian}
	\begin{equation}\label{spectralmixturekernel}
	k_{SM}(\tau) = \sum_{q=1}^Q w_q \prod_{d=1}^D \exp(-2 \pi^2 \tau_d^2 v_{q}^{(d)})
	\cos(2 \pi \tau^{(d)} \mu_q^{(d)}),
	\end{equation}
	which is constructed explicitly as the Fourier dual of mixtures of multivariate normal densities. In \cite{wilson2013gaussian}, it is argued that the spectral mixture kernel approximates many of the kernels in \cite{Rasmussen06}, given enough mixtures $Q$.
\end{itemize}

\subsection{Derived kernels}
Although PSD functions are relatively hard to find, even with the use of Bochner's Theorem, one can prove that many compositions of base PSD functions are themselves PSD, thus providing many new classes of kernels.
Given $\mathcal{X}$ an arbitrary set, $k_1$,$k_2$ PSD functions on $\mathcal{X}$, and $k_3$ a PSD function on a set $\mathcal{Y}$, we have:
\begin{itemize}
	\item $k(x,x') := k_1(x,x') + k_2(x,x')$ is a PSD function, since, for $\mathbf{x} = (x_1,...,x_N) \in \mathcal{X}^N$, $K(\mathbf{x},\mathbf{x}) = K_1(\mathbf{x},\mathbf{x}) + K_2(\mathbf{x},\mathbf{x})$, the sum
	of two PSD matrices, hence a PSD matrix. Similarly, $k(x,x') := k_1(x,x') k_2(x,x')$ is a PSD function, since $K(\mathbf{x},\mathbf{x}) = K_1(\mathbf{x},\mathbf{x}) \odot K_2(\mathbf{x},\mathbf{x})$ (where $A \odot B$ denotes the Hadamard product between $A$ and $B$), and, according to Schur product theorem \cite{hogben14}, $K(\mathbf{x},\mathbf{x})$ is also PSD as the Hadamard product of two PSD matrices.
	\item $k([x,y],[x',y']) = k_1(x,x') + k_3(y,y')$ is a PSD function, and so is $k([x,y],[x',y']) = k_1(x,x') k_3(y,y')$. This follows from the fact that both sum and Hadamard product of PSD matrices are PSD.
	\item For a map $f : \mathcal{Y} \to \mathcal{X}$, $k(y,y') = k_1(f(x),f(x'))$ is aPSD. This follows directly from the fact that if $y_i = f(x_i), \ i=1,...,n$,
	$K(\mathbf{y},\mathbf{y}) = K_1(f(\mathbf{x}),f(\mathbf{x}'))$. This property allows us to construct non-stationary kernels from stationary kernels, by using input warping functions. Moreover,
	this implies that we can substitute $r = ||\mathbf{x} - \mathbf{x}'||_2$ for $r = \sqrt{(\mathbf{x} - \mathbf{x})^T A^{-1} (\mathbf{x} - \mathbf{x})}$, where $A$ is a positive definite matrix, by setting $f(x) = A^{-1/2} x$
\end{itemize}
The last item allows us to construct kernels with general \textit{outputscale} and \textit{lengthscale} from stationary kernels. That is, if $k_0$ is an autocovariance function such that $k_0(0) = 1$, then
\begin{equation}\label{scaledkernels}
k(x-x') = \theta k\left(\frac{x_1-x'_1}{l_1},\ldots,\frac{x_D-x'_D}{l_D}\right)
\end{equation}
is a kernel with outputscale $\theta$ and lengthscales $l_1,\ldots,l_D$. We call such kernels \textit{anisotropic} (although strictly speaking, every non-isotropic kernel is anisotropic).

\section{Model selection}
In the above discussion, the model $M = (m,k)$ was assumed to be fixed. In practice, since we have many different kernel functions, each parameterized by a continuous set of parameters (called \textit{hyperparameters}), we need a way to choose the correct model. In the noisy measurement case, we also need to deal with the noise distribution parameters. Fortunately, the Bayesian framework gives a natural way to choose the model.

Assume $p(y|f(x)) = \mathcal{N}(f(x),\sigma_n^2)$. Then,
\begin{displaymath}
\mathcal{D}|M,\sigma_n = \mathbf{y} | \mathbf{x},M,\sigma_n \sim
\mathcal{N}(m(\mathbf{x}),K(\mathbf{x},\mathbf{x}) + \sigma_n \mathbf{I}).
\end{displaymath}
Therefore, the likelihood for the model is given by
\begin{equation}\label{loglikelihoodGP}
\begin{split}
\log p(\mathcal{D}|M,\sigma_n) = & -\frac{1}{2}(\mathbf{y} - m(\mathbf{x}))^T (K(\mathbf{x},\mathbf{x}) + \sigma_n \mathbf{I})^{-1} (\mathbf{y} - m(\mathbf{x})) + \\
&-\frac{1}{2} \log \det (K(\mathbf{x},\mathbf{x}) + \sigma_n \mathbf{I}) - \frac{1}{2} N \log(2\pi).
\end{split}
\end{equation}
The important thing to notice is that the likelihood above is actually a marginal likelihood, since
\begin{equation}
p(\mathcal{D}|M,\sigma_n) = \Ev_{f \sim GP(m,k)} \left[p(\mathcal{D}|f,\sigma_n) \right].
\end{equation}
Hence, it should display the Occam's razor effect. In fact, the log-determinant term does exactly this, acting as a sort of regularizer. However, this does not mean that GP regression is protected from overfitting (see \cite{Mohammed_2017}). Moreover, since the objective function is non-convex, there may be local optima that returns spurious results.

A fully Bayesian approach to GP regression is desirable in order to incorporate fully the hyperparameter knowledge. However, when the number of data is considerably large, Monte Carlo methods becomes inefficient (although it can be used still, see for example \cite{Neal_1997,Petelin_2014}). In \cite{Osborne_2007}, an approach based on Bayesian Monte Carlo (to be presented in Chapter 4) is also explored. However, efficient marginalization of hyperparameters remains an open problem.

\section{Computational issues}
\subsection{Jittering}
If the noise $\sigma_n^2$ is zero, the matrix $K(\mathbf{x},\mathbf{x})$ may be ill-conditioned, which is usually the case for the SQE kernel. One way to mitigate this problem is to force the existence of an "artificial noise" on $K(\mathbf{x},\mathbf{x})$, that is, one substitute it for $K(\mathbf{x},\mathbf{x}) + \sigma^2_j I$, where $\sigma^2_j$ is not a real noise parameter now, but just an stabilizer. In this case, the error caused by the addition of artificial noise is considerably smaller than the error of numerical operations in ill-conditioned matrices, if they are able to be performed at all. The Cholesky decomposition (described below) also helps with the stability of inverse matrix operations.

\subsection{Scaling with data}\label{scalinggpsession}
The main issue with GP regression is that, given $N$ training points, for a fixed covariance function the evaluations in \eqref{meancovGPR} requires at least one operation with the inverse of a $N \times N$ matrix $K(\mathbf{x},\mathbf{x} + \sigma_n \mathbf{I})$, whose computational cost is of order $\mathcal{O}(N^3)$. The problem is worsened in the case of training a model, since this operation has to be done for each evaluation of $\log p(\mathcal{D}|M,\sigma_n,\theta)$ while training.

Since $K(\mathbf{x},\mathbf{x}) + \sigma_n \mathbf{I}$ is a positive definite matrix, one can try to mitigate the computational cost by computing the Cholesky decomposition
\begin{displaymath}
K(\mathbf{x},\mathbf{x}) + \sigma_n \mathbf{I} = L L^T.
\end{displaymath}
Then, given the decomposition, the inverse operations involve inverses of triangular matrices, whose operations costs are of order $\mathcal{O}(N^2)$, while the determinant term in $\log p(\mathcal{D}|M,\sigma_n,\theta)$ can be calculated as $\log \det (L L^T) = 2 \sum_i \log L_{ii}$. However, Cholesky decomposition, although faster than other methods like the LU decomposition, still has a computational cost of $\mathcal{O}(N^3)$, hence the scaling problem still exists.

\section{Online learning}\label{onlinelearningsection}
One interesting aspect of GPs is its ability to accumulate online data in a relatively simple manner, provided we do not change its hyperparameters. Consider fixed an initial data $\mathcal{D} = \{(x_i,y_i)\}_{i=1}^N$, and a GP model $(m,k)$, if we have a kernel matrix $K_{\mathcal{D}}$, and its Cholesky factor $L_{\mathcal{D}}$, resulting in a readily accessible posterior mean function $m_\mathcal{D}(x)$ and covariance function $k_\mathcal{D}(x,x')$. Now, suppose some new data $\mathcal{D}' = \{(x'_j,y'_j)\}_{j=1}^M$ is available, and the practitioner wants to incorporate into a new posterior mean $m_{\mathcal{D} \cup \mathcal{D}'}(x,x')$ and covariance $k_{\mathcal{D} \cup \mathcal{D}'}(x,x')$. A naive manner for doing this would be constructing a new kernel matrix $K_{\mathcal{D} \cup \mathcal{D}'}$ from scratch, and compute its Cholesky factor $L_{\mathcal{D} \cup \mathcal{D}'}$, resulting in a operation cost $\mathcal{O}((M + N)^3)$. Fortunately, there is a clever way to obtain $L_{\mathcal{D} \cup \mathcal{D}'}$ from $L_\mathcal{D}$ with $\mathcal{O}(M^3 + MN^2)$ cost (assuming $K_{\mathcal{D} \cup \mathcal{D}'}$ stays positive-definite). The following argument is adapted from \cite{Osborne_2012}, where it is considered the upper Cholesky factor.

To see this, consider $\mathbf{x}_N = (x_i)_{i=1}^N$, $\mathbf{x}_M = (x'_j)_{j=1}^M$ and $\mathbf{x} = \mathbf{x}_N \cup \mathbf{x}_M$ (here $\cup$ denotes concatenation). Then, noting $K_\mathcal{D} = K(\x_N,\x_N)$, we have
\begin{equation}
 K_{\mathcal{D}\cup\mathcal{D}'} = \left[ \begin{array}{cc}
K(\x_N,\x_N) & K(\x_N,\x_M) \\
K(\x_M,\x_N) & K(\x_M,\x_M) \end{array} \right].
\end{equation}
Then $L_{\mathcal{D}\cup\mathcal{D}'}$ must be of the form
\begin{equation}\label{formatcholexpanded}
L_{\mathcal{D}\cup\mathcal{D}'} = \left[ \begin{array}{cc}
L_\mathcal{D} & 0 \\
S & \tilde{L} \end{array} \right],
\end{equation}
with $\tilde{L}$ being lower triangular. This is because
\begin{equation}
\begin{split}
L_{\mathcal{D}\cup\mathcal{D}'} L_{\mathcal{D}\cup\mathcal{D}'}^T = \left[ \begin{array}{cc}
L_\mathcal{D} & 0 \\
S & \tilde{L} \end{array} \right] \left[ \begin{array}{cc}
L_\mathcal{D}^T & S^T \\
0 & \tilde{L}^T \end{array} \right] & = \left[ \begin{array}{cc}
L_\mathcal{D} L_\mathcal{D}^T & L_\mathcal{D} S^T \\
S L_\mathcal{D}^T & S S^T + \tilde{L} \tilde{L}^T \end{array} \right] \\
 & = \left[ \begin{array}{cc}
K(\x_N,\x_N) & K(\x_N,\x_M) \\
K(\x_M,\x_N) & K(\x_M,\x_M) \end{array} \right] = K_{\mathcal{D}\cup\mathcal{D}'}.
\end{split}
\end{equation}
This readily shows not only that $L_{\mathcal{D}\cup\mathcal{D}'}$ must be of the format in \eqref{formatcholexpanded}, but it gives a way to get $S$ and $\tilde{L}$: calculate $S= \big( L_\mathcal{D}^{-1} K(\x_N,\x_M) \big)^T$, and $\tilde{L}$ is the lower Cholesky factor of $K(\x_M,\x_M) - SS^T$, whose operations are of cost $\mathcal{O}(N^2 M)$ and $\mathcal{O}(M^3)$.


\problemname{Purple Rain}

\illustration{0.2}{purplerain/problem_statement/purplerain.png}

Purple rain falls in the magic kingdom of Linearland which is a
straight, thin peninsula.

On close observation however, Prof. Nelson Rogers finds that actually
it is a mix of Red and Blue drops.

In his zeal, he records the location and color of the raindrops in
different locations along the peninsula.
Looking at the data, Professor Rogers wants to know which part of
Linearland had the ``least'' purple rain.

After some thought, he decides to model this problem as follows.
Divide the peninsula into $n$ sections and number them West to East
from $1$ to $n$.  Then, describe the raindrops as a sequence of {\tt
R} and {\tt B}, depending on whether the rainfall in each section is
primarily red or blue.  Finally, find a subsequence of where the
difference between the number of {\tt R} and the number of {\tt B} is
maximized.

\section*{Input}

The input consists of a single line containing a string of $n$
characters ($1 \le n \le 10^5$), describing the color of the raindrops
in sections $1$ to $n$.

It is guaranteed that the string consists of uppercase ASCII letters
`{\tt R}' and `{\tt B}' only.

\section*{Output}

Print, on a single line, two space-separated integers that describe
the starting and ending positions of the part of Linearland that had
the least purple rain.

If there are multiple possible answers, print the one that has the
Westernmost (smallest-numbered) starting section.  If there are multiple
answers with the same Westernmost starting section, print the one with
the Westernmost ending section.

\section*{Examples}


\subsection{Cohen's Kappa}
\label{chp:fundamentals:sec:inter_rater_agreement:subsec:cohens_kappa}
Cohen's Kappa $\kappa$ was introduced by \textcite{Cohen:1960} in 1960.
He states the hypothesis that even if all raters are unaware of the correct answer and purely guess, nevertheless some datapoints are congruent.
In his opinion random congruency should be considered by agreement statistics.
To tackle this issue he introduced the kappa statistics to account for the random agreement among two raters.
Similar to other correlation statistics, it can take values in the range from -1 to 1.
0 indicates the agreement obtained by random choice, whereas 1 represents perfect agreement.
The kappa calculation includes two quantities.
$P_o$ is the proportion of observed agreement of raters and $P_e$ is the proportion of rating agreement expected to be obtained by chance.
The overall formula of Cohen's $\kappa$ is then given by the following equation:

\begin{equation}\label{eq:Cohens_kappa}
    \kappa = \frac{P_o - P_e}{1 - P_e}
\end{equation}

Consider two raters, A and B respectively, assigning $N$ datapoints to $C$ categories.
$n_{c_i, c_j}$ indicates how many datapoints were assigned to class $c_i$ by rater $A$ while the same datapoints were assigned to class $c_j$ by rater $B$.
$p_{A, c_i}$ represents the proportion of assignments which were assigned to class $c_i$ by rater $A$.
An overview for $C=3$ is given in \cref{tab:cohens_kappa_sample_definition}.

\begin{table}[htpb]
    \centering
    \begin{tabular}{l|l|c|c|c|c}
        \multicolumn{2}{c}{}&\multicolumn{3}{c}{B}&\\
        \cline{3-5}
        \multicolumn{2}{c|}{}&$c_1$&$c_2$&$c_3$&\multicolumn{1}{c}{Proportion}\\
        \cline{2-5}
        \multirow{3}{*}{A}& $c_1$ & $n_{c_1, c_1}$ & $n_{c_1, c_2}$ &$n_{c_1, c_3}$& $p_{A, c_1}$\\
        \cline{2-5}
        & $c_2$ & $n_{c_2, c_1}$ & $n_{c_2, c_2}$ &$n_{c_2, c_3}$&$p_{A, c_2}$\\
        \cline{2-5}
        & $c_3$ & $n_{c_3, c_1}$ & $n_{c_3, c_2}$ &$n_{c_3, c_3}$ & $p_{A, c_3}$\\
        \cline{2-5}
        \multicolumn{1}{c}{} & \multicolumn{1}{c}{Proportion} & \multicolumn{1}{c}{$p_{B, c_1}$} & \multicolumn{1}{c}{$p_{B, c_2}$} & \multicolumn{1}{c}{$p_{B, c_3}$} & \multicolumn{1}{c}{$1$}\\
    \end{tabular}
    \caption[Cohen's Kappa notation overview]{Example confusion matrix.}\label{tab:cohens_kappa_sample_definition}
\end{table}

$P_o$ is then given by: %TODO check here for unwanted page breaks
\begin{equation}\label{eq:Cohens_kappa:p_o}
    P_o = \frac{\sum_{i=1}^{C} n_{c_i, c_i}}{N}
\end{equation}

Whereas the expected agreement by chance $P_e$ is calculated according to \cref{eq:Cohens_kappa:p_e}:
\begin{equation}\label{eq:Cohens_kappa:p_e}
    P_e = \sum_{i=1}^{C} p_{A, c_i} p_{B, c_i}
\end{equation}

An example for the case $C=2$ is illustrated in \cref{tab:cohens_kappa_sample_data}.
Here two raters rated 50 datapoints as \textit{Good} or \textit{Bad}.

\begin{table}[htpb]
    \centering
    \begin{tabular}{l|l|c|c|c}
        \multicolumn{2}{c}{}&\multicolumn{2}{c}{A}&\\
        \cline{3-4}
        \multicolumn{2}{c|}{}&Good&Bad&\multicolumn{1}{c}{Proportion}\\
        \cline{2-4}
        \multirow{2}{*}{B}& Good & $20$ & $5$ & $0.5$\\
        \cline{2-4}
        & Bad & $10$ & $15$ & $0.5$\\
        \cline{2-4}
        \multicolumn{1}{c}{} & \multicolumn{1}{c}{Proportion} & \multicolumn{1}{c}{$0.6$} & \multicolumn{    1}{c}{$0.4$} & \multicolumn{1}{c}{$1$}\\
    \end{tabular}
    \caption[Cohen's Kappa sample data]{Example data of 2 raters assigning 50 datapoints to two categories.}\label{tab:cohens_kappa_sample_data}
\end{table}

Given \cref{tab:cohens_kappa_sample_data}, one can directly calculate $P_o = \frac{ 20 + 15 }{50} = 0.7$ and $P_e = 0.6 \cdot 0.5 + 0.4 \cdot 0.5 = 0.5$.
Now $P_o$ and $P_e$ are plugged into \cref{eq:Cohens_kappa} which yields a kappa $\kappa = \frac{0.7 - 0.5}{1 - 0.5} = 0.4$.


\subsection{Abelian groups}\label{subsec:abelian_groups}

\begin{definition}\label{def:abelian_group}
  A \hyperref[def:magma/commutative]{commutative} \hyperref[def:group]{group} is usually called an \term{abelian group}. It is conventional to use \hyperref[rem:additive_magma]{additive notation} for abelian groups.

  We denote by \( \cat{Ab} \) the category of abelian groups.
\end{definition}

\begin{proposition}\label{thm:abelian_normal_subgroups}
  All subgroups of an abelian group are \hyperref[def:normal_subgroup]{normal}.
\end{proposition}
\begin{proof}
  Let \( \mscrG \) be abelian and \( \mscrH \) be a subgroup of \( \mscrG \). Then \( x \mscrG x^{-1} = xx^{-1} \mscrH = \mscrH \) for any \( x \in G \) and thus \( \mscrH \) is normal.
\end{proof}

\begin{definition}\label{def:group_of_integers_modulo}
  The \hyperref[def:set_of_integers]{integers} \( \BbbZ \) notoriously form an abelian group under addition. Fix a positive integer \( n \). We define the group
  \begin{equation*}
    \BbbZ_n \coloneqq \{ 0, 1, \ldots, n - 1 \}
  \end{equation*}
  with the operation
  \begin{equation*}
    x \oplus y \coloneqq \rem(x + y, n)
  \end{equation*}
  so that
  \begin{equation*}
    x \oplus y \cong x + y \pmod n.
  \end{equation*}

  The group \( \BbbZ_n \) is called the \term{group of integers modulo} \( n \).
\end{definition}
\begin{proof}
  We will prove that \( \BbbZ_n \) is an abelian group.

  \SubProofOf{def:magma/associative} Addition in \( \BbbZ_n \) is associative since
  \begin{balign*}
    (x \oplus y) \oplus z
    &=
    \rem((x \oplus y) + z, n)
    = \\ &=
    \rem(\rem(x + y, n) + z, n)
    = \\ &=
    \rem(x + y - n \quot(x + y, n) + z, n)
    = \\ &=
    \rem(x + y + z, n)
    = \\ &=
    \ldots
    = \\ &=
    x \oplus (y \oplus z).
  \end{balign*}

  \SubProofOf{def:unital_magma} The zero is obviously the identity.

  \SubProofOf{def:unital_magma_inverse_element} Fix \( x \in \BbbZ_n \). If \( x = 0 \), its inverse is \( 0 \). If \( x > 0 \), its inverse is \( n - x \) since \( n - x \in \BbbZ_n \) and
  \begin{equation*}
    x \oplus (n - x) = x + (n - x) - n = 0.
  \end{equation*}

  \SubProofOf{def:magma/commutative} Commutativity follows from
  \begin{equation*}
    x \oplus y
    =
    \rem(x + y, n)
    =
    \rem(y + x, n)
    =
    y \oplus x.
  \end{equation*}
\end{proof}

\begin{proposition}\label{thm:integers_modulo_isomorphic_to_quotient_group}
  The group \( \BbbZ_n \) of \hyperref[def:group_of_integers_modulo]{integers modulo \( n \)} is isomorphic to the quotient of \( \BbbZ \) by \( n\BbbZ = \{ nz : z \in \BbbZ \} \), i.e.
  \begin{equation*}
    \BbbZ_n \cong \BbbZ / n\BbbZ.
  \end{equation*}
\end{proposition}
\begin{proof}
  Define the function
  \begin{align*}
    &\varphi: \BbbZ_n \to \BbbZ / n\BbbZ  \\
    &\varphi(x) \coloneqq x + n\BbbZ.
  \end{align*}

  It is a homomorphism because
  \begin{balign*}
    \varphi(x \oplus y)
    &=
    \varphi(\rem(x + y, n))
    = \\ &=
    \varphi(x + y - n \quot(x + y, n))
    = \\ &=
    x + y - n \quot(x + y, n) + n\BbbZ
    = \\ &=
    x + y + n\BbbZ
    = \\ &=
    (x + n\BbbZ) + (y + n\BbbZ)
    = \\ &=
    \varphi(x) + \varphi(y).
  \end{balign*}

  Furthermore, this shows that \( \varphi \) is also an isomorphism.
\end{proof}

\begin{example}\label{ex:lagranges_theorem_for_groups/direct_product_zn}
  \Fullref{thm:lagranges_theorem_for_groups} and \fullref{thm:integers_modulo_isomorphic_to_quotient_group} imply that, for any positive integer \( n \), there exists a bijection between \( n \BbbZ \times \BbbZ_n \) and \( \BbbZ \).

  This bijection, however, is not necessarily a group isomorphism because \eqref{thm:integers_modulo_isomorphic_to_quotient_group} may not hold.

  Let \( f: \BbbZ \to n \BbbZ \times \BbbZ_n \) be a bijection. Then, for every integer \( k \) there exist integers \( m_k \in \BbbZ \) and \( r_k \in \BbbZ_n \) such that
  \begin{equation*}
    f(k) = (n m_k, r_k).
  \end{equation*}

  For \( (mn, p) \in  \)
\end{example}

\begin{proposition}\label{thm:cyclic_group_isomorphic_to_integers_modulo_n}
  Let \( C \) be a cyclic \hyperref[def:cyclic_group]{group}. If \( C \) is finite of order \( n \), it is isomorphic to the group \( \BbbZ_n \) of integers modulo \( n \) (see \fullref{def:group_of_integers_modulo}).
\end{proposition}
\begin{proof}
  The homomorphism
  \begin{balign*}
     &\varphi: \BbbZ_n \to C_n \\
     &\varphi(k) \coloneqq a^k
  \end{balign*}
  and the analogous homomorphism for the infinite group, is an isomorphism.
\end{proof}

\begin{proposition}\label{thm:abelian_group_categorical_limits}
  We are interested in \hyperref[def:category_of_cones/limit]{categorical limits} and \hyperref[def:category_of_cones/colimit]{colimits} in the category \( \cat{Ab} \). Fix an indexed family  \( \{ \mscrX_k \}_{k \in \mscrK} \) of abelian groups.
  \begin{thmenum}
    \thmitem{thm:abelian_group_categorical_limits/product} Their \hyperref[def:discrete_category_limits]{categorical product} is the direct product as inherited from \fullref{thm:group_categorical_limits}.

    \thmitem{thm:abelian_group_categorical_limits/coproduct} Their \hyperref[def:discrete_category_limits]{categorical coproduct} is the \hyperref[def:group_direct_product]{direct sum} \( \oplus_{k \in \mscrK} \mscrX_k \), the embedding morphisms being
    \begin{align*}
       &\iota_m: \mscrX_m \to \oplus_{k \in \mscrK} \mscrX_k \\
       &\iota_m(x_m) \coloneqq \begin{dcases}
        \begin{drcases}
          x_m, &k = m \\
          0_k, &k \neq m
        \end{drcases}
      \end{dcases}_{k \in \mscrK}.
    \end{align*}
  \end{thmenum}
\end{proposition}

\begin{proposition}\label{thm:monoid_completion_to_abelian_group}\mcite{nLab:grothendieck_group_of_a_commutative_monoid}
  Every \hyperref[def:magma/commutative]{commutative} \hyperref[def:unital_magma/monoid]{monoid} can be \hyperref[def:first_order_homomorphism_invertibility/embedding]{embedded} into an abelian group using the \term{Grothendieck completion} presented in the proof.
\end{proposition}
\begin{proof}
  Let \( M \) be a commutative monoid. Define the relation \( \cong \) on tuples of members of \( M \) as
  \begin{equation*}
    (x_1, x_2) \cong (y_1, y_2) \iff \exists a: x_1 + y_2 + a = y_1 + x_2 + a.
  \end{equation*}

  This is an equivalence relation because
  \SubProofOf{def:binary_relation/reflexive}
  \begin{equation*}
    (x_1, x_2) \cong (x_1, x_2) \iff x_1 + x_2 + 0 = x_1 + x_2 + 0
  \end{equation*}

  \SubProofOf{def:binary_relation/symmetric} By commutativity,
  \begin{balign*}
    (x_1, x_2) \cong (y_1, y_2)
     & \iff
    \exists a: x_1 + y_2 + a = y_1 + x_2 + a
    \\ &\iff
    \exists a: y_1 + x_2 + a = x_1 + y_2 + a
    \\ &\iff
    (y_1, y_2) \cong (x_1, x_2)
  \end{balign*}

  \SubProofOf{def:binary_relation/transitive} Let \( (x_1, x_2) \cong (y_1, y_2) \) and \( (y_1, y_2) \cong (z_1, z_2) \). Thus, there exist \( a, b \in \BbbN \) such that
  \begin{equation*}
    [x_1 + y_2 + a = y_1 + x_2 + a] \T{and} [y_1 + z_2 + b = z_1 + y_2 + b]
  \end{equation*}

  Summing both sides, we have
  \begin{equation*}
    x_1 + y_2 + a + y_1 + z_2 + b = y_1 + x_2 + a + z_1 + y_2 + b
  \end{equation*}

  We reorder both sides to obtain
  \begin{equation*}
    (x_1 + z_2) + (y_1 + y_2 + a + b) = (x_2 + z_1) + (y_1 + y_2 + a + b),
  \end{equation*}
  which implies \( (x_1, x_2) \cong (z_1, z_2) \).

  Define \( G \coloneqq M^2 / \cong \) to be the equivalence partition\fullref{thm:equivalence_partition} of \( M \times M \). Define addition in \( G \) on members of \( M \times M \) by
  \begin{equation*}
    (x_1, x_2) + (y_1, y_2)
    \coloneqq
    (x_1 + y_1, x_2 + y_2).
  \end{equation*}

  This addition does not depend on the representative of the equivalence class since \( (x_1, x_2) \cong (x_1', x_2') \) and \( (y_1, y_2) \cong (y_1', y_2') \) implies the existence of \( k, m \in \BbbN \), such that
  \begin{balign*}
    x_1 + x_2' + a&= x_2 + x_1' + a,
    y_1 + y_2' + b&= y_2 + y_1' + b,
  \end{balign*}
  which, when combined, give
  \begin{balign*}
    (x_1 + x_2' + a) + (y_1 + y_2' + b)
    &=
    (x_2 + x_1' + a) + (y_2 + y_1' + b)
    \\
    (x_1 + y_1) + (x_2' + y_2') + (a + b)
    &=
    (x_2 + y_2) + (y_1 + x_1) + (a + b).
  \end{balign*}

  This implies
  \begin{balign*}
    (x_1 + y_1, x_2 + y_2)
    \cong
    (x_1' + y_1', x_2' + y_2').
  \end{balign*}

  The equivalence class \( [(0, 0)] \) is obviously an identity in \( G \) and contains exactly the pairs \( (x, x) \) of identical elements.

  For each member \( (x_1, x_2) \in M \times M \) we define its inverse as \( (x_2, x_1) \). It is indeed an inverse since
  \begin{equation*}
    (x_1, x_2) + (x_2, x_1) = (x_1 + x_2, x_2 + x_1),
  \end{equation*}
  which, by commutativity, belongs to \( [(0, 0)] \).

  If \( (x_1, x_2) \cong (x_1', x_2') \), then
  \begin{equation*}
    (x_1, x_2) + (x_2', x_1')
    =
    (x_1 + x_2', x_2 + x_1'),
  \end{equation*}
  where the two representatives of a pair of inverses are equal because of the equivalence \( \cong \).

  Thus, \( + \) is a well-defined commutative operation on \( G \) with identity, making it an abelian group.

  Furthermore, the function
  \begin{balign*}
     & \varphi: M \to G              \\
     & \varphi(x) \coloneqq [(x, 0)]
  \end{balign*}
  is a monoid homomorphism, hence \( M \) is indeed embedded in the group. Furthermore, any group that embeds \( G \) must also embed \( M \) since \( G \setminus \varphi(M) \) consists only of the \enquote{inverse} elements of \( \varphi(M) \).
\end{proof}

\begin{definition}\label{def:group_commutator}
  Let \( G \) be any group. The commutator of \( x, y \in G \) is defined as
  \begin{equation*}
    [x, y] \coloneqq xyx^{-1}y^{-1}.
  \end{equation*}

  The commutator subgroup of \( G \) is the subgroup \hyperref[def:group_presentation]{generated} by all the commutators in \( G \).
\end{definition}

\begin{proposition}\label{thm:quotient_by_commutator_subgroup}\mcite[prop. 7.4]{Knapp2016BasicAlgebra}
  The commutator group \( G' \) of any group \( G \) is \hyperref[def:normal_subgroup]{normal} and the quotient \( G / G' \) is \hyperref[def:abelian_group]{abelian}.
\end{proposition}


\section{Continuous data model}
\label{sec:continuousDataModel}
A general nonlinear mixed effects model for continuous data for $N$ subjects and $n_i$ measurements per subject $i$ reads as follows (\cite{Lavielle:2012b}):
\begin{align}
 \underbrace{ y_{ij}}_{\text{\parbox{2cm}{\centering Experimental \\[-4pt]  data}}} =
 \underbrace{ f(x_{ij}, \psi_{i})}_{\text{\parbox{2.5cm}{\centering Model \\[-4pt]  prediction}}} +
 \underbrace{ g(x_{ij}, \psi_{i}, \xi) \; \epsilon_{ij}}_{\text{\parbox{3cm}{\centering Error}}}
\quad 1\le i \le N, \quad 1\le j \le n_i \label{eq:nlmeModel}
 \end{align}
with
\begin{itemize}
\item
$y_{ij}$ -- $j^{th}$ observation for subject $i$
\item
$f$ -- structural model prediction
\item
$x_{ij}$ -- regression variables, e.g. $time$ or $concentration$
\item
$\psi_{i}$ -- individual parameters
\item
$\epsilon_{ij}$ -- residual error
\item
$g$ -- standard deviation of the residual error
\item
$\xi$ -- parameters of the residual model
\end{itemize}
With $\epsilon_{ij}$ being normal distributed with mean 0 and variance 1, $y_{ij}$ is also normally distributed with mean $ f(x_{ij}, \psi_{i})$ and the standard deviation $g(x_{ij}, \psi_{i}, \xi)$.


\subsection{Scalar fields}

A scalar field is a function on an underlying input which produces a real output.

Inputs are not limited to real numbers. In this section we consider functions on vector spaces.

\subsection{Del}

\(\nabla =(\sum_{i=1}^n e_i\dfrac{\delta }{\delta x_i})\)

Where \(e\) are the basis vectors.

This on its own means nothing. It is similar to the partial differentiation function.


\subsection{Integration}
\noindent
The line element is $\mathrm{d}s^2 = \mathrm{d}r^2 + r^2\mathrm{d}\theta^2$, meaning that $s = \int_{\theta_1}^{\theta_2}{\sqrt{r^2 + \left(\frac{\mathrm{d}r}{\mathrm{d}\theta}\right)}\mathrm{d}\theta}$.\\
The area element is $\mathrm{d}A = r\mathrm{d}r\mathrm{d}\theta$, meaning that $A = \int_{\theta_1}^{\theta_2}{r^2\mathrm{d}\theta}$.

\input{./curvilinearCoordinates/gaussianIntegral}


\documentclass{amsart}
\newcommand{\cC}{\mathcal {C}}
\newcommand{\RR}{\mathbb {R}}
\newcommand{\PP}{\mathbb {P}}
\newcommand {\bv} {\mathbf {v}}
\newcommand {\bw} {\mathbf {w}}
\newcommand {\be} {\mathbf {e}}
\usepackage{tikz-cd}
\title {Obnoxious notes to perspective drawing}
\author {Hei}
\begin{document}
\maketitle
These are our notes to studying perspective without the crutch of cubes. Oh, well, as we added some notes on how to apply this to drawing, there are cubes now.  We think this definitely helped with our
drawing skills.  Please let us know any inaccuracies.

\section{The Perspective Projection Map}
\label{sec:persp-proj-map}

We consider the $3$-dimensional Euclidean space $\RR^3$. Let $V$ be the viewing plane and $O$ be the eye point. Then the perspective drawing is, in essence,
the projection to $V$ with respect to $O$. More precisely, the image of a point $P$ in space is the point of
intersection of the line $OP$ and the plane $V$. Let $f$ denote this map.

[TODO: figure of map]

Without loss of generality, we set $O$ to be the origin and $V$ to be the plane $z=1$. Then the map $f$ is given by:
\begin{align*}
  f: \RR^3\backslash \{z=0\} &\rightarrow V \\
  (x,y,z) &\mapsto (x/z,y/z,1).
\end{align*}
We may identify the affine space $V$ with $\RR^2$ by taking the vector part, more precisely, by dropping the $z$-coordinate. In application, we restrict $f$ to the domain  $z>0$.

Let $W$ be a plane. As long as $W$ does not pass $O$, then $f|_W$ is a bijection.

Let $C$ be intersection of the locus of a degree $n$ polynomial $p$ with the plane $W$, i.e., the points on $C$ satisfies $p$ in addition to a degree $1$ polynomial. In this case, it is easy to see that points on the image of $C$ also satisfies a degree $n$ polynomial.

In particular, $f$  maps lines to lines and conics to conics. Signature is not preserved however. Thus it is possible for the types of conics to change. In application, the image of an ellipse must be an ellipse; otherwise, the source ellipse intersects the $xy$-plane. This can be seen by going back to the original meaning of conics: curves cut from cones by planes.

\section{The Vanishing Point}
\label{sec:vanishing-point}

Let $\bv$ be a vector. Let $\ell$ be a line passing through a point $P$ and parallel to $\bv$. Then points on $\ell$ can be parametrised as
\begin{equation*}
  P + t\bv,\qquad t\in\RR.
\end{equation*}
% Assume that when $t$ increases, the points go farther away from $V$. In particular, this assumes that $\bv$ is not parallel to $V$.
Let the coordinates of $P$ be given by $(x_P,y_P,z_P)$ and $\bv$ given by $(v_1,v_2,v_3)$. Then if $v_3\neq 0$, i.e., $\bv$ is not parallel to $V$, then
\begin{equation*}
  \lim_{t\rightarrow \infty} f (P + t\bv)
  = \lim_{t\rightarrow \infty} (\frac{x_P+tv_1}{z_P+tv_3},\frac{y_P+tv_2}{z_P+tv_3},1)
  = (\frac{v_1}{v_3},\frac{v_2}{v_3},1).
\end{equation*}
This point is called the vanishing point of $\ell$. If $v_3=0$, we should consider infinity to be the vanishing point. The meaning of infinity and which infinity point can be made precise by using projective geometry.
Let $\cC_\bv$ be the collection of lines parallel to $\bv$. In fact, the vanishing point depends only on $\bv$, not on the choice of $\ell$ in $\cC_\bv$. Let $\cC_\bv$ be the collection of lines parallel to $\bv$. Thus every line in $\cC_\bv$ has the same vanishing point. Thus their images appear to radiate from one point. Note that if we restrict to the $z>0$ part, the image is a ray.

[TODO: figure of VP for parallel lines.]

Let $\ell_O$ be the line which is parallel to $v$ and which contains $O$. Geometrically the vanishing point for $\cC_\bv$ is just the point of intersection of $\ell_O$ and $V$.

Thus we can answer the question of how many vanishing points there are in a drawing. The answer is the number of lines with different direction vectors. In scenes like cityscape, there are generally three sets of parallel lines with different direction vectors. Each produces a vanishing point as long as the set of lines are not parallel to $V$. We get the so-called 3-point, 2-point and 1-point perspectives.

Let $W$ be a plane. The set of vanishing points of lines contained in $W$ form a line. Geometrically it is given by intersecting the plane passing through $O$ and parallel to $W$ with $V$. Let us call it the vanishing line of $W$. In application, when $W$ is taken to be the ground plane, the  vanishing line  of $W$ is usually called the horizon or eye level.

[TODO: figure of VL for parallel planes.]

In practice, image of lines are easy to determine once we put down the vanishing points. If we know a point to be the intersections of two lines, we may find its image by finding the intersection of the images of the two lines. As line segments of the same length do not map to line segments of the same length under the perspective projection, this provides an easy way to determine location of image of a point. We can find the image of the centre of a rectangle by intersecting the diagonals of the image of the rectangle, for example. Once we know the image of the centre, we can find image of the disecting points and hence we know how to find the image of the rectangle that extends the original rectangle by one time of its width.

\section{Measuring Distance}
\label{sec:transfer-distance}

The aim of this section is to study images of line segments of equal length. In application, essentially we operate on $V$ only. One problem arises often. We have the image of a line segment as reference. How do we find the image of the same length along another line?

What we have available are the vanishing points and the reference line segment on $V$. Once we supply the information of the relative position of $O$ to the lines, then we can transfer the distance from one line to another.

Let $\ell$ be a line. Let $P$ and $Q$ be two points on $\ell$.  Let $\ell'$ be another line that passes through $P$. Let $R$ be a point on $\ell'$ such that $|PQ|=|PR|$. Let $A$ be the vanishing point of $\ell$ and $B$ that of $\ell'$. We would like to find $f (R)$ in terms of $f (P)$, $f (Q)$, $A$ and $B$.

One  observation is that if we know the vanishing point $C$ of $QR$ then $f (R)$ can be found as the intersection of the line connecting $CQ$ and the line connecting $BP$. However the location of $C$ cannot be determined by $f (P)$, $f (Q)$, $A$ and $B$ alone.  We need an additional piece of data,  the location of $O$ relative to $\ell$ and $\ell'$. We focus on the plane $W$ through $O$ and parallel to both $\ell$ and $\ell'$. We translate $\ell$ and $\ell'$ to $W$ along a vector that is parallel to $V$ and orthogonal to $AB$. We use superscript $t$ to denote the translated object. On $(\ell')^t$ find the point $S$ such that $|P^tS|=|P^tQ^t|$. Note that $QR$ is parallel to $Q^tS$. Then $C$ is the intersection of $AB$ and $Q^tS$.

[TODO: figure view of $V$.]

[TODO: figure view of $W$ to get $C$]

In practice, $\ell$ and $\ell'$ are orthogonal, forming two edges of a cube.

\section{Application to Drawing}
\label{sec:application-drawing}

\subsection{Centre of rectangle and extension of length}
\label{sec:centre-rect-extens}

This is already mentioned above.

\subsection{Shadow}
\label{sec:shadow}

This is an application of the intersection method. One common need is to find the cast shadow of a stick that stands vertically on the ground under sunlight. Sunlight can be considered as parallel light with given direction $\bv$. The location of shadow of the foot $F$ of the stick is obvious. Let $P$ be the top of the stick. The location of shadow $Q$ of top of stick lies on the line parallel to $\bv$ through $P$. It also lies on the ground plane $G$. Thus it lies on the line $L$ that is the intersection of the plane parallel to the vertical direction and $\bv$ through $P$ and the plane $G$. Also consider the line $M$ through the shadow point $Q$ on the ground $G$ perpendicular to $V$. If we consider the component $\bw$ of light that is parallel to $V$, then the line through $P$ along $\bw$ intersects $M$ and call the intersection $R$. Thus we get two lines whose intersection gives the shadow point $Q$.

To find the image of $Q$ on $V$, we consider the two families of parallel lines determined by $L$ and $M$ respectively. The family of parallel lines associated to $L$ produces a vanishing point, which we call the ground vanishing point of light (GVPL). Thus $f (Q)$ lies on the line connecting $f (F)$ and GVPL. The point $f (R)$ is easy to find, if we know $f (F)$ and the horizon line and the $V$-component $\bw$ of $\bv$. Let $C$ be the vanishing point for the receding lines that are parallel to the normal of $V$. Then $f (Q)$ also lies on the line connecting $C$ and $f (R)$. Then using the intersection of the two lines we find $f (Q)$.

Summary: Shadow of top=(GLVP--foot) $\cap$ (centre--horizontal shadow drop).

This can be generalised to finding cast shadow of any point to any plane which may not be orthogonal to the viewing plane. In this case direction of light can be encoded in vanishing point of shadow on the plane and vanishing point of component of light on the plane which is orthogonal to the given plane and which is parallel to the line of intersection of the given plane and the viewing plane. We should give this plane a name to avoid long-winded sentences.

\section{Projective Geometry}
\label{sec:projective-geometry}

It is cumbersome to always have to differentiate between the cases when family of parallel lines has image lines that radiate from one point or lines that remain parallel. It is best to add points at `infinity'. How many points of infinities are there? In $\RR^2$ the families of parallel lines are indexed by direction vectors. Thus we should glue on $\PP^1:= (\RR^2 - \{0\})/\RR^\times$ of them. This gives us the projective space $\PP^2$, in which $\RR^2$ is embedded as follows:
\begin{align*}
  \RR^2 &\rightarrow \PP^2 \\
  (x,y) &\mapsto (x:y:1)
\end{align*}
while points at infinity embed as follows:
\begin{align*}
  \PP^1 &\rightarrow \PP^2\\
  (a:b) &\mapsto (a:b:0).
\end{align*}
Once we call the point $(a:b:0)$ the vanishing point of family of parallel line along the vector $(a,b,0)$, then we can write the theory in a more uniform way.

\section{Intersection Method}
\label{sec:intersection-method}

One can use two VLs to determine VP of the intersection  line and one can also use VPs of two lines to determine VL of the plane parallel to both lines.

% For planes parallel to $V$, the effect of $f$ is obvious.
% \begin{equation*}
%   \begin{tikzcd}
%     \{z=z_0\} \ar [r] \ar [d,"\simeq",sloped] & V \ar [d,"\simeq", sloped]\\
%     \RR^2 \ar [r] & \RR^2 \\ [-10pt]
%     (x,y)\ar [r,mapsto] &(x/z_0,y/z_0),
%   \end{tikzcd}
% \end{equation*}
% which is just scaling. Here the two vertical arrows means dropping $z$-coordinate. Thus an equispaced grid does the job.

% Next we consider a line $\ell$ not parallel to $V$. Let $A$ be the vanishing point of $\ell$ and $P$ the intersection of $\ell$ and $V$. Take orthonormal basis for $V$ (more precisely, the vector part of $V$) where one basis vector $\be_1$ is along the direction of $f (\ell)$ and the other $\be_2$ orthogonal to $f (\ell)$. This produces a grid on $V$. The aim now is to locate the image of the point on $\ell$ that is at $x$ units to  $P$.

% Let $Q=P+x\be_1$ and $R=P+x\be_2$. Let $S$ be the point on $\ell$ at distance $x$ to $P$. Let $B$ be the vanishing point of $QS$. It lies on $f (\ell)$. Then we can locate $f (S)$ geometrically as follows. Let $T=Q+x\be_2$. Find the intersection of $AR$ and $BT$. Then project it orthogonally to $f (\ell)$. The image is $f (S)$. When we take $x$ to be integers, we produce a grid on $\ell$.

% [TODO: figure of how to find $S$.]

% In practice, one usually sets up vanishing points for lines corresponding to edges of a cube first. Then one may use the top down view to plot out the vanishing points for the 3 diagonal lines on facets of the cube. Then the vertices of the cube can be found using the method outlined above. In this way we can form a `3d grid' on $V$.

% [TODO: figure of grid on $V$.]
\end{document}


\section{Groups in univalent mathematics}

In this section we demonstrate a typical way to use the univalence axiom\index{univalence axiom}, showing that isomorphic groups can be identified.
This is an instance of the \emph{structure identity principle}\index{structure identity principle}, which is described in more detail in section 9.8 of \cite{hottbook}.
We will see that in order to establish the fact that isomorphic groups can be identified, it has to be part of the definition of a group that its underlying type is a set. This is an important observation: in many branches of algebra the objects of study are \emph{set-level} structures\index{set-level structure}\footnote{A notable exception is that of categories, which are objects at truncation level $1$, i.e., at the level of \emph{groupoids}. We will briefly introduce categories in \cref{sec:categories}. For more about categories we recommend Chapter 9 of \cite{hottbook}.}.

\subsection{Semi-groups and groups}
We introduce the type of groups in two stages: first we introduce the type of \emph{semi-groups}, and then we introduce groups as semi-groups that possess further structure. It will turn out that this further structure is in fact a property, and this fact will help us to prove that isomorphic groups are equal.

\begin{defn}
  A \define{semi-group}\index{semi-group} consists of a set $G$ equipped with a term of type $\hasassociativemul(G)$, which is the type of pairs $(\mu_G,\assoc_G)$ consisting of a binary operation
  \begin{equation*}
    \mu_G : G \to (G \to G)
  \end{equation*}
  and a homotopy
  \begin{equation*}
    \assoc_G : \prd{x,y,z:G}\mu_G(\mu_G(x,y),z)=\mu_G(x,\mu_G(y,z)).
  \end{equation*}
  We write $\semigroup$\index{Semi-Group@{$\semigroup$}} for the type of all semi-groups in $\UU$.
\end{defn}

\begin{defn}
  A semi-group $G$ is said to be \define{unital}\index{semi-group!unital}\index{unital semi-group} if it comes equipped with a \define{unit}\index{unit!of a unital semi-group} $e_G:G$ that satisfies the left and right unit laws\index{unit laws!for a unital semi-group}
  \begin{align*}
    \leftunit_G : \prd{y:G}\mu_G(e_G,y)=y \\
    \rightunit_G : \prd{x:G}\mu_G(x,e_G)=x.
  \end{align*}
  We write $\isunital(G)$\index{is-unital@{$\isunital$}} for the type of such triples $(e_G,\leftunit_G,\rightunit_G)$. Unital semi-groups are also called \define{monoids}\index{monoid}.
\end{defn}

The unit of a semi-group is of course unique once it exists. In univalent mathematics we express this fact by asserting that the type $\isunital(G)$ is a proposition for each semi-group $G$. In other words, being unital is a \emph{property} of semi-groups rather than structure on it. This is typical for univalent mathematics: we express that a structure is a property by proving that this structure is a proposition.

\begin{lem}
  For a semi-group $G$ the type $\isunital(G)$ is a proposition.\index{is-unital@{$\isunital$}!is a proposition}
\end{lem}

\begin{proof}
  Let $G$ be a semi-group. Note that since $G$ is a set, it follows that the types of the left and right unit laws are propositions. Therefore it suffices to show that any two terms $e,e':G$ satisfying the left and right unit laws can be identified. This is easy:
  \begin{equation*}
    e = \mu_G(e,e') = e'.\qedhere
  \end{equation*}
\end{proof}

\begin{defn}
  Let $G$ be a unital semi-group. We say that $G$ \define{has inverses}\index{unital semi-group!has inverses}\index{semi-group!has inverses} if it comes equipped with an operation $x\mapsto x^{-1}$ of type $G\to G$, satisfying the left and right inverse laws\index{inverse laws!for a group}
  \begin{align*}
    \leftinv_G : \prd{x:G}\mu_G(x^{-1},x)=e_G \\
    \rightinv_G : \prd{x:G}\mu_G(x,x^{-1}) = e_G.
  \end{align*}
  We write $\isgroup'(G,e)$\index{is-group'@{$\isgroup'$}} for the type of such triples $((\blank)^{-1},\leftinv_G,\rightinv_G)$, and we write\index{is-group@{$\isgroup$}}
  \begin{equation*}
    \isgroup(G)\defeq\sm{e:\isunital(G)}\isgroup'(G,e)
  \end{equation*}
  A \define{group}\index{group} is a unital semi-group with inverses. We write $\group$\index{Group@{$\group$}} for the type of all groups in $\UU$.
\end{defn}

\begin{lem}
  For any semi-group $G$ the type $\isgroup(G)$ is a proposition.\index{is-group@{$\isgroup$}!is a proposition}
\end{lem}

\begin{proof}
  We have already seen that the type $\isunital(G)$ is a proposition. Therefore it suffices to show that the type $\isgroup'(G,e)$ is a proposition\index{is-group'@{$\isgroup'$}!is a proposition} for any $e:\isunital(G)$.

  Since a semi-group $G$ is assumed to be a set, we note that the types of the inverse laws are propositions. Therefore it suffices to show that any two inverse operations satisfying the inverse laws are homotopic.

  Let $x\mapsto x^{-1}$ and $x\mapsto \bar{x}^{-1}$ be two inverse operations on a unital semi-group $G$, both satisfying the inverse laws. Then we have the following identifications
  \begin{align*}
    x^{-1} & = \mu_G(e_G,x^{-1}) \\
    & = \mu_G(\mu_G(\bar{x}^{-1},x),x^{-1}) \\
    & = \mu_G(\bar{x}^{-1},\mu_G(x,x^{-1})) \\
    & = \mu_G(\bar{x}^{-1},e_G) \\
    & = \bar{x}^{-1}
  \end{align*}
  for any $x:G$. Thus the two inverses of $x$ are the same, so the claim follows.
\end{proof}

\begin{eg}
  An important class of examples consists of \define{loop spaces}\index{loop space!of 1-type is a group@{of $1$-type is a group}}\index{group!loop space of 1-type@{loop space of $1$-type}} $x=x$ of a $1$-type $X$, for any $x:X$.
  We will write $\loopspace{X,x}$ for the loop space of $X$ at $x$.
  Since $X$ is assumed to be a $1$-type, it follows that the type $\loopspace{X,x}$ is a set. Then we have
  \begin{align*}
    \refl{x} & : \loopspace{X,x} \\
    \invfunc & : \loopspace{X,x} \to \loopspace{X,x} \\
    \concat & : \loopspace{X,x} \to (\loopspace{X,x}\to \loopspace{X,x}),
  \end{align*}
  and these operations satisfy the group laws, since the group laws are just a special case of the groupoid laws for identity types, constructed in \cref{sec:groupoid}.
\end{eg}

\begin{eg}
  The type $\Z$ of integers\index{Z@{$\Z$}!is a group}\index{group!Z@{$\Z$}} can be given the structure of a group, with the group operation being addition. The fact that $\Z$ is a set follows from \cref{thm:eq_nat,ex:set_coprod}. The group laws were shown in \cref{ex:int_group_laws}.
\end{eg}

\begin{eg}
  Our last class of examples consists of the \define{automorphism groups}\index{automorphism group}\index{group!automorphism group of set} on sets. Given a set $X$, we define\index{Aut@{$\Aut$}}
  \begin{equation*}
    \Aut(X)\defeq (X\simeq X).
  \end{equation*}
  The group operation of $\Aut(X)$ is just composition of equivalences, and the unit of the group is the identity function. Note however, that although function composition is strictly associative and satisfies the unit laws strictly, composition of equivalences only satisfies the group laws up to identification because the proof that composites are equivalences is carried along.

  Important special cases of the automorphism groups are the \define{symmetric groups}\index{symmetric groups}\index{Sn@{$\mathcal{S}_n$}}\index{group!Sn@{$\mathcal{S}_n$}}
  \begin{equation*}
    \mathcal{S}_n\defeq \Aut(\Fin(n)).
  \end{equation*}
\end{eg}

\subsection{Homomorphisms of semi-groups and groups}

\begin{defn}
  Let $G$ and $H$ be semi-groups. A \define{homomorphism}\index{homomorphism!of semi-groups}\index{semi-group!homomorphism} of semi-groups from $G$ to $H$ is a pair $(f,\mu_f)$ consisting of a function $f:G\to H$ between their underlying types, and a term
  \begin{equation*}
    \mu_f:\prd{x,y:G} f(\mu_G(x,y))=\mu_H(f(x),f(y))
  \end{equation*}
  witnessing that $f$ preserves the binary operation of $G$. We will write\index{hom(G,H) for semi-groups@{$\hom(G,H)$ for semi-groups}}
  \begin{equation*}
    \hom(G,H)
  \end{equation*}
  for the type of all semi-group homomorphisms from $G$ to $H$.
\end{defn}

\begin{rmk}\label{rmk:is-set-hom-semi-group}
  Since it is a property for a function to preserve the multiplication of a semi-group, it follows easily that equality of semi-group homomorphisms is equivalent to the type of homotopies between their underlying functions. In particular, it follows that the type of homomorphisms of semi-groups is a set.
\end{rmk}

\begin{rmk}\label{rmk:category-semi-group}
  The \define{identity homomorphism}\index{identity homomorphism!of semi-groups} on a semi-group $G$ is defined to be the pair consisting of
  \begin{align*}
    \idfunc & : G \to G \\
    \lam{x}{y}\refl{xy} & : \prd{x,y:G} xy = xy.
  \end{align*}
  Let $f:G\to H$ and $g:H\to K$ be semi-group homomorphisms. Then the composite function $g\circ f:G\to K$ is also a semi-group homomorphism\index{composition!of semi-group homomorphisms}, since we have the identifications
  \begin{equation*}
    \begin{tikzcd}
      g(f(xy)) \arrow[r,equals] & g(f(x)f(y)) \arrow[r,equals] & g(f(x))g(f(y)).
    \end{tikzcd}
  \end{equation*}
  Since the identity type of semi-group homomorphisms is equivalent to the type of homotopies between semi-group homomorphisms it is easy to see that semi-group homomorphisms satisfy the laws of a category, i.e., that we have the identifications
  \begin{align*}
    \idfunc\circ f & = f \\
    g\circ \idfunc & = g \\
    (h\circ g) \circ f & = h \circ (g \circ f)
  \end{align*}
  for any composable semi-group homomorphisms $f$, $g$, and $h$. Note, however that these equalities are not expected to hold judgmentally, since preservation of the semi-group operation is part of the data of a semi-group homomorphism.
\end{rmk}

\begin{defn}
  Let $G$ and $H$ be groups. A \define{homomorphism}\index{homomorphism!of groups}\index{group!homomorphism} of groups from $G$ to $H$ is defined to be a semi-group homomorphism between their underlying semi-groups. We will write\index{hom(G,H) for groups@{$\hom(G,H)$ for groups}}
  \begin{equation*}
    \hom(G,H)
  \end{equation*}
  for the type of all group homomorphisms from $G$ to $H$.
\end{defn}

\begin{rmk}\label{rmk:category-group}
  Since a group homomorphism is just a semi-group homomorphism between the underlying semi-groups, we immediately obtain the identity homomorphism\index{identity homomorphism!for groups}, composition\index{composition!of group homomorphisms}, and the category laws are satisfied.
\end{rmk}

\subsection{Isomorphic semi-groups are equal}

\begin{defn}
Let $h:\hom(G,H)$ be a homomorphism of semi-groups. Then $h$ is said to be an \define{isomorphism}\index{group homomorphism!isomorphism}\index{isomorphism!of groups} if it comes equipped with a term of type $\isiso(h)$\index{is-iso for semi-groups@{$\isiso$ for semi-groups}}, consisting of triples $(h^{-1},p,q)$ consisting of a homomorphism $h^{-1}:\hom(H,G)$ of semi-groups and identifications
\begin{equation*}
p:h^{-1}\circ h=\idfunc[G]\qquad\text{and}\qquad q:h\circ h^{-1}=\idfunc[H]
\end{equation*}
witnessing that $h^{-1}$ satisfies the inverse laws\index{inverse laws!for semi-group isomorphisms}We write $G\cong H$ for the type of all isomorphisms of semi-groups from $G$ to $H$, i.e.,
\begin{equation*}
G\cong H \defeq \sm{h:\hom(G,H)}{k:\hom(H,G)} (k\circ h = \idfunc[G])\times (h\circ k=\idfunc[H]).
\end{equation*}
\end{defn}

If $f$ is an isomorphism, then its inverse is unique. In other words, being an isomorphism is a property.

\begin{lem}
  For any semi-group homomorphism $h:\hom(G,H)$, the type
  \begin{equation*}
    \isiso(h)
  \end{equation*}
  is a proposition.\index{is-iso for semi-groups@{$\isiso$ for semi-groups}!is a proposition} It follows that the type $G\cong H$ is a set for any two semi-groups $G$ and $H$.
\end{lem}

\begin{proof}
  Let $k$ and $k'$ be two inverses of $h$. In \cref{rmk:is-set-hom-semi-group} we have observed that the type of semi-group homomorphisms between any two semi-groups is a set. Therefore it follows that the types $h\circ k=\idfunc$ and $k\circ h=\idfunc$ are propositions, so it suffices to check that $k=k'$. In \cref{rmk:is-set-hom-semi-group} we also observed that the equality type $k=k'$ is equivalent to the type of homotopies $k\htpy k'$ between their underlying functions. We construct a homotopy $k\htpy k'$ by the usual argument:
  \begin{equation*}
    \begin{tikzcd}
      k(y) \arrow[r,equals] & k(h(k'(y)) \arrow[r,equals] & k'(y).
    \end{tikzcd}\qedhere
  \end{equation*}
\end{proof}

\begin{lem}\label{lem:grp_iso}
  A semi-group homomorphism $h:\hom(G,H)$ is an isomorphism if and only if its underlying map is an equivalence. Consequently, there is an equivalence
  \begin{equation*}
    (G\cong H)\simeq \sm{e:G\simeq H}\prd{x,y:G}e(\mu_G(x,y))=\mu_H(e(x),e(y))
  \end{equation*}
\end{lem}

\begin{proof}
  If $h:\hom(G,H)$ is an isomorphism, then the inverse semi-group homomorphism also provides an inverse of the underlying map of $h$. Thus we obtain that $h$ is an equivalence. The standard proof showing that if the underlying map $f:G\to H$ of a group homomorphism is invertible then its inverse is again a group homomorphism also works in type theory.
\end{proof}

\begin{defn}
Let $G$ and $H$ be a semi-groups. We define the map\index{iso-eq for semi-groups@{$\isoeq$ for semi-groups}}
\begin{equation*}
\isoeq : (G=H)\to (G\cong H)
\end{equation*}
by path induction, taking $\refl{G}$ to isomorphism $\idfunc[G]$.
\end{defn}

\begin{thm}\label{thm:iso-eq-semi-group}
The map\index{identity type!of Semi-Group@{of $\semigroup$}}\index{Semi-Group@{$\semigroup$}!identity type}
\begin{equation*}
\isoeq : (G=H)\to (G\cong H)
\end{equation*}
is an equivalence for any two semi-groups $G$ and $H$.
\end{thm}

\begin{proof}
By the fundamental theorem of identity types \cref{thm:id_fundamental}\index{fundamental theorem of identity types} it suffices to show that the total space
\begin{equation*}
\sm{G':\semigroup}G\cong G'
\end{equation*}
is contractible. Since the type of isomorphisms from $G$ to $G'$ is equivalent to the type of equivalences from $G$ to $G'$ it suffices to show that the type
\begin{equation*}
  \sm{G':\semigroup}\sm{e:\eqv{G}{G'}}\prd{x,y:G}e(\mu_G(x,y))=\mu_{G'}(e(x),e(y)))
\end{equation*}
is contractible\footnote{In order to show that a type of the form
  \begin{equation*}
    \sm{(x,y):\sm{x:A}B(x)}\sm{z:C(x)}D(x,y,z)
  \end{equation*}
  is contractible, a useful strategy is to first show that the type $\sm{x:A}C(x)$ is contractible. Once this is established, say with center of contraction $(x_0,z_0)$, it suffices to show that the type $\sm{y:B(x_0)}D(x_0,y,z_0)$ is contractible.}. Since $\semigroup$ is the $\Sigma$-type
\begin{equation*}
  \sm{G':\Set}\hasassociativemul(G'),
\end{equation*}
it suffices to show that the types
\begin{align*}
  & \sm{G':\Set}\eqv{G}{G'} \\
  & \sm{\mu':\hasassociativemul(G)}\prd{x,y:G}\mu_G(x,y)=\mu'(x,y)
\end{align*}
is contractible. The first type is contractible by the univalence axiom. The second type is contractible by function extensionality.
\end{proof}

\begin{cor}
The type $\semigroup$ is a $1$-type.\index{Semi-Group@{$\semigroup$}!is a 1-type@{is a $1$-type}}
\end{cor}

\begin{proof}
It is straightforward to see that the type of group isomorphisms $G\cong H$ is a set, for any two groups $G$ and $H$.
\end{proof}

\subsection{Isomorphic groups are equal}

Analogously to the map $\isoeq$ of semi-groups, we have a map $\isoeq$ of groups. Note, however, that the domain of this map is now the identity type $G=H$ of the \emph{groups} $G$ and $H$, so the maps $\isoeq$ of semi-groups and groups are not exactly the same maps.

\begin{defn}
  Let $G$ and $H$ be groups. We define the map\index{iso-eq for groups@{$\isoeq$ for groups}}
  \begin{equation*}
    \isoeq : (G=H)\to (G\cong H)
  \end{equation*}
  by path induction, taking $\refl{G}$ to the identity isomorphism $\idfunc:G\cong G$.
\end{defn}

\begin{thm}
  For any two groups $G$ and $H$, the map\index{identity type!of Group@{of $\group$}}\index{Group@{$\group$}!identity type}
  \begin{equation*}
    \isoeq:(G=H)\to (G\cong H)
  \end{equation*}
  is an equivalence.
\end{thm}

\begin{proof}
  Let $G$ and $H$ be groups, and write $UG$ and $UH$ for their underlying semi-groups, respectively. Then we have a commuting triangle
  \begin{equation*}
    \begin{tikzcd}
      (G=H) \arrow[rr,"\apfunc{\proj 1}"] \arrow[dr,swap,"\isoeq"] & & (UG=UH) \arrow[dl,"\isoeq"] \\
      & (G\cong H)
    \end{tikzcd}
  \end{equation*}
  Since being a group is a property of semi-groups it follows that the projection map $\group\to\semigroup$ forgetting the unit and inverses, is an embedding. Thus the top map in this triangle is an equivalence. The map on the right is an equivalence by \cref{thm:iso-eq-semi-group}, so the claim follows by the 3-for-2 property.
\end{proof}

\begin{cor}
  The type of groups is a $1$-type.\index{Group@{$\group$}!is a 1-type@{is a $1$-type}}
\end{cor}

\subsection{Categories in univalent mathematics}
\label{sec:categories}

In our proof of the fact that isomorphic groups are equal we have made extensive use of the notion of group homomorphism. What we have shown, in fact, is that there is a category of groups which is \emph{Rezk complete} in the sense that the type of isomorphisms between two objects is equivalent to the type of identifications between those objects. In this final section we briefly introduce the notion of Rezk complete category. There are many more examples of categories, such as the categories of rings, or modules over a ring.

\begin{defn}
  A \define{pre-category}\index{pre-category} $\mathcal{C}$ consists of
  \begin{enumerate}
  \item A type $A$ of \define{objects}.\index{pre-category!objects}\index{objects}
  \item For every two objects $x,y:A$ a set
    \begin{equation*}
      \hom(x,y)
    \end{equation*}
    of \define{morphisms}\index{pre-category!morphisms}\index{morphism} from $x$ to $y$.
  \item For every object $x:A$ an \define{identity morphism}\index{identity morphism}\index{pre-category!identity morphism}
    \begin{equation*}
      \idfunc : \hom(x,x)
    \end{equation*}
  \item For every two morphisms $f:\hom(x,y)$ and $g:\hom(y,z)$, a morphism
    \begin{equation*}
      g\circ f :\hom(x,z)
    \end{equation*}
    called the \define{composition}\index{composition!of morphisms} of $f$ and $g$.
  \item the following terms
    \begin{align*}
      \leftunit_{\mathcal{C}} & : \idfunc \circ f = f \\
      \rightunit_{\mathcal{C}} & : g \circ \idfunc = g \\
      \assoc_{\mathcal{C}} & : (h \circ g) \circ f = h \circ (g \circ f)
    \end{align*}
    witnessing that the category laws\index{category laws}\index{laws!of a category} are satisfied.
  \end{enumerate}
\end{defn}

\begin{eg}
  Since the type $X\to Y$ of functions between sets is again a set, we have a pre-category of sets.\index{pre-category!of sets}
\end{eg}

\begin{eg}
  By \cref{rmk:category-semi-group,rmk:category-group} we have pre-categories of semi-groups and of groups.\index{semi-group!is a pre-category}\index{pre-category!of semi-groups}\index{group!is a category}\index{pre-category!of groups}
\end{eg}

\begin{eg}
  A pre-category satisfying the condition that every hom-set is a proposition is a \define{preorder}.\index{preorder}\index{pre-category!preorder}
\end{eg}

\begin{defn}
  Given a pre-category $\mathcal{C}$, a morphism $f:\hom(x,y)$ is said to be an \define{isomorphism}\index{isomorphism!in a pre-category} if there exists a morphism $g:\hom(y,x)$ such that
  \begin{align*}
    g \circ f & = \idfunc \\
    f \circ g & = \idfunc.
  \end{align*}
  We will write $\iso(x,y)$\index{iso(x,y)@{$\iso(x,y)$}} for the type of all isomorphisms in $\mathcal{C}$ from $x$ to $y$.
\end{defn}

\begin{rmk}
  Just as in the case for semi-groups and groups, the condition that $f:\hom(x,y)$ is an isomorphism is a property of $f$.
\end{rmk}

\begin{defn}
  A pre-category $\mathcal{C}$ is said to be \define{Rezk-complete}\index{Rezk-complete}\index{pre-category!Rezk complete} if the canonical map
  \begin{equation*}
    (x=y)\to \iso(x,y)
  \end{equation*}
  is an equivalence for any two objects $x$ and $y$ of $\mathcal{C}$. Rezk-complete pre-categories are also called \define{categories}.\index{category}
\end{defn}

\begin{eg}
  The pre-category of sets is Rezk complete by the univalence axiom, so it is a category.\index{sets!form a category}\index{category!of sets}
\end{eg}

\begin{eg}
  The pre-categories of semi-groups and groups are Rezk-complete. Therefore they form categories.\index{Semi-Group@{$\semigroup$}!is a category}\index{Group@{$\group$}!is a category}\index{category!of groups}\index{category!of semi-groups}
\end{eg}

\begin{eg}
  A pre-order is Rezk-complete if and only if it is anti-symmetric. In other words, a poset is precisely a category for which all the hom-sets are propositions. Thus, we see that the anti-symmetry axiom can be seen as a univalence axiom for pre-orders.\index{category!poset}\index{poset!is a category}
\end{eg}

\begin{exercises}
  \exercise Let $X$ be a set. Show that the map\index{equiv-eq@{$\equiveq$}!is a group isomorphism}
  \begin{equation*}
    \equiveq : (X=X)\to (\eqv{X}{X})
  \end{equation*}
  is a group isomorphism.
  \exercise \label{ex:groupop-embedding}
  \begin{subexenum}
  \item Consider a group $G$. Show that the function
    \begin{equation*}
      \mu_G:G\to (G\simeq G)
    \end{equation*}
    is an injective group homomorphism.
  \item Consider a pointed type $A$. Show that the concatenation function
    \begin{equation*}
      \concat:\loopspace{A}\to (\loopspace{A}\simeq\loopspace{A})
    \end{equation*}
    is an embedding.\index{concat@{$\concat$}!is an embedding}
  \end{subexenum}
  \exercise Let $f:\hom(G,H)$ be a group homomorphism. Show that $f$ preserves units and inverses, i.e., show that\index{group homomorphism!preserves units and inverses}
  \begin{align*}
    f(e_G) & = e_H \\
    f(x^{-1}) & = f(x)^{-1}.
  \end{align*}
  \exercise Give a direct proof and a proof using the univalence axiom of the fact that all semi-group isomorphisms between unital semi-groups preserve the unit. Conclude that isomorphic monoids are equal.\index{isomorphism!of semi-groups!preserves unit}
  \exercise Consider a monoid $M$ with multiplication $\mu:M\to (M\to M)$ and unit $e$. Write
  \begin{equation*}
    \bar{\mu}\defeq\foldlist(e,\mu):\lst(M)\to M
  \end{equation*}
  for the iterated multiplication operation (see \cref{ex:lists}). Show that the square
  \begin{equation*}
    \begin{tikzcd}[column sep=6em]
      \lst(\lst(M)) \arrow[r,"\flattenlist(M)"] \arrow[d,swap,"\lst(\bar{\mu})"] & \lst(M) \arrow[d,"\bar{\mu}"] \\
      \lst(M) \arrow[r,swap,"\bar{\mu}"] & M
    \end{tikzcd}
  \end{equation*}
  commutes.
  \exercise Construct the category of posets.\index{category!of posets}
  \exercise Consider the \define{walking isomorphism}, i.e., the pre-category $\mathcal{I}$ given by
  \begin{equation*}
    \begin{tikzcd}
      0 \arrow[r,bend left=15,"f"] & 1 \arrow[l,bend left=15,"{f^{-1}}"]
    \end{tikzcd}
  \end{equation*}
  satisfying $f\circ f^{-1}=\idfunc$ and $f^{-1}\circ f=\idfunc$. Show that for any pre-category $\mathcal{C}$ the following are equivalent:
  \begin{enumerate}
  \item The pre-category $\mathcal{C}$ is Rezk complete.
  \item The precomposition function
    \begin{equation*}
      \mathcal{C}\to\mathsf{Fun}(\mathcal{I},\mathcal{C})
    \end{equation*}
    is an equivalence.
  \end{enumerate}
\end{exercises}


\chapter{Power of filters}

\section{Germs of functions}

\begin{defn}
  Functions $f, g \in \mathbf{Rel} (\Ob \mathcal{X} , B)$
  \emph{are of the same $\mathcal{X}$-germ} for a filter object
  $\mathcal{X}$ iff there exists $X \in \up \mathcal{X}$ such that $f|_X
  = g|_X$.
\end{defn}

\begin{prop}
  Being of the same germ is an equivalence relation.
\end{prop}

\begin{proof}
  ~
  \begin{description}
    \item[Reflexivity] Take arbitrary $X \in \up \mathcal{X}$.

    \item[Symmetry] Obvious.

    \item[Transitivity] Let $f|_X = g|_X$ and $g|_Y = h|_Y$. Then $f|_{X \cap
    Y} = h|_{X \cap Y}$.
  \end{description}
\end{proof}

\begin{defn}
  A \emph{germ} is an equivalence class of being the same germ.
\end{defn}

\begin{obvious}
Every germ is a filter on $\mathbf{Set}$.
\end{obvious}

\begin{thm}
  Let $A$, $B$ be sets.

  The following are mutually inverse bijections between monovalued reloids $f
  : A \rightarrow B$ with $\dom f = \mathcal{X}$ and $\mathcal{X}$-germs
  $S$ of functions $A \rightarrow B$ for $\mathcal{X} \in \mathscr{F} A$:
  \begin{enumerate}
    \item $f \mapsto \up^{\mathbf{Set}} f$;
    \item $S \mapsto s|_{\mathcal{X}}$ if $s\in S$.
  \end{enumerate}
  The second bijection can also be written as $S \mapsto \left(\bigsqcap^{\mathsf{RLD}} S\right)|_{\mathcal{X}}$ or
  if $\card B\ne 1$ as $S \mapsto \bigsqcap^{\mathsf{RLD}} S$.
\end{thm}

\begin{rem}
$s|_{\mathcal{X}}$ is always defined because $S$ is nonempty (it is an equivalence class).
\end{rem}

\begin{proof}
  First prove that $\up^{\mathbf{Set}} f$ is an
  $\mathcal{X}$-germ. Really, $F \in \up^{\mathbf{Set}} f
  \Leftrightarrow F \sqsupseteq f \Leftrightarrow F|_{\mathcal{X}} = f
  \Leftrightarrow \exists X \in \up \mathcal{X} : F|_X \sqsupseteq f$;
  thus $F, G \in \up^{\mathbf{Set}} f \Rightarrow \exists X \in
  \up \mathcal{X} : F|_X \sqsupseteq f \wedge \exists Y \in \up
  \mathcal{X} : G|_Y \sqsupseteq f \Rightarrow \exists X \in \up
  \mathcal{X} : F|_{X \cap Y} \sqsupseteq f \wedge \exists Y \in \up
  \mathcal{X} : G|_{X \cap Y} \sqsupseteq f \Rightarrow \exists Z \in
  \up \mathcal{X} : (F|_Z \sqsupseteq f \wedge G|_Z \sqsupseteq f)
  \Rightarrow \exists Z \in \up \mathcal{X} : (F \sqcap G) |_Z
  \sqsupseteq f$ and $F \in \up^{\mathbf{Set}} f \wedge \exists
  X \in \up \mathcal{X} : F|_X = G|_X \Rightarrow F \sqsupseteq f \wedge
  F|_{\mathcal{X}} = G|_{\mathcal{X}} \Rightarrow G|_{\mathcal{X}} \sqsupseteq
  f \Rightarrow G \in \up^{\mathbf{Set}} f$. We have proved
  that $\up^{\mathbf{Set}} f$ is an equivalence class of the
  suitable equivalence relation, that is $\up^{\mathbf{Set}} f$
  is an $\mathcal{X}$-germ.

  That $\bigsqcap^{\mathsf{RLD}} S$ is a monovalued reloid is obvious.
  Also $\im \bigsqcap^{\mathsf{RLD}} S = \mathcal{X}$ is obvious.

  Now prove that our correspondences are mutually inverse.

  Let $f_0 : A \rightarrow B$ be a monovalued reloid and $\dom f =
  \mathcal{X}$. Let $S = \up^{\mathbf{Set}} f_0$ and $f_1 =
  s|_{\mathcal{X}}$ for an~$s\in S$. We need to prove $f_1 = f_0$. Really,
  $f_1 = F|_{\mathcal{X}}$ for an $F\in\up^{\mathbf{Set}}f_0$; thus $f_1=f_0$.

  Let $S_0$ be an $\mathcal{X}$-germ of functions $A \rightarrow B$. Let $f =
  s|_{\mathcal{X}}$ for an~$s\in S_0$ and $S_1 =
  \up^{\mathbf{Set}} f$. We need to prove $S_1 = S_0$. Really,
  \[
  S_1 = \up^{\mathbf{Set}}(s|_{\mathcal{X}}) =
  \setcond{F\in\mathbf{Set}}{F\sqsupseteq s|_{\mathcal{X}}} =
  \setcond{F\in\mathbf{Set}}{\exists X\in\up\mathcal{X}:F|_X\sqsupseteq s|_X} =
  \setcond{F\in\mathbf{Set}}{\exists X\in\up\mathcal{X}:F|_X=s|_X} = S_0.
  \]

  $\left(\bigsqcap^{\mathsf{RLD}} S\right)|_{\mathcal{X}} =
  \bigsqcap^{\mathsf{RLD}}_{s\in S} s|_{\mathcal{X}} = s|_{\mathcal{X}}$
  for every choice of~$s\in S$.

  We can assume that $B \neq \emptyset$ because otherwise the theorem is
  obvious. Thus we can assume $\card B>1$.

  If $\mathcal{X} = X$ then obviously $S$ has just one element $F$ and
  $\im \bigsqcap^{\mathsf{RLD}} S = \im F = X =
  \mathcal{X}$. Otherwise for every $X \in \up \mathcal{X}$ there are
  elements $F$, $G$ of $S$ such that $\dom (F \sqcap G) \sqsubseteq X$
  (using $\card B > 1$).

  By properties of generalized filter bases $X \times \top \sqsupseteq
  \bigsqcap^{\mathsf{RLD}} S \Leftrightarrow \exists F, G \in S : X
  \times \top \sqsupseteq F \sqcap G \Leftrightarrow X \sqsupseteq
  \mathcal{X}$. Thus $\im \bigsqcap^{\mathsf{RLD}} S =
  \mathcal{X}$.
\end{proof}

\section{Power of filters}

Let's define $\mathcal{Y}^{\mathcal{X}}$ for filters~$\mathcal{X}$,~$\mathcal{Y}$:

First define $Y^{\mathcal{X}}$ for a set~$Y$:
\[ Y^{\mathcal{X}} = \setcond{ f \in \mathsf{RLD} (\Ob \mathcal{X}
   , Y) }{ \dom f = \mathcal{X} \wedge f\text{ is monovalued} } . \]

Now $\mathcal{Y}^{\mathcal{X}} = \bigsqcap^{\mathsf{RLD}}_{Y \in
\up \mathcal{Y}} Y^{\mathcal{X}}$.

\cite{filt-cat}~defines an isomorphic to this way to define ``exponentiation'' of filters.

TODO: Check $\mathcal{Y}^1 \cong \mathcal{Y}$; $\mathcal{Z}^{\mathcal{X}
\times^{\mathsf{RLD}} \mathcal{Y}} \cong
(\mathcal{Z}^{\mathcal{X}})^{\mathcal{Y}}$; $\mathcal{Z}^{\mathcal{X} \amalg
\mathcal{Y}} \cong \mathcal{Z}^{\mathcal{X}} \times^{\mathsf{RLD}}
\mathcal{Z}^{\mathcal{Y}}$; $\mathcal{Y}^2 \cong \mathcal{Y}\times^{\mathsf{RLD}}\mathcal{Y}$;
$\mathcal{Y}^0 \cong 1$; $\mathcal{Y}^N \cong \prod^{\mathsf{RLD}}_{n\in N}\mathcal{Y}$.
More formulas at \url{https://en.wikipedia.org/wiki/Cartesian_closed_category}.

Andreas Blass says in a private email that it is not cartesian closed: ``Unfortunately, the two categories of filters in my paper are
not cartesian closed.  This is mentioned in a parenthetical comment
near the bottom of page 141.  The operation of cartesian product with
the cofinite filter on the natural numbers has no right adjoint,
because it does not preserve infinite coproducts.''
about~\cite{filt-cat}.

But it is probably a braided closed monoidal category?

See \cite{filt-cat} for more categorical properties of filters.


\section{Diagonalization of symmetric matrices}

\begin{outcome}
  \begin{enumerate}
  \item Compute the real eigenvalues of a symmetric matrix.
  \item Compute an orthogonal basis of eigenvectors for a symmetric
    matrix.
  \item Orthogonally diagonalize a symmetric matrix.
  \end{enumerate}
\end{outcome}

In Chapter~\ref{cha:eigenvalues}, we saw that some matrices are
diagonalizable, and others are not. In this section, we will see that
the theory of diagonalization is much nicer when the matrix to be
diagonalized is symmetric. Recall that a matrix $A$ is
\textbf{symmetric}%
\index{symmetric matrix}%
\index{matrix!symmetric} if $A=A^T$. We begin with some observations
about the eigenvalues and eigenvectors of a symmetric matrix.

\begin{proposition}{Eigenvalues and eigenvectors of symmetric matrices}{eigenvalues-symmetric}
  Let $A$ be a symmetric matrix with real entries. Then
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item All eigenvalues of $A$ are real (i.e., not complex).
    \item Eigenvectors for distinct eigenvalues of $A$ are orthogonal.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{proposition}

\begin{proof}
  (a) Suppose $\eigenvar$ is a (possibly complex) eigenvalue of $A$,
  with (possibly complex) eigenvector $\vect{v}$. We will show that
  $\eigenvar$ is in fact real. Let $\conjugate{\eigenvar}$ be the
  complex conjugate of $\eigenvar$, and let $\conjugate{\vect{v}}$ be
  the complex conjugate of $\vect{v}$, i.e., the vector obtained from
  $\vect{v}$ by taking the complex conjugate of each of its entries.
  Since $\vect{v}$ is an eigenvector for eigenvalue $\eigenvar$ of
  $A$, we have
  \begin{equation}\label{eqn:eigenvalues-symmetric-1}
    \conjugate{\vect{v}}^T A\vect{v}
    = \eigenvar \conjugate{\vect{v}}^T\vect{v}.
  \end{equation}
  Taking the complex conjugate of both sides of the equation, and
  using the fact that $\conjugate{A}=A$ (since $A$ is
  real), we get
  \begin{equation}
    \vect{v}^T A\conjugate{\vect{v}}
    = \conjugate{\eigenvar} \vect{v}^T\conjugate{\vect{v}}.
  \end{equation}
  Then, taking the transpose of both sides of the equation, and using
  the fact that $A^T=A$ (since $A$ is symmetric), we have
  \begin{equation}\label{eqn:eigenvalues-symmetric-2}
    \conjugate{\vect{v}}^T A\vect{v}
    = \conjugate{\eigenvar} \conjugate{\vect{v}}^T\vect{v}.
  \end{equation}
  Comparing equations {\eqref{eqn:eigenvalues-symmetric-1}} and
  {\eqref{eqn:eigenvalues-symmetric-2}}, we find that
  $\eigenvar \conjugate{\vect{v}}^T\vect{v} = \conjugate{\eigenvar}
  \conjugate{\vect{v}}^T\vect{v}$. Since
  $\conjugate{\vect{v}}^T\vect{v}$ is a non-zero scalar, it follows
  that $\eigenvar = \conjugate{\eigenvar}$, i.e., $\eigenvar$ is real.

  \noindent
  (b) Suppose $\vect{v}$ is an eigenvector for eigenvalue
  $\eigenvarA$, $\vect{w}$ is an eigenvector for eigenvalue
  $\eigenvarB$, and $\eigenvarA\neq\eigenvarB$.  We evaluate
  $\vect{v}^TA\vect{w}$ in two different ways. On the on hand, we have
  \begin{equation*}
    \vect{v}^TA\vect{w}
    = \vect{v}^T(A\vect{w})
    = \vect{v}^T(\eigenvarB\vect{w})
    = \eigenvarB \vect{v}^T\vect{w}.
  \end{equation*}
  On the other hand, we have
  \begin{equation*}
    \vect{v}^TA\vect{w}
    = (\vect{v}^TA)\vect{w}
    = (A^T\vect{v})^T\vect{w}
    = (A\vect{v})^T\vect{w}
    = (\eigenvarA\vect{v})^T\vect{w}
    = \eigenvarA\vect{v}^T\vect{w}.
  \end{equation*}
  Therefore,
  $\eigenvarB\vect{v}^T\vect{w} = \eigenvarA\vect{v}^T\vect{w}$, or
  equivalently $(\eigenvarA - \eigenvarB)\vect{v}^T\vect{w} =
  0$. Since by assumption, $\eigenvarA - \eigenvarB\neq 0$, we must
  have $\vect{v}^T\vect{w} = 0$, i.e., $\vect{v}\orth\vect{w}$.
\end{proof}

Recall that an $n\times n$-matrix $A$ is \textbf{diagonalizable}%
\index{diagonalizable matrix}%
\index{matrix!diagonalizable} if there exists an invertible matrix $P$
and a diagonal matrix $D$ such that $D = P^{-1}AP$. We say that $A$ is
\textbf{orthogonally diagonalizable}%
\index{orthogonally diagonalizable matrix}%
\index{diagonalizable matrix!orthogonally diagonalizable}%
\index{diagonalization!orthogonal diagonalization}%
\index{matrix!orthogonally diagonalizable}%
\index{matrix!diagonalizable!orthogonally} if $P$ can, moreover, be
chosen to be orthogonal. Orthogonal diagonalizability is a convenient
property, because when $P$ is orthogonal, then $P^{-1}=P^T$, and we
can interchangeably write $D = P^{-1}AP$ or $D = P^TAP$.  The
following is the main theorem about the diagonalization of real
symmetric matrices.

\begin{theorem}{Diagonalization of symmetric matrices}{diagonalization-symmetric}
  Every real symmetric matrix $A$ is orthogonally diagonalizable.
\end{theorem}

\begin{proof}
  By induction on the size of the matrix. For a $1\times 1$-matrix,
  there is nothing to show, as it is already diagonal. Now consider a
  real symmetric $n\times n$-matrix $A$ with $n\geq 2$. By the
  fundamental theorem of algebra, the characteristic polynomial has at
  least one root, so that $A$ has at least one (possibly complex)
  eigenvalue $\eigenvar$. By Proposition~\ref{prop:eigenvalues-symmetric}(a),
  $\eigenvar$ is real. Since we can solve the system of equations
  $(A-\eigenvar I)\vect{v}=\vect{0}$ over the real numbers, there
  exists a real eigenvector $\vect{v}$ for the eigenvalue
  $\eigenvar$. We can assume without loss of generality that
  $\vect{v}$ is normalized, because else we could replace $\vect{v}$
  by $\frac{1}{\norm{\vect{v}}}\vect{v}$. By the Gram-Schmidt method,
  we can find an orthonormal basis
  $\set{\vect{u}_1,\ldots,\vect{u}_n}$ of $\R^n$ such that
  $\vect{u}_1=\vect{v}$. Let $Q$ be the orthogonal matrix that has
  $\vect{u}_1,\ldots,\vect{u}_n$ as its columns, and consider
  $B=Q^{-1}AQ$. Since
  $B\vect{e}_1 = Q^{-1}AQ\vect{e}_1 = Q^{-1}A\vect{u}_1 = \eigenvar
  Q^{-1}\vect{u}_1 = \eigenvar\vect{e}_1$, the matrix $B$ is of the
  form
  \begin{equation*}
    B = \begin{mymatrix}{cccc}
      \eigenvar & b_{12} & \cdots & b_{1n} \\
      0 & b_{22} & \cdots & b_{2n} \\
      \vdots & \vdots & \ddots & \vdots \\
      0 & b_{n2} & \cdots & b_{nn} \\
    \end{mymatrix}.
  \end{equation*}
  Moreover, since $B=Q^{-1}AQ = Q^TAQ$, the matrix $B$ is symmetric.
  Therefore $b_{12},\ldots,b_{1n} = 0$, and $B$ is of the form
  \begin{equation*}
    B = \begin{mymatrix}{c|ccc}
      \eigenvar & 0 & \cdots & 0 \\\hline
      0 &  & & \\
      \vdots & & C & \\
      0 &  & & \\
    \end{mymatrix},
  \end{equation*}
  where $C$ is a symmetric matrix of dimension $n-1$. By induction
  hypothesis, $C$ is orthogonally diagonalizable, i.e., there exists
  an orthogonal matrix $R$ such that $R^{-1}CR=D$ is diagonal. Let
  \begin{equation*}
    S = \begin{mymatrix}{c|ccc}
      1 & 0 & \cdots & 0 \\\hline
      0 &  & & \\
      \vdots & & R & \\
      0 &  & & \\
    \end{mymatrix}.
  \end{equation*}
  Then $S$ is orthogonal by
  Proposition~\ref{prop:properties-orthogonal}(\ref{item:properties-orthogonal-d}).
  Let $E = S^{-1}BS$. Then
  \begin{equation*}
    E =
    S^{-1}BS =
    \begin{mymatrix}{c|ccc}
      \eigenvar & 0 & \cdots & 0 \\\hline
      0 &  & & \\
      \vdots & & R^{-1}CR & \\
      0 &  & & \\
    \end{mymatrix}
    =
    \begin{mymatrix}{c|ccc}
      \eigenvar & 0 & \cdots & 0 \\\hline
      0 &  & & \\
      \vdots & & D & \\
      0 &  & & \\
    \end{mymatrix}.
  \end{equation*}
  Therefore, $E$ is diagonal. Let $P=QS$. Then $P$ is orthogonal
  by
  Proposition~\ref{prop:properties-orthogonal}(\ref{item:properties-orthogonal-a}). Moreover,
  we have
  \begin{equation*}
    P^{-1}AP = S^{-1}Q^{-1}AQS = S^{-1}BS = E.
  \end{equation*}
  Therefore, $A$ is orthogonally diagonalizable.
\end{proof}

\begin{example}{Diagonalization of symmetric matrices}{diagonalization-symmetric}
  Orthogonally diagonalize the matrix
  \begin{equation*}
    A = \begin{mymatrix}{rr}
      3 & 2 \\
      2 & 6 \\
    \end{mymatrix},
  \end{equation*}
  i.e., find an orthogonal matrix $P$ and a diagonal matrix $D$ such
  that $D = P^{-1}AP$.
\end{example}

\begin{solution}
  We proceed in much the same way as in Chapter~\ref{cha:eigenvalues},
  except that at a crucial moment, we ensure that $P$ is orthogonal.
  We start by calculating the characteristic polynomial of $A$:
  \begin{equation*}
    \det(A-\eigenvar I)
    ~=~ \begin{absmatrix}{cc}
      3-\eigenvar & 2 \\
      2 & 6-\eigenvar
    \end{absmatrix}
    ~=~ (3-\eigenvar)(6-\eigenvar) - 4
    ~=~ \eigenvar^2 - 9\eigenvar + 14.
  \end{equation*}
  We find the roots using the quadratic formula. The roots of the
  characteristic polynomial, and therefore the eigenvalues of $A$, are
  $\eigenvar=7$ and $\eigenvar=2$. For the eigenvalue $\eigenvar=7$,
  we find the eigenvector
  \begin{equation*}
    \vect{v} = \begin{mymatrix}{r} 1 \\ 2 \end{mymatrix},
  \end{equation*}
  and for the eigenvalue $\eigenvar=2$, we find the eigenvector
  \begin{equation*}
    \vect{w} = \begin{mymatrix}{r} -2 \\ 1 \end{mymatrix}.
  \end{equation*}
  We note that these two eigenvectors are orthogonal to each other,
  exactly as predicted by Proposition~\ref{prop:eigenvalues-symmetric}. So
  the vectors $\set{\vect{v},\vect{w}}$ form an orthogonal set. We
  turn this into an orthonormal set by normalizing each eigenvector,
  i.e., the normalized eigenvectors are
  \begin{equation*}
    \vect{u}_1
    ~=~ \frac{1}{\norm{\vect{v}}}\vect{v}
    ~=~ \frac{1}{\sqrt{5}} \begin{mymatrix}{r} 1 \\ 2 \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{u}_2
    ~=~ \frac{1}{\norm{\vect{w}}}\vect{w}
    ~=~ \frac{1}{\sqrt{5}} \begin{mymatrix}{r} -2 \\ 1 \end{mymatrix}.
  \end{equation*}
  We let $P$ be the matrix that has columns $\vect{u}_1$ and
  $\vect{u}_2$, i.e.,
  \begin{equation*}
    P = \frac{1}{\sqrt{5}} \begin{mymatrix}{rr} 1 & -2 \\ 2 & 1 \end{mymatrix}.
  \end{equation*}
  Note that since $\vect{u}_1$ and $\vect{u}_2$ are orthonormal, $P$
  is automatically orthogonal by
  Proposition~\ref{prop:conditions-orthogonal-matrix}. Moreover, we
  have
  \begin{equation*}
    P^{-1}
    ~=~ P^T
    ~=~ \frac{1}{\sqrt{5}} \begin{mymatrix}{rr} 1 & 2 \\ -2 & 1 \end{mymatrix}.
  \end{equation*}
  Finally,
  \begin{equation*}
    P^{-1}AP
    ~=~ P^TAP
    ~=~ \frac{1}{5}
    \begin{mymatrix}{rr} 1 & 2 \\ -2 & 1 \end{mymatrix}
    \begin{mymatrix}{rr} 3 & 2 \\ 2 & 6 \end{mymatrix}
    \begin{mymatrix}{rr} 1 & -2 \\ 2 & 1 \end{mymatrix}
    ~=~ \begin{mymatrix}{rr} 7 & 0 \\ 0 & 2 \end{mymatrix}.
  \end{equation*}
\end{solution}

\begin{example}{Diagonalization of symmetric matrices}{diagonalization-symmetric2}
  Find an orthogonal matrix $P$ and a diagonal matrix $D$ such $D =
  P^{-1}AP$, where
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      3  & 1 & -2 \\
      1  & 3 &  2 \\
      -2 & 2 &  0 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  Again, we start by calculating the characteristic polynomial and its roots:
  \begin{equation*}
    \det(A-\eigenvar I)
    ~=~ \begin{absmatrix}{ccc}
      3-\eigenvar  & 1 & -2 \\
      1  & 3-\eigenvar &  2 \\
      -2 & 2 &  -\eigenvar  \\
    \end{absmatrix}
    ~=~ - \eigenvar^3 + 6\eigenvar^2 -32.
  \end{equation*}
  The roots are $\eigenvar=4$ and $\eigenvar=-2$. To find the
  eigenvectors for $\eigenvar=4$, we solve the system of equations
  \begin{equation*}
    (A-4I)\vect{v}
    =
    \begin{mymatrix}{rrr}
      -1  & 1 & -2 \\
      1  & -1 &  2 \\
      -2 & 2 &  -4 \\
    \end{mymatrix}\vect{v}
    = \vect{0}.
  \end{equation*}
  The solution space is 2-dimensional, with basis
  \begin{equation*}
    \vect{v}_1 = \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{v}_2 = \begin{mymatrix}{r} 0 \\ 2 \\ 1 \end{mymatrix}.
  \end{equation*}
  To find the eigenvectors for $\eigenvar=-2$, we solve the system of
  equations
  \begin{equation*}
    (A+2I)\vect{v}
    =
    \begin{mymatrix}{rrr}
      5  & 1 & -2 \\
      1  & 5 &  2 \\
      -2 & 2 &  2 \\
    \end{mymatrix}\vect{v}
    = \vect{0}.
  \end{equation*}
  The solution space is 1-dimensional, with basis
  \begin{equation*}
    \vect{v}_3 = \begin{mymatrix}{r} 1 \\ -1 \\ 2 \end{mymatrix}.
  \end{equation*}
  As predicted by Proposition~\ref{prop:eigenvalues-symmetric},
  $\vect{v}_3$ is orthogonal to $\vect{v}_1$ and
  $\vect{v}_2$. However, there is a slight complication: $\vect{v}_1$
  and $\vect{v}_2$ are not orthogonal to each other. This is because
  they are eigenvectors for the {\em same} eigenvalue ($\eigenvar=4$),
  not for {\em distinct} eigenvalues. We found basis
  $\set{\vect{v}_1,\vect{v}_2}$ for the eigenspace for $\eigenvar=4$,
  but it doesn't happen to be an orthogonal basis. However, we can fix
  this by applying the Gram-Schmidt method to
  $\set{\vect{v}_1,\vect{v}_2}$. This yields the orthogonal basis
  $\set{\vect{u}_1,\vect{u}_2}$ for the eigenspace for $\eigenvar=4$,
  where
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{u}_2 = \begin{mymatrix}{r} -1 \\ 1 \\ 1 \end{mymatrix}.
  \end{equation*}
  We now have an orthogonal basis
  $\set{\vect{u}_1,\vect{u}_2,\vect{v}_3}$ of eigenvectors of the
  matrix $A$. We normalize the three vectors and use them as the
  columns of $P$:
  \begin{equation*}
    \def\arraystretch{1.4}
    P = \begin{mymatrix}{ccc}
      \frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{3}} & \frac{1}{\sqrt{6}}  \\
      \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{3}}  & -\frac{1}{\sqrt{6}} \\
      0                  & \frac{1}{\sqrt{3}}  & \frac{2}{\sqrt{6}}  \\
    \end{mymatrix}.
  \end{equation*}
  It seems that inverting $P$ will not be all that easy, but in fact,
  since $P$ is orthogonal, the inverse is just the transpose:
  \begin{equation*}
    \def\arraystretch{1.4}
    P^{-1}
    ~=~ P^T
    ~=~ \begin{mymatrix}{ccc}
      \frac{1}{\sqrt{2}}  & \frac{1}{\sqrt{2}}  & 0 \\
      -\frac{1}{\sqrt{3}} & \frac{1}{\sqrt{3}}  & \frac{1}{\sqrt{3}} \\
      \frac{1}{\sqrt{6}}  & -\frac{1}{\sqrt{6}} & \frac{2}{\sqrt{6}} \\
    \end{mymatrix}.
  \end{equation*}
  We have
  \begin{equation*}
    P^{-1}AP
    ~=~ D
    ~=~ \begin{mymatrix}{ccc} 4 & 0 & 0 \\ 0 & 4 & 0 \\ 0 & 0 & -2 \end{mymatrix}.
  \end{equation*}
\end{solution}