\chapter{Gaussian Models}
\label{chap:MVN}

In this chapter, we discuss the \textbf{multivariate Gaussian} or \textbf{multivariate normal(MVN)}, which is the most widely used joint probability density function for continuous variables. It will form the basis for many of the models we will encounter in later chapters.

\section{Basics}
Recall from Section \ref{sec:MVN} that the pdf for an MVN in $D$ dimensions is defined by the following:
\begin{equation}
\mathcal{N}(\vec{x}|\vec{\mu},\Sigma) \triangleq \dfrac{1}{(2\pi)^{\frac{D}{2}}|\Sigma|^{\frac{1}{2}}}\exp\left[-\dfrac{1}{2}(\vec{x}-\vec{\mu})^T\Sigma^{-1}(\vec{x}-\vec{\mu})\right]
\end{equation}

The expression inside the exponent is the Mahalanobis distance between a data vector $\vec{x}$ and the mean vector $\vec{\mu}$, We can gain a better understanding of this quantity by performing an \textbf{eigendecomposition} of $\vec{\Sigma}$. That is, we write $\vec{\Sigma}=\vec{U}\vec{\Lambda}\vec{U}^T$, where $\vec{U}$ is an orthonormal matrix of eigenvectors satsifying $\vec{U}^T\vec{U}=\vec{I}$, and $\vec{\Lambda}$ is a diagonal matrix of eigenvalues. Using the eigendecomposition, we have that
\begin{equation}
\vec{\Sigma}^{-1}=\vec{U}^{-T}\vec{\Lambda}^{-1}\vec{U}^{-1}=\vec{U}\vec{\Lambda}^{-1}\vec{U}^T=\sum\limits_{i=1}^D \dfrac{1}{\lambda_i}\vec{u}_i\vec{u}_i^T
\end{equation}
where $\vec{u}_i$ is the $i$'th column of $\vec{U}$, containing the $i$'th eigenvector. Hence we can rewrite the Mahalanobis distance as follows:
\begin{align}
(\vec{x}-\vec{\mu})^T\Sigma^{-1}(\vec{x}-\vec{\mu}) & =(\vec{x}-\vec{\mu})^T\left(\sum\limits_{i=1}^D \dfrac{1}{\lambda_i}\vec{u}_i\vec{u}_i^T\right)(\vec{x}-\vec{\mu}) \\
    & =\sum\limits_{i=1}^D \dfrac{1}{\lambda_i}(\vec{x}-\vec{\mu})^T\vec{u}_i\vec{u}_i^T(\vec{x}-\vec{\mu}) \\
	& =\sum\limits_{i=1}^D \dfrac{y_i^2}{\lambda_i}
\end{align}
where $y_i \triangleq \vec{u}_i^T(\vec{x}-\vec{\mu})$. Recall that the equation for an ellipse in 2d is
\begin{equation}
\dfrac{y_1^2}{\lambda_1}+\dfrac{y_2^2}{\lambda_2}=1
\end{equation}

Hence we see that the contours of equal probability density of a Gaussian lie along ellipses. This is illustrated in Figure \ref{fig:2d-MVN}. The eigenvectors determine the orientation of the ellipse, and the eigenvalues determine how elogonated it is.

\begin{figure}[hbtp]
\centering
    \includegraphics[scale=.70]{2d-MVN.png}
\caption{Visualization of a 2 dimensional Gaussian density. The major and minor axes of the ellipse are defined by the first two eigenvectors of the covariance matrix, namely $\vec{u}_1$ and $\vec{u}_2$. Based on Figure 2.7 of (Bishop 2006a)}
\label{fig:2d-MVN}
\end{figure}

In general, we see that the Mahalanobis distance corresponds to Euclidean distance in a transformed coordinate system, where we shift by $\vec{\mu}$ and rotate by $\vec{U}$.

\subsection{MLE for a MVN}
\begin{theorem}(\textbf{MLE for a MVN})
If we have $N$ iid samples $\vec{x}_i \sim \mathcal{N}(\vec{\mu},\vec{\Sigma})$, then the MLE for the parameters is given by
\begin{align}
\bar{\vec{\mu}}    & =\dfrac{1}{N}\sum\limits_{i=1}^N \vec{x}_i \triangleq \bar{\vec{x}} \\
\bar{\vec{\Sigma}} & =\dfrac{1}{N}\sum\limits_{i=1}^N (\vec{x}_i-\bar{\vec{x}})(\vec{x}_i-\bar{\vec{x}})^T \\
                     & =\dfrac{1}{N}\left(\sum\limits_{i=1}^N \vec{x}_i\vec{x}_i^T\right)-\bar{\vec{x}}\bar{\vec{x}}^T
\end{align}
\end{theorem}

\subsection{Maximum entropy derivation of the Gaussian *}
In this section, we show that the multivariate Gaussian is the distribution with maximum entropy subject to having a specified mean and covariance (see also Section TODO). This is one reason the Gaussian is so widely used: the first two moments are usually all that we can reliably estimate from data, so we want a distribution that captures these properties, but otherwise makes as few addtional assumptions as possible.

To simplify notation, we will assume the mean is zero. The pdf has the form
\begin{equation}
f(\vec{x})=\dfrac{1}{Z}\exp\left(-\dfrac{1}{2}\vec{x}^T\vec{\Sigma}^{-1}\vec{x}\right)
\end{equation}

\section{Gaussian discriminant analysis}
One important application of MVNs is to define the the class conditional densities in a generative classifier, i.e.,
\begin{equation}
p(\vec{x}|y=c,\vec{\theta})=\mathcal{N}(\vec{x}|\vec{\mu}_c,\vec{\Sigma}_c)
\end{equation}

The resulting technique is called (Gaussian) \textbf{discriminant analysis} or \textbf{GDA} (even though it is a generative, not discriminative, classifier — see Section TODO for more on this distinction). If $\vec{\Sigma}_c$ is diagonal, this is equivalent to naive Bayes.

We can classify a feature vector using the following decision rule, derived from Equation \ref{eqn:Generative-classifier}:
\begin{equation}
y=\arg\max_{c} \left[\log p(y=c|\vec{\pi})+\log p(\vec{x}|\vec{\theta})\right]
\end{equation}

When we compute the probability of $\vec{x}$ under each class conditional density, we are measuring the distance from $\vec{x}$ to the center of each class, $\vec{\mu}_c$, using Mahalanobis distance. This can be thought of as a \textbf{nearest centroids classifier}.

\begin{figure}[hbtp]
\centering
\subfloat[]{\includegraphics[scale=.70]{2d-Gaussians-Visualization-a.png}} \\
\subfloat[]{\includegraphics[scale=.70]{2d-Gaussians-Visualization-b.png}}
\caption{(a) Height/weight data. (b) Visualization of 2d Gaussians fit to each class. 95\% of the probability mass is inside the ellipse.}
\label{fig:2d-Gaussians-Visualization}
\end{figure}

As an example, Figure \ref{fig:2d-Gaussians-Visualization} shows two Gaussian class-conditional densities in 2d, representing the height and weight of men and women. We can see that the features are correlated, as is to be expected (tall people tend to weigh more). The ellipses for each class contain 95\% of the probability mass. If we have a uniform prior over classes, we can classify a new test vector as follows:
\begin{equation}
y=\arg\max_{c} (\vec{x}-\vec{\mu}_c)^T\vec{\Sigma}_c^{-1}(\vec{x}-\vec{\mu}_c)
\end{equation}

\subsection{Quadratic discriminant analysis (QDA)}
By plugging in the definition of the Gaussian density to Equation \ref{eqn:Generative-classifier}, we can get
\begin{equation}\label{eqn:QDA}
p(y|\vec{x},\vec{\theta})=\dfrac{\pi_c|2\pi\vec{\Sigma}_c|^{-\frac{1}{2}}\exp\left[-\frac{1}{2}(\vec{x}-\vec{\mu})^T\vec{\Sigma}^{-1}(\vec{x}-\vec{\mu})\right]}{\sum_{c'}\pi_{c'}|2\pi\vec{\Sigma}_{c'}|^{-\frac{1}{2}}\exp\left[-\frac{1}{2}(\vec{x}-\vec{\mu})^T\vec{\Sigma}^{-1}(\vec{x}-\vec{\mu})\right]}
\end{equation}

Thresholding this results in a quadratic function of $\vec{x}$. The result is known as quadratic discriminant analysis(QDA). Figure \ref{fig:QDA} gives some examples of what the decision boundaries look like in 2D.

\begin{figure}[hbtp]
\centering
\subfloat[]{\includegraphics[scale=.50]{QDA-a.png}} \\
\subfloat[]{\includegraphics[scale=.50]{QDA-b.png}}
\caption{Quadratic decision boundaries in 2D for the 2 and 3 class case.}
\label{fig:QDA}
\end{figure}

\subsection{Linear discriminant analysis (LDA)}
\label{sec:Linear-discriminant-analysis}
We now consider a special case in which the covariance matrices are \textbf{tied} or \textbf{shared} across classes,$\vec{\Sigma}_c=\vec{\Sigma}$. In this case, we can simplify Equation \ref{eqn:QDA} as follows:
\begin{align}
p(y|\vec{x},\vec{\theta})& \propto \pi_c\exp\left(\vec{\mu}_c\vec{\Sigma}^{-1}\vec{x}-\dfrac{1}{2}\vec{x}^T\vec{\Sigma}^{-1}\vec{x}-\dfrac{1}{2}\vec{\mu}_c^T\vec{\Sigma}^{-1}\vec{\mu}_c\right) \nonumber \\
 & =\exp\left(\vec{\mu}_c\vec{\Sigma}^{-1}\vec{x}-\dfrac{1}{2}\vec{\mu}_c^T\vec{\Sigma}^{-1}\vec{\mu}_c+\log \pi_c\right) \nonumber \\
 & \quad \exp\left(-\dfrac{1}{2}\vec{x}^T\vec{\Sigma}^{-1}\vec{x}\right) \nonumber \\
 & \propto \exp\left(\vec{\mu}_c\vec{\Sigma}^{-1}\vec{x}-\dfrac{1}{2}\vec{\mu}_c^T\vec{\Sigma}^{-1}\vec{\mu}_c+\log \pi_c\right)
\end{align}

Since the quadratic term $\vec{x}^T\vec{\Sigma}^{-1}\vec{x}$ is independent of $c$, it will cancel out in the numerator and denominator. If we define
\begin{align}
\gamma_c& \triangleq -\dfrac{1}{2}\vec{\mu}_c^T\vec{\Sigma}^{-1}\vec{\mu}_c+\log \pi_c \\
\vec{\beta}_c& \triangleq \vec{\Sigma}^{-1}\vec{\mu}_c
\end{align}
then we can write
\begin{equation}\label{eqn:LDA}
p(y|\vec{x},\vec{\theta})=\dfrac{e^{\vec{\beta}_c^T\vec{x}+\gamma_c}}{\sum_{c'}e^{\vec{\beta}_{c'}^T\vec{x}+\gamma_{c'}}} \triangleq \sigma(\vec{\eta}, c)
\end{equation}
where $\vec{\eta} \triangleq (e^{\vec{\beta}_1^T\vec{x}}+\gamma_1,\cdots, e^{\vec{\beta}_C^T\vec{x}}+\gamma_C)$, $\sigma()$ is the \textbf{softmax activation function}\footnote{\url{http://en.wikipedia.org/wiki/Softmax_activation_function}}, defined as follows:
\begin{equation}
\sigma(\vec{q},i) \triangleq \dfrac{\exp(q_i)}{\sum_{j=1}^n \exp(q_j)}
\end{equation}

When parameterized by some constant, $\alpha > 0$, the following formulation becomes a smooth, differentiable approximation of the maximum function:
\begin{equation}
\mathcal{S}_{\alpha}(\vec{x}) = \dfrac{\sum_{j=1}^D x_je^{\alpha x_j}}{\sum_{j=1}^D e^{\alpha x_j}}
\end{equation}

$\mathcal{S}_{\alpha}$ has the following properties:
\begin{enumerate}
\item $\mathcal{S}_{\alpha} \rightarrow \max$ as $\alpha \rightarrow \infty$
\item $\mathcal{S}_0$ is the average of its inputs
\item $\mathcal{S}_{\alpha} \rightarrow \min$ as $\alpha \rightarrow -\infty$
\end{enumerate}

Note that the softmax activation function comes from the area of statistical physics, where it is common to use the \textbf{Boltzmann distribution}, which has the same form as the softmax activation function.

An interesting property of Equation \ref{eqn:LDA} is that, if we take logs, we end up with a linear function of $\vec{x}$. (The reason it is linear is because the $\vec{x}^T\vec{\Sigma}^{-1}\vec{x}$ cancels from the numerator and denominator.) Thus the decision boundary between any two classes, says $c$ and $c'$, will be a straight line. Hence this technique is called \textbf{linear discriminant analysis} or \textbf{LDA}.

An alternative to fitting an LDA model and then deriving the class posterior is to directly fit $p(y|\vec{x},\vec{W})=\text{Cat}(y|\vec{W}\vec{x})$ for some $C \times D$ weight matrix $\vec{W}$. This is called \textbf{multi-class logistic regression}, or \textbf{multinomial logistic regression}. We will discuss this model in detail in Section TODO. The difference between the two approaches is explained in Section TODO.

\subsection{Two-class LDA}
To gain further insight into the meaning of these equations, let us consider the binary case. In this case, the posterior is given by
\begin{align}
p(y=1|\vec{x},\vec{\theta})& =\dfrac{e^{\vec{\beta}_1^T\vec{x}+\gamma_1}}{e^{\vec{\beta}_0^T\vec{x}+\gamma_0}+e^{\vec{\beta}_1^T\vec{x}+\gamma_1}}) \\
  & =\dfrac{1}{1+e^(\vec{\beta}_0-\vec{\beta}_1)^T\vec{x}+(\gamma_0-\gamma_1)} \\
  & =\text{sigm}((\vec{\beta}_1-\vec{\beta}_0)^T\vec{x}+(\gamma_0-\gamma_1))
\end{align}
where sigm$(x)$ refers to the sigmoid function\footnote{\url{http://en.wikipedia.org/wiki/Sigmoid_function}}.

Now
\begin{align}
\gamma_1-\gamma_0& = -\dfrac{1}{2}\vec{\mu}_1^T\vec{\Sigma}^{-1}\vec{\mu}_1+\dfrac{1}{2}\vec{\mu}_0^T\vec{\Sigma}^{-1}\vec{\mu}_0 + \log(\pi_1/\pi_0) \\
 & =-\dfrac{1}{2}(\vec{\mu}_1-\vec{\mu}_0)^T\vec{\Sigma}^{-1}(\vec{\mu}_1+\vec{\mu}_0)+ \log(\pi_1/\pi_0)
\end{align}

So if we define
\begin{align}
\vec{w}& =\vec{\beta}_1-\vec{\beta}_0=\vec{\Sigma}^{-1}(\vec{\mu}_1-\vec{\mu}_0) \\
\vec{x}_0& =\dfrac{1}{2}(\vec{\mu}_1+\vec{\mu}_0)-(\vec{\mu}_1-\vec{\mu}_0)\dfrac{\log(\pi_1/\pi_0)}{(\vec{\mu}_1-\vec{\mu}_0)^T\vec{\Sigma}^{-1}(\vec{\mu}_1-\vec{\mu}_0)}
\end{align}
then we have $\vec{w}^T\vec{x}_0=-(\gamma_1-\gamma_0)$, and hence
\begin{equation}
p(y=1|\vec{x},\vec{\theta})=\text{sigm}(\vec{w}^T(\vec{x}-\vec{x}_0))
\end{equation}

(This is closely related to logistic regression, which we will discuss in Section TODO.) So the final decision rule is as follows: shift $\vec{x}$ by $\vec{x}_0$, project onto the line \vec{w}, and see if the result is positive or negative.

\begin{figure}[hbtp]
\centering
    \includegraphics[scale=.50]{2d-LDA.png}
\caption{Geometry of LDA in the 2 class case where $\vec{\Sigma}_1=\vec{\Sigma}_2=\vec{I}$.}
\label{fig:2d-LDA}
\end{figure}

If $\vec{\Sigma}=\sigma^2\vec{I}$, then $\vec{w}$ is in the direction of $\vec{\mu}_1-\vec{\mu}_0$. So we classify the point based on whether its projection is closer to $\vec{\mu}_0$ or $\vec{\mu}_1$ . This is illustrated in Figure \ref{fig:2d-LDA}. Furthemore, if $\vec{\pi}_1=\vec{\pi}_0$, then $\vec{x}_0=\frac{1}{2}(\vec{\mu}_1+\vec{\mu}_0)$, which is half way between the means. If we make $\vec{\pi}_1>\vec{\pi}_0$, then $\vec{x}_0$ gets closer to $\vec{\mu}_0$, so more of the line belongs to class 1 a \emph{priori}. Conversely if $\vec{\pi}_1<\vec{\pi}_0$, the boundary shifts right. Thus we see that the class prior, πc, just changes the decision threshold, and not the overall geometry, as we claimed above. (A similar argument applies in the multi-class case.)

The magnitude of $\vec{w}$ determines the steepness of the logistic function, and depends on how well-separated the means are, relative to the variance. In psychology and signal detection theory, it is common to define the \textbf{discriminability} of a signal from the background noise using a quantity called \textbf{d-prime}:
\begin{equation}
d' \triangleq \dfrac{\mu_1-\mu_0}{\sigma}
\end{equation}
where $\mu_1$ is the mean of the signal and $\mu_0$ is the mean of the noise, and $\sigma$ is the standard deviation of the noise. If $d'$ is large, the signal will be easier to discriminate from the noise.

\subsection{MLE for discriminant analysis}
\label{sec:MLE-for-discriminant-analysis}
The log-likelihood function is as follows:
\begin{equation}
p(\mathcal{D}|\vec{\theta})=\sum\limits_{c=1}^C{\sum\limits_{i:y_i=c}{\log\pi_c}}+\sum\limits_{c=1}^C{\sum\limits_{i:y_i=c}{\log\mathcal{N}(\vec{x}_i|\vec{\mu}_c,\vec{\Sigma}_c)}}
\end{equation}

The MLE for each parameter is as follows:
\begin{align}
\bar{\vec{\mu}}_c& = \dfrac{N_c}{N} \\
\bar{\vec{\mu}}_c& = \dfrac{1}{N_c}\sum\limits_{i:y_i=c}\vec{x}_i \\
\bar{\vec{\Sigma}}_c& = \dfrac{1}{N_c}\sum\limits_{i:y_i=c}(\vec{x}_i-\bar{\vec{\mu}}_c)(\vec{x}_i-\bar{\vec{\mu}}_c)^T
\end{align}

\subsection{Strategies for preventing overfitting}
The speed and simplicity of the MLE method is one of its greatest appeals. However, the MLE can badly overfit in high dimensions. In particular, the MLE for a full covariance matrix is singular if $N_c <D$. And even when $N_c >D$, the MLE can be ill-conditioned, meaning it is close to singular. There are several possible solutions to this problem:
\begin{itemize}
\item{Use a diagonal covariance matrix for each class, which assumes the features are conditionally independent; this is equivalent to using a naive Bayes classifier (Section \ref{sec:NBC})}.
\item{Use a full covariance matrix, but force it to be the same for all classes,$\vec{\Sigma}_c=\vec{\Sigma}$. This is an example of \textbf{parameter tying} or \textbf{parameter sharing}, and is equivalent to LDA (Section \ref{sec:Linear-discriminant-analysis}).}
\item{Use a diagonal covariance matrix and forced it to be shared. This is called diagonal covariance LDA, and is discussed in Section TODO.}
\item{Use a full covariance matrix, but impose a prior and then integrate it out. If we use a conjugate prior, this can be done in closed form, using the results from Section TODO; this is analogous to the “Bayesian naive Bayes” method in Section \ref{sec:Bayesian-naive-Bayes}. See (Minka 2000f) for details.}
\item{Fit a full or diagonal covariance matrix by MAP estimation. We discuss two different kindsof prior below.}
\item{Project the data into a low dimensional subspace and fit the Gaussians there. See Section TODO for a way to find the best (most discriminative) linear projection.}
\end{itemize}

We discuss some of these options below.

\subsection{Regularized LDA *}

\subsection{Diagonal LDA}

\subsection{Nearest shrunken centroids classifier *}
One drawback of diagonal LDA is that it depends on all of the features. In high dimensional problems, we might prefer a method that only depends on a subset of the features, for reasons of accuracy and interpretability. One approach is to use a screening method, perhaps based on mutual information, as in Section 3.5.4. We now discuss another approach to this problem known as the \textbf{nearest shrunken centroids} classifier (Hastie et al. 2009, p652).

\section{Inference in jointly Gaussian distributions}
\label{sec:Inference-in-jointly-Gaussian-distributions}
Given a joint distribution, $p(\vec{x}_1,\vec{x}_2)$, it is useful to be able to compute marginals $p(\vec{x}_1)$ and conditionals $p(\vec{x}_1|\vec{x}_2)$. We discuss how to do this below, and then give some applications. These operations take $O(D^3)$ time in the worst case. See Section TODO for faster methods.

\subsection{Statement of the result}

\begin{theorem}(\textbf{Marginals and conditionals of an MVN}). Suppose $X=(\vec{x}_1,\vec{x}_2)$is jointly Gaussian with parameters
\begin{equation}
\vec{\mu}=\left(\begin{array}{c}\vec{\mu}_1 \\
                                \vec{\mu}_2\end{array}\right),
\vec{\Sigma}=\left(\begin{array}{cc}
                   \vec{\Sigma}_{11} & \vec{\Sigma}_{12} \\
 				   \vec{\Sigma}_{21} & \vec{\Sigma}_{22} \end{array}\right),
\vec{\Lambda}=\vec{\Sigma}^{-1}=\left(\begin{array}{cc}
                   \vec{\Lambda}_{11} & \vec{\Lambda}_{12} \\
 				   \vec{\Lambda}_{21} & \vec{\Lambda}_{22} \end{array}\right),
\end{equation}

Then the marginals are given by
\begin{equation}
\begin{split}
p(\vec{x}_1)= \mathcal{N}(\vec{x}_1|\vec{\mu}_1,\vec{\Sigma}_{11})\\
p(\vec{x}_2)= \mathcal{N}(\vec{x}_2|\vec{\mu}_2,\vec{\Sigma}_{22})
\end{split}
\end{equation}
and the posterior conditional is given by
\begin{equation}\label{eqn:Marginals-and-conditionals-of-an-MVN}
  \boxed{\begin{split}
    p(\vec{x}_1|\vec{x}_2)& =\mathcal{N}(\vec{x}_1|\vec{\mu}_{1|2},\vec{\Sigma}_{1|2}) \\
    \vec{\mu}_{1|2}& = \vec{\mu}_1+\vec{\Sigma}_{12}\vec{\Sigma}_{22}^{-1}(\vec{x}_2-\vec{\mu}_2) \\
	               & = \vec{\mu}_1-\vec{\Lambda}_{11}^{-1}\vec{\Lambda}_{12}(\vec{x}_2-\vec{\mu}_2) \\
				   & = \vec{\Sigma}_{1|2}(\vec{\Lambda}_{11}\vec{\mu}_1-\vec{\Lambda}_{12}(\vec{x}_2-\vec{\mu}_2)) \\
	\vec{\Sigma}_{1|2}& = \vec{\Sigma}_{11}-\vec{\Sigma}_{12}\vec{\Sigma}_{22}^{-1}\vec{\Sigma}_{21}=\vec{\Lambda}_{11}^{-1}
  \end{split}}
\end{equation}
\end{theorem}

Equation \ref{eqn:Marginals-and-conditionals-of-an-MVN} is of such crucial importance in this book that we have put a box around it, so you can easily find it. For the proof, see Section TODO.

We see that both the marginal and conditional distributions are themselves Gaussian. For the marginals, we just extract the rows and columns corresponding to $\vec{x}_1$ or $\vec{x}_2$. For the conditional, we have to do a bit more work. However, it is not that complicated: the conditional mean is just a linear function of $\vec{x}_2$, and the conditional covariance is just a constant matrix that is independent of $\vec{x}_2$. We give three different (but equivalent) expressions for the posterior mean, and two different (but equivalent) expressions for the posterior covariance; each one is useful in different circumstances.

\subsection{Examples}
Below we give some examples of these equations in action, which will make them seem more intuitive.

\subsubsection{Marginals and conditionals of a 2d Gaussian}

\section{Linear Gaussian systems}
Suppose we have two variables, $\vec{x}$ and $\vec{y}$.Let $\vec{x} \in \mathbb{R}^{D_x}$ be a hidden variable, and $\vec{y} \in \mathbb{R}^{D_y}$ be a noisy observation of $\vec{x}$. Let us assume we have the following prior and likelihood:
\begin{equation}\label{eqn:Linear-Gaussian-system}
  \boxed{\begin{split}
    p(\vec{x})&=\mathcal{N}(\vec{x}|\vec{\mu}_x,\vec{\Sigma}_x) \\
	p(\vec{y}|\vec{x})&=\mathcal{N}(\vec{y}|\vec{W}\vec{x}+\vec{\mu}_y,\vec{\Sigma}_y)
  \end{split}}
\end{equation}
where $\vec{W}$ is a matrix of size $D_y \times D_x$. This is an example of a \textbf{linear Gaussian system}. We can represent this schematically as $\vec{x} \rightarrow \vec{y}$, meaning $\vec{x}$ generates $\vec{y}$. In this section, we show how to “invert the arrow”, that is, how to infer $\vec{x}$ from $\vec{y}$. We state the result below, then give several examples, and finally we derive the result. We will see many more applications of these results in later chapters.

\subsection{Statement of the result}
\begin{theorem}(\textbf{Bayes rule for linear Gaussian systems}).
Given a linear Gaussian system, as in Equation \ref{eqn:Linear-Gaussian-system}, the posterior $p(\vec{x}|\vec{y})$ is given by the following:
\begin{equation}\label{eqn:Linear-Gaussian-system-posterior}
  \boxed{\begin{split}
    p(\vec{x}|\vec{y})&=\mathcal{N}(\vec{x}|\vec{\mu}_{x|y},\vec{\Sigma}_{x|y}) \\
	\vec{\Sigma}_{x|y}&=\vec{\Sigma}_x^{-1}+\vec{W}^T\vec{\Sigma}_y^{-1}\vec{W} \\
	\vec{\mu}_{x|y}&=\vec{\Sigma}_{x|y}\left[\vec{W}^T\vec{\Sigma}_y^{-1}(\vec{y}-\vec{\mu}_y)+\vec{\Sigma}_x^{-1}\vec{\mu}_x\right]
  \end{split}}
\end{equation}
In addition, the normalization constant $p(\vec{y})$ is given by
\begin{equation}\label{eqn:Linear-Gaussian-system-normalizer}
  \boxed{
    p(\vec{y})=\mathcal{N}(\vec{y}|\vec{W}\vec{\mu}_x+\vec{\mu}_y,\vec{\Sigma}_y+\vec{W}\vec{\Sigma}_x\vec{W}^T)
  }
\end{equation}
\end{theorem}

For the proof, see Section 4.4.3 TODO.

\section{Digression: The Wishart distribution *}

\section{Inferring the parameters of an MVN}

\subsection{Posterior distribution of $\mu$}

\subsection{Posterior distribution of $\Sigma$ *}

\subsection{Posterior distribution of $\mu$ and $\Sigma$ *}
\label{sec:Posterior-distribution-of-mu-and-Sigma}

\subsection{Sensor fusion with unknown precisions *}


\section{Grammar $\rarr$ Automaton}

Given a \textbf{Right-Linear Grammar}, the automaton has as states the \emph{nonterminals}, the initial state is the axiom.
\begin{align*}
    A \rarr aB &\quad\Rightarrow\quad A \xrightarrow{a} B \\
    A \rarr B & \quad\Rightarrow\quad A \xrightarrow{\epsilon}B \\
    A \rarr \epsilon &\quad\Rightarrow\quad A \rarr \text{ (final)}
\end{align*}

Given a \textbf{Left-Linear Grammar}

LL $\xrightarrow[\text{grammar}]{\text{reverse}}$ RL $\xrightarrow[\text{autom}]{\text{make}}$ automaton $\xrightarrow[\text{automaton}]{\text{reverse}}$ result.


\section{Small imprecisions can become big imprecisions}
In this section we explore two ways in which very small starting imprecisions can become very large imprecisions in the final output of a program.

\subsection{When doing numerically unstable computations}\label{ss:numerically-unstable}
There are some types of computations which can transform small imprecisions into catastrophically large ones, and line intersection is one of them. Imagine you have four points $A,B,C,D$ which were obtained through an previous imprecise process (for example, their position can vary by a distance of at most $r = 10^{-6}$), and you have to compute the intersection of lines $AB$ and $CD$.

For illustration, we represent the imprecisions on the points with small disk of radius $r$: the exact position is the black dot, while the small gray disk contains the positions it could take because of imprecisions. The dashed circle gives an idea of where point $I$ might lie.

In the best case, when $A,B$ and $C,D$ aren't too close together, and not too far from the intersection point $I$, then the imprecision on $I$ isn't too big.

\centerFig{small-big0}

But if those conditions are not respected, the intersection $I$ might vary in a very wide range or even fail to exist, if given the imprecision lines $AB$ and $CD$ end up being parallel, or if $A$ and $B$ (or $C$ and $D$) end up coinciding.

\centerFig{small-big1}

This shows that finding the intersection of two lines defined by imprecise points is a task that is inherently problematic for floating-point arithmetic, as it can produce wildly incorrect results even if the starting imprecision is quite small.

\subsection{With large values and accumulation}\label{ss:accumulation}

Another way in which small imprecisions can become big is by accumulation. Problem ``Keeping the Dogs Apart'', which we treat in more detail in a case study in section~\ref{ss:dogs}, is a very good example of this. In this problem, two dogs run along two polylines at equal speed and you have to find out the minimum distance between them at any point in time.

Even though the problem seems quite easy and the computations do not have anything dangerous for precision (mostly just additions, subtractions and distance computations), it turns out to be a huge precision trap, at least in the most direct implementation.

Let's say we maintain the current distance from the start for both dogs. There are $10^5$ polyline segments of length up to $\sqrt{2}\times 10^4$, so this distance can reach $\sqrt{2}\times 10^9$. Besides, to compute the sum, we perform $10^5$ sum operations which can all bring a $2^{-53} \approx 1.11 \times 10^{-16}$ relative error if we're using \lstinline|double|. So in fact the error might reach
\[\left(\sqrt{2}\times 10^9\right) \times 10^5 \times 2^{-53} \approx 0.016\]

Although this is a theoretical computation, the error does actually get quite close to this in practice, and since the tolerance on the answer is $10^{-4}$ this method actually gives a WA verdict.

This shows that even when only very small precision mistakes are made ($\approx 1.11 \times 10^{-16}$), the overal loss of precision can get very big, and carefully checking the maximal imprecision of your program is very important.


\subsubsection{Some sanity checks}

Let's take an example, to see what we get for a typical case.  We can write the beta-model as a radially symmetric pressure model:

\begin{equation}
p(x) = {1\over{(1 + x^2)^{3\beta/2}}}
\end{equation}

In the particular case $\beta = 2/3$, the integrals become analytic.  In particular, we have

\begin{equation}
\integral{}{}{p(x)}{x} = \integral{}{}{{1\over{(1 + x^2)}}}{x} = \arctan{x}
\end{equation}

and

\begin{equation}
\integral{}{}{p(x)x^2}{x} = \integral{}{}{{x^2\over{(1 + x^2)}}}{x} = {x - \arctan{x}}
\end{equation}

whence

\begin{equation}
I(0) = \integral{\infty}{-\infty}{p(x)}{x} = \integral{\infty}{-\infty}{{1\over{(1 + x^2)}}}{x} = \pi
\end{equation}

(which I confirmed numerically)

and

\begin{equation}
\integral{x_R}{0}{p(x)4\pi x^2}{x} = 4\pi\integral{x_R}{0}{{x^2\over{(1 + x^2)}}}{x} = 4\pi ({x_R - \arctan{x_R}})
\end{equation}

(which I also confirm numerically, for $x_R = 1$).

From Equation~\ref{eq:mgas} then, we have:

\begin{eqnarray}
\Mgas(R) &=& m_p\mu_e {{y(0)}\over{I(0)}} \left({{m_e c^2}\over{k_B T_e(0)}}\right)\left({{D^2_A\theta^2_c}\over{\sigma_T}}\right)\integral{x_R}{0}{p(x)4\pi x^2}{x}\\\nonumber
     &=& y(0)\left({{T_e}\over{1~{\rm keV}}}\right)^{-1} \left({{D_A}\over{1~{\rm Gpc}}}\right)^2 \left({{\theta_c}\over{1~{\rm arcsec}}}\right)^2 ({x_R - \arctan{x_R}})\\\nonumber
&& \times 6.8\times 10^{14}\Msolar
\end{eqnarray}

A \betamodel\ fit to A1914 with $\beta = 2/3$ gives $\theta_c = 20$ arcsec and $\Delta T(0) = -2$~mK, or $y(0) \sim 3.7\times 10^{-4}$.

For A1914 we also have $T_e \sim 10~{\rm keV}$, $D_A \sim 0.6~$Gpc,
yielding

\begin{eqnarray}
\Mgas(R) &=& y(0) ({x_R - \arctan{x_R}}) \times 9.8\times 10^{15}\Msolar
\end{eqnarray}

In Tony's paper, he estimates an \mathR500\ of $R \sim 1.3$~Mpc, or $x_R = R/(D_A\theta_c) \sim 22$, yielding

\begin{eqnarray}
\Mgas(R) &=& 7.7\times 10^{13}\Msolar.
\end{eqnarray}

Note that this is not what Tony reports for A1914, but is within 30\% of his value ($11\times 10^{13}\Msolar$).


\chapter{Geometria solida}
\section{Cono}
\begin{tcolorbox}[sidebyside,righthand width=9cm,colback=white,colframe=white,fonttitle=\bfseries	]
	\includestandalone[width=2.2cm]{geometria/cono}
	\tcblower
	\begin{align}
V=&\dfrac{\pi r^2h}{3}&r=&\sqrt{\dfrac{3V}{\pi h}}&h=&\dfrac{3V}{\pi r^2}\\
a=&\sqrt{r^2+h^2}&S_{lat}=&\pi r a&S_{tot}=&S_{lat}+S_{ba}
	\end{align}
\end{tcolorbox}\index{Cono!volume}\index{Cono!raggio}
\section{Cilindro}
\begin{tcolorbox}[sidebyside,righthand width=9cm,colback=white,colframe=white,fonttitle=\bfseries	]
	\includestandalone[width=2.2cm]{geometria/cilindro}
	\tcblower
	\begin{align}
	V=&\pi r^2h&r=&\sqrt{\dfrac{V}{\pi h}}&h=&\dfrac{V}{\pi r^2}\\
	S_{lat}=&\pi r h&S_{tot}=&S_{lat}+S_{ba}
	\end{align}
\end{tcolorbox}
\section{Parallelepipedo rettangolo}
\begin{tcolorbox}[sidebyside,righthand width=9cm,colback=white,colframe=white,fonttitle=\bfseries	]
	\includestandalone[width=4cm]{geometria/parallepipedoret}
	\tcblower
	\begin{align}
	V=&abh&V=&S_{ba}h
	\end{align}
\end{tcolorbox}


% !TEX root = ../main.tex
%---------------------------------------------------------------------------------------------------
%---------------------------------------------------------------------------------------------------
\section{Mathematical formulation}
%---------------------------------------------------------------------------------------------------
%---------------------------------------------------------------------------------------------------
\begin{frame}{Dynamic programming}\vspace{0.25cm}
\heading{Policy evaluation}
\begin{align*}
  v^\pi_t(s_t) = \E_{s_t}^\pi\left[ \sum^{T - t}_{j = 0}  \delta^j\, u_{t + j}(s_{t + j}, a^\pi_{t + j}(s_{t + j})) \right]
\end{align*}
\heading{Optimality equations}
\begin{align*}
  v^{\pi^*}_t(s_t)  & = \max_{a_t \in A} \left\{ u_t(s_t, a_t) + \delta \E^{\pi^*}_{s_t}\! \left[ v^{\pi^*}_{t + 1}(s_{t + 1})\,\right] \right\}
\end{align*}

\end{frame}
%---------------------------------------------------------------------------------------------------
%---------------------------------------------------------------------------------------------------
\begin{frame}{Backward induction algorithm}\vspace{0.25cm}
\begin{algorithmic}
\For{$t = T, \hdots, 1$}
    \If{t = T}
        \State $v^{\pi^*}_T(s_T) =  \underset{a_T\in A}{\max} \bigg\{ u_T(s_T, a_T) \bigg\}\qquad \forall\, s_T\in S$
    \Else
        \State Compute $v^{\pi^*}_t(s_t)$ for each $s_t\in S$ by
        \State $\qquad v^{\pi^*}_t(s_t) = \underset{a_t\in A}{\max} \bigg\{ u_t(s_t, a_t) + \delta \E^\pi_{s_t}\! \left[ v^{\pi^*}_{t + 1}(s_{t + 1}) \,\right] \bigg\}$
        \State and set
        \State $\qquad a^{\pi^*}_t(s_t) = \underset{a_t\in A}{\argmax} \bigg\{ u_t(s_t, a_t) + \delta \E^\pi_{s_t}\! \left[ v^{\pi^*}_{t + 1}(s_{t + 1})\,\right] \bigg\}$
    \EndIf
\EndFor
\vspace{0.3cm}\end{algorithmic}
\end{frame}
%---------------------------------------------------------------------------------------------------
%---------------------------------------------------------------------------------------------------


\subsection{Fuzzy Logic} \label{fuzzy_logic}
Fuzzy Logic (FL) aims at modeling the human reasoning system. It finds its roots in multi-valued logic, which extends the classical two-valued logic by adding new truth values. Differently from other multi-valued logics which have a finite number of truth values, in FL a variable can assume the value of any real number in [0,~1], where 0 means completely false and 1 means completely true.
% This characteristic is also found in Probabilistic Logic, where the truth value of a formula is computed by probabilistic expressions.
If we think about the definition of logic, it \emph{``is the science of the formal principle of reasoning"}. From this perspective, FL is concerned with the formal principles of \textit{approximate reasoning}, since we deal with imprecise premises from which we derive imprecise conclusions~\cite{pal1991fuzzy}. This aspect reflects the fact that in human reasoning we have to deal with vagueness when classifying concepts such as a \textit{tall person} or a \textit{small number}~\cite{hajek1998basic, novak2012mathematical}. In this work, we will focus on the fuzzy extension of the First Order Logic (FOL).

Several FL implementations have been proposed over the years to approximate the logical operators and quantifiers in different ways. We now proceed with the description of some popular variants of the FL by assuming familiarity with the basic notions of the FOL. The definitions we will use in this section are taken from~\cite{analyzingDiffFuzzyLogic}. First of all, the semantics of the main fuzzy operators relies on the following functions:
\begin{itemize}
    \item \textbf{Fuzzy Negation:} function to compute the \textit{negation} of a truth value of a formula. A \textit{fuzzy negation} is a decreasing function $ N:[0,1]\rightarrow [0,1] $ \\
    such that $ N(1)=0 $ and $ \forall (x)\in [0,1], N(N(x)) \geq x $. $N$ is called \textit{strict} if it is strictly decreasing and continuous, and \textit{strong} if \\$\forall x \in [0,1], N(N(x))=x$. %In general, we will consider the classical negation $N(x) = 1 - x$.

    \item \textbf{Triangular Norms (t-norms):} function to compute the \textit{conjunction} of two truth values of a formula. A \textit{t-norm} is a function $ T:[0,1]^{2}\rightarrow [0,1] $ that is \textit{commutative} and \textit{associative}, where $\forall x \in [0,1], T(x,\cdot)$ is increasing (\textit{monotonicity}) and $\forall x \in [0,1], T(1,x)=x$ (\textit{neutrality}).

    \item \textbf{Triangular Conorms (t-conorm):} function to compute the \textit{disjunction} of two truth values of a formula. A \textit{t-conorm} (also known as s-norm) is a function $S:[0,1]^{2} \rightarrow [0,1] $ that is \textit{commutative} and \textit{associative}, where $\forall x \in [0,1], S(x,\cdot)$ is increasing (\textit{monotonicity}) and $\forall x \in [0,1], S(0,x)=x$ (\textit{neutrality}). Since t-conorms are obtained from t-norms by applying De Morgran's law from classical logic, it is possible to write the following equivalence
    \begin{gather*}
        S(a, b) = 1 - T(1 - a, 1 - b)
    \end{gather*}

    \item \textbf{Aggregation Operators:} function to compute $\forall$ and $\exists$ quantifiers.
    \begin{itemize}
        \item the $\forall$ quantifier can be interpreted as a function $A:[0,1]^{n} \rightarrow [0,1] $ computed as a conjunction over all arguments $x$, using a t-norm adapted to $n$-dimensional input:
        \begin{gather*}
            A_{T}()=0 \\
            A_{T}(x_{1},x_{2},...,x_{n}) = T(x_{1},A_{T}(x_{2},...,x_{n}))
        \end{gather*}

        \item the $\exists$ quantifier can be interpreted as a function $E:[0,1]^{n} \rightarrow [0,1] $ computed as a disjunction over all arguments $x$, using a t-conorm adapted to $n$-dimensional input:
        \begin{gather*}
            E_{S}()=0 \\
            E_{S}(x_{1},x_{2},...,x_{n}) = S(x_{1},E_{S}(x_{2},...,x_{n}))
        \end{gather*}
    \end{itemize}
\end{itemize}

Starting from these definitions, we can find a variety of implementations of fuzzy operators. In Table~\ref{tab:fuzzy_norm} are reported the operators of some popular fuzzy logics. The variants proposed by Łukasiewicz and Gödel can be considered among the most popular, but there are many other interpretations (e.g., Weber, Yager, Fodor). Table~\ref{tab:fuzzy_aggregation} reports some aggregation operators based on the presented t-norm and t-conorm operators.

\begin{table}
\centering
\caption{Some common t-norm and t-conorm in fuzzy logic}
\label{tab:fuzzy_norm}
\begin{tabular}{|c|cc|}
\hline
\textbf{Name} & \multicolumn{1}{c|}{\textbf{T-norm}} & \textbf{T-conorm}              \\ \hline
Gödel         & $T_{G}(a,b)=min(a,b)$                & $S_{G}(a,b)=max(a,b)$          \\ \hline
Product       & $T_{P}(a,b)=a \cdot b$               & $S_{P}(a,b)=a + b - a \cdot b$ \\ \hline
Łukasiewicz   & $T_{L}(a,b)=max(a+b-1,0)$            & $S_{L}(a,b)=min(a+b,1)$        \\ \hline
\end{tabular}
\end{table}

\begin{table}
\centering
\caption{Some common aggregation operators in fuzzy logic}
\label{tab:fuzzy_aggregation}
\resizebox{\columnwidth}{!}{\begin{tabular}{|c|cc|}
\hline
\textbf{Name}     & \multicolumn{1}{c|}{\textbf{Generalizes}} & \textbf{Aggregation operator}                                     \\ \hline
Minimum           & $T_{G}$                                   & $A_{T_{G}}(x_{1},...,x_{n}) = min(x_{1},...,x_{n})$               \\
Product           & $T_{P}$                                   & $A_{T_{P}}(x_{1},...,x_{n}) = \prod_{i=1}^{n} x_{i}$              \\
Łukasiewicz       & $T_{L}$                                   & $A_{T_{L}}(x_{1},...,x_{n}) = max(\sum_{i=1}^{n} x_{i}-(n-1), 0)$ \\ \hline
Maximum           & $S_{G}$                                   & $E_{S_{G}}(x_{1},...,x_{n}) = max(x_{1},...,x_{n})$               \\
Probabilistic Sum & $S_{P}$                                   & $E_{S_{P}}(x_{1},...,x_{n}) = 1 - \prod_{i=1}^{n} (1 - x_{i}) $   \\
Bounded Sum       & $S_{L}$                                   & $E_{S_{L}}(x_{1},...,x_{n}) = min(\sum_{i=1}^{n} x_{i}, 1)$       \\ \hline
\end{tabular}}
\end{table}


\section*{Exercises}

\begin{ex}
  For each of the following symmetric matrices, find the eigenvalues,
  an orthonormal basis for each eigenspace, and then orthogonally
  diagonalize the matrix.
  \begin{equation*}
    (a)\quad
    A = \begin{mymatrix}{rr}
      -1 & 2 \\
      2  & 2 \\
    \end{mymatrix},\quad
    (b)\quad
    A = \begin{mymatrix}{rr}
      3 & 1 \\
      1 & 3 \\
    \end{mymatrix},\quad
    (c)\quad
    A = \begin{mymatrix}{rrr}
      1 & 1 & 0 \\
      1 & 2 & 1 \\
      0 & 1 & 1 \\
    \end{mymatrix},\quad
    (d)\quad
    A = \begin{mymatrix}{rrr}
      2 & 1 & 2 \\
      1 & 2 & 2 \\
      2 & 2 & 5 \\
    \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    \begin{enumerate}
    \item $A=PDP^{-1}$ where $D=\begin{mymatrix}{rr}
        -2 & 0 \\
        0  & 3 \\
      \end{mymatrix}$,
      $P=\frac{1}{\sqrt{5}}\begin{mymatrix}{rr}
        2  & 1 \\
        -1 & 2 \\
      \end{mymatrix}$.
    \item $A=PDP^{-1}$ where $D=\begin{mymatrix}{rr}
        2 & 0 \\
        0 & 4 \\
      \end{mymatrix}$,
      $P=\frac{1}{\sqrt{2}}\begin{mymatrix}{rr}
        1  & 1 \\
        -1 & 1 \\
      \end{mymatrix}$.
    \item $A=PDP^{-1}$ where
      $D=\begin{mymatrix}{rrr}
        0 & 0 & 0 \\
        0 & 1 & 0 \\
        0 & 0 & 3 \\
      \end{mymatrix}$,
      $\def\arraystretch{1.3}
      P=\begin{mymatrix}{rrr}
        \frac{1}{\sqrt{3}} & \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \\
        -\frac{1}{\sqrt{3}} & 0 & \frac{2}{\sqrt{6}} \\
        \frac{1}{\sqrt{3}} & -\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \\
      \end{mymatrix}$.
    \item $A=PDP^{-1}$ where
      $D=\begin{mymatrix}{rrr}
        1 & 0 & 0 \\
        0 & 1 & 0 \\
        0 & 0 & 7 \\
      \end{mymatrix}$,
      $\def\arraystretch{1.3}
      P=\begin{mymatrix}{rrr}
        \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{3}} & \frac{1}{\sqrt{6}} \\
        -\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{3}} & \frac{1}{\sqrt{6}} \\
        0 & -\frac{1}{\sqrt{3}} & \frac{2}{\sqrt{6}} \\
      \end{mymatrix}$.
    \end{enumerate}
  \end{sol}
\end{ex}

\begin{ex}
  Prove the converse of Theorem~\ref{thm:diagonalization-symmetric}:
  if a matrix $A$ is orthogonally diagonalizable, then $A$ is
  symmetric.
  \begin{sol}
    Suppose $A$ is orthogonally diagonalizable. Then $A=PDP^{-1}$,
    where $D$ is diagonal and $P$ is orthogonal. Since $P$ is
    orthogonal, we have $P^{-1}=P^T$, and therefore
    $A=PDP^T$. Moreover, since $D$ is diagonal, we have $D=D^T$. It
    follows that $A^T = (PDP^T)^T = (P^T)^T D^T P^T = PDP^T = A$, so
    $A$ is symmetric.
  \end{sol}
\end{ex}


%!TEX root = /home/renaud/Documents/EPL/tfe/latex/tfe.tex
\section{Matrix formulation}
The generic compartment model described in the previous sections can be written in matrix form. For simplicity and because we will restrict ourselves to such systems in the next, we consider the case of an isolated system, i.e. $\phi_{i,e} = 0$. Let
\begin{equation} \label{eq:def_compartment_vars}
	\b c = \begin{pmatrix} C_1 \\ C_2 \\ \vdots \\ C_N \end{pmatrix}, \quad \b q = \begin{pmatrix} q_1 \\ q_2 \\ \vdots \\ q_N \end{pmatrix}, \quad \bs \omega = \begin{pmatrix} |\Omega_1| \\ |\Omega_2| \\ \vdots \\ |\Omega_N| \end{pmatrix}, \quad \mbox{and} \quad \bs \Omega = \diag(\bs \omega).
\end{equation}
The evolution of the concentrations in the compartments is given by
\begin{equation} \label{eq:compartmentmatrix}
	\bs \Omega \b{\dot{c}} = \bs \Omega \b q + \b A \b c,
\end{equation}
where $\b A$ is the \textit{interaction matrix}, which describes the advective and diffusive fluxes between the subdomains. Using the parameters introduced previously, $\b A$ can be expressed as
\begin{equation} \label{eq:generalA}
	\b A = 	\begin{pmatrix}
				-\sum_{j=1}^N \frac{1}{2}U_{1,j}+V_{1,j} & -\frac{1}{2}U_{1,2}+V_{1,2} & \cdots & -\frac{1}{2}U_{1,N}+V_{1,N}\\[.1cm]
				-\frac{1}{2}U_{2,1}+V_{2,1} & -\sum_{j=1}^N \frac{1}{2}U_{2,j}+V_{2,j} & \cdots & -\frac{1}{2}U_{2,N}+V_{2,N}\\
				\vdots & \vdots & \ddots & \vdots\\
				-\frac{1}{2}U_{N,1}+V_{N,1} & -\frac{1}{2}U_{N,2}+V_{N,2} & \cdots & -\sum_{j=1}^N \frac{1}{2}U_{N,j}+V_{N,j}
			\end{pmatrix}.
\end{equation}
In matrix formulation, the mean concentration is expressed as
\begin{equation} \label{eq:Cmeanmatrix}
	\bar C = \frac{1}{|\Omega|} \b 1^\t \bs \Omega \b c,
\end{equation}
where $\b 1$ is the N-dimensional unit column vector. Notice that $|\Omega| = \b 1^\t \bs \Omega \b 1$. In the next of this section, we show how properties~\ref{prop1_comp}, \ref{prop2_comp} and~\ref{prop3_comp} from section~\ref{sec:prop_comp} are expressed in matrix formulation.
\begin{property} \label{prop1bis_comp}
Each column of the interaction matrix sums to zero:
\begin{equation}
	\b 1^\t \b A = 0.
\end{equation}
\end{property}
\begin{proof}
	From equations~\eqref{eq:compartmentmatrix} and~\eqref{eq:Cmeanmatrix},
	\begin{equation}
		|\Omega| \dot{\bar{C}} = \b 1^\t \bs \Omega \dot{\b{c}} = \b 1^\t \bs \Omega \b q + \b 1^\t\b A \b c.
	\end{equation}
	This equation is valid for an isolated system. If furthermore the tracer is passive ($\b q=0$), we know by corollary \ref{cor:Cmeanconstant} that $\bar C$ must be constant. Hence
	\begin{equation} \label{eq:souris}
		|\Omega| \dot{\bar{C}} = \b 1^\t\b A \b c = 0.
	\end{equation}
	Equation~\eqref{eq:souris} must be valid for any concentration vector $\b c$, thus $\b 1^\t \b A = 0$.
\end{proof}

\begin{property} \label{prop2bis_comp}
	Each row of the interaction matrix sums to zero:
	\begin{equation}
		\b A \b 1 = 0.
	\end{equation}
\end{property}
\begin{proof}
	Let $C_0>0$ be a constant and $\b c = C_0 \b 1$. Introducing the latter in~\eqref{eq:compartmentmatrix} with $\b q = \b 0$ yields
	\begin{equation}
		0 = C_0 \b A \b 1.
	\end{equation}
\end{proof}

\begin{property} \label{prop3bis_comp}
	The interaction matrix is negative definite.
\end{property}
\begin{proof}
	Let $\b q = 0$. Since we consider an isolated domain, $\bar C$ is a constant.
	Consider $\hat{\b c} = \b c - \bar C \b 1$, the deviation of the concentration with respect to $\bar C$. Since under those conditions,~\eqref{eq:compartmentmatrix} is linear and homogeneous in $\b c$, it applies to any perturbation of $\b c$ by a constant, and thus to $\hat{\b c}$:
	\begin{equation} \label{eq:prp3matrix}
		\bs \Omega \dot{\hat{\b c}} = \b A \hat{\b c}.
	\end{equation}
	The box variance is expressed as
	\begin{equation}\label{eq:sigmamatrix}
		\sigma^2_{box} = \frac{1}{|\Omega|} \hat{\b c}^\t \bs \Omega \hat{\b c}.
	\end{equation}
	A time-differentiation of~\eqref{eq:sigmamatrix} yields
	\begin{equation}
		\frac{d \sigma^2_{box}}{dt} = \frac{2}{|\Omega|} \hat{\b c}^\t \bs \Omega \dot{\hat{\b c}},
	\end{equation}
	and finally by~\eqref{eq:prp3matrix}
	\begin{equation}
		\frac{|\Omega|}{2} \frac{d \sigma^2_{box}}{dt} = \hat{\b c}^\t \b A \hat{\b c}.
	\end{equation}
	By property~\ref{prop3_comp}, the latter must be negative (or zero if $\hat{\b c} = \b 0$), which shows that $\b A$ is negative definite.
\end{proof}


\section{Axioms for Quantum Mechanics}
Now we start to try and make the theory more mathematical.
\subsection{Vector Space and Inner Product}
As we have seen, we can collect $0$ and the set of possible wavefunctions (for fixed time $t$) as a vector space $\mathcal H$ which is a subspace of $L^2(\mathbb R^3,\mathbb C)$.
Linear maps involved are operators and the inner product is what we have seen earlier:
\begin{definition}
    The inner product of two wavefunctions $\psi,\phi$ is defined by
    $$(\psi,\phi)=\int_{\mathbb R^3}\psi^*\phi\,\mathrm dV$$
\end{definition}
\begin{lemma}
    If $\psi,\phi\in L^2(\mathbb R^3,\mathbb C)$, then $(\psi,\phi)$ exists.
\end{lemma}
\begin{proof}
    Cauchy-Schwartz.
\end{proof}
\begin{proposition}
    1. $(\psi,\phi)=(\phi,\psi)^*$\\
    2. $(\psi,\lambda_1\phi_1+\lambda_2\phi_2)=\lambda_1(\psi,\phi_1)+\lambda_2(\psi,\phi_2)$ and $(\mu_1\psi_1+\mu_2\psi_2,\phi)=\mu_1^*(\psi_1,\phi)+\mu_2^*(\psi_2,\phi)$.\\
    3. $( \psi,\psi)\ge 0$ and equals $0$ iff $\psi=0$.
\end{proposition}
\begin{proof}
    Obvious.
\end{proof}
\begin{definition}
    A wavefunction $\psi$ is normalised if $(\psi,\psi)=1$.
    If $(\psi,\phi)=0$ then we say $\psi,\phi$ are orthogonal.
    A set of wavefunctions $\{\psi_n\}$ is orthogonal if $(\psi_m,\psi_n)=\delta_{mn}$.
    It is complete if an other wavefunction $\phi$ can be written as
    $$\phi=\sum_{n=0}^\infty c_n\psi_n$$
    for some $c_n\in\mathbb C$.
\end{definition}
It then follows that
\begin{proposition}
    If we have a complete and orthogonal set of wavefunctions $\{\psi_n\}$ and $\phi=\sum_n c_n\psi_n$, then $c_n=(\psi_n,\phi)$.
\end{proposition}
\begin{proof}
    Straight from definition.
\end{proof}
The probabilistic interpretation of inner product is that if we take $\phi$ to be the desired outcome of a measurement and $\psi$ is the actual state of the particle at the time of measurement, then $|( \psi,\phi)|^2$ can be taken as the probability of measuring $\phi$ as an outcome, that is the overlap between the two states at time $t$.
\subsection{Hermitian Operators}
An operator on $\mathcal H$, the vector space of possible wavefunctions and $0$, is a linear map from $\mathcal H$ to itself (or a vector space containing it).
\begin{example}
    Linear differential and translation are operators.
    We can also have the parity operator $\phi(x)\mapsto\psi(-x)$.
\end{example}
\begin{definition}
    The Hemitian conjugate $\hat{A}^\dagger$ of an operator $\hat{A}$ is an operator such that $(\hat{A}^\dagger\psi_1,\psi_2)=(\psi_1,\hat{A}\psi_2)$ for any $\psi_1,\psi_2\in\mathcal H$.
\end{definition}
Consequently, $(a_1\hat{A}_1+a_2\hat{A}_2)^\dagger=a_1^*\hat{A}^\dagger+a_2^*\hat{A}_2^\dagger$ and $(\hat{A}\hat{B})^\dagger=\hat{B}^\dagger\hat{A}^\dagger$.
\footnote{Yes, we haven't proved the existence and uniqueness of a conjugate. And yes, the lecturer still hasn't proved it.}
\begin{definition}
    A Hermitian operator is an operator $\hat{A}$ such that $(\hat{A}\psi_1,\psi_2)=(\psi_1,\hat{A}\psi_2)$.
\end{definition}
Equivalently $\hat{A}=\hat{A}^\dagger$.
\begin{example}
    Physical quantities (observables) like $\hat{x},\hat{p},\hat{H}$ are Hermitian operators.
\end{example}
\begin{theorem}
    The eigenvalues of Hermitian operators are real.
\end{theorem}
\begin{proof}
    Let $\hat{A}$ be Hermitian and $\psi$ a normalised eigenfunction of it with eigenvalue $a$, that is $\hat{A}\psi=a\psi$.
    Now
    \begin{align*}
        a&=a(\psi,\psi)=( \psi,a\psi)=(\psi,\hat{A}\psi)=( \hat{A}^\dagger\psi,\psi)=(\hat{A}\psi,\psi)\\
        &=( a\psi,\psi)=a^*(\psi,\psi)\\
        &=a^*
    \end{align*}
    So $a$ has to be real.
\end{proof}
\begin{theorem}
    Let $\hat{A}$ be a Hermitian operator and $\psi_1,\psi_2$ normalised eigenfunctions with different eigenvalues $a_1\neq a_2$, then $\psi_1,\psi_2$ are orthogonal.
\end{theorem}
\begin{proof}
    We have
    \begin{align*}
        a_1(\psi_1,\psi_2 )&=a_1^*(\psi_1,\psi_2)=( a_1\psi_1,\psi_2)=( \hat{A}\psi_1,\psi_2)\\
        &=( \hat{A}^\dagger\psi_1,\psi_2)=(\psi_1,\hat{A}\psi_2)=(\psi_1,a_2\psi_2)\\
        &=a_2(\psi_1,\psi_2)
    \end{align*}
    But $a_1\neq a_2$, so it has to be the case that $(\psi_1,\psi_2)=0$.
\end{proof}
\begin{theorem}
    The set of eigenfunctions, discrete or continous, of any Hermitian operator forms a complete orthogonal set.
\end{theorem}
\begin{proof}
    Haha.
\end{proof}
\begin{corollary}
    Every solution of TDSE can be written as a superposition of the stationary states.
\end{corollary}
\begin{proof}
    The stationary states are eigenfunctions of the Hamiltonian.
\end{proof}
\subsection{Quantum Measurements}
We have a few postulates for quantum mechanics.
\begin{postulate}
    1. Any quantum observable $O$ is represented by a Hermitian operator $\hat{O}$.\\
    2. The possible outcomes of measurement of $O$ are eigenvalues of $\hat{O}$.\\
    3. if $\hat{O}$ has a discrete set $\{\psi_i\}$ of normalised eigenfunctions with distinct eigenvalues $\{\lambda_i\}$, then the measurement of $O$ on a particle in state $\psi=\sum_ia_i\phi_i$, then the probability of outcome $\lambda$ is $p(\lambda_i)=|a_i|^2$.
    In particular, if $\psi=\psi_i$, then the measurement is $\lambda_i$ with probability $1$.\\
    4. If $\{\psi_i\}_{i\in I}$ is a subset of the eigenfunctions with common eigenvalue $\lambda$, then $p(\lambda)=\sum_{i\in I}|a_i|^2$.\\
    5. (consequence of 3 and 4) $\sum_i|a_i|^2=1$.\\
    6. (projection postulate, collapse of wavefunction) If $O$ is measured on $\psi$ at time $t$ and outcome of the measure is $\lambda_i$, then the wavefunction instaneously becomes $\psi_1$ at time $t$.\\
    7. If $\hat{O}$ has an eigenvalue $\lambda$ with eigenfunctions $\{\psi_i\}_{i\in I}$, then if $O$ is measured with outcome $\lambda$ the wavefunction instaneously becomes a superposition of $\{\psi_i\}_{i\in I}$.
\end{postulate}
\begin{definition}
    The projection operator $\hat{p}_i$ sends $\psi=\sum_ja_j\psi_j$ to $a_i\psi_i$.
\end{definition}
\subsection{Expected Values}
Consider the measurement of observable $O$ on state $\psi$ and the corresponding Hermitian operator $\hat{O}$ has a discrete set of eigenfunctions $\{\psi_i\}$ with eigenvalues $\lambda_i$, then $\{\lambda_i\}$ gives us all possible outcomes of measures of $O$ with $p(\lambda_i)=|\langle\psi,\psi_i\rangle|^2$.
So we can defined\begin{definition}
    The expected value of $O$ is
    $$\langle\hat{O}\rangle_\psi=\sum_ip_i\lambda_i=\sum_i|(\psi,\psi_i)|^2\lambda_i=(\psi,\hat{O}\psi)=\int_{\mathbb R^3}\psi^*\hat{O}\psi\,\mathrm dV$$
\end{definition}
Then easily $\langle\cdot\rangle_\psi$ is a linear form on the (real) vector space of Hermitian operators on $\mathcal H$.
\subsection{Commutators}
\begin{definition}
    The commutator of two operators $\hat{A},\hat{B}$ is the operator $[\hat{A},\hat{B}]=\hat{A}\hat{B}-\hat{B}\hat{A}$.
    Correspondingly, the anti-commutator is $\{\hat{A},\hat{B}\}=\hat{A}\hat{B}+\hat{B}\hat{A}$
\end{definition}
Easly $[\cdot,\cdot]$ is a antisymmetric and
$$[\hat{A},\hat{B}\hat{C}]=[\hat{A},\hat{B}]\hat{C}+\hat{B}[\hat{A},\hat{C}],[\hat{A}\hat{B},\hat{C}]=\hat{A}[\hat{B},\hat{C}]+[\hat{A},\hat{C}]\hat{B}$$
\begin{example}
    We have $[\hat{x},\hat{p}]=i\hbar\hat{I}$ where $\hat{I}$ is the identity operator.
\end{example}
\begin{definition}
    Two Hemitian operators $\hat{A},\hat{B}$ are simultaneously diagonalisable in $\mathcal H$ if there is a complete basis of joint eigenfunctions $\{\psi_i\}$ with $\hat{A}\psi_i=a_i\psi_i$ and $\hat{B}\psi_i=b_i\psi_i$ for all $i$ where $a_i,b_i$ are the respective eigenvalues.
\end{definition}
\begin{theorem}
    Two Hemitian operators $\hat{A}$ and $\hat{B}$ are simultaneously diagonalisable iff $[\hat{A},\hat{B}]=0$
\end{theorem}
\begin{proof}
    The ``only if'' direction is trivial.
    Conversely, if $[\hat{A},\hat{B}]=0$, then $\hat{A}\hat{B}=\hat{B}\hat{A}$, so $A(B\psi_i)=a_i(\hat{B}\psi_i)$ for all $i$.
    This means that $\hat{B}$ maps any eigenspace $E$ of $\hat{A}$ to itself.
    But $\hat{B}$ is also Hermitian on $E$, so we can have an eigenspace $E$ in which $\hat{B}$ acts diagonally.
    Collect them together gives the desired complete set of basis.
\end{proof}
\subsection{Heisenberg's Uncertainty Principle}
\begin{definition}
    The uncertainty in mass of $A$ on a state $\psi$ is defined as
    $$(\Delta_\psi A)^2=\langle (\hat{A}-\langle\hat{A}\rangle_\psi\hat{I})^2\rangle_\psi=\langle\hat{A}^2\rangle_\psi-(\langle\hat{A}\rangle_\psi)^2$$
    where $\hat{I}$ is the identity operator.
\end{definition}
\begin{lemma}
    $(\Delta_\psi A)^2\ge 0$ and $\Delta_\psi A=0$ iff $\psi$ is an eigenfunction of $\hat{A}$.
\end{lemma}
\begin{proof}
    Write $\phi=(\hat{A}-\langle\hat{A}\rangle_\psi\hat{I})\psi$, then
    \begin{align*}
        (\Delta_\psi A)^2&=\langle (\hat{A}-\langle\hat{A}\rangle_\psi\hat{I})^2\rangle_\psi\\
        &=((\hat{A}-\langle\hat{A}\rangle_\psi\hat{I})\psi,(\hat{A}-\langle\hat{A}\rangle_\psi\hat{I})\psi)\\
        &=(\phi,\phi)\ge 0
    \end{align*}
    and equality holds iff $\phi=0$ which happens iff $\phi=0$, but this is just another way of saying $\phi$ is an eigenfunction of $\hat{A}$.
\end{proof}
\begin{theorem}[Schwartz Inequality]
    If $\phi,\psi$ are any two normalisable wavefunctions, then $|(\phi,\psi)|^2\le(\phi,\phi)(\psi,\psi)$ with equality iff $\phi,\psi$ are linearly dependent.
\end{theorem}
\begin{proof}
    Just copy the proof of the usual Cauchy-Schwartz Inequality.
\end{proof}
So we can safely write $\Delta_\psi A=\sqrt{(\Delta_\psi A)^2}$.
\begin{theorem}[Generalised Uncertainty Theorem]
    If $A,B$ are observables and $\phi\in\mathcal H$, then
    $$(\Delta_\psi A)(\Delta_\psi B)\ge\frac{1}{2}|(\psi,[\hat{A},\hat{B}]\psi)|$$
\end{theorem}
\begin{proof}
    Write $\hat{A}'=\hat{A}-\langle\hat{A}\rangle_\psi\hat{I}$ which is also Hermitian, then we have $(\Delta_\psi A)^2=(\hat{A}'\psi,\hat{A}'\psi)$.
    Similarly $\hat{B}'=\hat{B}-\langle\hat{B}\rangle_\psi\hat{I}$ gives $(\Delta_\psi B)^2=(\hat{B}'\psi,\hat{B}'\psi)$.
    Note that we have $[\hat{A}',\hat{B}']=[\hat{A},\hat{B}]$.
    So Schwartz Inequality gives
    $$(\Delta_\psi A)^2(\Delta_\psi B)^2=(\hat{A}'\psi,\hat{A}'\psi)(\hat{B}'\psi,\hat{B}'\psi)\ge|(\hat{A}'\psi,\hat{B}'\psi)|^2=|(\psi,\hat{A}'\hat{B}'\psi)|^2$$
    But we can write $\hat{A}'\hat{B}'=([\hat{A}',\hat{B}']+\{\hat{A}',\hat{B}'\})/2$.
    Consequently,
    $$(\Delta_\psi A)^2(\Delta_\psi B)^2\ge \frac{1}{4}|(\psi,[\hat{A}',\hat{B}']\psi)+(\psi,\{\hat{A}',\hat{B}'\}\psi) |^2$$
    But
    $$(\psi,\{\hat{A}',\hat{B}'\}\psi)=(\{\hat{A}',\hat{B}'\}^\dagger\psi,\psi)=(\{\hat{A}',\hat{B}'\}\psi,\psi)=(\psi,\{\hat{A}',\hat{B}'\}\psi)^*$$
    so $(\psi,\{\hat{A}',\hat{B}'\}\psi)$ has to be real.
    Similarly $(\psi,[\hat{A}',\hat{B}']\psi)$ has to be purely imaginary.
    Therefore
    \begin{align*}
        (\Delta_\psi A)^2(\Delta_\psi B)^2&\ge \frac{1}{4}|(\psi,[\hat{A}',\hat{B}']\psi)+(\psi,\{\hat{A}',\hat{B}'\}\psi) |^2\\
        &=\frac{1}{4}|(\psi,[\hat{A}',\hat{B}']\psi)|^2+|(\psi,\{\hat{A}',\hat{B}'\}\psi) |^2\\
        &\ge\frac{1}{4}|(\psi,[\hat{A}',\hat{B}']\psi)|^2\\
        &=\frac{1}{4}|(\psi,[\hat{A},\hat{B}]\psi)|^2
    \end{align*}
    Taking square root on both sides shows the theorem.
\end{proof}
If $[\hat{A},\hat{B}]=0$, then the bound is just zero.
This is interpreted as we can measure $A,B$ simultaneously.
\begin{corollary}[Heisenberg's Uncertainty Principle]
    $$(\Delta_\psi x)(\Delta_\psi p)\ge\frac{\hbar}{2}$$
\end{corollary}
\begin{proof}
    Just take $\hat{A}=\hat{x},\hat{B}=\hat{p}$ and be reminded that $[\hat{x},\hat{p}]=i\hbar\hat{I}$.
\end{proof}
\begin{example}
    1. We can see a particle with light of wavelength $\lambda\sim\Delta x$ around its de Broglie wavelength $h/p$, so $\Delta p\sim p\sim\hbar/(\Delta x)$ which means $\Delta x\Delta p\sim h=2\pi\hbar>\hbar/2$.\\
    2. Consider the unnormalisable plane wave solution $\psi_p=e^{ipx/\hbar}$ for a free particle, then $\Delta_{\psi_p}p=0,\Delta_{\psi_p}x=\infty$.
    Recall that we need to make a certain superposition of it normalisable by introducing the Gaussian wavepackage
    $$\psi_{\rm GP}(x,t)=\sqrt[4]{\frac{\sigma}{\pi(\sigma^2+\hbar^2t^2/m^2)}}\exp\left( -\frac{\sigma(x-\hbar k_0t/m)^2}{2(\sigma^2+\hbar^2t^2/m^2)} \right)$$
    where we actually have $(\Delta_{\psi_{\rm GP}}x)(\Delta_{\psi_{\rm GP}}p)=\hbar/2$.
\end{example}
We can obtain the equality in the second example by calculation of course, but the gist is actually the following:
\begin{lemma}
    If $\hat{x}\psi=ia\hat{p}\psi$ for some $a\in\mathbb R$, then $(\Delta_\psi x)(\Delta_\psi p)=\hbar/2$.
\end{lemma}
This condition is easily seen to be necessary.
\begin{proof}
    The condition shows that we have the equality case in Schwartz inequality in the proof of the preceding theorem.
    Furthermore,
    \begin{align*}
        (\psi,\{\hat{x},\hat{p}\}\psi)&=(\psi,\hat{x}\hat{p}\psi)+(\psi,\hat{p}\hat{x}\psi)\\
        &=(\hat{x}\psi,\hat{p}\psi)+(\hat{p}\psi,\hat{x}\psi)\\
        &=(ia\hat{p}\psi,\hat{p}\psi)+(\hat{p}\psi,ia\hat{p}\psi)\\
        &=(-ia+ia)(\hat{p}\psi,\hat{p}\psi)\\
        &=0
    \end{align*}
    which implies the lemma.
\end{proof}
\begin{lemma}
    $\hat{x}\psi=ia\hat{p}\psi$ iff $\psi(x)\propto e^{-bx^2}$ for some $b>0$.
\end{lemma}
\begin{proof}
    Obvious.
\end{proof}
\subsection{Ehrenfest's Theorem}
\begin{theorem}[Ehrenfest's Theorem]
    Let $\hat{A}$ be an operator, then
    $$\frac{\mathrm d}{\mathrm dt}\langle\hat{A}\rangle_\psi=\frac{i}{\hbar}\langle [\hat{H},\hat{A}]\rangle_\psi+\langle\partial\hat{A}/\partial t\rangle_\psi$$
\end{theorem}
\begin{proof}
    Just expand.
    \begin{align*}
        \frac{\mathrm d}{\mathrm dt}\langle\hat{A}\rangle_\psi&=\frac{\mathrm d}{\mathrm dt}\int_{-\infty}^\infty\psi^*\hat{A}\psi\,\mathrm dx\\
        &=\int_{-\infty}^\infty\frac{\mathrm d}{\mathrm dt}(\psi^*\hat{A}\psi)\,\mathrm dx\\
        &=\int_{-\infty}^\infty \left( \frac{\partial\psi^*}{\partial t}\hat{A}\psi+\psi^*\hat{A}\frac{\partial\psi}{\partial t} \right)\,\mathrm dx+\langle\partial\hat{A}/\partial t\rangle_\psi\\
        &=\frac{i}{\hbar}\int_{-\infty}^\infty (\psi^*\hat{H}\hat{A}\psi-\psi^*\hat{A}\hat{H}\psi)\,\mathrm dx+\langle\partial\hat{A}/\partial t\rangle_\psi\\
        &=\frac{i}{\hbar}\langle[\hat{H},\hat{A}]\rangle_\psi+\langle\partial\hat{A}/\partial t\rangle_\psi
    \end{align*}
    which is what we wanted.
\end{proof}
\begin{example}
    Take $\hat{A}=\hat{H}$, then $[\hat{H},\hat{H}]=0$, so $\mathrm d\langle\hat{H}\rangle_\psi/\mathrm dt=0$ which is equivalent to the conservation of total energy in quantum mechanics.\\
    Take $\hat{A}=\hat{p}$, then $[\hat{H},\hat{p}]=i\hbar\partial U/\partial x$, therefore,
    $$\frac{\mathrm d\langle\hat{p}\rangle_\psi}{\mathrm dt}=-\left\langle\frac{\mathrm dU}{\mathrm dx}\right\rangle_\psi$$
    which is analogous to the classical case.\\
    Take $\hat{A}=\hat{x}$, ten we obtain $[\hat{H},\hat{x}]=-i\hbar\hat{p}/m$,
    $$\frac{\mathrm d\langle\hat{x}\rangle_\psi}{\mathrm dt}=\frac{\langle\hat{p}\rangle_\psi}{m}$$
    which is again analogous to classical mechanics.
\end{example}
\subsection{The Harmonic Oscillator Revisited}
For a harmonic oscillator, the Hamiltonian is
\begin{align*}
    \hat{H}&=\frac{\hat{p}^2}{2m}+\frac{1}{2}m\omega^2\hat{x}^2\\
    &=\frac{1}{2m}(\hat{p}+im\omega\hat{x})(\hat{p}-im\omega\hat{x})+\frac{i\omega}{2}[\hat{p},\hat{x}]\\
    &=\frac{1}{2m}(\hat{p}+im\omega\hat{x})(\hat{p}-im\omega\hat{x})+\frac{\hbar\omega}{2}
\end{align*}
Write $\hat{a}=(\hat{p}-im\omega\hat{x})/\sqrt{2m}$ (called the ladder operator), then $\hat{a}^\dagger=(\hat{p}+im\omega\hat{x})/\sqrt{2m}$ and $\hat{H}=\hat{a}^\top\hat{a}+\hbar\omega/2$.
Also $[\hat{a},\hat{a}^\dagger]=\hbar\omega\hat{I}$, therefore $\hat{a}$ is Hermitian.
In addition $[\hat{H},\hat{a}]=-\hbar\omega\hat{a}$ and $[\hat{H},\hat{a}^\dagger]=\hbar\omega\hat{a}^\dagger$.
Suppose $X$ is an eigenfunction of $\hat{H}$ with eigenvalue $E$, that is $\hat{H}X=EX$, then $\hat{H}\hat{a}X=[\hat{H},\hat{a}]X+\hat{a}\hat{H}X=(-\hbar\omega+E)\hat{a}X$.
Similarly $\hat{a}^\dagger =(E+\hbar\omega)\hat{a}^\dagger X$.
Then by induction $\hat{a}^nX$ is an eigenfunction of $\hat{H}$ with eigenvalue $E-n\hbar\omega$ and $\hat{a}^\dagger X$ an eigenfunction with eigenvalue $E+n\hbar\omega$.
For $U\ge 0$, we have $\langle H\rangle_\psi\ge 0$, so we can choose the lowest positive eigenfunction $X_0$ and we must then have $\hat{a}X_0=0$.
This is just a first order differential equation, which can be easily solved to give $X_0(x)\propto\exp(-m\omega x^2/(2\hbar))$.
So $X_0$ has eigenvalue $\hbar\omega/2$.
To find the excited states, we simply need to compute
$$X_n=(\hat{a}^\dagger)^nX_0\propto\frac{1}{\sqrt{2m}}(\hat{p}+im\omega\hat{x})^n\exp(-m\omega x^2/(2\hbar))$$
which involves Hermite polynomials.
They then have eigenvalue $E_n=(n+1/2)\hbar\omega$.


\lab{Trust-Region Methods}{Trust-Region Methods}
\label{lab:trust_region}

\objective{Explore Trust-Region methods for optimization.}

When it comes to optimizing high-dimensional functions, a common strategy is to break
the problem up into a series of smaller, easier tasks, leading to a sequence of
successive approximations to the optimizer. This is the approach taken by Line-Search
algorithms such as Newton's method, as discussed in Lab \ref{lab:line_search}.
The class of algorithms known as Trust-Region methods are also based on this
strategy, although they differ from Line-Search methods in some important ways.

\section*{Overview of the Trust-Region Approach}
Suppose we have a function $f$ that we wish to minimize over its entire domain.
Give that we have some particular point $x_k$ in the domain of $f$, how do we
select a new point $x_{k+1}$ that better minimizes the function? A Line-Search
algorithm solves this sub-problem by first choosing a search direction $d_k$
(often related to the gradient of $f$), and then a step length $\alpha_k$ so
as to minimize $f$ along the direction $d_k$. The next point, then, is simply
\[
x_{k+1} := x_k + \alpha_k d_k.
\]

A Trust-Region algorithm, on the other hand, does away with a search direction and
step length, and instead approximates the function $f$ with some simpler function
$m_k$ (called the \emph{model function}). The model $m_k$ will likely not be close to $f$ over the entire
domain, and so we must restrict our attention to a ball of radius $r_k$ centered at
the point $x_k$, inside of which $m_k$ is reasonably close to $f$. We then minimize
$m_k$ over this ball, and set $x_{k+1}$ equal to this minimizer. That is, solve the sub-problem
\[
x_{k+1} := \underset{x \in B(x_k, r_k)}{\text{argmin}} m_k(x).
\]
The ball $B(x_k, r_k)$ is called the \emph{trust region} because we trust that the
model function $m_k$ gives a reasonably accurate approximation of $f$ on this region.
Note that it is also possible to use other types of trust regions, such as
ellipsoidal or box-like regions.

\subsection*{The Model Function}
The model function $m_k$ is commonly taken to be a linear or quadratic approximation of
$f$ based on its Taylor Series expansion about the point $x_k$. In the linear case,
our model function has the form
\[
m_k(y) = f(x_k) + (y-x_k)^T \nabla f(x_k).
\]
In the quadratic case, we simply add on a quadratic term to obtain
\[
m_k(y) = f(x_k) + (y-x_k)^T \nabla f(x_k) + \frac{1}{2}(y - x_k)^T B_k (y-x_k),
\]
where $B_k$ is the Hessian matrix of $f$ at $x_k$, or some approximation thereof.
Given a trust region with radius $r_k$, note that our sub-problem can be
written in the following way:
\begin{align*}
x_{k+1} &= \underset{x \in B(x_k, r_k)}{\text{argmin}} m_k(x)\\
&= x_k + \underset{\|p\| < r_k}{\text{argmin}}\, \{f(x_k) + p^T \nabla f(x_k) + \frac{1}{2}p^T B_k p\}.
\end{align*}

\subsection*{The Trust-Region}
Discuss how choice of the radius $r_k$ can have large impact on performance of the algorithm. Balance between
too small and too large of a trust region. How to determine whether the radius should change in the next iteration.

\section*{The Dogleg Method}
This is a method to approximately solve the sub-problem, appropriate when the model Hessian is positive definite.
SciPy has an implementation of this algorithm in the optimization package.

\section*{Newton-CG Trust-Region Method}
This is a trust-region approach to the Newton-CG algorithm for large unconstrained optimization.
SciPy has an implementation of this algorithm in the optimization package.

\section*{Levenberg-Marquardt Algorithm}
This is a trust-region approach to the Gauss-Newton method for nonlinear least squares.


\section{Features}

Features are sources of information that hopefully allow conclusions towards some kind of quantity of interest. Some people like to call them independent variables.

\subsection{Dense Features, Sparse Features}
Sparsity and density refer to the fraction of a matrix that is zeros. In the context of features, sparse features typically refer to feature vectors that have many zeros in them. I.e., $[1,2,5,2,6,3,0]$ would be a dense feature vector and $[2,0,0,0,0,3,0,0,1]$ would be a sparse feature vector. Typically, sparsity is touted as an advantage because it allows for lossless compressed representations of high-dimensional data, which has computational advantages. Another advantage, though, is that sparse representations can be more interpretable by containing information in an inherently more condensed fashion. They are also thought to prevent overfitting: many regularization techniques aim to minimize the number of parameters or features used in a predictive model, which amounts to biasing an algorithm towards learning sparse coefficients. An example is ridge regression.

\section{Multicollinearity}
So far this section is based on \citeasnoun{ncssridgeregression}.

Multicollinearity, or collinearity, occurs when features are linearly correlated with each other. The effects are horrible. They include inaccurate estimates of the regression coefficients, higher standard errors of the regression coefficients, lower partial t-tests for the regression coefficients, falsely insignificant p-values and decreased predictive power of the model. And other things.

\subsubsection{Detection}

\subparagraph{Scatter Plots}
Scatterplots provide a visual test for collinearity by hopefully exposing relationships between independent variables. This is subjective and unreliable, but people love plots.

\subparagraph{Variance Inflation Factors (VIF)}
A VIF over 10 it said to indicate collinear variables.

\subparagraph{Eigenvalues of the Correlation Matrix}
Linear relationships between two or more variables cause the corresponding rows of the correlation matrix to be identical or very similar. Correspondingly, the matrix will be singular or near-singular, which will manifest itself through zero or near-zero eigenvalues. The conditioning number, given by the largest eigenvalue divided by the smallest eigenvalue, are a quick way to test for this. A large conditioning number indicates collinearity.

\subparagraph{Regression Coefficients}
Collinearity increases the standard error of the regression coefficients because it allows for the variation of the dependent variable to be explained in terms of a greater variety of different weights assigned to the collinear variables. Counterintuitive results for the regression coefficients may be the result of collinearity.

\subsection{Sources}

\subparagraph{Data Collection}
\subparagraph{Physical Constraints}
\subparagraph{Over-defined Model}
\subparagraph{Model Choice or Specification}
\subparagraph{Outliers}

\subsubsection{Remedies}

\subparagraph{Dimensionality Reduction}
SVD, PCA, NNMF and other dimensionality reduction techniques allow for the feature space to be shrunk in a way that aims to optimally preserve information. Any technique worth its salt will either collapse or filter collinear variables into a reduced-rank representation of the information in the dataset.

\subparagraph{Regularization}
Certain forms of regularization are similar in spirit to dimensionality reduction, except that, rather than addressing the dataset, they reduce the parameters used by a model to fit to the data. The canonical example is ridge regression. Ridge regression penalizes the use of a larger number of parameters and, if two variables are collinear, will tend to push the weight of one of them towards zero. It is important to standardize the variables before fitting the model, so that the regression weights for different variables are on the same scale. Ridge regression remains controversial \cite{ncssridgeregression}.


\problemname{Study Pair}
%\illustration{.5}{filename}{Image by \href{url}{Author}}

There are two classes, class $A$ and class $B$.
Each class has $N$ students.
The teacher wants to make study pairs between the two classes.
Each study pair consists of one student from class $A$ and the other from class $B$.
Every student's knowledge level on the course topic can be described as an integer.
Namely, the $i$-th student from class $A$ has knowledge level $A_i$, and the $i$-th student from class $B$ has knowledge level $B_i$.
To achieve best study results, the knowledge levels of the two students in a study pair shall not differ too much.
Besides, the sum of the knowledge levels of the two students in a pair shall not be too large (Let strong students help the others!).
More formally, given two constants $X, Y$, for every study pair $(A_i, B_j)$, it must satisfy $|A_i - B_j| \leq X$ and $A_i + B_j \leq Y$.

The teacher wonders what is the maximum number of study pairs she can make?

\section*{Input}

The first line of the input has an integer $T$, the number of test cases.\\
Each case has three lines.
The first line has three integers $N, X, Y$.
The second line has $N$ integers, $A_1, A_2, ... A_N$.
The third line has $N$ integers, $B_1, B_2, ... B_N$.

\section*{Output}

For each case, output the maximum number of study pairs that can be made.

\section*{Constraints}
\begin{itemize}
\item $1 \leq T\leq 15$
\item $1 \leq N \leq 10^5$
\item $0 \leq X, Y \leq 10^9$
\item $0 \leq A_i, B_i \leq 10^9$ for $1\leq i \leq N$
\end{itemize}

\section*{Subtasks}
\begin{itemize}
\item $N \leq 50$
\item Original constraints
\end{itemize}


\section{The \maxelement algorithm}
\Label{sec:maxelement}

The \maxelement algorithm in the \cxx Standard Library \cite[\S 28.7.8]{cxx-17-draft}
searches the maximum of a general sequence.
The signature of our version of \maxelement reads:

\begin{lstlisting}[style = acsl-block]

  size_type max_element(const value_type* a, size_type n);
\end{lstlisting}

The function finds the largest element in the range
\inl{a[0..n-1]}.
More precisely, it returns the unique valid index \inl{i} such that:
\begin{enumerate}
\item for each index \inl{k} with \inl{0 <= k < n} the condition
\inl{a[k] <= a[i]} holds and
\item for each index \inl{k} with \inl{0 <= k < i} the condition
\inl{a[k] < a[i]} holds.
\end{enumerate}
The return value of \maxelement is \inl{n} if and only if there is no
maximum, which can only occur if \inl{n == 0}.

\subsection{Formal specification of \maxelement}

The following listings shows the formal specification of \specref{maxelement}.
Note that we have subdivided the specification of \maxelement into the two
behaviors~\inl{empty} and \inl{not_empty}.
The behavior \inl{empty} contains the specification for the case that the
range contains no elements.
The behavior \inl{not_empty} applies if the range has a positive length.

The ensures clause \inl{max} of behavior \inl{not_empty} indicates that
the returned valid index \inl{k} refers to a maximum value of the array.
The postcondition \inl{first} expresses that \inl{k} is indeed the
\emph{first} occurrence of a maximum value in the array.

\input{Listings/max_element.h.tex}

\subsection{Implementation of \maxelement}

In our description, we concentrate on the \emph{loop annotations}
of the implementation of \implref{maxelement}.

\input{Listings/max_element.c.tex}

The loop invariant \inl{max} is needed to prove the  postcondition
\inl{result} of the behavior \inl{not_empty} of \specref{maxelement}.
Using loop invariant \inl{upper} we prove the postcondition \inl{upper}
of the behavior \inl{not_empty} of \specref{maxelement}.
Finally, the postcondition \inl{first} of this behavior can be
verified with the loop invariant \inl{first}.

\clearpage


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS622: Theory of Formal Languages
% Copyright 2014 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 2}

Let $A$ be an alphabet and let $b$ be a symbol such that $b \notin A$.

\begin{enumerate}[label=(\alph*)]
	\item construct a transition system that accepts the language $ L = b A^* b $.
	\item construct an equivalent deterministic finite automaton that accepts $L$.
\end{enumerate}

\subsection*{Solution}

\begin{enumerate}[label=(\alph*)]

	\item
	A transition system that accepts the language $b A^* b$ is shown in Figure \ref{fig:DR2}.
	Starting from initial state $q_0$, a word is accepted by this transition system only if it starts with $b$, follows a possible number of symbols in alphabet $A$ and end with $b$.

	\begin{figure}[H]\centering
		\begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,node distance=3cm,semithick]
			\tikzstyle{final}=[circle,thick,draw=black,fill=gray!40,text=black]
			\node[state,initial] (0) {$q_0$};
			\node[state] (1) [right of=0] {$q_1$};
			\node[state, final] (2) [right of=1] {$q_2$};
			\path
				(0) edge [bend left=0] node {b} (1)
				(1) edge [loop above] node {$A$} (1)
					edge [bend left=0] node {b} (2);
		\end{tikzpicture}
		\caption{Graph of a transition system that accepts the language $bA^*b$}
		\label{fig:DR2}
	\end{figure}

	\item
	To achieve an equivalent \textit{dfa} with minimal number of states, we take advantage of the fact that the transition system given in Figure \ref{fig:DR2} has no $\lambda$-transition.
	Therefore, it is also a non-deterministic finite automaton.
	To construct an equivalent \textit{dfa} from the transition system/\textit{ndfa} given in Figure \ref{fig:DR2}, Table \ref{tab:TB1} is constructed where $\mathcal{K}\left(S\right)$ is set of all accessible states.

	\begin{table}[H]\centering
		\begin{tabular}{|c|c||c|c|}
			\hline
			S & $\mathcal{K}\left(S\right)$ & S & $\mathcal{K}\left(S\right)$ \\
			\hline
			$\emptyset$ & $\emptyset$ & $\{q_0,q_1\}$ & $Q$ \\
			$\{q_0\}$ & $Q$ & $\{q_1,q_2\}$ & $\{q_1,q_2\}$ \\
			$\{q_1\}$ & $\{q_1,q_2\}$ & $\{q_0,q_2\}$ & $Q$ \\
			$\{q_2\}$ & $\{q_2\}$ & $\{q_0,q_1,q_2\}$ & $Q$\\
			\hline
		\end{tabular}
		\caption{Accessible States From All Possible Sets of States}
		\label{tab:TB1}
	\end{table}

	Based on Table \ref{tab:TB1}, set of states in an equivalent \textit{dfa} will be extracted as $\{Q,\{q1,q2\}, \{q2\}, \emptyset\}$ \ref{tab:TB1}.
	The equivalent \textit{dfa} can now be constructed by assigning $Q$ as initial state and $q_2$ as final state.
	The graph of such equivalent \textit{dfa} has been given in Figure \ref{fig:DR3}.

	\begin{figure}[H]\centering
		\begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,node distance=3cm,semithick]
			\tikzstyle{final}=[circle,thick,draw=black,fill=gray!40,text=black]
			\node[state,initial] (0) {$Q$};
			\node[state] (1) [right of=0] {$\{q_1,q_2\}$};
			\node[state, final] (2) [right of=1] {$\{q_2\}$};
			\node[state] (3) [right of=2] {$\emptyset$};
			\path
				(0) edge [bend right=0] node {b} (1)
					edge [bend left=40] node {A} (3)
				(1) edge [loop above] node {$A$} (1)
					edge [bend right=0] node {b} (2)
				(2) edge [bend right=0] node {$\{b\} \cup A$} (3)
				(3) edge [loop right] node {$\{b\} \cup A$} (3);
		\end{tikzpicture}
		\caption{Graph of an equivalent \textit{dfa} that accepts te language $bA^*b$}
		\label{fig:DR3}
	\end{figure}

\end{enumerate}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% UMB-CS240-2016S: Programming in C
% Copyright 2016 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/UMB-CS240-2016S
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 1}

Write a program \texttt{bmi.c} that takes your weight in pounds and height in inches and calculates your Body Mass Index (BMI) according to Equation \ref{eq1}.
To evaluate your BMI, program should as well indicate under which group you are classified according to Table \ref{tab1} obtained from the Department of Health and Human Services/National Institution of Health.

\begin{equation}
BMI = \frac{weightInPounds \times 703}{heightInInches^2}
\label{eq1}
\end{equation}

\begin{table}[H]\centering
\begin{tabular}{|r|l|}
\hline
Group & BMI index \\
\hline
Underweight & less than 18.5 \\
Normal & between 18.5 and 24.9 \\
Overweight & between 25 and 29.9 \\
Obese & greater than or equal to 30 \\
\hline
\end{tabular}
\caption{BMI classification}\label{tab1}
\end{table}

Following is the expected output of a sample run of your program.

\begin{terminal}
$ gcc bmi.c -o bmi
$ ./bmi
Your height (in): 72
Your weight (lb): 145
Your BMI is 19.66.
You are classified as normal.
\end{terminal}


$C[a, b]$ - set of all continuous functions

\section{Metric spaces}

\begin{definition}[Metric space]
    $X$ set. Function $d: x\times X \to [0, \infty)$ is called a metric if the following condition are met:
    \begin{enumerate}
        \item $d(x, y) = 0 \iff x = y$
        \item $d(x, y) = d(y, x)$
        \item $d(x,z) \leq d(x,y) + d(y, z)$ - the inequality of the triangle
    \end{enumerate}
\end{definition}
\begin{definition}
    $(X, d)$ - metric space.
\end{definition}
\begin{example}[Discrete space]
    $X$ - arbitrary.
    \[
    d(x, y) = \begin{cases}
            1 & x \not= y \\
            0 & x = y
        \end{cases}
    .\]
\end{example}
\begin{example}[Real line]
    $X = \mathbb{R}, \;\; d(x, y) = \left| x - y \right| $
\end{example}
\begin{example}[n-dimentional space]
    $X = \mathbb{R}^{n} = \{x = \left( x_1, \ldots, x_n \right) \mid x_i \in \mathbb{R}, 1 \leq i \leq n \} $
    \[
    d(x, y) = \sqrt{\sum_{i=1}^{n} \left( x_i - y_i \right)^2 }
    .\]
\end{example}

\begin{example}
    $d_1(x, y) = \sum_{i=1}^{n} \left| x_i - y_i \right|$ - metric on $\mathbb{R}^{n}$

    \begin{proof}
    \[
    d_1(x, z) = \sum_{i=1}^{n} \left| x_i - z_i \right| \leq \sum_{i=1}^{n} \left( \left| x_i - _i \right| + \left| y_i - z_i \right|  \right) =
    d_1(x, y) + d_(y, z)
    .\]
    \end{proof}

    $d_{\infty} (x, y) = \underset{1\leq i \leq n}{\max} \left| x_i - y_i \right|$ - metric on $\mathbb{R}^{n}$

    \begin{proof}
        \begin{align*}
        d_{\infty}(x, y) = 0 \iff \forall i : x_i = y_i \iff x = y \\
        d_{\infty}(x, z) = \underset{1\leq i \leq n}{\max} \left| x_i - z_i \right|  \leq d_{\infty}(x, y) + d_{\infty}(y, z) \\
        \left| x_i - z_i \right| \leq \left| x_i - y_i \right| + \left| y_i - z_i \right|
        .\end{align*}
    \end{proof}
    \[
    1 \leq p < \infty : d_{p} (x, y) = \left( \sum_{i=1}^{n} \left| x_i - y_i \right| ^{p} \right) ^{\frac{1}{p}}
    .\]
    \[
    0 < p < 1 : d_{p}(x, y) = \sum_{i=1}^{n} \left| x_i - y_i \right| ^{p}
    .\]

\end{example}

\begin{example}
$C[a, b]$ set of all continuous functions.
$f: [a, b] \to \mathbb{R}$
\[
d(f, g) = \underset{a \leq t \leq b}{\sup} \left| f(t) - g(t) \right|
.\]
$d(f, g)$ is a metric on $C[a, b]$.
\end{example}

\begin{example}
    $C_b[\mathbb{R}]$ - a set of all continuous and limited functions $f: \mathbb{R} \to \mathbb{R}$.
    \[
    d(f, g) = \underset{t \in \mathbb{R}}{\sup} \left| f(t) - g(t) \right|
    .\]
\end{example}
\begin{example}
    $(X, d)$ - metric space, $Y \subset X$
    \[
    d(y_1, y_2), y_1, y_2 \in Y
    .\]
    $(Y, d)$ - subspace $X$.
\end{example}

\begin{definition}
    $\left( X, d \right) $ - metric space, $\left\{ x_n : n\geq 1 \right\} $
    series of $X$ elements. $\left\{ x_n : n\geq 1 \right\} $ converges to $x \in X$ if $\underset{n\to\infty}{\lim} d(x_n, x) = 0$.
    \[
        \left( \forall \varepsilon > 0 \quad \exists N \quad \forall n \geq N \quad d(x_n, x) < \varepsilon \right)
    .\]
    \[
    x = \lim_{n \to \infty} x_n
    .\]
\end{definition}

\begin{theorem}
    In metric space convergent sequence has only one boundary.
\end{theorem}
\begin{proof}
    Let $\lim_{n \to \infty} x_n = x, \; \lim_{n \to \infty} x_n = y $.
    \[
    0 \leq d(x, y) \leq d(x, x_n) + d(x_n, y) \to 0
    .\]
    \[
    \Rightarrow d(x, y) = 0 \Rightarrow x = y
    .\]
\end{proof}

$\left( X, d_x \right), \left( Y, d_y \right) $ - metric spaces. $f : X \to Y$.

\begin{definition}
    \begin{enumerate}
        \item $f$ continuous in point $x_0 \in X$ if
            \[
            x_n \to x_0 \text{ in $X$ } \Rightarrow f(x_n) \to f\left( x_0 \right) \text{ in $Y$ }
            .\]
        \item $f$ continuous on $X$ if $f$ continuous in every point $x_0 \in X$.
    \end{enumerate}
\end{definition}

\begin{remark}
    $f$ continuous in $x_0 \in X$ then and only then if
    \[\forall \varepsilon > 0 \quad \exists \delta > 0 : d_x(x, x_0) < \delta \Rightarrow d_x(f(x), f(x_0)) < \varepsilon \]
\end{remark}

\begin{definition}
    \begin{enumerate}
        \item $f: X \to Y$ is called homeomorphism if $f$ is bijective, continuous and $f^{-1}$ is continuous.
        \item $f: X \to Y$ isometric if
            \[
            d_y (f(x), f(x')) = d_x(x, x')
            .\]
            Isometrie is always continuous.
    \end{enumerate}
\end{definition}

$x \in X, \;\; r > 0$
\begin{definition}
    Open ball \[
    \mathbb{B}(x, y) = \{ y \in X : d(y, x) < r \}
    .\]
\end{definition}
\begin{definition}
    Closed ball \[
    \overline{\mathbb{B}} = \{ y \in X : d(y, x) \leq r \}
    .\]
\end{definition}

Convergence can be rewritten using the last two definitions:
\[
x_n \to x \iff \forall \varepsilon > 0 \quad \exists N \quad
\forall n \geq N \quad x_n \in \mathbb{B}(x, \varepsilon)
.\]

\begin{figure}[ht]
    \centering
    \incfig{convergence-in-terms-of-open-and-closed-ball-definitions}
    \caption{Convergence in terms of open and closed ball definitions}
    \label{fig:convergence-in-terms-of-open-and-closed-ball-definitions}
\end{figure}

\begin{definition}
    $A \subset X$. Point $x$ is tangent to set $A$ if \[
    \forall \varepsilon > 0 \quad \mathbb{B}(x, \varepsilon) \cap A \not= \varnothing
    .\]
\end{definition}

\begin{example}
    $X = \mathbb{R}$, $A = (a, b)$. $a$ and $b$ are tangent to $A$. \\
    All elements from set $A$ are tangent to $A$. \\
    If there's some $\exists c > b$ then we can pick some ball around $c$
    of radius $r$. In that ball there would be no elements from A.
\end{example}
\begin{figure}[ht]
    \centering
    \incfig{tangent-point-example}
    \caption{Tangent point example}
    \label{fig:tangent-point-example}
\end{figure}

\begin{definition}
    $\overline{A} = \{ x \in X : x \text{ tangent to } A \} $ closure of set $A$.
\end{definition}

\begin{theorem}[Properties of closure]
    Set $A$ with closure $\overline{A}$ has the following properties:
    \begin{enumerate}
        \item $A \subset \overline{A}$
        \item $\overline{\overline{A}} = \overline{A}$ --- idempotence
        \item $A \subset B \Rightarrow \overline{A} \subset \overline{B}$
        \item $\overline{A \cup B} = \overline{A} \cup \overline{B}$
    \end{enumerate}
\end{theorem}
\begin{proof}
    \begin{itemize}
        \item[1] \begin{align*}
        x \in A \Rightarrow \mathbb{B}(x, \varepsilon) \cap A \not= \varnothing \text{ since it contains $x$ }
        .\end{align*}
    \item[3] \begin{align*}
        x \in \overline{A} \Rightarrow \mathbb{B}(x, \varepsilon) \cap A \not= \varnothing \Rightarrow \\
        \Rightarrow \mathbb{B}(x, \varepsilon) \cap B \not= \varnothing \Rightarrow
        x \in \overline{B}
        .\end{align*}
        \item[2] \begin{align*}
            \overline{A} \subset \overline{\overline{A}} \\
            \text{ need to show that } \overline{\overline{A}} \subset \overline{A} \\
            x \in \overline{\overline{A}}, \varepsilon > 0. \quad
            \mathbb{B}(x, \varepsilon) \cap \overline{A} \not= \varnothing \\
            \text{ exists point } y \in \mathbb{B}(x, \varepsilon) \cap \overline{A}.
        .\end{align*}
        \begin{figure}[ht]
            \centering
            \incfig{eps-ball-y-in-eps-ball-x-with-point-from-a}
            \caption{Eps-Ball Y in eps-Ball x with point from A}
            \label{fig:eps-ball-y-in-eps-ball-x-with-point-from-a}
        \end{figure}

        Lets show that $\mathbb{B}(y, \varepsilon - d(x, y)) \subset \mathbb{B}(x, \varepsilon)$.
        \begin{gather*}
            z \in \mathbb{B}(y, \varepsilon - d(x, y)) \\
            \text{ for $z$ the following is met } d(z, y) < \varepsilon - d(x, y) \\
            \varepsilon > d(z, y) + d(y, x) \geq d(z, x) \Rightarrow
            z \in \mathbb{B}(x, \varepsilon) \\
            \mathbb{B}(y, \varepsilon - d(x, y)) \cap A \not= \varnothing \Rightarrow
            \mathbb{B}(x, \varepsilon) \cap A \not= \varnothing \\
            \Rightarrow x \in \overline{A}.
        \end{gather*}

    \item[4] \[
    A \subset A \cup B \Rightarrow \overline{A} \subset \overline{A \cup B}; \; \overline{B} \subset \overline{A \cup B}
    \]
    \[
    \Rightarrow \overline{A} \cup \overline{B} \subset \overline{A \cup B}
    .\]
\begin{figure}[ht]
    \centering
    \incfig{for-stupid-idiots}
    \caption{For stupid idiots}
    \label{fig:for-stupid-idiots}
\end{figure}

    Need to prove $\overline{A \cup B} \subset \overline{A} \cup \overline{B}$

    (by contradiction) \\
    Let $x \in \overline{A \cup B}$ and $x \not\in \overline{A}$, $x \not\in \overline{B}$.
    \[
    \exists \varepsilon_1 > 0 : \mathbb{B}\left( x, \varepsilon_1 \right) \cap A = \varnothing
    .\]
    \[
    \exists \varepsilon_2 > 0 : \mathbb{B}\left( x, \varepsilon_2 \right) \cap B = \varnothing
    .\]
    \[
    \varepsilon = \min(\varepsilon_1, \varepsilon_2). \quad
    \mathbb{B}(x, \varepsilon) \cap \left( A \cup B \right) = \varnothing
    .\]
    \[
    \Rightarrow \overline{A \cup B} = \overline{A} \cup \overline{B}
    .\]
    \end{itemize}
\end{proof}

\begin{theorem}
    $x \in \overline{A} \iff$ in set A exists series $\left( x_n : n \geq 1 \right) $ that converges to $x$.
\end{theorem}
\begin{proof}
    \begin{itemize}
        \item[($\Rightarrow$)] Let $x\in \overline{A}$
            \[
            \forall \varepsilon > 0 \quad \mathbb{B}(x, \varepsilon) \cap A \not= \varnothing
            .\]
            let $\varepsilon_n = \frac{1}{n} \quad$ \\
            $\forall n \geq 1$ exists point $x_n \in A \cap \mathbb{B}(x, \frac{1}{n}$ \\
            \[
            0 \leq d(x, x_n) < \frac{1}{n} \to 0. \quad \lim_{n \to \infty} x_n = x
            .\]
        \item[($\Leftarrow$)] Let $\lim_{n \to \infty} x_n = x$ and $x_n \in A$. \\
            \[
            \forall \varepsilon > 0 \quad \exists N \quad \forall n \geq N \quad d(x_n ,x) < \varepsilon
            .\]
            \[
            x_n \in \mathbb{B}(x, \varepsilon) \cap A \not= \varnothing
            .\]
            \[
            \Rightarrow x \in \overline{A}
            .\]
    \end{itemize}
\end{proof}

\begin{definition}
    A is dense in set B if $B \subset \overline{A}$ (any B element can be approached to elements of A)
\end{definition}
\begin{definition}
    A dense everywhere if $\overline{A} = X$.
\end{definition}
\begin{definition}
    Metric space $\left( X, d \right) $ is separable if exists dense everywhere countable set.
\end{definition}

\begin{example}
    \begin{enumerate}
        \item $\mathbb{R}$ - separable space. $\overline{\mathbb{Q}} = \mathbb{R}$
        \item $\mathbb{R}^{n} $ - separable space related to any metric $d_{p}, \; 0 < p \leq \infty$. $\overline{\mathbb{Q}^{n}} = \mathbb{R}^{n}$
        \item $X, d$ - discrete. $\mathbb{B}(x, \varepsilon) = \{ y : d(x, y) < \varepsilon \} $. But if $0 < \varepsilon < 1$ then
            \[
            \mathbb{B}(x, \varepsilon) \cap A \not= \varnothing \iff x \in A.
            .\]
            \[
             \Rightarrow \overline{A} = A
            .\]
        The only dense everywhere set is X.
    \item $C[a, b]$; $d(f, g) = \underset{t \in [a, b]}{\sup} \left| f(t) - g(t) \right| $ \\
        By Weierstrasse theorem $\forall f \in C[a, b] \quad \forall \varepsilon > 0$ exists polynomial
    \[ P(t) = a_0 + a_1 t + \ldots + a_d t^{d} : \underset{t \in [a, b]}{\sup}
    \left| f(t) - P(t) \right| < \varepsilon \]
    Dense everywhere set is set of polymonials with rational coefficients.

    \item $C_{b}(\mathbb{R}), d(f, g) = \underset{t \in \mathbb{R}}{\sup} \left| f(t) - g(t) \right| $ - is not separable metric space.

\begin{figure}[ht]
    \centering
    \incfig{line-for-example-of-not-separable-metric-space}
    \caption{Line for example of not separable metric space}
    \label{fig:line-for-example-of-not-separable-metric-space}
\end{figure}

\[ A \subset \mathbb{Z} \quad f_{A}(n) =
 \begin{cases}
     1 & n \in A \\
     0 & n \in \mathbb{Z} \backslash A
 \end{cases}
.\]
\[
A \not= A' \quad n \in A \backslash A' \text{ or } n \in A' \backslash A
.\]
\[
d(f_{A}, f_{A'}) = 1
.\]
\[
\mathbb{B}(f_{A}, \frac{1}{2}) \cap \mathbb{B}(f_{A'}, \frac{1}{2}) = \varnothing
.\]
In space $C_b(\mathbb{R})$ exists a continuum family of open balls that do not intersect.

If dense everywhere set exists than in every open ball must be an element of the one.
\begin{figure}[ht]
    \centering
    \incfig{family-of-open-balls-each-containing-an-element}
    \caption{Family of open balls each containing an element}
    \label{fig:family-of-open-balls-each-containing-an-element}
\end{figure}
    \end{enumerate}
\end{example}


\section{Measurement Uncertainty Analysis}

\subsection{Uncertainty for natural angular frequency}

To estimate type-A uncertainty of $(10T)$,
the standard deviation of the average value can be calculated as

\[
\begin{split}
&t_{0.95}=3.18,\quad n=4,\\
&s_{\overline{10T}}=\sqrt{\frac{1}{n(n-1)}\sum_{i=1}^n((10T)_i-\overline{10T})^2}.\\
&\Delta_{10T,A}=\frac{3.18}{\sqrt{4}}\times0.0076811\approx 0.017s.
\end{split}
\]

The type-B uncertainty of $(10T)$ is 0.001s. Hence, the uncertainty of $(10T)$ is

$$ u_{10T}=\sqrt{\Delta_A^2+\Delta_B^2}=\sqrt{0.017^2+0.001^2}\approx 0.017s $$

For the uncertainty of $\omega_0$,

$$\omega_0=\frac{20\pi}{10T},\quad \frac{\partial \omega_0}{\partial
  (10T)}=-\frac{20\pi}{(10T)^2}$$

\[
\begin{split}
&u_{\omega_0}=\sqrt{\frac{\partial \omega_0}{\partial
    (10T)}^2(u_{10T})^2}=\frac{20\pi}{(10T)^2}u_{(10T)}\\
&=\frac{20\times3.1416}{15.485^2} \times 0.017 = 0.004rad/s,\\
\end{split}
\]

$$ u_{\omega_0,r}=\frac{u_{\omega_0}}{\omega_0}\times100\%
=\frac{0.004}{4.058}=0.10\% $$

\subsection{Uncertainty of damping coefficient}

The type-A uncertainty of $\ln(\theta_i/\theta_{i+5})$ can be determined by
calculating $s\times t_{0.95}/\sqrt{n}$.

Let $k = s \times t_{0.95}/\sqrt{n}$.

$$ t_{0.95}=2.78,\quad n=5$$

$$ \Delta_{A,k}=t_{0.95}/\sqrt{5}\times
\sqrt{\frac{1}{5-1}\sum_{i=1}^5(k_i-\bar{k})^2}=2.78/\sqrt{5}\times
0.00444 = 0.005 $$

Then, for the type-B uncertainty,

$$ \frac{\partial k}{\partial \theta_{i}}=\frac{1}{\theta_i} $$
$$ \frac{\partial k}{\partial \theta_{i+5}}=-\frac{1}{\theta_{i+5}} $$
$$ \Delta_{B,k}=\sqrt{\frac{\partial k}{\partial
    \theta_{i}}^2(u_{\theta_i})^2+\frac{\partial k}{\partial
    \theta_{i+5}}^2(u_{\theta_{i+5}})^2}
=\sqrt{(\frac{u_{\theta_i}}{\theta_i})^2+(\frac{u_{\theta_{i+5}}}{\theta_{i+5}})^2}  $$

When $i=0$, $\theta_0=89\degree$, $u_{\theta_2}=1\degree$,
$\theta_5=54\degree$, $u_{\theta_5}=1\degree$.

\[
\Delta_{B,k}=\sqrt{(\frac{1}{89})^2+(\frac{1}{54})^2}=0.022
\]

Considering the type-B uncertainty of $\ln(\theta_i/\theta_{i+5})$, the combined
uncertainty is

\[
\begin{split}
u_{k}&=\sqrt{(0.005)^2+(0.022)^2}= 0.022,\\
u_{k,r}&=\frac{u_k}{\bar{k}}\times100\%=\frac{0.022}{0.511}\times100\%=4\%
\end{split}
\]

Hence

$$ \ln(\theta_i/\theta_{i+5})=0.456\pm0.005, \quad u_r=1.09\% $$

For $10T=15.792 \pm 0.001s$, we know that
\[
T=1.5792\pm0.0001s, \quad u_{r,T}=0.006\%.
\]

Then calculate the uncertainty of $\beta=\ln(\theta_i/\theta_{i+5})/(5T)$.

$$ \frac{\partial \beta}{\partial T}=-\frac{k}{5T^2}$$

$$ \frac{\partial \beta}{\partial k}=\frac{1}{5T} $$

\[
\begin{split}
u_{\beta}&=\sqrt{(\frac{\partial \beta}{\partial T})^2(u_T)^2+(\frac{\partial
    \beta}{\partial k})^2(u_k)^2}
=\sqrt{(\frac{k^2}{25T^4})(u_T)^2+(\frac{1}{25T^2})(u_k)^2}\\
&=\sqrt{(\frac{0.511^2}{25\times1.5792^4})(0.0001^2)+(\frac{0.022^2}{25\times1.579
    2^2})}  \\
&= 0.003s^{-1}
\end{split}
\]

where $k=\ln(\theta_i/\theta_{i+5})$.

Thus,

\[
\beta=0.0648\pm 0.003s^{-1}, \quad u_{\beta,r}=5\%
\]

% TODO:

\subsection{Uncertainty of $\theta_{st}$ vs. $\omega$ and $\varphi$ vs. $\omega$}

We denote that $r=\omega/\omega_0$.
To determine the uncertainty of ratio, we know that

$$ r=\frac{20\pi}{10T\omega_0} $$
$$ \frac{\partial r}{\partial (10T)}=-\frac{20\pi}{(10T)^2\omega_0} $$
$$ \frac{\partial r}{\partial \omega_0}=-\frac{20\pi}{(10T)\omega_0^2} $$

\[
\begin{split}
u_{r}&=\sqrt{(\frac{\partial r}{\partial
    (10T)})^2(u_{10T})^2+(\frac{\partial r}{\partial
    \omega_0})^2(u_{\omega_0})^2}\\
&=\sqrt{\frac{400\pi^2}{(10T)^4\omega_0^2}(u_{10T})^2+\frac{400\pi^2}{(10T)^2\omega_0^4}(u_{\omega_0})^2}.
\end{split}
\]

As is calculated before, $\omega_0=3.9787 \pm 0.006 \ rad/s$.

For instance, when $10T= 15.79225  \pm 0.001s$, the uncertainty of $r$ is
\[
u_{r}=\sqrt{\frac{400\times3.1416^2}{15.79225 ^4\times 3.9787^2}(0.001)^2
  +\frac{400\times3.1416^2}{15.79225^2\times3.9787^4}(0.004)^2}=0.0009
\]

For the uncertainty of $\varphi$ and $\theta$, these two data only obtain type-B
uncertainty, so the combined uncertainty of them are both equal to
$\Delta_B=1\degree$, thus

$$ u_{\varphi}=1\degree $$
$$ u_{\theta}=1\degree. $$


\subsection{K-Means}
\label{sec:ctm-km}

The first clustering algorithm we tried was K-Means \cite{macqueen1967some}. It iteratively tries to group unlabeled data points with similar features together by minimizing the mean distance between each point in each group. Figure~\ref{fig:ctm-km} visualizes the groups labeled by the K-Means algorithm on the 2017 class for the Non-Blocking IO assignment.

The K-Means algorithm constructs each group to minimize the average distance from the group's centroid to every point in its corresponding group. Since the majority of students in our data set received full marks, we assumed the majority of the data points in each group should also represent full-mark solutions. As a result, the volume of full-mark solution points should draw the centroid of each group towards \textquote{regions} representing higher marks. Based on this idea, we calculated a student's score based on their distance to their corresponding group's centroid; the closer a student was to their group's centroid, the higher their probability of receiving a high score.

Let $P_i$ denote the point representing student $i$ and $C_i$ denote the centroid of the student's group. The distance from centroid $d_i$ is defined as:
\begin{equation*}
d_i = \norm{P_i - C_i}
\end{equation*}

As mentioned in Figure~\ref{fig:ctm-km}, the scales in the axes have no tangible interpretations. Therefore, the distances from centroid $d_i$ we calculated for each point also had no meaningful interpretation. As a result, we tried to evaluate each point's centroid distance relative to all other points. In other words, we calculated each student's score based on their performance relative to the rest of their class.

Let $\mu$ denote the mean and $\sigma$ denote the standard deviation of centroid distances of every student. We calculated student $i$'s score as follows:
\begin{equation*}
\text{Score}_i = 100 - 20 \cdot \frac{\abs{d_i - \mu}}{\sigma}
\end{equation*}

This equation deducted up to 20 points for each standard deviation of centroid distance a student was from their corresponding group's centroid. We chose to deduct 20 points per standard deviation arbitrarily after experimenting with different values. The main idea was to deduct marks based on how far the solution was from the centroid, where we assumed majority of the full mark solutions were located.

\begin{figure}
\includegraphics[width=\textwidth]{conversion-to-mark/marking_paster_nbio_ece459-a1-w2017_km}
\caption[K-Means Clustering]{Each data point represents a student solution to the Non-Blocking IO assignment from the 2017 class. It is important to note that this is a 2D projection of a multidimensional data set. Since we preprocessed the data with PCA, the first two dimensions presented here already capture the majority of the data variance. The scales in the axes are omitted because they have no tangible interpretations; they are meant for interpreting relative distances. The K-Means algorithm assigned each point a group, represented by their corresponding colour. Each set of concentric circles represent an arbitrary distance from the centroid of each group.}
\label{fig:ctm-km}
\end{figure}