\subsection{Curl in tensor form}
The curl of a vector function can be expressed in tensor form as
$$\mathop{\rm curl}{\bf F}=\epsilon_{ijk}\,{\partial F_k\over\partial x_j}$$
where $\epsilon_{ijk}$ is the Levi-Civita tensor.
The following script demonstrates that this formula is equivalent
to computing curl the old fashioned way.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
-- Define epsilon
epsilon = zero(3,3,3)
epsilon[1,2,3] = 1
epsilon[2,3,1] = 1
epsilon[3,1,2] = 1
epsilon[3,2,1] = -1
epsilon[1,3,2] = -1
epsilon[2,1,3] = -1
-- F is a generic vector function
F = (FX(),FY(),FZ())
-- A is the curl of F
A = outer(epsilon,d(F,(x,y,z)))
A = contract(A,3,4) --sum across k
A = contract(A,2,3) --sum across j
-- B is the curl of F computed the old fashioned way
BX = d(F[3],y)-d(F[2],z)
BY = d(F[1],z)-d(F[3],x)
BZ = d(F[2],x)-d(F[1],y)
B = (BX,BY,BZ)
-- Are A and B equal? Subtract to find out.
A-B
\end{Verbatim}

Here is the result when the script runs.

$\displaystyle \begin{bmatrix}0\\0\\0\end{bmatrix}$

The following is a variation on the previous script.
The product $\epsilon_{ijk}\,\partial F_k/\partial x_j$
is computed in just one line of code.
In addition, the outer product and the contraction across $k$
are now computed with a dot product.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
F = (FX(),FY(),FZ())
epsilon = zero(3,3,3)
epsilon[1,2,3] = 1
epsilon[2,3,1] = 1
epsilon[3,1,2] = 1
epsilon[3,2,1] = -1
epsilon[1,3,2] = -1
epsilon[2,1,3] = -1
A = contract(dot(epsilon,d(F,(x,y,z))),2,3)
BX = d(F[3],y)-d(F[2],z)
BY = d(F[1],z)-d(F[3],x)
BZ = d(F[2],x)-d(F[1],y)
B = (BX,BY,BZ)
-- Are A and B equal? Subtract to find out.
A-B
\end{Verbatim}

This is the result when the script runs.

$\displaystyle \begin{bmatrix}0\\0\\0\end{bmatrix}$


\subsection{Method} \label{subsec:method_logistic_regression}

The goal of classification is to assign an input vector $\bold{x}$ to one of K classes $\mathcal{C}_k$ where $k=0,1,...,K$. Linear regression can be extended to binary classification by two changes \cite{murphy_machine_2012}. First is the change of the Gaussian distribution for $y$ with a Bernoulli distribution, for the case of binary response, $y_i \in \{0,1\}$.

\begin{align}
     p(y \mid \mathbf{x}, \boldsymbol{\beta})&=Ber(y \mid p(y=1\mid \boldsymbol{x},\boldsymbol{\beta}))) \\
        &= p^{y}\left(y=1\mid  \mathbf{x}, \boldsymbol{\beta} \right)\left(1-p(y=1\mid  \mathbf{x}, \boldsymbol{\beta} )\right)^{1-y}
\end{align}

Second is to pass the linear combination of the inputs and parameters to a function that outputs in the range $[0,1]$. This is achieved by using a logit function also known as sigmoid function, $p(y=1\mid \bold{x}, \boldsymbol{\beta}) = sigm(\boldsymbol{\beta}^T \bold{x})$.
\newline\newline
We define sigmoid function as,
\begin{equation}
    sigm(z ) = \frac{1}{1 + e^{-z}}
\end{equation}

Putting these two steps together we get,
\begin{equation}
    p(y \mid \mathbf{x}, \boldsymbol{\beta})=Ber(y|sigm(\boldsymbol{\beta}^T \mathbf{x})))
\end{equation}
For a multivariate set of data $\mathcal{D} = \{y_i, \mathbf{x_i}\}$ with $y_i$ being a binary response $y_i \in \{0,1\}$ and $\mathbf{x_i}$ is a vector of size $1\times n$ where n is the number of features, the likelihood for all possible outcome is,
\begin{equation}\label{eq:likelihood}
    P(\mathcal{D}\mid \boldsymbol{\beta}) =  \Pi_i p(y_i \mid \mathbf{x}_i, \boldsymbol{\beta})
\end{equation}

From maximum likelihood estimation principle, the most probable observed data can be obtained from the maximum log-likelihood function where we maximizes with respect to $\boldsymbol\beta$. The log-likelihood for equation (\ref{eq:likelihood}) is given by,
\begin{equation}
  \mathrm{LL}(\boldsymbol{\beta}) =\sum_{i=1}^{N}\left[y_{i} \log p(y=1\mid  \mathbf{x}, \boldsymbol{\beta} )+\left(1-y_{i}\right) \log \left(1-p(y=1\mid  \mathbf{x}, \boldsymbol{\beta} )\right)\right]
\end{equation}

For logistic regression, the cost function is just the negative log-likelihood. We want a cost function to be minimized hence a negative sign on the log-likelihood.

\begin{align}
      C(\boldsymbol{\beta}) =-\sum_{i=1}^{N}\left[y_{i} \log p(y=1\mid  \mathbf{x}, \boldsymbol{\beta} )+\left(1-y_{i}\right) \log \left(1-p(y=1\mid  \mathbf{x}, \boldsymbol{\beta} )\right)\right]
\end{align}

This is also known as the cross-entropy error function. Unlike linear regression, the maximum likelihood estimate (MLE) for the parameters is no longer in closed form, so we need an optimization algorithm to compute it. One famous simple algorithm for unconstrained optimization is gradient descent, also known as steepest descent. It is given as follows,

\begin{align}
    g_k &= -\mathbf{X}^T(\mathbf{y}-\mathbf{\mu}) \\
    \beta_{k+1} &= \beta_k - \eta_k g_k
\end{align}

where $\eta_k$ is the learning rate, and $g_k$ is the gradient of the loss function with respect to $\mathcal{\beta}$.  Several hyperparameters can be utilized in controlling the training. We use the following tuning parameters in our code: learning rate, number of epochs, and number of batches. Learning rate is just the step size on each iteration as it approaches the minimum of the cost function. The number of batches is the number of samples processed before updating the learnable parameter. And the number of the epoch is a complete cycle over the training data. After completing one epoch, the optimal parameter $\boldsymbol{\beta}$ can be obtained. This can be used to calculate a continuous response $t =\boldsymbol{X}^T \boldsymbol\beta$. If we set a threshold value for $t$ we can create a decision rule. In our case, we use the following decision rule:

\begin{equation}
    \hat{y}=\begin{cases}
        1 & \text{ if } t \geq 0 \\
        0 & \text{ if } t < 0
    \end{cases}
\end{equation}

There are limitations of gradient descent. It can get stuck in local minima, it is sensitive to the initial condition, and for a large dataset, it is computationally expensive \cite{mehta2019high}. It is alleviated through stochasticity, for instance, by only taking the gradient of a subset of data called mini-batches.

\subsubsection{Stochastic Gradient descent with mini-batches}\label{subsec:SGD}
If there are n samples in total, and the mini-batch size set is M, there are n/M possible mini-batches $B_k $ where $k \in (1,2,..n/M)$. In this case, the gradient descent is approximated in each cycle over the mini-batches using a single mini-batch. The approximated gradient descent over a single batch , $c_i(\mathbf{x_i},\beta)$, is the gradient used to update the parameter. We denote the approximated gradient by $\nabla_\beta C^{MB}(\beta)$ written as,

\begin{equation}
    \nabla_{\beta} C^{MB}(\beta) = \sum_{i\in B_k} \nabla_\beta c_i(\textbf{x}_i, \beta)
\end{equation}

The SGD with mini-batches equation is given by,

\begin{align}
    g_k &=  \nabla_{\beta} C^{MB}(\beta)\\
\beta_{k+1} &= \beta_k - \eta_k g_k
\end{align}

\subsubsection{Support Vector Machine}\label{subsubsec:SVM}
Fundamentally support vector machine (SVM) is a two-class classifier. Since the problem at hand consists of two classes, benign and malignant, the Wisconsin breast cancer data is a good candidate to test SVM. SVM classifier determines a decision boundary by choosing parameters that maximize the margin. This constraint optimization problem is solved using Lagrange multipliers that follow Karush-Kuhn-Tucker conditions \cite{bishop2006pattern}. A dual representation of the Lagrangian form is expressed in terms of the kernel and obtained by taking the gradient of the Lagrangian form with respect to the parameters and setting this gradient to zero. The new data points are classified using the trained model by evaluating the signs of the output. To test SVM, we use scikit's built-in functionality.


\chapter{Separation of Variables}
Separation of variables is a technique which can be used to solve both ODEs and PDEs. The basic idea for an equation in two variables is to rewrite the equation so that each of the two variables is located on different sides of an equality sign, and since both sides of the equation depend on different variables, the two sides must be equal to a constant. We introduce this idea with the simple first order linear ODE
\begin{eqnarray}\label{eq:simpleode}
\ \frac{dy}{dt}=y.
\end{eqnarray}
As long as $y(t) \neq 0$ for any value of $t$, we can formally separate variables and rewrite eq.\ \eqref{eq:simpleode} as
\begin{eqnarray}
\ \frac{dy}{y}=dt.
\end{eqnarray}
Now we can solve for $y(t)$ by integrating both sides
\begin{eqnarray}
\int{\frac{dy}{y}}&=&\int{dt} \\
\ln{y}+a&=&t+b \\
e^{\ln{y}+a}&=&e^{t+b} \\
e^{\ln{y}}e^{a}&=&e^te^b \\
y&=&\frac{e^b}{e^a} e^t \\
y(t)&=&c e^t.
\end{eqnarray}
Where $a$, $b$, and $c$ are arbitrary constants of integration.

We now perform a similar example for a linear partial differential equation. The heat equation is
\begin{equation}
u_t=-u_{xx}.
\end{equation}
We suppose that $u=X(x)T(t)$, so that we obtain
\begin{equation}
X(x)\frac{\mathrm{d}T}{\mathrm{d}t}(t)=-\frac{\mathrm{d}^2X}{\mathrm{d}x^2}(x)T(t).
\end{equation}
We can rewrite this as
\begin{equation}
\frac{\frac{\mathrm{d}T}{\mathrm{d}t}(t)}{T(t)}=\frac{\frac{\mathrm{d}^2X}{\mathrm{d}x^2}(x)}{X(x)}=-C,
\end{equation}
where $C$ is a constant independent of $x$ and $t$. The two sides can be integrated separately to get $T(t)=\exp(-Ct)$ and either $X(x)=\sin(\sqrt{C}x)$ or $X(x)=\cos(\sqrt{C}x)$. Since the heat equation is linear, one can then add different solutions to the heat equation and still obtain a solution of the heat equation. Hence solutions of the heat equation can be found by
\begin{equation}
\sum_n \alpha_n\exp(-C_nt)\sin(\sqrt{C_n}x) + \beta_n\exp(-C_nt)\cos(\sqrt{C_n}x)
\end{equation}
where the constants $\alpha_n$, $\beta_n$ and $C_n$ are appropriately chosen. Convergence of such series to an actual solution is studied in mathematics courses on analysis (see for example Evans~\cite{Eva10} or Renardy and Rogers~\cite{RenRog04}), however the main ideas necessary to choose the constants, $\alpha_n$, $\beta_n$ and $C_n$ and hence construct such solutions are typically encountered towards the end of a calculus course or at the beginning of a differential equations course, see for example Courant and John~\cite{CouJoh98} or Boyce and DiPrima~\cite{BoyDip10}. Here, we consider the case where $x\in[0,2\pi]$, and for which we have periodic boundary conditions. In this case $\sqrt{C_n}$ must be integers, which we choose to be non-negative to avoid redundancies. At time $t=0$, we shall suppose that the initial condition is given by
\begin{equation}
u(x,t=0)=f(x).
\end{equation}
Now,
\begin{align}
&{} \int_0^{2\pi}\sin(nx)\sin(mx)=\left\{{\pi\quad m=n}\atop{0 \quad m\ne n}\right.,
\\&{} \int_0^{2\pi}\cos(nx)\cos(mx)=\left\{{\pi\quad m=n}\atop{0 \quad m\ne n}\right.,
\end{align}
and
\begin{align}
&{} \int_0^{2\pi}\cos(nx)\sin(mx)=0.
\end{align}
Thus we can consider the trigonometric polynomials as being orthogonal vectors. It can be shown that a sum of these trigonometric polynomials can be used to approximate a wide class of periodic functions on the interval $[0,2\pi]$; for well behaved functions, only the first few terms in such a sum are required to obtain highly-accurate approximations. Thus, we can suppose that
\begin{equation}
f(x)=\sum_n  \alpha_n\sin(\sqrt{C_n}x) + \beta_n\cos(\sqrt{C_n}x).
\end{equation}
Multiplying the above equation by either $\sin(\sqrt{C_n}x)$ or $\cos(\sqrt{C_n}x)$ and using the orthogonality of the functions, we deduce that
\begin{equation}
\alpha_n=\frac{\int_0^{2\pi}f(x)\sin(\sqrt{C_n}x)\mathrm{d}x}{\int_0^{2\pi}\sin^2(\sqrt{C_n}x)\mathrm{d}x}
\end{equation}
and
\begin{equation}
\beta_n=\frac{\int_0^{2\pi}f(x)\cos(\sqrt{C_n}x)\mathrm{d}x}{\int_0^{2\pi}\cos^2(\sqrt{C_n}x)\mathrm{d}x}.
\end{equation}

Most ODEs and PDEs of practical interest will not be separable. However, the ideas behind separation of variables can be used to allow one to find series solutions to a wide class of PDEs. These series solutions can also be found numerically and are what we will use to find approximate solutions to PDEs, and so the ideas behind this simple examples are quite useful.

%%%%%%%
%  Section
%%%%%%%
\section{Exercises}
\begin{enumerate}
\item[1)] Solve the ordinary differential equation
$$u_t=u(u-1)\quad u(t=0)=0.8$$
using separation of variables.
\item[2)]
\begin{enumerate}
\item[a)] Use separation of variables to solve the partial differential equation
$$u_{tt}=u_{xx}$$
with
$$u(x=0,t)=u(x=2\pi,t),$$
$$u(x,t=0)=\sin(6x)+\cos(4x)$$
and
$$u_t(x,t=0)=0.$$
\item[b)] Create plots of your solution at several different times and/or create an animation of the solution you have found.\footnote{Your solution should involve only a few modes and so you should be able to use a wide variety of software to create plots, for example a graphing calculator, a spreadsheet program such as Excel, Mathematica, Wolfram Alpha, Matlab, Maple, Python, Sage etc. You can use Wolfram Alpha and Sage online.}
\item[c)]  The procedure required to find the coefficients in the Fourier series expansion for the initial condition can become quite tedious/intractable. Consider the initial condition $u(x,t=0)=\exp(\sin(x))$. Explain why it would be difficult to compute the Fourier coefficients for this by hand. Also explain why it would be nice to have an algorithm or computer program that does this for you.
\end{enumerate}
\end{enumerate}


\section{Random Vectors}

A random vector $\mathbf X= \left(X^{(1)},\dots ,X^{(d)}\right)^ T$ of dimension $d \times 1$ is a vector-valued function from a probability space $\omega$ to $\mathbb {R}^ d$:\\

$  \mathbf{X}\, \, :\,  \Omega \longrightarrow   \mathbb {R}^ d$\\

$ \omega  \longrightarrow  \begin{pmatrix}  X^{(1)}(\omega ) \\ X^{(2)}(\omega )\\ \vdots \\ X^{(d)}(\omega )\end{pmatrix}$\\

where each $\, X^{(k)}\ $, is a (scalar) random variable on $\Omega$. \\

PDF of $\mathbf X$: joint distribution of its components $X^{(1)},\, \ldots ,\, X^{(d)}$. \\

CDF of $\mathbf X$:\\

$\mathbb {R}^ d \rightarrow   [0,1]$\\

$ \mathbf{x}  \mapsto   \mathbf{P}(X^{(1)}\leq x^{(1)},\ldots ,\, X^{(d)}\leq x^{(d)}).$\\

The sequence $\mathbf{X}_1, \mathbf{X}_2,\ldots$ converges in probability to $\mathbf{X}$ if and only if each component of the sequence $\, X_1^{(k)},X_2^{(k)},\ldots \,$ converges in probability to $\, X^{(k)}$.

\subsection*{Expectation of a random vector}
The expectation of a random vector is the elementwise expectation. Let $\mathbf X$  be a random vector of dimension $d \times 1$.\\

$   \mathbb E[\mathbf X] =  \begin{pmatrix} \mathbb E[X^{(1)}]\\ \vdots \\ \mathbb E[X^{(d)}]\end{pmatrix}.$\\

The expectation of a random matrix is the expected value of each of its elements. Let $X=\{X_{ij}\}$ be an $n \times p$ random matrix. Then $\mathbb{E}[X]$, is the $n \times p$ matrix of numbers (if they exist):\\

$\mathbb{E}[X]= \begin{bmatrix}
   \mathbb{E}[X_{11}]       & \mathbb{E}[X_{12}] & \dots & \mathbb{E}[X_{1p}] \\
   \mathbb{E}[X_{21}]       & \mathbb{E}[X_{22}] & \dots & \mathbb{E}[X_{2p}] \\
    \vdots & \vdots &\ddots & \vdots \\
     \mathbb{E}[X_{n1}]       & \mathbb{E}[X_{n2}] & \dots & \mathbb{E}[X_{np}] \\
\end{bmatrix}$\\

Let $X$ and $Y$ be random matrices of the same dimension, and let $A$ and $B$ be conformable matrices of constants.\\

$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y]$\\
$\mathbb{E}[AXB] = A \mathbb{E}[X] B$\\

\subsection*{Covariance Matrix}
Let $X$  be a random vector of dimension $d \times 1$ with expectation $\mu _{X}$.

Matrix outer products!\\

$\Sigma =\mathbb E[(X- \mu _{X})(X- \mu _{X})^ T] =$\\

$\mathbb{E}  \begin{pmatrix} \begin{bmatrix}

X_1 - \mu_1\\
X_2 - \mu_2\\
\ldots\\
X_d - \mu_d\\

\end{bmatrix} \begin{bmatrix} X_1 - \mu_1, X_2 - \mu_2,\ldots, X_d - \mu_d \end{bmatrix} \end{pmatrix}$\\

$\Sigma = Cov (X) = \begin{bmatrix}
\sigma_{11} & \sigma_{12} &\ldots & \sigma_{1d} \\
\sigma_{21} & \sigma_{22} &\ldots & \sigma_{2d} \\
\vdots & \vdots &\ddots & \vdots \\
\sigma_{d1} & \sigma_{d2} &\ldots & \sigma_{dd} \\

\end{bmatrix}$\\

The covariance matrix $\Sigma$ is a $d \times d$ matrix. It is a table of the pairwise covariances of the elemtents of the random vector. Its diagonal elements are the variances of the elements of the random vector, the off-diagonal elements are its covariances. Note that the covariance is commutative e.g. $\sigma_{12} = \sigma_{21}$ \\

Alternative forms:\\

$\Sigma = \mathbb {E}[XX^ T] - \mathbb {E}[X]\mathbb {E}[X]^ T =\\ = \mathbb {E}[XX^ T] - \mu _{X}\mu _{X}^ T$\\

Let the random vector $X \in \mathbb{R}^d$ and $A$ and $B$ be conformable matrices of constants.\\

$Cov(AX + B) = Cov(AX) = A Cov(X) A^T = A \Sigma A^T$

Every Covariance matrix is positive definite.\\

$\Sigma \prec 0$\\

\subsection*{Gaussian Random Vectors}

A random vector $\mathbf{X}=(X^{(1)},\ldots ,X^{(d)})^ T\,$ is a Gaussian vector, or multivariate Gaussian or normal variable, if any linear combination of its components is a (univariate) Gaussian variable or a constant (a “Gaussian" variable with zero variance), i.e., if $\alpha ^ T \mathbf{X}$ is (univariate) Gaussian or constant for any constant non-zero vector $\alpha \in \mathbb {R}^ d$.

\subsection*{Multivariate Gaussians}

The distribution of, $X$ the $d$-dimensional Gaussian or normal distribution, is completely specified by the vector mean $\mu =\mathbb E[\mathbf{X}]= (\mathbb E[X^{(1)}],\ldots ,\mathbb E[X^{(d)}])^ T$ and the $d \times d$ covariance matrix $\Sigma$. If $\Sigma$ is invertible, then the pdf of $X$ is:\\

$   f_{\mathbf X}(\mathbf x) = \frac{1}{\sqrt{\left(2\pi \right)^ d \text {det}(\Sigma )}}e^{-\frac{1}{2}(\mathbf x-\mu )^ T \Sigma ^{-1} (\mathbf x-\mu )}, ~ ~ ~ \\ \mathbf x\in \mathbb {R}^ d$\\

Where $\text {det}(\Sigma )$ is the determinant of $\Sigma$, which is positive when $\Sigma$ is invertible.

If $\mu = 0$ and $\Sigma$ is the identity matrix, then $X$ is called a standard normal random vector .

If the covariant matrix $\Sigma$ is diagonal, the pdf factors into pdfs of univariate Gaussians, and hence the components are independent.\\

The linear transform of a gaussian $X \thicksim N_d(\mu,\Sigma)$ with conformable matrices $A$ and $B$ is a gaussian:\\

$AX + B = N_d(A\mu + b, A \Sigma A^T)$

\subsection*{Multivariate CLT}

Let $X_1, \ldots, X_d \in \mathbb{R}^d$ be independent copies of a random vector $X$
such that $\mathbb{E}[x] = \mu$ ($d \times 1$ vector of expectations) and $Cov(X)= \Sigma$\\

$\sqrt(n)(\bar{X_n}-\mu) \xrightarrow[n \rightarrow \infty]{(d)} N(0,\Sigma)$\\

$\sqrt(n) \Sigma^{-1/2} \bar{X_n}-\mu \xrightarrow[n \rightarrow \infty]{(d)} N(0,I_d)$\\

Where $\Sigma^{-1/2}$ is the $d \times d$ matrix such that $\Sigma^{-1/2} \Sigma^{-1/2} = \Sigma^{1}$ and $I_d$ is the identity matrix.\\

\subsection*{Multivariate Delta Method}

Gradient Matrix of a Vector Function:\\

Given a vector-valued function $f:\mathbb {R}^ d \to \mathbb {R}^ k$, the gradient or the gradient matrix of $f$, denoted by $\nabla f$ , is the $d \times k$ matrix:\\

$   \nabla f =\\ = \begin{pmatrix} |& |& |& |\\ \nabla f_1&  \nabla f_2& \ldots & \nabla f_ k\\ |& |& |& |\\ \end{pmatrix} = \\
= \begin{pmatrix} \frac{\partial f_1}{\partial x_1}& \ldots & \frac{\partial f_ k}{\partial x_1}\\ \vdots & \ldots & \vdots \\ \frac{\partial f_1}{\partial x_ d}& \ldots & \frac{\partial f_ k}{\partial x_ d} \end{pmatrix}.$\\

This is also the transpose of what is known as the Jacobian matrix $\mathbf{J}_ f$ of $f$.\\

General statement, given\\

\begin{itemize}
	\item $\left(\mathbf{T}_ n\right)_{n\geq 1}$ a sequence of random vectors
	\item satisfying $\,  \sqrt{n} \left(\mathbf{T}_ n - \vec{\theta } \right) \xrightarrow [n\to \infty ]{(d)} \mathbf{T}$,

	\item a function $\mathbf{g}:\mathbb {R}^ d\to \mathbb {R}^ k$  that is continuously differentiable at $\vec{\theta }$,

\end{itemize}

then\\

$   \sqrt{n} \left(\mathbf{g}(\mathbf{T}_ n) - \mathbf{g}(\vec{\theta }) \right) \xrightarrow [n\to \infty ]{(d)} \nabla \mathbf{g}(\vec{\theta })^ T\, \mathbf{T}\qquad$\\

With multivariate Gaussians and Sample mean:\\

Let $\mathbf{T}_ n=\overline{\mathbf{X}}_ n$ where $\overline{\mathbf{X}}_ n$ is the sample average of $\, \mathbf{X}_1,\ldots ,\mathbf{X}_ n\, \stackrel{iid}{\sim }\mathbf{X},\,$ and $\, \vec{\theta }=\mathbb E[\mathbf{X}].\, \,$ The (multivariate) CLT then gives $\, \mathbf{T}\sim \mathcal{N}(\mathbf{0},\Sigma _\mathbf{X})$ where $\, \Sigma _\mathbf{X}\,$ is the covariance of $\mathbf{X}.\, \,$ In this case, we have:\\

$\sqrt{n} \left(\mathbf{g}(\mathbf{T}_ n) - \mathbf{g}(\vec{\theta }) \right) \xrightarrow [n\to \infty ]{(d)} \nabla \mathbf{g}(\vec{\theta })^ T\mathbf{T}\,$\\

$\nabla \mathbf{g}(\vec{\theta })^ T\mathbf{T} \sim \,  \mathcal{N}\left(0, \nabla \mathbf{g}(\vec{\theta })^ T \Sigma _{\mathbf{X}} \nabla \mathbf{g}(\vec{\theta })\right)\qquad$\\

$(\mathbf{T}\sim \mathcal{N}(\mathbf{0},\Sigma _\mathbf{X}))$\\


\subsection{Motivation}
\begin{definition}
	A first order ordinary differential equation is called \textit{separable} if it can be written in the form $y' = f\p{t}g\p{y}$.
\end{definition}
Separable equations appear relatively often in real life. One scenario is in radioactive decay of nuclei.
\par\bigskip
Suppose $N\p{t}$ is the amount of a nuclei at a time $t$. In our model, we assume that the amount of decay is proportional to the amount of nuclei $N$ and the length of time $\Delta t$. Then
\normalskip
\begin{align*}
	\Delta N &= N\p{t + \Delta t} - N\p{t} \approx -\lambda N\p{t} \Delta t \\
		  N' &= \lim_{\Delta t\to0} \frac{N\p{t + \Delta t} - N\p{t}}{\Delta t} = -\lambda N
\end{align*}
The equation can then be solved as follows:
\begin{align*}
	N' &= -\lambda N \\
	\frac{\diff{N}}{\diff{t}} &= -\lambda N \\
	\frac{\diff{N}}{N} &= -\lambda \, \diff{t}, \qquad N \neq 0\text{, but note that it is a solution to the ODE} \\
	\int \frac{\diff{N}}{N} &= \int -\lambda \, \diff{t} \\
	\ln\abs{N} &= -\lambda t + C_0 \\
	\abs{N} &= Ce^{-\lambda t}, \qquad C = e^{C_0} > 0 \\
	N &=
	\begin{cases}
		\phantom{-}Ce^{-\lambda t}, & N\p{t_0} > 0 \\
		-Ce^{-\lambda t}, & N\p{t_0} < 0
	\end{cases}
\end{align*}
If we allow $C$ to take on any real value, then the general solution to the ODE can be written as \[ N\p{t} = Ce^{-\lambda t}. \]
\clearpage
\begin{example}
	\noskip
	\begin{flalign*}
		y' &= t^2y^2 \\
		\frac{\diff{y}}{y^2} &= t^2 \, \diff{t} \\
		\int \frac{\diff{y}}{y^2} &= \int t^2 \, \diff{t} \\
		-\frac{1}{y} &= \frac{1}{3}t^3 + C \\
		y &= -\frac{1}{\frac{1}{3}t^3 + C}
	\end{flalign*}
	Note that $y\p{t} = 0$ is also a solution.
\end{example}

\subsection{``Proof''}
Suppose we have the separable ODE \[ y' = \frac{f\p{t}}{g\p{y}}. \] Then
\normalskip
\begin{align*}
	g\p{y} y' &= f\p{t} \\
	\int g\p{y} y' \, \diff{t} &= \int f\p{t} \, \diff{t} \\
	\int g\p{y} \, \diff{y} &= \int f\p{t} \, \diff{t}, \qquad y = y\p{t} \implies \diff{y} = \frac{\diff{y}}{\diff{t}} \, \diff{t}
\end{align*}
which is the same result from using separation of variables.

\subsection{Using Definite Integration (Newton's Law of Cooling)}
Newton's Law of Cooling states that the change in temperature of an object over time is directly proportional to the difference between the object's current temperature $T$ and the ambient temperature $A$. Mathematically, \[ \frac{\diff{T}}{\diff{t}} = -k\p{T - A}. \] Suppose we had the initial condition $T\p{t_0} = T_0$. Then we can solve the IVP using separation of variables:
\begin{align*}
	\frac{\diff{T}}{\diff{t}} &= -k\p{T - A} \\
	\frac{\diff{T}}{T - A} &= -k \, \diff{t}
\end{align*}
Instead of finding the general solution and then solving for the constant, we can use definite integration to get the particular solution directly. As time goes from $t_0$ to $t$, temperature goes from $T_0$ to $T\p{t}$. Thus,
\begin{align*}
	\int_{T_0}^{T\p{t} }\frac{\diff{u}}{u - A} &= \int_{t_0}^{t} -k \, \diff{v} \\
	\ln\abs{T - A} - \ln\abs{T_0 - A} &= -k\p{t - t_0} \\
	T\p{t} &= A + \p{T_0 - A}e^{-k\p{t - t_0}}
\end{align*}

\subsection{Implicitly Defined Solutions}
Sometimes, we are unable to find an explicit solution to an ODE. I.e., we are unable to get a solution in the form of $y\p{t} = f\p{t}$. Consider the separable differential equation \[ y' = \frac{g\p{t}}{f\p{y}} \] and that $f\p{y}$ and $g\p{t}$ have antiderivatives $F\p{y}$ and $G\p{t}$, respectively. Separation of variables yields
\begin{align*}
	\int f\p{y} \, \diff{y} &= \int g\p{t} \, \diff{t} + C \\
	F\p{y} &= G\p{t} + C \\
	y &= F^{-1}\p{G\p{t} + C}
\end{align*}
If we are unable to find $F^{-1}$, then we will not be able to find an explicit solution for the differential equation.
\par\pagebreak
\begin{example}
	\noskip
	\begin{align*}
		x' &= \frac{x^2}{\p{1 + x}t} \\
		\int \frac{1}{x^2} + \frac{1}{x} \, \diff{x} &= \int \frac{1}{t} \, \diff{t} \\
		-\frac{1}{x} + \ln\abs{x} &= \ln\abs{t} + C
	\end{align*}
	We are unable to solve for $x$, so we cannot find an explicitly defined solution.
\end{example}


\section{Geometric meaning of transpose}

Let's consider two vectors, $a$ and $b$ and an action $M$.
We apply the action $M$ to the first vector $a$ and
consider the scalar product $\langle Ma, b\rangle$.
What should be the equivalent action $N$ on $b$ that whould result
in exactly the same scalar product?
This action $N$ can be found from the equation
\[
\langle Ma, b \rangle = \langle a, Nb \rangle.
\]

We call the action $N$ as transpose of the action $M$.
Our definition explains immediately why it is called transpose:
the letter $M$ has changed its position.
This definition is consistent with classical one, as
\[
\langle Ma, b\rangle = (Ma)^T b = a^T (M^T b) = \langle a, M^T b \rangle
\]

Let's find some transposes using geometric approach.

Example 1.

Consider the action $R$ on $\mathbb{R}^2$ that rotates arbitrary vector $a$ clockwise by the angle $\alpha$.
What is the equivalent action on vector $b$? We can compensate the clockwise rotation $R$ of $a$
by a counterclockwise rotation of $b$. Under both rotations the lengths are preserved
and the angle between $a$ and $b$ is changed in the same way.

So the transpose of clockwise rotation is the counterclockwise rotation by the same angle.

Example 2.

Consider the action $U$ on $\mathbb{R}^3$ that reflects arbitrary vector $a$ across plane $\alpha$.
The equivalent action on the other vector $b$ would be this reflection $U$ itself!
Consider two scalar products, $\langle Ua, b \rangle$ and $\langle a, Ub \rangle$.
We see that vectors $a$ and $Ub$ are reflected vectors $Ua$ and $b$.
So the scalar products are equal.

The transpose of reflection is the same reflection, so $U^T=U$.

Example 3.

Consider the action $H$ on $\mathbb{R}^3$ that projects arbitrary vector $a$ on the plane $\alpha$.
Let $\hat a$ and $\hat b$ be the projections of vectors $a$ and $b$.
Scalar product is a linear function, and $\hat a$ is orthogonal to $b - \hat b$, so
\[
\langle Ha, b \rangle = \langle \hat a, \hat b + (b-\hat b) \rangle = \langle \hat a, \hat b \rangle
\]
By the same reasoning $\langle a, Hb \rangle = \langle \hat a, \hat b \rangle$.

And the transpose of projection is the same projection, $H^T=H$.

The equality $\langle Ha, b\rangle = \langle \hat a, \hat b \rangle$ holds only for orthogonal projection,
and not for the oblique one.
So $H^T \neq H$ for oblique projections.

The idea of a transpose may be also illustrated if one knows about singular value decomposition.
From singular value decomposition $A = U\Sigma V^T$ every action $A$ can be decomposed in rotation followed by scaling and
second rotation.
The transpose of rotation is inverse rotation. The transpose of scaling is the same scaling.
So the transpose of every action $A$ is the inverse of second rotation, the same scaling and inverse of first rotation.
It follows by the way that $\det A = \det A^T$ as rotations do not change volume and scaling is unaffected by transpose.

TODO: add pictures

TODO: Why $H^2=H$ and $H^T=H$ is sufficient for projection?

\url{https://math.stackexchange.com/questions/598258}


\subsection{The orthogonal projection}

in inner product space, orthogonal projection

\(p_uv = \dfrac{<u,v>}{{v,v}}v\)

we then know that \(o=v-p_uv\) is orthogonal to \(u\).

\subsection{Orthogonal set / Orthogonal basis}

if a set of vectors are all orthogonal, they form an orthogonal set
if the set spans the vector space, it is an orthogonal basis.

\subsection{The Gram-Schmidt process}

can we form an orthogonal basis from a non-orthogonal basis? yes, using gram schmidt

we have \(x_1\), \(x_2\) \(x_3\) etc
we want to make \(v_1\), \(v_2\) etc orthognal

\(v_1 = x_1\)
\(v_2 = x_2 - p_{x_2}v_1\)
\(v_3 = x_3 - p_{x_3}v_1 - p_{x_3}v_2v\)


\subsection{Softplus}

\subsubsection{The function}

\(a(z)=\ln (1+e^z)\)

\subsubsection{The derivative}

Its derivative is the sigmoid function:

\(a'(z)=\dfrac{1}{1+e^{-z}}\)

\subsubsection{Notes}

The softplus function is a smooth approximation of the ReLU function.

Unlike the ReLU function, Softplus does not induce sparcity.


\section{Calculus of Variations}
Example: shortest path between two points
$(x_1, y_1)$ and $(x_2, y-2)$.
Each segment is of length
\begin{align}
    ds &= \sqrt{dx^2 + dy^2}\\
    &= dx \sqrt{1 + {\left( \frac{dy}{dx} \right)}^2}
\end{align}
so the total length is
\begin{align}
    L &=
    \int ds\\
    &=
    \int_{x_1}^{x_2} dx\,
    \sqrt{1 + {\left( \frac{dy}{dx} \right)}^2}.
\end{align}
The question is:
How to find the $y(x)$ for which $L$ is smallest?

For any function $y(x)$ the integral will have some value,
so $L$ is a function of the function $y(x)$.
We say that $L$ is a ``functional'' of $y(x)$.
A functional is just a function of a function.

Let $y=y_0(x)$ be the function that minimizes $L$.
Let us consider infinitesimal deviations from the path
\begin{align}
    y(x) &= y_0(x) + \epsilon \eta(x)
\end{align}
for some arbitrary $y$ except that
$\eta(x_1) = \eta(x_2) = 0$.

By definition $y_0$ is the shortest path,
so any other path much be longer.
\begin{align}
    L\left( y_0 + \epsilon\eta(x) \right) > L(y_0)
\end{align}
for any $\eta(x)$.

Expanding out to first order in $\epsilon$,
we have
\begin{align}
    L\left( y_0 + \epsilon\eta \right)
    &=
    \int dx\,
    \sqrt{1 + {\left[ y_0'(x) + \epsilon\eta'(x) \right]}^2}\\
    &=
    \int dx\,
    \sqrt{%
        1 + y_0'^2 + 2\epsilon y_0' \eta'
    }
    + \mathcal{O}\left( \epsilon^2 \right)
\end{align}
Let's write this in a more convenient form
\begin{align}
    L\left( y_0 + \epsilon \eta \right) &=
    \int dx\, \sqrt{%
        1 + {\left( y_0' \right)}^2
    }
    {\left[
        1 + \frac{2\epsilon y_0' \eta'}{1 + y_0'^2}
    \right]}^{1/2}
\end{align}
Now we expand using the binomial theorem.
\begin{align}
    L\left( y_0 + \epsilon \eta \right) &=
    \underbrace{%
        \int dx\, \sqrt{%
            1 + y_0'^2
        }
    }_{L(y_0(x))}
    +
    \int dx\,
    \sqrt{1 + y_0'^2}\left[
        \frac{\epsilon y_0' \eta'}{1 + y_0'^2}
    \right]\\
    &=
    L[y_0(x)]
    +
    \int_{x_1}^{x_2} dx\,
    \frac{\epsilon y_0' \eta'}{\sqrt{1 + y_0'^2}}
\end{align}
So we have this thing depending on the derivative of $\eta$,
but it is arbitrary except at the end points.
So the trick is to use integration by parts,
and we know $\eta$ vanishes at the boundaries.
So let's integrate this by parts.
We're moving the derivative to the other terms,
and that leaves us with a boundary term that goes to zero when
$\eta(x_1)=\eta(x_2)=0$.
Once we integrate by parts,
the first term doesn't change
\begin{align}
    L\left( y_0 + \epsilon \eta \right) &=
    L[y_0(x)]
    -
    \epsilon
    \int_{x_1}^{x_2} dx\,
    \frac{d}{dx}\left[
    \frac{y_0'}{\sqrt{1 + y_0'^2}}
    \right]\eta
\end{align}
Now we have just evaluated this term
$L\left( y_0 + \epsilon \eta \right)$

Now what we are requiring is that
$L\left( y_0 + \epsilon\eta(x) \right) > L(y_0)$.
On the other hand,
we have that it is $L(y_0)$ plus some small quantity with an $\epsiilon$
in it.
And $\epsilon$ can be either positive or negative,
and if that integral was positive,
then I could choose $\epsilon$ to be negative,
and that inequality would be violated.
Conversely,
if the integral was negative,
then I could choose $\epsilon$ to be positive
and that inequality would again be violated.
So the only way for the inequality to hold is that this is stationary with
respect to $\epsilon$.
This means that the integral must be zero,
and the $\mathcal{O}\left( \epsilon^2 \right)$
terms are what gives you the extremising behaviour.

Recall in ordinary calculus,
if you have a function $y(x)$,
you can expand
\begin{align}
    y(x + \Delta x) &=
    y(x) + \Delta x \frac{dy}{dx}
    + \frac{1}{2}\left( \Delta x \right)^2 \frac{d^2y}{dx^2}
\end{align}
and the only way this could be a maximum is if
$\frac{dy}{dx}=0$.
Otherwise, I could always choose $\Delta x$ to be the right sign
to violate the fact that $x$ is the maximum.
That's why you must take the second derivative to determine whether it is a
maximum or a minimum.

Something similar is true here in the functional case.
The $\epsilon^2$ does not depend on the sign of $\epsilon$.
The only thing you can really be sure is that you have found an extremum.
The fact that it is a minimum comes from common sense.

What we're going to do is require that the integral is equal to zero.
\begin{align}
    L\left( y_0 + \epsilon \eta \right) &=
    L[y_0(x)]
    -
    \epsilon
    \underbrace{
        \int_{x_1}^{x_2} dx\,
        \frac{d}{dx}\left[
        \frac{y_0'}{\sqrt{1 + y_0'^2}}
        \right]\eta
    }_{=0}\\
    \int_{x_1}^{x_2} dx\,
    \frac{d}{dx}\left[
    \frac{y_0'}{\sqrt{1 + y_0'^2}}
    \right]\eta
    &= 0
\end{align}
Let me stop there now.
I'm not actually proving it's a minimum,
I'm just proving it's a stationary point.
To prove it's a minimum,
you need to show one more term.
You can do it,
but it will be more work,
which is not necessary to do.

It's kind of similar to functions.
The first derivative tells you if it's an extremum,
and to tell you if it's a maximum or minimum,
you need to evaluate one more derivative.

\begin{question}
    Technical question.
    Wouldn't it make more sens if you wrote the original inequality
    as a greater than or equal to sign?
\end{question}
The fact that I've dropped the $\epsilon^2$ terms.

\begin{question}
    Is it useful to say that you found an extremum but not necessarily a
    minimum.
\end{question}
In this question,
it's physical insight that tells us any extremum you find is a minimum.
In other cases it's not so obvious.
In this kind of problem,
once you've found your stationary point,
to prove it's a maximum or minimum,
you need to evaluate the second term.
It's a lot more work and we usually never do it.

There's this famous story that may be appropriate.
When they were designing the famous SR-71 blackbird,
they wanted to maximize the range.
They computed the stationary point for the configuration of the shape,
but they never bothered to compute the second derivative,
and it turned out to be the least efficient.
These theoreticians.
I'm not sure it's true though.
It's classified so we will never know.
Good thing about working in the defence industry.

Since the sign of $\epsilon$ is arbitrary,
the second term can be either positive or negative
and so
\begin{align}
    \int_{x_1}^{x_2}dx\,
    \frac{d}{dx}\left[
    \frac{y_0'}{\sqrt{1 + \left( y_0' \right)^2}}
    \eta(x)
    \right]
\end{align}
has to be true for any arbitrary function $\eta(x)$.
This is crucial.
And the only way for this to be true for any arbitrary function
$\eta(x)$
is if
\begin{align}
    \frac{d}{dx}\left[
    \frac{y_0'}{\sqrt{1 + \left( y_0' \right)^2}}
    \right] = 0
\end{align}
for every point on the path.

So here's the logic.
Suppose this was non-zero at some point.
Let's say at that point,
this thing was non-zero.
Then you could choose an $\eta(x)$
which is zero everywhere except that near point.
Then this would be non-zero.
So any point where this thing is non-zero,
just choose an $\eta(x)$ that is non-zero at that point,
then you'll make it non-zero.
You have so much freedom.
That's why the only way this integral is zero
is if this thing is zero at every single point.

So we have reduced the problem we started with
to solving a one-dimensional differential equation
\begin{align}
    \frac{d}{dx}\left[
    \frac{y_0'}{\sqrt{1 + \left( y_0' \right)^2}}
    \right] = 0
\end{align}
which we can integrate to get
\begin{align}
    \frac{y_0'}{\sqrt{1 + \left( y_0' \right)^2}}
    = C
\end{align}
for some constant $C$.
After some algebra, you get
\begin{align}
    y_0''^2 &=
    C(1 + y_0'^2)
\end{align}
and since $C$ is arbitrary,
we conclude
\begin{align}
    y_0'^2 &= \text{constant}\\
    \implies \qquad y_0' &= \text{constant} = m
\end{align}
which means
\begin{align}
    \frac{dy_0}{dx} = m
\end{align}
and so
\begin{align}
    y_0 = mx + c
\end{align}
where $m$ and $c$ are constants.
And that's just a straight line!
So you have convinced yourself that the shortest path between two points is a
straight line.

\section{Euler-Lagrange Equations}
Consider the integral
\begin{align}
    S &=
    \int_{x_1}^{x_2}
    f\left[
        y(x), y'(x), x
    \right]
    \,dx
\end{align}
and you're integrating that over $x$,
where $y(x)$ is a curve that connects
$(x_1,y_1)$ and $(x_2,y_2)$.
so that
$y(x_1)=y_1$ and $y(x_2)=y_2$.
In other words,
the end points are fixed.
The question is:
What is the function $y(x)$ that extremizes the functional $S$.

So $S$ is a functional and the question is,
what is the function $y(x)$ on which this is extremized,
subject the conditions of the values of $y(x)$ on the boundaries.
Very similar to the problem we just saw.

In the shortest path problem we just studied,
this $f$ was
\begin{align}
    f(x) &= \sqrt{1 + \left( y' \right)^2}
\end{align}
and it didn't depend on $y$ or $x$ explicitly.

Now we are studying the general case.
We did the special case with the shortest path problem,
now we are doing it with a general function $f$.

Now how do we extremize the problem?
We will just how to do it.

Let $y=y_0(x)$ correspond to the extremum.
Consider infinitesimal deformations
\begin{align}
    y(x) &= y_0(x) + \epsilon \eta(x)
\end{align}
where $\epsilon$ is an infinitesimal number
and $\eta(x)$ is an arbitrary function.

If $s$ is extremized for $y=y_0(x)$,
then $\frac{ds}{dz}=0$
at $\epsilon = 0$
for arbitrary $\eta(x)$.

So I'm going to call this thing
\begin{align}
    S(y_0 + \epsilon\eta) &=
    \int dx\, f\left(y + \epsilon\eta, y' + \epsilon\eta', x\right)
\end{align}
This is a crucial step.
This is $S$ for an arbitrary function $f$.
If I choose $y$ to be $y_0 + \epsilon\eta$,
then the above is true.

Now,
\begin{align}
    f(y + \epsilon\eta, y' + \epsilon\eta', x) &=
    f(y, y', \eta)
    + \epsilon \eta {\left( \frac{\partial f}{\partial y} \right)}_{y',x'}
    + \epsilon \eta' {\left( \frac{\partial f}{\partial y'} \right)}_{y, x}
\end{align}
Then
\begin{align}
    S[y_0 + \epsilon\eta] - S[y_0] &=
    \epsilon\int dx\, \left.
        \left[
        \eta \left( \frac{\partial f}{\partial y} \right)
        + \eta'\left( \frac{\partial f}{\partial y'} \right)
        \right]
    \right|_{y=y_0(x)}
\end{align}
Now just like what we did before,
we integrate by parts
and we end up with
\begin{align}
    S[y_0 + \epsilon\eta] - S[y_0] &=
    \epsilon \int dx\,
    \eta
    \left[
        \frac{\partial f}{\partial y}
        -
        \frac{d}{dx}\left(
            \frac{\partial f}{\partial y'}
        \right)
    \right]
    =0
\end{align}
Since $\eta$ is arbitrary,
it must be that at every point
\begin{align}
    \frac{\partial f}{\partial y}
    -
    \frac{d}{dx}\left(
        \frac{\partial f}{\partial y'}
    \right)
    = 0
\end{align}
and this equation has a name.
It is the famous \emph{Euler-Lagrange equation}.
This is one of the equations you are going to have to know.

This is the equation that is satisfied by the function $f$
that extremizes the functional $S[f]$ subject to the boundary conditions.

The steps we went through are actually exactly the same as that for the shortest
path straight line.
We're just following the problem more generally.
Previously we solved it when this $f(y,y',x) = \sqrt{1 + (y')62}$.
But now we used an arbitrary function and applied the stationary point
condition.

Any questions?
I want to stop here before I give examples.

\begin{example}[Shortest path]
    We want the minimize the length
    \begin{align}
        L &= \int_{x_1}^{x_2} dx\,
        \sqrt{1 + (y')^2}.
    \end{align}
\end{example}
\begin{proof}
    The function is
    \begin{align}
        f\left( y, y', x \right) &=
        \sqrt{1 + \left( y' \right)^2}.
    \end{align}
    Taking derivatives,
    \begin{align}
        \left( \frac{\partial f}{\partial y} \right)_{y', x} &= 0\\
        \left( \frac{\partial f}{\partial y} \right)_{y', x} &=
        \frac{y'}{\sqrt{1 + y'^2}}
    \end{align}
    Then using the Euler-Lagrange equation,
    \begin{align}
        \frac{d}{dx}\left\{
            \frac{y'}{\sqrt{1 + y'^2}}
        \right\}
        = 0
    \end{align}
    which is what we got before.
\end{proof}

\subsection{Brachistochrone problem}
I want to show the Brachistochrone problem,
which is the most famous problem in the calculus of variations.
It's famous because this is the problem that led to
the entire theory of the calculus of variations.
Bernoulli was trying the solve this problem and
he invented the calculus of variations for this problem.

Let's say I have a particle that slides down under gravity along some path
without friction.
Out of all possible paths,
with end points fixed,
what is the path that minimizes the time it takes to slide down?

This was a well known-problem 18th Century scientists were trying to solve.
Which is the time that it takes to get from A to B fastest.
Scientist were struggling to solve this problem,
and the way Bernoulli solved it was by inventing the calculus of variations.

Let's first write out the problem.

Given 2 points 1 and 2,
with 1 higher above the ground,
what shape should a track be built so that a particle released from
point 1 reach point 2 in the shortest time?

Let's make point 1 be the origin $(0, 0)$.
The total time is
\begin{align}
    T &=
    \int \frac{ds}{v}
\end{align}
where $v$ is the velocity.
You can work it out from the conservation of energy with
\begin{align}
    \frac{1}{2}mv^2 &= mgy\\
    v &= \sqrt{2 gy}
\end{align}
so the total time is
\begin{align}
    T &=
    \int
    \frac{dx\sqrt{1 + y'^2}}{\sqrt{2gy}},
\end{align}
which should be easy to put into the Euler-Lagrange equation.
The function is
\begin{align}
    f &=
    \sqrt{\frac{1 + y'^2}{2gy}}
\end{align}
which has derivatives
\begin{align}
    \frac{\partial f}{\partial y'} &=
    \frac{y'}{\sqrt{2gy(1 + y'^2)}}\\
    \frac{\partial f}{\partial y} &=
    -\frac{1}{2}
    \frac{\sqrt{1 + y'^2}}{\sqrt{2gy}}
\end{align}
so the Euler-Lagrange equation is
\begin{align}
    \frac{d}{dx}\left[
        \frac{y'}{\sqrt{y(1 + y'^2)}}
    \right]
    +
    \frac{\sqrt{1 + y'^2}}{2\sqrt{y^2}}
    &= 0
\end{align}
which simplifies to
\begin{align}
    \frac{y'^2}{1 + y'^2} =
    \underbrace{\text{constant}}_{-\frac{1}{2a}}\times y
\end{align}
and after some algebra,
you get
\begin{align}
    \frac{dy}{dx} &=
    \sqrt{\frac{2a - y}{y}}.
\end{align}
Then if you do a change of coordinates,
the solution is
\begin{align}
    x &= a(\theta - \sin\theta)\\
    y &= a ( 1 - \cos\theta)
\end{align}
which is called a \emph{cycloid}.


\section{Softmax Regression}
A softmax regression, also called a multiclass logistic regression, is
used to generalize logistic regression when there are more than 2 outcome
classes.
\subsection{Softmax Function}
The softmax function creates a probability distribution over a set of $k$
classes for a training example $x$, with $\theta_k$ denoting the set of
parameters to be optimzed for the $k$-th class.
\begin{equation}
  p(y=k | x; \theta) = \frac{\exp \left(\theta_k^T x \right)}{\sum_j \exp \left( \theta_j^T x \right)}
\end{equation}
\subsection{MLE and Cost Function}
We can write the maximum likelihood function for softmax regression as:
\begin{equation}
  L(\theta) = \prod_{i=1}^m \prod_k p(y=k|x; \theta)^{\mathbf{1}\{y_i=k\}}
\end{equation}
Where $\mathbf{1} \{y_i=k\}$ is the indicator function which is $1$ if its
argument is true, $0$ otherwise.
By taking the negative log likelihood:
\begin{equation}
  \begin{aligned}
    \ell ( \theta ) \quad & = \quad -\log L ( \theta ) \\
    & = \quad - \log \prod_{i=1}^m \prod_k p(y=k|x; \theta)^{\mathbf{1}\{y_i=k\}} \\
    & = \quad -\sum_{i=1}^m \sum_k \left( \mathbf{1}\{y_i=k\} \cdot \left( \theta^T_k x_i - \log \left( \sum_j \exp \left( \theta_j^T x_i \right) \right) \right) \right) \\
    & = \quad \sum_{i=1}^m -\theta^T_{y_i} x_i + \log \left( \sum_j \exp \left( \theta_j^T x_i \right) \right) \\
  \end{aligned}
\end{equation}
This is known as the cross-entropy loss function.
\subsection{Gradient Descent}
To perform gradient descent, we must take the derivative of our cost function,
but it is important to note that the derivative for the correct class is
different than the other classes.
\begin{equation}
  \begin{aligned}
    \nabla_{\theta_j} \ell(\theta) \quad & = \quad \nabla_{\theta_j} \left( \sum_{i=1}^m -\theta^T_{y_i} x_i + \log \left( \sum_k \exp \left( \theta_k^T x_i \right) \right) \right) \\
      & = \quad \sum_{i=1}^m \nabla_{\theta_j} \left( -\theta^T_{y_i} x_i \right)  +  \nabla_{\theta_j} \left( \log \left( \sum_k \exp \left( \theta_k^T x_i \right) \right) \right) \\
      & = \quad \sum_{i=1}^m \mathbf{1}\{y_i=j\} \cdot (-x_i) + \frac{\exp(\theta_j^T x_i)}{\sum_k \exp(\theta_k^T x_i)} \cdot x_i \\
      & = \quad \sum_{i=1}^m \left(\frac{\exp(\theta_j^T x_i)}{\sum_k \exp(\theta_k^T x_i)} -  \mathbf{1}\{y_i=j\} \right) \cdot x_i \\
  \end{aligned}
\end{equation}
And our update equation for the $j$-th parameter weights is:
\begin{equation}
  \theta_{j} : = \theta_{j} - \alpha \left( \frac{\exp(\theta_j^T x_i)}{\sum_k \exp(\theta_k^T x_i)} -  \mathbf{1}\{y_i=j\} \right) \cdot x_i
\end{equation}
Note that since each class has a set of weights, our gradient is a matrix known
as the jacobian $\mathbf{J}$, with $k$ classes each with $n$ feature weights.
\begin{equation}
  \mathbf{J}_{\theta} =
    \left[
      \begin{array} { c c c } {\frac { \partial \ell(\theta)  } { \partial \theta _ { 1 } } } & { \cdots } & { \frac { \partial \ell(\theta) } { \partial \theta _ { k } } } \end{array}
    \right] = \left[
      \begin{array} { c c c } {\frac { \partial \ell(\theta) } { \partial \theta _ { 11 } } } & { \cdots } & { \frac { \partial \ell(\theta) } { \partial \theta _ { k1} } } \\ { \vdots } & { \ddots } & { \vdots } \\ { \frac { \partial \ell(\theta) } { \partial \theta _ { 1n } } } & { \cdots } & { \frac { \partial \ell(\theta) } { \partial \theta _ { kn } } } \end{array}
    \right]
\end{equation}


\lab{Algorithms}{Givens rotations}{Givens rotations}
\objective{Use orthogonal transformations to perform QR decomposition.}

The matrix $\begin{pmatrix} cos(\theta) & -sin(\theta) \\ sin(\theta) & cos(\theta) \end{pmatrix}$ rotates a vector counterclockwise by $\theta$. Given a vector $x = \begin{pmatrix} a \\ b \end{pmatrix}$, we can rotate $x$ into the range of $e_1$ by choosing the correct $\theta$. The problem is equivalent to solving for $c = cos(\theta)$ and $s = sin(\theta)$ in the system
\begin{equation}
\label{eq:Givens rotation system}
\begin{pmatrix} c & -s \\ s & c \end{pmatrix} \begin{pmatrix}a\\b\end{pmatrix}
= \begin{pmatrix}r\\0\end{pmatrix}
\end{equation}
 In fact, it's not necessary to compute $\theta$; we solve for $c$ and $s$ directly. An obvious solution is $c = \frac{a}{\sqrt{a^2 + b^2}}$ and $s = \frac{-b}{\sqrt{a^2+b^2}}$.

\section*{Givens triangularization}

Like Householder, the Givens $QR$ algorithm, \emph{triangularizes} a matrix $A$ using \emph{orthogonal transformations}. The Householder $QR$ algorithm worked one column at a time; Givens works one element at a time. Each nonzero below the main diagonal can be zeroed out with a Givens rotation. If we solve for $c$ and $s$ as above, the matrix
\begin{equation*}
G(i,j,c,s) =
\begin{pmatrix}
1   & \cdots & 0 & \cdots & 0 & \cdots & 0 \\
 \vdots & \ddots & \vdots &  & \vdots & & \vdots \\
0   & \cdots &    c   & \cdots &    -s   & \cdots &    0   \\
 \vdots &        & \vdots & \ddots & \vdots &        & \vdots \\
 0   & \cdots &   s   & \cdots &    c   & \cdots &    0   \\
\vdots &        & \vdots &        & \vdots & \ddots & \vdots \\
0   & \cdots &    0   & \cdots &    0   & \cdots &    1
\end{pmatrix}
\end{equation*}
where
\[ \begin{array}{ll}
g_{i\, i} = c  & g_{j\, i}= -s   \\
g_{j\, j} = c &  g_{i\, j}= s  \\
\end{array}\]
rotates by $\theta$ in the $i,j$ plane. Also, it's easy to check that $G(i,j,c,s)$ is orthogonal. Left-multiplying $A$ by $G(i,j,c,s)$ zeroes out $A_{i\, j}$. For example,

\[
\begin{array}{ccccccc}
\begin{pmatrix}
*&*&*\\
*&*&*\\
*&*&*
\end{pmatrix}
&
\underrightarrow{G(3,1)}
&\begin{pmatrix}
*&*&*\\
*&*&*\\
0&*&*
\end{pmatrix}
&
\underrightarrow{G(2,1)}
&\begin{pmatrix}
*&*&*\\
0&*&*\\
0&*&*
\end{pmatrix}
&
\underrightarrow{G(3,2)}
&\begin{pmatrix}
*&*&*\\
0&*&*\\
0&0&*
\end{pmatrix}
\end{array}
\]

The Givens QR method is often slower than Householder, because it works one element at a time. But it is faster than Householder when $A$ is sparse, and it is very parallelizable.

\begin{problem}
Write a script that uses Givens rotations to do $QR$ decomposition. By performing successive Givens rotations, triangularize $A$ to find $R$. You can find $Q$ using the chain of rotations.

To find each rotation, you will have to solve \eqref{eq:Givens rotation system} for $c$ and $s$. Let $b$ be the element you want to zero out, say $A_{i\,j}$. Then $a$ is in the same \emph{column} as $b$, and in the \emph{row} you want to rotate into (it's like we're squishing the whole vector into $a$'s spot.) You can choose $a$ to be the diagonal element above $b$.
\end{problem}

\begin{problem}
Compare the MGS, Householder, and Givens algorithms for $QR$ decomposition on various matrices. Try different sizes and different levels of sparsity. Which is the fastest? Which is the most stable?
\end{problem}


\section{Minkowski sum and difference}

Given two points $\point{p_1} = (x_1, y_1, z_1)$ and $\point{p_2} = (x_2, y_2, z_2)$ we define
their sum as usual as the point
$\point{p_1} + \point{p_2} = (x_1+x_2, y_1+y_2, z_1+z_2)$,
and their difference as the point
$\point{p_1} - \point{p_2} = (x_1-x_2, y_1-y_2, z_1-z_2)$.
Given two polytopes $P_1$ and $P_2$, the Minkowski sum is a polytope $P_1 \oplus P_2$
defined as
$$
P_1 \oplus  P_2 = \{ \point{p_1} + \point{p_2} \ \vert\ \point{p_1} \in P_1, \point{p_2} \in P_2 \}.
$$
Similarly, the Minkowski difference is defined as
$$
P_1 \ominus P_2 = \{ \point{p_1} - \point{p_2} \ \vert\ \point{p_1} \in P_1, \point{p_2} \in P_2 \}.
$$
An obvious property of the Minkowski difference is the following.
\begin{property}
\label{pro:mink}
Two polytopes $P_1$,$P_2$
are intersecting if and only if
$\point{0} \in (P_1 \ominus P_2).$
\end{property}

Consider now translating $P_1$ and $P_2$ by
respectively vectors $\point{v_1}$ and $\point{v_2}$.
We have that $\point{v_1}+P_1$ and $\point{v_2}+P_2$
intersect if and only if
$\point{0} \in ((\point{v_1}+P_1) \ominus (\point{v_2}+P_2))$.
The following chain of relations hold
\begin{eqnarray}
\nonumber
& & \point{0} \in ((\point{v_1}+P_1) \ominus (\point{v_2}+P_2)) \\
\nonumber
& \Leftrightarrow & \point{0} \in \{ (\point{p_1}+\point{v_1}) - (\point{p_2}+\point{v_2}) \ \vert\ (\point{p_1} + \point{v_1}) \in P_1, (\point{p_2} + \point{v_2}) \in P_2 \} \\
\nonumber
& \Leftrightarrow & -\point{v_1} \in \{ \point{p_1} - (\point{p_2}+\point{v_2}) \ \vert\ \point{p_1} \in P_1, (\point{p_2} + \point{v_2}) \in P_2 \} \\
\nonumber
& \Leftrightarrow & (\point{v_2} - \point{v_1}) \in \{ \point{p_1} - \point{p_2} \ \vert\ \point{p_1} \in P_1, \point{p_2} \in P_2 \} \\
\label{eq:chain}
& \Leftrightarrow & (\point{v_2} - \point{v_1}) \in (P_1 \ominus P_2)
\end{eqnarray}
We interpret (\ref{eq:chain}) as follows: take two vectors
$\point{v_1}$, $\point{v_2}$ for translating
$P_1$ and $P_2$ respectively. The two translated polytopes
intersect if and only if the difference
vector is contained in the Minkowski difference.


\textbf{Let $f(x)=3/(5-4\cos(x))$. Compute its Fourier approximation for $N = 20 : 10 : 500$ and plot the error using \textsl{loglog}. What is the largest strip around the real axis for which $f$ can be extended to an analytic function? How does this relate to the error bound for analytic functions derived in class?}
\newline

In this case the function $f$ is $2\pi$-periodic, as we can see in figure 1, and analytic inside the strip $|\Im(z)|\leq a$. To determine the value of $a$, we solve
\begin{align*}
5-4\cos(z)=0 \Rightarrow z=\cos^{-1}(5/4)\approx\ln(2)i.
\end{align*}
Hence, $a\approx \ln(2)$.
\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=1]{P5_1.eps}}
\caption{Graph of the function to approximate.}
\end{figure}

Given that the function is $2\pi$-periodic and analytic inside the strip $|\Im(z)|\leq a$, the error is bounded as follows
\begin{align*}
\left|\mathcal{F}[u](x)-\mathcal{F}_N[u](x)\right|\leq \frac{4\pi M}{a}e^{-aN} = \frac{4\pi M}{\ln(2)}e^{-\ln(2)N} = \frac{4\pi M}{\ln(2)}2^{-N} \propto 2^{-N},
\end{align*}
where $M = \max_{x\in[0,2\pi]}|f(x-ai)|$. We can check in the last figure how this convergence is much faster. The points that separate from the error bound are due to round-off errors.
\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=1]{P5_2.eps}}
\caption{Approximation error and its bound.}
\end{figure}

\subsection*{Matlab code for this problem}\begin{verbatim}
%% Problem 5
clear variables; close all; clc
figformat='epsc';

% Plot function

x0 = -2*pi;
xf = 2*pi;
x = chebfun('x',[x0 xf]);
f = 3./(5-4*cos(x));

figure
plot(f,'b-','linewidth',2)
hold on
plot([0 0],[0 3.5],'r--')
grid on
xlabel('$x$','interpreter','latex')
ylabel('$f(x)=\frac{3}{5-4\cos(x)}$','interpreter','latex')
set(gca,'fontsize',14)
set(gca,'XTick',x0:pi/2:xf)
xticklabels({'-2\pi','-3\pi/2','-\pi','-\pi/2','0','\pi/2','\pi','3\pi/2','2\pi'})
txt='Latex/FIGURES/P5_1';
saveas(gcf,txt,figformat)

%%
% Approximation and error
N = 20:10:500;
x0 = 0;
xf = 2*pi;
x = chebfun('x',[x0 xf]);
f = 3./(5-4*cos(x));

parfor j = 1:length(N)
    A = exp(1i*x*(-N(j):N(j)));
    lambda = 1/(2*pi)*A'*f;
    fn = A*lambda;
    err(j) = norm(f-fn,Inf);
end
%%
figure
loglog(N,err,'*','MarkerSize',12)
hold on
loglog(N,2.^(-abs(N)),'r','linewidth',2)
grid on
axis([10 500 1e-16 1e-1])
xlabel('$N$','interpreter','latex')
ylabel('Error','interpreter','latex')
legend('Approximation Error','Error Bound')
set(gca,'fontsize',14)
txt='Latex/FIGURES/P5_2';
saveas(gcf,txt,figformat)
\end{verbatim}


\section{Logistic Regression}

Logistic Regression is a generalized linear model suitable for fitting probabilities, i.e. dependent variables that vary $y\in (0,1)$.

\subsection{Binary Response Case}
Assume that the dependent variable is a scalar and not a vector (i.e. only a single probability is being predicted). Then, assume that the dependence of that probability on the exogenous variable $\mathbf{x}$ and the parameters of the model $\mathbf{b}$ is given by:

\begin{equation}
p(\mathbf{x}) = \frac{1}{1+\exp(\mathbf{b\cdot x})}
\end{equation}

This is, of course, the sigmoid function, which also shows up as the Fermi-Dirac distribution. It creates a symmetric mapping from the interval $(-\infty,\infty)$ to $(0,1)$. It amounts to treating the log-odds as a linear function of the coefficients:

\begin{equation}
\ln\left(\frac{p(\mathbf{x})}{1-p(\mathbf{x})}\right) = \mathbf{b \cdot x}
\end{equation}

Equivalently:

\begin{equation}
\frac{p(\mathbf{x})}{1-p(\mathbf{x})} = \prod_j e^{b_j x_j}
\end{equation}

So the model assumes that different values of the independent variables will have a multiplicative effect on the likelihood ratio of the two binary outcomes. There is a deep reason for this choice of mapping, which is why it shows up as the Fermi-Dirac distribution in physics. In physics, it emerges from looking at the probability of energy levels being occupied by fermions. This is done by considering the number of equally possible configurations that correspond to a set of probabilities of a certain energy level being occupied. The Fermi-Dirac distribution is the distribution that maximizes the number of equally possible configurations, and so I suppose the sigmoid function has some maximum entropy property, which makes it the "maximally naive guess".

Assuming that the data is generated by independent trials, the likelihood of observing the data is given by multiplying the probability of all the datapoints together:

\begin{equation}
\mathscr{L}(\mathrm{data}|p(\mathbf{x})) = \prod_{y_i=1} p(\mathbf{x_i}) \prod_{y_i=0} (1-p(\mathbf{x_i}))
\end{equation}

Strictly speaking, I think there should be a factor of $1/N!$ here to account for the probability of seeing a specific permutation of the individual data points (which are distinguishable), but that factor will be independent of $p(\mathbf{x})$ and so it will not matter to estimation. In the literature I haven't seen a factor show up.

The expression $-2 \ln(\mathscr{L})$ is called the \textit{deviance}, which is the analog to residual sum of squares in ordinary linear regression. In OLS, the log likelihood is: $\ln\mathscr{L} = -\sum|\hat{y}-y|^2 + ...$, while the deviance is $-2\ln\mathscr{L} = -\sum_i \ln\left[\left(p(\mathbf{x_i})^{y_i}(1-p(\mathbf{x_i}))^{y_i-1}\right)^2\right]$.

Logistic regression is then fit by finding the coefficients that minimize the deviance / maximize the likelihood.

Question for another day: in OLS, there is an assumption that the data will deviate from the mean, where the probability of observing a certain deviation is given by a normal distribution. This has the consequence, for example, that outliers have an outsize impact on the fitted mean, because the normal distribution otherwise models them as exceedingly unlikely. If the normal distribution is the error distribution for OLS, what is the error distribution for logistic regression? I believe it looks like the Fermi-Dirac Distribution, where the Fermi-Energy is the 50\% decision boundary.

\subsection{Pseudo-$R^2$}
The deviance can be used to create an error metric analogous to $R^2$ for OLS, named Pseudo-$R^2$. It compares the deviance to the deviance of simply assuming the average probability based on the class balance.

\begin{equation}
\mathrm{pseudo}-R^2 = 1 - \frac{\mathrm{deviance}}{\mathrm{deviance\ when\ }p=\mathrm{\ average\ class}}
\end{equation}


\section{Gradient descent}
Gradient descent is a first-order optimization algorithm for finding the minimum of a function \cite{??}. It works iteratively by taking steps proportional to the gradient of the function evaluated at each iteration.\\

For a differentiable function $F(x)$, the iterative scheme of gradient ascent is:
\begin{equation}
\bfx^{(t+1)} = \bfx^{(t)} + \rho^{(t)} \nabla F(\bfx^{(t)})
\end{equation}
At each iteration, the gradient is re-evaluated and a step is performed in the direction of the gradient $\nabla F$. The step size is controlled by $\rho^{(t)}$, a parameter called the learning rate, which can be adjusted at each iteration $t$. \\
With certain assumptions on the function $F$ and particular choices of $\rho$ \cite{??}, gradient descent is guaranteed to converge to a local minimum. Hence, if $f$ is not convex, the algorithm is sensible to the initialisation $\bfx^0$.  \\

Gradient descent is appealing because of its simplicity, but in practice other methods are more efficient, including conjugate gradients and quasi-Newton methods \cite{Gill et al., 1981; Fletcher, 1987; Nocedal and Wright, 1999}. Additionally, Standard gradient ascent becomes prohibitive slow with large data sets \cite{}, because of the computational cost (both in terms of time and memory) associated with the calculation of gradients at each iteration ~\citep{Robbins-Monro1951, Bottou2011-by, Spall2003-fl} .

\section{Stochastic gradient descent}
Assuming  there is redundancy in the data set, an approximated yet fast gradient can be calculated using a random subset of the data (i.e. a minibatch). Under large N conditions, the stochastic version can converge faster than traditional gradient descent \cite{??}. If the cost function is convex, stochastic gradient descent is also guaranteed to find a global minimum. \\

As in standard, gradient descent, the iterative training schedule proceeds by taking small steps of size $\rho$ in the direction of the approximate gradient $\hat{\nabla}F$:
\begin{equation}
\bfx^{(t+1)} = \bfx^{(t)} + \rho^{(t)} \hat{\nabla} F(\bfx^{(t)})
\end{equation}
The step size $\rho^{(t)}$ can also be adjusted at each iteration $t$. When this series satisfies the Robbins-Monro conditions: $\sum_t \rho^{(t)} = \infty \text{ and } \sum_t (\rho^{(t)})^2 < \infty$, $f$ is guaranteed to converge to a local minimum ~\citep{Robbins-Monro1951}.

The computation of $\hat{\nabla}F$ requires an additional hyperparameter that establishes the size of the minibatch. A trade off exists between getting closer to the true gradient by increasing the size of the minibatch at the cost of speed and smoother convergence.


\subsection{Surface integrals}
A surface integral is like adding up all the wind on a sail.
In other words, we want to compute
$$\int\!\!\!\int{\bf F\cdot n}\,dA$$
where ${\bf F\cdot n}$ is the amount of wind normal to a tiny parallelogram $dA$.
The integral sums over the entire area of the sail.
Let $S$ be the surface of the sail parameterized by $x$ and $y$.
(In this model, the $z$ direction points downwind.)
By the properties of the cross product we have the following for the unit normal $\bf n$
and for $dA$.
$${\bf n}=\frac{{\frac{\partial S}{\partial x}\times\frac{\partial S}{\partial y}}}
{{\left|\frac{\partial S}{\partial x}\times\frac{\partial S}{\partial y}\right|}}\qquad
dA=\left|\frac{\partial S}{\partial x}\times\frac{\partial S}{\partial y}\right|\,dx\,dy$$
Hence
$$\int\!\!\!\int{\bf F\cdot n}\,dA=\int\!\!\!\int{\bf F}\cdot
\left({\frac{\partial S}{\partial x}\times\frac{\partial S}{\partial y}}\right)\,dx\,dy$$

\bigskip
\noindent
The following exercise is from
{\it Advanced Calculus} by Wilfred Kaplan, p.~313.
Evaluate the surface integral
$$\int\!\!\!\int_S{\bf F\cdot n}\,d\sigma$$

\noindent
where ${\bf F}=xy^2z{\bf i}-2x^3{\bf j}+yz^2{\bf k}$, $S$ is the surface
$z=1-x^2-y^2$, $x^2+y^2\le1$ and $\bf n$ is upper.

\bigskip
\noindent
Note that the surface intersects the $xy$ plane in a circle.
By the right hand rule, crossing $x$ into $y$ yields $\bf n$ pointing upwards hence
$${\bf n}\,d\sigma=\left({\frac{\partial S}{\partial x}\times\frac{\partial S}{\partial y}}\right)\,dx\,dy$$

\noindent
The following Eigenmath code computes the surface integral.
The symbols $f$ and $h$ are used as temporary variables.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
M1 = ((0,0,0),(0,0,-1),(0,1,0))
M2 = ((0,0,1),(0,0,0),(-1,0,0))
M3 = ((0,-1,0),(1,0,0),(0,0,0))
M = (M1,M2,M3)
cross(u,v) = dot(u,M,v)
z = 1 - x^2 - y^2
F = (x y^2 z,-2 x^3,y z^2)
S = (x,y,z)
f = dot(F,cross(d(S,x),d(S,y)))
h = sqrt(1 - x^2)
defint(f,y,-h,h,x,-1,1)
\end{Verbatim}

\noindent
$\displaystyle \tfrac{1}{48}\pi$


\subsubsection{Defective Matrix}
If $A$ has an eigenvalue $\lambda$ with multiplicity $k$ that does not generate $k$ corresponding linearly independent eigenvectors, the the matrix $A$ is defective.
To generate enough vectors, we need to extend eigenvectors.
\begin{definition}
	Let $A$ be a square matrix.
	A nonzero vector $\vec{v}$ satisfying
	\begin{equation*}
		(A-\lambda I)^n \vec{v} = \vec{0}
	\end{equation*}
	for some eigenvalue $\lambda$ and some positive integer $n$ is a generalized eigenvector or rank $n$.
\end{definition}
\noindent
These generalized eigenvectors also make up solutions.
\begin{theorem}
	Let $\vec{v}$ be an generalized eigenvector of a square matrix $A$ corresponding to an eigenvalue $\lambda$ with multiplicity $k$.
	Then
	\begin{equation*}
		e^{At}\vec{v} = e^{\lambda t}\left(\vec{v} + t(A-\lambda I)\vec{v} + \frac{t^2}{2!}(A-\lambda I)^2\vec{v} + \ldots\right)
	\end{equation*}
	is the corresponding fundamental solution.
	Further, the above sequence will terminate after $k$ or fewer terms (all subsequent terms are 0).
\end{theorem}

\begin{example}
	Find the general solution to the system
	\begin{equation*}
		\vec{x}' = \begin{bmatrix}
			1 & 0 & 0 \\
			1 & 3 & 0 \\
			0 & 1 & 1
		\end{bmatrix} \vec{x}.
	\end{equation*}
\end{example}
\noindent
Since $A$ is diagonal, the eigenvalues are simply the diagonal entries
\begin{equation*}
	\lambda = 3, 1 \text{(double root)}.
\end{equation*}
Finding the eigenvector for $\lambda  = 3$,
\begin{equation*}
	(A - 3I)\vec{v} = \vec{0} \implies \vec{v} = C_1 \begin{bmatrix}
		0 \\
		2 \\
		1
	\end{bmatrix}.
\end{equation*}
Finding the eigenvector for $\lambda = 1$,
\begin{equation*}
	(A - I)\vec{v} = \vec{0} \implies \vec{v} = C_2 \begin{bmatrix}
		0 \\
		0 \\
		1
	\end{bmatrix}.
\end{equation*}
Since $\lambda  =1$ had multiplicity 2, we need to find a generalized eigenvector.
\begin{equation*}
	(A - I)^2\vec{v} = \vec{0} \implies \vec{v} = C_2 \begin{bmatrix}
		0 \\
		0 \\
		1
	\end{bmatrix} + C_3 \begin{bmatrix}
		-2 \\
		1 \\
		0
	\end{bmatrix}.
\end{equation*}
Note how then eigenvector showed up again as a generalized eigenvector of rank 2.
This is because the eigenvector makes $(A - I)\vec{v} = 0$, so $(A - I)^2\vec{v} = (A - I)\vec{0} = \vec{0}$.
So, our solution is
\begin{align*}
	\vec{x} &= C_1e^{3t}\begin{bmatrix}
		0 \\
		2 \\
		1
	\end{bmatrix} + C_2e^{t}\begin{bmatrix}
		0 \\
		0 \\
		1
	\end{bmatrix} + C_3e^{t}\left(\begin{bmatrix}
		-2 \\
		1 \\
		0
	\end{bmatrix} + (A - I)t\begin{bmatrix}
		-2 \\
		1 \\
		0
	\end{bmatrix}\right) \\
	&= C_1e^{3t}\begin{bmatrix}
		0 \\
		2 \\
		1
	\end{bmatrix} + C_2e^{t}\begin{bmatrix}
		0 \\
		0 \\
		1
	\end{bmatrix} + C_3e^{t}\begin{bmatrix}
		-2 \\
		1 \\
		t
	\end{bmatrix}.
\end{align*}
We can rewrite this a bit more cleanly as
\begin{equation*}
	\vec{x} = \begin{bmatrix}
		0 & 0 & -2e^t \\
		2e^{2t} & 0 & e^t \\
		e^{3t} & e^t & te^t
	\end{bmatrix} \begin{bmatrix}
		C_1 \\
		C_2 \\
		C_3
	\end{bmatrix}.
\end{equation*}
This square matrix is called the fundamental matrix of the system.


\subsection{Dynamical Systems and Eigenvectors}

In general, a discrete dynamical system can be modeled as:

\[\boxed{\tb{x}(t+1)=A\tb{x}(t)}\]

where the transformation undergone by the system is $\tb{x}(t)\rightarrow\tb{x}(t+1)$ with matrix $A$.
Additionally, note that $\tb{x}(t)=\begin{bmatrix}c(t)\\r(t)\end{bmatrix}$ where $c(t)$ and $r(t)$ are some closed formulas.

Finding $\tb{x}(t)$ for an arbitrary integer $t>0$:

\[\boxed{\tb{x}(t)=A^t\tb{x}(0)=A^t\tb{x}_0}\]

Repeat definition of eigenvector from below:

\begin{framed}
    If $A$ is an $n\times n$ matrix, an eigenvector of $A$ is a nonzero vector $\tb{v}$
    that has the property that $\tb{v}$ and $A\tb{v}$ are parallel.
    Same as saying that $A\tb{v}=\lambda\tb{v}$, so $\lambda$ is an eigenvalue.
\end{framed}

\subsection{Dynamical Systems Example}

Following equations model transformation from $t$ to $t+1$:

\[\boxed{
    \begin{array}{l}
        c(t+1)=0.86 c(t)+0.08 r(t) \\
        r(t+1)=-0.12 c(t)+1.14 r(t)
        \end{array}
}\]

Is discrete dynamical linear system: changed over discrete time interval and dynamic as
variables change according to $t$. As a matrix-vector equation:

\[\left(\begin{array}{cc}
    0.86 & 0.08 \\
    -0.12 & 1.14
    \end{array}\right)\left(\begin{array}{l}
    c(t) \\
    r(t)
    \end{array}\right)=\left(\begin{array}{l}
    c(t+1) \\
    r(t+1)
    \end{array}\right)\]

$\tb{x}(t)=\left(\begin{array}{l}c(t) \\r(t)\end{array}\right)$
is the \textbf{state vector} at time $t$. $\tb{x}(0)=\left(\begin{array}{l}c_{0} \\r_{0}\end{array}\right)$
is the \textbf{initial state vector}. Calculating arbitrary state vector:

\[\left(\begin{array}{cc}
    0.86 & 0.08 \\
    -0.12 & 1.14
    \end{array}\right)^{t} \tb{x}_{0}=\tb{x}(t)\]

In this example, $c(t)=(100) 1.1^{t}$ and $r(t)=(300) 1.1^{t}$, so next
state vector is 1.1 times the current. However, for $\tb{x}_{0}=\left(\begin{array}{c}1000 \\1000\end{array}\right)$
there exists no such scalar pattern. Can use the basis of 2 (scalar pattern respected) vectors:

\[\mathfrak{B}=\left\{\left(\begin{array}{l}
    100 \\
    300
    \end{array}\right),\left(\begin{array}{l}
    200 \\
    100
    \end{array}\right)\right\}\]

Writing this state vector as a lin. combination:

\[\left(\begin{array}{l}
    1000 \\
    1000
    \end{array}\right)=2\left(\begin{array}{l}
    100 \\
    300
    \end{array}\right)+4\left(\begin{array}{l}
    200 \\
    100
    \end{array}\right)\]

After applying coefficient matrix to both sides and simplifying:

\[\left(\begin{array}{cc}
    0.86 & 0.08 \\
    -0.12 & 1.14
    \end{array}\right)^{t}\left(\begin{array}{c}
    1000 \\
    1000
    \end{array}\right)=2(1.1)^{t}\left(\begin{array}{c}
    100 \\
    300
    \end{array}\right)+4(0.9)^{t}\left(\begin{array}{c}
    200 \\
    100
    \end{array}\right)\]

Thus,

\[\begin{array}{l}
    c(t)=200(1.1)^{t}+800(0.9)^{t} \\
    r(t)=600(1.1)^{t}+400(0.9)^{t}
    \end{array}\]

Different trajectories for various initial state vectors:

\begin{center}
    \includegraphics[scale=0.5]{state-vectors.png}
\end{center}

Called a \textbf{phase portrait} for a discrete dynamical system.
Indicates performing of system based on initial states.
The 2 state vectors in the basis are \textbf{eigenvectors}.

\begin{framed}
    If $A$ is an $n\times n$ matrix, an eigenvector of $A$ is a nonzero vector $\tb{v}$
    that has the property that $\tb{v}$ and $A\tb{v}$ are parallel.
    Same as saying that $A\tb{v}=\lambda\tb{v}$, so $\lambda$ is an eigenvalue.
\end{framed}

If there exists an $n\times n$ matrix $A$ with $\lambda=0$, then kernel of $A$ must be nontrivial
because $A\tb{v}=\tb{0}=0\tb{v}$, therefore $A$ is singular.


\section{Inequalities for Probabilities}

\subsection{Markov's Inequality}

For any $t>0$:
\begin{equation}
\mathbb{P}(X > t) \leq \frac{\mathbb{E}(X)}{t}
\end{equation}

This is a general result, independent of the distribution of $X$.

\subsection{Chebyshev's Inequality}

\begin{equation}
\mathbb{P}(|X-\mu| \geq t) \leq \frac{\sigma^2}{t^2}
\end{equation}

and

\begin{equation}
\mathbb{P}(|Z| \geq k) \leq \frac{1}{k^2}
\end{equation}

where $Z = (X-\mu)/\sigma$. This is a general result that can be proven using Markov's inequality.

\subsection{Hoeffding's Inequality}
Let $Y_1,...,Y_n$ be independent observations so that $\mathbb{E}Y_i = 0$ and $a_i \leq Y_i \leq b_i$. Then, for any $t>0$,

\begin{equation}
\mathbb{P}\left( \sum_{i=1}^n Y_i \geq \epsilon \right) \leq e^{-t\epsilon}\prod_{i=1}^n e^{t^2 (b_i - a_i)^2/8}
\end{equation}

Hoeffding's inequality provides a sharper bound than Markov's inequality and relies on bounds of the random variables rather than their variance.

\subsubsection{Example: Bernoulli Random Variables}
Let $X_1,...,X_n \sim \mathrm{Bernoulli}(p)$. Assume that the goal is to measure the parameter $p$, which is estimated by the sample mean. This could for example be the error rate of a binary classifier. For the Bernoulli Random variable, $0\leq X_1 \leq 1$, so that the probability that the estimate of the error rate is off by more than $\epsilon$ is given by:

\begin{equation}
\mathbb{P}(|\overline(X)_n - p| > \epsilon) \leq 2e^{-2n\epsilon^2}
\end{equation}

\subsection{Mill's Inequality}
Mill's inequality is useful for normal random variables. Let $Z \sim \mathscr{N}(0,1)$, then the probability that $|Z|$ will be greater than some value $t$ is bounded by:

\begin{equation}
\mathbb{P}(|Z|>t) \leq \frac{2}{\pi}\frac{e^{-t^2/2}}{t}
\end{equation}

\section{Inequalities for Expectations}

\subsection{Cauchy-Schwarz Inequality}

\begin{equation}
\mathbb{E}|XY|\leq \sqrt{\mathbb{E}(X^2) \mathbb{E}(Y^2)}
\end{equation}

\subsection{Jensen's Inequality}

Jensen's inequality provides bounds for expectations for non-linear functions of a random variable. Given a random variable $X$ and a function $g(X)$, if $g(X)$ is convex:

\begin{equation}
\mathbb{E}g(X) \geq g(\mathbb{E}X)
\end{equation}

If $g(X)$ is concave:

\begin{equation}
\mathbb{E}g(X) \leq g(\mathbb{E}X)
\end{equation}


\subsection{Stokes' theorem}
Stokes' theorem says that in typical problems a surface integral can be
computed using a line integral.
(There is some fine print regarding continuity and boundary conditions.)
This is a useful theorem because usually the line integral is easier to
compute.
In rectangular coordinates the equivalence between a line integral
on the left and a surface integral on the right is
%\bigskip
%\noindent
%$\displaystyle{\oint_C P\,dx+Q\,dy+R\,dz}$
%$$=
%\int\!\!\!\int_S
%\left({\partial Q\over\partial x}-{\partial P\over\partial y}\right)\,dx\,dy
%+
%\left({\partial R\over\partial y}-{\partial Q\over\partial z}\right)\,dy\,dz
%+
%\left({\partial P\over\partial z}-{\partial R\over\partial x}\right)\,dz\,dx
%$$
%
%\noindent
%Curve $C$ is the perimeter around $S$.
%The theorem can be also be written as

$$\oint P\,dx+Q\,dy+R\,dz
=\int\!\!\!\int_S(\mathop{\rm curl}{\bf F})\cdot{\bf n}\,d\sigma
$$

where ${\bf F}=(P,Q,R)$.
For $S$ parametrized by $x$ and $y$ we have
$${\bf n}\,d\sigma=\left(
{\partial S\over\partial x}\times{\partial S\over\partial y}
\right)dx\,dy$$

Example:
Let ${\bf F}=(y,z,x)$ and let $S$ be the part of the paraboloid
$z=4-x^2-y^2$
that is above the $xy$ plane.
The perimeter of the paraboloid is the circle $x^2+y^2=2$.
The following script computes both the line and surface integrals.
It turns out that we need to use polar coordinates for the
line integral so that {\it defint} can succeed.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
-- Surface integral
z = 4-x^2-y^2
F = (y,z,x)
S = (x,y,z)
f = dot(curl(F),cross(d(S,x),d(S,y)))
x = r*cos(theta)
y = r*sin(theta)
defint(f*r,r,0,2,theta,0,2pi)
-- Line integral
x = 2*cos(t)
y = 2*sin(t)
z = 4-x^2-y^2
P = y
Q = z
R = x
f = P*d(x,t)+Q*d(y,t)+R*d(z,t)
f = circexp(f)
defint(f,t,0,2pi)
\end{Verbatim}

This is the result when the script runs.
Both the surface integral and the line integral
yield the same result.

$\displaystyle -4\pi$

$\displaystyle -4\pi$


In this (sub)section we will explore a special kind of relation called an \textbf{equivalence relation}. Suppose $A$ is a set. Roughly speaking, there are two useful ways of viewing an equivalence relation $A$.
\begin{itemize}
	\item We can view the relation as a classifier. That is, the relation divides the elements of $A$ into groups and classifies them as being the same if they are in the same group.
	\item Alternatively (and equivalently), the relation compares two elements $a$ and $b$ and determines whether the two elements are ``the same'' or not the same.
\end{itemize}

In this section we will see how these two views describe the same thing. In short this relationship can be more formally described as how we can use an equivalence relation to partition $A$ into several \textit{equivalence classes}. We will fully define these terms more formally later. First we begin with the definition of an equivalence relation.

\begin{definition}
Let $A$ be a set. An equivalence relation $\sim \subset A \times A$ is a relation with the following properties:
\begin{enumerate}
	\item For all $a \in A$, $a \sim a$.
	\item If $a \sim b$ for $a, b \in A$, then $b \sim a$.
	\item If $a \sim b$ and $b \sim c$, then $a \sim c$.
\end{enumerate}

Moreover, these conditions have specific names.
\begin{enumerate}
	\item Any relation satisfying the first condition is called \textit{reflexive}.
	\item Any relation satisfying the second condition is called \textit{symmetric}.
	\item Any relation satisfying the third condition is called \textit{transitive}.
\end{enumerate}
\end{definition}

Let us unpack the criterion for an equivalence relation $\sim$ on $A$. The first condition simply states that every element in $a$ is related to itself. The second condition models the condition that if $a$ is related to $b$, then $b$ should be related to $a$ as well. The third condition asserts the \textit{transitivity} of the relation $\sim$. That is, if $a$ is related to $b$, and $b$ is related to $c$, then $a$ should be related to $c$; that is, we can chain relations to get other relations.

\begin{example}
The notion of object equality is an equivalence relation. More formally, the $=$ symbol can be seen as a relation on any set $A$. We check that this satisfies all the properties that an equivalence relation should satisfy:
\begin{enumerate}
	\item $a = a$ for all $a \in A$.
	\item If $a = b$, then $b = a$ as well.
	\item If $a = b$ and $b = c$, then $a = c$ as well.
\end{enumerate}
\end{example}
The verifications we made above are true because they are in fact properties of \textit{equality} itself. In fact, the definition of an equivalence relation was formed to encode more general notions of equality. We will describe more examples of equivalence relations later in the section and explore for each relation in what way the elements are the same.

\begin{example}\label{apt1_eqr}
Suppose we have an apartment building $A$. Let $S$ be the set of all residents in $A$. We will define an equivalence relation $\sim$ on $S$ as follows. For $s, t \in S$ (ie, any two residents of the building), we say that $s \sim t$ if $s$ and $t$ live at the same apartment number. We can verify that this is an equivalence relation by verifying the three properties an equivalence relation needs to satisfy:
\begin{enumerate}
	\item For any $s \in S$, $s \sim s$ by default because they are the same person and hence they live at the same apartment number.
	\item If $s$ and $t$ live in the same apartment number this is the same as saying that $t$ and $s$ live at the same apartment number. So if $s \sim t$, then $t \sim s$.
	\item If $s \sim t$ and $t \sim r$, then $s$, $t$, and $r$ all live at the same apartment number. It follows that $s \sim r$ as desired.
\end{enumerate}
\end{example}

\begin{example}\label{even_odd_eqr}
For the set $\mathbb{Z}$ define a relation $\sim$ as follows. We have $a \sim b$ if $a - b$ is even. We can verify that this relation is an equivalence relation below:
\begin{enumerate}
	\item $a-a = 0$, which is even (it is $2 \times 0$), so $a \sim a$.
	\item If $a \sim b$, then $a - b$ is even. It follows that $b - a = -(a-b)$ is even as well. For if $a - b = 2m$ for some integer $m$ then $b - a = -(a-b) = -2m = 2(-m)$ can be seen to be an even integer. So $b \sim a$.
	\item If $a \sim b$ and $b \sim c$, then $a - b$ and $b - c$ are both even. So $a - b = 2m$ and $b - c = 2n$ for some integers $m$ and $n$. It can be seen that their sum must be even as well. But we have
	\[a - c = (a - b) + (b - c) = 2m + 2n = 2(m + n)\]
	so it follows that $a - c$ is even as well. Hence $a \sim c$ and the relation is transitive.
\end{enumerate}

So it follows that $\sim$ is an equivalence relation. We can verify that $a \sim b$ only if $a$ and $b$ are both odd or both even. So in this case, $\sim$ defines whether or not items are the same based on whether or not they have the same remainder when integer dividing by $2$.

We will also show that a similar looking relation is not an equivalence relation. Suppose instead we define $a \sim b$ if $a \sim b$ is odd. One can check that this relation is symmetric, but it is neither reflexive nor transitive.
\end{example}

\begin{example}\label{modm_eqr}
We can generalize the previous example as follows. For the set $\mathbb{Z}$ we can define a relation $\sim$ with the property that $a \sim b$ if $b - a$ is divisible by a positive integer $m$ (where $m$ is given in advance). One can check as above that this forms an equivalence relation.
\end{example}

At the beginning of this (sub)section we described two different ways of looking at an equivalence relation. We have made the second way (an equivalence relation as determining whether two elements are the ``same'' in some sense) above. We now make the first notion (an equivalence relation as a classifier) precise.

\begin{definition}
Suppose $S$ is a set. Then a partition $P$ of $S$ is a collection of subsets $P_n$ (possibly infinite) where any two $P_i$ and $P_j$ are mutually disjoint and $\bigcup_{j = 1}^nP_j = S$.
\end{definition}

One can use the word ``partition'' either as a noun indicating a set partition, as defined above, or as a verb describing the action of dividing a set into disjoint subsets. In the example below ``partition'' is used in its verb form.

\begin{example}
We can partition the set $S = \{1, 2, 3, 4, 5, 6\}$ into the even elements $E = \{2, 4, 6\}$ and odd elements $O = \{1, 3, 5\}$. One can easily see that $E$ and $O$ are disjoint and $E \cup O = S$.
\end{example}

Now we will describe how given an equivalence relation we can partition a set in a natural way. The idea is to include elements in the same subset if they are related to each other. A typical example is the previous example. If $\sim$ represents the equivalence relation $a \sim b$ if $a - b$ is even, then all the elements in $E$ are equivalent and all the elements in $O$ are equivalent. To generalize this notion to any equivalence relation, we will introduce the notion of an \textit{equivalence class}.

\begin{definition}
Suppose that $S$ is a set and $\sim$ is an equivalence relation on $S$. Let $a \in S$ be any element. We define
\[C_a = \{s \in S \mid s \sim a\}.\]
In other words, $C_a$ is the subset of $S$ whose elements are all equivalent to $a$.
\end{definition}

\begin{example}\ref{even_odd2_eqr}
In example \ref{even_odd_eqr} we defined an equivalence relation on $\mathbb{Z}$ with $a \sim b$ if $a - b$ was an even number. For this example we consider what $C_0$ is. If $a \sim 0$, then $a - 0 = a$ is an even number, and conversely. So $C_0 = \{\dots,-4, -2, 0, 2, 4, \dots,\}$. Similarly, we see that $C_1 = \{\dots,-3, -1, 1, 3, \dots\}$. We might observe the following:
\begin{itemize}
	\item $C_0$ and $C_1$ are disjoint. Their union is all of $\mathbb{Z}$, so $C_0$ and $C_1$ form a partition of the integers.
	\item We might notice that $C_4$, $C_2$, and $C_0$ are all the same sets. Indeed, we can convince ourselves that in this example, no matter what element $i$ we take from $C_0$, it will be the case that $C_i = C_0$. So an equivalence class seems to not depend on the \textit{choice of representative}.
\end{itemize}
\end{example}

Let's further explain what we mean by a \textit{choice of representative}. In defining an equivalence class $C_a$, we make a choice of element $a$. But as the above example demonstrates, it seems like we can choose any element from an equivalence class and the resulting equivalence class will turn out to be same. In fact, this happens for any general equivalence relation and we will explain why below. The reasoning follows from a particular claim:

\begin{proposition}
Let $S$ be a set, and let $\sim$ be an equivalence relation on $S$. For any $a, b \in S$, two possibilities happen.
\begin{enumerate}
	\item $C_a \cap C_b = \varnothing$, or the two equivalence classes are disjoint.
	\item $C_a = C_b$, ie, the two equivalence classes are the same.
\end{enumerate}
\end{proposition}

To show that this claim is true, we examine the two subsets $C_a$ and $C_b$. Maybe it is the case that $C_a \cap C_b = \varnothing$. Then this possibility has happened and there is nothing left to show. The other possibility is that $C_a \cap C_b \neq \varnothing$. Then to show that the claim is true we need to show that in fact $C_a = C_b$.

First we show that $a \sim b$. Since $C_a \cap C_b \neq \varnothing$ there is an element $c \in C_a \cap C_b$. By definition of set intersection, $c \in C_a$, and also $c \in C_b$. By definition of $C_a$ and $C_b$, we have that
\[c \sim a, c \sim b.\]
By symmetry of the equivalence relation, $a \sim c$, and hence by transitivity we have $a \sim b$.

Now that we know that $a \sim b$ it is pretty easy to show that $C_a \subset C_b$ and vice versa. To show that $C_a \subset C_b$, suppose $d$ is any element of $C_a$. By definition, $d \sim a$. But since $a \sim b$, by transitivity, $d \sim b$. So $d \in C_b$. It follows that $d \in C_b$. So $C_a \subset C_b$ as desired. To show $C_b \subset C_a$ is the same reasoning except we use the fact that $b \sim a$ (which is true by the symmetry property, since $a \sim b$).

%%%still need to explain how this gives you a partition of S
We are almost done. Now we just need to answer the following question: how does this claim help us generate a partition of the set $S$? To answer this partially we might describe how to partition $S$ using an iterative process:
\begin{enumerate}
	\item Choose any element $a \in S$. We take $C_a$.
	\item Choose any element $b \in S$, $b \not\in C_a$. This ensures that $C_a \cap C_b = \varnothing$ (why?).
	\item Choose any element $c \in S$, $c \not\in C_a \cup C_b$. This ensures that $C_c$ is disjoint from the first two sets.
	\item Keep going like this until $S$ is exhausted of elements.
\end{enumerate}
In more informal terms, we think about $S$ as a big box of elements, and we keep taking out elements of $S$ in bags (which are the equivalence classes), until $S$ is empty. The reason why this is not quite complete is that of course $S$ may be infinite, and the equivalence classes finite, so of course one might never reach a complete partition of $S$ by doing this. But the idea is still the same.

To partition any general set $S$ with an equivalence relation $\sim$ we first observe that the following identity holds:
\[S = \bigcup_{a \in S}C_a.\]
To see why this is true, we first observe that since all the $C_a \subset S$, then the backwards inclusion is true, ie $S \supset \bigcup_{a \in S}C_a$. For the forwards inclusion we observe that for any $c \in S$, $c \in C_c$ because $c \sim c$ by reflexivity. It follows that the forward inclusion holds, and hence the set equality holds.

Now we are tempted to say that the collection $\{C_a\}_{a \in S}$ is the required partition, because by the claim we showed above, the $C_a$ are either the same or disjoint. However, since some of the $C_a$ may be the same this is not generally a partition yet. What we do now is consider all collections $C_a, C_b, \dots$ which are the same, and discard all but one of them. After this process is completed we have our required partition.

\begin{example}\label{apt2_eqr}
In Example \ref{apt1_eqr}, we showed that the equivalence relation $p \sim q$ on two apartment building residents if $p$ and $q$ live at the same apartment number was an equivalence relation. The equivalence class $C_p$ consists of the set of all people who are roommates with $p$. In general, by our reasoning above, the collection of distinct and disjoint equivalence classes partitions the set of apartment residents $A$. Ignoring the representative of any equivalence class, we see that any subset used in our partition is just the collection of all the people living in a particular apartment number.

If we are to view this equivalence relation as a classifier, then roughly speaking this equivalence relation classifies people based on what apartment number they reside at.
\end{example}

\begin{example}\label{mod2_classes_eqr}
In Example \ref{even_odd2_eqr} we divided the integers $\mathbb{Z}$ based on whether they were odd or even. So in terms of a classifier this equivalence relation classifies integers based on whether or not they are even or odd.
\end{example}

\subsubsection{Exercises}

\begin{enumerate}
	\item \input{Ch3/2_ex/problem_1}
	\item \input{Ch3/2_ex/problem_2}
	\item Suppose Lily presents to you the following argument that a relation having the transitive and symmetric property is in fact reflexive: they explain that since $a \sim b$, then by symmetry, $b \sim a$. But then by transitivity, $a \sim a$. So the relation must be reflexive. Is Lily right? If not, what is wrong with their argument?
	\item \input{Ch3/2_ex/problem_5}
	\item \input{Ch3/2_ex/problem_6}
\end{enumerate}


\subsection{Time Series Normalization} \label{time_series_normalization}

Literature on time series mining \cite{ding2008querying,spiegel2015diss} suggests to normalize all (sub-)sequences before measuring their pair-wise dissimilarity by means of a distance measure. There are multiple ways to normalize time series, where two common techniques \cite{das1998rule} are compared  in this study.

Given is a time series $Q = (q_1, \dots, q_n)$ of length $n$, we can compute its mean $\mu$ and standard deviation $\sigma$ as followed:

\begin{equation*}
\mu = \frac{1}{n} \sum_{i=1}^{n} q_i 	\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; 	\sigma = \frac{1}{n} \sum_{i=1}^{n} (q_i - \mu)^2
\end{equation*}

Having defined the mean $\mu$ and standard deviation $\sigma$, we can normalize each data point $q_i$ (with $1\leq i \leq n$) of a time series $Q = (q_1, \dots, q_n)$ in one of the two following ways \cite{das1998rule}:

\begin{equation}
\eta(q_i) = q_i - \mu
\end{equation}

\begin{equation}
z(q_i) = \frac{q_i - \mu}{\sigma}
\label{eqn:zscore}
\end{equation}

Equation \ref{eqn:zscore} is commonly known as the Z-score. For the sake of simplicity we refer to $\eta$ and $z$ normalization \cite{das1998rule} for the rest of the paper.


%% 5
\section{ Monoids. }

\subsection{Natural numbers.} Natural numbers form a monoid with respect to addition.
To prove that, we need to show that in $\mathbb{N}$ we have an operation $+$ that has a left and right identity.
This identity element is zero, as indeed
\begin{align*}
\forall n \in \mathbb{N},&& 0 + n &= n && \text{by definition of addition} \\
                         && n + 0 &= n && \text{Lemma 2.2.2 in Tao's {\em Analysis I}}
\end{align*}
Addition is associative, too, as per Proposition 2.2.5 (Exercise 2.2.1) in {\em Analysis I}\cite{tao2016}.
We omit the proof here, as in the next part of the question we'll need to prove the same for a list --- and we can always
prove associativity of naturals by reduction to lists, if we choose a (length of) string of 1 as a representation of natural numbers.

\subsection{A string of 0 and 1.}
Strings form a monoid with respect to concatenation. Concatenation of two strings is a string, so the type checks;
concatenation with an empty string is the original string; and, intuitively, concatenation seems to be associative.

\newcommand\cat{+\kern-1.3ex+\kern0.8ex} % https://tex.stackexchange.com/questions/4194/how-to-typeset-haskell-operator-and-friends
More formally, concatenation is Haskell is defined\cite{base-src} as (we omit compiler directives here for clarity)
\begin{minted}{haskell}
(++) :: [a] -> [a] -> [a]
(++) []     ys = ys
(++) (x:xs) ys = x : xs ++ ys
\end{minted}
Ignoring infinite lists for the sake of this exercise, we can see that from the first clause, \mintinline{haskell}{[]} is indeed a left unit.

Right unit is trickier, but we can follow Tao's Lemma 2.2.2: The base case, $[] \cat [] = []$ holds as we know that \mintinline{haskell}{[]}
is a left unit. Now suppose inductively that $xs \cat [] = xs.$ We wish to show that $(x:xs) \cat [] = (x:xs)$.
From the second clause in the function definition, $(x:xs) \cat [] = x : (xs \cat []) = x : xs,$ and that closes the induction.

Tao leaves the proof of associativity to the reader, so we cannot avoid proving that. Since the string, being a list in Haskell, is an inductive data type,
it is natural to use induction again. We consider concatenation $as \cat bs \cat cs$ and would like to show that $as \cat (bs \cat cs) = (as \cat bs) \cat cs.$
Base case, $[] \cat (bs \cat cs) = ([] \cat bs) \cat cs,$ is trivial as \mintinline{haskell}{[]} is a left unit by definition of concatenation.
Now suppose that associativity holds for $as \cat (bs \cat cs) = (as \cat bs) \cat cs.$ We want to show that $(a : as) \cat (bs \cat cs) = ((a : as) \cat bs) \cat cs:$
\begin{align*}
(a : as) \cat (bs \cat cs) &= a : (as \cat (bs \cat cs)) = && \text{second clause of definition} \\
                           &= a : ((as \cat bs) \cat cs) = && \text{from induction hypothesis} \\
                           &= (a : (as \cat bs)) \cat cs = && \text{reversed second clause} \\
                           &= ((a : as) \cat bs) \cat cs   && \text{reversed second clause again}
\end{align*}
and the induction is closed.

Note that we never used the actual list elements here; indeed, the function definition is generic, and does not know what the element type is.
So if we use \mintinline{haskell}{Either One Zero} for \mintinline{haskell}{a}, our proof stays true for strings of ones and zeros.


\subsection{Properties of determinants}

\subsubsection{Identity}

\(\det I = 1\)

\subsubsection{Multiplication}

\(\det (AB)=\det A \det B\)

\subsubsection{Inverse}

\(\det (M^{-1})=\dfrac{1}{\det M}\)

We know this because:

\(\det (MM^{-1})=\det I = 1\)

\(\det M \det M^{-1}=1\)

\(\det (M^{-1})=\dfrac{1}{\det M}\)

\subsubsection{Complex cojugate}

\(\det (M^*)=\overline {\det M}\)

\subsubsection{Transpose}

\(\det (M^T)=\det M\)

\subsubsection{Addition}

\(\det (A+B)=\det A + \det B\)

\subsubsection{Scalar multiplication}

\(\det cM = c^n\det M\)

\subsubsection{Determinants and eigenvalues}

The determinant is equal to the product of the eigenvalues.


\subsection{Linear algebra}
The function $dot$ is used to multiply vectors and matrices.
Let

\[
A=\begin{pmatrix}1&2\\3&4\end{pmatrix}
\quad{\rm and}\quad x=\begin{pmatrix}x_1\\x_2\end{pmatrix}
\]

The following example computes $Ax$.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
A = ((1,2),(3,4))
x = (x1,x2)
dot(A,x)
\end{Verbatim}

$\displaystyle \begin{bmatrix}x_1+2x_2\\3x_1+4x_2\end{bmatrix}$

The following example shows how to use $dot$ and $inv$ to solve for
the vector $X$ in $AX=B$.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
A = ((3,7),(1,-9))
B = (16,-22)
X = dot(inv(A),B)
X
\end{Verbatim}

$\displaystyle X=\begin{bmatrix}-\frac{5}{17}\\ \frac{41}{17}\end{bmatrix}$

%One might wonder why the $dot$ function is necessary.
%Why not simply use $X=inv(A)*B$ like scalar multiplication?
%The reason is that the software normally reorders factors internally to
%optimize processing.
%For example, $inv(A)*B$ in symbolic form is changed to $B*inv(A)$
%internally.
%Since the dot product is not commutative, this reordering would give the
%wrong result.
%Using a function to do the multiply avoids the problem because
%function arguments are not reordered.

The $dot$ function can have more than two arguments.
For example, $dot(A,B,C)$ can be used for the dot product of three tensors.

Square brackets are used for component access.
Index numbering starts with 1.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
A = ((a,b),(c,d))
A[1,2] = -A[1,1]
A
\end{Verbatim}

$\displaystyle \begin{bmatrix}a&-a\\c&d\end{bmatrix}$

The following example demonstrates the relation
$A^{-1}=\mathop{\rm adj}{A}/\mathop{\rm det}{A}$.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
A = ((a,b),(c,d))
inv(A)
\end{Verbatim}

$\displaystyle \begin{bmatrix}\frac{d}{ad-bc} & -\frac{b}{ad-bc}\\-\frac{c}{ad-bc} & \frac{a}{ad-bc}\end{bmatrix}$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
adj(A)
\end{Verbatim}

$\displaystyle \begin{bmatrix}d & -b\\-c & a\end{bmatrix}$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
det(A)
\end{Verbatim}

$\displaystyle ad-bc$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
inv(A) - adj(A)/det(A)
\end{Verbatim}

$\displaystyle \begin{bmatrix}0 & 0\\0 & 0\end{bmatrix}$

Sometimes a calculation will be simpler if it can be reorganized to use
$adj$ instead of $inv$.
The main idea is to try to prevent the determinant from appearing as a
divisor.
For example, suppose for matrices $A$ and $B$ you want to check that
$${A}-{B}^{-1}=0$$
Depending on the complexity of $\mathop{\rm det}B$, the software
may not be able to find a simplification that yields zero.
Should that occur, the following alternative formulation can be tried.
$$(\mathop{\rm det}{B})\cdot{A}-\mathop{\rm adj}{B}=0$$

The adjunct of a matrix is related to the cofactors as follows.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
A = ((a,b),(c,d))
C = ((0,0),(0,0))
C[1,1] = cofactor(A,1,1)
C[1,2] = cofactor(A,1,2)
C[2,1] = cofactor(A,2,1)
C[2,2] = cofactor(A,2,2)
C
\end{Verbatim}

$\displaystyle C=\begin{bmatrix}d&-c\\-b&a\end{bmatrix}$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
adj(A) - transpose(C)
\end{Verbatim}

$\displaystyle \begin{bmatrix}0&0\\0&0\end{bmatrix}$