\chapter{Lattice gauge theory}
\label{chap:latticeqcd}

\section{Gauge invariance}
Nevertheless, a naive discretization of the Yang-Mills action from Equation \cref{yangmills}, in which one replaces all the partial derivatives appearing in the field strengths $F^{\mu\nu}$ with finite differences leads to the loss of gauge invariance \cite{schwartz, peskin}.

\begin{remark}
This is a consequence of imposing local gauge symmetry, with a $\textsf{SU}(3)$ gauge transformation given by Equation \cref{gaugetransf}, of the fields
\begin{align*}
    \phi(x)\mapsto \textsf{U}(x)\phi(x).
\end{align*}
Fields at different space-time points cannot directly be compared. Because of this, the partial derivative, which contains the difference between fields at different points, is not well defined. By introducing a quantity which gauge transform as
\begin{align}\label{latt3}
    \textsf{W}(x,y)\mapsto\textsf{U}(x)\textsf{W}(x,y)\textsf{U}^\dagger(y),
\end{align}
one may now properly define the covariant derivative as\footnote{Notice that with this definition, the covariant derivative transforms as
\begin{align}\label{latt1}
    \textsf{D}_\mu\phi(x)\mapsto\textsf{U}(x)\textsf{D}_\mu\phi(x).
\end{align}
}

\begin{definition}[Covariant derivative]
\begin{align}\label{latt2}
    \textsf{D}_{\mu} \phi(x)\overset{\Delta}{=} \lim _{\delta \epsilon^{\mu} \rightarrow 0} \frac{\textsf{W}(x, x+\epsilon) \phi(x+\epsilon)-\phi(x)}{\epsilon^\mu}.
\end{align}
\end{definition}
\noindent We choose $\textsf{W}(x,x)=\mathds{1}$ and then write an expansion in terms of the gauge fields\footnote{If we plug in this relation back in Equation \cref{latt2} and then in Equation \cref{latt1}, we deduce the gauge transformation of the gauge fields from Equation \cref{gaugefields}.}
\begin{align*}
    \textsf{W}(x,x+\epsilon)=1+ig\epsilon^\mu A_\mu+\mathcal{O}(\epsilon^2).
\end{align*}
\end{remark}

This enables us to express the {\sffamily\color{maincolor}Wilson line} as\footnote{Where the fields may further be written as $A_{\mu}=A_{\mu}^aT^a$ in the fundamental representation.}
\boxedeqlabel{latt5}{
    \textsf{W}(x,y)=\mathcal{P}\exp{i g \int_{y}^{x} \mathrm{d} z^{\mu} A_{\mu}(z) }
}

\begin{remark}
The {\sffamily\color{maincolor}path-ordering} operator $\mathcal{P}\{\ldots\}$ is necessary due to the non-Abelian nature of the fields. More clearly, after writing a Taylor expansion, the path-ordering operator acts as

\begin{align*}
    \begin{aligned}
    \textsf{W}(x, y)=& 1+i g \int_{0}^{1} \frac{\mathrm{d} z^{\mu}_\lambda}{\mathrm{d} \lambda} A_{\mu}^{a}(z_\lambda^\mu) T^{a} \mathrm{d} \lambda-\frac{1}{2} g^{2} \int_{0}^{1} \mathrm{d} \lambda \int_{0}^{1} \mathrm{d} \tau \frac{\mathrm{d} z^{\mu}_\lambda}{\mathrm{d} \lambda} \frac{\mathrm{d} z^{\nu}_\tau}{\mathrm{d} \tau}\times\\
    & \times A_{\mu}^{a}(z_\lambda^\mu) A_{\nu}^{b}(z_\tau^\nu)\left[T^{a} T^{b} \theta(\lambda-\tau)+T^{b} T^{a} \theta(\tau-\lambda)\right]+\cdots.
\end{aligned}
\end{align*}
\end{remark}

A Wilson line taken along a closed path $\gamma$ is called a {\sffamily\color{maincolor}Wilson loop} and it's given by
\boxedeqlabel{latt4}{
    \textsf{W}_\gamma=\mathcal{P}\exp{i g \oint\limits_{\gamma} \mathrm{d} x^{\mu} A_{\mu}(x)}
}

It is important to emphasize that the trace of a Wilson loop is gauge invariant\footnote{This may easily be proven by inserting Equation \cref{latt4} back in Equation \cref{latt3} and using the invariance of the trace under cyclic permutations.}.

\section{Real-time lattice gauge theory}
Let us begin by discretizing the Minkowski space-time on a hypercubic lattice\footnote{The lattice spacing along direction $\mu$, with the unit vector is $\hat{e}^\mu$, is $a^\mu$. This enables us to write $\hat{a}^{\mu}=a^\mu\hat{e}^\mu$. More concisely, $a^0$ is the time step and $a^i$ denote the spatial lattice spacings.} whose points are given by
\begin{align*}
    \textsf{X}^4=\left\{x \mid x=\sum_{\mu=0}^{3} n_{\mu} \hat{a}^{\mu}, \quad n_{\mu} \in \mathbb{Z}\right\}.
\end{align*}

A field $\phi(x)$ which resides on the lattice will be denoted as $\phi_x$, with $x\in\textsf{X}^4$. The gauge transformation $\textsf{U}(x)$ will become, upon discretization, $\textsf{U}_x$. The discretized action and corresponding field equations written in terms of $A_\mu$ will not remain gauge invariant.

Nevertheless, a gauge invariant lattice action may be constructed. Instead of using the gauge fields $A_\mu$, along with the corresponding conjugate momenta $P_\mu$, as the main degrees of freedom, one may simply seek for a more suitable quantity which is already gauge invariant and preserves gauge invariance upon discretization. An action built from such a quantity will inherently be gauge invariant. The simplest choice is the trace of a Wilson line, as given in Equation \cref{latt4}. We already saw that such a construction is gauge invariant.

% Diagrams included directly as .tikz images generated from Tikzit
\begin{figure}
    \centering
    \subfloat[\centering {\sffamily 2d} plaquette]{{\includegraphics[width=0.45\textwidth]{tikzit/plaquette_2d.tikz}}}%
    \qquad
    \subfloat[\centering {\sffamily 3d} plaquette]{{\includegraphics[width=0.45\textwidth]{tikzit/plaquette_3d.tikz}}}%
    \caption{Schematic representations of a plaquette, the shortest Wilson loop on a rectangular lattice.}%
    \label{fig:plaquettes}%
\end{figure}

% Diagrams included using .pdf files saved from Tikzit
%\begin{figure}
%    \centering
%    \subfloat[\centering {\sffamily 2d} plaquette]{{\includegraphics[width=0.45\textwidth]{plaquette_2d.pdf}}}%
%    \qquad
%    \subfloat[\centering {\sffamily 3d} plaquette]{{\includegraphics[width=0.45\textwidth]{plaquette_3d.pdf}}}%
%    \caption{Schematic representations of a plaquette, the shortest Wilson loop on a rectangular lattice.}%
%    \label{fig:plaquettes}%
%\end{figure}

A Wilson line taken between two neighboring lattice points, namely $x$ and $x+\hat{a}^{\mu}$, is called a {\sffamily\color{maincolor}gauge link} and it's given by\footnote{Here $\overline{\mathcal{P}}\{\ldots\}$ denotes the anti-path-ordering. We introduce the notation
\begin{align*}
    \textsf{W}_{x,\mu}\overset{\Delta}{=}\textsf{W}(x,x+\hat{a}^{\mu})
\end{align*}
and similarly for $A_{x,\mu}$.
}
\begin{align*}
    \textsf{W}_{x, \mu}=\overline{\mathcal{P}} \exp{i g \int\limits_{x}^{x+\hat{a}^{\mu}} d x^{\mu} A_{x,\mu}}.
\end{align*}

\begin{remark}
In a similar manner, we may also introduce a gauge link along the opposite direction, which would yield
\begin{align}\label{latt6}
    W_{x,\mu}^{\dagger}=\textsf{W}_{x+\mu,-\mu}.
\end{align}
A gauge link transforms as
\begin{align*}
    \textsf{W}_{x, \mu} \rightarrow \textsf{U}_{x} \textsf{W}_{x, \mu} \textsf{U}_{x+\mu}^{\dagger}.
\end{align*}
One may express a gauge link and afterwards expand it, see Equation \cref{latt5}, as
\begin{equation*}
    \begin{aligned}
     \textsf{W}_{x,\mu}&=\exp{iga^\mu A_{x,\mu}}\approx \mathds{1}+iga^\mu A_{x,\mu(x)}-\frac{1}{2}g^2a^\mu a^\nu A_{x,\mu}A_
     {x,\nu}+\mathcal{O}(a^3).
\end{aligned}
\end{equation*}
\end{remark}

The simplest non-trivial\footnote{The shortest Wilson line would just go back and forth between two neighbouring sites but such a combination would simply yield the trivial result
\begin{align*}
    \textsf{W}_{x,\mu}\textsf{W}_{x+\mu,-\mu}\stackrel{\text{\cref{latt6}}}{=\joinrel=}\mathds{1}.
\end{align*}
} Wilson loop on the lattice may be constructed along the path connecting neighbouring points on a rectangular {\sffamily\color{maincolor}plaquette}, see Figure \cref{fig:plaquettes}, as
\begin{align*}
    \textsf{W}_{x, \mu \nu}&=\textsf{W}_{x, \mu} \textsf{W}_{x+\mu, \nu} \textsf{W}^\dagger_{x+\mu,\mu}\textsf{W}^\dagger_{x,\nu}\\
    &\stackrel{\text{\cref{latt6}}}{=\joinrel=}\textsf{W}_{x, \mu} \textsf{W}_{x+\mu, \nu} \textsf{W}_{x+\mu+\nu,-\mu} \textsf{W}_{x+\nu,-\nu},
\end{align*}
which may further be expressed as
\begin{align*}
    \textsf{W}_{x, \mu \nu} \approx \exp{ig a^{\mu} a^{\nu} F_{x,\mu \nu}+\mathcal{O}\left(a^{3}\right)}.
\end{align*}

\begin{proof}
We may write the discretized Wilson loop as\footnote{
Using the Campbell-Baker-Hausdorff formula
\begin{align*}
    \exp{\textsf{A}}\exp{\textsf{B}}\approx\exp{\textsf{A}+\textsf{B}+\frac{1}{2}[\textsf{A}, \textsf{B}]+\cdots}.
\end{align*}
}
\begin{align*}
    \textsf{W}_{x,\mu\nu}&\approx\exp\Bigg\{ig(A_{x,\mu}+A_{x+\mu,\nu}-A_{x+\nu,\mu}-A_{x,\nu})+\\
    &\phantom{\approx\exp\Bigg\{}+\frac{g^2}{2}\Big(\big[A_{x,\nu}+A_{x+\nu,\mu},A_{x+\mu,\nu}+A_{x,\mu}\big]-\\
    &\phantom{\approx\exp\Bigg\{}-\big[A_{x,\nu},A_{x+\nu,\mu}\big]-\big[A_{x+\mu,\nu},A_{x,\mu}\big]\Big)\Bigg\}.
\end{align*}
By making use of the expansion
\begin{align*}
    A_{x+\mu,\nu}\approx A_{x,\nu}+a^\mu\partial_\mu A_{x,\nu}+\mathcal{O}(a^2),
\end{align*}
we may then derive
\begin{align*}
     \textsf{W}_{x,\mu\nu}&\approx\exp\Big\{iga^\mu a^\nu\underbrace{\big(\partial_\mu A_{x,\nu}-\partial_\nu A_{x,\mu}-ig[A_{x,\nu},A_{x,\mu}]\big)}_{\mathclap{\textstyle F_{x,\mu\nu}}}+\mathcal{O}(a^3)\Big\}.
\end{align*}
This may further be approximated as
\begin{align*}
    \textsf{W}_{x,\mu\nu}=\mathds{1}+ig a^\mu a^\nu F_{x,\mu \nu}-\frac{1}{2}(ga^\mu a^\nu)^2 F_{x,\mu \nu}^2+\mathcal{O}(a^5).
\end{align*}
in the limit of small lattice spacings.
\end{proof}

Therefore, we may construct a gauge invariant quantity, since in contains Wilson lines traced over, under the discretized gauge transformation $\textsf{U}_x$ as

\begin{align}\label{latt7}
    \textsf{Tr}\big\{2-\textsf{W}_{x, \mu \nu}-\textsf{W}_{x, \mu \nu}^{\dagger}\big\} \approx\left(g a^{\mu} a^{\nu}\right)^{2} \textsf{Tr}\big\{F_{x,\mu \nu}^{2}\big\}+\mathcal{O}(a^{6}).
\end{align}

The Yang-Mills action from Equation \cref{yangmills} may be split into an electric and a magnetic part
\begin{align*}
    \textsf{S}=\underbrace{\int \mathrm{d}^{4}x \sum_{i}\textsf{Tr}\big\{F_{0i}^2(x)\big\}}_{\mathclap{\textstyle \textsf{S}_\textsf{E}}}+\underbrace{\int \mathrm{d}^{4}x\sum_{i, j} \frac{1}{2} \textsf{Tr}\big\{F_{ij}^2(x)\big\}}_{\mathclap{\textstyle \textsf{S}_\textsf{B}}}.
\end{align*}
Upon discretization, they become\footnote{By making the replacement
\begin{align*}
    \int \mathrm{d}^{4}x(\ldots)\mapsto \underbrace{\prod\limits_{\mu}a^\mu}_{\mathclap{\textstyle \textsf{V}}}\sum_x(\ldots)
\end{align*}
and using the result from Equation \cref{latt7}.
}
\begin{align*}
    \textsf{S}_\textsf{E}& \approx V \sum_{x} \sum_{i} \frac{1}{\left(g a^{0} a^{i}\right)^{2}} \textsf{Tr}\big\{2-\textsf{W}_{x, 0 i}-\textsf{W}_{x, 0 i}^{\dagger}\big\}, \\
    \textsf{S}_\textsf{B}& \approx V \sum_{x} \sum_{i, j} \frac{1}{2\left(g a^{i} a^{j}\right)^{2}} \textsf{Tr}\big\{2-\textsf{W}_{x, i j}-\textsf{W}_{x, i j}^{\dagger}\big\}.
\end{align*}
Thus, the Yang-Mills action on the lattice is given by

\shadedeq{
\textsf{S}=V \sum_{x}\Big( \sum_{i} \frac{1}{\left(g a^{0} a^{i}\right)^{2}} \textsf{Tr}\big\{2-\textsf{W}_{x, 0 i}-\textsf{W}_{x, 0 i}^{\dagger}\big\} -\sum_{i, j} \frac{1}{2\left(g a^{i} a^{j}\right)^{2}}\textsf{Tr}\big\{2-\textsf{W}_{x, i j}-\textsf{W}_{x, i j}^{\dagger}\big\}\Big)
}


%!TEX root = main.tex
\paragraph{Boundary integral formulation of electrostatics in molecular solvation}\label{s:formulation}

Biomolecules in an ionic solvent or water can be represented with a model where the solvent is a continuous dielectric: the so-called implicit-solvent model.
The molecule (solute) is a dielectric cavity with partial charges at the atomic locations,  represented as a collection of Dirac delta functions.
In  the sketch of Figure \ref{fig:implicit_solvent}, the molecular cavity is region $\Omega_1$, the infinite medium  is $\Omega_2$, and the point charges are $q_k$.
The interface $\Gamma$ represents the molecular surface and can be determined through several approaches: van der Waals radii, Gaussian surface, solvent-accessible surface, and solvent-excluded surface.
We use the latter.
A solvent-excluded surface is built by tracking the contact points of a (virtual) spherical probe of  $\sim$ 1.4 \AA\ in radius, the size of a water molecule, as it rolls around the solute's atoms (with their corresponding van der Waals radii).
In this setup, we can compute the change in electrostatic potential as the interior region is charged up with the solute's partial charges.
The solvent usually consists of water (dielectric constant $\epsilon_2\approx80$) with salt ions that are free to move around, forced by the electric field.
At equilibrium, the salt ions are in a Boltzmann distribution, leading to the linearized Poisson-Boltzmann equation for the potential in $\Omega_2$, considering a screening factor known as the inverse of the Debye length ($\kappa$).
The electrostatic potential in the solute cavity follows Poisson's equation in a low dielectric medium ($\epsilon_1\approx2\textrm{--}4$), with the solute's point charges as sources.
With interface conditions on the molecular surface $\Gamma$, where the potential and electric displacement must be continuous, this results in the following system of equations:
%
\begin{align} \label{eq:pde}
\nabla^2\phi_1 &= \frac{1}{\epsilon_1}\sum_k q_k\delta(\mathbf{r},\mathbf{r}_k) \text{ in the solute ($\Omega_1$),}\nonumber\\
(\nabla^2-\kappa^2)\phi_2 &= 0 \text{ in the solvent ($\Omega_2$),}\nonumber\\
\phi_1 &= \phi_2 \quad \epsilon_1\frac{\partial \phi_1}{\partial\mathbf{n}} = \epsilon_2\frac{\partial \phi_2}{\partial\mathbf{n}} \text{ on the interface ($\Gamma$)}.
\end{align}
%

%
\begin{figure}
\centering
\includegraphics[width=0.2\textwidth]{implicit_solvent.pdf}
\caption{Representation of a dissolved molecule with the implicit-solvent model. The solute ($\Omega_1$) and solvent ($\Omega_2$) regions are interfaced by the solvent-excluded surface ($\Gamma$), and $q_k$ and $r_{vdW}$ are the atomic charge and radii, respectively.}
\label{fig:implicit_solvent}
\end{figure}

Equation \eqref{eq:pde} can be re-written as an integral equation on $\Gamma$ via Green's second identity, yielding:
%
\begin{align} \label{eq:volume_potential}
\phi_{1}+ K_{L}^{\Omega_1}(\phi_{1,\Gamma}) -  V_{L}^{\Omega_1} \left(\frac{\partial}{\partial \mathbf{n}}  \phi_{1,\Gamma}  \right) & = \frac{1}{\epsilon_1} \sum_{k=0}^{N_q}  \frac{q_k}{4\pi|\mathbf{r}_{\Omega_1} - \mathbf{r}_k|}  \quad \text{on $\Omega_1$,} \nonumber \\
\phi_{2} - K_{Y}^{\Omega_2}(\phi_{2,\Gamma}) + V_{Y}^{\Omega_2} \left( \frac{\partial}{\partial \mathbf{n}} \phi_{2,\Gamma} \right) & = 0 \quad \text{on $\Omega_2$,}
\end{align}
%
where $\phi_{1,\Gamma} = \phi_1(\mathbf{r}_\Gamma)$ and $\phi_{2,\Gamma} = \phi_2(\mathbf{r}_\Gamma)$ are evaluated on $\Gamma$ approaching from $\Omega_1$ and $\Omega_2$, respectively. $K$ and $V$ are the double- and single-layer potentials for the Laplace (subscript $L$) and Yukawa (subscript $Y$) kernels, defined as:
%
\begin{align}\label{eq:single_double}
V^\Omega_{L,Y}(\varphi) = \oint_\Gamma g_{L,Y}(\mathbf{r}_\Omega,\mathbf{r}')\varphi(\mathbf{r}')d\mathbf{r}'\nonumber\\
K^\Omega_{L,Y}(\varphi) = \oint_\Gamma \frac{\partial g_{L,Y}}{\partial\mathbf{n}'}(\mathbf{r}_\Omega,\mathbf{r}')\varphi(\mathbf{r}')d\mathbf{r}',\nonumber\\
\end{align}
%
where $\varphi(\mathbf{r})$ is a distribution over $\Gamma$, and $g_L(\mathbf{r},\mathbf{r}')=\frac{1}{4\pi|\mathbf{r}-\mathbf{r}'|}$ and $g_Y(\mathbf{r},\mathbf{r}')=\frac{e^{-\kappa|\mathbf{r}-\mathbf{r}'|}}{4\pi|\mathbf{r}-\mathbf{r}'|}$ are the free-space Green's function of the Laplace and linearized Poisson-Boltzmann equations, respectively.

We can use Equation \eqref{eq:volume_potential} to compute $\phi_\Gamma$ and $\partial\phi_\Gamma/\partial\mathbf{n}$ with either the \emph{direct}~\cite{YoonLenhoff1990} or \emph{derivative}~\cite{JufferETal1991} (also known as \emph{Juffer}) formulations.
The simpler direct formulation results from evaluating $\phi_1$ and $\phi_2$ in the limit as $\mathbf{r}$ approaches $\Gamma$, and applying the interface conditions from Equation \eqref{eq:pde}, giving:
%
\begin{align} \label{eq:direct}
\frac{\phi_{1,\Gamma}}{2}+ K_{L}^{\Gamma}(\phi_{1,\Gamma}) -  V_{L}^{\Gamma} \left(\frac{\partial}{\partial \mathbf{n}}  \phi_{1,\Gamma}  \right) & = \frac{1}{\epsilon_1} \sum_{k=0}^{N_q}  \frac{q_k}{4\pi|\mathbf{r}_{\Gamma} - \mathbf{r}_k|} \nonumber \\
\frac{\phi_{1,\Gamma}}{2} - K_{Y}^{\Gamma}(\phi_{1,\Gamma}) + \frac{\epsilon_1}{\epsilon_2}V_{Y}^{\Gamma} \left( \frac{\partial}{\partial \mathbf{n}} \phi_{1,\Gamma} \right) & = 0
\end{align}
%
This formulation is ill-conditioned, since the condition number of the resulting matrix grows unbounded with the number of discretization elements.
An alternative with better conditioning was derived by Juffer \emph{et al.} \cite{JufferETal1991} by taking the normal derivative of Equation \eqref{eq:volume_potential}, and coupling both $\phi$ and $\partial\phi/\partial\mathbf{n}$ on the boundary as follows:

\begin{align}\label{eq:juffer}
    &\begin{multlined}[t][0.48\textwidth] \frac{\phi_{1,\Gamma}}{2}\left(1+\frac{\epsilon_2}{\epsilon_1}\right) - \left(\frac{\epsilon_2}{\epsilon_1}K_Y^\Gamma - K_L^\Gamma\right)(\phi_{1,\Gamma}) \\
    + \left(V_Y^\Gamma - V_L^\Gamma\right)\left( \frac{\partial}{\partial \mathbf{n}} \phi_{1,\Gamma} \right) = \sum_{k=0}^{N_q}  \frac{q_k}{4\pi\epsilon_1|\mathbf{r}_{\Gamma} - \mathbf{r}_k|}
    \end{multlined} \nonumber \\
    &\begin{multlined}[t][0.48\textwidth] - \left(W_Y^\Gamma - W_L^\Gamma\right)(\phi_{1,\Gamma}) +  \frac{1}{2}\frac{\phi_{1,\Gamma}}{\partial\mathbf{n}}\left(1+\frac{\epsilon_1}{\epsilon_2}\right) \\
    + \left(\frac{\epsilon_1}{\epsilon_2}K_Y^{\prime\Gamma} - K_L^{\prime\Gamma}\right)\left( \frac{\partial}{\partial \mathbf{n}} \phi_{1,\Gamma} \right) = \sum_{k=0}^{N_q}  \frac{\partial}{\partial\mathbf{n}_\mathbf{r}}\left(\frac{q_k}{4\pi\epsilon_1|\mathbf{r}_{\Gamma} - \mathbf{r}_k|}\right)
    \end{multlined}
\end{align}
%
Here, we use the adjoint double-layer ($K'$) and hypersingular ($W$) operators, which are defined as
%
\begin{align}\label{eq:adj_hyp}
K^{\prime\Gamma}_{L,Y}(\varphi) = \oint_\Gamma \frac{g_{L,Y}}{\partial\mathbf{n}}(\mathbf{r}_\Gamma,\mathbf{r}')\varphi(\mathbf{r}')d\mathbf{r}'\nonumber\\
W^\Gamma_{L,Y}(\varphi) = \oint_\Gamma \frac{\partial^2 g_{L,Y}}{\partial\mathbf{n}'\partial\mathbf{n}}(\mathbf{r}_\Gamma,\mathbf{r}')\varphi(\mathbf{r}')d\mathbf{r}'\nonumber\\
\end{align}
%
A slightly modified version of Equation \eqref{eq:juffer} is used in the work from Lu and coworkers~\cite{LuETal2006,LuETal2009,ZhangETal2019}, where they scale the expressions by $\epsilon_1/\epsilon_2$, and solve for the exterior field. This gives
%
\begin{align}\label{eq:lu}
    &\begin{multlined}[t][0.48\textwidth] \frac{\phi_{2,\Gamma}}{2}\left(\frac{\epsilon_1}{\epsilon_2}+1\right) - \left(K_Y^\Gamma - \frac{\epsilon_1}{\epsilon_2}K_L^\Gamma\right)(\phi_{2,\Gamma}) \\
    + \left(V_Y^\Gamma - V_L^\Gamma\right)\left( \frac{\partial}{\partial \mathbf{n}} \phi_{2,\Gamma} \right) = \sum_{k=0}^{N_q}  \frac{q_k}{4\pi\epsilon_2|\mathbf{r}_{\Gamma} - \mathbf{r}_k|}
    \end{multlined} \nonumber \\
    &\begin{multlined}[t][0.48\textwidth] -\frac{\epsilon_1}{\epsilon_2}\left(W_Y^\Gamma - W_L^\Gamma\right)(\phi_{2,\Gamma}) +  \frac{1}{2}\frac{\phi_{2,\Gamma}}{\partial\mathbf{n}}\left(1+\frac{\epsilon_1}{\epsilon_2}\right) \\
    + \left(\frac{\epsilon_1}{\epsilon_2}K_Y^{\prime\Gamma} - K_L^{\prime\Gamma}\right)\left( \frac{\partial}{\partial \mathbf{n}} \phi_{2,\Gamma} \right) = \sum_{k=0}^{N_q}  \frac{\partial}{\partial\mathbf{n}_\mathbf{r}}\left(\frac{q_k}{4\pi\epsilon_2|\mathbf{r}_{\Gamma} - \mathbf{r}_k|}\right)
    \end{multlined}
\end{align}

As we charge up the cavity, the solvent ions rearrange and polarize.
The resulting electrostatic potential is called a \emph{reaction} potential ($\phi_{reac}$), and we can write the following decomposition in $\Omega_1$ :
%
\begin{equation}
\phi_1 = \phi_{reac} + \phi_{coul},
\end{equation}
%
where $\phi_{coul}$ is the Coulombic potential from the solute point charges only.
Having $\phi_{1,\Gamma}$ and $\partial\phi_{1,\Gamma}/\partial\mathbf{n}$ from Equation \eqref{eq:direct} or Equation \eqref{eq:juffer}, we can compute $\phi_{reac}$ by subtracting out the Coulombic contribution in the right-hand side of Equation \eqref{eq:volume_potential}:
%
\begin{equation}\label{eq:phi_reac}
\phi_{reac} = -K_{L}^{\Omega_1}(\phi_{1,\Gamma}) +  V_{L}^{\Omega_1} \left(\frac{\partial}{\partial \mathbf{n}}  \phi_{1,\Gamma}  \right)
\end{equation}

The thermodynamic work required to dissolve a molecule, known as solvation free energy, is usually divided into nonpolar and polar components.
The nonpolar part generates the empty solute-shaped cavity in the solvent, which is then charged by placing the partial charges inside the cavity, giving rise to a polar term in the energy.
The work in charging is performed under $\phi_{reac}$, and it can be computed as:
%
\begin{equation} \label{eq:energy}
\Delta G^{polar}_{solv} = \frac{1}{2}\int_{\Omega_1} \rho\phi_{reac}d\mathbf{r} = \frac{1}{2}\sum_{k=1}^{N_q}q_k\phi_{reac}(\mathbf{r}_k).
\end{equation}


\newpage

\section*{Part C: Implementing SVM Variation [20 points] (Yichong and Prakhar) }
\textbf{Please attach your code as appendix to this problem.}\\
In the last homework, we derived a variant of SVM that explicitly maximizes the margin.
You can use any library on quadratic optimization (e.g., CVXOPT for python or \textsf{quadprog} for MATLAB) for this problem. Here is an instruction on CVXOPT.

Installation instructions of CVXOPT can be found \href{http://cvxopt.org/install/index.html}{here}. CVXOPT provides an easy interface for quadratic programming: The function \textsf{qp(P, q[, G, h[, A, b]])} solves the optimization problem
\begin{align*}
\text{minimize}_{x \in \mathbb{R}^n}\;  & (1/2) x^TPx+q^Tx \\
\text{subject to } & Gx \preceq h \\
& Ax=b
\end{align*}
for $P\in \mathbb{R}^{n\times n}, q\in \mathbb{R}^n, G\in \mathbb{R}^{m_1\times n}, h \in \mathbb{R}^{m_1}, A \in \mathbb{R}^{m_2\times n}, b \in \mathbb{R}^{m_2}$.
Here $\preceq$ means pointwise less than or equal; i.e., if $a \preceq b $ for $a,b \in \mathbb{R}^n$, then $a_i\leq b_i \forall i=1,2,...,n$, where the subscript indicates coordinates. Look into \href{http://cvxopt.org/examples/tutorial/qp.html}{here} for a concrete example including all details.

In last homework we want to solve the following primal problem:
\begin{align}
\min_{\boldsymbol{w}, \boldsymbol{\xi}, \rho}&\quad \frac{1}{2} {\boldsymbol{||w||}}_2^2 + \frac{1}{2}b^2- \rho + \frac{\lambda}{2} \sum_{i=1}^n \xi_i^2\label{eqn:primal}\\
\text{subject to} &\quad y_i(w^T \boldsymbol{x_i} + b) \ge \rho - \xi_i,\quad i = 1,\dotsc,n 	\nonumber
\end{align}

We derived a dual program of (\ref{eqn:primal}) in homework 2. (Have a look at the solutions if you did not solve it - we will release it soon.) Implement the dual program using CVXOPT. Compute results for both $\lambda=1$ and $\lambda=10$. The data is a toy dataset *train\_data.csv* in {\color{red}csv} format of size $\mathbb{R}^{100\times 2}$, and the training label is contained in the file *train\_label.csv* of size $\mathbb{R}^{100\times 1}$. Be careful that CVXOPT uses a slightly different matrix format than numpy; so either create your matrix in CVXOPT format, or use a numpy array and convert it into CVXOPT format using \textsf{A = cvxopt.matrix(A)}. \\
\textbf{Answer:}\\
From last homework, we know the dual form of this variant SVM as below:
$$\argmin_{\vec{\alpha}}\frac{1}{2}||\sum_{i=1}^{n}\alpha_iy_i\vec{x}_i||_2^2 + \frac{1}{2}(\sum_{i=1}^{n}\alpha_iy_i)^2 + \frac{1}{2\lambda}\sum_{i=1}^{n}\alpha_i^2 $$
s.t.
$$\sum_{i=1}^{n}\alpha_i=1$$
$$\alpha_i \geq 0$$
Therefore, we can use CVXOPT to solve this dual problem to get $\vec{\alpha}$:

Assume the data matrix and label matrix as below:
$$$$
\begin{equation}
\nonumber
\begin{array}{rcl}
X & = & [\vec{x}_1,\vec{x}_2,\dots,\vec{x}_n]^T \\
Y & = & \left[\begin{array}{cccc}
y_1 & 0 & \dots & 0 \\
0 & y_2 & \dots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \dots & y_n
\end{array}\right]
\end{array}
\end{equation}

Then we can convert the dual form into the CVXOPT form using the following configuration:
\begin{equation}
\nonumber
\begin{array}{rcl}
P & = & Y^TXX^TY+\vec{y}\vec{y}^T+I/\lambda \\
\vec{q} & = & \vec{0} \\
G & = & -I \\
h & = & \vec{0} \\
A & = & [1,1,\dots,1] \\
b & = & 1 \\
\end{array}
\end{equation}

(a) \textbf{[4 points]} Suppose you use $\boldsymbol{\alpha}$ as the Lagrange multipliers in dual program. Given the dual solution $\boldsymbol{\alpha}$, compute the primal solution $\boldsymbol{w}, \boldsymbol{\xi}, \rho$ in terms of training data, $\lambda$ and $\boldsymbol{\alpha}$. (Hint: This has been computed in last homework, and you do not need to redo them.) \\
\textbf{Answer:}\\
From last homework, we know that:
\begin{equation}
\nonumber
\sum_{i=1}^{n}\alpha_i \vec{z}_i=\left[
\begin{array}{c}
\vec{w}^* \\
b^* \\
\sqrt{\lambda}\vec{\xi}
\end{array}\right],~~where~\vec{z}_{i}=\left[\begin{array}{c}
y_{i}\vec{x}_{i}\\
y_{i}\\
\frac{1}{\sqrt{\lambda}}\vec{e}_{i}
\end{array}\right]
\end{equation}
Therefore,
\begin{equation}
\nonumber
\begin{array}{rcl}
\vec{w} & = & X^TY\vec{\alpha}\\
b & = & \vec{\alpha}^T\vec{y}\\
\vec{\xi} & = & \vec{\alpha}\frac{1}{\lambda} \\
\end{array}
\end{equation}
For any $\alpha_i\neq0$, $y_i(\vec{w}^T\vec{x}_i+b)=\rho-\xi_i$ (complementary slackness); therefore, we can derive $\rho$ as below:
$$\rho = \min_i(y_i(\vec{w}^T\vec{x}_i+b)+\xi_i)$$

(b) \textbf{[8 points]} For each value of $\lambda$, draw a scatter plot of the data and plot the decision border  (where
the predicted class label changes) as well as the boundaries of the margin (the area in which
there is a nonzero penalty for predicting any label). Use different colors for margins and border. Also use different colors for positive ($y=1$) and negative ($y=-1$) samples.\\
\textbf{Answer:}\\
See Fig.\ref{fig:svm}
\begin{figure}[!h]
	\centering
	\includegraphics[width=0.8\textwidth]{./img/svm.eps}
	\caption{The Figure of Problem C.b}
	\label{fig:svm}
\end{figure}

(c) \textbf{[4 points]} Report the test error on the test set *test\_data.csv* and *test\_label.csv* for each value of $\lambda$.\\
\textbf{Answer:}\\
\begin{itemize}
	\item $\lambda = 1$: 0.15
	\item $\lambda = 10$: 0.13
	\item $\lambda = 100$: 0.15
	\item $\lambda = 1000$: 0.15
\end{itemize}

(d) \textbf{[4 points]} What is the difference between the result obtained in $\lambda=1$ and $\lambda=10$? Why is that?
\textbf{Answer:}\\
From the figure, we know that the margin area shrinks as the $\lambda$ increases, because higher $\lambda$ will less tolerate the slackness and thus shrink the margin area to remove more points from it. The test error of the SVM with $\lambda=10$ is lower than that with $\lambda=1$; however, this decreasing trend does not hold for the SVMs with $\lambda = 100$ and $\lambda = 1000$.

\newpage

\subsection*{Appendix: Code}

\lstinputlisting[language=Python]{./Python/hw3.py}


\subsection{Notation}

$E_{21}$ is the location at row 2 and column 1, used to eliminate this value.

\subsection{Inverse}

\[AA^{-1}=I=A^{-1}A\]

Matrix multiplication is not commutative:

\[\left(A B^{-1}\right)\left(B A^{-1}\right)=A I A^{-1}=I\]

Transpose inverse fact:

\[\boxed{(A^{-1})^TA^T=I}\]

\subsection{Concept}

Given $E_{21}A=U$, where $U$ is upper-triangular, $E^{-1}_{21}A=E^{-1}_{21}U$ gives:

\[\boxed{A=LU\text{ where }L=E^{-1}_{21}}\]


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 10.3 Complex Fourier Transform

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

This subroutine computes Fourier transforms for complex data in up to
6~dimensions using the fast Fourier transform. The relation between values $z$
and Fourier coefficients $\zeta $ is defined by%
\begin{multline*}
 z(j_1,j_2,\ldots,j_{ND})= \sum_{k_1=0}^{N_1-1} \cdots\\
\sum_{k_{ND}=0}^{N_{ND}-1}
\zeta (k_1,k_2,...,k_{ND})W_1^{j_1k_1} \cdots
W_{ND}^{j_{ND}k_{ND}},\text{ and}
\end{multline*}\vspace{-20pt}
\begin{multline*}
\zeta(k_1,k_2,\ldots,k_{ND})=\frac 1N_1 \cdots \frac 1N_{ND}
\sum_{j_1=0}^{N_1-1} \cdots\\
\sum_{j_{ND}=0}^{N_{ND}-1}
 z(j_1,\ldots,j_{ND}) W^{-j_1k_1} \cdots W^{-j_{ND}k_{ND}}
\end{multline*}
where $N_\ell =2^{\text{M}(\ell)}$, $W_\ell =e^{2\pi i/N_\ell}$, $%
0\leq j_\ell,\ k_\ell \leq N_\ell -1$, and $z$ and $\zeta $ are complex.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision}

\begin{description}
\item[\bf COMPLEX]  \ {\bf A}$(N_1$, $N_2$, ..., $\geq N_{ND})$
\ \ $[N_k=2^{\text{M}(k)}]$
\item[\bf REAL]  \ {\bf S}$(\geq \max (\nu _1${\bf , $\nu _2$, ..., $\nu
_{ND})-1)$} $[\nu _k=2^{\text{M}(k)-2}]$
\item[\bf INTEGER]  \ {\bf M}$(\geq {\textstyle ND})${\bf , ND, MS}
\item[\bf CHARACTER]  \ {\bf MODE$*(\geq {\textstyle ND})$}
\end{description}
On the initial call set MS to~0 to indicate the array S($)$ does not yet
contain a sine table. Assign values to A(), MODE, M, and ND.
$$
\fbox{{\bf CALL SCFT(A, MODE, M, ND, MS, S)}}
$$
A() will contain computed results. S($)$ will contain the sine table used in
computing the Fourier transform. MS may have been changed.

\subsubsection{Argument Definitions}

\begin{description}
\item[A()]  [inout] If the argument MODE selects analysis in all dimensions,
A() contains values $z$ on entry, and Fourier coefficients $\zeta $ on exit.
If MODE selects analysis in all dimensions, A() contains Fourier
coefficients $\zeta $ on entry, and values $z$ on exit. When A() contains $z$%
, A($j_1+1$, $j_2+1$, ..., $j_{ND}+1)=z(j_1$, $j_2$, ..., $j_{ND})$, and
when A() contains $\zeta $, A($k_1+1$, $k_2+1$, ..., $k_{ND}+1)=\zeta (k_1$,
$k_2$, ..., $k_{ND})$, 0\ $\leq j_i,k_i\leq 2^{\text{M}(i)}-1$, $i=0$, 1, ..., ND.

\item[MODE]  [in] The character MODE$(k{:}k)$ selects Analysis or Synthesis
in the $k^{th}$ dimension. 'A' or 'a' selects Analysis, transforming $%
z$'s to $\zeta $'s. 'S' or 's' selects Synthesis,
transforming $\zeta $'s to $z^{\prime }s.$

\item[M($)$]  [in] Defines $N_k=2^{\text{M}(k)}$, the number of complex data points
in the $k^{th}$ dimension. Require 0$\ \leq \text{M}(k)\leq 30$ for
all $k$. No action is taken in dimensions for which M($k)=0.$

\item[ND]  [in] Number of dimensions. Require 1 $\leq $ ND $\leq $ 6.

\item[MS]  [inout] Gives the state of the sine table in $S()$.  Let
$\text{MS}_{in}\text{ and MS}_{out}$ denote the values of MS on entry
and return respectively. If the sine table has not previously been
computed, set $\text{MS}_{in} = 0$ or $-$1 before the call. Otherwise
the value of $\text{MS}_{out}$ from the previous call using the same
S() array can be used as $\text{MS}_{in}$ for the current call.

Certain error conditions described in Section E cause the subroutine
to set $\text{MS}_{out} = -2$ and return.  Otherwise, with $\max _i
\{\text{M}(i)\} > 0$, the subroutine sets $\text{MS}_{out} = \max ($M(1),
M(2), ..., M(ND), $\text{MS}_{in}).$

If $\text{MS}_{out} > \max (2, \text{MS}_{in}),$ the subroutine sets
NT = $2^{\text{MS}_{out}-2}$ and fills S() with NT $-$ 1 sine values.

If $\text{MS}_{in}=-1$, the subroutine returns after the above
actions, not transforming the data in A().  This is intended to allow
the use of the sine table for data alteration before a subsequent Fourier
transform, as discussed in Section G of Chapter~16.0.

\item[S($)$]  [inout] When the sine table has been computed, S($j)=\sin \pi
j/(2\times {\textstyle NT})$, $j=1$, 2, ..., NT $-$ 1, see MS above.
\end{description}
\subsubsection{Modifications for Double Precision}

Change SCFT to DCFT and the REAL type statement to DOUBLE PRECISION. If it
is available, one can change the COMPLEX type statement to DOUBLE PRECISION
COMPLEX. For portability, (or out of necessity) one can change the COMPLEX
statement to DOUBLE PRECISION and change the first dimension to be twice as
big. The data should then be stored in A with the imaginary parts of the
complex numbers following immediately after the real parts.  This
representation is compatible with the representation used in Fortran
90, and with most compilers that extend Fortran~77 to provide a double
precision complex type.

\subsection{Examples and Remarks}

Estimate the spectral composition of%
$$
f(t)=[\sin 2\pi (t+0.1)+4\cos 2\pi (\sqrt{2}\,t+0.3)]+0i
$$
where we make the same assumptions and use the same $\Delta t$ and N as in
the example for SRFT1. Differences between the results given here and those
obtained for SRFT1 are due to the use of sigma factors in SRFT1. (It is more
efficient to use SRFT1 when $f$ is a real function. A real function was used
here to show the effect of the sigma factors.) Note that the peaks are
slightly sharper here than they are for SRFT1, but that as one leaves the
peaks the coefficients do not tend to zero nearly as rapidly as when the
sigma factors are used. The program to do these calculations and the results
are given at the end of this chapter.

\subsection{Functional Description}

The multi-dimensional complex transform involves calling SFFT to compute
one-dimensional complex transforms with respect to each dimension. For ND $=
1$, the formulas are given by Eqs.\,(9) and (10) in Chapter~16.0. For ${%
\textstyle ND} \geq 1$, the formula for $z$ given $\zeta $ is given in
Purpose above.  More details can be found in \cite{Krogh:1970:CFT}.

\bibliography{math77}
\bibliographystyle{math77}

\subsection{Error Procedures and Restrictions}

Require 1 $\leq $ ND $\leq $ 6 and 0 $\leq $ M($k)\leq 30$ for all $k$.
MODE must have one of its allowed values. If any of these conditions are
violated, the subroutine will issue an error message using the error
processing procedures of Chapter~19.2 with severity level~2
to cause execution to stop. A return is made with $\text{MS}=-2$
instead of stopping if the statement ``CALL\ ERMSET($-$1)'' is executed
before calling this subroutine.

If the sine table does not appear to have valid data, an error message is
printed, and the sine table and then the transform are computed.

\subsection{Supporting Information}

The source language is ANSI Fortran~77.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DCFT & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
 DCFT, DFFT, ERFIN, ERMSG, IERM1, IERV1\rule[-5pt]{0pt}{8pt}}\\
SCFT & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
 ERFIN, ERMSG, IERM1, IERV1, SCFT, SFFT}\\
\end{tabular}

Subroutine designed and written by: Fred T. Krogh, JPL, October~1969,
revised January~1988.

\begcodenp

\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSCFT}\vspace{0pt}
\lstinputlisting{\codeloc{scft}}

\vspace{5pt}\centerline{\bf \large ODSCFT}\vspace{5pt}
\lstset{language={}}
\lstinputlisting{\outputloc{scft}}
\end{document}


\documentclass[letterpaper]{article}

\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{September 26, 2014}
\maketitle
if $G$ is a group, and $A\subseteq G$ and $B\subseteq G$ then $AB=\{ab|a\in A, b\in B\}\subseteq G$.

\section*{proposition}

let $G$ be a group, then $H,K$ subgroups of $G$. Assume that $h^{-1}kh\in K$ for all $h\in H$, $k\in K$ then $HK$ is a subgroup of $G$ that contain s both $H$ and $K$, in fact, $HK$ is the smallest subgroup of $G$ that contains both $H$ and $K$. Assumption only important if we are not dealing with abelian groups.

\subsubsection*{proof}
$a,b\in HK$. Write $a=h_1k_1,b=h_2k_2$ with $h_i\in H,k_i\in K$ then $a\cdot b=h_1k_1h_2k_2=h_1h_2(h_2^{-1}k_1h_2)k_2\in HK$

$a=hk, a^{-1}=(hk)^{-1}=k^{-1}h^{-1}=h^{-1}(hk^{-1}h^{-1})\in HK$

\subsection*{examples}
$S_3, H=\{(1),(12)\}, K=\{(1),(123),(132)\}, (12)(123)=(23)\in HK, (12)(132)=(13)\in HK$ so $HK=G$ and is therefore contained by G

$(\mathbb{Z},+)$, $H=a\mathbb{Z}, k=b\mathbb{Z}$, let $d=(a,b)$

claim: $a\mathbb{Z}+b\mathbb{Z}=d\mathbb{Z}$. clearly $a\mathbb{Z}\subseteq d\mathbb{Z}$, $b\mathbb{Z}\subseteq d\mathbb{Z}$.

$a\mathbb{Z}+b\mathbb{Z}$ is the smallest subgroup that contains both $a\mathbb{Z}$ and $b\mathbb{Z}$. so $a\mathbb{Z}+b\mathbb{Z}\subseteq d\mathbb{Z}$.

$d=\gcd(a,b)$ so we can write $d=ma+nb$. let $\alpha\in d\mathbb{Z}$ and write $\alpha=dt, t\in \mathbb{Z}$ then $\alpha=dt=mat+nbt\in a\mathbb{Z}+b\mathbb{Z}$. so $d\mathbb{Z}\subseteq a\mathbb{Z}+b\mathbb{Z}$
\end{document}


\section{The Ergodic Theorem}
We want to study the long-term average behaviour of a Markov chain.
Recall the strong law of large numbers:
\begin{theorem}[Strong Law of Large Numbers]
    Let $(Y_i)_{i=0,1,\ldots}$ be a sequence of i.i.d. non-negative random variables, with $\mathbb EY_i=\mu\in[0,\infty]$, then
    $$\mathbb P\left[ \frac{Y_1+\cdots+Y_n}{n}\to\mu\text{ as }n\to\infty \right]=1$$
\end{theorem}
\begin{proof}
    Omitted.
\end{proof}
The ergodic theorem is a sorta similar thing, but with some twists.
Write $V_i(n)=\sum_{k=0}^{n-1}1_{X_k=i}$ as the number of visits to $i$ before $n^{th}$ step, then
\begin{theorem}[Ergodic Theorem]\label{ergodic}
    Let $P$ be irreducible and $\lambda$ be any distribution.
    Take a Markobv chain $(X_n)\sim\operatorname{Markov}(\lambda,P)$, then
    $$\mathbb P\left[ \frac{V_i(n)}{n}\to\frac{1}{m_i}\text{ as }n\to\infty \right]=1$$
\end{theorem}
\begin{remark}
    1. This does not follow directly from the law of large numbers as $1_{X_k=i}$ are not i.i.d. random variables.\\
    2. In particular, if $P$ is positive recurrent, then we know that $\pi_i=1/m_i$, so it translated to
    $$\mathbb P\left[ \frac{V_i(n)}{n}\to\pi_i\text{ as }n\to\infty \right]=1$$
    which is pretty intuitive.
\end{remark}
\begin{proof}
    If $P$ is transient, then $\mathbb P[V_i<\infty]=1$ where $V_i=\sum_{k=0}^\infty 1_{X_n=i}$ is the total number visits to $i$.
    Therefore $m_i=\infty$ for any $i$, which means
    $$\mathbb P\left[ \frac{V_i(n)}{n}\le\frac{V_i}{n}\to 0=\frac{1}{m_i} \right]=1$$
    as desired.
    If $P$ is recurrent and $\lambda=\delta_i$, we shall show that
    $$\mathbb P\left[ \frac{n}{V_i(n)}\to m_i\text{ as }n\to\infty \right]=1$$
    Let $S_i^{(r)}$ be the $r^{th}$ excursion length between visits to $i$.
    We know $S_i^{(1)},S_i^{(2)},\ldots$ are i.i.d. from Theorem \ref{strong_markov}.
    Also $\mathbb ES_i^{(r)}=m_i$, so by strong law of large numbers,
    $$\mathbb P_i\left( \frac{S_i^{(1)}+\cdots S_i^{(n)}}{n}\to m_i\text{ as }n\to\infty \right)$$
    To see this implies what we want, note that $S_i^{(1)}+\cdots S_i^{(V_i(n))}\ge n$ and $S_i^{(1)}+\cdots S_i^{(V_i(n)-1)}\le n-1$, so by dividing both sides of both inequalities by $V_i(n)$ and using $\mathbb P[V_i(n)\to\infty]=1$, we get the desired result.\\
    For the general case for a recurrent $P$, note that we have $\mathbb P[T_i<\infty]=1$.
    Therefore consider $(X_{T_i+n})_{n\ge 0}\sim\operatorname{Markov}(\delta_i,P)$ by Theorem \ref{strong_markov}.
    It is indepenedent of $X_0,\ldots,X_{T_i}$ by the same theorem.
    The result then follows since the limit of $V_i(n)/n$ as $n\to\infty$ is not affected if $(X_n)_{n\ge 0}$ is replaced by $(X_{T_i+n})_{n\ge 0}$.
\end{proof}
\begin{corollary}
    In the case where $P$ is positive recurrent, for any bounded function $f:I\to\mathbb R$,
    $$\mathbb P\left[ \frac{1}{n}\sum_{k=0}^{n-1}f(X_k)\to\bar{f}\text{ as }n\to\infty\right]=1,\bar{f}=\sum_{i\in I}\pi_if(i)$$
\end{corollary}
This gives us ways to estimate the theorectical average of $f$ in the Markov chain by observing how it runs.
\begin{proof}
    WLOG $|f|\le 1$.
    Then for any $J\subset I$ we have
    \begin{align*}
        \left|\frac{1}{n}\sum_{k=0}^{n-1}f(X_k)-\bar f\right|&=\left|\sum_{i\in I}\left( \frac{V_i(n)}{n}-\pi_i \right) f(i)\right|\\
        &\le \sum_{i\in J}\left|\frac{V_i(n)}{n}-\pi_i\right|+\sum_{i\notin J}\left( \frac{V_i(n)}{n}+\pi_i \right)\\
        &\le 2\sum_{i\in J}\left|\frac{V_i(n)}{n}-\pi_i\right|+2\sum_{i\notin J}\pi_i
    \end{align*}
    For any $\epsilon>0$ choose $J\subset I$ finite such that $\sum_{i\notin J}\pi_i<\epsilon$ (for example just take $J=I$) and a random variable $N=N(\omega)$ large enough such that
    $$\mathbb P\left[ \sum_{i\in J}\left|\frac{V_i(n)}{n}-\pi_i\right|<\epsilon\text{ for }n\ge N \right]=1$$
    Therefore
    $$\mathbb P\left[ \left|\frac{1}{n}\sum_{k=0}^{n-1}f(X_k)-\bar f\right|<4\epsilon\text{ for }n\ge N \right]=1$$
    which shows the theorem.
\end{proof}


\section{Correlation Estimation}
\subsection{Biased and unbiased ACF and correlogram spectral}
Fig.\ref{fig:1_3_a} illustrates both unbiased and biased estimations of autocorrelation function (ACF) and correlogram spectral with WGN, filtered WGN and noisy sinusoidal signals.  Observing the ACF diagrams, the biased and unbiased estimations are same when the lag $k$ is approximately less than 200. As the lag $k$ increases, the tendency is getting to separate in aspect of biased estimation increasing in value and the unbiased tending to zero, which verifies the Eq. \ref{proof:biase} and\ref{proof:unbiase} based on the Eq. (12)-(13) in instruction.
\begin{align}
\text{biased: } \mathbb{E}\left\{\hat r_{xx}(k)\right\} & =\frac{1}{N} \sum_{n=k+1}^{N} \mathbb{E} \left\{x(n)x^*(n-k) \right \} = \frac{N-k}{N}\ r_{xx}\label{proof:biase}\\
\text{unbiased: }\mathbb{E}\left\{\hat r_{xx}(k)\right\} & =\frac{1}{N-k} \sum_{n=k+1}^{N} \mathbb{E} \left\{x(n)x^*(n-k) \right \} = r_{xx}\label{proof:unbiase}
\end{align}
As to  correlograms of these two ACF, the biased ACF guarantees the non-negative PSD due to the positive semi-definite of ACF. However, the unbiased ACF accords to true mean of PSD, resulting to highly erratic for large lags $k$. Thus, the ACF is not positive definite and causes negative values of the PSD, which are inappropriate with theory.
\begin{figure}[htb]
\centering
\includegraphics[width=\textwidth]{fig/13/13a.eps}
\caption{Standard and Bartlett Method of EEG Periodograms}
\label{fig:1_3_a}
\end{figure}\\
\subsection{PSD estimation with several realisations}
The PSD estimations of the signal $x(n)$ in Eq.\ref{proof:x} are plotted in Fig.\ref{fig:1_3_b} with 100 realisitions.
\begin{equation}
x(n)=sin(2\pi2n)+sin(2\pi3n)+1.5*sin(2\pi4n)+ \omega (n) \quad \omega \sim \mathcal{N} (0,1)
\label{proof:x}
\end{equation}
The mean and standard deciation of the PSD are highlighted. The signal $x(n)$ has three frequency components with $f_1=2 Hz$, $f_2=3 Hz$ and $f_3=4Hz$ which are successfully detected in the PSD. It is obvious that the variance and noise are reduced by taking mean and standard deviation. Howerver, the peak value of the PSD is to sharp. Thus, the interval between realisitions is narrow and indistinct when the PSD estimations is in magnitude scale.
\begin{figure}[htb]
     \centering
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/13/13b1.eps}
     \end{subfigure}
     ~
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/13/13b2.eps}
     \end{subfigure}
        \caption{PSD estimations with mean and standard deviation}
        \label{fig:1_3_b}
\end{figure}
\subsection{PSD estimation in dB}
Repeating the process in previous section, the PSD estimations are shown in Fig.\ref{fig:1_3_c} in decibels scale. Observing the peak value of realisations, it was compressed to a large extent.  As a consequence, the amplitudes of three sinusoid signals seem to be same. Meanwhile, the noise fluctuations are significantly amplified, which increases the variance of the PSD due to the logarithm. Overall, it is an admissible and advantageous presentations to plot the PSD estimations in dB since both large and small features are visible and distinct.
\begin{figure}[htbp]
     \centering
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/13/13c1.eps}
     \end{subfigure}
     ~
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/13/13c2.eps}
     \end{subfigure}
        \caption{PSD estimations in dB}
        \label{fig:1_3_c}
\end{figure}
\subsection{Peak detection with window resolution}
Fig.\ref{fig:1_3_d1} depicts the periodograms of peak detection of complex-valued signal by varying the number of sample. It is obvious that the two peaks are becoming with incremental of samples ($N$), since the frequency resolution is proportional to $\frac{1}{N}$. In this experiment, the rectangular window was applied whose 3$dB$ bandwidth is defined as $0.89(\frac{2\pi}{N})$. Therefore, with two frequencies in $0.3Hz$ and $0.32Hz$ in radian, the theoretical number of samples are $N =0.89/(0.32-0.3)=44.5$. Hence, the peaks can be successfully identified when the samples are larger than 45. However, when $N$ is 40 as shown in Fig.\ref{fig:1_3_d1}, peaks are still detected which is probably caused by the sidelobes of the window.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=1\textwidth]{fig/13/13d1.eps}
    \caption{Periodogram: peak detection of varying samples}
    \label{fig:1_3_d1}
\end{figure}\\
Fig.\ref{fig:1_3_d2} illustrates the peak detection with varying frequency interval from $0.32\sim0.35$. When fixing $N$ to 30, the theoretical frequency interval $\Delta f= 0.89/30 \approx 0.3$. As shown in Fig.\ref{fig:1_3_d2}, the results consist with the theory analyse where the peaks are detectable with $f2 \ge 0.33$.\\
\begin{figure}[htbp]
    \centering
    \includegraphics[width=1\textwidth]{fig/13/13d2.eps}
    \caption{Periodogram: peak detection of varying frequencies}
    \label{fig:1_3_d2}
\end{figure}
\subsection{Frequency estimation by MUSIC}
The Multiple Signal (MUSIC) algorithm is an subspace method which focusing on the eigenvectors. A complex signal with AWGN can be expressed as $\mathbf x(n)=\mathbf A \mathbf e +\mathbf w$ on vector notation. Thus, its autocorrelation matrix is calculated and decomposed into the sum of signal subspace and noise subspace, as shown below.
\begin{equation}
\mathbb {E}(\mathbf{x x}^H)=\mathbf R_{xx}=\mathbf{EDE^H+\sigma^2I}=\mathbf{E_sD_sE^H_s+E_nD_nE^H_n}
\label{proof:Rxx}
\end{equation}
where $\mathbf{E_s=[e_1,...e_p]}$ is the eigenvectors of signal, $\mathbf{E_s=[v_{p+1},...v_M]}$ is the eigenvectors of noise and $\mathbf{D}=diag\mathbf {[A_1,...A_p,\sigma^2_{p+1}...\sigma^2_M]}$ is the eigenvalues of subspace. Due to independence between signal and noise vectors, the signal subspace and noise subspace are orthogonal, expressing in $\mathbf {e_k^Hv_i=0}$. Therefore, the MUSIC algorithm is introduced by
\begin{equation}
{\hat P_{MU}(\omega)}=\frac {1}{\sum_{i=p+1}^M \left | \mathbf{e}^H \mathbf{v}_i\right |^2}
\label{proof:music}
\end{equation}\\
In this experiment, the function \texttt{corrmtx} is used to calculated the autocorrelation matrix $\mathbf R_{xx}$ in Eq.\ref{proof:Rxx} and \texttt{pmusic} is the MUSIC function to get the peak frequency in Eq.\ref{proof:music}. Meanwhile, the $M$ is 14 and $p$ is 2 which are defined as the total dimension of the subspace and the dimension of the signal subspace respectively.\\
Fig.\ref{fig:1_3_e} depicts the estimation of the MUSIC function which is successfully detect the peaks. Comparing with the periodogram method, the MUSIC is using biased estimation and has less variance. Moreover, it can deal with less samples signal, while the periodogram method could only use long length signal due to the window resolution. However, the dimension of signal subspace $p$ need to be determined in advance, which is not practical in general cases. On the contrast, the periodogram method does not require the information of the signal.
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.6\textwidth]{fig/13/13e.eps}
    \caption{Periodogram: MUSIC algorithm}
    \label{fig:1_3_e}
\end{figure}


\subsection{A Functional to obtain Initial Conditions}
	The interesting thing about the equation (\ref{kolmogorov}), is that there is no standard way to define an initial condition. For this problem, a functional is defined that acts in the initial condition, and because there are different ways of defining this functional, the method may change. For this work, the following functional was chosen
	\begin{align*}
		u^{z_0}_0 (g) := g(z_0), \hspace{2mm} \text{for fixed} \hspace{2mm} z_0 \in [0, 1].
	\end{align*}

	To construct the initial condition, the following set of points is considered
	\begin{align*}
		P = \{ z_i, \hspace{1mm} 0 \leq i \leq p \hspace{1mm} : \hspace{1mm} z_0 = 0, \hspace{1mm} z_p = 1 \}
	\end{align*}

	\noindent Then for each point $z_i \in P$ such that $X_0 (z_i) = X(0, z_i)$ set $u_0 (x)$ as the evaluation functional $z_i \longrightarrow X^x_t (z_i)$. Then from (\ref{solution_kolmogorov}) we obtain
	\begin{align}
		u(0, x) = \mathbb{E}[u^{z_i}_0 (X^x_0)] = X^x (0, z_i) = x(z_i)
	\end{align}
	For other hand
	\begin{align*}
		u (0, x) = \displaystyle \sum _{n \in \mathcal{J}^{M, N}} u_{n}(0) H_n (x)
	\end{align*}
	multiplying for $H_m (x)$ and integrating over space $\mathcal{L}^2 (\mathcal{H}, \mu)$
	\begin{align*}
		u_m (0) = \displaystyle \int_{\mathcal{H}} x(z_i) H_m (x) \mu (dx)
	\end{align*}

	\noindent Note that in the direction of the eigenfunction $e_k$ the expression $x$ can be written as $(x, e_k )_{\mathcal{H}} e_k$, then we can write $H_m (x) x (z_i)$ in the direction $e_k$ as $P_{m_k} (\xi_k) (x, e_k )_{\mathcal{H}} e_k (z_i)$ with $\xi_k = (x, \Lambda^{-\frac{1}{2}} e_k) = \| \lambda_k \| (x, e_k )_{\mathcal{H}}$ and $P_{m_k}$ is given by (\ref{hermite_polynomials}). Then we have
	\begin{align*}
		u^{z_i}_m (0) &= \displaystyle \int_{\mathcal{H}} x(z_i) H_m (x) \mu (dx) \\
		&= \int_{\mathbb{R}^N} \sum_{k=1}^{\infty} P_{m_k} (\xi_k) (x, e_k )_{\mathcal{H}} e_k (z_i) \mu (d\xi_1, d\xi_2, \cdots) e_k \\
		&= \int_{\mathbb{R}^N} \sum_{k=1}^{\infty} P_{m_k} (\xi_k) \frac{\xi_k}{\lambda_k} e_k (z_i) \mu (d\xi_1, d\xi_2, \cdots) e_k \\
		&= \sum_{k=1}^{\infty} \frac{e_k}{\lambda_k} \int_{\mathbb{R}}  P_{m_k} (\xi_k) \xi_k (z_i) \mu (d\xi_k)
	\end{align*}
	truncating the above expression we have
	\begin{align}
		\label{IC_approx}
		u^{z_i}_m (0) \approx \displaystyle \sum_{k=1}^{M} \frac{e_k}{\lambda_k} \int_{\mathbb{R}}  P_{m_k} (\xi_k) \xi_k (z_i) \mu (d\xi_k)
	\end{align}

	\noindent Setting the equation (\ref{IC_approx}) for each element from $u^{z_i}_m$ as $u^{z_i}_{m_j} (0) = u_j (0)$, $1 \leq j \leq M$ and by (\ref{solution_finite_system}) evaluated for $t=0$, then the initial condition can be written as
	\begin{equation*}
		\begin{pmatrix}
			u_1 (0) \\ u_2 (0) \\ \vdots \\ u_{M-1} (0) \\	u_M (0)
		\end{pmatrix}
		=
		\begin{pmatrix}
			V1 & V2 & \dots & V_{M-1} & V_M
		\end{pmatrix}
		\begin{pmatrix}
			c_1 \\ c_2 \\ \vdots \\ c_{M-1} \\ c_M
		\end{pmatrix}
	\end{equation*}
	and the constants $c_j$ are calculated as
	\begin{equation*}
		\begin{pmatrix}
			c_1 \\ c_2 \\ \vdots \\ c_{M-1} \\ c_M
		\end{pmatrix}
		=
		\begin{pmatrix}
			V1 & V2 & \dots & V_{M-1} & V_M
		\end{pmatrix}^{-1}
		\begin{pmatrix}
			u_1 (0) \\ u_2 (0) \\ \vdots \\ u_{M-1} (0) \\	u_M (0)
		\end{pmatrix}
	\end{equation*}


\section{Discrete Distributions}
\begin{definition}
    Let $(\Omega,\mathscr F,\mathbb P)$ be a probability space where $\Omega$ is a countable set (we will most likely take it as finite), $\Omega=\{\omega_1,\omega_2,\ldots\}$ and $\mathscr F=2^\Omega$.
    In order to determine $\mathbb P$, it sufficed to determine all $p_i=\mathbb P(\{\omega_i\})$.
    We call $p_i$ as a discrete distribution.
\end{definition}
Note that $p_i\ge 0$ and $\sum_ip_i=1$.
\subsection{Examples of Useful Distributions}
\begin{definition}
    $\Omega=\{0,1\}$ and $p_1=p,p_0=1-p$ gives the Bernoulli distribution $\operatorname{Bern}(p)$.
    This comes from flipped a $p$-coin.
\end{definition}
\begin{definition}
    Toss $N$ $p$-coins and count the number of $1$'s (heads).
    So $\Omega=\{0,1,\ldots,N\}$ and
    $$p_k=\binom{N}{k}p^k(1-p)^{N-k}$$
    this is called the Binomial distribution $\operatorname{Bin}(N,p)$.
\end{definition}
Note that
$$\sum_{i=0}^Np_k=\sum_{i=0}^N\binom{N}{k}p^k(1-p)^{N-k}=1$$
by Binomial Theorem.
\begin{definition}
    Consider $k$ boxes and we throw $N$ independent balls in them randomly, and $p_i$ is the probability that one of the balls fall into box $i$.
    So $\Omega=\{(n_1,n_2,\ldots,n_k)\in\mathbb N_0^k:\sum_{r=1}^kn_r=N\}$ and we have
    $$\mathbb P((n_1,n_2,\ldots,n_k))=\binom{N}{n_1,n_2,\ldots,n_k}p_1^{n_1}\cdots p_k^{n_k}$$
    This is the multinomial distribution.
\end{definition}
\begin{definition}
    Toss a fair coin until we reach a head.
    So $\Omega=\{1,2,\ldots\}$ and
    $$p_k=\mathbb P(\text{tossed $k$ coins until there is a head})=(1-p)^{k-1}p$$
    Sometimes we shift it by $1$ and say $\Omega=\{0,1,\ldots\}$ and
    $$p_k=\mathbb P(\text{tossed $k$ coins before there is a head})=(1-p)^kp$$
    This is called the geometric distribution $\operatorname{Geom}(p)$.
\end{definition}
\begin{definition}
    The Poisson distribution is used to model the number of occurences of events in a given period of time.
    For example, number of customer entering a shop in a day.\\
    We take $\Omega=\{0,1,2\ldots\}$ and
    $$\mathbb P(\{k\})=e^{-\lambda}\frac{\lambda^k}{k!}$$
    We call this the Poisson distribution $\operatorname{Pois}(\lambda)$ of parameter $\lambda$.
\end{definition}
Consider the partition of $[0,1]$ in $n$ intervals of length $1/n$ and in each interval customer arrives with probability $p$ and at most $n$ customers will arrive, then
$$\mathbb P(\text{$k$ customers arrived})=\binom{n}{k}p^k(1-p)^{n-k}$$
which is $\operatorname{Bin}(n,p)$.
But if we take $p=\lambda/n$, we have
\begin{proposition}
    $\operatorname{Bin}(n,\lambda/n)\to\operatorname{Pois}(\lambda)$ as $n\to\infty$.
\end{proposition}
\begin{proof}
    Fix $k$, then
    \begin{align*}
        \mathbb P(\{k\})&=\binom{n}{k}\frac{\lambda^k}{n^k}\left(1-\frac{\lambda}{n}\right)^{n-k}\\
        &=\frac{\lambda^k}{k!}\frac{n!}{n^k(n-k)!}(1+\frac{-\lambda}{n})^{n-k}\\
        &\to\frac{\lambda_k}{k!}e^{-\lambda}
    \end{align*}
    as $n\to\infty$, which is exactly the Poisson distribution.
\end{proof}
\subsection{Random Variables}
\begin{definition}
    Let $(\Omega,\mathscr F,\mathbb P)$ be a probability space, a random variable $X$ is a function $\Omega\to\mathbb R$ such that\\
    1. $\forall x\in\mathbb R, \{X\le x\}=\{\omega\in\Omega:X(\omega)\le x\}\in\mathscr F$.
    \footnote{Sometimes we also write $\{X\in A\}=\{\omega\in\Omega:X(\omega)\in A\}$}
\end{definition}
\begin{example}
    Given $A\in\mathscr F$, the indicator of $A$ is a function $1_A:\Omega\to\{0,1\}$ with
    $$1_A(\omega)=\begin{cases}
        1\text{, if $\omega\in A$}\\
        0\text{, otherwise}
    \end{cases}$$
    This is a random variable
\end{example}
\begin{definition}
    Let $X$ be a random variable, then the probability distribution function of $X$ to be a function $F_X:\mathbb R\to [0,1]$ given by $F_X(x)=\mathbb P(X\le x)=\mathbb P(\{\omega\in\Omega:X(\omega)\le x\})$.
\end{definition}
\begin{proposition}
    1. $F_X(x)\to 1$ as $x\to\infty$ and $F_X(x)\to 0$ as $x\to -\infty$.\\
    2. $F_X$ is increasing.\\
    3. $F_X$ is right continuous, that is,
    $$\lim_{u\to 0^+}F_X(x+u)=F_X(x)$$
\end{proposition}
\begin{proof}
    Immediate.
\end{proof}
\begin{definition}
    Let $(\Omega,\mathscr F,\mathbb P)$ be a probability space and $(X_1,X_2,\ldots,X_n)$ is called a random variable in $\mathbb R^n$ if it is a function $\Omega\to\mathbb R^n$ and for any $(x_1,x_2\ldots,x_n)\in\mathbb R^n$,
    $$\{X_1\le x_1,\ldots,X_n\le x_n\}=\{\omega\in\Omega:X_1(\omega)\le x_1,\ldots,X_n(\omega)\le x_n\}\in\mathscr F$$
    Or equivalently, each $X_i$ is a random variable in $\mathbb R$.
\end{definition}
\subsection{Discrete Random Variables}
\begin{definition}
    We call $X$ a discrete random variable if it is a real random variable and the probablity space is discrete.\\
    In this case, we define the function $p_k=\mathbb P(X=x)=\mathbb P(\{\omega\in\Omega:X(\omega)=x\})$ as the probability mass function of $X$.
\end{definition}
\begin{definition}
    The discrete random variables $X_1,X_2,\ldots,X_n$ are independent if
    $$\mathbb P(X_1=x_1,\ldots,X_n=x_n)=\mathbb P(X_1=x_1)\cdots\mathbb P(X_n=x_n)$$
\end{definition}
\begin{example}
    If we toss a $p$-coin $n$ times independently, then $p_{\omega_1,\ldots,\omega_n}=\sum_{k=1}^n p^{\omega_k}(1-p)^{1-\omega_k}$ where $(\omega_1,\ldots,\omega_n)\in\{0,1\}^n$.\\
    Define $X_k(\omega_1,\ldots,\omega_n)=\omega_k$, then $\mathbb P(X_k=1)=p$ and $\mathbb P(X_k=0)=p$, so $X_k$ has distribution $\operatorname{Bern}(p)$.
    Furthermore, any $X_k$ are independent.\\
    For $\omega=\omega_1,\ldots,\omega_n$, let $S_n(\omega)=\sum_kX_k(\omega)$, so $S_n$ counts the number of heads and has distribution $\operatorname{Bin}(n,p)$ as we have $\mathbb P(S_n=k)=\binom{n}{k}p^k(1-p)^{n-k}$.
\end{example}
\subsection{Discrete Expectation}
Consider a discrete probability space $(\Omega,\mathscr F,\mathbb P)$ and a random variable $X:\Omega\to\mathbb R$, then
\begin{definition}
    $X$ is called nonnegative if $X(\Omega)\subset\mathbb R_{\ge 0}$.
\end{definition}
\begin{definition}
    For a non-negative random variable $X$, the expectation of $X$ is the sum
    $$\mathbb E[X]=\sum_{\omega\in\Omega}X(\omega)\mathbb P(\{\omega\})$$
    Since $X$ is nonnegative, this sum is either $0$ or approaches $+\infty$.
\end{definition}
\begin{lemma}\label{expectation_formula}
    $$\mathbb E[X]=\sum_{x\in\Omega_X}x\mathbb P(X=x)$$
\end{lemma}
\begin{proof}
    Write $\Omega_X=\{X(\omega):\omega\in\Omega\}$, then we immediately have
    $$\Omega=\bigcup_{x\in\Omega_X}\{X=x\}\left( =\coprod_{x\in\Omega_X}\{X=x\} \right)$$
    from definition.
    Use this to expand the formula yields
    \begin{align*}
        \mathbb E[X]&=\sum_{\omega\in\Omega}X(\omega)\mathbb P(\{\omega\})\\
        &=\sum_{x\in\Omega_X}\sum_{\omega\in\{X=x\}}x\mathbb P(\{\omega\})\\
        &=\sum_{x\in\Omega_X}x\sum_{\omega\in\{X=x\}}\mathbb P(\{\omega\})\\
        &=\sum_{x\in\Omega_X}x\mathbb P(X=x)
    \end{align*}
    As desired.
\end{proof}
As one may expect, the expectation can be interpreted as a weighted average of the values taken by the random variable by the respective probabilities.
\begin{example}
    1. Suppose $X\sim\operatorname{Bin}(n,p)$, then
    \begin{align*}
        \mathbb E[X]&=\sum_{k=0}^nk\binom{n}{k}p^k(1-p)^{n-k}\\
        &=\sum_{k=1}^nn\binom{n-1}{k-1}p^k(1-p)^{n-k}\\
        &=np\sum_{i=0}^{n-1}\binom{n-1}{i}p^i(1-p)^{n-i-1}\\
        &=np(p+1-p)^{n-1}=np
    \end{align*}
    2. Suppose $X\sim\operatorname{Pois}(\lambda)$, then
    \begin{align*}
        \mathbb E[X]&=\sum_{n=0}^\infty n\frac{\lambda^n}{n!}e^{-\lambda}\\
        &=\lambda\sum_{n=1}^\infty\frac{\lambda^{n-1}}{(n-1)!}e^{-\lambda}\\
        &=\lambda
    \end{align*}
\end{example}
Let $X$ be a general random variable, we can try to define its expectation by decomposing the positive and negative parts, that is
\begin{definition}
    We decompose $X$ into $X_+=\max\{X,0\}=(X+|X|)/2$ and $X_-=\max\{-X,0\}=(|X|-X)/2$, then $X=X_+-X_-$ and $|X|=X_++X_-$.
    If either $\mathbb E[X_+]$ or $\mathbb E[X_-]$ is finite, we define $\mathbb E[X]=\mathbb E[X_+]-\mathbb E[X_-]$.\\
    Also, if $\mathbb E[|X|]<\infty$, we call $X$ integrable..
\end{definition}
\begin{lemma}
    We still have Lemma \ref{expectation_formula}.
\end{lemma}
\begin{proof}
    Direct calculation.
\end{proof}
\begin{proposition}
    1. $X\ge 0\implies \mathbb E[X]\ge 0$.
    In particular, if $X\ge 0$ and $\mathbb E[X]=0$, then $\mathbb P(X=0)=1$.\\
    2. Let $c\in\mathbb R$, then $\mathbb E[cX]=c\mathbb E[X]$ and $\mathbb E[c+X]=\mathbb c+E[X]$.
    In particular $\mathbb E[c]=c$.\\
    3. Let $X,Y$ be random variables, then $\mathbb E[X+Y]=\mathbb E[X]+\mathbb E[Y]$.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
2 and 3 are called \textit{linearity of expectation}.
And in fact, we can extend it and say for a sequence of random variables $X_i$ we have $\mathbb E\left[\sum_nX_n\right]=\sum_n\mathbb E[X_n]$.
\begin{proposition}
    1. $\forall A\in\mathscr F$, we have $\mathbb E[1_A]=\mathbb P(A)$.\\
    2. (Law Of The Unconscious Statistician, LOTUS) Let $g:\mathbb R\to\mathbb R$ be a function and $X$ a random variable, then
    $$\mathbb E[g(X)]=\sum_{x\in\Omega_X}g(x)\mathbb P(X=x)$$
    3. If a random variable $X$ only takes integral value, then
    $$\mathbb E(X)=\sum_{k=1}^\infty\mathbb P(X\ge k)=\sum_{k=0}^\infty\mathbb P(X>k)$$
\end{proposition}
\begin{proof}
    1 is from definition.
    For 2, we have
    \begin{align*}
        \mathbb E[g(X)]&=\sum_{y\in\Omega_{g(X)}}y\mathbb P(Y=y)\\
        &=\sum_{y\in\Omega_{g(X)}}y\mathbb P(\{\omega:g(X(\omega))=y\})\\
        &=\sum_{y\in\Omega_{g(X)}}y\mathbb P(\{\omega:X(\omega)\in g^{-1}(\{y\})\})\\
        &=\sum_{y\in\Omega_{g(X)}}\sum_{x\in g^{-1}(\{y\})}y\mathbb P(X=x)\\
        &=\sum_{x\in\Omega_X}g(x)\mathbb P(X=x)
    \end{align*}
    3 is obvious since we can decompose $X=\sum_{k=1}^\infty 1_{\{X\ge k\}}$.
\end{proof}
One of the usage of 1 above is to give the following proof of Inclusion-Exclusion.
\begin{proof}[Another Proof of Inclusion-Exclusion]
    The indicator function satisfies $1_{A^c}=1-1_A,1_{A_1\cap\cdots\cap A_n}=1_{A_1}\cdots 1_{A_n}$, therefore
    $$1_{A_1\cup\cdots\cup A_n}=1_{(A_1^c\cap\cdots\cap A_n^c)^c}=1-\prod_{i=1}^n(1-1_{A_i})$$
    Expanding and taking expectation gives the result.
\end{proof}
\begin{definition}
    Let $X$ be a random variable and $n\in\mathbb N$, we call $\mathbb E[X^n]$ the $n^{th}$ moment of $X$, if it is well-defined.
\end{definition}
\subsection{Variance}
\begin{definition}
    Let $X$ be a random variable such that $\mathbb E[X]$ exists and is finite, then the variance of $X$ is defined as $\operatorname{Var}(X)=\mathbb E[(X-\mathbb E[X])^2]$.
\end{definition}
Intuitively, the variance is the measure of how ``spread out'' the random variable is.
So a smaller variance would mean that the random variable is largely concentrated in its expected value.
Indubitably $\operatorname{Var}(X)\ge 0$, so we can define
\begin{definition}
    The standard deviation is defined as $\operatorname{SD}(X)=\sqrt{\operatorname{Var}(X)}$.
\end{definition}
\begin{proposition}
    1. If $\operatorname{Var}(X)=0$, then $\mathbb P(X=\mathbb E[X])=1$.\\
    2. Let $c\in\mathbb R$, then $\operatorname{Var}(cX)=c^2\operatorname{Var}(X)$.\\
    3. $\operatorname{Var}(X)=\mathbb E[X^2]-(\mathbb E[X])^2$.\\
    4. $\forall x\in\mathbb R,\operatorname{Var}(X)\le \mathbb E[(X-x)^2]$ and equality hold iff $x=\mathbb E[X]$.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
\begin{example}
    1. For $X\sim\operatorname{Bin}(n,p)$, we have
    \begin{align*}
        \operatorname{Var}(X)&=\mathbb E[X^2]-(\mathbb E[X])^2\\
        &=\mathbb E[X(X-1)]+\mathbb E[X]-(\mathbb E[X])^2\\
        &=\sum_{k=0}^nk(k-1)\binom{n}{k}p^k(1-p)^{n-k}+np-n^2p^2\\
        &=p^2n(n-1)\sum_{k=0}^n\binom{n-2}{k-2}p^{k-2}(1-p)^{n-k}+np-n^2p^2\\
        &=np(1-p)
    \end{align*}
    2. For $X\sim\operatorname{Pois}(\lambda)$, we can calculate
    \begin{align*}
        \operatorname{Var}(X)&=\mathbb E[X(X-1)]+\mathbb E[X]-(\mathbb E[X])^2\\
        &=\sum_{k=0}^\infty k(k-1)\frac{\lambda^k}{k!}e^{-\lambda}+\lambda-\lambda^2\\
        &=\lambda^2+\lambda-\lambda^2\\
        &=\lambda
    \end{align*}
\end{example}
\begin{definition}
    Let $X,Y$ be two random variables, we define the covariance to be
    $$\operatorname{Cov}(X,Y)=\mathbb E[(X-\mathbb E[X])(Y-\mathbb E[Y])]$$
\end{definition}
The covariance is a measure of the interdependence of $X,Y$.
\begin{proposition}
    1. $\operatorname{Cov}(X,Y)=\operatorname{Cov}(Y,X)$.\\
    2. $\operatorname{Cov}(X,X)=\operatorname{Var}(X)$.\\
    3. $\operatorname{Cov}(X,Y)=\mathbb E[XY]-\mathbb E[X]\mathbb E[Y]$.\\
    4. Suppose $c$ is a constant, then $\operatorname{Cov}(cX,Y)=c\operatorname{Cov}(X,Y),\operatorname{Cov}(c+X,Y)=\operatorname{Cov}(X,Y)$.\\
    5. $\operatorname{Var}(X+Y)=\operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y)$.
    Easy to generalize it to finite sums\\
    6. For any constant $c$, $\operatorname{Cov}(c,X)=0$.\\
    7. $\operatorname{Cov}(X+Z,Y)=\operatorname{Cov}(X,Y)+\operatorname{Cov}(Z,Y)$.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
\begin{definition}
    $X,Y$ are called independent if $\mathbb P(X=x,Y=y)=\mathbb P(X=x)\mathbb P(Y=y)$.
\end{definition}
\begin{proposition}
    Let $X,Y$ be independent and $f,g$ be nonnegative functions, then $\mathbb E[f(X)g(Y)]=\mathbb E[f(X)]\mathbb E[g(Y)]$
\end{proposition}
\begin{proof}
    \begin{align*}
        \mathbb E[f(X)g(Y)]&=\sum_{x,y}f(x)g(y)\mathbb P(X=x,Y=y)\\
        &=\sum_{x,y}f(x)g(y)\mathbb P(X=x)\mathbb P(Y=y)\\
        &=\left( \sum_xf(x)\mathbb P(X=x) \right)\left( \sum_yf(y)\mathbb P(Y=y) \right)\\
        &=\mathbb E[f(X)]\mathbb E[g(Y)]
    \end{align*}
    As desired.
\end{proof}
In particular, if $X,Y$ are independent, then $\operatorname{Cov}(X,Y)=0$.
The converse, however, is not true.
\begin{example}[Non-example]
    Let $X_1,X_2,X_3\sim\operatorname{Bern}(1/2)$.
    Define $Y_1=2X_1-1,Y_2=2X_2-1,Z_1=Y_1X_3,Z_2=Y_2X_3$.
    Now $\operatorname{Cov}(Z_1,Z_2)=0$ but $Z_1,Z_2$ are not independent as $\mathbb E[Z_1=0,Z_2=0]=1/2\neq1/4=\mathbb E[Z_1=0]\mathbb E[Z_2=0]$.
\end{example}


\documentclass[]{article}
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage[cache=false]{minted}
\usepackage{graphicx}
\usepackage{caption}
\graphicspath{ {./images/} }
\usepackage{multirow}
\usepackage{framed}
\usepackage{cancel, xcolor}
%
\usepackage{tikz}
\usetikzlibrary{calc,arrows}
\newcommand{\tikzmark}[1]{%
	\tikz[overlay,remember picture]\node(#1){};}
%
\newcommand\hcancel[2][black]{\setbox0=\hbox{$#2$}%
	\rlap{\raisebox{.45\ht0}{\textcolor{#1}{\rule{\wd0}{1pt}}}}#2}
%
\begin{document}
\section{Properties and Operations}
\subsection{Special Matrices}
	\begin{align*}
		O_{mn} &=
			\begin{bmatrix}
				0_{11} & \cdots & 0_{1n} \\
				\vdots & \ddots & \vdots \\
				0_{m1 }& \cdots & 0_{mn} \\
			\end{bmatrix}
		\\
		I &=
			\begin{bmatrix}
				1      & 0      & \cdots & 0      \\
				0      & 1      & \ddots & \vdots \\
				\vdots & \ddots & 1      & 0      \\
				0      & \cdots & 0      & 1      \\
			\end{bmatrix}
	\end{align*}
\subsection{List of Basic Property Theorems}
\subsubsection{Properties of Matrix Addition and Scalar Multiplication}
	\begin{align}
		A + B &= B + A \\
		A + (B + C) &= (A + B) + C \\
		(cd)A &= c(dA) \\
		1A &= A \\
		c(A + B) &= cA + cB \\
		(c + d)A &= cA + dA
	\end{align}
\subsubsection{Properties of Zero Matrices}
	\begin{align}
		A + O_{mn} &= A \\
		A + (-A) &= O_{mn} \\
		\text{If } cA &= O_{mn} \text{ then } c = 0 \text{ or } A = O_{mn}
	\end{align}
\subsubsection{Properties of Matrix Multiplication}
	\begin{align}
		A ( BC ) &= ( AB ) C \\
		A ( B + C ) &= AB + AC \\
		( A + B ) C &= AC + BC \\
		c ( AB ) = ( &cA ) B = A ( cB )
	\end{align}
\subsubsection{Properties of Identity Matrix}
	\begin{align}
		AI_{n} &= A \\
		I_{m}A &= A
	\end{align}
\subsection{Transposition}
	\begin{align*}
	A &=
		\begin{bmatrix}
			a & b & c \\
			d & e & f \\
			g & h & i \\
		\end{bmatrix}
	\\
	A^T &=
		\begin{bmatrix}
			a & d & g \\
			b & e & h \\
			c & f & i \\
		\end{bmatrix}
	\end{align*}
\subsubsection{Properties of Transposed Matrix}
	\begin{align}
		( A^T )^T &= A \\
		( A + B )^T &= A^T + B^T \\
		( cA )^T &= c ( A^T ) \\
		( AB )^T &= B^T A^T
	\end{align}
\subsection{Inverse Matrices}
\subsubsection{Properties of Inverse Matrices}
	\begin{align}
		( A^{−1} )^{−1} &= A  \\
		( A^k )^{−1} = A^{−1} A^{−1}  &\cdots A^{−1}  = ( A^{−1})^k \\
		( cA )^{−1} &= \frac{1}{c}A^{−1} \\
		( A^T )^{−1} &= ( A^{−1} )^T \\
		(AB)^{−1} &= B^{−1}A^{−1} \\
		AC = BC &\rightarrow A = B \text{  if C is inversible} \\
		CA = CB &\rightarrow A = B \text{  if C is inversible}
	\end{align}
Systems of Equations
$Ax = y$
have a unique solution
$x = A^{-1}b$.
	\begin{framed}
		Example: \\
		\begin{align*}
			a_{11}x_1 + a_{12}x_2 + a_{13}x_3 &= y_1 \\
			a_{21}x_1 + a_{22}x_2 + a_{23}x_3 &= y_2 \\
			a_{31}x_1 + a_{32}x_2 + a_{33}x_3 &= y_3
		\end{align*}
		\begin{center}
		$
			A^{-1}
			\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix} =
			\begin{bmatrix} x_1 & x_2 & x_3 \end{bmatrix}
		$
		\end{center}
	\end{framed}

\subsubsection{Using Gauss-Jordan Elimination}
	\begin{align*}
	A &=
		\begin{bmatrix}
			a_{11} & a_{12} & a_{13} \\
			a_{21} & a_{22} & a_{23} \\
			a_{31} & a_{32} & a_{33} \\
		\end{bmatrix}
	\\
		\begin{bmatrix}
			a & b & c &1&0&0 \\
			d & e & f &0&1&0 \\
			g & h & i &0&0&1 \\
		\end{bmatrix}
	&\rightarrow
		\begin{bmatrix}
			1&0&0& a\prime & b\prime & c\prime  \\
			0&1&0& d\prime & e\prime & f\prime  \\
			0&0&1& g\prime & h\prime & i\prime  \\
		\end{bmatrix}
	\\
	\begin{bmatrix} A & I \end{bmatrix}
	&=
	\begin{bmatrix} I & A^{-1} \end{bmatrix}
	\\
	A^{-1} &=
		\begin{bmatrix}
			a\prime & b\prime & c\prime  \\
			d\prime & e\prime & f\prime  \\
			g\prime & h\prime & i\prime  \\
		\end{bmatrix}
	\end{align*}

\subsubsection{2x2 Quick Solution}
First is a square matrix which can be inverted using a simple rearrangement and multiplication by the inverse of the \textbf{Determinant}.
	\begin{align*}
		A &=
			\begin{bmatrix}
				a & b \\
				c & d \\
			\end{bmatrix}
		\\
		A^{-1} &=
			\frac{1}{ad - bc}
			\begin{bmatrix}
				 d &-b \\
				-c & a \\
			\end{bmatrix}
	\end{align*}
\section{Elementary Matrices}
Are one single operation away from the I matrix: \\
$$
\begin{bmatrix}
	1      & 0      & \cdots & 0      \\
	0      & 1      & \ddots & \vdots \\
	\vdots & \ddots & 1      & 0      \\
	0      & \cdots & 0      & 1      \\
\end{bmatrix}
$$\\
Every elementary matrix $E$ is invertible and $E^{-1}$ is also an elementary matrix.
\subsection{Matrices as Products of Elementary Matrices}

\subsection{LU Factorization}
$ A = LU $ \\
Matrix $A$ is equal to the product of a Lower corner and Upper corner factor. \\
$$ A =
\begin{bmatrix}
	1 & -3 & 0 \\
	0 &  1 & 3 \\
	2 & -10 & 2 \\
\end{bmatrix}
$$
	$$
	\begin{bmatrix}
		1 & -3 & 0 \\
		0 &  1 & 3 \\
		0 & -4 & 2 \\
	\end{bmatrix}
	\hspace{1.5em}
	R_3+(-2)R_1 \rightarrow R_3
	\hspace{2em} E_1 =
	\begin{bmatrix}
		 1 & 0 & 0 \\
		 0 & 1 & 0 \\
		-2 & 0 & 0 \\
	\end{bmatrix}
	$$\\
	$$
	\begin{bmatrix}
		1 & -3 & 0 \\
		0 &  1 & 3 \\
		0 &  0 & 14 \\
	\end{bmatrix}
	\hspace{1.5em}
	R_3+(4)R_2 \rightarrow R_3
	\hspace{2em} E_2 =
	\begin{bmatrix}
		1 & 0 & 0 \\
		0 & 1 & 0 \\
		0 & 4 & 0 \\
	\end{bmatrix}
	$$\\
The final matrix on the left is $L$.\\
$ L = E_n \cdots E_2E_1A $ \\
$ U = E_1^{-1} E_2^{-1} \cdots E_n^{-1} $
\section{Determinants}

\subsection{Minors}
Minors are the determinant of entries in a matrix that do not share a column or row with the current position. \\
$$
\begin{bmatrix}
	a_{11} & \hcancel[red]{a_{12}} & a_{13} \\
	\hcancel[red]{a_{21}} & \textcolor{red}{a_{22}} & \hcancel[red]{a_{23}} \\
	a_{31} & \hcancel[red]{a_{32}} & a_{33} \\
\end{bmatrix}
\rightarrow
M_{22} =
\begin{vmatrix}
	a_{11} & a_{13} \\
	a_{31} & a_{33} \\
\end{vmatrix}
$$
$$
\begin{bmatrix}
	\textcolor{red}{a_{11}} & \hcancel[red]{a_{12}} & \hcancel[red]{a_{13}} \\
	\hcancel[red]{a_{21}} & a_{22} & a_{23} \\
	\hcancel[red]{a_{31}} & a_{32} & a_{33} \\
\end{bmatrix}
\rightarrow
M_{11} =
\begin{vmatrix}
	a_{22} & a_{23} \\
	a_{32} & a_{33} \\
\end{vmatrix}
$$
\subsection{Cofactors}
$ C_{ij} = (-1)^{i+j} M_{ij} $ \\
This follows the following sign pattern: \\
$$
	\begin{bmatrix}
		+&-&+&-&+&-&\cdots\\
		-&+&-&+&-&+&\cdots\\
		+&-&+&-&+&-&\cdots\\
		-&+&-&+&-&+&\cdots\\
		+&-&+&-&+&-&\cdots\\
		\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\ddots\\
	\end{bmatrix}
$$
\subsection{Determinants of Larger Matrices}
The generalized Equation: \\
$$
det(A) = |A| =
\sum_{j=1}^{n}a_{1j}C_{1j} =
a_{11}C_{11} + a_{12}C_{12} + \cdots + a_{1n}C_{1n}
$$ \\
This can be applied to any single row or column. \emph{Choose wisely.}
\subsubsection{Alternate Method}
$$
det(A) =
\left|A\right| =
	\begin{vmatrix}
		\tikzmark{s1}a_{11} & a_{12}\tikzmark{n2} \\
		\tikzmark{s2}a_{21} & a_{22}\tikzmark{n1} \\
	\end{vmatrix}
= a_{11}\tikzmark{e1}a_{22} - a_{12}\tikzmark{e2}a_{21} $$
	\begin{tikzpicture}[overlay,remember picture]
		\draw[->,blue]
			($(s1.north west)+(0.1,0.1)$) --
			($(n1.south east)+(0.1,-0.1)$) -|
			($(e1.south)+(-0.1,0)$) ;
		\draw[->,red]
			($(s2.south east)+(-0.1,0.1)$) --
			($(n2.north west)+(0.1,0.2)$) -|
			($(e2.north)+(0,0.1)$) ;
	\end{tikzpicture}

\hspace{5em} Subtract these products
$$
	\begin{bmatrix}
		\tikzmark{downS}a_{11} & a_{12} & \tikzmark{upE}a_{13} & a_{11} & a_{12} \\
		a_{21} & a_{22} & a_{23} & a_{21} & a_{22} \\
		\tikzmark{upS}a_{31} & a_{32} & \tikzmark{downE}a_{33} & a_{31} & a_{32} \\
	\end{bmatrix}
$$
\begin{tikzpicture}[overlay,remember picture]
	\draw[->,blue]($(upS.south west)+(0,0)$) -> ($(upE.north east)+(0.2,0.1)$);
	\draw[->,blue]($(upS.south west)+(1,0)$) -> ($(upE.north east)+(1.2,0.1)$);
	\draw[->,blue]($(upS.south west)+(2,0)$) -> ($(upE.north east)+(2.2,0.1)$);
	\draw[->,red]($(downS.north west)+(0,0.1)$) -> ($(downE.south east)+(0.1,0)$);
	\draw[->,red]($(downS.north west)+(1,0.1)$) -> ($(downE.south east)+(1.1,0)$);
	\draw[->,red]($(downS.north west)+(2,0.1)$) -> ($(downE.south east)+(2.1,0)$);
\end{tikzpicture}
\hspace{5em} Add these products \\
$ ∣A∣=
a_{11}a_{22}a_{33} +
a_{12}a_{23}a_{31} +
a_{13}a_{21}a_{32} −
a_{31}a_{22}a_{13} −
a_{32}a_{23}a_{11} −
a_{33}a_{21}a_{12} $

\subsection{Adjoint Matrices}
Adjoint matrices are the transpose of a matrix of Cofactors: \\
$ adj(A) =
\begin{bmatrix}
	C_{11} & C_{21} & C_{n1} \\
	C_{12} & C_{22} & C_{n2} \\
	C_{1n} & C_{2n} & C_{nn} \\
\end{bmatrix} $

\subsection{Properties of Determinants}
\begin{align}
	|A||B| &= |AB| \\
	|cA| &= c^n|A| \\
	|A| &= |A^T| \\
	|A^{-1}| &= \frac{1}{|A|} \\
	A^{-1} &= \frac{1}{|A|}adj(A) \\
\end{align}

\subsection{Cramer's Rule}
For a system of equations: \\
\begin{align*}
	a_{11}x_1 + a_{12}x_2 + a_{13}x_3 &= b_1 \\
	a_{21}x_1 + a_{22}x_2 + a_{23}x_3 &= b_2 \\
	a_{31}x_1 + a_{32}x_2 + a_{33}x_3 &= b_3 \\
\end{align*}
\begin{align*}
	x_1 &= \frac{|A_1|}{|A|} \\
	x_2 &= \frac{|A_2|}{|A|} \\
		&\cdots \\
	A_1 &=
		\begin{bmatrix}
			\color{red}{b_1} & a_{12} & a_{13} \\
			\color{red}{b_{2}} & a_{22} & a_{23} \\
			\color{red}{b_{3}} & a_{32} & a_{33} \\
		\end{bmatrix} \\
	A_2 &=
		\begin{bmatrix}
			a_{11} & \color{red}{b_{1}} & a_{13} \\
			a_{21} & \color{red}{b_{2}} & a_{23} \\
			a_{31} & \color{red}{b_{3}} & a_{33} \\
		\end{bmatrix}
\end{align*}

\subsection{Applications of Determinants}
\subsubsection{Area of a Triangle}
For a triangle with points $(x_1,y_1)$,$(x_2,y_2)$, and $(x_3,y_3)$ \\
\begin{align*}
	\text{Area} = \pm\frac{1}{2}
		\begin{vmatrix}
			x_{1} & y_{1} & 1 \\
			x_{2} & y_{2} & 1 \\
			x_{3} & y_{3} & 1 \\
		\end{vmatrix}
\end{align*} \\
This can likewise be used to solve for the volume of a
tetrahedron in 3-dimensional space. \\
\begin{align*}
	\text{Area} = \pm\frac{1}{6}
		\begin{vmatrix}
			x_{1} & y_{1} & z_1 & 1 \\
			x_{2} & y_{2} & z_2 & 1 \\
			x_{3} & y_{3} & z_3 & 1 \\
			x_4 & y_4 & z_4 & 1
		\end{vmatrix}
\end{align*} \\

\subsubsection{Colinearity}
If \\
\begin{align*}
	\begin{vmatrix}
		x_{1} & y_{1} & 1 \\
		x_{2} & y_{2} & 1 \\
		x_{3} & y_{3} & 1 \\
	\end{vmatrix}
	= 0
\end{align*} \\
Then the 3 points are on the same line.
Likewise solving the following for $x$ and $y$
will give the equation of the line that passes through the other two points. \\
\begin{align*}
	\begin{vmatrix}
		x_{1} & y_{1} & 1 \\
		x_{2} & y_{2} & 1 \\
		x & y & 1 \\
	\end{vmatrix}
	= 0
\end{align*} \\
This can also be used to test for Coplanar points in 3D space
using the same expansion to 4x4 as seen above.

\section{Vector Space}

\subsection{Properties of Vector Addition and Scalar Multiplication in a Plane}

Let $v$, $u$, and $w$  be vectors and $c$ and $d$ be scalars. \\
\begin{align}
	v + u &= u + v \\
	( u + v ) + w &= u + ( v + w ) \\
	v + u &= (v_1 + u_1),(v_2 + u_2) \\
	u + 0 &= u \\
	u + (-u) &= 0 \\
	c ( u + v ) &= cu + cv \\
	( c + d ) u &= cu + du \\
	c ( du ) &= ( cd ) u \\
\end{align} \\
\textbf{Summary of important Vector Spaces} \\
\begin{align*}
R &= \text{set of all real numbers}\\
R^2 &= \text{set of all ordered pairs}\\
R^3 &= \text{set of all ordered triples}\\
R^n &= \text{set of all n-tuples}\\
C ( − \inf , \inf ) &=
		\text{set of all continuous functions defined on the real number line} \\
C [ a, b ] &=
		\text{set of all continuous functions defined on a closed interval }[ a, b ] , \\
		& \text{where } a \neq b \\
P &= \text{set of all polynomials}\\
P_n &= \text{set of all polynomials of degree} ≤ n
		\text{(together with the zero polynomial)} \\
M_{m,n} &= \text{set of all } m × n \text{matrices} \\
M_{n,n} &= \text{set of all } n × n \text{square matrices} \\
\end{align*} \\

\subsection{Subspaces}

\textbf{The Test for Subspace} \\
\begin{enumerate}
	\item If Subspace is non-empty. (Includes the 0 vector)
	\item If $u$ and $v$ are in $W$, then $u+v$ is in $W$. (Closed by Addition)
	\item If $u$ is in $W$ and $c$ is any scalar, then $cu$ is in $W$ (Closed by Multiplication)
\end{enumerate}

\subsection{Linear Combinations of Vectors}
Vector v in vector space V is a linear combination of vectors if
it can be written in the following form: \\
$v = c_1 u_1 + c_2 u_2 + \ldots + c_k u_k$ \\
Solvable by matrix method: \\
$$ u_1 = ( u_{11}, u_{12}, u_{13}) \\
\begin{bmatrix}
	u_{11} & u_{21} & u_{31} & v_1 \\
	u_{12} & u_{22} & u_{32} & v_2 \\
	u_{13} & u_{23} & u_{33} & v_3 \\
\end{bmatrix} $$
** Note that the matrix is built transposed from the way in
which most matrices have been built in this course. \\
Once built, solve for row-echelon form. If the bottom row is
0s then the remainder has infinitely many solutions with a form
of $c_it + \ldots$ from the rest of the matrix solution.

\subsection{Spanning Sets}

A set spans $R^2$ if its coefficient matrix has a non-zero determinant. \\
Same is true of $R^3$.

\subsection{Linear Dependence and Independence}

For a set $S = {v_1, v_2, \ldots, v_n}$ \\
where there exists a non-trivial solution to: \\
$ c_1v_1 + c_2v_2 + \ldots + c_nv_n = 0 $ \\
(The trivial solution is for $c_{1 \cdots n}=0$ )\\
Is Linearly Dependant. \\
If only the trivial solution exists it is linearly-independent. \\
\textbf{Testing for Dependence} \\
If the coefficient matrix can reduced to identity matrix: \\
$$\begin{bmatrix}
	u_{11} & u_{21} & u_{31} & v_1 \\
	u_{12} & u_{22} & u_{32} & v_2 \\
	u_{13} & u_{23} & u_{33} & v_3 \\
\end{bmatrix}
\rightarrow
\begin{bmatrix}
	1 & 0 & 0 & 0 \\
	0 & 1 & 0 & 0 \\
	0 & 0 & 1 & 0 \\
\end{bmatrix}$$ \\
There will only be the trivial solution. \\
If the Gaussian reduction yields a all zero row there are infinitely many solutions, therefor a non-trivial solution, and the set is linearly-dependent.

\subsection{Basis for Vector Space}
The standard basis for $R^n \text{is } M_{n,n} $ in the identity matrix form.
The standard basis for $P_n \text{is } S={1,x,x^2,\ldots,x^n} $.

\textbf{Testing for Alternative Bases} \\
And alternative basis will be Linearly-Independent
and will be a spanning set. \\

\section{Inner Product Spaces}

\subsection{Length and Dot Product}
Vector Length is the result of a Pythagorean-like series. \\
$\left\|V_n\right\| =\sqrt{v_1^2 + v_2^2 + v_3^2 \cdots v_n^2}$ \\

Inner Product: \\
$\left\langle X,Y\right\rangle = x_1y_1 + x_2y_2 + \cdots x_ny_n$ \\

Unit Vectors have the same direction as the source vector but a length of 1. \\
$ u = \frac{v}{\left\|v\right\| } $ \\
Shown by: \\
\begin{align*}
	\left\| u\right\| &= \left\|\frac{v}{\left\|v\right\|}\right\|  \\
		&= \frac{1}{\left\|v\right\|}\left\|v\right\| \\
		&= 1 \\
	\text{*Note that: } \\
	\left\|cv\right\| &= |c|\left\|v\right\|  \\
\end{align*} \\

The distance between two vectors in $R^n$ space is:
$ d(u,v) =
\left\| u-v \right\| =
\sqrt{(u_1-v_1)^2 + (u_2-v_2)^2 \cdots (u_n-v_n)^2} $\\

\subsection{Properties of Dot Products}
\begin{align*}
	u \cdot v &= u_1v_1 + u_2v_2 + \ldots + u_nv_n \\
	u \cdot v &= v \cdot u \\
	u \cdot (v + w) &= u \cdot v + u \cdot w \\
	c(u \cdot v) = (cu) &\cdot v = u \cdot (cv) \\
	v \cdot v &= \left\|v\right\|^2 \\
	v \cdot v \geq 0 \text{, and } v \cdot v &= 0 \text{ if and only if } v = 0
\end{align*}

\subsection{Cauchy-Schwarz Inequality}
If $u$ and $v$ are vectors in $R^n$, then \\
$\left|u \cdot v\right| \leq \left\|u\right\| \left\|v\right\|$ \\
where $\left|u \cdot v\right|$
denotes the \emph{absolute value} of $u \cdot v$.
\\

\subsection{Angle Between Vectors}
The Cauchy-Schwarz inequality must be verified in order to use the following method to find the angle between two vectors. \\
$ \cos\theta = \frac{u \cdot v}{\left\| u \right\| \left\| v \right\|} $,
$ 0 < \theta < \pi$\\

\includegraphics[width=\linewidth]{AngleRef}
\includegraphics[width=0.5\linewidth]{UnitCircle}
%\includegraphics[width=0.5\linewidth]{TrigFuncRef}
*Remember $(cos\theta,sin\theta)$ \\

\textbf{Orthogonal Unit Vectors:} \\
In $R^2$, $v = (v_1, v_2)$, the Orthogonal vector is: $(v_2, −v_1)$ \\
Therefor the Orthogonal Unit vectors are:
$ \left\langle
	\frac{v_1}{\left\| v\right\| },
  - \frac{v_2}{\left\| v\right\| }
  \right\rangle $ and
$ \left\langle
  - \frac{v_1}{\left\| v\right\| },
	\frac{v_2}{\left\| v\right\| }
\right\rangle $

\subsection{Inner Products}
Axioms of Inner Products:
\begin{align*}
	\left\langle u,v \right\rangle &=
	\left\langle v,u \right\rangle \\
	\left\langle u,v+w \right\rangle &=
	\left\langle u,v \right\rangle + \left\langle u,w \right\rangle \\
	c\left\langle u,v \right\rangle &=
	\left\langle cu,v \right\rangle \\
	\left\langle v,v \right\rangle \geq 0 \text{, and }
	\left\langle v,v \right\rangle &= 0
	\text{ if and only if } v=0 \\
\end{align*}

\subsection{Definitions in Inner Product Space}
\begin{align*}
	\text{\textbf{Length} of $u$: } \left\| u\right\| &=
		\sqrt{\left\langle u,u \right\rangle} \\
	\text{\textbf{Distance} between $u$ and $v$: } d(u,v) &=
		\left\| u-v \right\|  \\ &=
		\sqrt{\left\langle u-v,u-v \right\rangle} \\
	\text{\textbf{Angle} between $u$ and $v$: } \cos\theta &=
		\frac{\left\langle u,v \right\rangle}
			{\left\| u\right\| \left\| v\right\|} \\
	\text{$u$ and $v$ are orthogonal when }\left\langle u,v\right\rangle &=0 \\
	\text{Cauchy-Schwarz Inequality: }
		\left| \left\langle u,v \right\rangle \right| &\leq
		\left\| u\right\| \left\| v\right\| \\
	\text{Triangle inequality: } \left\| u + v\right\| &\leq
		\left\| u\right\| + \left\| v\right\| \\
	\text{Pythagorean Theorem: $u$ and }&
	\text{$v$ are orthogonal if and only if} \\
		\left\| u + v\right\|^2 &= \left\| u\right\|^2 \left\| v\right\|^2 \\
\end{align*}

\subsubsection{Orthogonal Projection}
Let $u$ and $v$ be vectors in an inner product space $V$, such that $v \neq 0$.
Then the orthogonal projection of $u$ onto $v$ is \\
$\text{proj}_vu = \frac
	{\left\langle u,v \right\rangle}
	{\left\langle v,v \right\rangle}
v $

\subsection{Gram-Schmitz Orthonormalization Process}
\begin{enumerate}
	\item Let $B = \left\lbrace  v_1,v_2, \ldots, v_n\right\rbrace $
		be a basis for product space $V$.
	\item Let $B' = \left\lbrace w_1,w_2, \ldots, w_n\right\rbrace $, where \\
		$w_1 = v_1$ \\
		$w_2 = v_2 - \frac{v_2 \cdot w_1}{w_1 \cdot w_1}w_1 $ \\
		$w_3 = v_3 - \frac{v_3 \cdot w_1}{w_1 \cdot w_1}w_1
				   - \frac{v_3 \cdot w_2}{w_2 \cdot w_2}w_2 $ \\
		$w_n = v_n - \frac{v_n \cdot w_1}{w_1 \cdot w_1}w_1
		           - \frac{v_n \cdot w_2}{w_2 \cdot w_2}w_2 - \cdots
		           - \frac{v_n \cdot w_n}{w_n \cdot w_n}w_n $ \\
		$B'$ is the \emph{orthoganal basis} for $V$.
	\item Let $u_i = \frac{w_i}{\left\| w_i\right\| } $ and \\
		$B" = \left\lbrace  u_1,u_2, \ldots, u_n\right\rbrace $ \\
		$B"$ is the \emph{orthonormal basis} for $V$.
\end{enumerate}

\section{Eigenvectors and Eigenvalues}
\subsection{Definitions}
$ Ax = \lambda x $ \\
Where $A$ is an $n \times n$ matrix, \\
$\lambda$ is the Eigenvalue, and \\
$x$ is the non-zero Eigenvector.

\begin{align*}
	\left| \lambda I - A \right| &= 0 \\
	( \lambda I - A )x &= 0 \\
\end{align*}

The \textbf{characteristic equation} is $\left| \lambda I - A = 0 \right| $ solving this from polynomial form will give Eigenvalues.

\subsubsection{Examples}
\textbf{Eigenvalues:}\\
$$ A =
	\begin{bmatrix}
		a & b \\ c & d \\
	\end{bmatrix} $$ \\
The characteristic polynomial is: \\
$$ \left| \lambda I - A \right| =
	\begin{vmatrix}
		\lambda -a & -b \\ -c & \lambda -d \\
	\end{vmatrix} =
	\lambda^2 -a\lambda -d\lambda + (-b)(-c) =
	(\lambda + i)(\lambda + j) = 0 $$ \\
The result is that Matrix $A$ has the Eigenvalues $\lambda_1 = -i$
and $\lambda_2 = -j$. \\
\textbf{Eigenvectors:}\\
$$ (\lambda_1)I-A =
	\begin{bmatrix}
		\lambda_1 -a & -b \\ -c & \lambda_1 -d \\
	\end{bmatrix} $$
Reduce this matrix into Row-Echelon form, or as near as possible.
$$ 	\begin{bmatrix}
		\lambda_1 -a & -b \\ -c & \lambda_1 -d \\
	\end{bmatrix}
		\rightarrow
	\begin{bmatrix}
		i & j \\ 0 & 0 \\
	\end{bmatrix} $$
Each row is set equal to zero and a variable will be assigned. \\
In this case $x_2 = t $ \\
$$ x_1 - 4x_2 = 0 \\
	x =
	\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} =
	\begin{bmatrix} 4t \\ t \end{bmatrix} =
	t\begin{bmatrix} 4 \\ 1 \end{bmatrix},
	t \neq 0 $$ \\
This must be done for each Eigenvalue.

\section{Linear Programming}

\subsection{Systems of Linear Equations}
\textbf{Linear Equation between two points:} \\
$ (y_1-y_2)x + (x_2-x_1)y + x_1y_2-x_2y_1 = 0 $

\end{document}


\documentclass[]{article}
\usepackage{amssymb,amsmath}
\begin{document}
\section{Complete Elliptic Integrals of First, Second, and Third
Kind}
\begin{eqnarray*}
\mathbf{G}(k',p,a,b) & = & \displaystyle
\int_0^\infty \frac{a+b\xi^2}{(1+p\xi^2)\sqrt{(1+\xi^2)(1+{k'}^2\xi^2)}}
\, d\xi  \\[3mm]
& = & \displaystyle \int_0^{\pi/2} \frac{a \cos^2 \phi + b \sin^2 \phi}
{\cos^2 \phi + p \sin^2 \phi} \frac{d\phi}
{\sqrt{\cos^2 \phi + {k'}^2 \sin^2 \phi}} \qquad ({k'}^2 > 0).
\end{eqnarray*}
For $p < 0$, this integral is defined by its principal value.
See {\bf Notes} for special cases.

{\bf The functions K(k) and E(k):}
\begin{eqnarray*}
\mathrm{K}(k) & = & \displaystyle \int_0^{\pi/2}
\frac{d\psi}{\sqrt{1-k^2\sin^2 \psi}} \qquad (|k| < 1), \\
\mathrm{E}(k) & = & \displaystyle \int_0^{\pi/2}
\sqrt{1-k^2\sin^2 \psi} \, d\psi \qquad (|k| \le 1).
\end{eqnarray*}

Other common definitions of the complete elliptic integrals and their
relations to $\mathbf{F}_1^*$, $\mathbf{F}_2^*$, $\mathbf{F}_3^*$ are
listed here for convenience ($k^2+{k'}^2 = 1$): \\[2mm]
{\bf First kind:}
$$ \begin{array}{rcl}
F(k,\pi/2) & = & \mathrm{K}(k) \quad = \quad \mathbf{F}_1^*(k') \qquad
(|k| < 1), \\[6mm]
\widehat{F}(1,k) & = & \displaystyle \int_0^1
\frac{d\eta}{\sqrt{(1-\eta^2)(1-k^2\eta^2)}} \quad = \quad
\mathbf{F}_1^*(k') \qquad (|k| < 1).
\end{array} $$
{\bf Second kind:}
$$ \begin{array}{rcl}
E(k,\pi/2) & = & \mathrm{E}(k) \quad = \quad \mathbf{F}_2^*(k',1,{k'}^2)
\qquad (|k| \le 1), \\[6mm]
\widehat{E}(1,k) & = & \displaystyle \int_0^1
\sqrt{\frac{1-k^2 \eta^2}{1-\eta^2}} \, d\eta \quad = \quad
\mathbf{F}_2^*(k',1,{k'}^2) \qquad (|k| \le 1).
\end{array} $$
{\bf Third kind:}
$$ \begin{array}{rclcl}
\Pi(\pi/2,h,k) & = & \displaystyle \int_0^{\pi/2}
\frac{d\psi}{(1+h\sin^2 \psi)\sqrt{1-k^2\sin^2 \psi}} & = &
\mathbf{F}_3^*(k',h+1) \qquad (|k| < 1), \\[6mm]
\widehat{\Pi}(1,h,k) & = & \displaystyle \int_0^1
\frac{d\eta}{(1+h\eta^2)\sqrt{(1-\eta^2)(1-k^2\eta^2)}} & = &
\mathbf{F}_3^*(k',h+1) \qquad (|k| < 1).
\end{array} $$

{\tt FUNCTION} subprograms \\
User Entry Names:

The redundant parameter {\tt AK2} in {\tt RELI3C} and {\tt DELI3C}
permits improved accuracy when $k^2$ is small, i.e. $k' \approx 1$. In
this case, $\mathtt{AK2} = k^2$ should be calculated using
higher-precision arithmetic and then truncated before calling the
subprogram.
Special examples are
$$\begin{array}{rcl}
\mathrm{K}(k)                           & = & \mathbf{G}(k',1,1,1), \\[1mm]
\mathrm{E}(k)                           & = & \mathbf{G}(k',1,1,{k'}^2)\\[2mm]
(\mathrm{K}(k)-\mathrm{E}(k))/k^2       & = & \mathbf{G}(k',1,0,1), \\[3mm]
(\mathrm{K}(k)-{k'}^2\mathrm{E}(k))/k^2 & = & \mathbf{G}(k',1,1,0), \\[4mm]
\Pi(h,k)                                & = & \mathbf{G}(k',h+1,1,1),\\[5mm]
(\mathrm{K}(k)-\Pi(h,k))/h              & = & \mathbf{G}(k',h+1,0,1),
\end{array} $$
If $ab \ge 0$ then $\mathbf{G}$ will evaluate any linear
combination of K$(k)$, E$(k)$, $\Pi(h,k)$ without cancellation
(such as would occur, for example, if (K$(k)-$E$(k))/k^2$ were to be
computed from values of K$(k)$ and E$(k)$ which had been computed
separately.

Other functions which can be represented by $\mathbf{G}$ are the Jacobian
Zeta function $\mathbf{Z}(\Phi,k)$ and the Heuman Lambda function
$\Lambda_0(\Phi,k)$ (see Ref. 5):
$$\begin{array}{rcl@{\qquad}l}
\mathbf{Z}(\Phi,k) & = & \displaystyle k^2 \,
\frac{\sin \Phi \cos \Phi}{\mathrm{K}(k)} \, \mathbf{G}(k',q,0,\sqrt{q})
& (q = \cos^2 \Phi + {k'}^2 \sin^2 \Phi) \\[4mm]
\Lambda_0(\Phi,k) & = & \displaystyle \frac{2}{\pi}
\sqrt{q} \sin \Phi \ \mathbf{G}(k',q,1,{k'}^2) &
(q = 1 + k^2 \tan^2 \Phi).
\end{array} $$
{\it (Quoted from Ref. 3, slightly modified)}.\\

The subprograms for $\mathbf{F}_1^*$, $\mathbf{F}_2^*$ are based on the
Algol60 procedures {\it cel1, cel2} in Ref. 1, those for
$\mathbf{F}_3^*$ on {\it cel3} in Ref. 2, and those for $\mathbf{G}$
on {\it cel} in Ref. 3.

\begin{enumerate}
\item R. Bulirsch, Numerical calculation of elliptic integrals and
elliptic functions, Numer. Math. {\bf 7} (1965) 78--90.
\item R. Bulirsch, Numerical calculation of elliptic integrals and
elliptic functions II, Numer. Math. {\bf 7} (1965) 353--354.
\item R. Bulirsch, Numerical calculation of elliptic integrals and
elliptic functions III, Numer. Math. {\bf 13} (1969) 305--315.
\item W.J. Cody, Chebyshev approximations for the complete elliptic
integrals $K$ and $E$, Math. Comp. {\bf 19} (1965) 105--112.
\item P.F. Byrd and M.D. Friedman, Handbook of elliptic integrals
for engineers and scientists, 2nd ed., Springer-Verlag Berlin (1971)
33--37.
\end{enumerate}
\end{document}


\subsection{Other}

\subsubsection{Production functions}

A firm produces \(Q\) using inputs \(X\).

\(Q=f(X_1,...,X_n)\)

\subsubsection{Marginal products}

This is the marginal utility, adapted for the production setting.

\(MP=\dfrac{\delta }{\delta x_1}f(\mathbf x)\)

\subsubsection{Diminishing marginal returns}

This says that marginal returns decrease as the use of a factor increases.

\(\dfrac{\delta^2 }{{\delta x_1}^2}f(\mathbf x)<0\)


\section{Decision Trees}
\begin{frame}
   \frametitle{Decision Trees}
   Decision tree inducers provide an algorithm to solve classification and regression problems.
   \begin{itemize}
      \item  A binary decision tree $T$ splits the data: $s_T(x) = x_{i_T} \leq t_{T}$ unless $T$ is a leaf
      \item If $T$ is a leaf, it predicts $T(x) = c_T$ for some constant $c_T$
      \item If $T$ is not a leaf, it creates two subtrees $T_{left}$ and $T_{right}$, and predicts:
      \[
      T(x) =
            \begin{cases}

               T_{left}(x) &\quad\text{if}\:  s_T(x) = \text{TRUE} \\
               T_{right}(x) &\quad\text{if}\:  s_T(x) = \text{FALSE} \\
            \end{cases}
      \]
      \item This defines the capacity of the model
   \end{itemize}
\end{frame}

\begin{frame}
   \frametitle{Decision Trees - Example}
   \begin{center}
      \includegraphics[height=150px]{img/true_vs_learned_regulartree.png}
   \end{center}
   \textit{Left:} Training data of an artificial classification problem.
   \textit{Right:} Learned function of a decision tree
\end{frame}

\begin{frame}
   \frametitle{Decision Trees - Learning}
   \begin{itemize}
   \item Fitting a tree is an optimization problem
   \item Many exact solutions are NP hard: e.g. finding a minimal tree that fits the data
   \item Instead of exact solutions: greedy algorithms (bottom up vs top down)
   \item Different loss functions have been proposed: \newline
   InformationGain, Gini Index,  Likelihood-Ratio Chi–Squared Statistics, DKM Criterion, Gain Ratio, ...- see \cite{rokach_decision_2005}
   \item This project evaluates the Information Bottleneck as loss function
   \end{itemize}
\end{frame}

\begin{frame}[fragile]
   \frametitle{Decision Trees - Top Down}
   \begin{lstlisting}[language=Python, basicstyle=\small]
class DecisionTree():
  def fit(self, X, Y):
    best_loss = infinity
    for d in range(X.shape[1]): # O(D)
      loss, thresh, left_split, right_split = \
        best_split(X[:,d], Y) # O(N)
      if loss_d < best_loss:
        update best_loss, X_l, Y_l, X_r, Y_r, t_T, i_T
    if stopping_criterion is fulfilled:
      self.c_T = best_constant_estimator(Y)
      return self
    self.left = DecisionTree().fit(X_l, Y_l)
    self.right = DecisionTree().fit(X_r, Y_r)
    self.prune()
    return self
\end{lstlisting}

Important property of the loss: $J(Y, T(X)) = \frac{|Y_{left}|}{|Y|}J(Y_{left}, T(X_{left})) + \frac{|Y_{right}|}{|Y|}J(Y_{right}, T(X_{right}))$
\end{frame}


\subsection{Numerical Experiments}
Now that we've finally slogged through the presentation of the theory, we can
produce a few numerical experiments. Take, for example, the linear least squares
regression problem with $A \in \mathbb{R}^{n \times m}, x \in \mathbb{R}^m$ and
$b \in \mathbb{R}^n$:
\[
  f(x) = \frac{1}{2}\norm{Ax-b}_2^2
  = \sum_{k=1}^n
  \underbrace{ \frac{1}{2} (A_ix - b_i)^2 }_{f_i(x)}
\]
This is a particularily interesting example, as when we examing the stochastic
descent direction of this loss function:
\[
  \nabla f_i(x) = A_{i*}^T(A_{i*}x - b_i),
  \text{ where }
  A =
  \begin{bmatrix}
    A_{1*} \\
    \vdots \\
    A_{n*}
  \end{bmatrix}
\]
we notice that the sparsity pattern of $\nabla f_i(x)$ is entirely controlled by
by the sparsity pattern of the rows of $A$. This makes this problem
particularily convinient when trying to understand the effect of sparsity on the
asynchronous noise generated by \hogwild. The original paper \cite{2011NRRW}'s
results required strict assumptions on the sparsity pattern of $\nabla f_i(x)$
in order to guarantee convergence, but as we saw in Theorem
\ref{thm:convexconv}, as long as certain regularity properties are satisfied,
there's no need for such an assumption. Indeed $f$ does satisfy the above,
assuming that $A$ is of full rank%
\footnote{
  One subtle note is that the second condition is actually equivalent to having
  upper bounded maximal eigenvalue. I think we proved this back when we were
  discussing Nesterov's method, and also can be found at this stackexchange
  \url{https://math.stackexchange.com/a/1699082/245618}.
}. Therefore, when $A$ is both sparse and dense, we should see
a $\mathcal{O}(1/k)$ convergence rate, and indeed see Figure
\ref{fig:convergence} for the validation of that.
\begin{figure}[!htb]
  \centering
  \includegraphics[width=.8\textwidth]{./resources/convergence}
  \caption{
    The matrix Sparse$(d = 0.1)$ is generated by the command {\tt
    sprand(n,m,.01)}. Playing with the density parameter doesn't change the
    shape of the convergence line, but note that denser matrices (since the
    entries are Gaussian) produce a higher initial loss and suffer from more
    asynchronous noise, and therefore require more iterations.
  } \label{fig:convergence}
\end{figure}

\subsubsection{Asynchronous Noise as a Function of Bandedness}
So now that we've seen that, indeed, dense updates enjoy the same convergence
properties as sparser ones, we can also investigate the relationship of
asynchronous noise as a function of sparsity. Since the sampling of the next
data point to be used in a stochastic iterate is done uniformly and
independently, we note that in the interest of minimizing asynchronous noise,
the exact pattern matters less than the number of non-zero elements. Therefore,
an interesting way to represent different sparsity patterns in our linear
regression, is in choosing $A$ banded, and then varying the size of that band.
\begin{definition}
  Let $k \geq 1$ and $A \in \mathbb{R}^{n \times n}$. We denote $A$ as
  a $(k-1)$-banded matrix if $k$ is the maximal integer greater than zero such
  that either the $k$th or the $-k$th diagonal are not-identically zero.
\end{definition}
Now consider the operation of \hogwild\ with two simultaneous threads on the
loss function defined above, with $A$ $k$-banded. Then an lower-bound to the
probability that these two threads do not share an component of $x_i$, which
they need to read/write from/to, is (assuming sampling with replacement) the
probability of picking $i_1, i_2$ uniformly from $[|k+1, \dots, n-k|]$ such that
$[|i_1-k, i_1+k|] \cap [|i_2-k, i_2+k|] = \varnothing$. Via a simple counting
argument, we find that this is:
\[
  \Prob{[|i_1-k, i_1+k|] \cap [|i_2-k, i_2+k|] = \varnothing}
  =
  \frac{\left((n-2k)-(2k+1)\right)_+}{n-2k}
  =
  \frac{\left(n-4k-1\right)_+}{n-2k}
\]
where $(\cdot)_+ = \max(0, \cdot)$. This lower-bound gives us an upperbound on
the probability that they do share a necessary component, one minus the above.
This confirms something we already know, that if $k << n$, then the two threads
are very unlikely to have asynchronous noise. However, with even just $k = n/4$,
then the above probability is zero, and this is just for $P = 2$. I had wanted
to calculate the $k$ taking the above to zero as a function of $P$, but the
analysis quickly becomes intractible for $P \geq 3$, barring a nice counting
argument I don't see. Regardless, as long as we can measure asynchronous noise,
we can get an idea of how it increases as a function of $k$.

To construct an experiment to see this, let $x_k$ ($k$ not a component, but an
parameter) be the final iterate of \hogwild, as applied to the linear regression
problem with $A$ $k$-banded. Then supposing we hold some solution $x^*$ constant
among all $k$, then we can view $f(x_k)$ as a random-variable, whose variance
characterizes the amount of asynchronous noise experienced throughout
computation, as the sequential algorithm (assuming the random choice of
stochastic data points is seeded between intervals) will have $\Var{f(x_k)} = 0,
\forall k$. See Figure \ref{fig:variances} for the results of such an
experiment.
\begin{figure}[!htb]
  \centering
  \includegraphics[width=.8\textwidth]{./resources/banded_asyncnoise}
  \caption{
  } \label{fig:variances}
\end{figure}