\section{Introduction}
We are going to analyse the ring of $p$-adic integers $\hat{\zz}_p$ for a given prime number $p \geq 2$. To establish this algebraic ring, there are several possibilities.
\subsection{Basic ring theory}
For clarity, we are going to repeat some important definitions of ring theory. We make usage of categorical notations as found in (Strickl. 2000, Formal groups and formal group laws).
\subsubsection{Rings and prime ideals}
We call a set $R$ a ring if it is an abelian group $(R,m_+,0,S_-)$ with structure maps
$$\bao{rrcl}
+ :& R \times R&\longrightarrow& R\\
&(r,s) &\longmapsto&r + s\\
&&&\\
0 :& \ast & \longrightarrow&R\\
&x&\longmapsto&0\\
&&&\\
- :& R &\longrightarrow&R\\
&r &\longmapsto&-r\\
\ea$$
and $(R,m_\cdot)$ is a semigroup such that the following diagram commutes:
$$\bao{cc}\xymatrix{
R^4 \ar[rrr]^{id_R \times \tau_{R^2} \times id_R} &&& R^4 \ar[d]^{m_\cdot \times m_\cdot}&\\
R^3 \ar[d]_{id_R \times m_+} \ar[u]^{\Delta \times id_{R^2}} &&&R^2\ar[d]^{m_+}\\
R^2\ar[rrr]^{m_\cdot}&&&R\\
} & \bao{c}
\trm{where}\bao{rrcl}
\Delta : & R & \longrightarrow& R^2\\
&r&\longmapsto&(r,r)\\
&&&\\
\tau_{R^2} :& R^2 &\longrightarrow&R^2\\
&(r,s)&\longmapsto&(s,r).\\
\ea\ea
\ea$$
This is just a rephrasing of distributivity. Clearly, $m_+$ and $m_\cdot$ are addition and multiplication, $S$ is the additive inversion and $0$ is the zero map. If no ambiguity can occur, we simply use the standard symbols for those operation maps.  Let $(R,+_R, 0_R, -_R,\cdot_S)$ and $(S,+_S,0_S,-_S,\cdot_S)$ be two rings. We call a map $f : R \longrightarrow S$ a ring homomorphism if
$$\xymatrix{
R^2 \ar[r]^{f \times f} \ar[d]_m & R^2\ar[d]^m\\
R \ar[r]_{f}&R\\
}$$
commutes for both operation maps $+, \cdot : R^2 \longrightarrow R$. The class of all rings forms a category with morphisms called ring homomorphisms. We call a ring unital, if there is a map $1 : \ast \longrightarrow R, x \longmapsto 1$ such that $(R,\cdot,1)$ is a monoid. Again, if no ambiguity can arise we simply denote a ring by its set. For a unital ring $R$, we denote by $R^\times$ the group of units:
$$R^\times := \{x \in R : \exists y \in R, x y = 1\} = \pi_1(\mathcal{M}_1),$$
$$\ \pi_1 : R \times R \longrightarrow R, (r,s) \longmapsto r\ \trm{and}\ \mathcal{M}_1 := \{(x,y) \in R^2 : x y = 1\}.$$
\subsubsection{Modules and ideals}
\indent For a ring $R$, we call an abelian group $(M,+_M,0_M,-_M)$ an left $R$-module if there is a map
$m_\cdot : R \times M \longrightarrow M$ such that both diagrams
$$\bao{c}
\xymatrix{
R^4 \times M^2 \ar[rrr]^{id_{R^2} \times \tau_{R^2 \times M} \times id_{M^2}}& && (R^2 \times M)^2 \ar[d]^{+_R \times id_M \times +_R \times id_M}\\
R^2 \times M^2 \ar[d]_{+_R \times +_M} \ar[u]^{\Delta_{R^2} \times id_{M^2}} &&&(R \times M)^2\ar[d]^{m_+}\\
R\times M\ar[rrr]^{m_\cdot}&&&M\\
}\\
\xymatrix{
R^2 \times M \ar[r]^{id_R \times m_\cdot} \ar[d]_{\cdot_R \times id_M} & R \times M\ar[d]^{m_\cdot}\\
R \times M \ar[r]_{m_\cdot} &M\\
}
\ea$$
commute. We remark that the first diagram represents distributivity in both, $R$ and $M$. The latter represents the successive scalar multiplication commutes with multiplication of $R$ (i.e. $m_\cdot(r,m_\cdot(s,m)) = m_\cdot(rs, m)$). The map $m_\cdot$ is called the outer left product or scalar left multiplication. Similarily, we define a right $R$-module for $m_\cdot : M \times R \longrightarrow M$. An $R$-submodule $N$ of $M$ is a subset that is itself a module (i.e. the restrictions of the structure maps have images in product sets of $N$). The class of $R$-modules forms a category, with $R$-linear maps as morphisms.
\paragraph{Ideal}
An ideal $I \subset R$ is simply an $R$-submodule of $R$. Alternatively, we could have defined an ideal $I$ to be an abelian subgroup of $R$, that is stable under $R$-action: $R.I \subset I$. We call $R$ simple if $\left<0\right>$ and $R$ (if unital) are the only ideals in $R$. We call an ideal $I$ proper if $I \neq R = \left<1\right>$. A proper ideal $I$ is called prime if for all $a, b \in R$ holds $a b \in I$ implies $a \in I$ or $b \in I$. We call a proper ideal $I$ maximal if for any ideal $J \supset I$  we have either $J = I$ or $J = R$. Two ideals $I, J \subset R$ are called coprime if
$$I + J = \left<1\right>.$$
\begin{prop}
In a unital commutative ring, there always exists a maximal ideal and every maximal ideal is prime.
\end{prop}
\bws Proving the first statement we are considering the set $R^0 := R\backslash R^\times$. Then we construct the set of ideals $\mathcal{I}$ in $R^0$. Clearly, $\{0\} \in \mathcal{I}$. Then any element $x \in R^0 \backslash \{0\}$ generates an ideal in $R^0$. Thus
$$\mathcal{I} \supset \mathcal{I}_0 := \{R.x : x \in R^0\}.$$
Now we pick $I = R.x$ and $J = R.x'$ being not coprime then its sum is a proper ideal $I + J \subset R^0$. Therefore, we may attach
$$\mathcal{I}_0 \cup \left\{\sum_{x \in R^0} R.x : \sum R.x \cap R^\times = \emptyset\right\} =: \mathcal{I}.$$
It is not obvious that this is already the family of all ideals in $R^0$. However, since each $R$-combination of elements in $R^0$ is either a unit or a non-unit we have that
$$\forall I \in \mathcal{I} \wedge \forall y \in I \Rightarrow y \in R^0.$$
%We may repeat the this procedure in the following way:
%$$R^i = \bigcup_{I \in \mathcal{I}_i} I,\ \mathcal{I}_i = \mathcal{I}_{i-1} \cup \left\{\sum_{x \in R^{i-1}} R.x : \sum R.x \cap R^\times = \emptyset\right\}$$
%until no more proper ideals are left to be added.
Now we get a partially ordered set
$$(\mathcal{I}, \leq),\ \leq = \left\{(I,J) \in \mathcal{I}^2 : I \supset J\right\}.$$
Each chain of ideals $\mathcal{I}_I := \{I_i : \ldots \supset I_i \supset I_{i+1} \supset \ldots\}$ is a totally ordered set. Thus, by Zorns lemma we have that each $\mathcal{I}_I \subset \mathcal{I}$ has a lower bound. By definition it is the largest $R$-submodule in $R^0$ or equivalently a maximal ideal.\\
\indent To see that a maximal ideal is prime we simply mention that the factor ring of a maximal ideal is an algebraic field, hence an integral domain.
\paragraph{Spectrum}
%For a ring $R$ we can call an ideal $I$ prime if and only if $R/I$ is an integral domain, i.e. zero is the only $R$-torsion. Furthermore, an ideal $I$ is maximal if and only if $R/I$ is an algebraic field.
Prime ideals are usually denoted by $\mathfrak{p}$, $\mathfrak{q}$ - maximal ideals by $\mathfrak{m}$. Let $\mathcal{M}_R := \mathcal{I} \cup \{\left<1\right>\}$ denote all ideals in $R$. We call the set of prime ideals
$$\trm{Spec}(R) := \{\mathfrak{p} \in \mathcal{M}_R : \trm{Ann}_R(R/\mathfrak{p}) = \left<0\right>\}$$
the spectrum of $R$. We remark that the set of maximal ideals is a subset of $\trm{Spec} R$ in all unital ring, as each field is obviously an integral domain. Lastly, we have that
$$R \simeq \prod_{\mathfrak{p} \in \trm{Spec} R} R/\mathfrak{p}.$$


\documentclass[11pt, a4paper]{article}

\usepackage{tikz}
\usepackage{amsmath}
\usepackage{placeins}
\usepackage{booktabs}

\begin{document}

\title{INDEPENDENT COMPONENT ANALYSIS}
\date{}
\maketitle

Independent Component Analysis is a statistical generative model which aims to reveal hidden independent additive components from multi-dimensional data.

\section{Mathematical Preliminaries}

\subsection{Single Random Variable}

Let the Probability Mass Function of a discrete random variable X be defined as follows.

\begin{table}[htbp]
	\centering
	\begin{tabular}{|c|c|c|c|c|}
		\toprule
		         & $x=1$ & $x=2$ & $x=3$ & $x=4$ \\
		\midrule
		$P(X=x)$ & 1/6   & 1/3   & 1/3   & 1/6   \\
		\hline
	\end{tabular}
\end{table}

The expectation $E(X)$ of this random variable is defined as follows.

\begin{align*}
	E(X) & = \sum_k x_k P(X=x_k)                                                                       \\
	     & = 1 \times \frac{1}{6} + 2 \times \frac{1}{3} + 3 \times \frac{1}{3} + 4 \times \frac{1}{6} \\
	     & = \frac{15}{6}
\end{align*}

If a new variable $Y=2X+1$ is defined, the PMF of $Y$ becomes

\begin{table}[htbp]
	\centering
	\begin{tabular}{|c|c|c|c|c|}
		\toprule
		                & $y=2 \times 1  + 1$ & $y=2 \times 2  + 1$ & $y=2 \times 3  + 1$ & $y=2 \times 4  + 1$ \\
		\midrule
		$P(Y=y=2x + 1)$ & 1/6                 & 1/3                 & 1/3                 & 1/6                 \\
		\hline
	\end{tabular}
\end{table}

\begin{align*}
	E(Y) & = \sum_k y_k P(Y=y_k)         \\
	     & = \sum_k (2 x_k + 1) P(Y=y_k) \\
	     & = \sum_k (2x_k + 1) P(X=x_k)  \\
	     & = 2E(X) + 1
\end{align*}

Hence, expectation is a linear operator which satisfies $E(aX + b) = aE(X) + b$.

The variance of a discrete random variable $X$ is defined as

\begin{align*}
	Var(X) & = E((X-E(X))(X-E(X)))           \\
	       & = E(X^2 - 2E(X)X + E(X)E(X))    \\
	       & = E(X^2) - 2E(X)E(X) + E(X)E(X) \\
	       & = E(X^2) - E(X)^2
\end{align*}

In case of continuous random variables, instead of Probability Mass Function a Probability Density Function is defined because the sample space is continuous and infinite as opposed to discrete and finite.

The expectation and variance of a continuous random variable are defined as

\begin{align*}
	E(X)   & = \int x P(X=x) dx            \\
	Var(X) & = \int (x - E(X))^2 P(X=x) dx
\end{align*}

Just as in the case of discrete random variables where the sum of probabilities of all events in sample space must sum to 1, the area under PDF curve in the case of continuous random variable must also integrate to 1.

An example of a well known continuous random variable is the Gaussian random variable whose PDF is defined as follows.

\begin{align*}
	P(X=x) & = \frac{1}{\sqrt{2\pi\sigma^2}} e^{\frac{-(x-\mu)^2}{2\sigma^2}} \\
\end{align*}

with the following expectation and variance.

\begin{align*}
	E(X)   & = \mu      \\
	Var(X) & = \sigma^2
\end{align*}

\subsection{A Pair of Random Variables}

Let the joint Probability Mass Function of a pair of discrete random variables X  and Y be defined as follows.

\begin{table}[htbp]
	\centering
	\begin{tabular}{|c|c|c|c|}
		\toprule
		P(X=x,\ Y=y) & $x=1$ & $x=2$ & $x=3$ \\
		\midrule
		$y=1$        & 3/8   & 3/16  & 3/16  \\
		$y=2$        & 1/8   & 1/16  & 1/16  \\
		\hline
	\end{tabular}
\end{table}

As usual, the sum of all probabilities is equal to 1.

Given the joint distribution of a set of random variables, one can also find out the distribution of a subset of those random variables and the distribution of such a subset is called the marginal distribution. In the above example, the marginal distribution of $X$ and $Y$can be computed by summing up each column and row respectively in the table above.

\begin{align*}
	P(X=x_k) = \sum_{y_l} P(X=x_k,\ y=y_l) \\
	P(Y=y_k) = \sum_{x_l} P(X=x_l,\ y=y_k) \\
\end{align*}

In the case of a pair of continuous random variables, their joint Probability Distribution Function becomes a surface in three dimensions. The marginal distributions take the following formulae.

\begin{align*}
	P(X=x) = \int_y P(X=x,\ y=y)dy \\
	P(Y=y) = \int_x P(X=x,\ y=y)dx \\
\end{align*}

Two random variables are said to statistically independent if and only if

\begin{align*}
	P(X=x,\ Y=y) = P(X=x)P(Y=y)
\end{align*}

For independent random variables, the following result holds.

\begin{align*}
	E(f(X)g(Y)) & = \int \int f(x)g(y)P(X=x,\ Y=y)dxdy      \\
	            & = \int \int f(x)g(y) P(X=x) P(Y=y) dxdy   \\
	            & = \int f(x) P(X=x) dx \int g(y) P(Y=y) dy \\
	            & = E(f(X))E(g(Y))
\end{align*}

A weaker form of independence is uncorrelatedness. If $f(X)$ and $g(Y)$ are defined such that $f(X)=X$ and $g(Y) = Y$ and $E(XY) = E(X)E(Y)$ holds true, then two variables are said to be uncorrelated.

\subsection {Sum of Random Variables}

If $X$ denotes the outcome of roll of a dice and $Y$ denotes the outcome of roll of another dice, then $Z=X+Y$ denotes a random variable whose value is equal to the sum of the two rolls. If PMFs of $X$ and $Y$ are known, what is the PMF of $Z=X+Y$?

\begin{align*}
	P(Z=X+Y=z) & = \sum_x P(X=x,\ Y=z-x) \\
	           & = \sum_y P(X=z-y,\ Y=y)
\end{align*}

It is easy to see why the above equation holds. To find $P(Z=z)$, one has to sum the probabilities $P(X=x,\ Y=y)$ of all possible scenarios where $x+y=z$. In the case of continuous random variables the formulae become

\begin{align*}
	P(Z=X+Y=z) & = \int P(X=x,\ Y=z-x) dx \\
	           & = \int P(X=z-y,\ Y=y) dy
\end{align*}

Random variable sums are commutative and associative.

\begin{align*}
	X + Y     & = Y + Z       \\
	X + Y + Z & = (X + Y) + Z \\
	          & = X + (Y + Z)
\end{align*}

The expectation of a sum of two random variables is the sum of expectations of individual random variables.

\begin{align*}
	E(X+Y) & = \int \int (x+y)P(X=x,\ Y=y)dxdy                                     \\
	       & = \int \int xP(X=x,\ Y=y)dxdy + \int yP(X=x,\ Y=y)dxdy                \\
	       & = \int x\ \int P(X=x,\ Y=y) dy\ dx + \int y\ \int P(X=x,\ Y=y) dx\ dy \\
	       & = \int xP(X=x)dx + \int yP(Y=y)dy                                     \\
	       & = E(X) + E(Y)
\end{align*}

Interestingly, If $X$ and $Y$ are independent Gaussian random variables then the above result holds for variances too.

\begin{align*}
	E(X+Y)   & = E(X) + E(Y)     \\
	Var(X+Y) & = Var(X) + Var(Y)
\end{align*}

\section{Motivation}

Suppose there are $N$ people speaking at a party and there are $N$ microphones placed at different locations recording sound. Each microphone is recording a simultaneous mix of all speakers' sounds depending on the relative position of the microphone from the speakers. Looking at the microphone recordings alone, is it possible to decipher what each speaker said?

\section{Problem Statement}

Denote each speaker's sound as a random variable $S_i$ (S stands for signal). A square mixing matrix $A$ is assumed to be responsible for linearly transforming $N$ signals into another set of $N$ random variables observed one each by a microphone.

\begin{align*}
	\begin{pmatrix}
	X_1 \\
	X_2 \\
	. \\
	. \\
	. \\
	X_N
	\end{pmatrix} &=
	\begin{pmatrix}
	a_{11} & a_{12} & ... & a_{1N} \\
	a_{21} & a_{22} & ... & a_{2N} \\
	.      & .      & ... & .      \\
	.      & .      & ... & .      \\
	.      & .      & ... & .      \\
	a_{N1} & a_{N2} & ... & a_{NN} \\
	\end{pmatrix}
	\begin{pmatrix}
	S_1 \\
	S_2 \\
	. \\
	. \\
	. \\
	S_N
	\end{pmatrix} \\
	\\
	X &= A S
\end{align*}

If the $j$th column of the matrix $A$ is denoted by $a_j$, then the equation becomes

\begin{align*}
	X = \sum_j a_j S_j
\end{align*}

A set of observations corresponding to $X$ are given and the goal is to recover $S$.

\begin{align*}
	X & = A S      \\
	S & = A^{-1} X \\
	S & = W X
\end{align*}

The goal is to find this un-mixing matrix $W$.

\section{Assumptions}

\begin{itemize}
	\item The signals to be recovered must be mutually independent. This assumption gives basis to any ICA algorithm. Find a linear combination $W$ of the observed random variables to yield ``signal" random variables that are as independent as possible.
	\item The signals must be non-Gaussian. The reason for this restriction will be explained later.
	\item The signals have zero expectation. This is for mathematical convenience. The observed random variables can be centered to impose this restriction. How?
	      \begin{align*}
	      	if\  E(X_i)  & = 0\ \forall\  i       \\
	      	then\ E(S_i) & = E(\sum_j w_{ij}S_j)  \\
	      	             & =  \sum_j w_{ij}E(S_j) \\
	      	             & = 0\ \forall \ i
	      \end{align*}

\end{itemize}

\section{Ambiguities}

\begin{itemize}
	\item Variances of the signals can't be calculated.
	      \begin{align*}
	      	\begin{pmatrix}
	      	X_1 \\
	      	X_2
	      	\end{pmatrix} & =
	      	\begin{pmatrix}
	      	2             & 3 \\
	      	4             & 5
	      	\end{pmatrix}
	      	\begin{pmatrix}
	      	S_1 \\
	      	S_2
	      	\end{pmatrix} \\
	      	              & =
	      	\begin{pmatrix}
	      	1             & 3 \\
	      	2             & 5
	      	\end{pmatrix}
	      	\begin{pmatrix}
	      	2S_1 \\
	      	S_2
	      	\end{pmatrix}
	      \end{align*}

	      Hence the variances of all signals are assumed to be equal to unity.
	\item The order of signals can't be recovered.

	      \begin{align*}
	      	\begin{pmatrix}
	      	X_1 \\
	      	X_2
	      	\end{pmatrix} & =
	      	\begin{pmatrix}
	      	2             & 3 \\
	      	4             & 5
	      	\end{pmatrix}
	      	\begin{pmatrix}
	      	S_1 \\
	      	S_2
	      	\end{pmatrix} \\
	      	              & =
	      	\begin{pmatrix}
	      	3             & 2 \\
	      	5             & 4
	      	\end{pmatrix}
	      	\begin{pmatrix}
	      	S_2 \\
	      	S_1
	      	\end{pmatrix}
	      \end{align*}
\end{itemize}

\section{Example}

Let $S_1$ and $S_2$ be two independent signals with a uniform density of $1/2$ from $-1$ to $1$. The PDF of each signal looks like

\begin{figure}[htbp]
	\centering
	\begin{tikzpicture}

		\draw[-latex] (-2,0) -- (2,0) node[right]{$s$};
		\draw[-latex] (0,-1) -- (0,1) node[above]{$P(S=s)$};

		\draw[line width=0.15em] (-1.8,0) -- (-1,0);
		\draw[line width=0.15em] (-1,0) -- (-1,0.5);
		\draw[line width=0.15em] (-1,0.5) -- (1,0.5);
		\draw[line width=0.15em] (1,0.5) -- (1,0);
		\draw[line width=0.15em] (1,0) -- (1.8,0);

		\draw (-1, 0.5) node [above left] {(-1, 1/2)};
		\draw (1, 0.5) node [above right] {(1, 1/2)};

	\end{tikzpicture}
\end{figure}

Their joint distribution takes the the following form.

\begin{align*}
	P(S_1=s_1,\ S_2=s_2) & = \frac{1}{4}\ \ \ if\ |s_1| <= 1\ and\ |s_2| <= 1 \\
	                     & = 0\ \ \ otherwise
\end{align*}

\begin{figure}[htbp]
	\centering
	\begin{tikzpicture}

		\draw[-latex] (-2,0) -- (2,0) node[right]{$s_1$};
		\draw[-latex] (0,-2) -- (0,2) node[above]{$s_2$};

		\draw [fill=gray, opacity=0.3] (-1,-1) rectangle (1,1);

		\draw (-1, -1) node [below left] {(-1, -1)};
		\draw (-1, 1) node [above left] {(-1, 1)};
		\draw (1, -1) node [below right] {(1, -1)};
		\draw (1, 1) node [above right] {(1, 1)};
	\end{tikzpicture}
\end{figure}

\FloatBarrier

Consider the following linear transformation of these signals.

\begin{align*}
	\begin{pmatrix}
	X_1 \\
	X_2
	\end{pmatrix} & =
	\begin{pmatrix}
	1             & 1 \\
	3             & 1
	\end{pmatrix}
	\begin{pmatrix}
	S_1 \\
	S_2
	\end{pmatrix}
\end{align*}

The joint density of the observed signals becomes

\begin{align*}
	P(X_1=x_1,\ X_2=x_2) & = \frac{1}{2} \times \frac{1}{4}
	= \frac{1}{\begin{vmatrix}
	1                    & 1                                                                    \\
	3                    & 1
	\end{vmatrix}} \times \frac{1}{4} \ \ \ if\ (x_1, x_2)\ lie\ in\  parallelogram\ below \\
	                     & = 0\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ otherwise
\end{align*}

\begin{figure}[htbp]
	\centering
	\begin{tikzpicture}

		\draw[-latex] (-4,0) -- (4,0) node[right]{$x_1$};
		\draw[-latex] (0,-4) -- (0,4) node[above]{$x_2$};

		\draw [fill=gray,  opacity=0.3] (2,4) -- (0,2) -- (-2,-4) -- (0,-2) -- cycle;

		\draw (2, 4) node [above right] {(2, 4)};
		\draw (0, 2) node [above left] {(0, 2)};
		\draw (-2, -4) node [below left] {(-2, -4)};
		\draw (0, -2) node [below right] {(0, -2)};
	\end{tikzpicture}
\end{figure}

\FloatBarrier

One edge of the parallelogram points in the direction $(1, 1)$ and the other in $(1, 3)$. These are the same as the columns in the mixing matrix. This gives some hope that ICA could exploit some properties of the joint distribution to recover the original signals like in this case where the edges of the joint density give the mixing matrix away.

\section{Density Under Linear Transformation}

Consider a random vector $X$ with the following PDF.

\begin{figure}[htbp]
	\centering
	\begin{tikzpicture}

		\draw[-latex] (-2,0) -- (2,0) node[right]{$x$};
		\draw[-latex] (0,-1) -- (0,2) node[above]{$P(X=x)$};

		\draw[line width=0.15em] (-1.8,0) -- (-1,0);
		\draw[line width=0.15em] (-1,0) -- (0,1);
		\draw[line width=0.15em] (0,1) -- (1,0);
		\draw[line width=0.15em] (1,0) -- (1.8,0);

		\draw (-1, 0) node [below] {(-1, 0)};
		\draw (1, 0) node [below] {(1, 0)};
		\draw (0, 1) node [above right] {(0, 1)};
	\end{tikzpicture}
\end{figure}

\FloatBarrier

Consider another random vector $Y=2X$. How is the PDF of $Y$ related to that of $X$? One can proceed in the following manner.

\begin{align*}
	P(Y=y) & = P(2X = y)          \\
	       & = P(X= \frac{y}{2}))
\end{align*}

The calculated PDF is drawn below.

\begin{figure}[htbp]
	\centering
	\begin{tikzpicture}

		\draw[-latex] (-4,0) -- (4,0) node[right]{$y$};
		\draw[-latex] (0,-1) -- (0,2) node[above]{$P(Y=y)$};

		\draw[line width=0.15em] (-3.8,0) -- (-2,0);
		\draw[line width=0.15em] (-2,0) -- (0,1);
		\draw[line width=0.15em] (0,1) -- (2,0);
		\draw[line width=0.15em] (2,0) -- (3.8,0);

		\draw (-2, 0) node [below] {(-2, 0)};
		\draw (2, 0) node [below] {(2, 0)};
		\draw (0, 1) node [above right] {(0, 1)};
	\end{tikzpicture}
\end{figure}

On closer inspection, the calculated PDF is wrong as the area under it is 2. Thus a division by 2 is necessary. Thus,

\begin{align*}
	Y      & = \alpha X                               \\
	X      & = \frac{1}{\alpha} Y                     \\
	P(Y=y) & = \frac{1}{\alpha} P(X=\frac{y}{\alpha}) \\
	P(X=x) & = \alpha P(Y=\alpha x)
\end{align*}

More generally,

\begin{align*}
	Y                                 & = \alpha X                                                                \\
	\begin{pmatrix} Y_1               &                                                                           \\ Y_2 \\ ... \\Y_N \end{pmatrix} &= \begin{pmatrix} \alpha_1^T \\ \alpha_2^T \\ ... \\  \alpha_N^T \end{pmatrix} \begin{pmatrix} X_1 \\ X_2 \\... \\  X_N \end{pmatrix}  \\
	P(X_1=x_1, X_2=x_2, ..., X_N=x_N) & =  \begin{vmatrix} \alpha_1^T                                             \\ \alpha_2^T \\ ... \\  \alpha_N^T \end{vmatrix}  P(Y_1=\alpha_1^T X, Y_2=\alpha_2^T X, ..., Y_N=\alpha_N^T X) \\
	P(X_1=x_1, X_2=x_2, ..., X_N=x_N) & =  |\alpha|  P(Y_1=\alpha_1^T X, Y_2=\alpha_2^T X, ..., Y_N=\alpha_N^T X)
\end{align*}

\section{Relation to PCA}

Given a centered random vector $X$, PCA defines a linear transform $Y\ =\ P X$ such that the resulting vector has uncorrelated components. $P$ is a matrix of orthonormal eigenvectors of the covariance matrix of $X$.

\begin{align*}
	E(XX^T) & = P^TDP
\end{align*}

The covariance matrix of the transformed random vector is diagonalized.

\begin{align*}
	E(YY^T) & = E(PXX^TP^T) \\
	        & = PE(XX^T)P^T \\
	        & = PP^TDPP^T   \\
	        & = D
\end{align*}

One can go a step further to define $Y\ =\ D^{-1/2}PX$. Now the covariance matrix of the transformed random vector becomes identity.

\begin{align*}
	E(YY^T) & = D^{-1/2}PE(XX^T)P^TD^{-1/2T} \\
	        & = D^{-1/2}PP^TDPP^TD^{-1/2T}   \\
	        & = I
\end{align*}

Such a transformation of $X$ is called whitening. Whitening is often a preprocessing step of ICA. Intuitively, whitening makes the transformed components uncorrelated and then ICA will make them independent. So whitening solves half the problem of ICA. More formally, whitening restricts the space of mixing matrix to orthogonal matrices thus making the problem easier. To understand why, consider $Y\ =\  AS$ such that $Y$ is white.

\begin{align*}
	E(YY^T) = I      \\
	AE(SS^T) A^T = I \\
\end{align*}

Since signals are independent with unit variances as per an assumption above,

\begin{align*}
	E(SS^T) & = I \\
	AA^T    & = I \\
\end{align*}

proving that $A$ must be orthogonal. Hence, in case of a random vector of size $N$, instead of estimating $N^2$ parameters of the matrix $A$, ICA only has to estimate $N(N-1)/2$ parameters since $A$ is orthogonal and an orthogonal matrix has $N(N-1)/2$ degrees of freedom. This is equal to estimating almost half the parameters. Thus, literally too, whitening solves half the problem of ICA.

\section{Why Gaussian signals can't be recovered?}

Consider a Gaussian random variable,

\begin{align*}
	P(X=x) & = \frac{1}{\sqrt{2\pi\sigma^2}} e^{\frac{-(x-\mu)^2}{2\sigma^2}} \\
\end{align*}

Consider another random variable $Y=\alpha X$. What does the PDF of $Y$ look like?

\begin{align*}
	P(Y=y) & = \frac{1}{\alpha} P(X=\frac{y}{\alpha})                                                     \\
	P(Y=y) & = \frac{1}{\alpha \sqrt{2\pi\sigma^2}} e^{\frac{-(\frac{y}{\alpha}-\mu)^2}{2\sigma^2}}       \\
	       & = \frac{1}{\sqrt{2\pi \alpha^2 \sigma^2}} e^{\frac{-(y- \alpha \mu)^2}{2 \alpha^2 \sigma^2}}
\end{align*}

It is clear that $Y$ is also Gaussian with mean $\alpha \mu$ and variance $\alpha^2 \sigma^2$. Another well known result with respect to Gaussians is that sum of two Gaussians is also a Gaussian with mean as sum of individual means and variance as sum of individual variances.

\begin{align*}
	\mathcal{N}(\mu_1, \sigma_1^2) + \mathcal{N}(\mu_2, \sigma_2^2) \sim \mathcal{N}(\mu_1 + \mu_2, \sigma_1^2 + \sigma_2^2)
\end{align*}

These properties of Gaussian signals make them unrecoverable. Consider two Gaussian signals of zero mean and unit variance as per ICA assumptions and an orthogonal mixing matrix. An orthogonal matrix is considered because it is assumed that whitening has already been done and thus the search space of ICA is only orthogonal matrices.

\begin{align*}
	\begin{pmatrix}
	a & b                 \\
	c & d
	\end{pmatrix}
	\begin{pmatrix}
	\mathcal{N}(0, 1) \\
	\mathcal{N}(0, 1)
	\end{pmatrix} \\
	  & = \begin{pmatrix}
	\mathcal{N}(0, a^2) + \mathcal{N}(0, b^2) \\
	\mathcal{N}(0, c^2) + \mathcal{N}(0, d^2)
	\end{pmatrix}  \\
	  & = \begin{pmatrix}
	\mathcal{N}(0, a^2+b^2)  \\
	\mathcal{N}(0, c^2 + d^2)
	\end{pmatrix}
\end{align*}

Due to orthogonality of the mixing matrix $a^2+b^2=c^2+d^2=1$.

\begin{align*}
	\begin{pmatrix}
	a & b                 \\
	c & d
	\end{pmatrix}
	\begin{pmatrix}
	\mathcal{N}(0, 1) \\
	\mathcal{N}(0, 1)
	\end{pmatrix}
	  & = \begin{pmatrix}
	\mathcal{N}(0, 1) \\
	\mathcal{N}(0, 1)
	\end{pmatrix}
\end{align*}

So, upon the transformation due to any orthogonal matrix the signals remain unchanged. Hence, in case of Gaussian signals only whitening is possible.

\section{Algorithm}

$T$ training samples are given of the form,

\begin{align*}
	\begin{pmatrix}
	x_1^1 & x_1^2 & ... & x_1^T \\
	x_2^1 & x_2^2 & ... & x_2^T \\
	... \\
	x_N^1 & x_N^2 & ... & x_N^T
	\end{pmatrix}
\end{align*}

The log-likelihood can be written as,

\begin{align*}
	L = log \left( \prod_i P(X_1 = x_1^i, X_2=x_2^i, ..., X_N=x_N^i) \right)
\end{align*}

Rewriting ICA equation,

\begin{align*}
	S   &= WX              \\
	\begin{pmatrix} S_1 \\ S_2 \\ ... \\S_N \end{pmatrix} &= \begin{pmatrix} w_1^T \\ w_2^T \\ ... \\  w_N^T \end{pmatrix} \begin{pmatrix} X_1 \\ X_2 \\... \\  X_N \end{pmatrix}  \\
\end{align*}

Using the density transformation rule,

\begin{align*}
	L(W) = log \left( \prod_i \left( P(S_1 = w_1^{T} X^i, S_2=w_2^{T} X^i, ..., S_N= w_N^{T} X^i) |W| \right) \right)
\end{align*}

Since the signals are independent,

\begin{align*}
	L(W)= log \left( \prod_i \left( \prod_j  P(S_j = w_j^{T}X^i) |W| \right) \right) \\
	L(W) = \sum_i \left( \sum_j  log (P(S_j = w_j^{T}X^i)) + log(|W|) \right)
\end{align*}

$P(S_j = w_j^{T}X^i)$ is estimated by the sigmoid function.

\begin{align*}
	P(S_j = w_j^{T}X^i) = \frac{e^{-w_j^{T}X^i}}{1 + e^{-w_j^{T}X^i}}
\end{align*}

The log-likelihood equation becomes,

\begin{align*}
	L(W) = \sum_i \left( \sum_j  log \left( \frac{e^{-w_j^{T}X^i}}{1 + e^{-w_j^{T}X^i}} \right) + log(|W|) \right)
\end{align*}

To maximize $L(W)$ gradient descent can be applied. At convergence, the optimal $W$ can be multiplied to $X$ to recover sought $S$.

\end{document}

Independent Component Analysis/Independent Component Analysis.tex


\documentclass[a4paper,12pt]{article}

% .......................................... Preamble

% Use \usepackage[fleqn]{amsmath} to make the math environments left-aligned instead of centered
\usepackage[fleqn]{amsmath}
\usepackage{dsfont} % Used for representing Real numbers (\mathds{R})

% Custom commands to define global variables/symbols in the document
\newcommand{\vectorSpace}{\mathds{R}}
\newcommand{\VSet}{\mathcal{V}}
\newcommand{\PSet}{\mathcal{P}}

% .......................................... End

\begin{document}

	\section*{Maths}

	This is $x^2 =\frac{1}{4}$ in inline mode.
	Here it is in display mode:
	\begin{displaymath}
		x^2 = \frac{1}{4}
	\end{displaymath}

	\section*{More maths}

	Fraction: $\frac{dy}{dx} = \dfrac{dy}{dx}$ \newline
	Sum: $\sum_{n=1}^{\infty} = \displaystyle\sum_{n=1}^{\infty}$ \newline
	Limit: $\lim_{n \to \infty} = \displaystyle\lim_{n \to \infty}$ \newline
	Integral: $\int_a^b = \displaystyle\int_a^b$

	\section*{Matrices}

	\begin{displaymath}
		A = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix},~
		B = \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix},~
		|A| = \begin{vmatrix} 1 & 2 \\ 3 & 4 \end{vmatrix}
	\end{displaymath}

	\section*{More frightening equations}

	\begin{align*}
		f(x) &= e^{x-1}\\
		\int_1^2 f(x)\;dx &= e^{x-1} ~\Bigr|_1^2\\
		&= e - 1
	\end{align*}

	\begin{align}
		2x - 4y - 7z + 8w &= \pi\\
		3x + 5y + 9z &= 213
	\end{align}

	\section*{Math symbols}

	A projection defines a function $(f)$ that transforms data in $\vectorSpace^{\eta}$ to $\vectorSpace^{\kappa}$ (where $\eta \geq \kappa$)\dots\ $f$ transforms the point set $\VSet$ to another set $\PSet$ can be denoted as a mapping of a data point $(\chi)$ in $\vectorSpace^{\eta}$ to another point $(\rho)$ in $\vectorSpace^{\kappa}$ that is subject to:
	\begin{eqnarray}
		f_{\chi} & = & \rho \left\{
		\begin{array}{llr}
			\VSet = (\chi_{0}, \chi_{1}, \dots, \chi_{i}) & \forall{\chi_{i}} \in \vectorSpace^{\eta} & 0 \leq i < N \\
			\PSet = (\rho_{0}, \rho_{1}, \dots, \rho_{j}) & \forall{\rho_{j}} \in \vectorSpace^{\kappa} & 0 \leq j < n \\
			\eta \geq \kappa & {} & {} \\
			N \geq n & {} & {}
		\end{array} \right.
	\end{eqnarray}

\end{document}


\section{Linear Discriminant Analysis}
\label{chp:lindisana}

In chapter \ref{chp:logreg} it was described how logistic regression could be used for binary classification. It was also stated how it could be extended to  a classification problem with more than two classes. In practice logistic regression is though rarely used for classification problems with more than two classes. Furthermore a problem with logistic regression is that it does not perform very stable under some circumstance. One big problem with logistic regression is that it might not converge if the classes are well-separated. An example could be the data in the table below. Here we see that the classes are well-separated as $X<0$ then $Y=0$ and if $X>0$ then $Y=1$. Because of this perfect separation the algorithm will never converge because it can keep optimizing.
\begin{center}
	\begin{tabular}{rll}
		\multicolumn{1}{c}{\textbf{Y}} &
		\multicolumn{1}{c}{\textbf{X}} \\ \hline
		0     &  -2  \\[0.05cm]
		0     &  -3  \\[0.05cm]
		0     &  -1  \\[0.05cm]
		0     &  -5  \\[0.05cm]
		1     &    3  \\[0.05cm]
		1     &    1  \\[0.05cm]
		1     &    5  \\[0.05cm]
		1     &    6  \\[0.05cm]
	\end{tabular}
\end{center}

Instead a method called Linear Discriminant Analysis(LDA) can be used. In logistic regression we directly model $Pr(Y|X)$. With LDA $P(X|Y)$ is modeled and prediction is done by using Bayes theorem to achieve $P(Y|X)$ and find the most probable class given the predictors. The following variables are used:

\begin{itemize}
	\item Let $\pi_k$ represent the prior probability $P(Y=k)$ for K classes.
	\item Let $f_k(x)$ represent the probability $P(X=x|Y=k)$. So $f_k(x)$ is the probability of the predictors when the class Y is given.
\end{itemize}

By inserting these in Bayes formula and normalizing to get $P(Y=k|X=x)$ the following can be written.

\begin{equation}
	P(Y=k|X = x) = \dfrac{\pi_k f_k(x)}{\sum_{l=1}^{K} \pi_l f_l(x)}
\end{equation}

For the equation the two variables $\pi_k$ and $f_k(x)$ still needs to be estimated. $pi_k$ can be estimated by computing the fraction of the training data that belongs to the k'th class. While $f_k(x)$ can be estimated as a Gaussian. This means that the equation differs when using only 1 predictor to using multiple. When only using one predictor and taking log of the equation it can be rewritten as the following

\begin{equation}
	\delta_k(x) = x  \dfrac{\mu_k}{\sigma^2} - \dfrac{\mu_k^2}{2\sigma^2} + log(\pi_k)
\end{equation}

Then the equation is calculated for each class k and is classified for the k that results in the highest probability. The equation for multiple predictors instead uses a $mu$ matrix and a covariance matrix and can be written as following

\begin{equation}
	\delta_k(x) = x^T \Sigma^{-1} \mu_k -  \frac{1}{2} \mu_k^{T} \Sigma^{-1} \mu_k + log \pi_k
\end{equation}

Note here that LDA assumes the observation to have a Gaussian distribution. If this is the case then LDA is likely to outperform Logistic regression, while if it is not, then Logistic regression is likely to outperform LDA.

The means of the Gaussian are simply calculated by summing all values with a specific class and dividing by the total number.
The variance is calculated across all the data with a specific class.

\section{Lab 4.6.3 - Linear Discriminant Analysis}
First a model linear discriminant analysis model is created with the python library sklearn. The model is created based on the two predictors \emph{Lag1} and \emph{Lag2}.

\begin{lstlisting}[language=Python, label=lst:ldaModel, caption=creating linear discriminant analysis model sklearn]
X_train = data[:'2004'][['Lag1', 'Lag2']]
Y_train = data[:'2004']['Direction']

X_test = data['2005':][['Lag1', 'Lag2']]
Y_test = data['2005':]['Direction']

lda = LinearDiscriminantAnalysis()
fit = lda.fit(X_train,Y_train)
\end{lstlisting}

After creating the model the calculated prior probability, means and coefficients can be read. The prior probability for \emph{Down} is $\pi_1=0.492$ while the prior probability for \emph{Up} is $\pi_2=0.508$. So based on the data there are more days where the stock market went up than down.

The means $\mu_k$ are described by a matrix. The matrix describes a separate mean for both \emph{Lag1} and \emph{Lag2} and for each predictor a mean for \emph{Up} and \emph{Down}. The mean value for \emph{Lag1} and \emph{Lag2} has a tendency to be positive on days when the stock went up while they are negative when the stock went down.  $\bigl( \begin{smallmatrix} 	 & Lag1 & Lag2	\\ 	 Down & 0.0428 & 0.0339 \\ 	 Up & -0.0395 & -0.0313 \end{smallmatrix} \bigr) $. The coefficients are estimated to $-0.0554$ and $-0.0443$. These are the multipliers for \emph{Lag1} and \emph{Lag2}.

\todo{Er det vigtigt at vise hvordan vi printer disse værdier?}
\begin{lstlisting}[language=Python, label=lst:ldaProperties, caption=printing lda prior mean and coefficients]
print "Prior probabilities:"
print lda.priors_

print "Mean:"
print lda.means_

print "Coefficients:"
print lda.coef_
\end{lstlisting}

In listing \ref{lst:ldaScore} the score is printet. The score show that the LDA model predicted correctly in $56\%$ of the predictions on the test data.
\begin{lstlisting}[language=Python, label=lst:ldaScore, caption=printing lda score]
print(lda.score(X_test, Y_test))
\end{lstlisting}

This shows how linear discriminant analysis performs very similar to logistic regression on the stock market data.

\section{Quadratic discriminant analysis}
Similar to LDA the Quadratic discriminant analysis (QDA) also assumes that the observations are drawn from a Gaussian distribution and also uses Bayes theorem for prediction.

\myFigure{lda_vs_qda.PNG}{LDA vs QDA}{fig:qdaCompare}{0.6}

In Figure \ref{fig:qdaCompare} the difference between LDA and QDA is illustrated. It shows that the LDA can only learn linear boundaries, while the QDA can learn quadratic boundaries. This makes the QDA more flexible.

\subsection{Lab 4.6.4}
Again the stock market data is used for prediction with \emph{Lag1} and \emph{Lag2} as predictors. The QDA model is estimated with sklearn. This is shown in listing \ref{lst:qdaModel}.

\begin{lstlisting}[language=Python, label=lst:qdaModel, caption=Creating quadratic discriminant analysis model sklearn]
X_train = data[:'2004'][['Lag1', 'Lag2']]
Y_train = data[:'2004']['Direction']

X_test = data['2005':][['Lag1', 'Lag2']]
Y_test = data['2005':]['Direction']

qda = QuadraticDiscriminantAnalysis()
fit = qda.fit(X_train,Y_train)
\end{lstlisting}

In listing \ref{lst:qdaScore} the score for the QDA model is printed. The print shows that on the test data the prediction was correct in $59.9\%$ of the cases. With the stock market data it actually performed better than both logistic regression and linear discriminant analysis.
\begin{lstlisting}[language=Python, label=lst:qdaScore, caption=Printing qda score]
print(qda.score(X_test, Y_test))
\end{lstlisting}

\section{K-Nearest Neighbors}
K-Nearest Neighbors or \emph{KNN} is a classification technique that classifies observations depending on distance to k other observations.

Say you have a dataset with two classes of observations A and B, with 10 observations each. Then a test observation is desired to be classified with $k=5$, which means the five nearest observations will be examined. If three of the examined observations are A and the other two are B then the estimated probabilities are $3/5$ for class A and $2/5$ for class B.

\subsection{Lab 4.6.5}

In this lab exercise the method \emph{K-nearest Neighbors} is used. To perform this method, sklearn has a function called \emph{KNeighborsClassifier}. This takes one parameter, which is number of n neighbors. After this the function fit is called, that fits the data.
The code is shown in listing

\begin{lstlisting}[caption={Python K-Nearest neighbors function, where K is set to 1}, label=lst:kneighbor, mathescape=true]
X_train = data[:'2004'][['Lag1', 'Lag2']]
Y_train = data[:'2004']['Direction']

X_test = data['2005':][['Lag1', 'Lag2']]
Y_test = data['2005':]['Direction']

knn = KNeighborsClassifier(n_neighbors=1)
fit = knn.fit(X_train, Y_train)
\end{lstlisting}

The function is used to predict the market's movement from the dates in 2005. If the value K is set to 1, 50\% of the predictions are correctly predicted which is not very good.
If K is change to 3, the predictions turns out to be correct in 53\% of the cases. So increasing the number of k from 1 to 3 made the model a bit better. All kinds of different values can be chosen for K, but it is important not to change the number to often because the model then will be customized to the training data which is not good.

If the results are compared with the results that \emph{QDA} produces, the \emph{QDA} provides better results than K-nearest neighbors.


\lab{Data Structures III: K-D Trees}{K-D Trees}
\label{lab:DS3}

\objective{Nearest neighbor search is an optimization problem that arises in applications such as computer vision, pattern recognition, internet marketing, and data compression.
In this lab we solve the problem efficiently using a k-d tree, then use SciPy's k-d tree to implement a handwriting recognition algorithm.}

\section*{The Nearest Neighbor Search Problem} % ==============================

Suppose you move into a new city with several post offices.
Since your time is valuable, you wish to know which post office is closest to your home.
This is called the nearest neighbor search problem, and it has many applications.

In general, suppose that $X$ is a collection of data, called a \emph{training set}.
Let $y$ be any point (often called the \emph{target} point) in the same space as the data in $X$.
The nearest neighbor search problem determines the point in $X$ that is closest to $y$.
For example, in the post office problem the set $X$ could be addresses or latitude and longtitude data for each post office in the city.
Then $y$ would be the data that represents your new home, and the task is to find the closest post office in $X$ to $y$.

\begin{problem} % Euclidean Distance
Roughly speaking, a function that measures distance between two points in a set is called a \emph{metric}.\footnote{Metrics and metric spaces are examined in detail in Chapter 5 of Volume I.}
The \emph{euclidean metric} measures the distance between two points in $\mathbb{R}^n$ with the familiar distance formula:
\[
d(\x,\y) = \sqrt{\displaystyle\sum_{i=1}^n (x_i - y_i)^2} = \| \x - \y \|_2
\]

Write a function that accepts two 1-dimensional NumPy arrays and returns the euclidean distance between them.
Raise a \li{ValueError} if the arrays don't have the same number of entries.
\\
(Hint: NumPy already has some functions to help do this quickly.)
\end{problem}

Consider again the post office example.
One way to find out which post office is closest is to drive from home to each post office, measure the mileage, and then choose the post office that is the closest.
This is called an \emph{exhaustive search}.
More precisely, we measure the distance of $y$ to each point in $X$, and choose the point with the smallest distance.
This method is ineffecient however, and only feasible for relatively small training sets.

\begin{problem} % Exhaustive search method.
Write a function that solves the nearest neighbor search problem by exhaustively checking all of the distances between a given point and each point in a data set.
The function should take in a set of data points (as an $m \times k$ NumPy array, where each row represents one of the $m$ points in the data set) and a single target point (as a 1-dimensional NumPy array with $k$ entries).
Return the point in the training set that is closest to the target point and its distance from the target.

The complexity of this algorithm is $O(mk)$, where $k$ is the number of dimensions and $m$ is the number of data points.
\end{problem}

\section*{K-D Trees} % ========================================================

% \footnote{This lab is a sequel to the Data Structures II lab, and should not be attempted until the reader has successfully implemented a binary search tree in Python.}
A \href{https://en.wikipedia.org/wiki/K-d_tree}{\emph{$k$-d tree}} is a special kind of binary search tree for high dimensional data (i.e., more dimensions than 1).
While a binary search tree excludes regions of the number line from a search until the search point is found, a $k$-d tree works on regions of $\mathbb{R}^k$.
So long as the data in the tree meets certain dimensionality requirements, similar efficiency gains may be made.

Recall that to search for a point in a binary search tree, we start at the root, and if the point we are searching for is less than the root we proceed down the left branch of the tree.
If it is larger, we proceed down the right branch.
By doing this, we exclude a region of the number line (and therefore the subtree in the opposite direction) from our search.
By eliminating this region from consideration, we have far fewer points to search and the efficiency of our search is greatly increased.
% See Figure \ref{fig:k-binary-search} for an illustration.

Like a binary search tree, a $k$-d tree starts with a root node with a depth, or level, of 0.
At the $i^{th}$ level, the nodes to the left of a parent have a lower value in the $i^{th}$ dimension.
Nodes to the right have a greater value in the $i^{th}$ dimension.
At the next level, we do the same for the next dimension.
For example, consider data in $\mathbb{R}^3$.
The root node partitions the data according to the first dimension.
The children of the root partition according to the second dimension, and the grandchildren along the third.
See Figure \ref{fig:k-binary-search} for an example in $\mathbb{R}^2$.

As with any other data structure, the first task is to construct a node class to store data.
A \li{KDTNode} is similar to a \li{BSTNode}, except it has another attribute called \li{axis}.
The \li{axis} attribute tells us which dimension of $\mathbb{R}^k$ to split on.

% TODO: A figure actually depicting the resulting KDT, like the BST figures of the Data Structures II lab.

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/kdpic1.png}
\caption{A regular binary search tree partitions $\mathbb{R}$, but a $k$-d tree partitions $\mathbb{R}^{k}$. The above graph illustrates the partition for a $k$-d tree loaded with the points (5, 5), (8, 4), (3, 2), (7, 7), (2, 6), and (9, 2), in that order. To find the point (9, 2), we start at the root. Since the $x$-coordinate of (9, 2) is greater than the $x$-coordinate of (5, 5), we move into the region to the right of the middle blue line, thus excluding all points ($x$, $y$) with $x < 5$. Next we compare (9, 2) to the root's right child, (8, 4). Since the $y$-coordinate of (9, 2) is less than the $y$-coordinate of (8, 4), we move below the red line on the right, thus excluding all points ($x$, $y$) with $y > 4$. We have now found (9, 2), since it is the left child of (8, 4).}
\label{fig:k-binary-search}
\end{figure}

\begin{comment} % KDT Node class
\begin{lstlisting}
# import the BSTNode class from the previous lab.

class KDTNode(BSTNode):
    """Node class for K-D Trees. Inherits from BSTNode.
    Attributes:
        left (KDTNode): a reference to this node's left child.
        right (KDTNode): a reference to this node's right child.
        parent (KDTNode): a reference to this node's parent node.
        data (ndarray): a coordinate in k-dimensional space.
        axis (int): the dimension to make comparisons on.
    """
    def __init__(self, data):
        """Construct a K-D Tree node containing 'data'. The left, right,
        and prev attributes are set in the constructor of BSTNode.
        """
        BSTNode.__init__(self, data)
        self.axis  = 0
\end{lstlisting}
\end{comment}

\begin{problem} % KDTNode class
Copy the \li{BSTNode} class from the previous lab.%\footnote{If the file containing the \li{BSTNode} class is in a different directory than your solutions file for this lab, see \url{https://docs.python.org/2/tutorial/modules.html\#packages} for instructions on packages and imports.}
Write a \li{KDTNode} class that inherits from \li{BSTNode}.

Modify the constructor so that a \li{KDTNode} can only hold a NumPy array (of type \li{np.ndarray}).
If any other data type is given, raise a \li{TypeError}.
Also create an \li{axis} attribute (set it to \li{None} or $0$ for now).

\begin{comment} % Magic methods (outdated)
\item Write the \li{__sub__} magic method so that \li{x} - \li{y} returns the euclidean distance between the data in node \li{x} and the data in node \li{y}.

\item Write the \li{__eq__} magic method so that \li{x == y} is \li{True} if and only if \li{x} and \li{y} have the same data (Hint: \li{np.allclose()})

\item Finally, write the \li{__lt__} and \li{__gt__} magic methods so that the $<$ and $>$ operators compare the $i^{th}$ entry of the data, where $i$ is the \li{axis} attribute of the node on the \emph{right side} of the operator.
For example,

\begin{lstlisting}
>>> x = KDTNode(np.array([1,2]))
>>> y = KDTNode(np.array([3,1]))
>>> y.axis = 0            # Compare the '0th' entry of the data
>>> x < y                # True, since 1 < 3
True
>>> x > y
False

>>> y.axis = 1            # Compare the '1st' entry of the data
>>> x < y                # False, since 2 > 1
False
>>> x > y
True
\end{lstlisting}
\end{comment}
\end{problem}

% Now we construct the $k$-d tree class.
% For an optimal $k$-d tree, the data needs to be inserted in a very particular order.
% However, inserting at random still usually produces a good tree.
% Here we simply insert the data in the order that it is given.

% TODO: put this elsewhere (in the problem statement?).

The major difference between a $k$-d tree and a binary search tree is how the data is compared at each depth level.
Though we don't need to use a \li{find()} method in solving the nearest neighbor problem, we provide the $k$-d tree version of \li{find()} as an instructive example.

In the \li{find()} method, every comparison in the recursive \li{_step()} function compares the data of \li{target} and \li{current} based on the \li{axis} attribute of \li{current}.
This way if each existing node in the tree has the correct \li{axis}, the correct comparisons are made as we descend through the tree.

\begin{lstlisting}
# Copy or import the BST class from the previous lab.

class KDT(BST):
    """A k-dimensional binary search tree object.
    Used to solve the nearest neighbor problem efficiently.

    Attributes:
        root (KDTNode): the root node of the tree. Like all other
            nodes in the tree, the root houses data as a NumPy array.
        k (int): the dimension of the tree (the 'k' of the k-d tree).
    """

    def find(self, data):
        """Return the node containing 'data'. If there is no such node
        in the tree, or if the tree is empty, raise a ValueError.
        """

        # Define a recursive function to traverse the tree.
        def _step(current):
            """Recursively step through the tree until the node containing
            'data' is found. If there is no such node, raise a Value Error.
            """
            if current is None:                     # Base case 1: dead end.
                raise ValueError(str(data) + " is not in the tree")
            elif np.allclose(data, current.value):
                return current                      # Base case 2: data found!
            elif data[current.axis] < current.value[current.axis]:
                return _step(current.left)          # Recursively search left.
            else:
                return _step(current.right)         # Recursively search right.

        # Start the recursion on the root of the tree.
        return _step(self.root)
\end{lstlisting}

\begin{problem} % Implement the K-D Tree!
Finish implementing the \li{KDT} class.
\begin{enumerate}
\item Override the \li{insert()} method.
To insert a new node, find the node that should be the parent of the new node by recursively descending through the tree as in the \li{find()} method (see figure \ref{fig:k-insert} for a geometric example).
Note that the \li{k} attribute of the will have to be set at some point.

The \li{axis} attribute of the new node will be one more than that axis of the parent node.
If the last dimension of the data has been reached, start \li{axis} over at 0.

\item To solve the nearest neighbor search problem, we need only create the $k$-d tree once.
Then we can use it multiple times with different target points.
To prevent the user from altering the tree, disable the \li{remove()} method.
Raise a \li{NotImplementedError} if the method is called, and allow it to receive any number of arguments.
\end{enumerate}
\end{problem}

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/kdpic2.png}
\caption{To insert the point (4, 7) into the $k$-d tree of figure \ref{fig:k-binary-search}, we find the node that will be the new node's parent. Start at the root, (5, 5). Since the $x$-coordinate of (4, 7) is less than the $x$-coordinate of (5, 5), we move into the region to the left of the middle blue line, to the root's left child, (3, 2). The $y$-coordinate of (4, 7) is greater than the $y$-coordinate of (3, 2), so we move above the red line on the left, to the right child (2, 6). Now we return to comparing the $x$-coordinates, and since $4 > 2$ and (2, 6) has no right child, we install (4, 7) as the right child of (2, 6).}
\label{fig:k-insert}
\end{figure}

Using a $k$-d tree to solve the nearest neighbor search problem requires some care.
At first glance, it appears that a procedure similar to \li{find()} or \li{insert()} will immediately yield the result.
However, this is not always the case (see Figure \ref{fig:bad-search}).

To correctly find the nearest neighbor we will keep track of the target point, the current search node, current best point, and current minimum distance.
Start at the root node.
Then the current search node and current best point will be root, and the current minimum distance will be the euclidean distance from \li{root} to \li{target}.
We then proceed recursively as in the \li{find()} method.
As we find better points (nearer neighbors), we update the appropriate variables accordingly.

Once we have reached the bottom of the tree, we will have a good guess for the nearest neighbor.
However, we are not guaranteed to have arrived at the correct point.
One way to ensure that we have arrived at the correct point is to draw a hypersphere with a radius of the current minimum distance around the candidate nearest neighbor.
If this hypersphere does not intersect any of the hyperplanes that split the $k$-d tree, then we know that have found a best point.

\begin{figure}[H]
\includegraphics[width=\textwidth]{figures/kdpic3.png}
\caption{Suppose we want to find the point in the $k$-d tree of figure \ref{fig:k-insert} that is closest to (8, 4.5). First we record the distance from the root to the target as the current minimum distance (about 3.04), then travel down the tree to the right. The right child, (8, 4), is only .5 units away from the target (the green circle), so we update the minimum distance. Since (8, 4) is not a leaf in the tree, we could continue down to the left child, (7, 7). However, this leaf node is much further from the target (the purple circle). To ensure that we terminate the algorithm correctly we check to see if the hypersphere of radius .5 around the current node (the green circle) intersects with any other hyperplanes. Since it does not, we stop descending down the tree and conclude (correctly) that (8, 4) is the nearest neighbor.}
\label{fig:bad-search}
\end{figure}

\begin{comment}
\begin{figure}
\caption{We need a figure here.  Something similar to what is found at http://web.stanford.edu/class/cs106l/handouts/assignment-3-kdtree.pdf, page 8.  This needs a great caption.
I can write it when the figure is done. Of course we will need to have a reference to this document.}
\label{fig:hyperplanes}
\end{figure}
\end{comment}

While we can not easily draw the correct hypersphere, there is an equivalent procedure that has a straightforward implementation in Python.
Before we finally decide to descend in one direction, we add the minimum distance to the $i^{th}$ entry of the target point's data, where $i$ is the \li{axis} of the candidate nearest neighbor.
If this sum is greater that the $i^{th}$ entry of the current search node, then the hypersphere would necessarily intersect one of the hyperplanes drawn by the tree (why?).

We summarize the algorithm below.

\begin{algorithm}[h] % K-d search algorithm.
\begin{algorithmic}[1]
\State Given a set of data and a \texttt{target}, build a $k$-d tree out of the data set.
\Procedure{Search}{\texttt{current, neighbor, dist}}
\If {\texttt{current} is \texttt{None}}
    \Comment{Base case.}
    \State \pseudoli{return} \texttt{neighbor, dist}
\EndIf
\State \texttt{index} $\gets$ \texttt{current.axis}
\If {metric(\texttt{current, target}) $<$ \texttt{dist}}
    \State \texttt{neighbor} $\gets$ \texttt{current}
        \Comment{Update the best estimate.}
    \State \texttt{dist} $\gets$ metric(\texttt{current, target})
\EndIf
\If {\texttt{target[index]} $<$ \texttt{current.value[index]}}
    \Comment{Recurse left.}
    \State \texttt{neighbor, dist} $\gets$ Search(\texttt{current.left, neighbor, dist})
    \If {\texttt{target[index]} + \texttt{dist} $\geq$ \texttt{current.value[index]}}
        \State \texttt{neighbor, dist} $\gets$ Search(\texttt{current.right, neighbor, dist})
    \EndIf
\Else
    \Comment{Recurse right.}
    \State \texttt{neighbor, dist} $\gets$ Search(\texttt{current.right, neighbor, dist})
    \If{\texttt{target[index]} - \texttt{dist} $\leq$ \texttt{current.value[index]}}
        \State \texttt{neighbor, dist} $\gets$ Search(\texttt{current.left, neighbor, dist})
    \EndIf
\EndIf
\State \pseudoli{return} \texttt{neighbor, dist}
\EndProcedure
\State Start \textsc{Search()} at the root of the tree.
\end{algorithmic}
\caption{$k$-d tree nearest neighbor search}
\label{alg:kdneighborz}
\end{algorithm}

\begin{problem} % K-d nearest neighbor search algorithm.
Use Algorithm \ref{alg:kdneighborz} to write a function that solves the nearest neighbor search problem by searching through a $k$-d tree (your \li{KDT} object).
The function should take in a data set and a single target point.
Return the nearest neighbor in the data set and the distance from the nearest neighbor to the target point, as in Problem 2 (be sure to return a NumPy array, not a KDTNode for the neighbor).

To test your function, use Scipy's built-in \li{KDTree} object.
This structure behaves like the \li{KDT} class, but its operations are heavily optimized.
To solve the nearest neighbor problem, initialize the tree with data, then `query' the tree with the target point.
The \li{query} method returns a tuple of the minimun distance and the index of the nearest neighbor in the data.

\begin{lstlisting}
>>> from scipy.spatial import KDTree

# Initialize the tree with data (in this example we use random data).
>>> data = np.random.random((100,5))
>>> target = np.random.random(5)
>>> tree = KDTree(data)

# Query the tree and print the minimum distance.
>>> min_distance, index = tree.query(target)
>>> print(min_distance)
0.309671532426

# Print the nearest neighbor by indexing into the tree's data.
>>> print(tree.data[index])
[ 0.68001084  0.02021068  0.70421171  0.57488834  0.50492779]
\end{lstlisting}
\end{problem}

\section*{Handwriting Recognition} % ==========================================

\subsection*{Classification} % ------------------------------------------------

Suppose that we are given a training set of data as well as a set of \emph{labels} that describe each datum in the training set.
For example, suppose that we had a training set containing the incomes and debt levels of $N$ individuals.
Along with this data, we have a set $N$ labels that state whether the individual has filed for bankruptcy.
The classification problem is to try and assign the correct label to an unlabelled data point.

\subsection*{$k$-Nearest Neighbors} % -----------------------------------------

In our previous work, we used a $k$-d tree to find the nearest neighbor of a target point.
A more general problem is to find the \emph{k} nearest neighbors to a point (using some metric to measure ``distance'' between data points).
In classification, we find the $k$ nearest neighbors, we let each neighbor ``vote'' to decide what label to give the new point.
For example, consider the bankrupty case in the previous section.
If we find the 10 nearest neighbors to a new individual, and 8 of them went bankrupt, then we would predict that the individual will also go bankrupt.
On the other hand, if 7 of the nearest neighbors had not filed for bankruptcy, we would predict that the individual was at low risk for bankruptcy.

\subsection*{The Handwriting Recognition Problem} % ---------------------------

The problem of recognizing handwritten letters and numbers with a computer has many applications.
A computer image may be thought of a vector in $\mathbb{R}^n$, where $n$ is the number of pixels in the image and the entries represent how bright each pixel is.
If two people write the same number, we would expect the vectors representing a scanned image of those number to be close in the euclidean metric.
This insight means that given a training set of scanned images along with correct labels, we may confidently infer the label of a new scanned image.

\subsection*{sklearn} % -------------------------------------------------------

%More information about this package can be found at http://scikit-learn.org/stable/modules/neighbors.html.
The \li{sklearn} module contains powerful tools for solving the nearest neighbor problem.
To start nearest neighbors classification, we import the neighbors module from sklearn.
This module has a class for setting up a k-nearest neighbors classifier.

\begin{lstlisting}
# Import the neighbors module
>>> from sklearn import neighbors

# Create an instance of a k-nearest neighbors classifier.
# 'n_neighbors' determines how many neighbors to give votes to.
# 'weights' may be 'uniform' or 'distance.'  The 'distance' option
#     gives nearer neighbors more weight.
# 'p=2' instructs the class to use the euclidean metric.
>>> nbrs = neighbors.KNeighborsClassifier(n_neighbors=8, weights='distance', p=2)
\end{lstlisting}

The \li{nbrs} object has two useful methods for classification.
The first, \li{fit}, will take arrays of data (the training set) and labels and put them into a $k$-d tree.
This can then be used to find k-nearest neighbors, much like the \li{KDT} class that we implemented previously.

\begin{lstlisting}
# 'points' is some NumPy array of data
# 'labels' is a NumPy array of labels describing the data in points.
>>> nbrs.fit(points, labels)
\end{lstlisting}

The second method, \li{predict}, will do a k-nearest neighbor search on the $k$-d tree and use the result to attach a label to unlabelled points.

\begin{lstlisting}
# 'testpoints' is an array of unlabeled points.
# Perform the search and calculate the accuracy of the classification.
>>> prediction = nbrs.predict(testpoints)
>>> np.average(prediction/testlabels)
\end{lstlisting}

\begin{problem} % Postal data problem.
The United States Postal Service has made a collection of labeled hand written digits available to the public, provided in \texttt{PostalData.npz}.
We will use this data for k-nearest neighbor classification.
This data set may be loaded by using the following command:

\begin{lstlisting}
labels, points, testlabels, testpoints = np.load('PostalData.npz').items()
\end{lstlisting}

This contains a training set and a test set.
The first entry of each array is a name, so \li{points[1]} and \li{labels[1]} are the actual points and labels to use.
Each point is an image that is represented by a flattened $28 \times 28$ matrix of pixels.
The corresponding label indicates which number was written.

Classify the testpoints with \li{n_neighbors} as 1, 4 or 10, and with \li{weights} as \li{'uniform'} or \li{'distance'}.
For each trial print a report indicating how your classifier performs in terms of percentage of correct classifications.
Which combination gives the most correct classifications?
(Hint: define an inner function that takes in \li{n_neighbors} and \li{weights} as arguments calls the neighbors functions appropriately)

A similar classification process is used by the United States Postal Service to automatically determine the zip code to send a letter to.

\begin{figure}[H]
\includegraphics[width=.25\textwidth]{figures/Example.png}
\caption{An example of the number 6 taken from the data set}
\end{figure}
\end{problem}

\begin{comment}
\newpage

\section*{Additional Material} % ==============================================

% TODO (there's already some material on this in the oldLab/ folder)
\subsection*{The Curse of Dimensionality} % -----------------------------------

\end{comment}


\section{Finding eigenvalues}

\begin{outcome}
  \begin{enumerate}
  \item Find the characteristic polynomial, eigenvalues, and
    eigenvectors of a matrix.
  \item Find the eigenvalues of a triangular matrix.
  \end{enumerate}
\end{outcome}

In the previous section, we saw how to find the eigenvectors
corresponding to a given eigenvalue $\eigenvar$, if $\eigenvar$ is already
known. But we have not yet seen how to find the eigenvalues of a
matrix. However, the calculations in
Examples~\ref{exa:find-eigenvectors-given-eigenvalue} and
{\ref{exa:basis-eigenspace}} suggest a way forward. We can see that
the following are equivalent:
\begin{enumerate}
\item $\eigenvar$ is an eigenvalue of $A$.
\item There exists a non-zero vector $\vect{v}$ such that
  $A\vect{v}=\eigenvar\vect{v}$.
\item The homogeneous system of equations
  $(A-\eigenvar I)\vect{v}=\vect{0}$ has a non-trivial solution.
\end{enumerate}
Indeed, the equivalence between 1 and 2 is just the definition of an
eigenvalue, and the equivalence between 2 and 3 is just algebra.
By Corollary~\ref{cor:determinant-homogeneous}, we know that the
system $(A-\eigenvar I)\vect{v}=\vect{0}$ has a non-trivial solution if
and only if $\det(A-\eigenvar I)=0$. Therefore, we have proved the
following theorem:

\begin{theorem}{Eigenvalues}{eigenvalues}
  Let $A$ be a square matrix, and let $\eigenvar$ be a scalar. Then
  $\eigenvar$ is an eigenvalue of $A$ if and only if
  \begin{equation*}
    \det(A-\eigenvar I)=0.
  \end{equation*}
\end{theorem}

\begin{example}{Finding the eigenvalues}{finding-eigenvalues}
  Find the eigenvalues of the matrix
  \begin{equation*}
    A = \begin{mymatrix}{rr}
      -5 & 2 \\
      -7 & 4 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  By Theorem~\ref{thm:eigenvalues}, a scalar $\eigenvar$ is an
  eigenvalue of $A$ if and only if $\det(A-\eigenvar I)=0$. We calculate
  the determinant:
  \begin{eqnarray*}
    \det(A-\eigenvar I)
    &=&
    \begin{absmatrix}{cc}
      -5-\eigenvar & 2 \\
      -7 & 4-\eigenvar \\
    \end{absmatrix} \\
    &=& (-5-\eigenvar)(4-\eigenvar) + 14 \\
    &=& \eigenvar^2 + \eigenvar - 6.
  \end{eqnarray*}
  Therefore, $\eigenvar$ is an eigenvalue if and only if
  $\eigenvar^2 + \eigenvar - 6 = 0$. We can find the roots of this
  equation using the quadratic formula, or equivalently, by factoring
  the left-hand side:
  \begin{equation*}
    \eigenvar^2 + \eigenvar - 6 = 0
    \iff
    (\eigenvar+3)(\eigenvar-2) = 0.
  \end{equation*}
  Therefore, the eigenvalues are $\eigenvar=-3$ and $\eigenvar=2$.
\end{solution}

\begin{example}{Finding the eigenvalues}{finding-eigenvalues2}
  Find the eigenvalues of the matrix
  \begin{equation*}
    A=\begin{mymatrix}{rrr}
      5 & -4 & 4 \\
      2 & -1 & 2 \\
      0 &  0 & 2 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  Once again, we calculate $\det(A-\eigenvar I)$:
  \begin{eqnarray*}
    \det(A-\eigenvar I)
    &=&
        \begin{absmatrix}{ccc}
          5-\eigenvar & -4 & 4 \\
          2 & -1-\eigenvar & 2 \\
          0 &  0 & 2-\eigenvar \\
        \end{absmatrix} \\
    &=&
        (5-\eigenvar)(-1-\eigenvar)(2-\eigenvar) - 2(-4)(2-\eigenvar) \\
    &=& -\eigenvar^3 + 6\eigenvar^2 - 11\eigenvar + 6 \\
    &=& (3-\eigenvar)(1-\eigenvar)(2-\eigenvar).
  \end{eqnarray*}
  The eigenvalues are the roots of this polynomial, i.e., the
  solutions of the equation
  $(\eigenvar-3)(\eigenvar-1)(2-\eigenvar)=0$. Therefore, the eigenvalues of
  $A$ are $\eigenvar=1$, $\eigenvar=2$, and $\eigenvar=3$.
\end{solution}

\begin{example}{No real eigenvalue}{no-real-eigenvalue}
  Find the eigenvalues of the matrix
  \begin{equation*}
    A=\begin{mymatrix}{rr}
      0 & -1 \\
      1 &  0 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  We have
  \begin{equation*}
    \det(A-\eigenvar I)
    ~=~
        \begin{absmatrix}{cc}
          -\eigenvar & -1 \\
          1 & -\eigenvar \\
        \end{absmatrix} \\
    ~=~
        \eigenvar^2 + 1.
  \end{equation*}
  Since $\eigenvar^2+1 = 0$ does not have any solutions in the real
  numbers, the matrix $A$ has no real eigenvalues. (However, if we
  were working over the field of complex numbers rather than real
  numbers, this matrix would have eigenvalues $\eigenvar=\pm i$).
\end{solution}

As the examples show, the quantity $\det(A-\eigenvar I)$ is always a
polynomial in the variable $\eigenvar$. A \textbf{polynomial}%
\index{polynomial} is an expression of the form
\begin{equation*}
  p(\eigenvar) = a_n\eigenvar^n + a_{n-1}\eigenvar^{n-1} + \ldots + a_1\eigenvar + a_0,
\end{equation*}
where $a_0,\ldots,a_n$ are constants called the \textbf{coefficients}%
\index{coefficient!of a polynomial}%
\index{polynomial!coefficient} of the polynomial. The polynomial
$\det(A-\eigenvar I)$ has a special name:

\begin{definition}{Characteristic polynomial}{characteristic-polynomial}
  Let $A$ be a square matrix. The expression
  \begin{equation*}
    p(\eigenvar) = \det(A-\eigenvar I)
  \end{equation*}
  is called the \textbf{characteristic polynomial}%
  \index{characteristic polynomial}%
  \index{polynomial!characteristic|see{characteristic polynomial}}%
  \index{matrix!characteristic polynomial} of $A$.
\end{definition}

\begin{example}{Characteristic polynomial}{characteristic-polynomial}
  Find the characteristic polynomial of the matrix
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      3  & 0 &  2 \\
      6  & 4 &  3 \\
      -4 & 0 & -3 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  The characteristic polynomial is
  \begin{eqnarray*}
    \det(A-\eigenvar I) \\
    &=&
        \begin{absmatrix}{ccc}
          3-\eigenvar  & 0 &  2 \\
          6  & 4-\eigenvar &  3 \\
          -4 & 0 & -3-\eigenvar \\
        \end{absmatrix} \\
    &=&
        (3-\eigenvar)\begin{absmatrix}{ccc}
          4-\eigenvar & 3 \\
          0 & -3-\eigenvar \\
        \end{absmatrix}
    + 2 \begin{absmatrix}{ccc}
          6  & 4-\eigenvar \\
          -4 & 0 \\
        \end{absmatrix} \\
    &=& (3-\eigenvar)(4-\eigenvar)(-3-\eigenvar) + 8(4-\eigenvar) \\
    &=& -\eigenvar^3 + 4\eigenvar^2 + \eigenvar - 4.
  \end{eqnarray*}
  \vspace{-8ex}\par
\end{solution}

It is time to summarize the method for finding the eigenvalues and
eigenvectors of a matrix.

\begin{procedure}{Finding eigenvalues and eigenvectors}{find-eigenvalues-vectors}
  \index{eigenvalue!calculating}%
  \index{eigenvector!calculating}%
  \index{matrix!eigenvalue!calculating}%
  \index{matrix!eigenvector!calculating}%
  Let $A$ be an $n\times n$-matrix. To find the eigenvalues and
  eigenvectors of $A$:
  \begin{enumerate}
  \item Calculate the characteristic polynomial $\det(A-\eigenvar I)$.
  \item The eigenvalues are the roots of the characteristic polynomial.
  \item For each eigenvalue $\eigenvar$, find a basis for the
    eigenvectors by solving the homogeneous system
    \begin{equation*}
      (A-\eigenvar I)\vect{v} = \vect{0}.
    \end{equation*}
  \end{enumerate}
  To double-check your work, make sure that $A\vect{v}=\eigenvar\vect{v}$
  for each eigenvalue $\eigenvar$ and associated eigenvector $\vect{v}$.
\end{procedure}

\begin{example}{Finding eigenvalues and eigenvectors}{finding-eigenvalues-eigenvectors}
  Find the eigenvalues and eigenvectors of the matrix
  \begin{equation*}
    A = \begin{mymatrix}{rrr}
      3  & 0 &  2 \\
      6  & 4 &  3 \\
      -4 & 0 & -3 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  We already found the characteristic polynomial in
  Example~\ref{exa:characteristic-polynomial}. It is
  \begin{equation*}
    p(\eigenvar)
    = \det(A-\eigenvar I)
    = -\eigenvar^3 + 4\eigenvar^2 + \eigenvar - 4.
  \end{equation*}
  Finding the roots%
  \index{characteristic polynomial!root}%
  \index{matrix!characteristic polynomial!root}%
  \index{root!of a polynomial}%
  \index{polynomial!root}
  of a cubic polynomial can be a bit tricky, but
  with some trial and error, we can find that $\eigenvar=1$ is a
  root. We can therefore factor out $(\eigenvar-1)$:
  \begin{equation*}
    p(\eigenvar) = (\eigenvar-1)(-\eigenvar^2+3\eigenvar+4).
  \end{equation*}
  Then we can use the quadratic formula to find the remaining two
  roots:
  \begin{equation*}
    \eigenvar = \frac{-3\pm\sqrt{9+16}}{-2},
  \end{equation*}
  which yields the two roots $\eigenvar=-1$ and $\eigenvar=4$. Therefore,
  we have
  \begin{equation*}
    p(\eigenvar) = -(\eigenvar-1)(\eigenvar+1)(\eigenvar-4),
  \end{equation*}
  and the eigenvalues of $A$ are $\eigenvar=1$, $\eigenvar=-1$, and
  $\eigenvar=4$. We now find the eigenvectors for each eigenvalue.
  \begin{itemize}
  \item {\bf{\underline{For $\eigenvar=1$:}}} We must solve
    $(A-I)\/\vect{v}=\vect{0}$, i.e.,
    \begin{equation*}
      \begin{mymatrix}{rrr}
        2  & 0 &  2 \\
        6  & 3 &  3 \\
        -4 & 0 & -4 \\
      \end{mymatrix}\vect{v}=\vect{0}.
    \end{equation*}
    The basic solution is
    \begin{equation*}
      \vect{v}_1 = \begin{mymatrix}{r} -1 \\ 1 \\ 1 \end{mymatrix}.
    \end{equation*}
  \item {\bf{\underline{For $\eigenvar=-1$:}}} We must solve
    $(A+I)\/\vect{v}=\vect{0}$, i.e.,
    \begin{equation*}
      \begin{mymatrix}{rrr}
        4  & 0 &  2 \\
        6  & 5 &  3 \\
        -4 & 0 & -2 \\
      \end{mymatrix}\vect{v}=\vect{0}.
    \end{equation*}
    The basic solution is
    \begin{equation*}
      \vect{v}_2 = \begin{mymatrix}{r} -1 \\ 0 \\ 2 \end{mymatrix}.
    \end{equation*}
  \item {\bf{\underline{For $\eigenvar=4$:}}} We must solve
    $(A-4I)\vect{v}=\vect{0}$, i.e.,
    \begin{equation*}
      \begin{mymatrix}{rrr}
        -1  & 0 &  2 \\
        6  & 0 &  3 \\
        -4 & 0 & -7 \\
      \end{mymatrix}\vect{v}=\vect{0}.
    \end{equation*}
    The basic solution is
    \begin{equation*}
      \vect{v}_3 = \begin{mymatrix}{r} 0 \\ 1 \\ 0 \end{mymatrix}.
    \end{equation*}
  \end{itemize}
\end{solution}

\begin{example}{A zero eigenvalue}{zero-eigenvalue}
  Let
  \begin{equation*}
    A=\begin{mymatrix}{rrr}
      2 & 2 & -2 \\
      1 & 3 & -1 \\
      -1 & 1 & 1 \\
    \end{mymatrix}.
  \end{equation*}
  Find the eigenvalues and eigenvectors of $A$.
\end{example}

\begin{solution}
  To find the eigenvalues of $A$, we first compute the characteristic
  polynomial.
  \begin{equation*}
    \det(A-\eigenvar I) =
    \begin{absmatrix}{ccc}
      2-\eigenvar & 2 & -2 \\
      1 & 3-\eigenvar & -1 \\
      -1 & 1 & 1-\eigenvar \\
    \end{absmatrix}
    = -\eigenvar^3+6 \eigenvar^2-8\eigenvar.
  \end{equation*}
  You can verify that the roots of this polynomial are
  $\eigenvar_1 = 0$, $\eigenvar_2 = 2$, $\eigenvar_3 = 4$.  Notice that
  while eigenvectors can never equal $0$, it is possible to have an
  eigenvalue equal to $0$.  Now we will find the basic
  eigenvectors.
  \begin{itemize}
  \item {\bf{\underline{For $\eigenvar_1 =0$:}}} We must solve the
    equation $(A-0I)\vect{v} = \vect{0}$. This equation becomes
    $A\vect{v}=\vect{0}$. We write the augmented matrix for this
    system and reduce to {\ef}:
    \begin{equation*}
      \begin{mymatrix}{rrr|r}
        2 & 2 & -2 & 0 \\
        1 & 3 & -1 & 0 \\
        -1 & 1 & 1 & 0 \\
      \end{mymatrix}
      \roweq\ldots\roweq
      \begin{mymatrix}{rrr|r}
        1 & 0 & -1 & 0 \\
        0 & 1 &  0 & 0 \\
        0 & 0 &  0 & 0 \\
      \end{mymatrix}.
    \end{equation*}
    The basic solution is
    \begin{equation*}
      \vect{v}_1
      =
      \begin{mymatrix}{r} 1 \\ 0 \\ 1 \end{mymatrix}.
    \end{equation*}
  \item {\bf{\underline{For $\eigenvar_2=2$:}}} We solve the
    equation $(A-2I)\vect{v} = \vect{0}$:
    \begin{equation*}
      \begin{mymatrix}{rrr|r}
        0  & 2 & -2 & 0 \\
        1  & 1 & -1 & 0 \\
        -1 & 1 & -1 & 0 \\
      \end{mymatrix}
      \roweq\ldots\roweq
      \begin{mymatrix}{rrr|r}
        1  & 0 &  0 & 0 \\
        0  & 1 & -1 & 0 \\
        0  & 0 &  0 & 0 \\
      \end{mymatrix}.
    \end{equation*}
    The basic solution is
    \begin{equation*}
      \vect{v}_2
      =
      \begin{mymatrix}{r} 0 \\ 1 \\ 1 \end{mymatrix}.
    \end{equation*}
  \item {\bf{\underline{For $\eigenvar_3=4$:}}} We solve the
    equation $(A-4I)\vect{v} = \vect{0}$:
    \begin{equation*}
      \begin{mymatrix}{rrr|r}
        -2 & 2  & -2 & 0 \\
        1  & -1 & -1 & 0 \\
        -1 &  1 & -3 & 0 \\
      \end{mymatrix}
      \roweq\ldots\roweq
      \begin{mymatrix}{rrr|r}
        1 & -1 & 0 & 0 \\
        0 &  0 & 1 & 0 \\
        0 &  0 & 0 & 0 \\
      \end{mymatrix}.
    \end{equation*}
    The basic solution is
    \begin{equation*}
      \vect{v}_3
      =
      \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix}.
    \end{equation*}
  \end{itemize}
  Thus we have found the eigenvectors $\vect{v}_1$ for $\eigenvar_1$,
  $\vect{v}_2$ for $\eigenvar_2$, and $\vect{v}_3$ for $\eigenvar_3$.
  We can double-check our answers by checking the equation
  $A\vect{v}=\eigenvar\vect{v}$ in each case:
  \begin{eqnarray*}
    A\vect{v}_1
    &=&
    \begin{mymatrix}{rrr}
      2 & 2 & -2 \\
      1 & 3 & -1 \\
      -1 & 1 & 1 \\
    \end{mymatrix}
    \begin{mymatrix}{r} 1 \\ 0 \\ 1 \end{mymatrix}
    =
    \begin{mymatrix}{r} 0 \\ 0 \\ 0 \end{mymatrix}
    = 0\vect{v}_1,
    \\
    A\vect{v}_2
    &=&
    \begin{mymatrix}{rrr}
      2 & 2 & -2 \\
      1 & 3 & -1 \\
      -1 & 1 & 1 \\
    \end{mymatrix}
    \begin{mymatrix}{r} 0 \\ 1 \\ 1 \end{mymatrix}
    =
    \begin{mymatrix}{r} 0 \\ 2 \\ 2 \end{mymatrix}
    = 2\vect{v}_2,
    \\
    A\vect{v}_3
    &=&
    \begin{mymatrix}{rrr}
      2 & 2 & -2 \\
      1 & 3 & -1 \\
      -1 & 1 & 1 \\
    \end{mymatrix}
    \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix}
    =
    \begin{mymatrix}{r} 4 \\ 4 \\ 0 \end{mymatrix}
    = 4\vect{v}_3.
  \end{eqnarray*}
  Therefore, our eigenvectors and eigenvalues are correct.
\end{solution}

We conclude this section by considering the eigenvalues of a
triangular matrix. Recall from
Definition~\ref{def:triangular-matrices} that a matrix is
\textbf{upper triangular}%
\index{matrix!upper triangular!eigenvalues of}%
\index{matrix!triangular!eigenvalues of}%
\index{upper triangular matrix!eigenvalues}%
\index{triangular matrix!eigenvalues}%
\index{eigenvalue!of triangular matrix}
if all entries below the main diagonal are
zero, and \textbf{lower triangular}%
\index{matrix!lower triangular!eigenvalues of}%
\index{lower triangular matrix!eigenvalues}
if all entries above the main
diagonal are zero.

\begin{example}{Eigenvalues of a triangular matrix}{eigenvalues-triangular-matrix}
  Find the eigenvalues of
  \begin{equation*}
    A=\begin{mymatrix}{rrr}
      1 & 2 & 4 \\
      0 & 4 & 7 \\
      0 & 0 & 6 \\
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  We calculate $\det(A - \eigenvar I) = 0$ as follows:
  \begin{eqnarray*}
    \det (A - \eigenvar I) =
    \det \begin{mymatrix}{ccc}
      1-\eigenvar & 2 & 4 \\
      0 & 4-\eigenvar & 7 \\
      0 & 0 & 6-\eigenvar \\
    \end{mymatrix} = (1-\eigenvar)(4-\eigenvar)(6-\eigenvar).
  \end{eqnarray*}
  Solving the equation $(1-\eigenvar)(4-\eigenvar)(6-\eigenvar) = 0$
  results in the eigenvalues $\eigenvar_1 = 1$, $\eigenvar_2 = 4$, and
  $\eigenvar_3 = 6$.  Thus the eigenvalues are the entries on the main
  diagonal of $A$.
\end{solution}

Clearly, the same is true for any (upper or lower) triangular
matrix. We therefore have the following proposition:

\begin{proposition}{Eigenvalues of a triangular matrix}{eigenvalues-triangular-matrix}
  Let $A$ be an upper or lower triangular matrix. Then the eigenvalues
  of $A$ are the entries on the main diagonal.
\end{proposition}


\chapter{Batch Normalization}
In this chapter, we discuss the batch normalization (BN) technique. The aim of BN is to avoid gradient explosion or disappearance, which makes the training process unstable, i.e. the gradient is either too large or too small to be a reasonable update.

\section{The Definition of Batch Normalization}
BN can be applied to any layer of a neural network, where it normalizes, scales and shifts an internal computation result. It in fact includes two parts, one for the training procedure and one for the test.

\subsection{Model w.r.t loss function: an idea to ``understand" BN}
In fact, BN is just a new model, and then use the general SGD w.r.t mini-batch type! The new model of BN is like:
\begin{equation}\label{BN-model}
L(\Theta) = \sum_{i \in I} \frac{1}{|N|} \|\tilde {f}^J(x_i;\Theta) - y_i\|^2,
\end{equation}
as $I = \{1, 2,\cdots,N\}$ is the {\bf all data}.
We can rewrite it as:
\begin{equation}\label{BN-model1}
L(\Theta) = \mathbb{E}_{i} \|\tilde {f}^J(x_i;\Theta) - y_i\|^2.
\end{equation}
Now we define $\tilde {f}^J(x_i;\Theta)$ as:
\begin{equation}
\tilde f^j = \theta^j \circ \text{\text{BN}}_{I} ( g^j\circ\tilde f^{j-1}).
\end{equation}
or, as it may perform better in practice,
\begin{equation}
\tilde f^j = \theta^j \circ g^j \circ \text{\text{BN}}_{I} (\tilde f^{j-1}),
\end{equation}
So what we will explain is based the second form.
At layer $j$, with a scaling factor $\gamma^j\in\mathbb{R}$ and a shifting $\beta^j \in \mathbb{R}$, $\text{BN}_I$ is defined as
\begin{equation}
\text{BN}_{I}(x)= \gamma^j\frac{x - \mu_I^j}{\sigma_I^j}+\beta^j \bm{1},
\end{equation}
where $\mu^j$ is the mean value and $\sigma^j$ is the standard deviation of the all date i.e.
\begin{align}
\mu_I^j &= \frac{\sum_{i\in I}  x_i }{|I|},\\
\sigma_I^j  &= \sqrt{ \delta + \frac{ \sum_{i\in I} ( x_i - \mu_I^j)^2}{|I|-1} }.
\end{align}
where $\delta$ is an extra term for numerical stability, and the square in $(  x_i - \mu^j)^2$ is an element-wise square operation.

Note that $\gamma^j$ and $\beta^j$ are parameters to be trained.

If BN is added after the linear transformation, i.e.
\begin{equation}
\text{BN}_I(f^{j-1}(X_i))=\text{BN}_{I}(WX_i + b),
\end{equation}
the bias $b$ is in fact removed by BN, thus
\begin{equation}
\text{BN}_I(f^{j-1}(X_i))=\text{BN}_I(WX_i).
\end{equation}

\subsection{Training Process:  approximated SGD}
Now to train BN model, we still use SGD, so we just sample a subset of data as $S \subset I$ and compute gradient on this mini-batch.

{\bf But to compute the $\mu^j_I$ is very expensive, and we can simply use the idea in statistics i.e use the sampled data $S$ to approximate $I$.}

So for every step, the corresponding object function to be take gradient can be written as:
\begin{equation}
	\sum_{i \in S} \|\tilde {f}^J(x_i) - y_i\|^2,
\end{equation}
where for each layer $\tilde f^j$,
\begin{equation}
\tilde f^j = \theta^j \circ \text{\text{BN}}_{S} ( g^j\circ\tilde f^{j-1}).
\end{equation}
or, as it may perform better in practice,
\begin{equation}
\tilde f^j = \theta^j \circ g^j \circ \text{\text{BN}}_{S} (\tilde f^{j-1}),
\end{equation}
So what we will explain is based the second form.

At layer $j$, with a scaling factor $\gamma^j\in\mathbb{R}$ and a shifting $\beta^j \in \mathbb{R}$, $\text{BN}_S$ is defined as
\begin{equation}
\text{BN}_{S}(x)= \gamma^j\frac{x - \mu_S^j}{\sigma_S^j}+\beta^j \bm{1},
\end{equation}
where $\mu^j$ is the mean value and $\sigma^j$ is the standard deviation of the input mini-batch, i.e.
\begin{align}
\mu^j_I \approx \mu_S^j &= \frac{\sum_{i\in S}  x_i }{|S|},\\
\mu_I^j \approx \sigma_S^j  &= \sqrt{ \delta + \frac{ \sum_{i\in S} ( x_i - \mu_S^j)^2}{|S|-1} }.
\end{align}
where $\delta$ is an extra term for numerical stability, and the square in $(  x_i - \mu^j)^2$ is an element-wise square operation.

Note that $\gamma^j$ and $\beta^j$ are parameters to be trained.

With the trainable parameters in $\text{BN}_S$, the derivative now becomes
\begin{align}
\frac{\partial \tilde f^j}{ \partial \theta^{j-1}} &= \theta^j \circ (g^j)' \circ \frac{\partial \text{BN}_{S} (\tilde f^{j-1})}{ \partial \theta^{j-1}} \\
&=  \theta^j \circ (g^j)' \circ \left(\frac{\partial \text{BN}_{S} (\tilde f^{j-1})}{ \partial \mu^j}\frac{\partial \mu^j}{\partial \theta^j} + \frac{\partial \text{BN}_{S} (\tilde f^{j-1})}{ \partial \sigma^j}\frac{\partial \sigma^j}{\partial \theta^j} + \frac{\partial \text{BN}_{S} (\tilde f^{j-1})}{ \partial \tilde f^j}\frac{\partial \tilde f^j}{\partial \theta^j}  \right).
\end{align}
The gradient of $\gamma$ and $\beta$ are trivial. The back-propagation process can then be applied to optimize the model.

\subsection{Testing Process: recover $\mu_I$ and $\sigma_I$ from all samples}
Considering that a mini-batch is an approximation of the whole dataset, the mean value $\mu$ and standard deviation $\sigma$ of BN in each layer, should be the same, all being of the whole dataset.

%at the testing time, they are replaced by those of the whole dataset, i.e.
%\begin{align}\label{bnbatch}
%	\mu_I&\approx \mu = \frac{1}{n_\mathcal{B}}\sum_{B\in\mathcal{B}}\mu_B,\\
%	\sigma^2_I&\approx \sigma^2 = \frac{1}{n_\mathcal{B}}\sum_{B\in\mathcal{B}}\sigma^2_{\mathcal{B}},
%\end{align}
%where $\mathcal{B}$ is the set of all mini-batches.
Thus, after optimizing the model,  use the moving average scheme as:
\begin{equation}\label{BN-timeaverage}
\mu_t=\rho*\mu_t+(1-\rho)*\bar\mu_{t-1} \quad \sigma_t=\rho*\beta_t+(1-\rho)*\bar\sigma_{t-1},
\end{equation}
with $\mu_t$ and $\sigma_t$ is the $\mu_{S_t}$ and $\sigma_{S_t}$ in step t. {\bf But we don't use the updated $\mu_{S_t}$ or $\sigma_{S_t}$ in training phase, we just note them and use for approximate the real $\mu_I$ and $\sigma_I$ in previous BN model.}

The parameters $\gamma$ and $\beta$ will be kept.

\subsection{Implementation of batch normalization}
Batch normalization (BN) can be viewed as a function $\text{BN}(x)$, where

\begin{enumerate}
	\item in 1D case, $x\in\mathbb{R}^{m\times C\times n}$,
	\item in 2D case, $x\in\mathbb{R}^{m\times C\times H \times W}$,
	\item in 3D case, $x\in \mathbb{R}^{m\times C\times D \times H \times W}$.
\end{enumerate}
What matters is $m$, the number of input samples, and $C$, the number of channels.

$H$, $W$, $D$ means height, width, and depth respectively, in 2D or 3D case, and $n$ simply means the length of a sample in 1D case.

Because only $m$ and $C$ are necessary for BN, we will denote $x_{i,c}$ as a tensor, where $i$, $c$ is the first and the second index of $x$, i.e. the dimension of $n$, $H$, $W$, $D$ are omitted.

The definition of $\text{BN}(x)$, denoted as $y$, is
\begin{equation}
	\begin{aligned}
		y&=\text{BN}(x),\\
		y_{i,c}&=\gamma_c\frac{x_{i,c}-\text{mean}_i[x_{i,c}]}{\sqrt[]{\text{Var}_i[x_{i,c}]+\epsilon}}+\beta_c,
	\end{aligned}
\end{equation}
where $\text{mean}_i[x_{i,c}]$ is the mean of $x_{i,c},\ i\in I$ and $\text{Var}_i[x_{i,c}]$ is the variance of $x_i, i\ \in I$. $\gamma_c,\beta_c\in\mathbb{R}$ are the trainable parameters, often called the scale and the shift, respectively. $\epsilon$ is used for numerical stability.

In the training process, like all the other parameters such as $W$ and $b$, $\gamma,\beta$ will be updated with $\nabla_\gamma L$ and $\nabla_\beta L$, where $L$ is the loss function.

In practice, mean $\mu$ and var $\sigma$ are in fact the running estimates $\bar\mu$, $\bar\sigma$, i.e.
\begin{equation}
	\begin{aligned}
		&\text{Initialize running estimates, }&\bar\mu_0=0,\ \bar\sigma_0=1,\\
		&\text{At time step t, compute mean and var, }&\mu_t=\text{mean}_i[x_{i,c}],\ \sigma_t=\text{Var}_i[x_{i,c}],\\
		&\text{Update running estimates, }&\bar\mu_t=\rho*\mu_t+(1-\rho)*\bar\mu_{t-1},\\
		&&\bar\sigma_t=\rho*\beta_t+(1-\rho)*\bar\sigma_{t-1},\\
	\end{aligned}
\end{equation}
where $\rho$ is the momentum, $\rho\in(0,1)$ and $\rho\approx 0.1$ in practice.

In the testing process, the final $\bar\mu$ and $\bar\sigma$ are used to normalize the input.

BN is often used before activation functions, e.g. a one-hidden-layer neural network with BN can be written as
\begin{equation}
	\label{bn_1hidden}
	f(x)=\theta_2\circ \text{ReLU1}\circ\text{BN}\circ\theta_1\circ x.
\end{equation}

\begin{figure}[htbp]
	\centering{\includegraphics[width=10cm]{BN_NoBN.png}}
	\caption{Comparison of BN on ResNet10, CIFAR-10}
	\label{alo: large k}
\end{figure}

%{\bf For test phase}, to the $\hat \mu^j$ and $\hat \sigma^j$ are computed
%by averaging those $\mu^j$ and $\sigma^j$ in training phase. In training phase, suppose we have those next training process with mini-batch $S_t$ for $t = 1:T$. So, after $T$ steps gradient descent, those $\theta^j$ have been trained well, and we get:
%\begin{equation}
%\mu^j_{t}, \sigma^j_t, \quad t = 1:T,
%\end{equation}
%So the test model is
%\begin{equation}
%f(x) = \hat{f}^J(x),
%\end{equation}
%with
%\begin{equation}
% \hat{f}^j(x) = \theta^j \circ g^j \circ \hat{\text{BN}}^j(\hat f^{j-1})
%\end{equation}
%for
%\begin{equation}
%\hat{\text{BN}}^j( \hat f^{j-1}(x_i) )= \frac{\hat f^{j-1}(x_i) - \hat \mu^j}{\hat \sigma^j},
%\end{equation}
%where
%\begin{equation}
%\hat \mu^j =\frac{\sum_{i=1}^T \mu^j_t }{T},
%\end{equation}
%and
%\begin{equation}
%\hat \sigma^j  = \frac{\sum_{i=1}^T \sigma^j_t }{T}.
%\end{equation}
%
%In some situation, they may use the similar strategy like the momentum for SGD, then may get $\hat \mu^j$ and $\hat \sigma^j$ by: For $t= 1 :T$
%\begin{equation}
%\hat \mu^j = 0.9 \hat \mu^j + 0.1 \mu^j_t,
%\end{equation}
%and the similar operation for $\hat \sigma^j$

\subsection{An example}
Now consider a simple case in the Deep Learning Book, where
\begin{enumerate}
\item there's no activation function,
\item all layers have only one neuron and no bias, i.e. $\theta(x) = wx$.
\end{enumerate}
$f$ is further defined to have $J$ layers, i.e.
\begin{equation}
	f(x) = w_J w_{J-1} \cdots w_2 w_1 x,
\end{equation}
with the loss function
\begin{equation}
	L(w_1, \cdots, w_J) = \sum_{i = 1}^N\frac{1}{2} (f(x_i) - y_i)^2.
\end{equation}
The gradients can then be computed as
\begin{align}
	\frac{\partial L}{\partial f(x_i)}&=f(x_i)-y_i,\\
	\frac{\partial f}{\partial w_i}&=\frac{f(x)}{w_i}=x\prod_{j\neq i}w_j.
\end{align}
The problem is that, if $w_j,2\leq j\leq J$ are updated with $\epsilon\Delta w_j$, the update of $w_1$ would be
\begin{equation}
	\frac{\partial f}{\partial w_1}=x\prod_{j\geq2}(w_j+\epsilon\Delta w_j),
\end{equation}
which multiplies the change of parameters, and when $J$ is larger, the effect is stronger. If the change is scaling up, this may introduce gradient explosion, or disappearance if it's scaling down.

One may adjust the learning rate $\epsilon$ to moderate this effect, but this is hard in practice.

%Now we suppose to decrease the value for $f$ by $0.1$ at $x$. With the first order approximation of $f$, we know that $f$ will decrease about
%\begin{equation}
%\epsilon \|\bm{g}\|^2,
%\end{equation}
%with
%\begin{equation}
%\bm{w} \leftarrow \bm{w} - \epsilon \bm{g}.
%\end{equation}
%
%However, for really situation, we have
%\begin{equation}
%f(x) = (w_J - \epsilon g_J)\cdots(w_1 - \epsilon g_1)x.
%\end{equation}
%For one term in seconder approximation like:
%\begin{equation}
%\epsilon^2 g_1 g_2 \Pi_{i=3}^J w_i x.
%\end{equation}
%Because we connot control the value of $\Pi_{i=3}^J w_i x$, it can be very small or extremely large so {\bf it it very difficult to choose suitable learning rate.}

With BN, the model is changed to
\begin{equation}
\tilde f(x) = w_J \tilde f^{J-1}(x),
\end{equation}
where the update is normalized.
%with $\tilde f^{J-1}$ has the stander statistics( or $\gamma$ and $\eta$ by adaptivity). If we don't use \text{BN}, we know that for any $w_j$ with $j = 1:J-1$ can effect the $f(x)$ remarkably like $w_j = 0$ makes this network degeneration or $w_j = -w_j$ changes the prediction of $f(x)$ totally. Without normalization, nearly every update would have an extreme effect on the statistics of $f^{J-1}$.
%
%Now suppose that $x$ is drawn from a unit Gaussian i.e:
%\begin{equation}
%x \sim \mathcal{N}(0,1).
%\end{equation}
% Then the original method $f$ is also Gaussian but not unit it depended w.r.t $\bm{w}$:
%\begin{equation}
%f \sim \mathcal{N}(\mu({\bm{w}}),\sigma(\bm{w})).
%\end{equation}
%But if we use \text{BN}, here $\tilde h^{J-1}$ becomes unit Gaussian again, i.e
%\begin{equation}
%\tilde f^{J-1} \sim \mathcal{N}(0,1).
%\end{equation}  So $\tilde f$ is an Gaussian only depends on $\omega_J$,
%\begin{equation}
%\tilde f \sim \mathcal{N}(\mu(w_J),\sigma(w_J)).
%\end{equation}  which means we just need to trained $w_J$ all $w_j$ for $j=1:J-1$ make no sense for this model. So, what we only need to train is only $w_J$, this make the training process very fast.

As it says in the book,
\begin{quotation}
	\emph{
		Batch normalization has thus made this model significantly easier to learn. In this example, the ease of learning of course came at the cost of making the lower layers useless. In our linear example, the lower layers no longer have any harmful effect, but they also no longer have any beneficial effect. This is because we have normalized out the first and second order statistics, which is all that a linear network can influence. In a deep neural network with nonlinear activation functions, the lower layers can perform nonlinear transformations of the data, so they remain useful. Batch normalization acts to standardize only the mean and variance of each unit in order to stabilize learning, but allows the relationships between units and the nonlinear statistics of a single unit to change.}
\end{quotation}

%\input{6DL/XN}

%\input{6DL/jinchao-XN}


\chapter{Sum of Random Variables of Probability Distribution}
\label{ch:sum_rv}

\section{Sum $R = X + Y$}

StatAnaly supports the sum of random variable in the format of $R = X + Y$, where $X$ and $Y$ are random variables of probability distributions.
The supported combinations of the distributions are listed in the section.

\subsection{Standard Uniform \& Standard Uniform (continuous)}
    \begin{equation}
        R = \sum_{i=1}^2 X_i \qquad
        X_i \sim \text{StdUniform}, \quad
        R \sim \text{Irwin-Hall}(2)
    \end{equation}

\subsection{Normal \& Normal}

    \begin{equation}
        R = \sum_{i=1}^2 X_i \qquad
        X_i \sim \text{Normal}(\mu_i,\sigma_i), \quad
        R \sim \text{Normal}\left(\sum _{i=1}^{2}\mu _{i},\sum _{i=1}^{2}\sigma _{i}^{2}\right)
    \end{equation}

\subsection{Cauchy \& Cauchy}

    \begin{equation}
        R = \sum_{i=1}^2 X_i \qquad
        X_i \sim \text{Cauchy}(m_i,b_i), \quad
        R \sim \text{Cauchy}\left(\sum _{i=1}^{2}m_{i},\sum _{i=1}^{2}b _{i}\right)
    \end{equation}

\subsection{Gamma \& Gamma}

    \begin{equation}
        R = \sum_{i=1}^2 X_i \qquad
        X_i \sim \text{Gamma}(\alpha_i,\theta), \quad
        R \sim \text{Gamma}\left(\sum _{i=1}^{2}\alpha_{i},\theta \right)
    \end{equation}

\subsection{Exponential \& Exponential}

    \begin{equation}
        R = \sum_{i=1}^2 X_i \qquad
        X_i \sim \text{Exponential}(\theta), \quad
        R \sim \text{Erlang}\left(2, \theta \right)
    \end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{List of supported sum $R = X^2 + Y^2$}

StatAnaly supports the sum of random variable in the format of $R = X^2 + Y^2$, where $X$ and $Y$ are random variables of probability distributions.
The supported combinations of the distributions are listed in the section.

\subsection{Normal \& Normal}

\begin{equation}
    R = \sum_{i=1}^2 X_i^2 \qquad
    X_i \sim \text{Normal}(0,1), \quad
    R \sim \text{Chi-Squared}\left( 2 \right)
\end{equation}

\begin{equation}
    R = \sum_{i=1}^2 X_i^2 \qquad
    X_i \sim \text{Normal}(\mu_i,1), \quad
    R \sim \text{Non-central Chi-Squared}\left(2, \sum _{i=1}^{2}\mu _{i}^2\right)
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{List of supported sum $R = \sqrt{X^2 + Y^2}$}

StatAnaly supports the sum of random variable in the format of $R = \sqrt{X^2 + Y^2}$, where $X$ and $Y$ are random variables of probability distributions.
The supported combinations of the distributions are listed in the section.

\subsection{Normal \& Normal}

\begin{equation}
    R = \sqrt{ \sum_{i=1}^2 X_i^2 } \qquad
    X_i \sim \text{Normal}(0,\sigma^2), \quad
    R \sim \text{Rayleigh}\left( \sigma \right)
\end{equation}

\begin{equation}
    R = \sqrt{ \sum_{i=1}^2 X_i^2 } \qquad
    X_i \sim \text{Normal}(\mu_i,\sigma^2), \quad
    R \sim \text{Rician}\left(\sqrt{\sum _{i=1}^{2}\mu _{i}^2}, \sigma \right)
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{List of supported sum $R = \sum_i X_i$}

StatAnaly supports the sum of random variable in the format of $R = \sum_i X_i$, where $X$ is random variable of a probability distribution.
The supported combinations of the distributions are listed in the section.

\subsection{Array of Standard Uniform}
    \begin{equation}
        R = \sum_{i=1}^n X_i \qquad
        X_i \sim \text{StdUniform}, \quad
        R \sim \text{Irwin-Hall}(n)
    \end{equation}

\subsection{Array of Normal}
    \begin{equation}
        R = \sum_{i=1}^n X_i \qquad
        X_i \sim \text{Normal}(\mu_i,\sigma_i), \quad
        R \sim \text{Normal}\left(\sum _{i=1}^{n}\mu _{i},\sum _{i=1}^{n}\sigma _{i}^{2}\right)
    \end{equation}

\subsection{Array of Cauchy}

    \begin{equation}
        R = \sum_{i=1}^n X_i \qquad
        X_i \sim \text{Cauchy}(m_i,b_i), \quad
        R \sim \text{Cauchy}\left(\sum _{i=1}^{n}m_{i},\sum _{i=1}^{n}b _{i}\right)
    \end{equation}

\subsection{Array of Gamma}

    \begin{equation}
        R = \sum_{i=1}^n X_i \qquad
        X_i \sim \text{Gamma}(\alpha_i,\theta), \quad
        R \sim \text{Gamma}\left(\sum _{i=1}^{n}\alpha_{i},\theta \right)
    \end{equation}

\subsection{Array of Exponential}

    \begin{equation}
        R = \sum_{i=1}^n X_i \qquad
        X_i \sim \text{Exponential}(\theta), \quad
        R \sim \text{Erlang}\left(n, \theta \right)
    \end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{List of supported sum $R = \sum_i X_i^2$}

StatAnaly supports the sum of random variable in the format of $R = \sum_i X_i^2$, where $X$ is random variable of a probability distribution.
The supported combinations of the distributions are listed in the section.

\subsection{Array of Normal}

\begin{equation}
    R = \sum_{i=1}^n X_i^2 \qquad
    X_i \sim \text{Normal}(0,1), \quad
    R \sim \text{Chi-Squared}\left( n \right)
\end{equation}

\begin{equation}
    R = \sum_{i=1}^n X_i^2 \qquad
    X_i \sim \text{Normal}(\mu_i,1), \quad
    R \sim \text{Non-central Chi-Squared}\left(2, \sum _{i=1}^{n}\mu _{i}^2\right)
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{List of supported sum $R = \sqrt{\sum_i X_i^2}$}

StatAnaly supports the sum of random variable in the format of $R = \sqrt{\sum_i X_i^2}$, where $X$ is random variable of a probability distribution.
The supported combinations of the distributions are listed in the section.

\subsection{Array of Normal}

\begin{equation}
    R = \sqrt{ \sum_{i=1}^n X_i^2 } \qquad
    X_i \sim \text{Normal}(0,1), \quad
    R \sim \text{Chi}\left( n \right)
\end{equation}

\begin{equation}
    R = \sqrt{ \sum_{i=1}^n X_i^2 } \qquad
    X_i \sim \text{Normal}(\mu_i,1), \quad
    R \sim \text{Non-central Chi} \left( 2, \sqrt{\sum _{i=1}^{n}\mu _{i}^2} \right)
\end{equation}