\lab{K-Means Clustering}{K-Means Clustering}
\objective{Understand the basics of \emph{k-means} clustering, and apply to the problem of clustering earthquake epicenters.}

\subsection*{Clustering}
In Lab \ref{lab:pca}, we analyzed the iris dataset using PCA; we have reproduced the first two principal components of the iris data in Figure \ref{fig:iris_data}.
Upon inspection, a human can easily see that there are two very distinct groups of irises.
Can we create an algorithm to identify these groups without human supervision?
This task is called \emph{clustering}, an instance of \emph{unsupervised learning}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{iris_pca.pdf}
\caption{The first two principal components of the iris dataset.}
\label{fig:iris_data}
\end{figure}

The objective of clustering is to find a partition of the data such that points in the same subset will be ``close'' according to some metric.
The metric used will likely depend on the data, but some obvious choices include Euclidean distance and angular distance.
Throughout this lab we will use the metric $d(x,y) = \|x-y\|_2$, the Euclidean distance between $x$ and $y$.

More formally, suppose we have a collection of $\mathbb{R}^K$-valued observations $X = \{x_1,x_2,\ldots,x_n\}$.
Let $N \in \mathbb{N}$ and let $\mathcal{S}$ be the set of all $N$-partitions of $X$, where an $N$-partition is a partition with exactly $N$ nonempty elements.
We can represent a typical partition in $\mathcal{S}$ as $S = \{S_1,S_2,\ldots,S_N\}$, where
\[
X = \bigcup_{i=1}^N S_i
\]
and
\[
|S_i| > 0, \qquad i=1,2,\ldots,N.
\]
We seek the $N$-partition $S^*$ that minimizes the within-cluster sum of squares, i.e.
\[
S^* = \underset{S\in\mathcal{S}}{\arg\min} \sum_{i=1}^N\sum_{x_j\in S_i}\|x_j-\mu_i\|_2^2,
\]
where $\mu_i$ is the mean of the elements in $S_i$, i.e.
\[
\mu_i = \frac{1}{|S_i|}\sum_{x_j\in S_i}x_j.
\]

\subsection*{The \emph{K-Means} Method}
Finding the global minimizing partition $S^*$ is generally intractable since the set of partitions can be very large indeed,
but the \emph{k-means} algorithm is a heuristic approach that can often provide reasonably accurate results.

We begin by specifying an initial cluster mean $\mu_i^{(1)}$ for each $i = 1, \cdots, N$ (this can be done by random initialization, or according to some heuristic).
For each iteration, we adopt the following procedure.
Given a current set of cluster means $\mu^{(t)}$, we find a partition $S^{(t)}$ of the observations such that
\begin{equation*}
S_{i}^{(t)} = \{x_j \; : \; \|x_j - \mu_{i}^{(t)}\|_2^2 \leq \|x_j - \mu_{l}^{(t)}\|_2^2,\,\,\,  l = 1, \cdots, N\}.
\end{equation*}
We then update our cluster means by computing for each $i = 1, \cdots, N$.
We continue to iterate in this manner until the partition ceases to change.

Examine Figure \ref{fig:iris_clusterings}, which shows two different clusterings of the iris data produced by the \emph{k-means} algorithm.
Note that the quality of the clustering can depend heavily on the initial cluster means.
We can use the within-cluster sum of squares as a measure of the quality of a clustering (a lower sum of squares is better).
Where possible, it is advisable to run the clustering algorithm several times, each with a different initialization of the means,
and keep the best clustering.
Note also that it is possible to have very slow convergence.
Thus, when implementing the algorithm, it is a good idea to terminate after some specified maximum number of iterations.

\begin{figure}[h]
	\centering
	\begin{tabular}{cc}
	\includegraphics[width=.49\textwidth]{iris_means_1.pdf} &
	\includegraphics[width=.49\textwidth]{iris_means_2.pdf}
	\end{tabular}
	\caption{Two different K-Means clusterings for the iris dataset.
            Notice that the clustering on the left predicts the flower species to a high degree of accuracy,
            while the clustering on the right is less effective.}
    \label{fig:iris_clusterings}
\end{figure}

\begin{problem}
Implement the \emph{k-means} algorithm using the following function declaration.

\begin{lstlisting}
def kmeans(data,n_clusters,init='random',max_iter=300):
    """
    Cluster a dataset using the k-means algorithm.

    Parameters
    ----------
    data : ndarray of shape (n,k)
        Each row is an observation.
    n_clusters : int
        The number of clusters.
    init : string or ndarray of shape (n_clusters,k)
        If init is the string 'random', then randomly initialize the cluster means.
        Else, the initial cluster means are given by the rows of init.
    max_iter : int
        The maximum allowable number of iterations.

    Returns
    -------
    means : ndarray of shape (n_cluster,k)
        The final cluster means, given as the rows.
    labels : ndarray of shape (n,)
        The i-th entry is an integer in [0,n_clusters-1] indicating
        which cluster the i-th row of data belongs to relative to
        the rows of means.
    measure : float
        The within-cluster sum of squares quality measure.
    """
    pass
\end{lstlisting}

Test your function on the first two principal components of the iris dataset.
Run it 10 times, using a different random initialization of the means each time.
Retain the clustering with the smallest within-cluster sum of squares.
Your clustering should be similar to the first clustering in Figure \ref{fig:iris_clusterings}.
\end{problem}

\subsection*{Detecting Active Earthquake Regions}
Suppose we are interested in learning about which regions are prone to experience frequent earthquake activity.
We could make a map of all earthquakes over a given period of time and examine it ourselves, but this, as an unsupervised learning problem, can be solved using our k-means clustering tool.

Our data is contained in 6 text files, each with earthquake data throughout the world covering a time period of one month, giving us data from January 2010 through June 2010.
These files contain a lot of information which isn't of interest to us at the present time; all we would like to extract from them is the location of each earthquake, which appears in characters $21$ through $33$ of each line.
Characters $21$ through $26$ contain the latitude of each epicenter, character $26$ denoting North or South, and characters $27$ through $33$ contain the longitude of each epicenter, character $33$ denoting East or West.
We need to divide each value by $1,000$ to represent these as degrees and decimals.
\begin{figure}
	\centering
	\includegraphics[width=\textwidth]{earthquakes.png}
	\caption{Earthquake epicenters over a 6 month period.}
	\label{fig:earthquakes}
\end{figure}
\begin{problem}
Load the earthquake data into a $n\times 2$ array, where each row gives the longitude and latitude of an earthquake in degrees.
Multiply South latitudes and West longitudes by $-1$.
Create a scatter plot of the resulting data. You should be able to see the outlines of some of the continents and tectonic plates (since these are often areas of significant seismic activity). Your plot should match Figure \ref{fig:earthquakes}.
\end{problem}

We want to cluster this data into active earthquake regions.
For this task, we might think that we can regard any epicenter as a point in $\mathbb{R}^{2}$ with coordinates being their latitude and longitude.
This, however, would be incorrect, because the earth is not flat. We must recognize that latitude and longitude are best viewed as a variation of spherical coordinates in $\mathbb{R}^{3}$, and we should interpret them as such.
Since our \emph{k-means} algorithm is based on Euclidean distance, we need to transform our data into 3-dimensional Euclidean coordinates.

A simple way to accomplish this transformation is to first transform the latitude and longitude values to spherical coordinates, and then to Euclidean coordinates.
Recall that a spherical coordinate in $\mathbb{R}^3$ is a triple $(r,\theta,\varphi)$, where $r$ is the distance from the origin, $\theta$ is the radial angle in the $xy$-plane from the $x$-axis,
and $\varphi$ is the angle from the $z$-axis. In our earthquake data, the longitude is already the appropriate $\theta$ value, and the $\varphi$ value (in degrees) is simply $90^\circ$ minus the latitude.
For simplicity, we can take $r=1$, since the earth is roughly a sphere.
We can then transform to Euclidean coordinates using the following relationships:
\begin{align*}
r & = \sqrt{x^{2} + y^{2} + z^{2}} & x & = r \sin \varphi \cos \theta \\
\varphi & = \arccos \frac{z}{r} & y & = r \sin \varphi \sin \theta \\
\theta & = \arctan \frac{y}{x} & z & = r \cos \varphi
\end{align*}

\begin{problem}
Transform your earthquake data into three dimensional Euclidean coordinates.
Be sure to consider if and when you need to transform your data from degrees to radians.
\end{problem}

We are now ready to cluster the earthquake data using the Euclidean coordinates.
We need to address one further issue, however.
Notice that each earthquake data point has norm 1 in Euclidean coordinates, since it lies on the surface of a sphere of radius 1.
We also need to ensure that our cluster means have norm 1.
Otherwise, the means can't be interpreted as locations on the surface of the earth.
Furthermore, the \emph{k-means} algorithm will struggle to find good clusters.
A solution to this problem is to normalize the mean vectors at each iteration, so that they are always unit vectors.
Thus, we need to add optional functionality to our \li{kmeans} function.
\begin{figure}
	\centering
	\includegraphics[width=\textwidth]{earthquake_clusters.png}
	\caption{Earthquake epicenter clusters with $N = 15$.}
	\label{fig:earthquakeclusters}
\end{figure}

\begin{problem}
Add a keyword argument \li{normalize=False} to your \li{kmeans} function, and add code to normalize the means at each iteration, should this argument be set to \li{True}.
Use your function to cluster the earthquake data into 15 clusters. Run this 10 times, keeping the best clustering.
Transform the cluster means back to latitude and longitude coordinates (when calculating $\theta$ using the inverse tangent, use \li{numpy.arctan2} or \li{math.arctan2},
so that that correct quadrant is chosen).
Create a scatter plot showing each cluster mean, along with the earthquake epicenters color-coded according to their cluster. Your plot should resemble that of Figure \ref{fig:earthquakeclusters}.
\end{problem}

Though plotting our results in two dimensions gives us a good picture, we can see that this is not entirely accurate.  There are points that appear to be closer to a different cluster center than the one to which they belong.  This comes from viewing the results in only two dimensions.  When viewing in three dimensions, we can see more clearly the accuracy of our results.
\begin{problem}
Add a keyword argument \li{3d=False} to your \li{kmeans} function, and add code to show the three-dimensional plot instead of the two-dimensional scatter plot should this argument be set to \li{True}.  Maintain the same color-coding scheme as before.  Use \li{mpl_toolkits.mplot3d.Axes3D} to make your plot.
\end{problem}

\subsection*{Spectral Clustering}
We now turn to another method for solving a clustering problem, namely that of Spectral Clustering.  As you can see in Figure ???, it can cluster data not just by its location on a graph, but can even separate shapes that overlap others into distinct clusters.  It does so by utilizing the spectral properties of a Laplacian matrix.  Different types of Laplacian matrices can be used.  In order to construct a Laplacian matrix, we first need to create a graph of vertices and edges from our data points.  This graph can be represented as a symmetric matrix $W$ where $w_{ij}$ represents the edge from $x_i$ to $x_j$.  In the simplest approach, we can set $w_{ij} = 1$ if there exists an edge and $w_{ij} = 0$ otherwise.  However, we are interested in the similarity of points, so we will weight the edges by using a \emph{similarity measure}.  Points that are similar to one another are assigned a high similarity measure value, and dissimilar points a low value.  One possible measure is the \emph{Gaussian similarity function}, which defines the similarity between distinct points $x_i$ and $x_j$ as
\begin{equation*}
s(x_i,x_j) = e^{- \frac{\| x_i - x_j \| ^2}{2 \sigma ^2}}
\end{equation*}
for some set value $\sigma$.

Note that some similarity functions can yield extremely small values for dissimilar points.  We have several options for dealing with this possibility.  One is simply to set all values which are less than some $\epsilon$ to be zero, entirely erasing the edge between these two points.  Another option is to keep only the $T$ largest-valued edges for each vertex.  Whichever method we choose to use, we will end up with a weighted \emph{similarity matrix} $W$.  Using this we can find the diagonal \emph{degree matrix} $D$, which gives the number of edges found at each vertex.  If we have the original fully-connected graph, then $D_{ii} = n-1$ for each $i$.  If we keep the $T$ highest-valued edges, $D_{ii} = T$ for each $i$.

As mentioned before, we may use different types of Laplacian matrices.  Three such possibilities are:
\begin{enumerate}
    \item The \emph{unnormalized Laplacian}, $L = D - W$
    \item The \emph{symmetric normalized Laplacian}, $L_{sym} = I - D^{-1/2}WD^{-1/2}$
    \item The \emph{random walk normalized Laplacian}, $L_{rw} = I - D^{-1}W$.
 \end{enumerate}

Given a similarity measure, which type of Laplacian to use, and the desired number of clusters $k$, we can now proceed with the Spectral Clustering algorithm as follows:

\begin{itemize}
    \item Compute $W$, $D$, and the appropriate Laplacian matrix.
    \item Compute the first $k$ eigenvectors $u_1, \cdots , u_k$ of the Laplacian matrix.
    \item Set $U = [u_1, \cdots , u_k]$, and if using $L_{sym}$ or $L_{rw}$ normalize $U$ so that each row is a unit vector in the Euclidean norm.
    \item Perform $k$-means clustering on the $n$ rows of $U$.
    \item The $n$ labels returned from your \li{kmeans} function correspond to the label assignments for $x_1, \cdots, x_n$.
\end{itemize}

As before, we need to run through our $k$-means function multiple times to find the best measure when we use random initialization.  Also, if you normalize the rows of $U$, then you will need to set the argument \li{normalize = True}.

\begin{problem}
Implement the Spectral Clustering Algorithm by calling your \li{kmeans} function, using the following function declaration:
\begin{lstlisting}
def specClus(measure,Laplacian,args,arg1=None,kiters=10):
    """
    Cluster a dataset using the k-means algorithm.

    Parameters
    ----------
    measure : function
        The function used to calculate the similarity measure.
    Laplacian : int in {1,2,3}
        Which Laplacian matrix to use. 1 corresponds to the unnormalized,
        2 to the symmetric normalized, 3 to the random walk normalized.
    args : tuple
        The arguments as they were passed into your k-means function,
        consisting of (data, n_clusters, init, max_iter, normalize). Note
        that you will not pass 'data' into your k-means function.
    arg1 : None, float, or int
        If Laplacian==1, it should remain as None
        If Laplacian==2, the cut-off value, epsilon.
        If Laplacian==3, the number of edges to retain, T.
    kiters : int
        How many times to call your kmeans function to get the best
        measure.

    Returns
    -------
    labels : ndarray of shape (n,)
        The i-th entry is an integer in [0,n_clusters-1] indicating
        which cluster the i-th row of data belongs to.
    """
    pass
\end{lstlisting}
\end{problem}

You may use the following code to test your function.  This will generate random data for up to six concentric circles, and will show the results using both Spectral Clustering and $k$-means.  Note that it may take several tries to correctly classify the data, and will take a long time to run.  As arguments for this function, you need to pass in two lists giving the radii and number of points for each concentric circle, respectively, which Laplacian, and the additional argument \li{arg1} as described previously. If using \li{radii = [2.,6.]} and \li{pts = [300,700]},  your results should be similar to Figure ???.
\begin{lstlisting}
def test_specClus(radii,pts,Laplacian,arg1=None,kiters=10):
    n_clusters = len(radii)
    def measure(xi,xj):
        return np.exp(-(la.norm(xi-xj)**2))

    def circles(radii,pts):
        ttl_pts = np.sum(pts)
        positions = np.random.choice(range(ttl_pts),ttl_pts,replace=False)
        data = np.empty((ttl_pts,2))
        place = 0
        for i in xrange(n_clusters):
            randnessx = np.random.random(pts[i])-.5
            randnessy = np.random.random(pts[i])-.5
            data0 = 2*np.pi*np.random.random(pts[i])
            datax = radii[i]*np.cos(data0)+randnessx
            datay = radii[i]*np.sin(data0)+randnessy
            data[positions[place:place+pts[i]],0] = datax
            data[positions[place:place+pts[i]],1] = datay
            place += pts[i]
        return data

    data = circles(radii,pts)
    init = 'random'
    max_iters = 300
    normalize = False
    if Laplacian != 1:
        normalize = True
    args = (data,n_clusters,init,max_iters,normalize)

    spec_clus_labels = specClust(measure,Laplacian,args,arg1,kiters)
    kmeans_labels = kmeans(data,n_clusters,init,max_iters,normalize)[1]
    colors = ['r.','g.','b.','k.','c.','y.']

    plt.subplot(121)
    for i in xrange(n_clusters):
        rel_data_pos = np.where(spec_clus_labels==i)[0]
        plt.plot(data[rel_data_pos,0],data[rel_data_pos,1],colors[i])
    plt.subplot(122)
    for i in xrange(n_clusters):
        rel_data_pos = np.where(kmeans_labels==i)[0]
        plt.plot(data[rel_data_pos,0],data[rel_data_pos,1],colors[i])
    plt.show()
\end{lstlisting}


\section{Nonlinear Support Vector Machines}

When applying our SVC to \emph{linearly separable} data in~\eqref{eq:svc_max_wolfe_dual}, we have started by creating a matrix $Q$ from the dot product of our input variables:

\begin{equation} \label{eq:svc_hessian}
	Q_{ij} = y_i y_j k(x_i,x_j)
\end{equation}

or, a matrix $K$ from the dot product of our input variables in the SVR case~\eqref{eq:svr_min_wolfe_dual}:

\begin{equation} \label{eq:svr_hessian}
	K_{ij} = k(x_i,x_j)
\end{equation}

where $k(x_i,x_j)$ is an example of a family of functions called \emph{kernel functions}.

For any positive definite kernel function $k$ (a so called Mercer kernel), it is guaranteed that there exists a mapping $\phi$ into a Hilbert space $\mathcal{H}$, such that:

\begin{equation} \label{eq:kernel_function}
	k(x_i,x_j) = \langle \phi(x_i), \phi(x_j) \rangle = \phi(x_i)^T \phi(x_j)
\end{equation}

where $\langle \cdot, \cdot \rangle$ denotes the inner product in the Hilbert space and $\phi(\cdot)$ is the identity function.

The reason that this \emph{kernel trick} is useful is that there are many classification/regression problems that are nonlinearly separable/regressable in the \emph{input space}, which might be in a higher dimensionality \emph{feature space} given a suitable mapping $x \rightarrow \phi(x)$.

\subsection{Polynomial kernel}

The \emph{polynomial} kernel is defined as:

\begin{equation} \label{eq:poly_kernel}
	k(x_i,x_j)=(\gamma \langle x_i, x_j\rangle + r)^d
\end{equation}

where $\gamma$ define how far the influence of a single training example reaches (low values meaning ‘far’ and high values meaning ‘close’).

\begin{figure}[h!]
	\centering
	\begin{subfigure}{.49\textwidth}
		\centering
		\includegraphics[width=\textwidth]{img/poly_dual_l1_svc_hyperplane}
		\caption{Polynomial SVC hyperplane}
		\label{fig:poly_dual_l1_svc_hyperplane}
	\end{subfigure}
	\begin{subfigure}{.49\textwidth}
		\centering
		\captionsetup{justification=centering}
		\includegraphics[width=\textwidth]{img/poly_dual_l1_svr_hyperplane}
		\caption{Polynomial SVR hyperplane}
		\label{fig:poly_dual_l1_svr_hyperplane}
	\end{subfigure}
\caption{Polynomial SVM hyperplanes}
\end{figure}

\pagebreak

\subsection{Gaussian kernel}

The \emph{gaussian} kernel is defined as:

\begin{equation} \label{eq:gaussian_kernel1}
	k(x_i,x_j)=\exp(-\frac{\|x_i-x_j\|_2^2}{2\sigma^2})
\end{equation}

or, equivalently:

\begin{equation} \label{eq:gaussian_kernel2}
	k(x_i,x_j)=\exp(-\gamma \|x_i-x_j\|_2^2)
\end{equation}

where $\displaystyle \gamma=\frac{1}{2\sigma^2}$ define how far the influence of a single training example reaches (low values meaning ‘far’ and high values meaning ‘close’).

\begin{figure}[h!]
	\centering
	\begin{subfigure}{.49\textwidth}
		\centering
		\includegraphics[width=\textwidth]{img/gaussian_dual_l1_svc_hyperplane}
		\caption{Gaussian SVC hyperplane}
		\label{fig:gaussian_dual_l1_svc_hyperplane}
	\end{subfigure}
	\begin{subfigure}{.49\textwidth}
		\centering
		\captionsetup{justification=centering}
		\includegraphics[width=\textwidth]{img/gaussian_dual_l1_svr_hyperplane}
		\caption{Gaussian SVR hyperplane}
		\label{fig:gaussian_dual_l1_svr_hyperplane}
	\end{subfigure}
\caption{Gaussian SVM hyperplanes}
\end{figure}

\subsection{Laplacian kernel}

The \emph{laplacian} kernel is defined as:

\begin{equation} \label{eq:laplacian_kernel1}
	k(x_i,x_j)=\exp(-\frac{\|x_i-x_j\|_1}{2\sigma^2})
\end{equation}

or, equivalently:

\begin{equation} \label{eq:laplacian_kernel2}
	k(x_i,x_j)=\exp(-\gamma \|x_i-x_j\|_1)
\end{equation}

where $\displaystyle \gamma=\frac{1}{2\sigma^2}$ define how far the influence of a single training example reaches (low values meaning ‘far’ and high values meaning ‘close’).

\begin{figure}[h!]
	\centering
	\begin{subfigure}{.49\textwidth}
		\centering
		\includegraphics[width=\textwidth]{img/laplacian_dual_l1_svc_hyperplane}
		\caption{Laplacian SVC hyperplane}
		\label{fig:laplacian_dual_l1_svc_hyperplane}
	\end{subfigure}
	\begin{subfigure}{.49\textwidth}
		\centering
		\captionsetup{justification=centering}
		\includegraphics[width=\textwidth]{img/laplacian_dual_l1_svr_hyperplane}
		\caption{Laplacian SVR hyperplane}
		\label{fig:laplacian_dual_l1_svr_hyperplane}
	\end{subfigure}
\caption{Laplacian SVM hyperplanes}
\end{figure}


\section{Relative Growth Rates}
In many practical applications, like the run times of computer algorithms for example, we often want to know if a function grows slower, the same, or faster than another.
\begin{definition}
	If
	\begin{equation*}
		\lim_{x \to \infty}{\frac{f(x)}{g(x)}} = \infty \Leftrightarrow \lim_{x \to \infty}{\frac{g(x)}{f(x)}} = 0,
	\end{equation*}
	then $f$  grows faster than $g$.
	If
	\begin{equation*}
		\lim_{x \to \infty}{\frac{f(x)}{g(x)}} = c \Leftrightarrow \lim_{x \to \infty}{\frac{g(x)}{f(x)}} = \frac{1}{c}
	\end{equation*}
	for some non-zero constant $c$, then $f$ and $g$ grow at the same rate.
\end{definition}

\begin{example}
	Compare $e^x$ and $x^{100}$.
	Does one grow faster than the other, or do they grow at the same rate?
\end{example}
\begin{answer}
	\begin{align*}
		\lim_{x\to\infty}{\frac{e^x}{x^{100}}} &= \lim_{x\to\infty}{\frac{e^x}{100x^{99}}} \\
		&= \vdots \text{ (after many applications of L'H\^{o}pital's Rule)} \\
		&= \lim_{x\to\infty}{\frac{e^x}{100!}}
		&= \infty.
	\end{align*}

	So, $e^x$ grows faster than $x^{100}$.
	In fact, any exponential $b^x$ grows faster than any polynomial, as long as $b > 1$.
\end{answer}

\subsection{Transitive Grow Rates}
For sufficiently large $x$, growth rates are transitive.
That is, if $f$ grows the same/faster/slower/ than $g$, and $g$ grows the same/slower/faster than $h$, then $f$ also grows the same/faster/slower than $h$.

\begin{example}
	Show that $f(x)=\sqrt{x^2+5}$ and $g(x)=\left(2\sqrt{x}-1\right)^2$ grow at the same rate.
\end{example}
\begin{answer}
	We'll show that both $f$ and $g$ grow at the same rate as $h(x)=x$.
	Starting with $f$ and $h$,
	\begin{align*}
		\lim_{x\to\infty}{\frac{f(x)}{h(x)}} &= \lim_{x\to\infty}{\frac{\sqrt{x^2+5}}{x}} \\
		&= \lim_{x\to\infty}{\sqrt{\frac{x^2+5}{x^2}}} \\
		&= \sqrt{\lim_{x\to\infty}{\frac{x^2+5}{x^2}}} \text{ (by the Power Rule)} \\
		&= \sqrt{1} \\
		&= 1.
	\end{align*}

	So, $f$ and $h$ grow at the same rate.
	Moving on to $g$ and $h$,
	\begin{align*}
		\lim_{x\to\infty}{\frac{g(x)}{h(x)}} &= \lim_{x\to\infty}{\frac{\left(2\sqrt{x}-1\right)^2}{x}} \\
		&= \lim_{x\to\infty}{\left(\frac{2\sqrt{x}-1}{\sqrt{x}}\right)^2} \\
		&= \left(\lim_{x\to\infty}{\frac{2\sqrt{x}-1}{\sqrt{x}}}\right)^2 \text{ (by the Power Rule)} \\
		&= \left(2\right)^2 \\
		&= 4.
	\end{align*}

	So, $g$ and $h$ grow at the same rate.
	Since $f$ and $g$ both grow at the same rate as $h$, $f$ and $g$ must grow at the same rate as each other.
\end{answer}

\subsection{Growth Rate Hierarchy ($n^n$FEPL)}
For most of the common types of functions we see, we can establish families of functions and rank these families by their growth rates from fastest-growing to slowest-growing.
If two functions are in different families, we can be sure that one grows faster than the other.
If two functions are in the same family, we'll have to do more work to compare them.
These families are summarized by the acronym $n^n$FEPL\footnote{You might recognize these families as a sort of Big-O family from computer science.}.
\begin{itemize}[align=left, leftmargin=0.66in]
	\item[$\textbf{n}^\textbf{n}$] These are functions that have a variable both in the base and exponent.
	\item[\textbf{F}actorials] These are functions that have an $n!$ term.
	\item[\textbf{E}xponentials] These are functions that have a constant base and a variable exponent.
		Note that if the variable base if less than 1, the function actually gets smaller for larger $n$.
	\item[\textbf{P}olynomials] These are functions with a variable base and constant exponent.
		Certain polynomials can still grow faster than others.
		For example, $x^2$ grows faster than $x$, which grows faster than $\sqrt{x}$.
	\item[\textbf{L}ogarithms] These are functions that have a log of a polynomial.
\end{itemize}

Although these rules are indeed true, don't just apply them blindly.
You should try to simplify a function first before figuring out to which family it belongs.
For example, although $\ln{x^x}$ contains an $x^x$ and a $\ln$, it's neither in the $n^n$ family nor in the logarithms family.
In fact, although this function is not a polynomial, it grows faster than $x$ but slower than $x^2$. \\

A function belongs to the family of its fastest-growing positive term.
Negative terms can either be ignored or used to simplify other terms.
For example, although $x^3 + e^x$ contains a polynomial $x^3$ term, for very large $x$, the $e^x$ term dominates the growth, meaning this function is part of the exponentials family.


\subsection{The box product}

In this section, we explore another application of the cross product.
Recall that we can use the cross product to find the the area of a
parallelogram. As we will now show, we can also use the cross product
together with the dot product to find the volume of a
parallelepiped. We begin with a definition.

\begin{definition}{Parallelepiped}{parallelepiped}
  The parallelepiped%
  \index{parallelepiped} determined by three vectors $\vect{u}$,
  $\vect{v}$, and $\vect{w}$ consists of the set of points of the form
  \begin{equation*}
    r\/\vect{u}+s\/\vect{v}+t\/\vect{w},
  \end{equation*}
  where $r,s,t$ are real numbers between $0$ and $1$, inclusive. The
  parallelepiped is a $3$-dimensional body bounded by parallelograms
  as shown in this picture.
  \begin{center}
    \begin{tikzpicture}[scale=1.2]
      \fill[blue!20] (0,0,0) -- (2,0,0) -- (3.5,1.5,0) -- (1.5,1.5,0) -- cycle;
      \fill[blue!35] (2,0,0) -- ++(1.5,1.5,0) -- ++(1,0,-1.5) -- ++(-1.5,-1.5,0) -- cycle;
      \fill[blue!12] (1.5,1.5,0) -- ++(2,0,0) -- ++(1,0,-1.5) -- ++(-2,0,0) -- cycle;
      \draw[blue,->](0,0,0) -- node [below, near end] {$\vect{u}$} (2,0,0); %% u
      \draw[red,dashed,->](0,0,0) -- node [above left, near end] {$\vect{v}$} (1,0,-1.5); %% v
      \draw[green!50!black,->](0,0,0) -- node [left, near end] {$\vect{w}$} (1.5,1.5,0); %% w
      \draw[dashed](1,0,-1.5)--(3,0,-1.5);
      \draw[dashed](1,0,-1.5)--(2.5,1.5,-1.5);
      \draw(1.5,1.5,0)--(3.5,1.5,0)--(4.5,1.5,-1.5)--(2.5,1.5,-1.5)--(1.5,1.5,0);
      \draw(3.5,1.5,0)--(2,0,0)--(3,0,-1.5)--(4.5,1.5,-1.5);
    \end{tikzpicture}
  \end{center}
\end{definition}

Notice that the base of the parallelepiped is the parallelogram
determined by the vectors $\vect{u}$ and $\vect{v}$. Therefore, its
area is equal to $\norm{\vect{u}\times\vect{v}}$. The height of the
parallelepiped is $\norm{\vect{w}}\cos\theta$, where $\theta$ is the
angle between $\vect{w}$ and $\vect{u}\times\vect{v}$, as shown in
this picture.
\begin{center}
  \begin{tikzpicture}[scale=1]
    \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (45:8mm) arc (45:90:8mm) -- cycle;
    \fill[blue!20] (0,0,0) -- (2,0,0) -- (3.5,1.5,0) -- (1.5,1.5,0) -- cycle;
    \fill[blue!35] (2,0,0) -- ++(1.5,1.5,0) -- ++(1,0,-1.5) -- ++(-1.5,-1.5,0) -- cycle;
    \fill[blue!12] (1.5,1.5,0) -- ++(2,0,0) -- ++(1,0,-1.5) -- ++(-2,0,0) -- cycle;
    \draw[blue,->](0,0,0) -- node [below, near end] {$\vect{u}$} (2,0,0); %% u
    \draw[red,dashed,->](0,0,0) -- node [above left, near end] {$\vect{v}$} (1,0,-1.5); %% v
    \draw[green!50!black,->](0,0,0) -- node [left, near end] {$\vect{w}$} (1.5,1.5,0); %% w
    \draw[->](0,0,0) -- node[left] {$\vect{u}\times\vect{v}$} (0,2.5,0);
    \draw[dashed](1,0,-1.5)--(3,0,-1.5);
    \draw[dashed](1,0,-1.5)--(2.5,1.5,-1.5);
    \draw(1.5,1.5,0)--(3.5,1.5,0)--(4.5,1.5,-1.5)--(2.5,1.5,-1.5)--(1.5,1.5,0);
    \draw(3.5,1.5,0)--(2,0,0)--(3,0,-1.5)--(4.5,1.5,-1.5);
    \node at (67.5:5mm){$\theta$};
  \end{tikzpicture}
\end{center}
The volume of this parallelepiped is the area of the base times the
height which is just
\begin{equation*}
  \norm{\vect{u}\times \vect{v}} \norm{\vect{w}} \cos \theta =
  (\vect{u}\times\vect{v}) \dotprod \vect{w}.
\end{equation*}
This expression is known as the \textbf{box product}%
\index{box product} and is sometimes written as
$\boxprod{\vect{u},\vect{v},\vect{w}}$.

Consider what happens if you interchange $\vect{v}$ with $\vect{w}$ or
$\vect{u}$ with $\vect{w}$.  Geometrically, we can see that this
merely introduces a minus sign. We find that the box product of three
vectors equals the volume of the parallelepiped determined by the
three vectors if the three vectors form a right-handed system, and the
negative of the volume if the vectors form a left-handed system.
We summarize this in the following proposition:

\begin{proposition}{The box product}{box-product}
  Let $\vect{u}, \vect{v}, \vect{w}$ be three vectors in $\R^3$ that
  define a parallelepiped. The box product
  $(\vect{u}\times\vect{v}) \dotprod \vect{w}$ is equal to:
  \begin{itemize}
  \item The volume%
    \index{box product!volume of parallelepiped}%
    \index{volume!of parallelepiped}%
    \index{parallelepiped!volume} of the parallelepiped, if
    $\vect{u}, \vect{v}, \vect{w}$ form a right-handed system.
  \item The negative of the volume of the parallelepiped, if
    $\vect{u}, \vect{v}, \vect{w}$ form a left-handed system.
  \end{itemize}
  In any case, the volume of the parallelepiped can be computed as the
  absolute value of the box product, given by
  $\abs{(\vect{u}\times\vect{v}) \dotprod \vect{w}}$.
\end{proposition}

\begin{example}{Volume of a parallelepiped}{parallelepiped-volume}
  Find the volume of the parallelepiped determined by the vectors
  \begin{equation*}
    \vect{u}
    =
    \begin{mymatrix}{r}
      1 \\
      2 \\
      -5
    \end{mymatrix}, \quad
    \vect{v}
    =
    \begin{mymatrix}{r}
      1 \\
      3 \\
      -6
    \end{mymatrix}, \quad
    \vect{w}
    =
    \begin{mymatrix}{r}
      3 \\
      2 \\
      3
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  According to the above discussion, we can take the cross product of
  any two of these vectors, and then the dot product with the third
  vector. The result will be either plus or minus the desired
  volume. Therefore we can obtain the volume by taking the absolute value.

  We first compute the cross product of $\vect{u}$ and $\vect{v}$:
  \begin{equation*}
    \vect{u} \times \vect{v}
    =
    \begin{mymatrix}{r}
      1 \\
      2 \\
      -5
    \end{mymatrix}
    \times
    \begin{mymatrix}{r}
      1 \\
      3 \\
      -6
    \end{mymatrix} \\
    =\begin{mymatrix}{c}
      (2)(-6) - (-5)(3) \\
      (-5)(1) - (1)(-6) \\
      (1)(3)  - (2)(1)  \\
    \end{mymatrix}
    =
    \begin{mymatrix}{r}
      3 \\
      1 \\
      1
    \end{mymatrix}
  \end{equation*}
  Then we take the dot product of this vector with $\vect{w}$:
  \begin{equation*}
    (\vect{u} \times \vect{v}) \dotprod \vect{w}
    =
    \begin{mymatrix}{r}
      3 \\
      1 \\
      1
    \end{mymatrix}
    \dotprod
    \begin{mymatrix}{r}
      3 \\
      2 \\
      3
    \end{mymatrix} \\
    = 9+2+3
    = 14.
  \end{equation*}
  Thus, the volume of the parallelepiped is 14 cubic units.
\end{solution}

The following is a consequence of Proposition~\ref{prop:box-product}:

\begin{corollary}{Right- and left-handed systems of vectors}{box-product-right-handed}
  The box product $(\vect{u}\times\vect{v}) \dotprod \vect{w}$ is:
  \begin{itemize}
  \item Positive, if $\vect{u},\vect{v},\vect{w}$ form a right-handed system.
  \item Negative, if $\vect{u},\vect{v},\vect{w}$ form a left-handed system.
  \item Zero, if $\vect{u},\vect{v},\vect{w}$ are coplanar.
  \end{itemize}
\end{corollary}

\begin{example}{Right- and left-handed systems of vectors}{box-product-right-handed}
  Which of the following systems of vectors
  $\vect{u},\vect{v},\vect{w}$ is right-handed? Which one is left-handed?
  Which one is coplanar?
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item $\vect{u}=\mat{1,2,0}^T$, $\vect{v}=\mat{0,0,1}^T$, $\vect{w}=\mat{1,-1,1}^T$.
    \item $\vect{u}=\mat{1,1,1}^T$, $\vect{v}=\mat{1,2,3}^T$, $\vect{w}=\mat{0,1,1}^T$.
    \item $\vect{u}=\mat{0,1,2}^T$, $\vect{v}=\mat{1,2,2}^T$, $\vect{w}=\mat{1,1,0}^T$.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{example}

\begin{solution}
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item We have
      $\boxprod{\vect{u},\vect{v},\vect{w}} = (\vect{u}\times\vect{v})
      \dotprod \vect{w} = \mat{2,-1,0}^T\dotprod\mat{1,-1,1}^T=3$, so
      the box product is positive and the system of vectors
      $\vect{u},\vect{v},\vect{w}$ is right-handed.
    \item We have
      $\boxprod{\vect{u},\vect{v},\vect{w}} = (\vect{u}\times\vect{v})
      \dotprod \vect{w} = \mat{1,-2,1}^T\dotprod\mat{0,1,1}^T=-1$, so
      the box product is negative and the system is left-handed.
    \item We have
      $\boxprod{\vect{u},\vect{v},\vect{w}} = (\vect{u}\times\vect{v})
      \dotprod \vect{w} = \mat{-2,2,-1}^T\dotprod\mat{1,1,0}^T=0$, so
      the box product is zero and the vectors are coplanar.
    \end{enumerate}
  \end{enumialphparenastyle}
\end{solution}

We finish this section with a law involving the dot product and the
cross product. It represents a fundamental observation that comes
directly from the geometric definition of the box product.

\begin{proposition}{Box product law}{order-of-product}
  Let $\vect{u}$, $\vect{v}$, and $\vect{w}$ be vectors. Then
  $(\vect{u}\times\vect{v}) \dotprod \vect{w}=\vect{u}\dotprod
  (\vect{v}\times \vect{w})$.
\end{proposition}

\begin{proof}
  This follows from observing that both
  $(\vect{u}\times \vect{v}) \dotprod \vect{w}$ and
  $\vect{u}\dotprod (\vect{v}\times \vect{w})$ compute the same
  box product, i.e., they either both give the volume of the
  parallelepiped or they both give the negative of the volume.

  Alternatively, we can calculate each product explicitly:
  \begin{eqnarray*}
    (\vect{u}\times \vect{v}) \dotprod \vect{w}
    &=&
        u_2v_3w_1 - u_3v_2w_1
        + u_3v_1w_2 - u_1v_3w_2
        + u_1v_2w_3 - u_2v_1w_3, \\
    \vect{u}\dotprod (\vect{v}\times \vect{w})
    &=&
        u_2v_3w_1 - u_3v_2w_1
        + u_3v_1w_2 - u_1v_3w_2
        + u_1v_2w_3 - u_2v_1w_3.
  \end{eqnarray*}
  In Chapter~\ref{cha:determinants}, you will learn that these
  expressions are a special case of a determinant.
\end{proof}


%\section*{Appendix II: Zeros and Poles}
%
The zeros and poles of a 2nd order low-shelving filter are
\begin{align}
s_{0} = \omega_\mathrm{c} \:g_\mathrm{B}^{\pm\frac{1}{4}} \: \e^{\im\alpha}\qquad
s_{\infty} = \omega_\mathrm{c} \: g_\mathrm{B}^{\mp\frac{1}{4}} \: \e^{\im\alpha},
\end{align}
and the respective complex conjugates $s_{0}^{\ast}$ and $s_{\infty}^{\ast}$.
The sign in the exponents is again determined
by the shelving level $G_\mathrm{B}$.
The polar angle $\alpha$ in the complex $s$-plane (Laplace domain)
relates to
\begin{align}
Q = \frac{-1}{2\cos\alpha}.
\end{align}
For stable and causal filters,
the poles must be in the left-half $s$-plane $\Re(s_{\infty}) < 0$,
therefore $\frac{\pi}{2} < \alpha < \frac{3\pi}{2}$.
The corresponding $Q$-factor is always positive
and varies within $[\frac{1}{2}, \infty)$.
This paper mainly considers Butterworth \cite{Ballou2008} $Q=\frac{1}{\sqrt{2}}$
which leads to $\alpha=\frac{3}{4} \pi$
as illustrated in Fig.~\ref{fig:pzmap}.
If $Q=\frac{1}{2}$, i.e. $\alpha=\pi$,
the poles and zeros lie on the real axis,
thus double real poles $s_{0} = s_{0}^{\ast}$ and
$s_{\infty} = s_{\infty}^{\ast}$ are obtained.
%
In this case the 2nd order frequency
response is identical to that of the 1st order shelving filter, although
obtained by different pole/zero-configurations.
%
Thus, it is sufficient to discuss the present filter design based on
2nd order filters
%
\NewL In the case of high-shelving filters, the poles and zeros are exchanged. %,
%\begin{align}
%s_{0} &= \omega_{c,\mu} \: g_\mathrm{B}^{\mp\frac{1}{4}} \: e^{i\alpha}\\
%s_{\infty} &= \omega_{c,\mu} \: g_\mathrm{B}^{\pm\frac{1}{4}} \: e^{i\alpha}.
%\end{align}
Notice that a high-shelving filter with a shelving gain $\pm g_\mathrm{B}$
has the same poles and zeros as the low-shelving filter with $\mp g_\mathrm{B}$.
The system functions only differ by a scaling factor of $\pm g_\mathrm{B}$
due to the leading coefficients $b_{2}$ and $a_{2}$.


\subsection{Planes}
\noindent
A plane can also be formed using a point in the plane, $P$, and a vector perpendicular to the plane, $\vec{n}$. All vectors $\langle x,y,z \rangle$ that originate from $P$ and remain in the plane must be perpendicular to $\vec{n}$, so their dot product with $\vec{n}$ would be 0. So, the point-normal form of a plane is\footnote{Note: Conventionally, $\vec{n}$ is a unit vector, $\hat{n}$.}
\begin{equation*}
	\vec{n}\cdot\left(\langle x,y,z \rangle - \vec{P}\right) = 0
\end{equation*}

\begin{figure}[h]
	\centering
	\includegraphics[scale=0.5]{Images/vectorValuedFunctions/PlaneNormalVector}
\end{figure}

\noindent
One can also construct a plane from 3 non-collinear points in the plane. One can still take advantage of point-normal form here by choosing 1 point to be $P_0$ and drawing vectors from this point to the two other points. The cross product of these two vectors is $\vec{n}$.
\begin{equation*}
	\left(\left(\vec{P_1} - \vec{P_0}\right) \times \left(\vec{P_2} - \vec{P_0}\right)\right) \cdot \left(\langle x,y,z \rangle - \vec{P_0}\right) = 0
\end{equation*}
where $P_0$, $P_1$, and $P_2$ are the three points in the plane.\\

\noindent
One can also construct a plane from a point in the plane, $P_0$, and a line in the plane, $\vec{r}(t) = \vec{P_1} + t\vec{v}$, that doesn't pass through $P_0$. One can get this setup into point-normal form by choosing a an output of $\vec{r}(t)$, like $\vec{P_1}$, and constructing a vector that points from $\vec{P_1}$ to $\vec{P_0}$, $\vec{P_1}-\vec{P_0}$, and crossing this with $\vec{v}$ to find $\vec{n}$.
\begin{equation*}
	\left(\vec{v} \times \left(\vec{P_1} - \vec{P_0}\right)\right) \cdot \left(\langle x,y,z \rangle - \vec{P_0}\right) = 0
\end{equation*}

\noindent
One can also construct a plane from two intersecting lines, $\vec{r_1}(t) = \vec{P_0} + t\vec{v_1}$ and $\vec{r_2}(t) = \vec{P_0} + t\vec{v_2}$, where $P_0$ is where the two lines intersect. One can cross $\vec{v_1}$ with $\vec{v_2}$ to get the normal vector.
\begin{equation*}
	\left(\vec{v_1} \times \vec{v_2}\right) \cdot \left(\langle x,y,z \rangle - \vec{P-0}\right) = 0
\end{equation*}


\chapter{0411notes}
Iterative methods for solving:
\begin{enumerate}
\item \begin{equation}
\min_{\theta} f(\theta)
 \end{equation}
\item \begin{equation}Au=g \end{equation}
\end{enumerate}

We have dynamic system approach:
(1) Optimization:
\begin{equation}\theta^*\in \arg\min_{\theta} f(\theta)\to \nabla f(\theta^*)=0 \end{equation}
\begin{equation}\frac{d\theta}{dt}=-\nabla f(\theta(t)) \end{equation}
By Euler's method,
\begin{equation}\frac{\theta^{i+1}-\theta^i}{\eta}=-\nabla f(\theta^i) \end{equation}
\begin{equation}\theta^{i+1}=\theta^i-\eta\nabla f(\theta^i) \end{equation}
(2) \begin{align}
Au&=g  \\\notag
u&=u(t) \\\notag
u_t&=g-Au\\\notag
u^*&=\lim_{t\to \infty}u(t)\\\notag
g-Au^*&=0
\end{align}

Moreover,
\begin{equation}Au=g \Leftrightarrow B(Au-g)=0, \end{equation} where $B$ is nonsingular.
A simple choice for $B$ is $B=D^{-1}$, where $D=\diag(A)$.

For Euler's method,
\begin{equation}\frac{u^{i+1}-u^i}{\eta}=B(g-Au^i)\equiv r_i \end{equation}
\begin{equation} u^{i+1}=u^i+\eta r_i. \end{equation}

Why do we consider the residual?
For $Au=g$, we want to know whether $u^i \to u^{i+1}$.
Consider $Ae=r=g-Au^i$, we have the update $u^{i+1}=u^i+e$.
After solving $Ae=r$ 'approximately', say
\begin{equation} \hat{e}=Br, B\approx A^{-1}. \end{equation}
We have
\begin{equation} u^{i+1}=u^i+\hat{e}=u^i+B(g-Au^i) \end{equation}

Apply the numerical ODE methods.
(1) Apply the machine learning to Euler's method.
For $\min_{\theta} f(\theta)$, the explicit Euler's method is
\begin{equation} \theta^{i+1}=\theta^i-\eta_i\nabla f(\theta^i). \end{equation}
The implicit Euler's method is
\begin{equation} \theta^{i+1}=\theta^i-\eta_i\nabla f(\theta^{i+1}). \end{equation}
(2) Multistep method.
\begin{equation} u^{i+1}=u^i+\eta_i r_i+\eta_{i-1} r_{i-1}+...+\eta_0 r_0). \end{equation}
Then we can apply the explicit Euler's method to ResNet and the multi-step ODE solvers to the Dense-Net.
There are some examples,
\begin{enumerate}
\item LeNet
\item AlexNet
\item ResNet
\item DenseNet
\item MgNet
\end{enumerate}


%
% Chapter 2.8
%

\section*{2.8 Related Rates}

\subsubsection*{Example}

Air is being pumped into a spherical balloon so that its volume increases at a rate of 100 cm\(^3\)/s. How fast is the radius of the balloon increasing when the diameter is 50 cm?

\subsubsection*{Solution}

Identify given information and the unknown value.
$$ \frac{dV}{dt} = 100 \text{ cm}^3/\text{s} $$
$$ \frac{dr}{dt} =\text{? when } r = 25 \text{ cm} $$
Relate \(V\) and \(r\) by a formula.
$$ V=\frac{4}{3} \pi r^3 $$
Differentiate both sides of the equation with respect to \(t\).
$$ \frac{dV}{dt}=\frac{dV}{dr}\frac{dr}{dt}=4 \pi r^2 \frac{dr}{dt} $$
Solve for the unknown value.
$$ \frac{dr}{dt} = \frac{1}{4 \pi r^2} \frac{dV}{dt} $$
If we put \(r=25\) and \(\frac{dV}{dt}=100\) in the equation, we get
$$ \frac{dr}{dt} = \frac{1}{4 \pi {(25)}^2}100=\frac{1}{25 \pi} \text{ cm/s} $$


\subsection{Characteristic of a ring}

The characteristic of a ring is the number of times the multiplicative identity must be added to get the additive identity.

If this never happens, the characteristic is \(0\).

\subsubsection{Example}

The integer mod \(2\) ring, the characteristic is \(2\).


\chapter{Week 1: The vector}

\section{What is a vector?}
A vector is an array of $d$ numbers, and also can be thought as functions that maps from $\{0, 1, \ldots, d-1\}$ to $\fF$ with $\fF^{d}$ as the notation.

A vector most of whose values are zero is called a \textsl{sparse} vector. If no more than $k$ of the entries are nonzero, we say the vector is $k$-sparse. A $k$-sparce vector can be represented using space proportional to $k$.

\section{Vector addition and scalar-vector multiplication}
\begin{definition}[Vector addition]
  \begin{equation*}
    [u_1, u_2, \ldots, u_n] + [v_1, v_2, \ldots, v_n] = [u_1+v_1, u_2+v_2, \ldots, u_n+v_n]
  \end{equation*}
\end{definition}

\begin{definition}[Zero vector]
  The $D$-vector whose entries are all zero is the zero vector, written $\vzero_D$ or just $\vzero$.
  \begin{equation*}
    \vv + \vzero = \vv
  \end{equation*}
\end{definition}

\begin{definition}[Associativity]
  \begin{equation*}
    (\vx + \vy) + \vz = \vx + (\vy + \vz)
  \end{equation*}
\end{definition}

\begin{definition}[Commutativity]
  \begin{equation*}
    \vx + \vy = \vy + \vx
  \end{equation*}
\end{definition}

For vectors, we refer to field elements as scalars, we use them to scale vectors: $\alpha\vv$. Greek letters (e.g. $\alpha,\beta,\gamma$) denote scalars.

\begin{definition}
  Multiplying a vector $\vv$ by a scalar $\alpha$ is defined as multiplying each entry of $\vv$ by $\alpha$.
  \begin{equation*}
    \alpha [\vv_1, \ldots, \vv_n] = [\alpha\vv_1, \ldots, \alpha\vv_n]
  \end{equation*}
\end{definition}

The set of points $\{ \alpha\vv : \alpha\in\R \}$ forms the line through the origin and $\vv$.

An expression of the form $\alpha\vu + \beta\vv$ where $0 \le \alpha \le 1, 0 \le \beta \le 1$, and $\alpha + \beta = 1$ is called a \emph{convex combination} of $\vu$ and $\vv$. An expression of the form $\alpha\vu + \beta\vv$ where $\alpha+\beta=1$ is called and \emph{affine combination} of $\vu$ and $\vv$.

% ================================================================

\section{Dot-product}
\begin{definition}[Dot-product of two $D$-vectors]
  Dot-product of two $D$-vectors is the sum of product of corresponding entries:
  \begin{equation*}
    \vu \cdot \vv = \sum_{k\in\{1, \ldots, D\}} \vu_k \vv_k
  \end{equation*}
\end{definition}

\subsection{Linear equations}
\begin{definition}
  A linear equation is an equation of the form
  \begin{equation*}
    \va \cdot \vx = \beta
  \end{equation*}
  where $\va$ is a vector, $\beta$ is a scalar, and $\vx$ is a vector of variables.
\end{definition}

Algebraic properties of dot-product:
\begin{itemize}
\item Commutativity $\vv \cdot \vx = \vx \cdot \vv$
\item Homogeneity $(\alpha\vu)\cdot\vv = \alpha(\vu\cdot\vv)$
\item Distributive law $(\vv_1 + \vv_2)\cdot\vx = \vv_1\cdot\vx + \vv_2\cdot\vx$
\end{itemize}


\section{Sequences}
\begin{definition}
	A sequence $\left\{a_n\right\} = \left\{a_1, a_2, \ldots, a_n\right\}$ is an ordered list of numbers.
	Each element of a sequences is called a term and is identified by its index in the sequence.
	Sequences can be finite or infinite.
\end{definition}

\begin{example}
	The nth term of a sequence is defined by the following formula:
	\begin{equation*}
		a_n = \frac{(-1)^n}{n^2+1}.
	\end{equation*}
	Find the 1st, 2nd, and 100th terms of the sequence.
\end{example}
\begin{answer}
	\begin{align*}
		a_1 &= \frac{(-1)^1}{1^2 + 1} = \frac{-1}{2} \\
		a_2 &= \frac{(-1)^2}{2^2 + 1} = \frac{1}{5} \\
		a_{100} &= \frac{(-1)^{100}}{100^2 + 1} = \frac{1}{10001}.
	\end{align*}
\end{answer}

The above sequence was defined explicitly, meaning that we have a formula for the nth term of the sequence only in terms of n.
However, sequences can also be defined recursively, meaning the formula for subsequent terms of the sequence contains previous terms.
For a recursive sequence to be properly defined, there need to be one or more base terms that aren't defined recursively.
For example, the Fibonacci sequence, one of the most famous recursive sequences, as $a_1$ and $a_2$ as base terms.
\begin{equation*}
	a_n = \begin{cases}
		1 & n = 1, 2 \\
		a_{n-1} + a_{n-2} & n \geq 3
	\end{cases}.
\end{equation*}

\subsection{Common Types of Sequences}
There are some common types of sequences that you should be familiar with.
You might recognize these types of sequences and some of the formulas surrounding them from previous math classes.

\subsubsection{Arithmetic Sequences}
\begin{definition}
	An arithmetic sequence is one where $a_{n+1} - a_{n} = d$, a common difference, for all terms.
\end{definition}

That is, to get the next term, we simply add some number $d$ (which could be negative) to the previous term.
Arithmetic sequences can be defined either explicitly or recursively.
Let $a_0$ be the starting term of the sequence.
\begin{align*}
	a_n &= dn + a_0 \\
	&= \begin{cases}
		a_0 & n = 0 \\
		d + a_{n-1} & n \geq 1
	\end{cases}.
\end{align*}
As we can see from the explicit formula, if we graphed values of an arithmetic sequence on in the plane with $x$ coordinate $n$ and $y$ coordinate $a_n$, all points would lie on a line with slope $d$ and $y$ intercept $a_0$.

\begin{example}
	Write an explicit formula for the following arithmetic sequence.
	\begin{equation*}
		\left\{\ln{2}, \ln{6}, \ln{18}, \ldots\right\}.
	\end{equation*}
\end{example}
\begin{answer}
	Since we are given that this sequence is arithmetic, we'll find the common difference.
	\begin{equation*}
		d = \ln{6} - \ln{2} = \ln{\frac{6}{2}} = \ln{3}.
	\end{equation*}

	So, applying the explicit formula for an arithmetic sequence with starting term $\ln{2}$ and common difference $\ln{3}$,
	\begin{equation*}
		a_n = \ln{(3)}n + \ln{2}, n\geq 0.
	\end{equation*}

	We might have also noticed that each term inside the $\ln$ is triple the previous one, meaning we can write an explicit formula and then simplify to the same answer as before.
	\begin{align*}
		a_n &= \ln{(3^{n}\cdot 2)}, n\geq 0 \\
		&= \ln{3^n} + \ln{2}, n \geq 0 \\
		&= \ln{(3)}n + \ln{2}, n \geq 0.
	\end{align*}
\end{answer}

\subsubsection{Geometric Sequences}
\begin{definition}
	A geometric sequence is one where $\frac{a_{n+1}}{a_n} = r$, a common ratio, for all terms.
\end{definition}

That is, to get the next term, we simply multiply some number $r$ (which could be negative) by the previous term.
Geometric sequences can also be defined explicitly or recursively.
Let $a_0$ be the starting term of the sequence.
\begin{align*}
	a_n &= a_0(r)^n \\
	&= \begin{cases}
		a_0 & n = 0 \\
		ra_{n-1} & n \geq 1
	\end{cases}.
\end{align*}

As we can see with the explicit formula, if we graphed terms of a geometric sequence for positive $r$, the points would lie on an exponential curve with $y$ intercept $a_0$ and exponential base $r$.

\begin{example}
	Write an explicit formula for the following geometric sequence.
	\begin{equation*}
		\left\{2,-6,18,-54,\ldots\right\}.
	\end{equation*}
\end{example}
\begin{answer}
	Since we are given that this sequence is geometric, we'll find the common ratio.
	\begin{equation*}
		r = \frac{-6}{2} = 3.
	\end{equation*}

	So, applying the explicit formula for a geometric sequence with starting term 2 and common ratio -3,
	\begin{equation*}
		a_n = 2(-3)^n.
	\end{equation*}
\end{answer}

\subsection{Limits of a Sequence}
Once we have a formula for a sequence, we might be interested to know if $a_n$ tends towards some value as $n$ gets large.
\begin{definition}
	Let $L$ be a real number, the sequence $\left\{a_n\right\}$ as limit $L$ as $n$ approaches infinity if given any real $\epsilon > 0$, there is some index $m$ such that for all $n > m$
	\begin{equation*}
		\abs{a_n - L} < \epsilon.
	\end{equation*}
	We notate this as
	\begin{equation*}
		\lim_{n\to\infty}{a_n} = L
	\end{equation*}
	and say the sequence converges to $L$.
	If the sequence does not have a limit, then we say the sequences diverges.
\end{definition}

The following rules we gave for limits of a function: Sum and Difference Rule, Product Rule, Constant Multiple Rule, and Quotient Rule, all still apply to limits of sequences.
The only rule that doesn't still hold is the Power Rule because of the following sort of problem:
\begin{align*}
	a_n &= (-1)^n \\
	\lim_{n\to\infty}{a_n^2} &= 1 \\
	\left(\lim_{n\to\infty}{a_n}\right)^2 &= \text{DNE} \\
	1 &\neq \text{DNE}.
\end{align*}

\subsubsection{The Sandwich Theorem for Sequences}
\begin{theorem}[Sandwich Theorem for Sequences]
	If $\lim_{n\to\infty}{a_n} = \lim_{n\to\infty}{c_n} = L$ and there is an integer $m$ such that $a_n \leq b_n \leq c_n$ for all $n > m$, then $\lim_{n\to\infty}{b_n} = L$.
\end{theorem}

This is essentially the same as the Sandwich Theorem for limits of a function.
The only added caveat is that we have to find some index $m$ for which the sandwiching inequality always holds for terms after the $m$th.

\begin{example}
	Determine if the following sequence converges or diverges.
	If it converges, find its limit.
	\begin{equation*}
		a_n = (-1)^n\frac{n-1}{2}, n\geq 1.
	\end{equation*}
\end{example}
\begin{answer}
	We see that as $n$ grows large $\abs{a_n}$ approaches 1.
	However, $a_n$ bounces between 1 and -1 depending on whether $n$ is even or odd.
	Thus, we could let $\epsilon = 1/2$, which would show that the limit diverges.
\end{answer}

\begin{example}
	Determine if the following sequences converges or diverges.
	If it converges, find its limit.
	\begin{equation*}
		a_n = \frac{\cos{n}}{n}, n\geq 1.
	\end{equation*}
\end{example}
\begin{answer}
	It might at first seem that this limit diverges because $\cos$ bounces between -1 and 1.
	However, we can use the Sandwich Theorem to show that the limit converges.
	\begin{align*}
		\frac{-1}{n} &\leq \frac{\cos{n}}{n} \leq \frac{1}{n}, n\geq 1 \\
		\lim_{n\to\infty}{\frac{-1}{n}} &\leq \lim_{n\to\infty}{\frac{\cos{n}}{n}} \leq \lim_{n\to\infty}{\frac{1}{n}} \\
		0 &\leq \lim_{n\to\infty}{\frac{\cos{n}}{n}} \leq 0 \\
		\lim_{n\to\infty}{\frac{\cos{n}}{n}} &= 0.
	\end{align*}
\end{answer}

\subsubsection{Absolute Value Theorem}
We can apply the Sandwich Theorem to show that sequences whose absolute value converges to 0 must also converge to 0.
\begin{theorem}[Absolute Value Theorem]
	If $\lim_{n\to\infty}{\abs{a_n}} = 0$, then $\lim_{n\to\infty}{a_n} = 0$.
\end{theorem}
\begin{proof}
	For all $n$,
	\begin{equation*}
		-\abs{a_n} \leq a_n \leq \abs{a_n}.
	\end{equation*}
	Applying the Sandwich Theorem and limit properties,
	\begin{align*}
		-\lim_{n\to\infty}{\abs{a_n}} &\leq \lim_{n\to\infty}{a_n} \leq \lim_{n\to\infty}{\abs{a_n}} \\
		-0 &\leq \lim_{n\to\infty}{a_n} \leq 0 \\
		\lim_{n\to\infty}{a_n} &= 0.
	\end{align*}
\end{proof}


\section{Quantifying uncertainty}

\subsection{Conditional probability}

\begin{equation}
    P(a|b) = \frac{a \wedge B)}{P(b)}  \implies P(A \wedge B) = P(a|b) P(b)
\end{equation}

\begin{equation}
    P(a) = \sum_{b}{P(a, b)} = \sum_{b}{P(a|b)P(b)}
\end{equation}

Chain rule
\begin{equation}
    P(x_1, x_2, \cdots, x_n) = \prod_{i}{P(x_i|x_{i-1}, x_{i-2}, \cdots x_1)}
\end{equation}

\subsection{Independence}

Two r.v. are independent if and only if $P(a, b) == P(a)P(b)$

\subsection{Bayes' Rule}

\begin{equation}
\begin{aligned}
    P(a|b) &= \frac{P(a, b)}{P(b)} \\
    &= \frac{P(b|a)P(a)}{P(b)} \\
    &= \frac{P(b|a)P(a)}{\sum_{i}{P(B|a_i)}} \\
    &\propto P(b|a)P(a) \thinspace  (w.r.t. a)
\end{aligned}
\end{equation}


%
% Chapter 5.5
%

\section*{5.5 Average Value of a Function}

The average value of \(x\) on \([a, b]\) is
\[ f_{avg} = \frac{1}{b-a} \int_a^b f(x)dx \]

\subsection*{Mean Value Theorem for Integrals}

If \(f\) is continuous on \([a, b]\), then there exists a number \(c\) in \([a, b]\) such that
\[ f(c) = f_{avg} = \frac{1}{b-a} \int_a^b f(x)dx \]


\subsection{Test 3}
\begin{enumerate}
	\item Evaluate each of the following integrals as they appear or by changing coordinate systems. Sketch and/or describe the region geometrically to help in choosing an appropriate coordinate system.
	\begin{enumerate}[label=\alph*.]
		\item \begin{equation*}
			\int_{0}^{3}{\int_{0}^{2}{\int_{0}^{1}{ze^{x+y+z^2}\mathrm{d}x}\mathrm{d}y}\mathrm{d}x}
		\end{equation*}
		We can use a simple u-substitution.
		\begin{align*}
			u &= z^2 + x + y, \mathrm{d}u = 2z\mathrm{d}z \\
			I &= \int_{0}^{3}{\int_{0}^{2}{\int_{x+y}^{x+y+1}{\frac{1}{2}e^{u}\mathrm{d}u}\mathrm{d}y}\mathrm{d}x} \\
			&= \frac{1}{2}\int_{0}^{3}{\int_{0}^{2}{e^{x+y+1} - e^{x+y}\mathrm{d}y}\mathrm{d}x} \\
			&= \frac{1}{2}\int_{0}^{3}{\left((e^{3+x}-e^{2+x}) - (e^{1+x}-e^{x})\right)\mathrm{d}x} \\
			&= \frac{1}{2}(e^6 - e^5 - e^4 + e^2 + e - 1)
		\end{align*}

		\item \begin{equation*}
			\int_{0}^{\sqrt{2}}{\int_{-\sqrt{2-x^2}}^{\sqrt{2-x^2}}{\int_{0}^{5}{z\mathrm{d}z}\mathrm{d}y}\mathrm{d}x}
		\end{equation*}
		The region we are integrating is a half-cylinder, so we will use cylindrical coordinates.
		\begin{align*}
			I &= \int_{-\pi/2}^{\pi/2}{\int_{0}^{\sqrt{2}}{\int_{0}^{5}{rz\mathrm{d}z}\mathrm{d}r}\mathrm{d}\theta} \\
			&= \pi\int_{0}^{\sqrt{2}}{r\mathrm{d}r}\cdot\int_{0}^{5}{z\mathrm{d}z} \\
			&= \pi\left(\frac{\sqrt{2}^{2}}{2} - \frac{0^2}{2}\right) \cdot \left(\frac{5^2}{2} - \frac{0^2}{2}\right) \\
			&= \frac{25\pi}{2}
		\end{align*}

		\item \begin{equation*}
			\int_{-3}^{3}{\int_{0}^{\sqrt{9 - x^2}}{\int_{0}^{\sqrt{9 - x^2 - y^2}}{z\mathrm{d}z}\mathrm{d}y}\mathrm{d}x}
		\end{equation*}
		The region we are integrating is a quarter sphere, so we will use spherical coordinates.
		\begin{align*}
			I &= \int_{0}^{3}{\int_{0}^{\pi}{\int_{0}^{\pi/2}{\rho^2\sin{\phi}\cos{\phi}\mathrm{d}\phi}\mathrm{d}\theta}\mathrm{d}\rho} \\
			&= \int_{0}^{3}{\rho^3\mathrm{d}\rho}\cdot\int_{0}^{\pi}{\mathrm{d}\theta} \cdot \int_{0}^{\pi/2}{\sin{\phi}\cos{\phi}\mathrm{d}\phi} \\
			&= \frac{3^4}{4} \cdot \pi \cdot \frac{1}{2} = \frac{81\pi}{8}
		\end{align*}
	\end{enumerate}

	\item Let $\Omega \subset \mathbb{R}^3$ be a spherical ball of radius $R$ centered at the origin. Set up and evaluate $\iiint\limits_{\Omega}{\mathrm{d}V}$.\\
	Since we are finding the volume of a ball, we'll use spherical coordinates.
	\begin{align*}
		\mathrm{d}V &= \rho^2\sin{\phi}\mathrm{d}\rho\mathrm{d}\theta\mathrm{d}\phi \\
		I &= \int_{0}^{R}{\int_{0}^{2\pi}{\int_{0}^{\pi}{\rho^2\sin{\phi}\mathrm{d}\phi}\mathrm{d}\theta}\mathrm{d}\rho} \\
		&= \int_{0}^{R}{\rho^2\mathrm{d}\rho} \cdot \int_{0}^{2\pi}{\mathrm{d}\theta} \cdot \int_{0}^{\pi}{\sin{\phi}\mathrm{d}\phi} \\
		&= \frac{R^3}{3} \cdot 2\pi \cdot 2 = \frac{4\pi}{3}R^3
	\end{align*}

	\item A plane lamina with density $\sigma(x,y) = \sqrt{x^2+y^2}$ occupies the region $D$, the region bounded by the Archimedian spiral $r = \theta$ and the half line $\theta = \alpha, r \geq 0$, where $\alpha$ is an unknown angle in radians. Find $\alpha$ such that the average density $\bar{\sigma}$ of the lamina is $\pi / 2$.
	\begin{equation*}
			\bar{\sigma} = \frac{\iint\limits_{D}{\sigma\mathrm{d}A}}{\iint\limits_{D}{\mathrm{d}A}} = \frac{\pi}{2}
	\end{equation*}
	\begin{equation*}
		D = \left\{(r,\theta) \mid 0 \leq \theta \leq \alpha, 0 \leq r \leq \theta \right\}, \sigma(r,\theta) = r
	\end{equation*}
	\begin{align*}
		\bar{\sigma} &= \frac{\int_{0}^{\alpha}{\int_{0}^{\theta}{r^2\mathrm{d}r}\mathrm{d}\theta}}{\int_{0}^{\alpha}{\int_{0}^{\theta}{r\mathrm{d}r}\mathrm{d}\theta}} \\
		&= \frac{\int_{0}^{\alpha}{\frac{\theta^3}{3}\mathrm{d}\theta}}{\int_{0}^{\alpha}{\frac{\theta^2}{2}\mathrm{d}\theta}} \\
		&= \frac{\alpha^4/12}{\alpha^3/6} = \frac{\alpha}{2} \\
		\frac{\alpha}{2} &= \frac{\pi}{2} \\
		&\implies \alpha = \pi
	\end{align*}

	\item Consider the 2D Gaussian function $f(x,y) = e^{-(x^2+y^2)}$. Evaluate $\iint\limits_{D}{f(x,y)\mathrm{d}A}$, where $D$ is a disk of radius $a$ centered at the origin. Use the results to evaluate $\iint\limits_{\mathbb{R}^2}{f(x,y)\mathrm{d}A}$.
	\begin{equation*}
		D = \left\{(r,\theta) \mid 0 \leq r \leq a, 0 \leq \theta \leq 2\pi \right\}
	\end{equation*}
	\begin{equation*}
		f(x,y) = \exp{(-x^2-y^2)} = \exp{(-r^2)}
	\end{equation*}
	\begin{align*}
		\iint\limits_{D}{f(x,y)\mathrm{d}A} &= \int_{0}^{2\pi}{\int_{0}^{a}{e^{-r^2}r\mathrm{d}r}\mathrm{d}\theta} \\
		&= \int_{0}^{2\pi}{\mathrm{d}\theta} \cdot \frac{-1}{2}\int_{0}^{-a^2}{e^{u}\mathrm{d}u} \\
		&= 2\pi \cdot \frac{-1}{2} \cdot \left(e^{-a^2} - 1\right) \\
		&= \pi\left(1 - e^{-a^2}\right)
	\end{align*}
	\begin{equation*}
		\iint\limits_{\mathbb{R}^2}{f(x,y)\mathrm{d}A} = \lim_{a \to \infty}{\pi\left(1 - e^{-a^2}\right)} = \pi
	\end{equation*}
\end{enumerate}


\subsection{Complex conjugate}

We have \(z=a+bi\).

The complex conjugate is:

\(\bar z=a-bi\)


\subsection{Maximising the likelihood function}

We have a likelihood function of the data.

\(L(\theta ; X)=P(X|\theta )\)

We choose values for \(\theta \) which maximise the likelihood function.

\(argmax_\theta P(X|\theta )\)

That is, for which values  of \(\theta \) was the observation we saw most likely?

This is a mode estimate.

\subsection{IID}

\(L(\theta ; X)=\prod_i P(x_i|\theta )\)

\subsection{Logarithms}

We can take logarithms, which preserve stationary points. As logarithms are defined on all values above \(0\), and all probabilities are also above zero (or zero), this preserves solutions.

The non-zero stationary points of:

\(\ln L(\theta ; X)=\ln \prod_i P(x_i|\theta )\)

\(\ln L(\theta ; X)=\sum_i \ln P(x_i|\theta )\)

\subsection{Example: Coin flip}

Let’s take our simple example about coins. Heads and tails are the only options, so \(P(H)+P(T)=1\).

\(P(H|\theta )=\theta \)

\(P(T|\theta )=1-\theta \)

\(\ln L(\theta ; X)=\sum_i \ln P(x_i|\theta )\)

If we had \(5\) heads and \(5\) tails we would have:

\(\ln L(\theta ; X)=5\ln (\theta )+ 5\ln (1-\theta )\)

So \(P(H)=\dfrac{1}{2}\) is the value which makes our observation most likely.


%%%%%%%%%%%%%
% Lp Spaces %
%%%%%%%%%%%%%

\section{\texorpdfstring{$L_p$}\ \ Spaces}

\begin{theorem}{}{H\"older's Inequality}

    Let $f,g: \Omega \to \overline{\mathbb{R}}$ be measurable functions, then

        \begin{align*}
            \int_{\Omega} |fg| \, d\mu \leq \Norm{f}_p \Norm{g}_q \quad \textrm{for } p \geq 1,
        \end{align*}

    where

        \begin{align*}
            q \coloneqq
            \begin{cases}
                \frac{p}{p - 1} & p > 1, \\
                \infty & p = 1
            \end{cases}.
        \end{align*}

\end{theorem}

\begin{theorem}{}{H\"older's Inequality for Expectations}

    Let $X,Y$ be random variables, then

        \begin{align*}
            E|XY| \leq (E|X|^p)^{\frac{1}{p}} (E|Y|^q)^{\frac{1}{q}}
        \end{align*}

    where

        \begin{align*}
            q \coloneqq
            \begin{cases}
                \frac{p}{p - 1} & p > 1, \\
                \infty & p = 1
            \end{cases}.
        \end{align*}

\end{theorem}

\begin{proposition}{}{Finite Second Momenta Implication}

    Let $X, Y$ be random variables with finite second momenta. Then $E|XY| < \infty$.

    \Hint Use H\"older's Inequality with $p = 2$ on $E|XY| = \int_{\Omega} |XY| \, dP$.

\end{proposition}

\begin{lemma}{4.4}{Borel-Cantelli Lemma}

    Let $(A)_{n=1}^{\infty}$ be a sequence of sets $A_n \in \CalF$ such that $\sum_{n=1}^{\infty} \mu(A_n) < \infty$, i.e. the series of measures of $A_n$ converges. Then for:

        \begin{align*}
            A \coloneqq \limsup_{n \to \infty} A_n \coloneqq \bigcap_{n=1}^{\infty} \bigcup_{k=n}^{\infty} A_k,
        \end{align*}

    we have $\mu(A) = 0$.

    \Hint Define $B_n \coloneqq \bigcup_{k=n}^{\infty} A_k$, then $(B_n)_{n=1}^\infty$ is decreasing and so $\bigcap_{n=1}^{\infty} B_n = \lim_{n \to \infty} B_n$ and realize that $\sum_{n=1}^{\infty} \mu(A_n) < \infty$ $\Rightarrow$ tail sums $\sum_{k=n}^{\infty} \mu(A_k) \to 0$ as $n \to \infty$.

\end{lemma}


In general, there are three kinds of claim ``structures'' which encompass the vast majority of claims that people want to prove. These are the ``object has property'' structure, the ``hypothesis implies conclusion'' structure, and the ``situations are equivalent'' structure. We expand more on these below.

An ``object has property'' structure concerns some specific object or collection of objects, with certain properties known beforehand. The nature of the claim is that the object has some other property, which is usually not totally obvious from the properties that are known beforehand. Here is an example of such a claim:
\begin{proposition}
The set of natural numbers $\mathbb{N}$ are countable.
\end{proposition}
Here the object in question is specific, being the set of natural numbers $\mathbb{N}$ and the property is the property of countability of a set. However, based on the definition of countability, this claim is trivially true based on the definition of countability. Is there anything we might glean from it? Here is another similar claim, but instead with a collection of objects.
\begin{proposition}
Any subset of the natural numbers $\mathbb{N}$ is countable.
\end{proposition}
Here, the property remains the same, but we have expanded to a collection of objects, namely the subset of natural numbers. However, this claim is still trivially true based on the definition of countability. Maybe there is a more general claim which is not immediately obvious? There is, once we replace $\mathbb{N}$ with any countable set.
\begin{proposition}
Any subset of a countable set is countable.
\end{proposition}
We use this set of examples to indicate that a claim involving a very specific object (the natural numbers) can be made into a claim that is much more general (and not immediately obvious). In this way, a lot of mathematics is built from the ``ground up'' from small examples which get more general.

\subsection{Hypothesis implies Conclusion}

The other type of claim usually investigated has a ``hypothesis implies conclusion'' structure. With these claims, the hypothesis usually involves one or several objects with various properties, and the claim is that some given conclusion will follow. That is, these types of claims usually go the following way: ``Suppose A, B, \ldots satisfy the following properties or relations. Then a certain conclusion follows.'' Here is a more specific example:

\begin{proposition}
Suppose $A$, $B$, and $C$ are sets with $A \subset B$ and $B \subset C$. Then $A \subset C$.
\end{proposition}

Here, the objects are $A$, $B$, and $C$ with some properties involving subset relations. The conclusion is another property involving subset relations.

\subsection{Situations are Equivalent}

Similarly to the ``hypothesis implies conclusion'' proof structure, the ``situations are equivalent'' structure involves a set of two or more hypotheses. The claim is that each of these hypotheses implies the other. Usually these claims will have the following form:

\begin{proposition}
The following are equivalent:
\begin{itemize}
	\item Situation A
	\item Situation B
\end{itemize}
\end{proposition}

To prove this proposition is true is the same as showing both the claims ``Situation A implies Situation B'' and ``Situation B implies Situation A''.

For the situation with $3$ or more hypotheses, one might imagine that one needs to show a lot more than two claims, and the situation may get out of hand quickly. However, this is not the case, since it turns out implication is transitive. That is, if ``Situation A implies Situation B'' and ``Situation B implies Situation C'', then it follows that ``Situation A implies Situation C''. So if we are to show situations A, B, and C are equivalent, one possible way to show this is to show the following claims to be true:

\begin{itemize}
	\item Situation A implies Situation B
	\item Situation B implies Situation C
	\item Situation C implies Situation A.
\end{itemize}


\section{Data normalization in DNNs and CNNs}

\subsection{Normalization for input data of DNNs}
Consider that we have the all training data as
\begin{equation}\label{eq:trainingdata}
(X,Y) := \{(x_i, y_i)\}_{i=1}^N,
\end{equation}
for $x_i \in \mathbb{R}^d$ and $y_i \in \mathbb{R}^k$.

Before we input every data into a DNN model, we will apply the following normalization
for all data $x_i$ for each component.
Let denote
\begin{equation}\label{key}
[x_i]_j \longleftrightarrow \text{ the j-th component of data } x_i.
\end{equation}
Then we have following formula of for all $j = 1, 2, \cdots, d$
\begin{equation}\label{key}
[\tilde x_i]_j  = \frac{[x_i]_j - [\mu_X]_j }{\sqrt{[\sigma_X]_j}},
\end{equation}
where
\begin{equation}\label{key}
[\mu_X]_j =\mathbb{E}_{x \sim X}[[x]_j] = \frac{1}{N}\sum_{i=1}^N [x_i]_j,
\quad  [\sigma_X]_j = \mathbb{V}_{x\sim X}[[x]_j] = \frac{1}{N} \sum_{i=1}^N ( [x_i]_j - [\mu_X]_j)^2.
\end{equation}
Here $x \sim X$ means that $x$ is a discrete random variable on $X$
with probability
\begin{equation}\label{key}
\mathbb P( x = x_i ) = \frac{1}{N},
\end{equation}
for any $x_i \in X$.

For simplicity, we rewrite the element-wise definition above as the following
compact form
\begin{equation}\label{eq:normlizationData}
\tilde x_i = \frac{x_i - \mu_X }{\sqrt{\sigma_X}},
\end{equation}
where
\begin{equation}\label{key}
x_i , \tilde x_i , \mu_X, \sigma_X \in \mathbb{R}^d,
\end{equation}
defined as before and all operations in \eqref{eq:normlizationData} are element-wise.

Here we note that, by normalizing the data set, we have the next properties
for new data $\tilde x \in \tilde X$ with component $j = 1,2,\cdots,d$,
\begin{equation}\label{key}
\mathbb{E}_{\tilde X}[[\tilde x]_j] = \frac{1}{N} \sum_{i=1}^N [\tilde x_i]_j = 0,
%\begin{pmatrix}
%0\\0\\\vdots\\0
%\end{pmatrix} \in \mathbb{R}^d,
\end{equation}
and
\begin{equation}\label{key}
\mathbb{V}_{\tilde X}[[\tilde x]_j] = \frac{1}{N} \sum_{i=1}^N ([\tilde x_i]_j - \mathbb{E}_{\tilde X}[[\tilde x]_j] )^2 = 1.
%\begin{pmatrix}
%1\\1\\\vdots\\1
%\end{pmatrix} \in \mathbb{R}^d.
\end{equation}

Finally, we will have a ``new'' data set
\begin{equation}\label{key}
\tilde X = \{\tilde x_1, \tilde x_2, \cdots, \tilde x_N \},
\end{equation}
with unchanged label set $Y$. For the next sections, without special notices, we use $X$ data set
as the normalized one as default.

\subsection{Data normalization for images in CNNs}
For images, consider we have a color image data set $(X,Y) := \{(x_i, y_i)\}_{i=1}^N$ where
\begin{equation}\label{key}
x_i \in \mathbb{R}^{3 \times m\times n}.
\end{equation}
We further denote these the $(s,t)$ pixel value for data $x_i$ at channel $j$ as:
\begin{equation}\label{key}
[x_i]_{j;st} \longleftrightarrow (s,t) \text{ pixel value for } x_i \text{ at channel } j,
\end{equation}
where $1\le i \le N, 1\le j\le 3, 1\le s \le m$, and  $1\le j\le n$.

Then, the normalization for $x_i$ is defined by
\begin{equation}\label{key}
[\tilde x_i]_{{j;st}} = \frac{[x_i]_{{j;st}} - [\mu_X]_j }{\sqrt{[\sigma_X]_j}},
\end{equation}
where %$\bm 1 \in \mathbb{R}^{m\times n}$ with all elements equal to $1$ and
\begin{equation}\label{key}
[x_i]_{{j;st}}, [\tilde x_i]_{{j;st}}, [\mu_X]_j, [\sigma_X]_j \in \mathbb{R}.
\end{equation}
Here
\begin{equation}\label{key}
[\mu_X]_j = \frac{1}{m\times n\times N} \sum_{ 1 \le i \le N} \sum_{1\le s \le m, 1 \le t \le n} [x_i]_{j;st}.
\end{equation}
and
\begin{equation}\label{key}
[\sigma_X]_j = \frac{1}{ N \times m\times n} \sum_{ 1 \le i \le N} \sum_{1\le s \le m, 1 \le t \le n} ([x_i]_{j;st} -[\mu_X]_j )^2.
\end{equation}
In batch normalization, we confirmed with Lian by both numerical test and code checking that BN also use the above
formula to compute the variance in CNN for each channel.

Another way to compute the variance over each channel is to compute the standard deviation on each channel for every data,
and then average them in the data direction.
\begin{equation}\label{key}
\sqrt{[\tilde \sigma_X]_j} = \frac{1}{ N} \sum_{ 1 \le i \le N}  \left( \frac{1}{m\times n}\sum_{1\le s \le m, 1 \le t \le n} ([x_i]_{j;st} - [\mu_i]_j )^2 \right)^{\frac{1}{2}},
\end{equation}
where
\begin{equation}\label{key}
[\mu_i]_j  = \frac{1}{m\times n} \sum_{1\le s \le m, 1 \le t \le n} [x_i]_{j;st}.
\end{equation}

\subsection{Comparison of $\sqrt{[\sigma_X]_j}$ and  $\sqrt{[\tilde \sigma_X]_j}$ on CIFAR10.}

They share the same $\mu_X$ as
\begin{equation}\label{key}
\mu_X = \begin{pmatrix}
0.49140105 & 0.48215663 & 0.44653168
\end{pmatrix}.
\end{equation}
But they had different standard deviation estimates:
\begin{equation}
\begin{aligned}
\sqrt{[\sigma_X]_j} &= \begin{pmatrix}
0.24703284 & 0.24348499 & 0.26158834
\end{pmatrix} \\
%\sqrt{[\bar \sigma_X]_j} &= \begin{pmatrix}
%0.12835675 & 0.12578563 & 0.1533168
%\end{pmatrix} \\
\sqrt{[\tilde \sigma_X]_j} &= \begin{pmatrix}
0.20220193 & 0.19931635 & 0.20086373
\end{pmatrix}
\end{aligned}
\end{equation}

\section{Initialization for deep neural networks}

\subsection{Xavier's Initialization}
The goal of Xavier initialization~\cite{glorot2010understanding} is to initialize the deep neural network to avoid gradient vanishing or blowup when the input is white noise.

Let us denote the DNN models as:
\begin{equation}
\begin{cases}
f^1(x) &= W^1 x + b^1 \\
f^{\ell}(x) &= W^\ell \sigma(f^{\ell-1}(x)) + b^\ell \quad \ell = 2:L \\
f(x) &= f^{L} \\
\end{cases},
\end{equation}
with $x \in \mathbb{R}^{n_0}$ and $f^{\ell} \in \mathbb{R}^{n_\ell}$. More precisely, we have
\begin{equation}\label{key}
W^\ell \in \mathbb{R}^{n_{\ell} \times n_{\ell-1}}.
\end{equation}

The basic assumptions that we make are:
\begin{itemize}
	%% \item The input $x$ is a mean $0$ random variable with identity covariance, i.e. $\mathbb{E}(x) = 0$, $\mathbb{E}(x_ix_j) = 0$ if $i\neq j$, and $\mathbb{E}(x_i^2) = 1$
	% \item \blue{The input $x$ is a mean $0$ random vector with the same variance for each component, i.e. $\mathbb{E}[[x]_i] = 0$ and $\mathbb{E}[[x]_i^2] (\mathbb{V} [[x]_i]) =$ $\mathbb{E}[[x]_j^2] (\mathbb{V}[[x]_j])$. }
	% (Here we notice that, after data normalization as discussed before this assumption holds directly.)
	\item The initial weights $W^\ell_{ij}$ are i.i.d symmetric random variables with mean $0$, namely the
	probability density function of $W^\ell_{ij}$ is even.
	\item The initial bias $b^\ell = 0$.
\end{itemize}

Now we choose the variance of the initial weights to ensure that the features $f^L$ and gradients don't blow up or vanish. To this end we have the following lemma.

\begin{lemma}\label{lemm:init}
	Under the previous assumptions $f^\ell_i$  is a symmetric random variable with $\mathbb{E}[f^\ell] = 0$.
	Moreover, we have the following identity
	\begin{equation}\label{eq:FWini}
	\mathbb{E}[(f^{\ell}_i)^2] = \sum_{k}\mathbb{E}[(W^\ell_{ik})^2]\mathbb{E}[\sigma(f^{\ell-1}_k)^2].
	\end{equation}
	% \begin{equation}\label{key}
	%  \mathbb V [f^{L}_i] =  \left(\Pi_{\ell=1}^{L} n_{\ell-1} {\rm Var} [W^\ell_{st}] \right)\mathbb V [x_j].
	% \end{equation}
\end{lemma}
%Note that here we don't have that $f^\ell_i$ and $f^\ell_j$ are actually independent, just that they are `linearly' independent.

Now, if $\sigma  = id$, we can prove by induction from $\ell = 1$ that
	\begin{equation}\label{key}
	\mathbb V [f^{L}_i] =  \left(\Pi_{\ell=2}^{L} n_{\ell-1} {\rm Var} [W^\ell_{st}] \right) \left(\mathbb{V}[W^1_{st}]\sum_{k}\mathbb{E}[( [x]_k)^2] \right).
	\end{equation}
We make this assumption that $\sigma  = id$, which is pretty reasonably since most activation functions in use at the time (such as the hyperbolic tangent) were close to the identity near $0$.

Now, if we set
\begin{equation}\label{key}
\mathbb{V}[W^\ell_{ik}] = \frac{1}{n_{\ell-1}}, \quad \forall \ell \ge 2,
\end{equation}
we will obtain
\begin{equation}
\mathbb V [f^{L}_i] = \mathbb V [f^{L-1}_j] = \cdots =  \mathbb V [f^{1}_k] = \mathbb{V}[W^1_{st}]\sum_{k}\mathbb{E}[( [x]_k)^2].
\end{equation}

Thus, in pure DNN models, it is enough to just control $\sum_{k}\mathbb{E}[( [x]_k)^2]$.

A similar analysis of the propagation of the gradient ($\frac{\partial L(\theta)}{\partial f^\ell}$) suggests that we set
\begin{equation}\label{key}
\mathbb{V}[W^\ell_{ik}] = \frac{1}{n_{\ell}}.
\end{equation}

Thus, the {\bf Xavier's initialization} suggests to initialize $W^\ell_{ik}$ with variance as:
\begin{itemize}
	\item To control $\mathbb V [f^{\ell}_i] $:
	\begin{equation}\label{key}
	{\rm Var}[W^\ell_{ik}] = \frac{1}{n_{\ell-1}}.
	\end{equation}
	\item To control $\mathbb{V}[\frac{\partial L(\theta)}{\partial f_i^\ell}]$:
	\begin{equation}\label{key}
	{\rm Var}[W^\ell_{ik}] = \frac{1}{n_{\ell}}.
	\end{equation}
	\item Trade-off to control $\mathbb{V} [\frac{\partial L(\theta)}{\partial W_{ik}^\ell}]$:
	\begin{equation}\label{key}
	{\rm Var}[W^\ell_{ik}] = \frac{2}{n_{\ell-1} + n_\ell}.
	\end{equation}
\end{itemize}

Here we note that, this analysis works for all symmetric type distribution around zero,
but we often just choose uniform distribution $\mathcal U(-a,a)$ and normal distribution $\mathcal N(0,s^2)$.
Thus, the final version of Xavier's initialization takes the trade-off type as
\begin{equation}
W^{\ell}_{ik} \sim \mathcal{U}(-\sqrt{\frac{6}{n_\ell+n_{\ell-1}}}, \sqrt{\frac{6}{n_\ell+n_{\ell-1}}}),
\end{equation}
or
\begin{equation}
W^{\ell}_{ik} \sim \mathcal{N}(0,  {\frac{2}{n_\ell+n_{\ell-1}}}).
\end{equation}

%\subsection{Variance analysis in backward propagation phase}
%See the separate file in {``6DL/HandWrittenNotes/InitBackward.pdf''}
%\includepdf[pages=-,pagecommand={}]{HandWrittenNotes/InitBackward.pdf}
%\includepdf[pages=-,pagecommand={}]{497/handwritten_notes/MultidimensionalVariance.pdf}

\subsection{Kaiming's initialization}
In~\cite{he2015delving}, Kaiming He and others extended this analysis to get an \textit{exact} result when the activation function is the {\bf ReLU}.

We first have the following lemma for symmetric distribution.
\begin{lemma}
	If $X_i \in \mathbb{R}$ for $i=1:n$ are i.i.d with symmetric probability density function $p(x)$, i.e. $p(x)$ is even.
	Then for any nonzero random vector $Y = (Y_1, Y_2, \cdots, Y_n) \in \mathbb{R}^n$ which is independent with $X_i$,
	the following random variable
	\begin{equation}\label{key}
	Z = \sum_{i=1}^n X_i Y_i,
	\end{equation}
	is also symmetric.
\end{lemma}

Then state the following result for ReLU function and random variable with
symmetric distribution around $0$.
\begin{lemma}
	If $X$ is a random variable on $\mathbb{R}$ with symmetric probability density $p(x)$ around zero, i.e.,
	\begin{equation}\label{key}
	p(x) = p(-x).
	\end{equation}
	Then we have $\mathbb{E} X = 0$ and
	\begin{equation}\label{key}
	\mathbb{E}[[{\rm ReLU}(X)]^2] = \frac{1}{2}{\rm Var}[X].
	\end{equation}
\end{lemma}

Based on the previous Lemma~\ref{lemm:init}, we know that $f^{\ell-1}_k$ is a symmetric distribution around $0$.
The most important observation in Kaiming's paper~\cite{he2015delving} is that:
\begin{equation}\label{key}
\mathbb{V}[ f^\ell_i ] = n_{\ell-1}  \mathbb{V}[W^\ell_{ij}] {\mathbb{E}[[\sigma(f^{\ell-1}_j)]^2]} = n_{\ell-1} \mathbb{V}[W^\ell_{ik}] {\frac{1}{2} \mathbb{V}[f^{\ell-1}_k]},
\end{equation}
{if $\sigma = {\rm ReLU}$}.
Thus, Kaiming's initialization suggests to take:
\begin{equation}\label{key}
\mathbb{V}[W^\ell_{ik}] = \frac{2}{n_{\ell-1}}, \quad \forall \ell \ge 2.
\end{equation}

For the first layer $\ell=1$, by definition
\begin{equation}\label{key}
f^1 = W^1 x + b^1,
\end{equation}
there is no ReLU, thus it should be $\mathbb{V}[W^1_{ik}] = \frac{1}{d}$.
For simplicity, they still use $\mathbb{V}[W^1_{ik}] = \frac{2}{d}$ in the paper~\cite{he2015delving}.
Similarly, an analysis of the propagation of the gradient suggests that we set
$\mathbb{V}[W^\ell_{ik}] = \frac{2}{n_{\ell}}$.
However, in paper~\cite{he2015delving} authors did not suggest to take the trade-off version, they just chose
\begin{equation}\label{key}
\mathbb{V}[W^\ell_{ik}] = \frac{2}{n_{\ell-1}},
\end{equation} as default.

Thus, the final version of Kaiming's initialization takes the forward type as
\begin{equation}
W^{\ell}_{ik} \sim \mathcal{U}(-\sqrt{\frac{6}{n_{\ell-1}}}, \sqrt{\frac{6}{n_{\ell-1}}}),
\end{equation}
or
\begin{equation}
W^{\ell}_{ik} \sim \mathcal{N}(0,  {\frac{2}{n_{\ell-1}}}).
\end{equation}

%And another difference is that Xavier use the uniform distribution but He use the Gaussian distribution.
%More precisely,
%However, in Pytorch implementation, the uniform distribution is also applied.

\subsection{Initialization in CNN models and experiments}
For CNN models, following the analysis above we have the next iterative scheme in CNNs
\begin{equation}\label{key}
f^{\ell,i} = K^{\ell,i} \ast \sigma (f^{\ell,i-1}),
\end{equation}
where $f^{\ell,i-1} \in \mathbb{R}^{c_\ell\times n_\ell \times m_\ell }$, $f^{\ell,i} \in \mathbb{R}^{h_\ell\times n_\ell \times m_\ell}$ and $K \in \mathbb{R}^{(2k+1) \times (2k+1) \times h_\ell \times c_\ell}$.
Thus we have
\begin{equation}\label{key}
[f^{\ell,i}]_{h;p,q} = \sum_{c=1}^{c_\ell}\sum_{s,t=-k}^k K^{\ell,i}_{h,c;s,t} \ast \sigma ([f^{\ell,i-1}]_{c;p+s,q+t}).
\end{equation}
Take variance on both sides, we will get
\begin{equation}\label{key}
\mathbb{V} [[f^{\ell,i}]_{h;p,q}] = c_\ell (2k+1)^2 \mathbb{V}[K^{\ell,i}_{h,o;s,t}] \mathbb{E}[([f^{\ell,i-1}]_{o;p+s,q+t})^2],
\end{equation}
thus we have the following initialization strategies:
\begin{description}
	\item[Xavier's initialization]
	\begin{equation}\label{key}
	\mathbb{V}[K^{\ell,i}_{h,o;s,t}] = \frac{2}{ (c_\ell + h_\ell) (2k+1)^2}.
	\end{equation}
	\item[Kaiming's initialization]
	\begin{equation}\label{key}
	\mathbb{V}[K^{\ell,i}_{h,o;s,t}] = \frac{2}{c_\ell (2k+1)^2}.
	\end{equation}
\end{description}

Here we can take this Kaiming's initialization as:
\begin{itemize}
	\item Double the Xavier's choice, and get
	\begin{equation}\label{key}
	\mathbb{V}[K^{\ell,i}_{h,o;s,t}] = \frac{4}{(c_\ell + h_\ell )(2k+1)^2}.
	\end{equation}
	\item Then pick $c_\ell$ or $h_\ell$ for final result
	\begin{equation}\label{key}
	\mathbb{V}[K^{\ell,i}_{h,o;s,t}] = \frac{4}{(c_\ell + h_\ell )(2k+1)^2} = \frac{2}{c_\ell (2k+1)^2}.
	\end{equation}
\end{itemize}

And they have the both uniform and normal distribution type.
\begin{figure}[H]
	\begin{center}
		\includegraphics[width=0.45\linewidth]{converge_22layers}
	\end{center}
	\caption{The convergence of a \textbf{22-layer} large model. The x-axis is the number of training epochs. The y-axis is the top-1 error of 3,000 random val samples, evaluated on the center crop. Use ReLU as the activation for both cases. Both Kaiming's initialization (red) and ``\emph{Xavier's}'' (blue) \cite{glorot2010understanding} lead to convergence, but Kaiming's initialization starts reducing error earlier.}
	\label{fig:converge_22layers}
	%\end{figure}
	%\begin{figure}[t]
	\begin{center}
		\includegraphics[width=0.45\linewidth]{converge_30layers}
	\end{center}
	\caption{The convergence of a \textbf{30-layer} small model (see the main text). Use ReLU as the activation for both cases. Kaiming's initialization (red) is able to make it converge. But ``\emph{Xavier's}'' (blue) \cite{glorot2010understanding} completely stalls - It is also verified that that its gradients are all diminishing. It does not converge even given more epochs.}
	\label{fig:converge_30layers}
\end{figure}

Given a 22-layer model, in cifar10 the convergence with Kaiming's initialization is faster than Xavier's,
but both of them are able to converge and the validation accuracies with two different initialization
are about the same(error is 33.82,33.90).

With extremely deep model with up to 30 layers,
Kaiming's initialization is able to make the model convergence. On the contrary, Xavier's method completely stalls the learning.

\section{Batch Normalization in DNN and CNN}
\subsection{Recall the original DNN model}
Consider the classical (fully connected)  artificial deep neural network (DNN) $f^L$,
\begin{equation}\label{nn}
\begin{cases}
f^1&= \theta^1 (x) := W^1 x+b^1,\\
f^\ell &= \theta^{\ell} \circ \sigma (f^{\ell-1}) :=W^\ell  \sigma (f^{\ell-1}) +b^\ell,\ \ell=2,...,L.
\end{cases}
\end{equation}
where $x\in\mathbb{R}^n$ is the input vector, $\sigma$ is a non-linear function (activation).

\subsection{``Real'' Batch Normalization and ``new'' model}
\paragraph{Definition of BN operation based on the batch}
Following the idea in normalization, we consider that we have the all
training data as
\begin{equation}\label{eq:trainingdata}
(X,Y) := \{x_i, y_i\}_{i=1}^N.
\end{equation}

Since the normalization is applied to
each activation independently, let us focus on a particular activation $ [f^\ell]_k$ and omit $k$ as $f^\ell$ for clarity.
We have $N$ values of this activation
in the batch,
$$ X=\{x_1, \cdots, x_N\}.$$
Let the normalized values be
$\hat f^\ell$, and their linear transformations be $\tilde f^\ell$.
\begin{equation}\label{def:BNeq}
\begin{aligned}
\mu^\ell_{ X} & \leftarrow \mathbb{E}_{x \sim X} [f^\ell(x)] =  \frac{1}{N}\sum_{i=1}^N f^\ell(x_i) & \text{ batch mean}& \\
\sigma^\ell_{ X} & \leftarrow \mathbb{E}_{x \sim X}  \left[(f^\ell(x)-\mathbb{E}_{x \sim X}[ f^\ell(x)])^2 \right] = \frac{1}{N}\sum_{i=1}^N (f^\ell(x_i)-\mu_{ X})^2 &  \text{ batch variance}&\\
\hat f^\ell (x) & \leftarrow \frac{f^\ell(x)-\mu^\ell_{ X}}{\sqrt{\sigma^\ell_{ X}+\epsilon}}   &\text{ normalize}&\\
\tilde f^\ell(x)  & \leftarrow \gamma^\ell \hat f^\ell (x) + \beta^\ell
&\text{ scale and shift}&
\end{aligned}
\end{equation}
Here we note that all these operations in the previous equation are defined by element-wise.
Then at last, we define the BN operation based on the batch set as
\begin{equation}\label{eq:BNop}
{\rm BN}_{X}({f^\ell(x)}) = \tilde f^\ell(x) := \gamma^\ell  \frac{f^\ell(x)-\mu^\ell_{ X}}{\sqrt{\sigma^\ell_{ X}+\epsilon}}  + \beta^\ell  ,
\end{equation}
where $\tilde f^\ell(x)$, $\mu^\ell_{ X}$ and $\sigma^\ell_{ X}$  are given above.

\paragraph{``New" model for BN}
In summary, we have the new DNN model with BN as:
\begin{equation}\label{nn-BN0}
\begin{cases}
\tilde f^1(x_i)&= (\theta^1 (x_i) ),\\
\tilde f^\ell &= \theta^{\ell} \circ \sigma \circ {\rm BN}_{ X}(\tilde f^{\ell-1}), \quad  \ell=2,...,L.
\end{cases}
\end{equation}
%Here we would like to notice that all these input data
%$x_i \in X$, it is already normalized over all the original data set with
%the same fashion.
%Thai is to say
%$$
%x_i = {\rm BN}_{\bar X}(\bar x_i),
%$$
%for the original data $\bar x_i \in \bar X$.

For a more comprehensive notation,
we can use the next notation
\begin{equation}
\sigma_{\rm BN} := \sigma \circ {\rm BN}_{ X}.
\end{equation}

Here one thing is important that we need to mention is that because of the new
scale $\gamma^\ell$ and shift $\beta^\ell$ added after the BN operation.
We can remove the basis $b^\ell$ in $\theta^\ell$, thus to say the real model we will compute should be
\begin{equation}\label{nn-BN}
\begin{cases}
\tilde f^1(x_i)&= W^1 x_i ,\\
\tilde f^\ell &= W^{\ell}  \sigma_{\rm BN}(\tilde f^{\ell-1}), \quad  \ell=2,...,L.
\end{cases}
\end{equation}

Combine the two definition, we note
\begin{equation}
\tilde \Theta := \{W, \gamma, \beta\},
\end{equation}
where
$W = \{W^1, \cdots, W^l \}$, $\gamma := \{\gamma^2, \cdots, \gamma^L\}$ and $\beta := \{\beta^2, \cdots, \beta^L\}$.

Finally, we have the loss function as:
\begin{equation}\label{eq:loss-BN}
\mathcal L(\tilde \Theta) = \mathbb{E}_{(x,y)\sim (X,Y)} \approx \frac{1}{N}\sum_{i=1}^N \ell(\tilde f^L(x_i; \tilde \Theta), y_i).
\end{equation}

A key observation in \eqref{eq:loss-BN} and the new BN model \eqref{nn-BN} is that
\begin{equation}\label{eq:threeExpectation}
\begin{aligned}
\mu^\ell_{ X}
%& \leftarrow \frac{1}{N}\sum_{i=1}^N f^\ell(x_i)
&= \mathbb{E}_{x \sim X} [f^\ell(x)],\\
\sigma^\ell_{ X}
%& \leftarrow \frac{1}{N}\sum_{i=1}^N (f^\ell(x_i)-\mu_{ X})^2
&=  \mathbb{E}_{x \sim X}  \left[(f^\ell(x)-\mathbb{E}_{x \sim X}[ f^\ell(x)])^2 \right], \\
\mathcal L(\tilde \Theta)
%&= \sum_{i=1}^N L(\tilde f^J(x_i; \tilde \Theta), y_i)
&=  \mathbb{E}_{(x,y)\sim (X,Y)}  \left[\ell(\tilde f^L(x_i; \tilde \Theta), y_i) \right].
\end{aligned}
\end{equation}
Here we need to mention that
$$
x \sim X
$$
means $x$ subject to the discrete distribution of all data $X$.
%For example:
%$$
%{p}_{X}(x = x_i) = \frac{1}{N}.
%$$
%or more mathematically we can use the Dirac distribution as:
%$$
%p_{X}(x) = \frac{1}{N} \sum_{i=1}^N\delta(x - x_i).
%$$

%Similar with the case from (batch) gradient descent to stochastic gradient descent.
%Therefore, BN makes the second simplification: since we use mini-batches
%in stochastic gradient training, {\em each mini-batch produces estimates of the mean and variance} of each
%activation for whole batch.

\subsection{BN: some ``modified" SGD on new batch normalized model}
Following the key observation in \eqref{eq:threeExpectation}, and recall the
similar case in SGD, we do the the sampling trick in \eqref{eq:loss-BN} and
obtain the mini-batch SGD:
\begin{equation}\label{eq:mini-batch-sample}
x \sim X \approx x \sim \mathcal B,
\end{equation}
here $\mathcal B$ is a mini-batch of batch $X$ with $\mathcal B \subset X$.

However, for problem in \eqref{eq:loss-BN}, it is very difficult to find some
subtle sampling method because of the composition of $\mu^\ell_{\mathcal X}$
and $[\sigma^\ell_{\mathcal X}]^2 $. However, one simple way for sampling
\eqref{eq:loss-BN} can be chosen as taking \eqref{eq:mini-batch-sample} for
all the expectation case in \eqref{eq:loss-BN} and \eqref{eq:threeExpectation}.

This is to say, in training process ($t$-th step for example), once we choose $B_t \subset X$
as the mini-batch, then the model becomes
\begin{equation}\label{nn-BN-training}
\begin{cases}
\tilde f^1(x_i)&= W^1 x_i ,\\
\tilde f^\ell &= W^{\ell}  \sigma_{\rm BN}(\tilde f^{\ell-1}), \quad  \ell=2,...,L.
\end{cases}
\end{equation}
where
\begin{equation}
\sigma_{\rm BN} := \sigma \circ {\rm BN}_{\mathcal B_t},
\end{equation}
or we can say that $X$ is replaced by $\mathcal B_t$ in this case.

Here ${\rm BN}_{\mathcal B_t}$ is defined by
\begin{equation}\label{def:BNeq-traning}
\begin{aligned}
\mu^\ell_{\mathcal B_t} & \leftarrow \frac{1}{m}\sum_{i=1}^m f^\ell(x_i) & \text{ mini-batch mean}& \\
\sigma^\ell_{\mathcal B_t} & \leftarrow \frac{1}{m}\sum_{i=1}^m (f^\ell(x_i)-\mu_{\mathcal B_t})^2 &  \text{ mini-batch variance}&\\
\hat f^\ell (x) & \leftarrow \frac{f^\ell(x)-\mu^\ell_{\mathcal B_t}}{\sqrt{\sigma^\ell_{\mathcal B_t}+\epsilon}}   &\text{ normalize}&\\
{\rm BN}_{\mathcal B_t}(\tilde f^{\ell}) := \tilde f^\ell(x) & \leftarrow \gamma^\ell \hat f^\ell (x) + \beta^\ell
&\text{ scale and shift}&
\end{aligned}
\end{equation}

Here BN operation introduce some new parameters as $\gamma$ and $\beta$.
Thus to say, for training phase, if we choose mini-batch as $\mathcal B_t$ in $t$-th training
step, we need to take gradient as
\begin{equation}
\frac{1}{m}\nabla_{\tilde \Theta} \sum_{i \in \mathcal B_t} \ell(\tilde f^L(x_i; \tilde \Theta), y_i),
\end{equation}
which needs us the to take gradient for $\mu_{B}^\ell$ or $[\sigma_B^\ell]^2$
w.r.t $w^i$ for $i \le \ell$.

{\bf Questions:} To derive the new gradient formula for BN step because of the fact that
$$
\mu^\ell_{\mathcal B_t}, \quad \text{and} \quad \sigma^\ell_{\mathcal B_t},
$$
contain the output of $\tilde f^{\ell-1}$.

This is exact the batch normalization method described in \cite{ioffe2015batch}.

\subsection{Testing phase in Batch-Normalized DNN}
One key problem is that, in the BN operator, we need to compute the mean and variance
in a data set (batch or mini-batch). However, in the inference step, we just input one data
into this DNN, how to compute the BN operator in this situation.

Actually,  the $\gamma$ and $\beta$ parameter is fixed after training, the only problem is
to compute the mean $\mu$ and variance $\sigma^2$.
All the mean $ \mu_{\mathcal B_t}  $ and variance $\sigma^2_{\mathcal B_t} $ during the training phase
are just the approximation of the mean and variance of whole batch i.e.  $ \mu_{X}  $ and $\sigma^2_{X}$
as shown in \eqref{eq:threeExpectation}.

One natural idea might be just use the BN operator w.r.t to the whole training data set, thus to say
just compute $\mu_X$ and $\sigma_X^2$ by definition in \eqref{def:BNeq}.

However, there are at least the next few problems:
\begin{itemize}
	\item computation cost,
	\item ignoring the statistical approximation (don't make use of the $ \mu_{\mathcal B_t}  $ and  $\sigma^2_{\mathcal B_t} $ in training phase).
\end{itemize}

Considering that we have the statistical approximation for $ \mu_{X}  $ and $\sigma^2_{X}$
during each SGD step, moving average might be a more straightforward way.
Thus two say, we define the $\mu^\ell$ and $[\sigma^\ell]^2$ for the inference (test) phase
as
\begin{equation}
\mu^\ell = \frac{1}{T}\sum_{t=1}^T \mu^\ell_{\mathcal B_t},
\quad \sigma^\ell =\frac{1}{T} \frac{m}{m-1}\sum_{t=1}^T\sigma^\ell_{\mathcal B_t}.
\end{equation}
Here we take Bessel's correction for unbiased variance.
The above moving average step is found in the original paper of BN in \cite{ioffe2015batch}.

Another way to do this is to call the similar idea in momentum.
At each time step we update the running averages for mean and variance using an exponential decay based on the momentum parameter:
\begin{equation}
\begin{aligned}
\mu^\ell_{\mathcal B_t} &=\alpha \mu^\ell_{\mathcal B_{t-1}} + (1-\alpha)\mu^\ell_{\mathcal B_t} \\
\sigma^\ell_{\mathcal B_t} &=\alpha \sigma^\ell_{\mathcal B_{t-1}} + (1-\alpha)\sigma^\ell_{\mathcal B_t},
\end{aligned}
\end{equation}
$\alpha$ is close to $1$, we can take it as $0.9$ generally. Then we all take bath mean and variance as
$\mu^\ell_X  \approx \mu^\ell_{\mathcal B_T}$  and $\sigma^\ell_X  \approx \sigma^\ell_{\mathcal B_T}$.

Many people argue that the variance here should also use Bessel's correction.

\subsection{Batch Normalization for CNN}
One key idea in BN is to do normalization with each scalar features (neurons)
separately along a mini-batch.
Thus to say, we need one to identify what is neuron in CNN.
This is a historical problem, some people think neuron in CNN
should be the pixel in each channel some thing that each channel is just
one neuron. BN choose the later one.
{\bf One (most ?) important reason for this choice is the fact of computation cost. }

For convolutional layers, BN additionally wants the normalization
to  obey the convolutional property -- so that different elements
of the same feature map, at different locations, are normalized in the
same way.
To compute $\mu^\ell_{\mathcal B_t}$, we take mean of the set of all values in a feature map across both the
elements of a mini-batch and spatial locations -- so for a mini-batch
of size $m$ and feature maps of size $m_\ell \times n_\ell$ (image geometrical size),
we use the effective mini-batch of size $ m m_\ell n_\ell$.
We learn a pair of parameters $\gamma_k$ and $\beta_k$ per feature map (k-th channel), rather than per activation.

For simplicity, then have the following BN scheme for CNN
\begin{equation}\label{def:BNeq-traningCNN}
\begin{aligned}
[\mu^\ell_{\mathcal B_t}]_{j} & \leftarrow \frac{1}{m \times m_\ell \times n_\ell }\sum_{i=1}^m \sum_{1\le s\le m_\ell, 1\le t \le n_\ell} [f^\ell(x_i)]_{j;st}
&&\text{ mean on channel }j \\
[\sigma^\ell_{\mathcal B_t}]_{j} & \leftarrow \frac{1}{m \times m_\ell \times n_\ell }\sum_{i=1}^m \sum_{1\le s\le m_\ell, 1\le t \le n_\ell}
([f^\ell(x_i)]_{j;st}-[\mu^\ell_{\mathcal B_t}]_j)^2   &&\text{ variance on channel }j\\
[\hat f^\ell (x)]_{j;st} & \leftarrow \frac{[f^\ell(x)]_{j,st}-[\mu^\ell_{\mathcal B_t}]_j}{\sqrt{[\sigma^\ell_{\mathcal B_t}]_j+\epsilon}}   &&\text{ normalize }\\
[{\rm BN}_{\mathcal B_t}(\tilde f^{\ell})]_{j;st} &:= [\tilde f^\ell(x)]_{j;st}  \leftarrow [\gamma^\ell]_j [\hat f^\ell (x)]_{j;st} + [\beta^\ell]_{j}
&&\text{ scale and shift on channel}
\end{aligned}
\end{equation}


\section{Exponentiation by Binary Decomposition}\label{sec:decomposition}
This section covers the computation of $x^n$ for a nonnegative integer $n$
as in section \texttt{11-3} of \emph{Hackers's Delight}
\cite{Warren:2012:HD:2462741} (p. \texttt{288}).
The base $x$ can be of any type
with an associative operation $\cdot$ defined on it.
Therefore the method presented here
is not only applicable to multiplication on integers and floating points,
but also i.e.\ concatenation of strings
($(\text{'ab'})^3 = \text{'ababab'}$).

The goal is to reduce the number of required operations
with the most primitive approach taking $(n-1)$ steps.
Whatever the operation may be,
it will be referred to as ``multiplication'' below.

Using the identities $x^{(a+b)} = x^a \cdot x^b$
and $(x^a)^b = x^{(a \cdot b)}$,
a decomposition of the exponent $n$ can be achieved
and thereby the number of required multiplications is reduced up to a minimum.
Consider these examples\footnote{
Henry Warren \cite{Warren:2012:HD:2462741} mentions $x^{15} = (x^3)^5$
which requires 6 (one more) multiplications.
The decomposition shown here was selected as a replacement to demonstrate that
simply chaining prime factors isn't necessarily the optimal solution.
}:
\[ x^{15} = x^3 \cdot ((x^3)^2)^2 \hspace{4ex} x^{27} = ((x^3)^3)^3 \]

\begin{quotation}\noindent
``Perhaps surprisingly, there is no known simple method that, for all $n$,
finds the optimal sequence of multiplications to compute $x^n$.
The only known methods involve an extensive search.''\\
\phantom{a} \hfill -- Hacker's Delight
\cite{Warren:2012:HD:2462741} (p. \texttt{289})
\end{quotation}

This circumstance leads to the question, which decomposition
is simple to obtain and a relatively good approximation of the optimum.
The binary representation of $n$ is cheap
when already operating in a low level coding environment.
Consider the same examples as before:
\[ x^{15} = x^8 \cdot x^4 \cdot x^2 \cdot x \hspace{4ex}
x^{27} = x^{16} \cdot x^8 \cdot x^2 \cdot x \]

In comparison, binary decomposition takes
for $n := 15$ a total of $6$ ($\log_2 8 + 3$)
instead of $5$ ($2 + 1 + 1 + 1$) operations
and for $n := 27$ a total of $7$ ($\log_2 16 + 3$)
instead of $6$ ($2 + 2 + 2$) multiplications.

The algorithm for this computation\footnote{
See \url{https://hackersdelight.org/hdcodetxt/iexp.c.txt}
\cite{Warren:HD:Website}
} scans the binary representation from right to left,
by isolating the least significant bit (\lstinline$n&1$)
and right-shifting it (\lstinline$n = n >> 1$).
For every new bit, a factor \lstinline$p$ (that starts as $x$)
is squared and therefore equals $x, x^2, x^4, \dots$ throughout the cycles.
It is multiplied onto the final result \lstinline$y$ (which starts as $1$)
if and only if the bit in that cycle is \lstinline$1$.