%!TEX root=report.tex
\subsection{Basis expansion}

Basis expansion is done by adding more columns to design matrix. Let us first motivate why this might be needed.

\subsubsection{Motivation}
It has previously been described how the design matrix $X$ was constructed such that we had
\begin{equation*}
\resizebox{\textwidth}{!}{$
X = \left[\begin{matrix}
	\mathbf{1} &
	\mathbf{t} &
	\frac{1}{2} \mathbf{t}^2 &
	\cos\left( \dfrac{2 \pi}{\frac{365.242}{1}} \mathbf{t} \right) &
	\sin\left( \dfrac{2 \pi}{\frac{365.242}{1}} \mathbf{t} \right) &
	\cdots &
	\cos\left( \dfrac{2 \pi}{\frac{365.242}{18}} \mathbf{t} \right) &
	\sin\left( \dfrac{2 \pi}{\frac{365.242}{18}} \mathbf{t} \right)
\end{matrix}\right].
$}
\end{equation*}
The problem with the above $X$ is that the periodic columns assumes the pattern accounts for the entire period. This causes problems in areas such as Denmark, where the winter comes later than ``usual'', or the summer is much colder/warmer etc..

This leads to higher variance of the residuals which decreases the accuracy of the velocity and acceleration parameter of the model. By adding columns to the design matrix, such that the periodic pattern is only assumed extend over one period, the accuracy can be improved.

\subsubsection{The Hinge-function}

One can achieve this basis expansion is by virtue of the hinge-function
\begin{equation}
f(x - \zeta)_+ = \begin{cases}
  f(x - \zeta) & \text{if } x - \zeta \ge 0 \\
  0                 & \text{otherwise}
\end{cases} \quad \forall x \in \mathbb{R}, \zeta \in \mathbb{R}
\end{equation}
$\zeta$ denotes the ``knot'' between the zero part and the $f$ part. By choosing $f$ and adding the hinge function to the design matrix, a basis expansion where terms are only active for some parts of the time series is obtained.

When $f$ is a polynomial one achieve a piecewise polynomial which is called a spline.
In this case each hinge function is guaranteed to be continuous and it can be shown that the spline's derivatives are continuous to order $M-2$ where $M$ is the degree of polynomial $f$ \cite[p.~144]{statistical-learning}. When dealing with polynomials, splines which are continuous up to and including its second order derivatives are called \textit{cubic splines} and they generate nicely looking curves when performing the regression. \cite[p.~143]{statistical-learning}.

\subsubsection{Knots and trigonometric functions}
To model each year with different behavior, in essence all that is needed is to create 9 knots (10 intervals requires 9 splits) and construct the hinge functions.
However, since the functions being used are sines and cosines, there is no guarantee of continuity at the knots (only continuity if the trigonometric function evaluates to zero at the knot).
In grim cases, when multiple discontinuous functions are combined, one can end up with cusps, that is a steep descent/ascend followed by the opposite movement in rapid succession.
If this phenomenon becomes too large, one should consider rejecting the model, this will be discussed in the result section.


\subsection{Rational numbers}

\subsubsection{Defining rational numbers}

We previously defined integers in terms of natural numbers. Similarly we can define rational numbers in terms of integers.

\(\forall ab \in \mathbb{I} (\neg (b=0)\rightarrow \exists c (b.c=a))\)

A rational is an ordered pair of integers.

\(\{\{a\},\{a,b\}\}\)

So that:

\(\{\{a\},\{a,b\}\}=\dfrac{a}{b}\)

\subsubsection{Converting integers to rational numbers}

Integers can be shown as rational numbers using:

\((i,1)\)

Integers can then be turned into rational numbers:

\(\mathbb{Q}=\dfrac{a}{1}\)

\(a=\dfrac{a_1}{a_2}\)

\(b=\dfrac{b_1}{b_2}\)

\(c=\dfrac{c_1}{c_2}\)

\subsubsection{Equivalence classes of rationals}

There are an infinite number of ways to write any rational number, as with integers. \(\dfrac{1}{2}\) can be written as \(\dfrac{1}{2}\), \(\dfrac{-2}{-4}\) etc.

The class of these terms form an equivalence class.

We can show these are equal:

\(\dfrac{a}{b}=\{\{a\},\{a,b\}\}\)

\(\dfrac{ca}{cb}=\{\{a\},\{a,b\}\}\)

\(\dfrac{ca}{cb}=\{\{ca\},\{ca,cb\}\}\)

\(\{\{a\},\{a,b\}\}=\{\{ca\},\{ca,cb\}\}\)


\section{Determinants and Inverses}
\subsection{Introduction}
Consider a linear map $\mathbb R^n\to\mathbb R^n$ given by the matrix $M$.
We want to define an $n\times n$ matrix $\tilde{M}=\operatorname{adj}M$ and a scalar $\det M$ with
$$M\tilde{M}=(\det M)I$$
Furthermore, $\det M$ is the factor by whcih an area in $\mathbb R^2$ or a volumn in $\mathbb R^3$ is scaled.
If $\det M\neq 0$, then  we will have
$$M^{-1}=\frac{1}{\det M}\tilde{M}$$
For $n=2$, we know that
$$\tilde{M}=
\begin{pmatrix}
    M_{22}&-M_{12}\\
    -M_{21}&M_{11}
\end{pmatrix},
\det M=
\begin{vmatrix}
    M_{11}&M_{12}\\
    M_{21}&M_{22}
\end{vmatrix}
=M_{11}M_{22}-M_{12}M_{21}
$$
works.
Note that $\det M\neq 0$ if and only if $M\underline{e_1},M\underline{e_2}$ are linearly independent if and only if $\operatorname{Im}M=\mathbb R^2$ if and only if $\operatorname{rank}M=2$.\\
For $n=3$, we recall that given any $\underline{a},\underline{b},\underline{c}\in\mathbb R^3$, the scalar $[\underline{a},\underline{b},\underline{c}]$ is the volumn of a parallelopiped spanned by $\underline{a},\underline{b},\underline{c}$.
We can also note that the standard basis vectors obey $[\underline{e_i},\underline{e_j},\underline{e_k}]=\epsilon_{ijk}$.
Note that for $M$ a real $3\times 3$ matrix, its columns are $M\underline{e_i}=M_{ji}\underline{e_j}$.
So the volumn is scaled by a factor
$$[M\underline{e_1},M\underline{e_2},M\underline{e_3}]=M_{i1}M_{j2}M_{k3}[\underline{e_i},\underline{e_j},\underline{e_k}]=M_{i1}M_{j2}M_{k3}\epsilon_{ijk}$$
We define this to be $\det M$.
We can also define $\tilde{M}$ by
$$\underline{R_1}(\tilde{M})=\underline{C_2}(M)\times\underline{C_3}(M)$$
$$\underline{R_2}(\tilde{M})=\underline{C_3}(M)\times\underline{C_1}(M)$$
$$\underline{R_3}(\tilde{M})=\underline{C_1}(M)\times\underline{C_2}(M)$$
So one can see immediately that
$$(\tilde{M}M)_{ij}=\underline{R_i}(\tilde{M})\cdot\underline{C_j}(M)=\det M\delta_{ij}$$
as desired.\\
How about when we consider $n$ in general?
\subsection{Alternating forms}
We first want to generalize our $\epsilon$ symbol to higher dimensions by considering the permutation.
\begin{definition}
    A permutation $\sigma:\{1,2,\ldots,n\}\to\{1,2,\ldots,n\}$ is a bijection of $\{1,2,\ldots,n\}$ to itself.
\end{definition}
So $\{1,2,\ldots,n\}=\{\sigma{1},\sigma{2},\ldots,\sigma{n}\}$.
It is immediate that the permutations on $n$ letters form a group $S_n$ under composition, and it is easy that $|S_n|=n!$.
\begin{definition}
    A permutaion $\tau\in S_n$ is called a transposition of $i,j\in\{1,2,\ldots,n\}$ if $\tau(i)=j,\tau(j)=i,\forall k\neq i,j, \tau(k)=k$.
    We denote $\tau$ by $(p\ q)$.
\end{definition}
\begin{proposition}
    Any permutation is a product of transpositions.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
The way we write it is not unique, but the number of transpositions is unique modulo $2$.
\begin{proposition}
    If some permutation $\sigma$ can be written as the product of $k$ transpositions and the product of $l$ transpositions, then $k\equiv l\pmod{2}$.
\end{proposition}
\begin{proof}
    Will see in groups (actually quite trivial).
\end{proof}
\begin{definition}
    We say the permutation $\sigma$ is even if it can be written as the product of an even number of transpositions, odd if otherwise.
    We define the sign, or signature function $\epsilon:S_n\to\{1,-1\}$ by
    $$\epsilon(\sigma)=
    \begin{cases}
        1\text{, if $\sigma$ is even}\\
        -1\text{, otherwise}
    \end{cases}
    $$
\end{definition}
Note that $\epsilon(\operatorname{id})=0$ and more generally $\epsilon(\sigma\circ\pi)=\epsilon(\sigma)\epsilon(\pi)$ for permutations $\sigma,\pi$.
\begin{definition}
    The $\epsilon$ symbol (or tensor) on $n$ letters is defined as
    $$
    \epsilon_{ij\ldots kl}=
    \begin{cases}
        \epsilon(\sigma)\text{, if $ij\ldots kl$ is a permutation $\sigma$ of $\{1,2,\ldots,n\}$}\\
        0\text{, otherwise. That is, some indices coincide.}
    \end{cases}
    $$
\end{definition}
So we now can define the alternating forms.
\begin{definition}
    Given vectors $\underline{v_1},\underline{v_2},\ldots,\underline{v_n}$ in $\mathbb R^n$ or $\mathbb C^n$.
    We define the alternating form to be the scalar
    $$[\underline{v_1},\underline{v_2},\ldots,\underline{v_n}]=\epsilon_{ij\ldots kl}(\underline{v_1})_i(\underline{v_2})_j\cdots(\underline{v_{n-1}})_k(\underline{v_n})_l$$
    One can check that the alternating forms when $n=2,3$ are exactly the same as we have defined them before.
\end{definition}
Note that the alternating form is multilinear, thus a tensor.
Also, it is totally antisymmetric: interchanging any two vectors changes the sign.
Equivalently,
$$[\underline{v_{\sigma(1)}},\underline{v_{\sigma(2)}},\ldots,\underline{v_{\sigma(n)}}]=\epsilon(\sigma)[\underline{v_1},\underline{v_2},\ldots,\underline{v_n}]$$
Moreover, $[\underline{e_1},\underline{e_2},\ldots,\underline{e_n}]=1$.\\
One can see immediately that
\begin{proposition}
    If the function $f:(F^n)^n\to F$ where $F=\mathbb R$ or $\mathbb C$ is multilinear, totally antisymmetric and $f(\underline{e_1},\underline{e_2},\ldots,\underline{e_n})=1$, then $f$ is uniquely determined.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
\begin{proposition}
    If some vector(s) is $\underline{v_1},\underline{v_2},\ldots,\underline{v_n}$ is a linear combination of others, then $[\underline{v_1},\underline{v_2},\ldots,\underline{v_n}]=0$.
\end{proposition}
\begin{example}
    In $\mathbb C^4$, let $\underline{v_1}=(i,0,0,2),\underline{v_2}=(0,0,5i,0),\underline{v_3}=(3,2i,0,0),\underline{v_4}=(0,0,-i,1)$, then $[\underline{v_1},\underline{v_2},\underline{v_3},\underline{v_4}]=10i$
\end{example}
\begin{proposition}
    $[\underline{v_1},\underline{v_2},\ldots,\underline{v_n}]\neq 0$ if and only if $\{\underline{v_i}\}$ is an independent set.
\end{proposition}
\begin{proof}
    We have already shown the ``only if'' part, so it remains to show the other direction.
    If $\{\underline{v_i}\}$ is independent, then it constituted a basis, hence if $[\underline{v_1},\underline{v_2},\ldots,\underline{v_n}]=0$, then by multilinearity the alternating form will be zero everywhere, which is a contradiction.
\end{proof}
\begin{definition}[Definition of determinant]
    Consider $M\in M_{n\times n}(F)$ where $F=\mathbb R$ or $\mathbb C$ with columns $\underline{C_a}$, then the determinant of $M$ is
    \begin{align*}
        \det M&=[\underline{C_1},\underline{C_2},\ldots,\underline{C_n}]\\
        &=[M\underline{e_1},M\underline{e_2},\ldots,M\underline{e_n}]\\
        &=\epsilon_{ij\ldots kl}M_{i1}M_{j2}\cdots M_{k(n-1)}M_{ln}\\
        &=\sum_{\sigma\in S_n}\epsilon(\sigma)M_{\sigma(1)1}M_{\sigma(2)2}\cdots M_{\sigma(n)n}
    \end{align*}
\end{definition}
\begin{example}
    1. The definition of determinant here in general coincides with the cases in $2$ and $3$ dimensional cases.\\
    2. If $M$ is diagonal, then $\det M$ is the product of all diagonal entries.
    So $\det I=1$.\\
    3. If we have
    $$M=\left(\begin{array}{@{}ccc|c@{}}
        &&&0\\
        &A&&\vdots\\
        &&&0\\
        \hline
        0&\dots&0&1
    \end{array}\right)$$
    Then $\det M=\det A$
\end{example}
\subsection{Properties of Determinants}
If we multiply one of the columns (or rows) of the matrix $M$ by a scalar $\lambda$ to produce $M'$, we have $\det M'=\lambda\det M$.
Furthermore, if we interchange two adjascent columns, the determinant is negated.
We can see these directly from the definitive property of determinants.
In addition, $\det M\neq 0$ if and only if the columns are linearly independent.
\begin{proposition}
    For any $n\times n$ matrix $M$, $\det M=\det M^\top$.\\
    Equivalently, $[\underline{C_1},\underline{C_2},\ldots,\underline{C_n}]=[\underline{R_1},\underline{R_2},\ldots,\underline{R_n}]$.
\end{proposition}
\begin{proof}
    We have
    \begin{align*}
        [\underline{C_1},\underline{C_2},\ldots,\underline{C_n}]&=\sum_{\sigma\in S_n}\epsilon(\sigma)M_{\sigma(1)1}M_{\sigma(2)2}\cdots M_{\sigma(n)n}\\
        &=\sum_{\sigma\in S_n}\epsilon(\sigma)M_{1\sigma^{-1}(1)}M_{2\sigma^{-1}(2)}\cdots M_{n\sigma^{-1}(n)}\\
        &=\sum_{\sigma'\in S_n}\epsilon(\sigma')M_{1\sigma'(1)}M_{2\sigma'(2)}\cdots M_{n\sigma'(n)}\\
        &=[\underline{R_1},\underline{R_2},\ldots,\underline{R_n}]
    \end{align*}
    Since the map $\sigma\to\sigma'=\sigma^{-1}$ is an automorphism on $S_n$ and $\epsilon(\sigma)=\epsilon(\sigma')$.
\end{proof}
We can evaluate determinants by expanding rows or columns.
\begin{definition}
    For $M$ an $n\times n$ matrix,
    Define $M^{ia}$ be the determinant of the $(n-1)\times (n-1)$ matrix obtained by deleting the row $i$ and column $a$ of $M$.
    This is called a minor.
\end{definition}
\begin{proposition}\label{det_formula}
    $$\forall a,\det{M}=\sum_i(-1)^{i+a}M_{ia}M^{ia}$$
    $$\forall i,\det{M}=\sum_a(-1)^{i+a}M_{ia}M^{ia}$$
\end{proposition}
\begin{proof}
    Trivial but see later sections for the proof.
\end{proof}
By some trivial computation, we discover
\footnote{We definitely did not see that coming}
that matrices having many zeros would be easier to calculate, so it brings us to the ways to simplify the determinants.\\
The first thing we could do is row/column operations.
If we modify $M$ by mapping $\underline{C_i}\mapsto \underline{C_i}+\lambda\underline{C_j}, i\neq j$ (or equivalently on rows), then the determinant is not changed.
This follows immediate from multilinearity and total antisymmetry.\\
Plus, as we stated above, if we interchange $\underline{C_i},\underline{C_j},i\neq j$ (same for rows), then the determinant changes sign.
This can help us simplify the calculation greatly since we can produce a lot of $0$'s from there.
\begin{theorem}
    For any $n\times n$ matrices $M,N$, $\det(MN)=\det(M)\det(N)$.
\end{theorem}
\begin{lemma}
    $$\epsilon_{i_1i_2\ldots i_n}M_{i_1a_1}M_{i_2a_2}\cdots M_{i_na_n}=\epsilon_{a_1a_2\ldots a_n}\det{M}$$
\end{lemma}
\begin{proof}
    Trivial.
\end{proof}
\begin{proof}[Proof of the multiplicativity of determinants]
    By the preceding lemma,
    \begin{align*}
        \det(MN)&=\epsilon_{i_1i_2\ldots i_n}(MN)_{i_11}\cdots(MN)_{i_nn}\\
        &=\epsilon_{i_1i_2\ldots i_n}M_{i_1k_1}N_{k_11}\cdots M_{i_nk_n}N_{k_nn}\\
        &=\epsilon_{i_1i_2\ldots i_n}M_{i_1k_1}\cdots M_{i_nk_n}N_{k_11}\cdots N_{k_nn}\\
        &=\det{M}\epsilon_{k_1k_2\ldots k_n}N_{k_11}\cdots N_{k_nn}\\
        &=\det{M}\det{N}
    \end{align*}
    As desired.
\end{proof}
Note that this would mean that $\det$ is a group homomorphism.
There are a few consequences of the multiplicative property:
\begin{proposition}
    1. If $M$ is invertible, then $\det(M^{-1})=\det(M)^{-1}$.\\
    2. If $M$ is orthogonal, then $\det(M)=\pm 1$.\\
    3. If $M$ is unitary, then $|\det(M)|=1$.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
\subsection{Minors, Cofactors and Inverses}
\begin{definition}
    Select a column $\underline{C_a}$ of matrix $M$, we can write $\underline{C_a}=M_{ia}\underline{e_i}$, so
    \begin{align*}
        \det M&=[\underline{C_1},\underline{C_2},\ldots,\underline{C_a},\ldots,\underline{C_n}]\\
        &=[\underline{C_1},\underline{C_2},\ldots,M_{ia}\underline{e_i},\ldots,\underline{C_n}]\\
        &=M_{ia}[\underline{C_1},\underline{C_2},\ldots,\underline{e_i},\ldots,\underline{C_n}]\\
        &=:\sum_iM_{ia}\Delta_{ia}
    \end{align*}
    So
    $$\Delta_{ia}=[\underline{C_1},\underline{C_2},\ldots,\underline{e_i},\ldots,\underline{C_n}]$$
    is called the cofactor.
    Note that the cofactor is exactly the determinant of the matrix removing the row $i$ and column $a$.
\end{definition}
\begin{proof}[Proof of Proposition \ref{det_formula}]
    We know, then, that
    $$\Delta_{ia}=(-1)^{i+a}M^{ia}$$
    by shuffling the rows and columns.
    Therefore we have
    $$\det M=\sum_iM_{ia}\Delta_{ia}=\sum_i(-1)^{i+a}M_{ia}M^{ia}$$
    since columns and rows do not have real difference (we can do a transpose anyways), the two statements are proved.
\end{proof}
Now we want to find our adjugate (or adjoint) $\tilde{M}$.
Note that in our above proof we can observe that $M_{ib}\Delta_{ia}=\det(M)\delta_{ab}$, so we can easily define $\tilde{M}_{ij}=\Delta_{ji}$.
So $\tilde{M}$ is the transpose of the matrix of cofactors, then the relation above becomes
$$M\tilde{M}=\det(M)I$$
as desired.
This justifies the existence of $\tilde{M}$ and $\det(M)$ in the full generality of $\mathbb C^N$ from beginning of this section.
\subsection{System of Linear Equations}
Consider a system of $n$ linear equations in $n$ unknowns $x_i$, written in vector-matrix form $A\underline{x}=\underline{b}$ where $A$ is some $n\times n$ matrix.
If $\det A$ is nonzero, then $A^{-1}$ exists, which implies an unique solution $\underline{x}=A^{-1}\underline{b}$.
But what if $\det A$ is zero?\\
We know that if $\underline{b}\notin\operatorname{Im}A$, then by definition there is no solution.
But if $\underline{b}\in\operatorname{Im}A$, then the entire (shifted) space $\underline{x_p}+\ker{A}$ where $A\underline{x_p}=\underline{b}$ is exactly all the solutions.
This can be seen from the linear superposition of the solutions.\\
Note that the formula $\underline{x_p}+\ker{A}$ also applied to the case for $\det A\neq 0$ since in that case $\ker{A}=\{\underline{0}\}$.
If $\underline{u_i}$ is a basis for $\ker{A}$, then the general solution is $\underline{x_p}+a_i\underline{u_i}$.
\begin{example}
    Consider $A\underline{x}=B$ where
    $$A=\begin{pmatrix}
        1&1&a\\
        a&1&1\\
        1&a&1
    \end{pmatrix},\underline{b}=\begin{pmatrix}
        1\\
        c\\
        1
    \end{pmatrix}$$
    Now $\det A=(a-1)^2(a+2)$.
    So if $a\notin \{1,-2\}$, then
    $$
    A^{-1}=\frac{1}{(a-1)(a+2)}
    \begin{pmatrix}
        -1&a+1&-1\\
        -1&-1&a+1\\
        a+1&-1&-1
    \end{pmatrix}$$
    So
    $$\underline{x}=A^{-1}\underline{b}=\frac{1}{(1-a)(a+2)}\begin{pmatrix}
        2-c-ca\\
        c-a\\
        c-a
    \end{pmatrix},c\in\mathbb R$$
    Geometrically, this solves to give a point.
    If $a=1$, then
    $$A=\begin{pmatrix}
        1&1&1\\
        1&1&1\\
        1&1&1
    \end{pmatrix}\implies\operatorname{Im}A=\left\{\lambda\begin{pmatrix}
        1\\
        1\\
        1
    \end{pmatrix}:\lambda\in\mathbb R\right\}$$
    So there is no solution if $c\neq 1$.
    For $c=1$, since $(1,0,0)^\top$ would be a particular solution, the solutions form the plane $(1,0,0)^\top+\ker A$, i.e. the general solution is of the form
    $$\begin{pmatrix}
        1-\lambda-\mu\\
        \lambda\\
        \mu
    \end{pmatrix},\lambda,\mu\in\mathbb R$$
    For $c=-2$, by again looking at the image we conclude that $c$ must be $-2$, in which case the same analysis gives us the general solution
    $$\begin{pmatrix}
        1+\lambda\\
        \lambda\\
        \lambda
    \end{pmatrix},\lambda\in\mathbb R$$
\end{example}
Let $\underline{R_1},\underline{R_2},\underline{R_3}$ be the rows of $A$, then
$$A\underline{u}=\underline{0}\iff \forall i\in\{1,2,3\},\underline{R_i}\cdot\underline{u}=0$$
Note that the latter system of equations described planes through the origin.
We know that the solution of the homogeneous problem is equivalent to finding the kernel of $A$.
If $\operatorname{rank}A=3$, then we must have $\underline{u}=\underline{0}$ since $\{\underline{R_i}\}$ would be independent.
If $\operatorname{rank}A=2$, then $\{\underline{R_i}\}$ spans a plane, so the kernel is living along the line of normal to the plane.
If $\operatorname{rank}A=1$, then the normals to the pairwise spanned planes are parallel, so the kernel is a plane.\\
Now consider instead $A\underline{u}=\underline{b}$, then it happens iff $\underline{R_i}\cdot \underline{u}=b_i$.
In this case, if $\operatorname{rank}A=3$, then the normals of the planes described by the the system intersects at a point, thus there is an unique solution.
If $\operatorname{rank}A=2$, then the planes may intersect, in which case the solution is a line, but it might not be the case.
If $\operatorname{rank}A=1$, then the planes may coincide, in which case they are all the same, thus we have a plane of solution.
But again this might not be the case.
Two of them may coincide but that is not enough.\\
But how do you solve the equations and find the kernels systematically?
\subsection{Gaussian Elimination and Echelon Form}
Consider $A\underline{x}=\underline{b}$ where $\underline{x}\in\mathbb R^n,\underline{b}\in\mathbb R^m$ and $A$ is an $m\times n$ matrix, then we can solve it by Gaussian Elimination.
In general, we can reorder rows by row operations to rewrite original system in simplier form.
Note that when we do row operations, we have to do it simultaneously on the matrix and on the vector.
Our aim is to finally transform the matrix to the following form
$$M=\begin{pmatrix}
    M_{11}&\star&\star&\dots&\star&\star&\dots&\star\\
    0&M_{22}&\star&\dots&\star&\star&\dots&\star\\
    0&0&M_{33}&\dots&\star&\star&\dots&\star\\
    \vdots&\vdots&\vdots&\ddots&\vdots&\vdots&\ddots&\vdots\\
    0&0&0&\dots&M_{kk}&\star&\dots&\star\\
    0&0&0&\dots&0&0&\dots&0\\
    \vdots&\vdots&\vdots&\ddots&\vdots&\vdots&\ddots&\vdots\\
    0&0&0&\dots&0&0&\dots&0
\end{pmatrix}$$
Note that the row rank equals the column rank.
We can use induction to prove that we can indeed use a way to obtain $M$ from $A$ in a finitely many number of row operations.
Note that $\det A=\pm\det M$


\documentclass{article}

\usepackage{mathrsfs, amsmath}    % need for subequations
\usepackage{verbatim}   % useful for program listings
\usepackage{color}      % use if color is used in text
\usepackage{hyperref}   % use for hypertext links, including those to external documents and URLs

\allowdisplaybreaks

\begin{document}
\section*{Feed Forward Neural Networks}
In this part of the tutorial I will go through the components of a fully connected feed forward neural network and show how gradient descent and backpropagation can be used to optimize the parameters of the neural network for a certain task. Our training data will be a collection of $N$ objects represented by input and output vectors $(\textbf{x}_1, \textbf{y}_1),...,(\textbf{x}_N, \textbf{y}_N)$. Each vector $\textbf{x}_n$ has length $M$, where each entry of the vector represent a different feature (or measurement) of object $n$. The network itself has trainable parameters $\textbf{W}^l$ and $\textbf{b}^l$. The weights of the network are stored in the series of matrices $\textbf{W}^l$, where the $W_{ij}^l$ entry of the tensor represents the weight of the connection from neuron $j$ in layer $l-1$ to neuron $i$ in layer $l$. The series of vectors $\textbf{b}^l$ represent the offset the neurons in layer $l$, where the entry $b_{i}^l$ is the offset of neuron $i$ in layer $l$. The activations of and inputs to the neurons in the network can be represented with the following equations
\begin{equation}
\begin{split}
\textbf{a}^0 &= \textbf{x} \\
\textbf{z}^l &= \textbf{W}^l \textbf{a}^{l-1} + \textbf{b}^l \\
\textbf{a}^l &= \sigma_l (\textbf{z}^l) \\
\hat{\textbf{y}} &= \textbf{a}^{L},
\end{split}
\end{equation}
where the subscript $n$, which denotes the training example, is suppressed so as to make the equations more transparent. Here $\textbf{z}^l$ are the inputs to the neurons in layer $l$. By applying a sigmoidal activation function $\sigma_l$ to the inputs $\textbf{z}^l$ the activations $\textbf{a}^l$ of the neurons in layer $l$ are obtained. In practice this activation function is usually a tanh, logistic, or relu function. This function can also be different for each layer of the network. For simplicity of notation the activations in the layer $l=0$ are the input vector $\textbf{x}$, and the activations of the last layer $l=L$ are the predicted output $\hat{\textbf{y}}$. In component form these equations are.
\begin{equation}
\label{eq:nn_components}
\begin{split}
a_j^0 &= x_j \\
z_i^l &= \sum_{j=0} W_{ij}^l a_j^{l-1} + b_i^l \\
a_i^l &= \sigma_l (z_i^l) \\
\hat{y} &= a_i^{L}
\end{split}
\end{equation}

\section*{Cost Functions}
The difference between the predicted output and the known output, for the $n^{th}$ training example, for a given set of parameters $\textbf{W}^l$ and $\textbf{b}^l$, can be quantified by defining the loss function $E_k(\textbf{x}_n, \textbf{y}_n ; \textbf{W}^l, \textbf{b}^l)$. For regression problems the cost function is often chosen to be the squared error loss
\begin{equation}
E_n = \frac{1}{2} \left |\hat{\textbf{y}}_n - \textbf{y}_n \right|^2
\end{equation}
For classification problems with multiple classes the cross entropy loss is used. This requires the use of the softmax activation function in the last layer to ensure the output neurons of the network output probabilities between $0$ and $1$. The softmax activation function is
\begin{equation}
\hat{\textbf{y}} = \sigma_L(\textbf{z}^{l-1}) = \frac{e^{\textbf{z}^{l-1}}}{\text{sum} \left( e^{\textbf{z}^{l-1}} \right)},
\end{equation}
where $\text{sum} \left( e^{\textbf{z}^{l-1}} \right)$ is the sum of elements of the vector $e^{\textbf{z}^{l-1}}$. The cross entropy loss of the $n^{th}$ training example is then given by the dot product between the true label vector $\textbf{y}_n$ and the the log of the predicted labels from the network $\log{\hat{\textbf{y}}_n}$
\begin{equation}
E_n = - \textbf{y}_n \cdot \log{\hat{\textbf{y}}_n}.
\end{equation}
The cost function for the entire training set is then the sum of the losses for each individual training example
\begin{equation}
E \left(\textbf{W}^l, \textbf{b}^l \right) = \frac{1}{N} \sum_{n=0}^N E_n(\textbf{x}_n, \textbf{y}_n ; \textbf{W}^l, \textbf{b}^l)
\end{equation}

\subsection*{Minimizing the Cost Function with Backpropagation}
Optimizing the network for a particular training set means minimizing the cost function $E \left(\textbf{W}^l, \textbf{b}^l \right)$ as a function of the weights and offsets parameters. In practice the number of trainable parameters can be very large so it is impractical to use brute force to minimize the cost function. Backpropagation is the traditional gradient descent algorithm combined with the use of the chain rule to calculate the derivatives of the cost function with respect to the training parameters. Gradient descent works off the observation that from any given point in parameter space the fastest way to get to a local minimum of the cost function is to travel in the negative direction of the gradient of the cost function at that point. Thus by updating the parameters according to the rules
\begin{equation}
\begin{split}
W^l_{ij} & \rightarrow W^l_{ij} - \alpha \frac{\partial E}{\partial W^l_{ij}} \\
b^l_i & \rightarrow b^l_{i} - \alpha \frac{\partial E}{\partial b^l_i}
\end{split}
\end{equation}
eventually the values of the trainable parameters will be such that the cost function is at a local minimum. The quantity $\alpha$ is called the learning rate. It should be tuned to a value that ensures the gradient descent algorithm reaches a local minimum in a reasonable amount of time.

From the above update rules it can be seen that gradient descent requires knowledge of the derivatives of the cost function with respect to the training parameters. In theory these derivatives can be calculated numerically, but this approach is inefficient and prone to numerical error. A better approach is to use the chain rule to write the derivatives as
\begin{equation}
\begin{split}
\frac{\partial E_n}{\partial W^l_{ij}} &= \frac{\partial E_n}{\partial z_i^l } \frac{\partial z_i^l}{\partial W^l_{ij}} \\
\frac{\partial E_n}{\partial b^l_{ij}} &= \frac{\partial E_n}{\partial z_i^l} \frac{\partial z_i^l}{\partial b^l_i}
\end{split}
\end{equation}
From the definition of $z_i^l$ the inputs to the neurons in layers $l$ can be written $z_i^l=\sum_{k=0} W_{ik}^l a_k^l + b_i^l$. Thus
\begin{equation}
\begin{split}
\frac{\partial z_i^l}{\partial W^l_{ij}} &= \frac{\partial}{\partial W^l_{ij}} \left( \sum_{k=0} W_{ik}^l a_k^{l-1} + b_i^l \right) = a_j^{l-1} \\
\frac{\partial z_i^l}{\partial b^l_i} &= \frac{\partial}{\partial b^l_{i}} \left( \sum_{k=0} W_{ik}^l a_k^{l-1} + b_i^l \right) = 1
\end{split}
\end{equation}
Furthermore the derivative $\frac{\partial E_n}{\partial z_i^l }$ has the interpretation of being the "error" of the network in layer $l$. This quantity is usually given the name $\delta_i^l = \frac{\partial E_n}{\partial z_i^l }$. Inserting these expressions into the above equations, the derivatives of the cost function at layer $l$ can be written in terms of the activation at layer $l-1$ and the errors at layer $l$.
\begin{equation}
\begin{split}
\frac{\partial E_n}{\partial W^l_{ij}} &= \delta_i^l a_j^{l-1} \\
\frac{\partial E_n}{\partial b^l_{ij}} &= \delta_i^l
\end{split}
\end{equation}
Now all that's left is to write the errors in terms of the activations using the chain rule. For a hidden layer this error can be written as
\begin{equation}
\delta_j^{l-1} = \frac{\partial E_n}{\partial z_j^{l-1} } = \sum_k \frac{\partial E_n}{\partial z_k^l } \frac{\partial z_k^l}{\partial z_j^{l-1} }
\end{equation}
Using the definition of the inputs to the neurons in layer $l$
\begin{equation}
\begin{split}
\frac{\partial z_k^l}{\partial z_j^{l-1}} &= \frac{\partial }{\partial z_j^{l-1}} \left(\sum_k W_{ik}^l a_k^{l-1} + b_k^l \right) \\
&= \frac{\partial }{\partial z_j^{l-1}} \left(\sum_k W_{ik}^l \sigma_l (z_k^{l-1}) + b_k^l \right) \\
&=W_{ij}^l \sigma_l '(z_j^{l-1})
\end{split}
\end{equation}
Thus
\begin{equation}
\begin{split}
\delta_j^{l-1}
&= \sum_k \frac{\partial E_n}{\partial z_k^l } W_{kj}^l \sigma_l '(z_j^{l-1}) \\
&= \sigma_l '(z_j^{l-1}) \sum_k \delta_k^l W_{kj}^l
\end{split}
\end{equation}
For the output layer
\begin{equation}
\delta_i^{L}
= \frac{\partial E_n}{\partial z_i^{L}}
= \frac{\partial E_n}{\partial \hat{y}_i} \frac{\partial \hat{y}_i}{\partial z_i^{L}}
\end{equation}
For the squared error loss function (suppressing the $n$ subscript) this becomes
\begin{equation}
\delta_i^{L}
= \frac{\partial}{\partial z_i^L} \left(\frac{1}{2} \left |\hat{\textbf{y}} - \textbf{y} \right|^2 \right) \frac{\partial \hat{y}_i}{\partial z_i^{L}}
= \left( \hat{y}_i - y_i \right) \sigma_L'(z_i^L)
\end{equation}
For the cross entropy loss this becomes
\begin{equation}
\delta_i^{L}
= - \frac{\partial}{\partial \hat{y}_i} \left(\textbf{y} \cdot \log{\hat{\textbf{y}}} \right) \frac{\partial \hat{y}_i}{\partial z_i^{L}}
= -\frac{y_i}{\hat{y}_i} \sigma_L'(z_i^L)
\end{equation}

\end{document}


\section{Spectral Radius}
\label{sec:spectral_radius}

The spectral radius is a key parameter that determines some properties of the
echo state network weight matrix, which we will here denote as
$\matr{A} \in \mathbb{C} ^{n\times n}$.
If $\matr{A}$ has eigenvalues $\lambda_1, ... , \lambda_p$, where $p \leq n$,
the spectral radius is defined as
\begin{equation}
  \rho(\matr{A}) = \max \{|\lambda_1|,...,|\lambda_p|\}.
\end{equation}

Thus, to obtain the spectral radius we somehow have to find a reasonably close estimate
for the absolute largest eigenvalue.
For very large matrices, it obviously becomes infeasible to calculate all eigenvalues
which has a computational complexity of $O(n^3)$.
One method of quickly computing the largest eigenvalue is called \textbf{inverse iteraion}
(or inverse power method), but before diving into the exact algorithm we will go
through a quick recap of the linear algebra that is necessary for inverse iteration.

If the matrix $\matr{A}$ is diagonalizable, we can find a matrix $\matr{S}$ such that
\begin{equation}
  \label{eq:diagonal_matrix}
  \matr{A} = \matr{S}^{-1} {\Lambda} \matr{S},
\end{equation}
where $\Lambda$ is a diagonal matrix containing the eigenvalues
$\lambda_1,...,\lambda_p$.
From Eq.~\ref{eq:diagonal_matrix} follows that:
\begin{equation}
  \label{eq:matrix_power}
  \matr{A} \matr{A} = (\matr{S}^{-1}\Lambda\matr{S})(\matr{S}^{-1}\Lambda\matr{S})
                    = \matr{S}^{-1}\Lambda^2\matr{S},
\end{equation}
which means that for any continuous function one can write:
\begin{equation}
  f(\matr{A}) = \matr{S}^{-1} f(\Lambda) \matr{S},
\end{equation}
because every continuous function can be approximated by a polynomial function.
In our case it will turn out to be very useful, that
\begin{equation}
  \label{eq:inverse_ev}
  (\matr{A} - \sigma \mathbb{I})^{-1} =
        \matr{S}^{-1} (\Lambda - \sigma \mathbb{I})^{-1} \matr{S},
\end{equation}

because only the eigenvalues (on the diagonal of $\Lambda$) are affected by
the inversion, but the eigenvectors stay exactly the same.

\subsubsection{Power Method}
\label{ssub:power_method}
Suppose $\vec{x} \in \mathbb{C} ^n$, that can be written as the sum of eigenvectors
and eigenvalues:

\begin{equation}
  \vec{x} = \lambda_1\vec{e}_1 + ... + \lambda_p\vec{e}_p,
\end{equation}

where $|\lambda_1| > |\lambda_2| > ... > |\lambda_p|$ it follows that:
\begin{equation}
  \label{eq:sum_ev}
  \matr{A}^j \vec{x} = \lambda_1^j\vec{e}_1 + ... + \lambda_p^j\vec{e}_p,
  =  \lambda_1^j \sum_{i=0}^{p} \bigg(\frac{\lambda_i}{\lambda_1}\bigg)^j \vec{e}_i,
\end{equation}

which means that for large enough $j$ Eq.~\ref{eq:sum_ev} converges to:
\begin{equation}
  \label{eq:power_method}
  \vec{\mu} = \matr{A}^j \vec{x} \rightarrow \lambda_1^j \vec{e_1}.
\end{equation}

Eq.~\ref{eq:power_method} is called the \textbf{power method} for finding eigenvalues.
From the estimate $\vec{\mu}$ of the largest eigenvector one can easily obtain the estimate
$\sigma$ of the corresponding eigenvalue by applying the Rayleigh quotient:
\begin{equation}
  \label{eq:rayleigh_quotient}
  \sigma = \frac{\vec{\mu}^* \matr{A} \vec{\mu}}{||\vec{\mu}^* \vec{\mu}||}
\end{equation}

From Eq.~\ref{eq:sum_ev} one can quickly see that it has an error of $O(|\lambda_2/\lambda_1|^j)$,
which means that it converges very slowly if the two largest eigenvalues are very
close to each other.

\subsubsection{Inverse Iteration}
\label{ssub:inverse_iteration}
The inverse iteration method aims to solve the problem of slow convergence of the
power method by manipulating the eigenvalues of the iterated matrix favourably.

Suppose we can find a reasonably close estimate $\sigma$ of an eigenvector $\lambda_i$
of $\matr{A}$, then the matrix $(\matr{A} - \sigma \mathbb{I})$ is almost singular,
because on of the elements of its diagonalized counterpart is close to zero.
\footnote{The estimate $\sigma$ is often called shift, which is why this method is also known
as the \textit{inverse shift method}.}
This means, according to Eq.~\ref{eq:inverse_ev} that
$(\matr{A} - \sigma \mathbb{I})^{-1}$ has one very large eigenvalue,
which we can exploit for a fast conversion of the power method.
Starting from a random vector $\vec{x}_0$, the iterative update equation for the
inverse power method reads:
\begin{equation}
  \label{eq:inverse_iteration}
  \vec{x}_{i+1} = (\matr{A} - \sigma \mathbb{I})^{-1} \vec{x}_i,
\end{equation}
The shift $\sigma$ is updated at each iteration by using the Rayleigh
quotient Eq.~\ref{eq:rayleigh_quotient}.
This results in an increasingly good estimate of the desired eigenvalue, which
in turn speeds up the convergence of the inverse iteration.
This only becomes a problem once the calculated shift hits the exact value of
an eigenvalue of $\matr{A}$. Then the shifted matrix becomes singular.
\\
Eq.~\ref{eq:inverse_iteration} can be rewritten in terms of a linear system solver:
\begin{equation}
  \vec{x}_{i+1} = \text{linearSolve}(\matr{A} - \sigma \mathbb{I}, \text{ }\vec{x}_i)
\end{equation}
in order to exploit the speedup of solving a linear system compared to an inverse
matrix computation, which results in especially large speedups as soon as the
matrix $\matr{A}$ is very sparse.

The last remaining problem to solve is a reasonable estimate for the largest eigenvalue
of $\matr{A}$, preferably and upper bound in order to make sure that the iteration
does not converge to the second largest eigenvalue.
The most straight forward thing to do here would be taking the Frobenius norm of $\matr{A}$,
which provides an intuitive upper bound of the spectral radius:
\begin{align}
  |\lambda|^k ||\vec{e}|| = ||\lambda^k \vec{e}||
  &= ||\matr{A}^k \vec{e}|| \leq ||\matr{A}^k|| \cdot ||\vec{e}|| \\
  |\lambda^k| &\leq ||A^k||
\end{align}
A faster way of doing this is based on the so called Gershgorin disks \cite{Gershgorin}.
If $a_{ij}$ is an element of the complex matrix $\matr{A}$ we can define
$R_i=\sum_{j\neq i}|a_{ij}|$ as the radius of a disk $D(a_{ii}, R_i)$ centered at
$a_{ii}$.
Gershgorin's circle theorem states that the eigenvalues of $\matr{A}$ have lie
within the Gershgorin disks.
Picking the the largest $R_i$ in combination with the largest $a_{ii}$ as an upper
bound thus provides an upper bound for the largest eigenvalue in much less than
$O(n^3)$ (complexity of the Frobenius norm).

This iterative inverse method typically converges in a few steps, which makes it
vastly superior to calculating every eigenvalue and picking its maximum in order
to find the spectral radius.\\

%%Fig.~\ref{fig:inv_iter_speedup} demonstrates that by this method we can obtain
%%the spectral radius of a matrix about 40 times faster than by calculating
%%all eigenvalues by hand and picking the largest one.
%
%%\begin{figure}[h]
%%  \centering
%%  \includegraphics[width=0.8\linewidth]{inv_iter_speedup.png}
%%  \caption{Speedup of inverse iteration algorithm compared to the Numpy eigenvalue
%%  calculation. CREDITS TO JAMES AVERY}
%%  \label{fig:inv_iter_speedup}
%%\end{figure}
%
%%if diagonalizaable:
%%else:
%%spectral theorem with defective part etc.
%
%% and: http://www.cs.cornell.edu/~bindel/class/cs6210-f09/lec26.pdf


\subsection{Model Selection}
\label{sec:evidence}

Given a model class $M$ and a vector of model parameters $\theta$, we
can factor the joint distribution $P(\theta,D|M)$ as:

\begin{equation}
P(\theta,D|M) = P(\theta|D,M)\, P(D|M) = P(D|\theta,M)\, P(\theta|M),
\end{equation}

which gives us Bayes' Theorem:

\begin{equation}
P(\theta|D,M) = {{P(D|\theta,M)\, P(\theta|M)}\over{P(D|M)}}
\end{equation}

We identify $P(\theta|M)$ as the prior, $P(D|\theta, M)$ as the
likelihood, and $P(\theta|D,M)$ as the posterior distribution for
$\theta$, of which the chain is an estimate.

The quantity $P(D|M)$ is known as the {\it evidence}, and can be seen
to play the role of a normalizing factor:

\begin{equation}
P(D|M) = \integral{}{}{P(D|\theta,M)\, P(\theta|M)}{\theta}.
\end{equation}

In the context of parameter fitting, $P(D|M)$ is usually left
unspecified, as only the ratio of the posterior is used to choose
between different values of parameters.

When comparing different classes of models, however, the evidence
plays a central role, as it represents the probability of the data
given a particular model, marginalized over all possible values of
that model's parameters $\theta$.

When considering two classes of models, the evidence ratio
$\mathcal{E}$ is defined as:

\begin{equation}
{{P(D|M_1)}\over{P(D|M_2)}} = {{\integral{}{}{P(D|\alpha,M_1)\, P(\alpha|M_1)}{\alpha}}\over{\integral{}{}{P(D|\beta, M_2)\, P(\beta|M_2)}{\beta}}}
\end{equation}

Note that the evidence ratio contains the usual ``goodness of fit''
criterion in the form of the likelihood $P(D|M)$; i.e.,
if our two ``hypotheses '' just consist of two sets of parameter
values $P(\alpha|M_1) = \delta(\alpha_0)$ and
$P(\beta|M_2) = \delta({\beta_0})$, the we recover the usual
likelihood ratio:

\begin{equation}
\mathcal{E} = {{P(D|\alpha_0)}\over{P(D|\beta_0)}}
\end{equation}

If the hypotheses are more complex, however, the evidence ratio
retains a dependence on the integral over the priors, something that
is often referred to as the {\it Ockham factor}; i.e., models are
penalized if only a small part of the prior parameter range matches
the data.  If for example, our hypotheses consisted of two different
priors for the same parameters $\vec{\alpha}$, where the likelihood
was 1 for $\vec{\alpha} < \vec{\alpha}_{max}$ and 0 everwhere
else, and where $H_1$ allowed only a fraction $f$ of parameter space
over which the likelihood was non-zero, while $H_2$ allowed the full
range of parameter space over which the likelihood was non-zero, then

\begin{equation}
\mathcal{E} = {{\integral{}{}{P(\alpha|M_1)}{\alpha}}\over{\integral{}{}{P(\alpha|M_2)}{\alpha}}} = f,
\end{equation}

that is, $H_1$ would be disfavored over $H_2$ by that same factor $f$,
even though both hypotheses contain some range of $\vec{\alpha}$ over
which the model fits the data equally well.

In the context of Markov chains, one way to estimate the evidence
proceeds as follows:

\begin{equation}
P(\theta|D,M)\, P(D|M) = {{P(D|\theta,M)\, P(\theta|M)}}
\end{equation}

or

\begin{equation}
P(D|M)\, {{P(\theta|D,M)}\over{P(D|\theta,M)}} = P(\theta|M)
\end{equation}

whence

\begin{equation}
P(D|M)\, \integral{}{}{{{P(\theta|D,M)}\over{P(D|\theta,M)}}}{\theta} = \integral{}{}{P(\theta|M)}{\theta} = 1.
\end{equation}

The term

\begin{equation}
\left<P(D|\theta,M)^{-1}\right> \equiv \integral{}{}{{{P(\theta|D,M)}\over{P(D|\theta,M)}}}{\theta}
\end{equation}

is just the average of $P(D|\theta,M)^{-1}$ over the posterior.
In the context of Markov chains, then, the quantity

\begin{equation}
\hat{Z} \equiv {{1}\over{\left<P(D|\theta,M)^{-1}\right>}},
\end{equation}

where $\left<P(D|\theta,M)^{-1}\right>$ is the average value of the
inverse likelihood over the chain, can be taken as an estimate of the
evidence.


% !TeX root = thoughts.tex
\begin{appendices}
\section{\label{APP:affinenormal}Affine transformations of normal distributions}
Proving the relationship's of the means and covariances between affine transformations is easily done from the definition of linear transformations and covariances.

For the means $\overline{\mathbf{x}}$ and $\overline{\mathbf{y}}$ of respectively the normal distributions $\mathbf{X}$ and $\mathbf{Y}$ we use the definition of the transformation of $\mathbf{X}$ to $\mathbf{Y}$. $E\left\{ a \right\}$ is the expected value of $a$.

\begin{align}
	\overline{\mathbf{y}} &= E\left\{ \mathbf{y} \right\}\nonumber\\
	&=E\left\{ \mathbf{L}\mathbf{x} + \bm{c} \right\}\nonumber\\
	&=\mathbf{L}E\left\{ \mathbf{x}  \right\}+ \bm{c}\nonumber\\
	&=\mathbf{L}	\overline{\mathbf{x}}+ \bm{c}\\
	\square \span \nonumber
\end{align}

For the covariances we use it's definition:

\begin{align}
	\mathbf{\Sigma}_\mathbf{y}
	&\triangleq
	E\left\{
		\left( \mathbf{y} -  \overline{\mathbf{y}} \right)
		\left( \mathbf{y} -  \overline{\mathbf{y}} \right)^T
	\right\} \nonumber \\
	&=
	E\left\{
	\left( \mathbf{L}\mathbf{x} + \bm{c} -   \mathbf{L}\overline{\mathbf{x}} + \bm{c} \right)
	\left( \mathbf{L}\mathbf{x} + \bm{c} -   \mathbf{L}\overline{\mathbf{x}} + \bm{c} \right)^T
	\right\}  \nonumber \\
	&=
	E\left\{
	\left( \mathbf{L}\mathbf{x} -   \mathbf{L}\overline{\mathbf{x}} \right)
	\left( \mathbf{L}\mathbf{x} -   \mathbf{L}\overline{\mathbf{x}} \right)^T
	\right\} \nonumber \\
	&=
	E\left\{
	\mathbf{L}(\mathbf{x} - \overline{\mathbf{x}})
	(\mathbf{x} - \overline{\mathbf{x}} )^T \mathbf{L}^T
	\right\} \nonumber \\
	&=
	\mathbf{L}\;
	E\left\{
	(\mathbf{x} - \overline{\mathbf{x}})
	(\mathbf{x} - \overline{\mathbf{x}} )^T
	\right\}
	\mathbf{L}^T \nonumber \\
	&=
	\mathbf{L}\;
	\mathbf{\Sigma}_\mathbf{x}
	\mathbf{L}^T\\
	\square \span \nonumber
\end{align}

\end{appendices}


\section{Orthogonal Complement and Adjoint Map}
\begin{definition}
    Suppose $V=U\oplus W$.
    The projection operator $\pi=\pi_W:V\to W$ into $W$ is defined via $u+w\mapsto w$ for any $u\in U,w\in W$.
\end{definition}
Easy to see that $\pi$ is linear and $\pi^2=\pi$
\begin{remark}
    We have $\pi_U=\operatorname{id}-\pi_W$.
\end{remark}
Of course, in the case where $U=W^\perp$, we can have something better.
\begin{lemma}
    Let $V$ be an inner product space and $W\le V$ finite dimensional subspace of $V$, then:\\
    (a) If $\{e_i\}$ is an orthonormal basis of $W$, then $\forall v\in V,\pi(v)=\sum_i\langle v,e_i\rangle e_i$.\\
    (b) $\forall v\in V,w\in W,\|v-\pi(v)\|\le\|v-w\|$ with equality iff $w=\pi(v)$.
\end{lemma}
\begin{proof}
    Just observe that $v-\pi(v)\in W^\perp$ which is known to be a complementary subspace of $W$.
    This gives (a) immediately, and for (b) we have $\|v-w\|^2=\|v-\pi(v)+\pi(v)-w\|^2=\|v-\pi(v)\|^2+\|\pi(v)-w\|^2\ge \|v-\pi(v)\|$.
\end{proof}
\begin{proposition}
    Let $V,W$ be finite dimensional inner product spaces and $\alpha\in L(V,W)$.
    Then there is a unique linear map $\alpha^\ast:W\to V$ such that $\forall v\in V,w\in W,\langle \alpha(v),w\rangle=\langle v,\alpha^\ast(w)\rangle$.
    Moreover, if $B,C$ are orthonormal bases of $V,W$, then $[\alpha^\ast]_{C,B}=(\overline{[\alpha]}_{B,C})^\top$.
\end{proposition}
\begin{proof}
    Brute-force computation.
\end{proof}
\begin{definition}
    This map $\alpha^\ast$ is called the adjoint of $\alpha$.
\end{definition}
\begin{remark}
    One might notice that we used the same notation for adjoint and dual of a map.
    This (intentional) abuse of notation can be justified by considering the linear isomorphisms $\psi_{R,V}:V\to V^\ast,\psi_{R,W}:W\to W^\ast$ via $\psi_{R,V}(v)=\langle \cdot,v\rangle,\psi_{R,W}(w)=\langle \cdot,w\rangle$ which immediately satisfies $\alpha^\ast_{\rm adjoint}=\psi_{R,V}^{-1}\circ\alpha^\ast_{\rm dual}\circ\psi_{R,W}$.
    \[
        \begin{tikzcd}
            W^\ast\arrow{r}{\alpha^\ast_{\rm dual}}&V^\ast\\
            W\arrow{u}{\psi_{R,W}}\arrow[swap]{r}{\alpha^\ast_{\rm adjoint}}&V\arrow[swap]{u}{\psi_{R,V}}
        \end{tikzcd}
    \]
\end{remark}
\begin{definition}
    Let $V$ be an inner product space.
    A map $\alpha\in L(V)$ is self-adjoint if $\alpha=\alpha^\ast$, i.e. $\forall v,w\in V,\langle\alpha(v),w\rangle=\langle v,\alpha(w)\rangle$.\\
    It is called an isometry if $\alpha^\ast\circ\alpha=\operatorname{id}$, or $\langle\alpha(v),\alpha(w)\rangle=\langle v,w\rangle$ for any $v,w\in V$.
\end{definition}
\begin{remark}
    By the polarisation identity, $\alpha$ is an isometry iff $\|\alpha(v)\|=\|v\|$ for any $v\in V$.
\end{remark}
\begin{lemma}
    Let $V$ be a finite dimensional inner product space over $\mathbb R$ (resp. $\mathbb C$).
    Then $\alpha\in L(V)$ is self-adjoint iff for any orthonormal basis $B$ of $V$, $[\alpha]_B$ is symmetric (resp. Hemitian).
    It is an isometry iff for any orthonormal basis $B$ of $V$, $[\alpha]_B$ is orthonormal (resp. unitary).
\end{lemma}
\begin{proof}
    Immediate.
\end{proof}
The collection of isometries are naturally subgroups of $L(V)$.
\begin{definition}
    Let $V$ be a finite dimensional inner product space over a field $F=\mathbb R$ or $\mathbb C$.
    The subgroup of isometries $\{\alpha\in L(V):\alpha^\ast\circ\alpha=\operatorname{id}\}\le L(V)$ is called the orthogonal group $O(V)$ of $V$ when $F=\mathbb R$ and the unitary group $U(V)$ of $V$ when $F=\mathbb C$.
\end{definition}
\begin{remark}
    Fix an orthonormal basis $\{e_i\}$ of $V$.
    Then there is a one-to-one correspondence between the isometries in $V$ and the orthonormal bases of $V$ via $\alpha\leftrightarrow \{\alpha(e_i)\}$.
\end{remark}


\section{The Classification of Finite Abelian Groups}
We will prove (the generalization of) the following theorem later when we look at modules:
\begin{theorem}\label{classify_fin_abe}
    Every finite abelian group is isomorphic to a product of cyclic groups.
\end{theorem}
In this section, we shall look at the uniqueness criterion of this statement.
In fact, this representation is not unique in general, but we can get some sort of a uniqueness statement.
\begin{lemma}
    If $m,n$ are coprime, then $C_m\times C_n\cong C_{mn}$.
\end{lemma}
\begin{proof}
    Let $g_m$ be the generator of $C_m$ and $g_n$ be that of $C_n$, then $(g_m,g_n)\in C_m\times C_n$ has order $\gcd(m,n)=mn$.
\end{proof}
\begin{corollary}
    Let $G$ be a finite abelian group, then $G\cong C_{n_1}\times C_{n_2}\times\cdots\times C_{n_k}$ such that any $n_i$ is a prime power
\end{corollary}
\begin{proof}
    Let $n$ be a positive integer, then $n=p_1^{e_1}\cdots p_r^{e_r}$ where $p_i$ are distinct primes, then
    $$C_n\cong C_{p_1^{e_1}}\times \cdots\times C_{p_r^{e_r}}$$
    by the preceding lemma.
    Combining this with the theorem gives the result.
\end{proof}
In fact, what we will prove is the following refinement of Theorem \ref{classify_fin_abe}.
\begin{theorem}\label{fin_abe_struct}
    Let $G$ be a finite abelian group, then $G\cong C_{d_1}\times C_{d_2}\times\cdots\times C_{d_t}$ such that $1<d_1|d_2|d_3|\cdots|d_{t-1}|d_t$.
\end{theorem}
\begin{remark}
    The integers $n_1,n_2,\ldots$ are up to a reordering, and $d_1,d_2,\ldots$ are uniquely determined by the group $G$.
    The proof (which we omit) works by counting the elements of $G$ with every possible order (it is enough to consider prime powers though).
\end{remark}
\begin{example}
    1. Abelian groups of order $8$ can only be $C_8,C_2\times C_4,C_2\times C_2\times C_2$.\\
    2. Abelian groups of order $12$ can only be $C_2\times C_2\times C_3\cong C_2\times C_6,C_4\times C_3\cong C_{12}$
\end{example}


\chapter{Multiplying Polynomials in Python}

At this point, you have created a nice toolbox of functions for
dealing with lists of coefficients as polynomials. Create a file called \filename{poly.py} and copy the folowing functions into it:
\begin{itemize}
\item \pyfunction{evaluate\_polynomial}
\item \pyfunction{polynomial\_to\_string}
\item \pyfunction{add\_polynomials}
\item \pyfunction{scalar\_polynomial\_multiply}
\item \pyfunction{subtract\_polynomial}
\end{itemize}

Now create another file in the same directory called \filename{test.py}. Type this into that file:
\begin{Verbatim}
import poly

polynomial_a = [9.0, -4.0, 3.0, -5.0]
print('Polynomial A =', poly.polynomial_to_string(polynomial_a))

polynomial_b = [-9.0, 0.0, 4.0, 2.0, 1.0]
print('Polynomial B =', poly.polynomial_to_string(polynomial_b))

# Evaluation
value_of_b = poly.evaluate_polynomial(polynomial_b, 3)
print('Polynomial B at 3 =', value_of_b)

# Adding
a_plus_b = poly.add_polynomials(polynomial_a, polynomial_b)
print('A + B =', poly.polynomial_to_string(a_plus_b))

# Scalar multiplication
b_scalar = poly.scalar_polynomial_multiply(-3.2, polynomial_b)
print('-3.2 * Polynomial B =', poly.polynomial_to_string(b_scalar))

# Subtraction
a_minus_b = poly.subtract_polynomial(polynomial_a, polynomial_b)
print('A - B =', poly.polynomial_to_string(a_minus_b))
\end{Verbatim}

When you run it, you should get the following:
\begin{Verbatim}
Polynomial A = -5.0x^3 + 3.0x^2 + -4.0x + 9.0
Polynomial B = 1.0x^4 + 2.0x^3 + 4.0x^2 + -9.0
Polynomial B at 3 = 162.0
A + B = 1.0x^4 + -3.0x^3 + 7.0x^2 + -4.0x
-3.2 * Polynomial B = -3.2x^4 + -6.4x^3 + -12.8x^2 + 28.8
A - B = -1.0x^4 + -7.0x^3 + -1.0x^2 + -4.0x + 18.0
\end{Verbatim}

Now you are ready to implement multiplication of polynomials. The function will look like this:
\begin{Verbatim}
def multiply_polynomials(a, b):
  ...Your code here...
\end{Verbatim}
It will return a list of coefficients.

In an exercise in the last chapter, you were asked `` Let's say I have
two polynomials, $p_1$ and $p_2$.  $p_1$ has degree 23.  $p_2$ has
degree 12.  What is the degree of their product?'' The answer was $23 +
12 = 35$.

In our implementation, a polynomial of degree 23 is held in a list of length 24.

In Python we wil be trying to multiply a polynomial $a$ and a
polynomial $b$ represented as lists. What is the degree of that product?
\begin{Verbatim}
      result_degree = (len(a) - 1) + (len(b) - 1)
\end{Verbatim}

Now, we need to create an array of zeros that is one longer than that. Here is a cute Python trick: if you have a list, you can replicate it using the * operator.
\begin{Verbatim}
a = [5,7]
b = a * 4
print(b)
# [5, 7, 5, 7, 5, 7, 5, 7]
\end{Verbatim}

Here's how you will get a list of zeros:
\begin{Verbatim}
      result = [0.0] * (result_degree + 1)
\end{Verbatim}

We will step through $a$ getting the index and value of each entry. You can do this in one line using \pyfunction{enumerate}:
\begin{Verbatim}
      for a_degree, a_coefficient in enumerate(a):
\end{Verbatim}
For each of those, we will step through the entire $b$ polynomial. As
you multiply together each term, you will add it to appropriate
coefficient of the result.

Here is the whole function:
\begin{Verbatim}
def multiply_polynomials(a, b): # What is the degree of the resulting
polynomial?  result_degree = (len(a) - 1) + (len(b) - 1)

    # Make a list of zeros to hold the coefficents result = [0.0] *
    (result_degree + 1)

    # Iterate over the indices and values of a for a_degree,
    a_coefficient in enumerate(a):

        # Iterate over the indices and values of b for b_degree,
        b_coefficient in enumerate(b):

            # Calculate the resulting monomial coefficient =
            a_coefficient * b_coefficient degree = a_degree + b_degree

            # Add it to the right bucket
            result[degree] = result[degree] + coefficient

    return result
\end{Verbatim}

Take a long look at that function.  When you understand it, type it into \filename{poly.py}.

In \filename{test.py}, try out the new function:
\begin{Verbatim}
# Multiplication
a_times_b = poly.multiply_polynomials(polynomial_a, polynomial_b)
print('A x B =', poly.polynomial_to_string(a_times_b))
\end{Verbatim}

This is an example of a \emph{nested loop}. The outer loop steps
through the polynomial $a$. For each step it takes, the inner loop
steps through the entire polynomial $b$.

\section{Something surprising about lists}

You can imagine that you might want to create two very similar polynomials. Let's say polynomial $c$ is $x^2 + 2x + 1$ and polynomial $d$ is $x^2 -2x + 1$.  You might think you are very clever to just alter that degree 1 coefficient like this:
\begin{Verbatim}
c = [1.0, 2.0, 1.0]
d = c
d[1] = -2.0
\end{Verbatim}

If you printed out $c$, you would get $[1.0, -2.0, 1.0]$.  Why? You
assigned two variables ($c$ and $d$) to the \emph{the same list}.  So
when you use one reference ($d$) to change the list, you see the
change if you look at the list from either reference. \emph{FIXME:
  Diagram of two references to the same list here.}

To create two separate lists, you would need to explicitly make a copy:
\begin{Verbatim}
c = [1.0, 2.0, 1.0]
d = c.copy()
d[1] = -2.0
\end{Verbatim}


\chapter{Matrices and Linear Maps}

Each linear map on a finite dimensional map corresponds to a matrix and
\[ A =
\begin{pmatrix}
	a_{11} & \dots &  a_{1n} \\
	\vdots & \ddots & \vdots \\
	a_{m1} & \dots & a_{mn}
\end{pmatrix}
 = (a_{ij})_{\substack{i=1,...,m \\ j=1,...n}}
\]

$m$ is the number of rows and $n$ is the number of columns.

Now we want to see how matrices can be related to linear mappings. Consider $T \in \mathcal{L}(U,V)$ where $V,W$ are finite dimensional vector spaces. Let $v_1,...v_n$ be a basis for $V$ and $w_1,...w_m$ be a basis of $W$. The linear mapping is well defined as soon as we know how it acts on the basis vectors. Because then we can express it as linear combination and extend it any vecor in the space.
\begin{align*}
v &= \lambda_1 v_1 + ... + \lambda_n v_n , v \in V \\
T(v) &= T(\lambda_1 v_1 + ... + \lambda_n v_n) \\
&= \lambda_1 T(v_1) + ... + \lambda_n T(v_n)
\end{align*}

A matrix is a compact way of doing the above!

Each image vector vector $T(v_j)$ can be expressed in the basis $w_1,...,w_m$. Here $a_{ij}...a_{mj} \in \Re$ :
\begin{equation*}
	T(v_j) = a_{1j} w_1 + ... + a_{mj} w_m
\end{equation*}

We now stack these coefficients in a matrix.

\[
\begin{pmatrix}
a_{11} & \dots & a_{ij} & \dots & a_{1n} \\
\vdots  & &  \vdots & & \vdots \\
a_{m1} & \dots & a_{mj} & \dots & a_{mn}
\end{pmatrix}
\]

This is the matrix of mapping $T$ with respect to the basis $v_1,...,v_n$ of $V$ and $w_1,...w_m$ of $W$.
There are $m$ rows, one for each basis vector of $W$. There are $n$ columns, one for each basis vector of $V$.  \\

\textbf{Notation}: Let $T : V \rightarrow W$ be a linear mapping and let $\mathcal{B}$ be a basis of $V$ and $\mathcal{C}$ be a basis of $W$. We denote $H(T, \mathcal{B}, \mathcal{C})$ the matrix corresponding to $T$ with basis $\mathcal{B}$ and $\mathcal{C}$ \\

\textbf{Convinient properties of matrices}: Let $V, W$ be vector spaces and consider the basis is fixed. Let $S, T \in \mathcal{L}(V, W)$.

\begin{itemize}
	\item $H(S+T) = H(S) + H(T)$
	\item $H(\lambda S) = \lambda H(S)$
	\item For $v = \lambda_1 v_1 + ... + \lambda_n v_n$ we have that $T(v) =   H(T) \begin{pmatrix}
	\lambda_1 \\
	\vdots \\
	\lambda_n
	\end{pmatrix}$, where $v_1,...v_n$ are the basis of $V$.
	\item $H(S \circ T) = H(S) \cdot  H(T)$
\end{itemize}

\section{Invertiable maps and matrices}

\begin{definition}
	Let $T \in \mathcal{L}(V,W)$ is called invertible if there exists another linear map $S \in \mathcal{L}(W,V)$ :
	\begin{center}
			$S \circ T = \mathcal{I}_v$ and $T \circ S = \mathcal{I}_w$
	\end{center}
\end{definition}
The map $S$ is called the inverse of $T$, denoted by $T^{-1}$.

\begin{proposition}
	Inverese maps are unique.
\end{proposition}

\begin{proposition}
	A linear map is invertible if and only if it is injective and surjective.
\end{proposition}

\begin{proof}
	Invertible $\implies$ injective: \\
	Suppose $T(u) = T(v)$. Then $u = T^{-1}(T(u)) = T^{-1}(T(v)) = v$. This means the mapping is injective. \\
	Invertible $\implies$ surjective: \\
	Let $w \in W$. Then $w = T(T^{-1}(w)) \implies w \in range(T)$. This means that mapping is surjective. \\
	 Injective and surjective $\implies$ invertible \\
	 Let $w \in W$, there exists unique $v \in V$ such that $T(v) = w$. Now define the mapping $S(w) = v$. Clearly we have $T \circ S = \mathcal{I}$. Let $v \in V$, then -
	 \begin{align*}
	 T((S \circ T)(v)) = (T \circ S)(Tv) = \mathcal{I} \circ Tv = Tv
	 \end{align*}
	 $\implies (S \circ T)v = v \implies S \circ T = \mathcal{I}$
\end{proof}

\subsection{Inverse Matrix}

\begin{definition}
	A square matrix $A \in F^{n \times m}$ is invertible if there exists a square matrix $B \in F^{n \times m}$ such that -
	\begin{equation*}
		A \cdot B = B \cdot A = \mathcal{I}
	\end{equation*}
	The matrix $B$ is called the inverse matrix and is denoted by $A^{-1}$
\end{definition}

\begin{proposition}
	The inverse matrix represents the inverse of the corresponding linear map, that is: $T: V \rightarrow V$
	\begin{equation*}
		M(T^{-1}) = (M(T))^{-1}
	\end{equation*}
\end{proposition}

Essentially we are saying that the matrix of the inverse map is the same taking the matrix of the map and then doing the inverse.


\section{Infinite Limits}
Although our limit definition works for finite values of $c$, it's also useful to think about what happens as $c$ goes to $\pm\infty$.
We'll need to add to our limit definition to incorporate infinite values, since it doesn't make sense to talk about neighborhoods at infinity.
\begin{definition}
	Let $f$ be a real-valued function defined on some subset $D \subseteq \R$ that contains arbitrarily large values.
	\begin{equation*}
		\lim_{x \to \infty}{f(x)} = L
	\end{equation*}
	if for every real $\epsilon > 0$, there is a real number $N > 0$ such that for all $x \in D$,
	\begin{equation}
		x > N \implies \abs{f(x) - L} < \epsilon.
	\end{equation}
\end{definition}

All the same properties that we described for finite limits, like the Sum and Difference Rule, still hold for infinite limits.

\subsection{End Behavior Model}
When x is numerically large, we can often model the behavior of a complicated function with a simplier one that behaves roughly the same for numerically large input values and is the same in the limit.
There are a few rules that these follow.
\begin{enumerate}
	\item For a polynomial, the end-behavior is highest-degree term.
	\item For a rational function, like a ratio of polynomials, the end behavior is the ratio of the highest degree terms.
	\item For more complicated functions, we may need to use some reasoning about the graph of the function and limit properties to determine end-behavior.
\end{enumerate}

\subsection{Horizontal Asymptotes}
Horizontal Asymptotes are a special type of end-behavior model.
\begin{definition}
	The line $y=b$ is a horizontal asymptote of $y = f(x)$ if $\lim_{x\to \infty}{f(x)} = b$ or $\lim_{x \to -\infty}{f(x)} = b$.
\end{definition}

We can determine horizontal asymptotes for rational functions (usually quotient of polynomials).
There are a few cases to consider
\begin{enumerate}
	\item If the numerator is a higher degree than the denominator, there is no horizontal asymptote, so we'll need a different method to calculate what happens at $\pm\infty$.
	\item If the denominator is a higher degree than the numerator, then there is a horizontal asymptote at $y = 0$.
	\item If the numerator and denominator have the same degree, there is a horizontal asymptote at $y = k$ where k is the ratio of the highest degree terms.
\end{enumerate}

\begin{example}
	Find the following limits, if they exist.\\
	\begin{table}[H]
	\begin{center}
	\begin{tabular}{ l l l}
		1. $\begin{aligned}[t]
			\lim_{x \to \infty}{\frac{x^3 - 6x + 1}{x^2 + 2x - 3}}
		\end{aligned}$ &
		2. $\begin{aligned}[t]
			\lim_{x\to -\infty}{\frac{x-9}{2x-x^2}}
		\end{aligned}$ &
		3. $\begin{aligned}[t]
			\lim_{x\to \infty}{\frac{6x^2-4x^5+7x-1}{12x^5-3x^2+2}}
		\end{aligned}$ \\
		\hspace{1pt} & \hspace{1pt}\\
		4. $\begin{aligned}[t]
			\lim_{x\to \infty}{\frac{3x+1}{\abs{x}+2}}
		\end{aligned}$ &
		5. $\begin{aligned}[t]
			\lim_{x \to \infty}{x + e^{-x}}
		\end{aligned}$ &
		6. $\begin{aligned}[t]
			\lim_{x \to -\infty}{x + e^{-x}}
		\end{aligned}$
	\end{tabular}
	\end{center}
	\end{table}
\end{example}
\begin{answer}
	\begin{enumerate}
		\item Since the numerator degree is bigger than the denominator degree, we'll need to use the end behavior model.
			The end behavior model tells us that the numerator term dominates and has positive values, so the limit evaluates to $\infty$.
		\item Since the denominator has higher degree than the numerator, there is a HA at $y=0$, so the limit evaluates to $0$.
		\item Since the numerator and denominator have the same degree, the limit is the ratio of the highest-degree coefficients, $\frac{-1}{3}$.
		\item The numerator and denominator have the same degree. For $x > 0$, $\abs{x}+2 = x+2$, so the limit is the ratio of highest-degree coefficients, $3$.
		\item Looking at the two terms, we can see that as $x$ gets large, $e^{-x}$ gets very small, contributing less and less to the overall value.
			So, we can say that this function as a right end behavior model of $x$, so the limit is $\infty$.
		\item Looking at the two terms, we that that as $x$ gets very large and negative, $e^{-x}$ changes much faster than $x$.
			That is, $e^{-x}$ contributes more and more to the overall value of the function compared to $x$.
			So, we can say that this function has a left end behavior model of $e^{-x}$, so the limit is $\infty$.
	\end{enumerate}
\end{answer}


\section{Exponential Distribution}
\label{sec:Exponential-Distribution}

The arrival and service processes can follow whichever probabilistic distribution. Most often, inter-arrival and service times follow the exponential distribution, because of its wide applicability and mathematical tractability.
The Exponential distribution is the continuous counterpart of the Geometric distribution. The Geometric distribution models the number of attempts until the first success, whereas the Exponential distribution models the time until the first success.

\begin{definition}[Exponential Distribution]
\label{def:Exponential-Distribution}
	A random variable $X$ is Exponentially distributed with rate\footnote{if $X \sim Exp(\lambda)$, $\lambda$ is called \textit{rate} because $\expected{X}=\frac{1}{\lambda}$.} $\lambda$ ($X \sim Exp(\lambda)$) if $X$ has the following probability density function:

	\begin{equation}
	\label{eqn:Exponential-PDF}
	f(x) = \left\{\begin{matrix}
		\lambda e^{-\lambda x} & x \geq 0\\
		0 & x \leq 0
	\end{matrix}\right.
	\end{equation}
\end{definition}

The p.d.f of $X \sim Exp(\lambda)$ is shown in \Cref{fig:Exponential-PDF}.

\begin{figure}[tp]
\label{fig:Exponential-PDF}
	\centering
	\includegraphics{fig/exponential-pdf}
	\caption{Exponential p.d.f.}
\end{figure}

From the \Cref{def:Exponential-Distribution} follows that the cumulative distribution function of $X \sim Exp(\lambda)$ is

\begin{equation}
\label{eqn:Exponential-CDF}
\begin{aligned}
	F(x) = \int_{- \infty}^{x} \partial y =
	\left\{\begin{matrix}
	1 - e^{-\lambda x} & x \geq 0\\
	0 & x \leq 0
	\end{matrix}\right.
\end{aligned}
\end{equation}

The Exponential distribution has moments

\begin{equation}
\label{eqn:Exponential-Mean}
\expected{X} = \int_{- \infty}^{\infty} xf(x) \partial x = \frac{1}{\lambda}
\end{equation}

\begin{equation}
\label{eqn:Exponential-Moment-2}
\expected{X^{2}} = \int_{- \infty}^{\infty} x^{2}f(x) \partial x = \frac{2}{\lambda^{2}}
\end{equation}

and variance

\begin{equation}
\label{eqn:Exponential-Variance}
	\variance{X} = \expected{X^{2}} - \expected{X}^{2} = \frac{1}{\lambda^{2}}
\end{equation}

The Exponential distribution is so convenient because it is memoryless. In particular, the Exponential distribution is the only continuous memoryless distribution.

\begin{theorem}[Exponential Memoryless]
\label{thm:Exponential-Memoryless}

	A random variable $X \sim Exp(\lambda)$ is memoryless.

	\begin{proof}
		\begin{equation*}
		\probability{X > s+t | X > s}=\frac{X > s+t}{X > s}=\frac{e^{- \lambda (s+t)}}{e^{- \lambda s}}=e^{- \lambda t}=\probability{X > t}
		\end{equation*}
	\end{proof}
\end{theorem}

The Exponential distribution is memoryless and with constant failure rate. It exposes also the following useful properties.

\begin{theorem}[Precedency of Exponentials]
\label{thm:Exponential-Precedency}
	Given independent $X_{1} \sim Exp(\lambda_{1})$ and $X_{2} \sim Exp(\lambda_{2})$, we have that

	\begin{equation}
	\label{eqn:Exponential-Precedency}
	\probability{X_{1} < X_{2}} = \frac{\lambda_{1}}{\lambda_{1} + \lambda_{2}}
	\end{equation}
\end{theorem}

\begin{theorem}[Minimum Exponentials]
\label{thm:Exponential-Minimum}
	Given independent $X_{1} \sim Exp(\lambda_{1})$ and $X_{2} \sim Exp(\lambda_{2})$, we have that

	\begin{equation}
	\label{eqn:Exponential-Minimum}
	\min(X_{1},X_{2}) \sim Exp(\lambda_{1} + \lambda_{2})
	\end{equation}
\end{theorem}


\section{Data Streams (30 points)}

In this problem, we study an approach to approximating the frequency of occurrences of different items in a data stream. Assume $S = \langle a_1, a_2, \ldots, a_t \rangle$ is a data stream of items from the set $\{1, 2, \ldots, n\}$. Assume for any $1\leq i\leq n$, $F[i]$ is the number of times $i$ has appeared in $S$. We would like to have good approximations of the values $F[i]$ ($1\leq i\leq n$) at all times.

A simple way to do this is to just keep the counts for each item $1\leq i\leq n$ separately. However, this will require $\mathcal{O}(n)$ space, and in many applications (e.g., think online advertising and counts of user's clicks on ads) this can be prohibitively large. We see in this problem that it is possible to approximate these counts using a much smaller amount of space. To do so, we consider the algorithm explained below.

\paragraph{Strategy.} The algorithm has two parameters $\delta, \epsilon >0$. It picks $\left\lceil \log\frac{1}{\delta}\right\rceil$ independent hash functions:
\[
\forall j \in \left\llbracket 1; \left\lceil \log\frac{1}{\delta}\right\rceil\right\rrbracket, \quad h_j:\{1,2,\ldots, n\} \rightarrow \{1,2, \ldots, \left\lceil \frac{e}{\epsilon} \right\rceil\},
\]
where $\log$ denotes natural logarithm. Also, it associates a count $c_{j,x}$ to any $1\leq j\leq  \left\lceil \log\frac{1}{\delta}\right\rceil$ and $1\leq x \leq  \left\lceil \frac{e}{\epsilon} \right\rceil$. In the beginning of the stream, all these counts are initialized to $0$. Then, upon arrival of each $a_k$ ($1\leq k \leq t$), each of the counts $c_{j, h_j(a_k)}$ ($1\leq j\leq \left\lceil \log\frac{1}{\delta}\right\rceil$) is incremented by $1$.

For any $1\leq i\leq n$, we define $\tilde{F}[i] = \min_{j} \{c_{j,h_j(i)}\}$. We will show that $\tilde{F}[i]$ provides a good approximation to $F[i]$.

\paragraph{Memory cost.} Note that this algorithm only uses $\mathcal{O}\left(\frac{1}{\epsilon}\log\frac{1}{\delta}\right)$ space.

\paragraph{Properties.} A few important properties of the algorithm presented above:
\begin{itemize}
\item For any $1\leq i\leq n$:
\[
	\tilde{F}[i]\geq F[i].
\]
\item For any $1\leq i\leq n$ and $1\leq j\leq \lceil \log(\frac{1}{\delta})\rceil$:
\[
	\mathsf{E}\left[c_{j,h_j(i)}\right] \leq F[i] + \frac{\epsilon}{e} (t - F[i]).
\]
\end{itemize}

\subquestion{(a) [10 Points]} Prove that:
\[
	\pr{\tilde{F}[i] \leq F[i] + \epsilon t} \geq 1-\delta.
\]
\emph{Hint: Use Markov inequality and the property of independence of hash functions.}

Based on the proof in part (a) and the properties presented earlier, it can be inferred that $\tilde{F}[i]$ is a good approximation of $F[i]$ for any item $i$ such that $F[i]$ is not very small (compared to $t$). In many applications (\emph{e.g.}, when the values $F[i]$ have a heavy-tail distribution), we are indeed only interested in approximating the frequencies for items which are not too infrequent. We next consider one such application.

\subquestion{(b) [20 Points]}

\paragraph{Warning.} This implementation question requires substantial computation time %- Python / Java / C / C++ implementations will be faster.
Python implementation reported to take 15min - 1 hour. Therefore, we advise you to start early.

\paragraph{Dataset.} %\url{http://snap.stanford.edu/class/cs246-data/HW4-q4.zip}
The dataset in \textbf{q4/data} contains the following files:
\begin{enumerate}
\item \texttt{words\_stream.txt} Each line of this file is a number, corresponding to the ID of a word in the stream.
\item \texttt{counts.txt} Each line is a pair of numbers separated by a tab. The first number is an ID of a word and the second number is its associated exact frequency count in the stream.
\item \texttt{words\_stream\_tiny.txt} and \texttt{counts\_tiny.txt} are smaller versions of the dataset above that you can use for debugging your implementation.
\item \texttt{hash\_params.txt} Each line is a pair of numbers separated by a tab, corresponding to parameters $a$ and $b$ which you may use to define your own hash functions (See explanation below).
\end{enumerate}

\paragraph{Instructions.}
Implement the algorithm and run it on the dataset with parameters $\delta = e^{-5}, \epsilon = e\times 10^{-4}$. (Note: with this choice of $\delta$ you will be using 5 hash functions - the 5 pairs $(a,b)$ that you'll need for the hash functions are in \texttt{hash\_params.txt}). Then for each distinct word $i$ in the dataset, compute the relative error $E_r[i] = \frac{\tilde{F}[i] - F[i]}{F[i]}$ and plot these values as a function of the exact word frequency $\frac{F[i]}{t}$. (\textbf{You do not have to implement the algorithm in Spark.})

The plot should use a logarithm scale both for the $x$ and the $y$ axes, and there should be ticks to allow reading the powers of 10 (e.g. $10^{-1}$, $10^0$, $10^1$ etc...). The plot should have a title, as well as the $x$ and $y$ axes. The exact frequencies $F[i]$  should be read from the counts file. Note that words of low frequency can have a very large relative error. That is not a bug in your implementation, but just a consequence of the bound we proved in question (a).

Answer the following question by reading values from your plot: What is an approximate condition on a word frequency in the document to have a relative error below $1 = 10^0$ ?

\paragraph{Hash functions.}
You may use the following hash function (see example pseudo-code), with $p = 123457$, $a$ and $b$ values provided in the hash params file and \texttt{n\_buckets} (which is equivalent to $\left\lceil \frac{e}{\epsilon} \right\rceil$) chosen according to the specification of the algorithm. In the provided file, each line gives you $a$, $b$ values to create one hash function.

\begin{verbatim}
# Returns hash(x) for hash function given by parameters a, b, p and n_buckets
def hash_fun(a, b, p, n_buckets, x)
{
	y = x [modulo] p
	hash_val = (a*y + b) [modulo] p
	return hash_val [modulo] n_buckets
}
\end{verbatim}
Note: This hash function implementation produces outputs of value from $0$ to $(\texttt{n\_buckets}-1)$, which is different from our specification in the \textbf{Strategy} part. You can either keep the range as $\{0, ..., \texttt{n\_buckets}-1\}$, or add 1 to the hash result so the value range becomes $\{1, ..., \texttt{n\_buckets}\}$, as long as you stay consistent within your implementation.

\subsection*{What to submit}
\begin{enumerate}[(i)]
	\item Proof that $\pr{\tilde{F}[i] \leq F[i] + \epsilon t} \geq 1-\delta$. [part (a)]
	\item Log-log plot of the relative error as a function of the frequency. Answer for which word frequencies is the relative error below 1. [part (b)]
	\item Submit the code on Gradescope submission site. [part (b)]
\end{enumerate}


\subsection{The geometric significance of the dot product}

The \textbf{included angle}%
\index{included angle} of two vectors $\vect{u}$ and $\vect{v}$ is the
angle $\theta$ between the vectors such that $0 \leq \theta \leq \pi$.
\begin{center}
  \begin{tikzpicture}
    \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (-30:8mm) arc (-30:45:8mm) -- cycle;
    \draw[->, thick, blue](0,0) -- node[above left] {$\vect{u}$} (45:1.5);
    \draw[->, thick, red](0,0) -- node[below left] {$\vect{v}$} (-30:2);
    \node at (7.5:5mm){$\theta$};
  \end{tikzpicture}
\end{center}
The dot product can be used to determine the included angle between
two vectors.

\begin{proposition}{The dot product and the included angle}{dot-product-angle}
  Let $\vect{u}$ and $\vect{v}$ be two vectors in $\R^n$, and let
  $\theta$ be the included angle. Then the following equation holds.
  \begin{equation*}
    \vect{u}\dotprod \vect{v}=\norm{\vect{u}} \norm{\vect{v}} \cos \theta.
  \end{equation*}
\end{proposition}

In words, the dot product of two vectors equals the product of the
magnitude (or length) of the two vectors multiplied by the cosine of
the included angle. Note that this gives a geometric description of
the dot product that does not depend explicitly on the coordinates of
the vectors.

\begin{example}{Find the angle between two vectors}{angle-two-vectors}
  Find the angle%
  \index{angle!between vectors} between the vectors
\begin{equation*}
  \vect{u}
  =
  \begin{mymatrix}{r}
    2 \\
    2
  \end{mymatrix}
  \quad\mbox{and}\quad
  \vect{v}
  =
  \begin{mymatrix}{r}
    0 \\
    3
  \end{mymatrix}.
\end{equation*}
\end{example}

\begin{solution}
  By Proposition~\ref{prop:dot-product-angle},
  \begin{equation*}
    \vect{u}\dotprod \vect{v}=\norm{\vect{u}} \norm{\vect{v}} \cos \theta.
  \end{equation*}
  Hence,
  \begin{equation*}
    \cos \theta =\frac{\vect{u}\dotprod \vect{v}}{\norm{\vect{u}}\norm{\vect{v}}}.
  \end{equation*}
  First, we compute $\vect{u}\dotprod \vect{v} = (2)(0) + (2)(3) = 6$.
  Then,
  \begin{equation*}
    \begin{array}{l}
      \norm{\vect{u}} = \sqrt{2^2+2^2}=\sqrt{8}, \\
      \norm{\vect{v}} = \sqrt{0^2+3^2}=3.
    \end{array}
  \end{equation*}
  Therefore, we have
  \begin{equation*}
    \cos \theta =\frac{6}{3\sqrt{8}} = \frac{1}{\sqrt{2}}.
  \end{equation*}
  Taking the inverse cosine of both sides of the equation, we find
  that $\theta =\frac{\pi}{4}$ radians, or 45 degrees.
\end{solution}

\begin{example}{Computing a dot product from an angle}{geometric-dot-product}
  Let $\vect{u},\vect{v}$ be vectors with $\norm{\vect{u}} = 3$ and $\norm{\vect{v}} = 4$.
  Suppose the angle between $\vect{u}$ and $\vect{v}$ is $\pi / 3$. Find $\vect{u}\dotprod \vect{v}$.
\end{example}

\begin{solution}
  From Proposition~\ref{prop:dot-product-angle}, we have
  \begin{equation*}
    \vect{u}\dotprod \vect{v}
    =\norm{\vect{u}} \norm{\vect{v}} \cos \theta
    =3\cdot 4\cdot \cos\paren{\frac{\pi}{3}}
    =3\cdot 4\cdot \frac{1}{2}=6.
\end{equation*}
\end{solution}


\section{Computer Math}

In this section, I introduce a few vocabulary terms that help you understand how computers do certain things, like add two (binary) numbers. In most respects, adding these numbers is no different from how you already add numbers. When the numbers in any column is too big to be held in that column, you must ``carry'' the extra number to the next larger column. The difference, and it is slightly awkward at first, is remembering that any column can only hold 0 or 1 -- if you reach 2, you have to carry the ``two'' to the next column.

\subsection*{Addition}

Adding two values is straightforward. Simply align the values on the least significant bit and add each column, moving any ``carry'' to the bit one position left, just like you already do with ``regular'' (base-10) arithmetic.

\begin{verbatim}
  Binary:     Decimal:

  0001 0110   (16 + 0 + 4 + 2 + 0) =   22
+ 0000 0011    (0 + 0 + 0 + 1 + 2) =    3
===========                          ====
  0001 1001   (16 + 8 + 0 + 0 + 1) =   25
\end{verbatim}

\bigskip

\noindent Your turn! Add these numbers in binary, column by column, carrying each ``2'' as needed. After you add them, convert each number to decimal to check your work.

\bigskip

\begin{tabular}{p{3in} | c  p{3in} }
\hline
\\
\begin{minipage}{2.95in}
\begin{verbatim}
   Binary:       Decimal:

  0000 0000
+ 0000 0011
===========      ====

___________      ____
\end{verbatim}
\end{minipage}

&&

\begin{minipage}{2.95in}
\begin{verbatim}
   Binary:       Decimal:

  0000 0010
+ 0000 0010
===========      ====

___________      ____
\end{verbatim}
\end{minipage}

\\
\hline
\\

\begin{minipage}{2.95in}
\begin{verbatim}
   Binary:       Decimal:

  0000 0110
+ 0000 0011
===========      ====

___________      ____
\end{verbatim}
\end{minipage}

&&

\begin{minipage}{2.95in}
\begin{verbatim}
   Binary:       Decimal:

  0000 1010
+ 0001 1011
===========      ====

___________      ____
\end{verbatim}
\end{minipage}

\\
\hline
\end{tabular}

There is more information about binary arithmetic in the Appendix -- you can see how to represent negative numbers, and how to subtract numbers, in binary, just the way a computer does. But for now, you know enough to move on.

\vfill

\input{./include/twoscomplement.tex}


\subsection{Probability density functions}

\subsubsection{Definition}

If continuous, probability at any point is \(0\). We instead look at probability density.

Derived from cumulative distribution function:

\(F_X(x)=\int_{-\infty}^x f_X(u)du\)

The density function is \(f_X(x)\).

\subsubsection{Conditional probability distributions}

For probability mass functions:

\(P(Y=y|X=x)=\dfrac{P(Y=y\land X=x)}{P(X=x)}\)

For probability density functions:

\(f_Y(y|X=x)=\dfrac{f_{X,Y}(x,y)}{f_X(x)}\)


\section{Power Series}
\subsection{Geometric Series}
First, we need to define what we mean by an infinite series.
\begin{definition}
	An infinite series is of the form
	\begin{equation*}
		a_1 + a_2 + \ldots + a_n + \ldots \text{ or equivalently, } \sum_{k=1}^{\infty}{a_k}.
	\end{equation*}
	Just like with finite series, each $a_i$ is a term, and $a_n$ is the nth term.
\end{definition}

We can describe the behavior of an infinite series by looking at how its value behaves after summing a finite number of terms.
We can define what it means for an infinite sum to have a value by looking at the limit of the partial sums as $n$ grows large.
\begin{definition}
	The nth partial sum of an infinite series is
	\begin{equation*}
		s_n = \sum_{k=1}^{n}{a_k}.
	\end{equation*}
	The infinite series converges to value $L$ if
	\begin{equation*}
		\lim_{n\to\infty}{s_n} = L.
	\end{equation*}
	Otherwise, the series diverges and does not have a value.
\end{definition}

\begin{example}
	State if the following infinite series converges or diverges.
	\begin{equation*}
		\frac{3}{10} + \frac{3}{100} + \ldots + \frac{3}{10^n} + \ldots.
	\end{equation*}
\end{example}
\begin{answer}
	Looking at the partial sums,
	\begin{align*}
		s_1 &= 0.3 \\
		s_2 &= 0.33 \\
		&\vdots \\
		s_n &= 0.\underbrace{33333\ldots}_{\text{$n$ total 3's}}
	\end{align*}

	So, it seems the limit of the partial sums tends towards a decimal with an infinite number of 3's.
	This value corresponds to the decimal expansion of 1/3, which clearly is real and finite, so the series converges.
\end{answer}

The above series is a geometric series since each subsequent term is 10 times smaller than the previous one (i.e $r=1/10$).
\begin{lemma}
	The geometric series
	\begin{equation*}
		\sum_{k=0}^{\infty}{a_0(r)^k}
	\end{equation*}
	converges to a value of $a_0/(1-r)$ if $\abs{r} < 1$ and diverges otherwise.
\end{lemma}
\begin{proof}
	We'll first find a formula for the partial sums and then find the limit of the partial sums for $1 < r < 1$.
	\begin{align*}
		s_n &= \sum_{k=0}^{n}{a_0(r)^k} \\
		&= a_0 + a_0r + a_0r^2 + \ldots + a_0r^n \\
		rs_n &= a_0r + a_0r^2 + a_0r^3 + \ldots + a_0r^n + a_0r^{n+1} \\
		&= -a_0 + s_n + a_0r^{n+1} \\
		s_n(r-1) &= a_0\left(r^{n+1} - 1\right) \\
		s_n &= a_0\frac{r^{n+1}-1}{r-1}.
	\end{align*}
	This formula for partial sums holds for all values of $r$.
	Now we'll take the limit of $s_n$ and see for what values of $r$ the limit exists.
	\begin{align*}
		\sum_{k=0}^{\infty}{a_0(r)^n} &= \lim_{n\to\infty}{s_n} \\
		&= \lim_{n\to\infty}{a_0\frac{r^{n+1}-1}{r-1}} \\
		&= \frac{-a_0}{r-1}, \abs{r} < 1 \\
		&= \frac{a_0}{1-r}, \abs{r} < 1.
	\end{align*}
\end{proof}

\subsection{Functions from Geometric Series}
What happens if rather than letting $r$ be some fixed value we know beforehand, we let $r$ be some variable $x$?
Applying the formula,
\begin{equation*}
	a_0 + a_0x + a_0x^2 + \ldots = \frac{a_0}{1-x}, \abs{x} < 1.
\end{equation*}
This sort of sum of powers of $x$ is called a power series, and the condition that $\abs{x}<1$ is called the interval of convergence.
Right now, this power series is centered at $x=0$, but we can generalize it a bit to be centered at $x=h$.
\begin{equation*}
	a_0 + a_0(x-h) + a_0(x-h)^2 + \ldots = \frac{a_0}{1-(x-h)}, \abs{x-h} < 1.
\end{equation*}
Note that this formula allows us to find the power series of any function $a_0/(mx+b)$.
\begin{equation*}
	\frac{a_0}{mx + b} = \frac{a_0}{1-(1-mx-b)} = a_0 + a_0(1-mx+b) + a_0(1-mx+b)^2 + \ldots, \abs{1-mx-b} < 1.
\end{equation*}
So,
\begin{align*}
	\frac{1}{x} &= 1 + (1-x) + (1-x)^2 + \ldots, \abs{1-x} < 1 \\
	\frac{1}{1-x} &= 1 + x + x^2 + x^3 + \ldots, \abs{x} < 1 \\
	\frac{1}{1+x} &= 1 - x + x^2 - x^3 + \ldots, \abs{x} < 1.
\end{align*}

\subsubsection{Term-By-Term Differentiation}
\begin{theorem}
	If the power series
	\begin{equation*}
		f(x) = \sum_{k=0}^{\infty}{c_k(x-a)^k} = c_0 + c_1(x-a) + c_2(x-a)^2 + \ldots
	\end{equation*}
	converges for $\abs{x-a} < R$, including $R=\infty$, then the power series
	\begin{equation*}
		\sum_{k=1}^{\infty}{kc_k(x-a)^{k-1}} = c_1 + 2c_2(x-a) + 3c_3(x-a)^2 + \ldots
	\end{equation*}
	also converges for $\abs{x-a} < R$ and is equal to $f^\prime(x)$ on that interval.
\end{theorem}

Applying the theorem,
\begin{align*}
	\frac{-1}{x^2} &= -1 - 2(1-x) - 3(1-x)^2 - \ldots, \abs{1-x} < 1 \\
	\frac{1}{(1-x)^2} &= 1 + 2x + 3x^2 + \ldots, \abs{x} < 1 \\
	\frac{-1}{(1+x)^2} &= -1 + 2x - 3x^2 + \ldots, \abs{x} < 1.
\end{align*}

\subsubsection{Term-By-Term Integration}
\begin{theorem}
	If the power series
	\begin{equation*}
		f(x) = \sum_{k=0}^{\infty}{c_k(x-a)^k} = c_0 + c_1(x-a) + c_2(x-a)^2 + \ldots
	\end{equation*}
	converges for $\abs{x-a} < R$, including $R=\infty$, then the power series
	\begin{equation*}
		\sum_{k=0}^{\infty}{c_k\frac{(x-a)^{k+1}}{k+1}} = c_0(x-a) + c_1\frac{(x-a)^2}{2} + c_2\frac{(x-a)^3}{3} + \ldots
	\end{equation*}
	also converges for $\abs{x-a} < R$ and represents the antiderivative of $f$ on that interval.
\end{theorem}

Applying the theorem,
\begin{align*}
	\ln{\abs{x}} &= 1 + \frac{(1-x)^2}{2} + \frac{(1-x)^3}{3} + \ldots, \abs{1-x} < 1 \\
	-\ln{\abs{1-x}} &= x + \frac{x^2}{2} + \frac{x^3}{3} + \ldots, \abs{x} < 1 \\
	\ln{\abs{1+x}} &= x - \frac{x^2}{2} + \frac{x^3}{3} - \frac{x^4}{4} + \ldots, \abs{x} < 1.
\end{align*}

This is pretty impressive: we now have an formula for $\ln{x}$ in terms of polynomials.
This also includes some pretty surprising identities.
For example\footnote{Technically, we're substituting $x=1$ which isn't in the interval of convergence. However, since we are dealing with an alternating sum, the series will also converge for $x=-1$ and $x=1$.},
\begin{equation*}
	\ln{2} = 1 - \frac{1}{2} + \frac{1}{3} - \ldots + \frac{(-1)^{n+1}}{n} + \ldots.
\end{equation*}

\begin{example}
	Find a power series for $\arctan{x}$ and state the interval of convergence.
\end{example}
\begin{answer}
	We know that
	\begin{equation*}
		\dd{}{x}\arctan{x} = \frac{1}{1+x^2}.
	\end{equation*}

	So, if we can find a power series that represents $1/(1+x^2)$, we can integrate term-by-term to get a power series for $\arctan{x}$.
	We also already know the following power series.
	\begin{equation*}
		\frac{1}{1+u} = 1 - u + u^2 - u^3 + \ldots, \abs{u} < 1.
	\end{equation*}

	Letting $u=x^2$,
	\begin{align*}
		\frac{1}{1+x^2} &= 1 - x^2 + x^4 - x^6 + \ldots, \abs{x^2} < 1 \\
		&= 1 - x^2 + x^4 - x^6 + \ldots, \abs{x} < 1.
	\end{align*}

	Integrating term-by-term,
	\begin{align*}
		\arctan{x} = x - \frac{x^3}{3} + \frac{x^5}{5} + \ldots + (-1)^n\frac{x^{2n+1}}{2n+1} + \ldots, \abs{x} < 1.
	\end{align*}

	This power series also gives rise to a pretty interesting identity\footnote{See footnote 1.}.
	\begin{equation*}
		\arctan{1} = \frac{\pi}{4} = 1 - \frac{1}{3} + \frac{1}{5} - \frac{1}{7} + \ldots + (-1)^n\frac{x^{2n+1}}{2n+1} \ldots.
	\end{equation*}
\end{answer}


\subsection{Integration}
\noindent
The volume element in cylindrical coordinates is $\mathrm{d}V = r\mathrm{d}r\mathrm{d}\theta\mathrm{d}z$.

[INSERT IMAGE]

\noindent
For example, let's evaluate $\int_{0}^{4}{\int_{0}^{\sqrt{16 - y^2}}{\int_{0}^{16 - x^2 - y^2}{\mathrm{d}z}\mathrm{d}x}\mathrm{d}y}$, the volume under the paraboloid $z = 16 - x^2 - y^2$ using cylindrical coordinates.
\begin{align*}
	&= \int_{0}^{4}{\int_{0}^{\pi/2}{\int_{0}^{16 - r^2}{r\mathrm{d}z}\mathrm{d}\theta}\mathrm{d}r} \\
	&= \int_{0}^{4}{\int_{0}^{\pi/2}{16r - r^2\mathrm{d}\theta}\mathrm{d}r} \\
	&= \frac{\pi}{2}\int_{0}^{4}{16r - r^3\mathrm{d}r} \\
	&= \frac{\pi}{2}\left[8r^2 - \frac{r^4}{4}\right]_0^4 \\
	&= 32\pi
\end{align*}

\noindent
For another example, let's find the average value of $f(x,y,z) = z$ on the domain $\Omega$ which is bounded by $z = \sqrt{6-x^2-y^2}$ and $z = x^2+y^2$ \footnote{Work omitted for brevity}.
\begin{equation*}
	\bar{f} = \frac{\int_{0}^{2\pi}{\int_{0}^{\sqrt{2}}{\int_{r^2}^{\sqrt{6 - r^2}}{zr\mathrm{d}z}\mathrm{d}r}\mathrm{d\theta}}}{\int_{0}^{2\pi}{\int_{0}^{\sqrt{2}}{\int_{r^2}^{\sqrt{6 - r^2}}{r\mathrm{d}z}\mathrm{d}r}\mathrm{d}\theta}}=\frac{11}{12\sqrt{6} - 17}
\end{equation*}


\subsection{Eigenvalues of Markov matrices}

The following is an important proposition.

\begin{proposition}{Eigenvalues of a migration matrix}{eigenvalue-migration-matrix}
Let $A=\mat{a_{ij}} $ be a migration matrix. Then $1$ is always an
eigenvalue for $A$.
\end{proposition}

\begin{proof} Remember that the determinant of a matrix always equals that of its transpose.
Therefore,
\begin{equation*}
\det (\eigenvar I - A) =\det ((\eigenvar I - A)
^{T}) =\det (\eigenvar I - A^T)
\end{equation*}
because $I^{T}=I$. Thus the characteristic equation for $A$ is the same as
the characteristic equation for $A^{T}$. Consequently, $A$ and $A^{T}$ have the same
eigenvalues. We will show that $1$ is an eigenvalue for $A^{T}$ and then it
will follow that $1$ is an eigenvalue for $A$.

Remember that for a migration matrix, $\sum_{i}a_{ij}=1$. Therefore, if
$A^{T}=\mat{b_{ij}} $ with $b_{ij}=a_{ji}$, it follows that
\begin{equation*}
\sum_{j}b_{ij}=\sum_{j}a_{ji}=1
\end{equation*}

Therefore, from matrix multiplication,
\begin{equation*}
A^{T}\begin{mymatrix}{r}
1 \\
\vdots \\
1
\end{mymatrix} =\begin{mymatrix}{c}
\sum_{j}b_{ij} \\
\vdots \\
\sum_{j}b_{ij}
\end{mymatrix} =\begin{mymatrix}{r}
1 \\
\vdots \\
1
\end{mymatrix}
\end{equation*}

Notice that this shows that $\begin{mymatrix}{r}
1 \\
\vdots \\
1
\end{mymatrix} $ is an eigenvector for $A^{T}$ corresponding to the eigenvalue, $\lambda =1$.
 As explained above, this shows that $\lambda =1$ is an
eigenvalue for $A$ because $A$ and $A^{T}$ have the same eigenvalues.
\end{proof}