\lab{QR Decomposition}{QR decomposition}
\label{lab:QRdecomp}
\objective{Use the Gram-Schmidt algorithm and orthonormal transformations to perform the QR decomposition.}

The QR decomposition of a matrix $A$ is a factorization $A=QR$, where $Q$ has orthonormal columns and $R$ is upper triangular.
This decomposition is useful for computing least squares and finding eigenvalues.
As stated in the following theorem, the QR decomposition of $A$ always exists when the rank of $A$ equals the number of columns of $A$.
\begin{theorem}
Let $A$ be an $m\times n$ matrix of rank $n$.  Then $A$ can be
factored into a product $Q R$, where $Q$ is an $m\times n$ matrix
with orthonormal columns and $R$ is a nonsingular $n \times n$ upper
triangular matrix.
\end{theorem}

In this lab we will only discuss real matrices.
All of these results can be extended to the complex numbers by replacing ``orthogonal matrix'' with ``unitary matrix,'' ``transpose'' with ``hermitian conjugate,'' and ``symmetric matrix'' with ``hermitian matrix.''

\section*{Modified Gram-Schmidt}
Let $A$ be an $m \times n$ matrix of rank $n$.
There are many methods for computing the QR decomposition.
First we will discuss the algorithm that computes $Q$ by applying Gram-Schmidt to the columns of $A$.

Let $\{\x_i\}_{i=1}^n$ be the columns of $A$ (the rank hypothesis implies that these are linearly independent vectors).
Then the Gram-Schmidt algorithm computes an orthonormal basis $\{\q_i\}_{i=1}^n$ for the span of the $\x_i$.
The Gram-Schmidt algorithm defines  \[ \q_1 = \frac{\x_1}{\norm{\x_1}}.\]
It recursively defines $\q_k$ for $k>1$ by
\[
\q_{k} = \frac{\x_k - \p_{k-1}}{\|\x_k - \p_{k-1}\|}, \,\,\,\, k=2,\ldots,n,
\]
where
\[
\p_{k-1} = \sum_{i=1}^{k-1} \langle \q_i, \x_k\rangle \q_i, \,\,\,\, k=2, \ldots, n
\]
and $\p_0 = 0$.

In the QR factorization of $A$, the matrix $Q$ has columns $\{\q_i\}_{i=1}^n$.
The matrix $R$ is then the upper triangular matrix with entries $r_{jk}$ where $r_{kk} = \|\x_k-\p_{k-1}\|$ and $r_{j k} = \langle \q_j, \x_k\rangle$ when $j < k$.
See [TODO: ref textbook] for a proof that $QR=A$ and $R$ is nonsingular.

When implemented with a computer, the Gram-Schmidt algorithm may produce a matrix $Q$ with columns that are not even close to orthonormal due to rounding errors.
We now introduce the modified Gram-Schmidt algorithm, which consistently produces matrices $Q$ whose columns are ``very close'' to orthonormal.

In the modified Gram-Schmidt algorithm, $\q_1$ is the normalization of $\x_1$ as before.
We then make each of the vectors $\x_2, \ldots, \x_n$ orthogonal to $\q_1$ by defining
\[
\x_k := \x_k - \langle \q_1,\x_{k} \rangle \q_1,\quad k=2,\ldots,n.
\]
(Compare this to the usual Gram-Schmidt algorithm, where we only made $\x_2$ orthogonal to $\q_1$.)
Next we define $\q_2 = \frac{\x_2}{\|\x_2\|}.$ Once again we make $\x_3, \ldots, \x_n$ orthogonal to $\q_2$ by defining
\[
\x_k := \x_k - \langle \q_2,\x_{k} \rangle \q_2,\quad k=3,\ldots,n.
\]
(Each of these new vectors is a linear combination of vectors orthogonal to $\q_1$, and hence is orthogonal to $\q_1$ as well.)
We continue this process until we have an orthonormal set $q_1, \ldots, q_k$.
The entire modified Gram-Schmidt algorithm is described in Algorithm \ref{Alg:gram_schmidt}.

%TODO: proofread this, explain notation, and make the LU decomposition alg. like this
\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Modified Gram-Schmidt}{$A$}
\State $m, n \gets \text{shape} \left( A \right)$
\State $Q \gets \text{copy} \left( A \right)$
\State $R \gets \text{zeros}((n,n))$
\For{$0 \leq i < n$}
    \State $R_{i,i} \gets \norm{Q_{:,i}}$
    \State $Q_{:,i} \gets Q_{:,i}/R_{i,i}$
    \For{$i+1 \leq j < n$}
        \State $R_{i,j} \gets Q_{:,j}^\mathsf{T}Q_{:,i}$
        \State $Q_{:,j} \gets Q_{:,j}-R_{i,j}Q_{:,i}$
	\EndFor
\EndFor
\State \pseudoli{return} $Q, R$
\EndProcedure
\end{algorithmic}
\caption{The modified Gram-Schmidt. This algorithm returns orthogonal $Q$ and upper triangular $R$ such that $A = QR$.}
\label{Alg:gram_schmidt}
\end{algorithm}

%TODO: put this section after the next? What algorithm does SciPy use?
\section*{QR decomposition in SciPy}
The linear algebra library in SciPy calculates the QR decomposition using a software package called LAPACK (Linear Algebra PACKage), which is incredibly efficient.
Here is an example of using SciPy to compute the QR decomposition of a matrix.

\begin{lstlisting}
>>> import numpy as np
>>> from scipy import linalg as la

>>> A = np.random.rand(4,3)
>>> Q, R = la.qr(A)
>>> Q.dot(R) == A
array([[ True, False, False],
       [ True, False, False],
       [ True,  True, False],
       [ True,  True, False]], dtype=bool)
\end{lstlisting}
 Note that \li{Q.dot(R)} does not equal \li{A} exactly because of rounding errors.
 However, we can check that the entries of \li{Q.dot(R)} are ``close'' to the entries of \li{A} with the NumPy method \li{np.allclose()}.
 This method checks that the elements of two arrays differ by less than a given tolerance, a tolerance specified by two keyword arguments \li{rtol} and \li{atol} that default to $10^{-5}$ and $10^{-8}$ respectively.
 You can read the documentation to learn how the tolerance is computed from these numbers.
\begin{lstlisting}
>>> np.allclose(Q.dot(R), A)
True
\end{lstlisting}
We can use the same method to check that \li{Q} is ``very close'' to an orthogonal matrix.
\begin{lstlisting}
>>> np.allclose(Q.T.dot(Q), np.eye(4))
True
\end{lstlisting}

\begin{problem}
\label{prob:QR}
Write a function that accepts as input a $m \times n$ matrix of rank $n$ and computes its QR decomposition, returning the matrices $Q$ and $R$.
Your function should use Algorithm \ref{Alg:gram_schmidt}.
Hint: Read about the function \li{np.inner()}.
Another hint: Check that your function works by using \li{np.allclose()}.
\end{problem}

\begin{problem}
Write a function that accepts a square matrix $A$ of full rank and returns $\abs{\det(A)}$.
Use the QR decomposition of $A$ to perform this calculation.
Hint: What is the determinant of an orthonormal matrix?
\end{problem}

\section*{Householder triangularization}
Another way to compute the QR decomposition of a matrix is with a series of orthonormal transformations.
Like the Modified Gram-Schmidt algorithm, orthonormal transformations are numerically stable, meaning that they are less susceptible to rounding errors.

\subsection*{Householder transformations}
The \emph{hyperplane} in $\mathbb{R}^n$ with normal vector $\mathbf{v}$ is the set $\{ \x \in \mathbb{R}^n \mid \langle \x, \mathbf{v} \rangle = 0 \}$.
Equivalently, the hyperplane defined by $\mathbf{v}$ is just the orthogonal complement $\mathbf{v}^{\perp}$.
See Figure \ref{fig:Householder_reflector}.

A \emph{Householder transformation} of $\mathbb{R}^n$ is a linear transformation that reflects about a hyperplane.
If a hyperplane $H$ has normal vector $\mathbf{v}$, let $\mathbf{u} = \mathbf{v}/\|\mathbf{v}\|$.
Then the Householder transformation that reflects about $H$ corresponds to the matrix $H_{\mathbf{u}} = I - 2 \mathbf{u}\mathbf{u}^T$.
You can check that $(I - 2 \mathbf{u}\mathbf{u}^T)^T(I - 2 \mathbf{u}\mathbf{u}^T)=I$, so Householder transformations are orthogonal.

%\newcommand{\ipt}[2]{\ensuremath{\left\langle #1,#2 \right\rangle}}
\begin{figure}
\begin{center}
\begin{tikzpicture}
%\draw[-, thick](0,-1.5)--(0,3); %x-axis
%\draw[-,thick](-1,0)--(4,0); %y-axis
%\node[draw, thick, minimum height=4.5cm, minimum width=5cm]()at(1.5,.75){};
\draw[-,dashed, gray](-2,-1.333)--(3, 2); % hyperplane
\draw[->, gray, >=stealth,ultra thick](0,0)--(.8,-1.2); % v
\draw[->, >=stealth, thick](0,0)--(2.815, .777); % H x
\draw[->, >=stealth, thick](0,0)--(1.8,2.3); % x
\node[draw=none](v)at(.65,-.6){$\v$};
\node[draw=none](x)at(.75,1.5){$\x$};
\node[draw=none](Hx)at(3, .5){$H_{\v}\x$};
\node[draw=none](H)at(-1.5,-.6){$H$};
%\node[draw=none](bullet)at(2.31, 1.53){\textbullet};
%\node[draw=none](vandx)at(4.0,1.5){$\x -
%\left \langle \dfrac{\v}{\|\v\|}, \x \right \rangle \dfrac{\v}{\|\v\|}$};
\end{tikzpicture}
\end{center}
\caption{This is a picture of a Householder transformation.
The normal vector $\mathbf{v}$ defines the hyperplane $H$.
The Householder transformation $H_{\mathbf{v}}$ of $\mathbf{x}$ is just the reflection of $\mathbf{x}$ across $H$.}
\label{fig:Householder_reflector}
\end{figure}

\subsection*{Householder triangularization}
The QR decomposition of an $m \times n$ matrix $A$ can also be computed with Householder transformations via the Householder triangularization.
Whereas Gram-Schmidt makes $A$ \emph{orthonormal} using a series of transformations stored in an \emph{upper triangular} matrix, Householder triangularization makes $A$ \emph{triangular} by a series of \emph{orthonormal} transformations.
More precisely, the Householder triangularization finds an $m \times m$ orthogonal matrix $Q$ and and $m \times n$ upper triangular matrix $R$ such that $QA = R$.
Since $Q$ is orthogonal, $Q^{-1}=Q^T$ so $A = QR$.
(Contrast this with the QR decomposition where $A = QR$, the matrix $Q$ is $m \times n$, and $R$ is $n \times n$).
If $A$ is square, then $A = Q^TR$ is the QR decomposition.

Let's demonstrate the idea behind Householder triangularization on a $4 \times 3$ matrix $A$.
Let $e_1, \ldots, e_4$ be the standard basis of $\mathbb{R}^4$.
First we find an orthonormal transformation $Q_1$ that maps the first column of A into the span of $e_1$.
This is diagrammed below, where $*$ represents an arbitrary entry.

\def\mc#1{\multicolumn{1}{c|}{#1}}
\def\lc#1{\multicolumn{1}{|c}{#1}}
\begin{equation*}
\begin{pmatrix}
* & * & * \\
* & * & * \\
* & * & * \\
* & * & *
\end{pmatrix}
\underrightarrow{Q_1}
\begin{pmatrix}

* & * & * & \\ \cline{2-3}
\mc{0} & * & \mc{*}& \\
\mc{0} & * & \mc{*} & \\
\mc{0}& * & \mc{*} & \\ \cline{2-3}
\end{pmatrix}
\end{equation*}
Let $A_2 = Q_1A$ be the matrix on the right above.
Now find an orthonormal transformation $Q_2$ that fixes $e_1$ and maps the second column of $A_2$ into the span of $e_1$ and $e_2$.
Notice that since $Q_2$ fixes $e_1$, the top row of $A_2$ will be fixed by $Q_2$, and only entries in the boxed submatrix will change.

%\def\mc#1{\multicolumn{1}{c|}{#1}}
%\begin{equation*}
%\begin{pmatrix}
%* & * & * & \\ \cline{2-3}
%\mc{0} & * & \mc{*}& \\
%\mc{0} & * & \mc{*} & \\
%\mc{0}& * & \mc{*} & \\ \cline{2-3}
%\end{pmatrix}
%
%\underrightarrow{Q_1}
%
%\begin{pmatrix}
%* & * & * & \\ \cline{2-3}
%\mc{0} & 0 & \mc{*}& \\
%\mc{0} & 0 & \mc{*} & \\
%\mc{0}& 0 & \mc{*} & \\ \cline{2-3}
%\end{pmatrix}
%\end{equation*}

Let $A_3 = Q_2Q_1A$ be the matrix on the right above.
Finally, find an orthonormal transformation $Q_3$ that fixes $e_1$ and $e_2$ and maps the third column of $A_3$ into the span of $e_1$, $e_2$, and $e_3$.
The diagram below summarizes this process, where boxed entries indicate those affected by the operation just performed.

\begin{equation*}
Q_3 Q_2 Q_1
\begin{pmatrix}
* & * & * \\
* & * & * \\
* & * & * \\
* & * & *
\end{pmatrix}
= Q_3 Q_2
\begin{pmatrix}  \cline{2-4}
&\lc{*} & * & \mc{*} & \\
&\lc{0} & * & \mc{*}& \\
&\lc{0} & * & \mc{*} & \\
&\lc{0}& * & \mc{*} & \\ \cline{2-4}
\end{pmatrix}
= Q_3
\begin{pmatrix}
* & * & * \\ \cline{2-3}
\mc{0} & * & \mc{*} \\
\mc{0} & 0 & \mc{*} \\
\mc{0} & 0 & \mc{*} & \\ \cline{2-3}
\end{pmatrix}
=
\begin{pmatrix}
* & * & * \\
0 & * & * \\ \cline{3-3}
0 & \mc{0} & \mc{*} \\
0 & \mc{0} & \mc{0} & \\ \cline{3-3}
\end{pmatrix}.
\end{equation*}

We've accomplished our goal, which was to triangularize $A$ using orthonormal transformations.
But how do we construct the matrices $Q_k$?

It turns out that we can choose each $Q_k$ to be a Householder transformation.
Suppose we have found Householder transformations $Q_1, \ldots, Q_{k-1}$ such that $Q_{k-1}\ldots Q_2Q_1 = A_k$ where
\[
A_k = \begin{pmatrix}
T & X' \\
0 & X''
\end{pmatrix}.
\]
Here, $T$ is a $(k-1) \times (k-1)$ upper triangular matrix.
Let $\x$ be the $k^{th}$ column of $A_k$.
Write $\x = \x' + \x''$ where $\x'$ is in the span of $e_1, \ldots, e_{k-1}$.
So $\x''$ looks like $k-1$ zeros followed by the first column of $X''$.
The idea is to reflect $\x''$ about a hyperplane into the span of $e_k$.
It turns out that there are two choices that will work (see Figure \ref{fig:two reflectors}).
These hyperplanes have normal vectors $\x'' + \| \x'' \|e_k$ and $\x'' - \| \x'' \|e_k$.
In fact, the more numerically stable transformation is to reflect about the hyperplane with normal vector $\mathbf{v}_k = \x'' +\sign(x_k) \| \x'' \|e_k$, where $x_k$ is the $k^{th}$ entry of $\x''$ (or the top left entry of $X''$).
(You can check that $\mathbf{v}_k$ is orthonormal to $e_1, \ldots, e_{k-1}$, so the plane orthonormal to $\mathbf{v}_k$ contains $e_1, \ldots, e_{k-1}$, and reflecting about it fixes $e_1, \ldots, e_{k-1}$.)
Thus, $Q_k$ is the Householder transformation $H_{\mathbf{v}_k}$.

The final question is to find an efficient algorithm for computing $Q = Q_nQ_{n-1} \ldots Q_1$ and $R = Q_nQ_{n-1} \ldots Q_1A$.
The idea is to start with $R=A$ and $Q = I$. Then we compute $Q_1$ and modify $R$ to be $Q_1A$ and $Q$ to be $Q_1$.
Next we compute $Q_2$ and modify $R$ to be $Q_2Q_1A$ and $Q$ to be $Q_2Q_1$, and so forth.
As we have already discussed, $Q_k$ fixes the first $k-1$ rows and columns of any matrix it acts on.
In fact, if $\x'' = (0, \ldots, 0, x_k, x_{k+1}, \ldots, x_n)$ as above, then $\mathbf{v}_k = (0, \ldots, 0, v_{k_0}, x_{k+1}, \ldots, x_n)$ where $v_{k_0} = x_k + \sign(x_k) \| \x'' \|$.
If $\mathbf{u}_k$ is the normalization of $(v_{k_0}, x_{k+1}, \ldots, x_n) \in \mathbb{R}^{n-(k-1)}$, then
\[
Q_k = I-\frac{2\x''(\x'')^T}{\|\x''\|^2} =  \begin{pmatrix}
I & 0 \\
0 & I-2\mathbf{u}_k\mathbf{u}_k^T
\end{pmatrix}.
\]
This means that, using block multiplication,
\[
Q_kA_k =  \begin{pmatrix}
I & 0 \\
0 & I-2\mathbf{u}_k\mathbf{u}_k^T
\end{pmatrix}\begin{pmatrix}
T & X' \\
0 & X''
\end{pmatrix} = \begin{pmatrix}
T & X' \\
0 & ( I-2\mathbf{u}_k\mathbf{u}_k^T)X''
\end{pmatrix}.
\]
So at each stage of the algorithm, we only need to update the entries in the bottom right submatrix of $A_k$, and these change via matrix multiplication by $ I-2\mathbf{u}_k\mathbf{u}_k^T$. Similarly,
\[
Q_kQ_{k-1}\ldots Q_1 = Q_k \begin{pmatrix}
A\\
B
\end{pmatrix} = \begin{pmatrix}
I & 0 \\
0 & I-2\mathbf{u}_k\mathbf{u}_k^T
\end{pmatrix}\begin{pmatrix}
A\\
B
\end{pmatrix} = \begin{pmatrix}
A\\
(I-2\mathbf{u}_k\mathbf{u}_k^T)B
\end{pmatrix},
\]
so to update $\prod Q_i$, we need only modify the bottom rows.
These also change via matrix multiplication by $I-2\mathbf{u}_k\mathbf{u}_k^T$.

These arguments produce Algorithm \ref{Alg:Householder}.

%To find $Q_1$, we first identify an appropriate hyperplane to reflect $x$ into the span of $e_1$.
%It turns out there are two hyperplanes that will work, as shown in figure \ref{fig:two reflectors}.
%(In the complex case, there are infinitely many such hyperplanes.)
%Between the two, the one that reflects $x$ further will be more numerically stable.
%This is the hyperplane perpendicular to $v = sign(x_1)\norm{x}_2 e_1 + x$.
%
%To see how this works, let $x$ be the first column of the submatrix that we want to project onto the span of $e_1$.
%In order for this to be a unitary operation, this will need to preserve the norm of $x$.
%This means that $\left( I - 2 v v^\mathsf{H} \right) x = \pm \norm{x} e_1$, or, in other words,
%
%\[ 2 v v^\mathsf{H} x =
%\begin{pmatrix}
%x_1 \pm \norm{x} \\
%x_2 \\
%x_3 \\
%\vdots \\
%x_n
%\end{pmatrix}\]
%
%Let $u$ be the vector on the right hand side of this expression.
%It can be shown that the vector  $\frac{u}{\norm{u}}$ is the proper choice for $v$.
%We will show that the vector $\frac{u}{\norm{u}}$ is the proper choice for $v$.
%Notice that:
%
%\[\norm{u}^2 = \norm{x}^2 \pm 2 \norm{x} x_1 + x_1^2 + x_2 + \dots + x_n^2 = 2 \norm{x}^2 \pm 2 \norm{x} x_1 \]
%
%and that
%
%\[\norm{x}^2 \pm \norm{x} x_1 = u^\mathsf{H} x \]
%
%So we have
%
%\begin{align*}
%2 v v^\mathsf{H} x &= 2 u \frac{\norm{x}^2 \pm x_1 \norm{x}}{\norm{u}^2} \\
%		&= 2 u \frac{u^\mathsf{H} x}{\norm{u}^2} \\
%		&= 2 \frac{u}{\norm{u}} \left( \frac{u}{\norm{u}} \right)^\mathsf{H} x
%\end{align*}
%
%So $\frac{u}{\norm{u}}$ is a proper choice of $v$ that will project $x$ into the span of $e_1$.

\begin{figure}
\begin{tikzpicture}

\draw[-, dashed, gray](-3,-1)--(3,1);
\draw[-, dashed, gray](-.8,2.4)--(.6,-1.8);
\draw[-, gray, thick](-4,0)--(4,0);
\draw[->, thick, >=stealth'](0,0)--(2,1.6);
\draw[<->, thick, >=stealth'](-2.6,0)--(2.6,0);
\draw[->, gray,  ultra thick, >=stealth](0,0)--(1.5,.5);
\draw[->, gray, ultra thick, >=stealth](0,0)--(-.5,1.5);

\node[draw=none, node distance=3.5cm]
	(dummy)at(2.5,.2){};
\node[draw=none, node distance=.5cm](Hvx)
	[below of=dummy]{$H_{\v_1}\x$};
\node[draw=none, node distance=2cm](x)
	[above left of=Hvx]{$\x$};
\node[draw=none](v1)[left of=x]{$\v_1$};
\node[draw=none, node distance=.55cm](v2)[below of=x]{$\v_2$};
\node[draw=none, node distance=4cm](hvx2)[left of=dummy]{$H_{\v_2}\x$};

\end{tikzpicture}
\caption{If we want to reflect $\x$ about a hyperplane into the span of $e_1$, there are two hyperplanes that will work.
The two choices are defined by the normal vectors $\v_1$ and $\v_2$.
Reflecting about the hyperplane defined by $\v_i$ produces  $H_{\v_i}\x$.}
\label{fig:two reflectors}
\end{figure}

%TODO: explain notation, fix algorithm
\begin{algorithm}
\caption{Householder triangularization.
This algorithm returns orthonormal $Q$ and upper triangular $R$ satisfying $A = QR$.}
\label{Alg:Householder}
\begin{algorithmic}[1]
\Procedure{Householder}{$A$}
\State $m, n \gets \text{shape} \left( A \right)$
\State $R \gets \text{copy} \left( A \right)$
\State $Q \gets I_m$
\For{$0 \leq k < n-1$}
    \State $u_k \gets \text{copy} \left( R_{k:,k} \right)$
    \State $u_{k_0} \gets u_{k_0} + \text{sign} \left( u_{k_0} \right) \norm{u_k}$
    \State $u_k \gets u_k / \norm{u_k}$
    \State $R_{k:,k:} \gets R_{k:,k:} - 2 u_k \left( u_k^\mathsf{T} R_{k:,k:} \right)$
    \State $Q_{k:} \gets Q_{k:} - 2 u_k \left( u_k^\mathsf{T} Q_{k:} \right)$
\EndFor
\State \pseudoli{return} $Q^\mathsf{H}, R$
\EndProcedure
\end{algorithmic}
\end{algorithm}

\begin{problem}
\label{prob:HouseholderQR}
Write a function that accepts as input a $m \times n$ matrix of rank $n$ and computes its QR decomposition, returning the matrices $Q$ and $R$.
Your function should use Algorithm \ref{Alg:Householder}.
Hint: Read about the function \li{np.outer()}.
\end{problem}

\begin{comment}
\subsection*{Stability of the Householder QR algorithm}
We will now examine the stability of the Householder QR algorithm.
We will use SciPy's built in QR factorization which uses Householder reflections internally.

Try the following.

\begin{lstlisting}
>>> Q, X = la.qr(np.random.rand(500,500)) # create a random orthonormal matrix:
>>> R = np.triu(np.random.rand(500,500)) # create a random upper triangular matrix
>>> A = np.dot(Q,R) # Q and R are the exact QR decomposition of A
>>> Q1, R1 = la.qr(A) # compute QR decomposition of A
\end{lstlisting}

Observe:

\begin{lstlisting}
>>> la.norm(Q1-Q)/la.norm(Q) # check error in Q
0.282842955725
>>> la.norm(R1-R)/la.norm(R) # check error in R
0.0428922016647
\end{lstlisting}

This is terrible!
This algorithm works in $16$ decimal points of precision, but $Q_1$ and $R_1$ are only accurate to $0$ and $1$ decimal points, respectively.
We've lost $16$ decimal points of precision!

Don't lose hope.
Check how close the product $Q_1 R_1$ is to $A$.
\begin{lstlisting}
>>> A1 = Q1.dot(R1)
>>> np.absolute(A1 - A).max()
3.9968028886505635e-15
\end{lstlisting}
We've now recovered $15$ digits of accuracy.
Considering the error relative to the norm of $A$ (using the 2-norm for matrices), we see that this relative error is even smaller.
\begin{lstlisting}
>>> la.norm(A1 - A, ord=2) / la.norm(A, ord=2)
8.8655568331889288e-16
\end{lstlisting}
The errors in $Q_1$ and $R_1$ were somehow ``correlated," so that they canceled out in the product.
The errors in $Q_1$ and $R_1$ are called \emph{forward errors}.
The error in $A_1$ is the \emph{backward error}.

In fact, the large errors in \li{Q1} and \li{R1} were not because the algorithm was bad, it was because $A$ was poorly conditioned.
The condition number for randomly generated upper triangular matrices is generally very high, and this was the case here.
This has, in turn, made the condition number of $A$ extremely large.

Try the following to compute the condition number of $A$.
In this case the condition number of $A$ and $R$ are computed to be different, though, in theory, they should be exactly the same.
\begin{lstlisting}
>>> from numpy.linalg import cond
>>> cond(A)
4.1426075832870472e+18
>>> cond(R)
3.1767577244363792e+19
\end{lstlisting}

Householder QR factorization is more numerically stable than Gram-Schmidt or even Modified Gram-Schmidt (MGS).
However, MGS is still useful for some types of iterative methods because it finds the orthonormal basis one vector at a time instead of all at once (for an example see Lab \ref{lab:EigSolve}).
\end{comment}

\section*{Upper Hessenberg form}
An upper Hessenberg matrix is a square matrix with zeros below the first subdiagonal.
Every  $n \times n$ matrix $A$ can be written $A = Q^THQ$ where $Q$ is orthonormal and $H$ is an upper Hessenberg matrix, called the Hessenberg form of $A$.

A fast algorithm for computing the QR decomposition of a Hessenberg matrix will be taught in Lab \ref{lab:givens}. This algorithm in turn leads to a fast algorithm for finding eigenvalues of a matrix, which will be discussed in Lab \ref{lab:EigSolve}.

For now, we will outline an algorithm for computing the upper Hessenberg form of any matrix.
Like Householder triangularization, this algorithm uses Householder transformations.
To find orthogonal $Q$ and upper Hessenberg $H$ such that $A = Q^THQ$, it suffices to find such matrices that satisfy $Q^TAQ=H$.
Thus, our strategy is to multiply $A$ on the right and left by a series of orthonormal matrices until it is in Hessenberg form.
If we use the same $Q_1$ as in the first step of the Householder algorithm, then with $Q_1 A$ we introduce zeros in the first column of $A$.
However, since we now have to multiply $Q_1 A$ on the left by $Q_1^T$, all those zeros are destroyed.

Instead, let's try choosing a $Q_1$ that fixes $e_1$ and reflects the first column of $A$ into the span of $e_1$ and $e_2$.
Because $Q_1$ fixes $e_1$, the product $Q_1A$ leaves the first row of $A$ alone, and $(Q_1A)Q_1^T$ leaves the first column of $(Q_1A)$ alone.
If $A$ is a $5 \times 5$ matrix, this looks like
\[
\begin{array}{ccccc}
\begin{pmatrix}
* & * & * & * & * \\
* & * & * & * & * \\
* & * & * & * & * \\
* & * & * & * & * \\
* & * & * & * & *
\end{pmatrix}
&\underrightarrow{Q_1 \cdot }&
\begin{pmatrix}
* & * & * & * & * \\
* & * & * & * & * \\
0 & * & * & * & * \\
0 & * & * & * & * \\
0 & * & * & * & *
\end{pmatrix}
&\underrightarrow{\cdot Q_1^T }&
\begin{pmatrix}
* & * & * & * & * \\
* & * & * & * & * \\
0 & * & * & * & * \\
0 & * & * & * & * \\
0 & * & * & * & *
\end{pmatrix}
\\
A & & Q_1A & & (Q_1 A) Q_1^T
  \end{array}
\]
We now iterate through the matrix until we obtain
\begin{equation*}
Q_3 Q_2 Q_1 A Q_1^T Q_2 ^T Q_3^T =
\begin{pmatrix}
* & * & * & * & * \\
* & * & * & * & * \\
0 & * & * & * & * \\
0 & 0 & * & * & * \\
0 & 0 & 0 & * & *
\end{pmatrix}.
\end{equation*}

The pseudocode for computing the Hessenberg form of a matrix with Householder transformations is shown in Algorithm \ref{Alg:Hessenberg}.
Although the Hessenberg form exists for any square matrix, this algorithm only works for full-rank square matrices.
Notice that this algorithm is very similar to Algorithm \ref{Alg:Householder}.

\begin{algorithm}
\caption{Reduction to Hessenberg form for a nonsingular matrix.
This algorithm returns orthogonal $Q$ and upper Hessenberg $H$ such that $A = Q^THQ$.}
\label{Alg:Hessenberg}
\begin{algorithmic}[1]
\Procedure{Hessenberg}{$A$}
\State $m, n \gets \text{shape}(A)$
\State $H \gets \text{copy}(A)$
\State $Q \gets I_m$
\For{$0 \leq k < n-2$}
    \State $u_k \gets \text{copy}\left(H_{k+1:, k}\right)$
    \State $u_{k_0} \gets u_{k_0} + \text{sign}(u_{k_0}) \norm{u_k}$
    \State $u_k \gets u_k/\norm{u_k}$
    \State $H_{k+1:,k:} \gets H_{k+1:,k:} - 2u_k(u_k^\mathsf{T} H_{k+1:,k:})$
    \State $H_{:,k+1:} \gets H_{:,k+1:} - 2(H_{:,k+1:} u_k) u_k^\mathsf{T}$
    \State $Q_{k+1:} \gets Q_{k+1:} - 2u_k(u_k^\mathsf{T} Q_{k+1:})$
\EndFor
\State \pseudoli{return} $Q, H$
\EndProcedure
\end{algorithmic}
\end{algorithm}

When $A$ is symmetric, its upper Hessenberg form is a tridiagonal matrix.
This is because the $Q_i$'s zero out everything below the first subdiagonal of $A$ and the $Q_i^T$'s zero out everything above the first superdiagonal.
Thus, the Hessenberg form of a symmetric matrix is especially useful, since as we saw in Lab \ref{lab:complexity}, tridiagonal matrices make computations fast.

%TODO: This doesn't work if $A$ is singular
\begin{problem}
\label{prob:hessenberg}
Write a function that accepts as input a nonsingular square matrix $A$ and computes its Hessenberg form, returning orthogonal $Q$ and upper Hessenberg $H$ satisfying $A = Q^THQ$.
Your function should use Algorithm \ref{Alg:hessenberg}.
What happens when you compute the Hessenberg factorization of a symmetric matrix?
\end{problem}

%Sources: http://www.cs.unc.edu/~krishnas/eigen/node5.html
% http://en.wikipedia.org/wiki/Givens_rotation
%http://en.wikipedia.org/wiki/QR_decomposition
%	Note the Operation count: Householder is 2/3 n^3, MGS is 2 n^3
%http://en.wikipedia.org/wiki/QR_algorithm
%Applied Numerical methods using MATLAB by Yang has some code written for this
%http://www.math.kent.edu/~reichel/courses/intr.num.comp.2/lecture21/evmeth.pdf
%	These are eigenvalue algorithms explained carefully
%http://en.wikipedia.org/wiki/Householder_transformation
%Numerical Linear Algebra, by Lloyd N. Trefethen and David Bau III, Chapters 10 and 16


\subsection{Definitions}
\begin{enumerate}
	\item $f(x_0, y_0)$ is a local maximum of $f$ if for some $\delta > 0$, $f(x_0, y_0) \geq f(x, y) \forall (x,y) \in N((x,y),\delta)$.
		That is, you can draw a circle in the domain of $f$ centered at $(x_0, y_0)$ such that the value of $f$ at every point in the circle besides $(x_0, y_0)$ is less than $(x_0, y_0)$.
	\item $f(x_0, y_0)$ is a local minimum of $f$ if for some $\delta > 0$, $f(x_0, y_0) \leq f(x,y) \forall (x,y) \in N((x,y),\delta)$.
		That is, you can draw a circle in the domain of $f$ centered at $(x_0, y_0)$ such that the value of $f$ at every point in the circle besides $(x_0, y_0)$ is greater than $(x_0, y_0)$.
	\item $f(x_0, y_0)$ is a global max of $f$ if $f(x_0, y_0) \geq f(x,y) \forall (x,y) \in D(f)$ where $D(f)$ is the domain of $f$.
	\item $f(x_0, y_0)$ is a global min of $f$ if $f(x_0, y_0) \leq f(x,y) \forall (x,y) \in D(f)$ where $D(f)$ is the domain of $f$.
\end{enumerate}
\begin{theorem}
	If $(x_0, y_0)$ is in the domain of $f$ and a local extrema of $f(x,y)$, then $f_x(x_0, y_0)$ and $f_y(x_0, y_0)$ is either 0 or undefined.
\end{theorem}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.9\textwidth]{./Images/differentialMultivariableCalculus/optimization.png}
	\caption{Critical points appear when the partial derivatives are 0 or undefined.}
\end{figure}


\section*{Exercises}

\begin{ex}
  Let $A = \begin{mymatrix}{rr}
    1 & 2 \\
    2 & 1
  \end{mymatrix}$. Find $A^{10}$ by diagonalization.
  \begin{sol}
    First we write $A = PDP^{-1}$.
    \begin{equation*}
      \def\arraystretch{1.3}
      \begin{mymatrix}{rr}
        1 & 2 \\
        2 & 1
      \end{mymatrix}
      =
      \begin{mymatrix}{rr}
        -1 & 1 \\
        1 & 1
      \end{mymatrix}
      \begin{mymatrix}{rr}
        -1 & 0 \\
        0 & 3
      \end{mymatrix}
      \begin{mymatrix}{rr}
        -\frac{1}{2} & \frac{1}{2} \\
        \frac{1}{2} & \frac{1}{2}
      \end{mymatrix}
    \end{equation*}
    Therefore $A^{10} = PD^{10}P^{-1}$.
    \begin{equation*}
      \def\arraystretch{1.3}
      \begin{mymatrix}{rr}
        1 & 2 \\
        2 & 1
      \end{mymatrix}^{10}
      ~=~
      \begin{mymatrix}{rr}
        -1 & 1 \\
        1 & 1
      \end{mymatrix}
      \begin{mymatrix}{rr}
        (-1)^{10} & 0 \\
        0 & 3^{10}
      \end{mymatrix}
      \begin{mymatrix}{rr}
        -\frac{1}{2} & \frac{1}{2} \\
        \frac{1}{2} & \frac{1}{2}
      \end{mymatrix} \\
      ~=~
      \frac{1}{2}\begin{mymatrix}{cc}
        3^{10}+1 & 3^{10}-1 \\
        3^{10}-1 & 3^{10}+1
      \end{mymatrix}.
    \end{equation*}
  \end{sol}
\end{ex}

\begin{ex}
  Let $A = \begin{mymatrix}{rrr}
    1 & 4 & 1 \\
    0 & 2 & 5 \\
    0 & 0 & 5
  \end{mymatrix}$. Find $A^{50}$ by diagonalization.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Let $A = \begin{mymatrix}{rrr}
    1 & -2 & -1 \\
    2 & -1 & 1 \\
    -2 & 3 & 1
  \end{mymatrix}$. Find $A^{100}$ by diagonalization.
  % \begin{sol}
  % \end{sol}
\end{ex}

\begin{ex}
  Let $A = \begin{mymatrix}{rr}
    -5 & -6 \\
    9 & 10 \\
  \end{mymatrix}$. Find a square root of $A$, i.e., find a matrix $B$
  such that $B^2=A$.
  \begin{sol}
    $B=\begin{mymatrix}{rr}
      -1 & -2 \\
      3  & 4 \\
  \end{mymatrix}$.
  \end{sol}
\end{ex}

\begin{ex}
  Let $A = \begin{mymatrix}{rrr}
    -2 & 0 & 6 \\
    -3 & 1 & 6 \\
    -3 & 0 & 7 \\
  \end{mymatrix}$. Find a square root of $A$.
  \begin{sol}
    $A = \begin{mymatrix}{rrr}
      0  &  0 & 2 \\
      -1 &  1 & 2 \\
      -1 &  0 & 3 \\
    \end{mymatrix}$.
  \end{sol}
\end{ex}


\section{Scalar multiplication}

\begin{outcome}
  \begin{enumerate}
  \item Multiply a scalar by a vector algebraically and geometrically.
  \item Use the laws of scalar multiplication to prove equalities
    between vector expressions.
  \end{enumerate}
\end{outcome}

Scalar multiplication of vectors in $\R^n$ is defined as
follows.

\begin{definition}{Scalar multiplication of vectors in $\R^n$}{vector-scalar-multiplication}
  If $k\in\R$ is a scalar and $\vect{u}\in \R^n$ is a vector, then
  their \textbf{scalar multiplication}%
  \index{vector!scalar multiplication}%
  \index{scalar multiplication!of a vector} $k\vect{u}\in \R^n$ is
  defined by
  \begin{equation*}
    k\vect{u}=k\begin{mymatrix}{c}
      u_1 \\
      \vdots \\
      u_n
    \end{mymatrix} = \begin{mymatrix}{c}
      ku_1 \\
      \vdots \\
      ku_n
    \end{mymatrix}.
  \end{equation*}
\end{definition}

For example $3 \mat{1, 2, 3}^T = \mat{3, 6, 9}^T$ and
$-2\mat{1, 2, 3}^T = \mat{-2, -4, -6}^T$.

\begin{example}{Geometric meaning of scalar multiplication}{geometric-scalar-multiplication}
  Let $\vect{u}=\mat{2,1}^T$, and draw the following vectors to scale:
  $2\vect{u}$, $\vect{u}$, $\frac{1}{2}\vect{u}$, $0\vect{u}$,
  $-\frac{1}{2}\vect{u}$, $-\vect{u}$, and $-2\vect{u}$.  What is the
  geometric meaning of scalar multiplication?
\end{example}

\begin{solution}
  Here is a picture of the seven vectors. We draw their tails in
  different places to make their relationship easier to see.
  \begin{center}
    \begin{tikzpicture}[scale=0.7]
      \draw[->, thick, blue] (-10.5,-1)    -- node[above]{$2\vect{u}$} +(4,2);
      \draw[->, thick, blue] (-6,-0.5)  -- node[above]{$\vect{u}$} +(2,1);
      \draw[->, thick, blue] (-3,-0.25) -- node[above]{$\frac{1}{2}\vect{u}$} +(1,0.5);
      \draw[fill, blue](0,0) circle [radius=3pt] node[above]{$0\vect{u}$};
      \draw[->, thick, blue] (3,0.25)   -- node[above]{$-\frac{1}{2}\vect{u}$} +(-1,-0.5);
      \draw[->, thick, blue] (6,0.5)    -- node[above]{$-\vect{u}$} +(-2,-1);
      \draw[->, thick, blue] (10.5,1)      -- node[above]{$-2\vect{u}$} +(-4,-2);
    \end{tikzpicture}
  \end{center}
  We see that the vector $k\vect{u}$ has the same direction as
  $\vect{u}$ when $k$ is positive, and the opposite direction when $k$
  is negative. Further, the length of the vector is scaled by a factor
  of $\abs{k}$. It increases if $\abs{k}>1$ and decreases if
  $\abs{k}<1$. For example, the vector $2\vect{u}$ is exactly twice as
  long as $\vect{u}$.  (It is because of this scaling property that
  scalars are called scalars%
  \index{scalar!for scaling}).
\end{solution}

Just as with addition, scalar multiplication of vectors satisfies
several important properties. These are outlined in the following
proposition.

\begin{proposition}{Properties of scalar multiplication}{vector-scalar-multiplication}
  The following properties hold for vectors
  $\vect{u},\vect{v}\in\R^n$ and $k,\ell$ scalars.%
  \index{vector!properties of scalar multiplication}%
  \index{vector!scalar multiplication!properties}%
  \index{properties of scalar multiplication!vector}%
  \begin{itemize}
  \item The distributive law over vector addition
    \index{distributive law!over vector addition}%
    \index{vector!distributive law}%
    \begin{equation*}
      k(\vect{u} + \vect{v}) = k\vect{u} + k\vect{v}.
    \end{equation*}
  \item The distributive law over scalar addition
    \index{distributive law!over scalar addition}%
    \begin{equation*}
      (k + \ell) \vect{u} = k\vect{u} + \ell\vect{u}.
    \end{equation*}
  \item The associative law for scalar multiplication
    \index{associative law!of scalar multiplication}%
    \index{vector!associative law of scalar multiplication}%
    \begin{equation*}
      k(\ell\vect{u}) = (k\ell)\vect{u}.
    \end{equation*}
  \item The rule for multiplication by $1$
    \index{rule for multiplication by 1}%
    \index{vector!rule for multiplication by 1}%
    \begin{equation*}
      1\vect{u}=\vect{u}.
    \end{equation*}
  \end{itemize}
\end{proposition}

\begin{proof}
  We will show the proof of:
  \begin{equation*}
    k(\vect{u} + \vect{v}) = k\vect{u} + k\vect{v}.
  \end{equation*}
  Assume $\vect{u}=\mat{u_1,\ldots,u_n}^T$ and
  $\vect{v}=\mat{v_1,\ldots,v_n}^T$. We have:
  \begin{equation*}
    \begin{array}{ll}
      k(\vect{u} + \vect{v}) & = k\mat{u_1 + v_1,\ldots,u_n + v_n}^T \\
                             & = \mat{k(u_1 + v_1), \ldots, k(u_n + v_n)}^T \\
                             & = \mat{ku_1 + k v_1, \ldots, ku_n + kv_n}^T \\
                             & = \mat{ku_1, \ldots, ku_n}^T + \mat{kv_1, \ldots, kv_n}^T \\
                             & = k\vect{u} + k\vect{v}.
    \end{array}
  \end{equation*}
\end{proof}


%
% Chapter 3.8
%

\section*{3.8 Newton's Method}

Newton's method (Newton-Raphson method) is a way for finding successively better approximations for roots of real-valued functions. The idea behind Newton's method is that the tangent line is close to the curve and so its \(x\)-intercept is close to the \(x\)-intercept of the curve (namely, root \(r\) that we are seeking).

\subsection*{Using Newton's Method}

To find the formula for \(x_2\), we use the fact that the slope of \(L\) is \(f'(x_1)\), so its equation is
$$ y-f(x_1)=f'(x_1)(x-x_1) $$
Since the \(x\)-intercept of \(L\) is \(x_2\), we know that the point \((x_2, 0)\) is on the line, and so
$$ 0-f(x_1)=f'(x_1)(x_2-x_1) $$
If \(f'(x_1) \neq 0\), we can solve this equation for \(x_2\):
We use \(x_2\) as the second approximation to \(r\).
Repeat this procedure with \(x_1\) being replaced by the second approximation \(x_2\) giving \(x_3\) and so on.
\\\\\\
In general if the \(n\)th approximation is \(x_n\) and \(f'(x) \neq 0\), then the next approximation is given by
$$ x_{n+1}=x_n-\frac{f(x_n)}{f'(x_n)} $$
If the numbers \(x_n\) become closer and closer to \(r\) as \(n\) becomes large, then we say that the sequence converges to \(r\) and we write
$$ \lim_{n \to \infty}x_n=r $$


\chapter{Complex numbers}
\label{app:complex}

Throughout history, mankind has invented more and more complicated
number systems in an effort to make algebra easier.
\begin{itemize}
\item In the beginning, there were the \textbf{natural numbers}%
  \index{natural number}%
  \index{number!natural} $\N = \set{1, 2, 3, \ldots}$. However,
  after a while, it became a problem that certain equations, such as
  $x+5=3$, do not have a solution in the natural numbers.
\item To solve this problem, {\em zero} and {\em negative numbers}
  were invented, resulting in the set of \textbf{integers}%
  \index{integer}%
  \index{number!integer} $\Z=\set{\ldots,-3,-2,-1,0,1,2,3,\ldots}$. In
  the integers, the equation $x+5=3$ has a solution, namely
  $x=-2$. But some other equations, such as $2x=1$, still do not have
  a solution in the integers.
\item To solve this problem, the \textbf{rational numbers}%
  \index{rational number}%
  \index{number!rational} $\Q$ were invented. In the rational
  numbers, the equation $2x=1$ has a solution, namely
  $x=\frac{1}{2}$. However, some other equations, such as $x^2=2$,
  still do not have a solution in the rational numbers.
\item To solve this problem, the \textbf{real numbers}%
  \index{real number}%
  \index{number!real} $\R$ were invented. In the real numbers,
  the equation $x^2=2$ has a solution, namely $x=\sqrt{2}$. However,
  some other equations, such as $x^2=-1$, still don't have a solution
  in the real numbers.
\item To solve this problem, the \textbf{complex numbers}%
  \index{complex number}%
  \index{number!complex|see{complex number}} were invented.
\end{itemize}
The purpose of this section is to summarize the most important facts
about the complex numbers. We will also see that the above process
does not continue. In the complex numbers, all non-trivial polynomial
equations have a solution, and therefore, no additional numbers are
``missing''.  This property is called the \textbf{fundamental theorem
  of algebra}%
\index{fundamental theorem of algebra}%
\index{algebra!fundamental theorem of}. Gauss is usually credited with
giving a proof of this theorem in 1797 but many others worked on it
and the first completely correct proof was due to Argand in 1806.


    \documentclass{article}
    \usepackage[utf8]{inputenc}
    \usepackage{amsmath}
    \begin{document}
 %Title
 \section{Collatz Sequence for \(n=25\) }
    %Collatz Function
    \[
        f(n)=
        \begin{cases}
        \frac{n}{2}, & n \mod 2=0
        \\
        3n+1, &n \mod 2=1
        \end{cases} \\
    \]
    % Path for given N
    \(\textbf{Path for f(25)}\\[3mm]f(n), n=25
 \\
 \Rightarrow 3(25) + 1
 \\
 \Rightarrow n=76
 \\[3mm]
f(n), n=76
 \\
 \Rightarrow \frac{76}{2}
 \\
 \Rightarrow n=38
 \\[3mm]
f(n), n=38
 \\
 \Rightarrow \frac{38}{2}
 \\
 \Rightarrow n=19
 \\[3mm]
f(n), n=19
 \\
 \Rightarrow 3(19) + 1
 \\
 \Rightarrow n=58
 \\[3mm]
f(n), n=58
 \\
 \Rightarrow \frac{58}{2}
 \\
 \Rightarrow n=29
 \\[3mm]
f(n), n=29
 \\
 \Rightarrow 3(29) + 1
 \\
 \Rightarrow n=88
 \\[3mm]
f(n), n=88
 \\
 \Rightarrow \frac{88}{2}
 \\
 \Rightarrow n=44
 \\[3mm]
f(n), n=44
 \\
 \Rightarrow \frac{44}{2}
 \\
 \Rightarrow n=22
 \\[3mm]
f(n), n=22
 \\
 \Rightarrow \frac{22}{2}
 \\
 \Rightarrow n=11
 \\[3mm]
f(n), n=11
 \\
 \Rightarrow 3(11) + 1
 \\
 \Rightarrow n=34
 \\[3mm]
f(n), n=34
 \\
 \Rightarrow \frac{34}{2}
 \\
 \Rightarrow n=17
 \\[3mm]
f(n), n=17
 \\
 \Rightarrow 3(17) + 1
 \\
 \Rightarrow n=52
 \\[3mm]
f(n), n=52
 \\
 \Rightarrow \frac{52}{2}
 \\
 \Rightarrow n=26
 \\[3mm]
f(n), n=26
 \\
 \Rightarrow \frac{26}{2}
 \\
 \Rightarrow n=13
 \\[3mm]
f(n), n=13
 \\
 \Rightarrow 3(13) + 1
 \\
 \Rightarrow n=40
 \\[3mm]
f(n), n=40
 \\
 \Rightarrow \frac{40}{2}
 \\
 \Rightarrow n=20
 \\[3mm]
f(n), n=20
 \\
 \Rightarrow \frac{20}{2}
 \\
 \Rightarrow n=10
 \\[3mm]
f(n), n=10
 \\
 \Rightarrow \frac{10}{2}
 \\
 \Rightarrow n=5
 \\[3mm]
f(n), n=5
 \\
 \Rightarrow 3(5) + 1
 \\
 \Rightarrow n=16
 \\[3mm]
f(n), n=16
 \\
 \Rightarrow \frac{16}{2}
 \\
 \Rightarrow n=8
 \\[3mm]
f(n), n=8
 \\
 \Rightarrow \frac{8}{2}
 \\
 \Rightarrow n=4
 \\[3mm]
f(n), n=4
 \\
 \Rightarrow \frac{4}{2}
 \\
 \Rightarrow n=2
 \\[3mm]
f(n), n=2
 \\
 \Rightarrow \frac{2}{2}
 \\
 \Rightarrow n=1
 \\[3mm]

    \section{Credits}
    %Maybe add link to gitrepo and other shit, idk
    This is created using Collatzer (https://github.com/Z1aaan/Collatzer).
    Created By: Z1aaan

    README:
    A program created to visualize and simulate a user-given value for \textit{N}
    and see what happens when it is put under the Collatz function.
    \end{document}


\documentclass{article}
\author{a1trl9}
\title{Probability Review}
\date{}

\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{environ}

\newcommand{\V}[1]{\boldsymbol{#1}}
\newcommand{\TM}{\bigtriangledown}

\NewEnviron{mulequation}{%
\begin{equation}\begin{split}
  \BODY
\end{split}\end{equation}
}

\setlength\parindent{0pt}
\counterwithin*{equation}{section}

\begin{document}
\maketitle

\section{Random Variables}

\subsection{Characteristics of Random Variables}

\subsubsection{Expected Value}

For discrete random variable \(X\), expected value is denfined as:

\begin{equation}
\sum_{i=1}^{\infty}x_ip_i
\end{equation}

Where \(p_i\) is the probability for \(X=x_i\).

For continuous random variable \(X\), expected value is defined as:

\begin{equation}
    \int_{\mathbf{R}}f(x)xdx
\end{equation}

Where \(f(x)\) is the probability density function for \(X=x\).

\vspace{2mm}
\textbf{Proposition 1:} For random variables \(X_1, X_2\), no matter if
they are independent, \(E(X_1+X_2)=E(X_1)+E(X_2)\).

\textbf{Proof}

If \(X_1, X_2\) are continuous variables. Then:

\begin{equation}
\begin{split}
    E(X_1 + X_2)&=\int_{-\infty}^{\infty}
    \int_{-\infty}^{\infty}f(x_1, x_2)(x_1+x_2)dx_1dx_2\\
    &=\int_{-\infty}^{\infty}x_1dx_1\int_{-\infty}^{\infty}f(x_1,x_2)dx_2\\
    &+\int_{-\infty}^{\infty}x_2dx_2\int_{-\infty}^{\infty}f(x_1,x_2)dx_1\\
    &=\int_{-\infty}^{\infty}x_1f(x_1)dx_1+\int_{-\infty}^{\infty}x_2f(x_2)dx_2\\
    &=E(X_1)+E(X_2)
\end{split}
\end{equation}

Noting it actually requires rigorous proof to show why the order of
integrations on \(x_1, x_2\) could be interchanged.

\vspace{2mm}
\textbf{Proposition 2:} If \(X_1, X_2\) are two independent random variables,
\(E(X_1X_2)=E(X_1)E(X_2)\).

\vspace{2mm}
\textbf{Proof}

\textit{Only prove when \(X_1, X_2\) are continuous variables.}

Since \(X_1, X_2\) are independent:
\begin{equation}
    f(x_1, x_2)=f(x_1)f(x_2)
\end{equation}

Therefore:

\begin{equation}
\begin{split}
    E(X_1X_2)&=\int_{-\infty}^{\infty}\int_{-\infty}^{\infty}
    x_1x_2f(x_1, x_2)dx_1dx_2\\
    &=\int_{-\infty}^{\infty}f(x_1)dx_1\int_{-\infty}^{\infty}f(x_2)dx_2\\
    &=E(X_1)E(X_2)
\end{split}
\end{equation}

\subsubsection{Variance}

\vspace{2mm}
\textbf{Proposition 3:} If \(X_1, X_2\) are two independent random variables,
\(Var(X_1+X_2)=Var(X_1)+Var(X_2)\).

\vspace{2mm}
\textbf{Proof}
\begin{equation}
\begin{split}
    &Var(X_1+X_2)=E\{[(X_1+X_2)-E(X_1+X_2)]^2\}\\
    &=E[(X_1+X_2)^2]-E^2(X_1+X_2)\\
    &=E(X_1^2)+E(X_2^2)+2E(X_1X_2)-E^2(X_1)-E^2(X_2)-2E(X_1)E(X_2)\\
    &=E(X_1^2)-E^2(X_1)+E(X_2^2)-E^2(X_2)\\
    &=Var(X_1)+Var(X_2)
\end{split}
\end{equation}

\section{Distributions}
\subsection{Discrete Distributions}
\subsubsection{Poisson Distribution}
\textbf{PDF}

\begin{equation}
p(k)=\frac{e^{-\lambda}\lambda^k}{k!}
\end{equation}

Where \(k \in \{0\} \cup \mathbb{N}\).

According to Taylor series (at 0, Maclaurin series):
\begin{equation}
\begin{split}
f(\lambda)&=e^{\lambda}=f(0)+\frac{f'(0)\lambda}{1!}+\frac{f''(0)\lambda^2}{2!}+\cdots\\
&=\frac{\lambda^0}{0!}+\frac{\lambda^1}{1!}+\frac{\lambda^2}{2!}+\cdots\\
&=\sum_{k=0}^\infty\frac{\lambda^k}{k!}
\end{split}
\end{equation}

Therefore:
\begin{equation}
\begin{split}
\sum_{k=0}^{\infty}\frac{e^{-\lambda}\lambda^k}{k!}&=e^{-\lambda}\sum_{k=0}^{\infty}\frac{\lambda^k}{k!}\\
&=e^{-\lambda}e^{\lambda}\\
&=1
\end{split}
\end{equation}

\subsection{Continuous Distributions}
\subsubsection{Exponential Distribution}
\textbf{PDF}

\begin{equation}
p(x)=\lambda e^{-\lambda x}
\end{equation}

It is obvious:
\begin{equation}
\begin{split}
\int_{0}^{\infty}\lambda e^{-\lambda x}&=-e^{-\lambda x}|^{\infty}_0\\
&=-e^{\infty}+e^0=0+1=1
\end{split}
\end{equation}

\subsubsection{Gamma Distribution}
\vspace{2mm}
\textbf{PDF}
\begin{equation}
\int_0^1x^{a-1}(1-x)^{b-1}dx
\end{equation}

\subsubsection{Beta Distribution}
Beta distribution is used as conjugate prior distibution for Bernoulli Distribution.

\vspace{2mm}
\textbf{PDF}
\begin{equation}
p(x)=x^{a-1}(1-x)^{b-1}dx
\end{equation}

\vspace{2mm}
\textbf{Further: Beta function}

Beta function holds an interesting property that:

\begin{equation}
\mathbf{B}(a, b) = \frac{\Gamma{(a)}\Gamma{(b)}}{\Gamma{(a, b)}}
\end{equation}

\vspace{2mm}
\textbf{Proof}

Let \(\Gamma(a)=\int_0^{\infty}e^{-x}x^{a-1}dx\) and \(\Gamma(b)=\int_0^{\infty}e^{-y}y^{b-1}dy\), then:
\begin{equation}
\begin{split}
\Gamma(a)\Gamma(b)&=\int_0^{\infty}e^{-x}x^{a-1}dx \cdot \int_0^{\infty}e^{-y}y^{b-1}dy\\
&=\int_0^{\infty}\int_0^{\infty}e^{-(x+y)}x^{a-1}y^{b-1}dxdy
\end{split}
\end{equation}

Let \(f(u, v): (u, v) \rightarrow (x, y), x=uv, y=u(1-v)\), then:
\begin{equation}
\Gamma{(a)}\Gamma{(b)}=\int_0^{\infty}\int_0^1e^{-u}(uv)^{a-1}(u(1-v))^{b-1}
|D(u, v)|dudv
\end{equation}

Where \(D(\mathbf{J}(u, v))\) is the determinant of Jacobian matrix \(\mathbf{J}\). As:
\begin{equation}
\mathbf{J}=
\begin{bmatrix}
v & 1 - v \\
u & -u
\end{bmatrix}
\end{equation}

\(|D(\mathbf{J}(u, v))|=|-uv-u(1-v)=-u|=u\). So:
\begin{mulequation}
\Gamma{(a)}\Gamma{(b)}=
&\int_0^{\infty}\int_0^1e^{-u}u^{a+b-2}v^{a-1}(1-v)^{b-1}ududv\\
&=\int_0^{\infty}\int_0^1e^{-u}u^{a+b-1}v^{a-1}(1-v)^{b-1}dudv\\
&=\int_0^{\infty}u^{a+b-1}du\int_0^1v^{a-1}(1-v)^{b-1}dv\\
&=\Gamma{(a+b)}\mathbf{B}(a, b)
\end{mulequation}

Here one fact is important:
\(\Gamma(a)=\int_0^{\infty}e^{-x}x^{a-1}dx\) is convergent, which can be proved
as:
\begin{equation}
\begin{split}
\Gamma{(a)}&=\int_0^{k}e^{-x}x^{a-1}dx+\int_k^{\infty}e^{-x}x^{a-1}dx\\
\end{split}
\end{equation}

And it can be shown \(\forall a, \exists k\):
\begin{equation}
\begin{split}
\int_k^{\infty}e^{-x}x^{a-1}dx
<
\int_k^{\infty}e^{\frac{-x}{2}}dx=-2e^{\frac{-x}{2}}|^{\infty}_1=2e^{\frac{-k}{2}}
\end{split}
\end{equation}

and:

\begin{equation}
\int_0^{k}e^{-x}x^{a-1}dx
\end{equation}

is constant.

\section{Linear Regression}
\textbf{Least Square Approach in the form of matrix}
As for:

\begin{equation}
\mathbf{y}=X\beta + \epsilon
\end{equation}

The least square approach tries to minimize:

\begin{equation}
f=||y-X\beta||^2
\end{equation}

Let each partial derivative of \(f\) on \(\beta\) is equal to zero. That is,
for \(\beta_j\):
\begin{equation}
\sum_{i=1}^nx_{ij}(y_i-\mathbf{x}_i\beta)=0
\end{equation}

It could be transformed to:
\begin{equation}
\begin{split}
&\sum_{i=1}^nx_{ij}y_i=\sum_{i=1}^nx_{ij}\mathbf{x}_i\beta\\
&\mathbf{x}'_j\mathbf{y}=\mathbf{x}'_jX\beta
\end{split}
\end{equation}

Now for all partial derivatives:
\begin{equation}
X'\mathbf{y}=X'X\beta
\end{equation}

That is:
\begin{equation}
\beta=(X'X)^{-1}X'\mathbf{y}
\end{equation}

\end{document}


\section{Mathematical Properties}
Consider $\mathcal{D}$ to be a set of observations, such that $\mathcal{D} = X \biguplus Y$, $\mathcal{D}$ is the set of two sets $X$ and $Y$ with no shared elements between them; $X$ is the set of normal observations, while $Y$ is the set of outliers; Denote $\epsilon$ as threshold; $Fe$ is the feature of a function $F$ of the an observation; Any function $F$ uses feature $Fe$ must satisfy the following properties:

\begin{def}\textbf{(Outlier Existence)} If there are at least two clusters then we should check for outliers:
\begin{align}
degree(D) \geq 2
\end{align}
\end{def}

\begin{def}\textbf{(Normality)} To check whether a certain observation is normal:
\begin{align}
x \in D \text{ is normal observation if } x \in X | d(x) \geq \epsilon
\end{align}
\end{def}

\begin{def}\textbf{(Outlier)} To check whether a certain observation is an outlier:
\begin{align}
y \in D \text{ is an outlier if } y \in Y | d(y) < \epsilon
\end{align}
\end{def}

\section{Motivation}
The motivation of this research was to compare two algorithms of outlier detection. Precisely, One Class Support Vector Machine and Local Outlier Factor, bringing together two advantages:
\begin{itemize}
\item Ability to know the algorithm with the higher efficiency.
\item Ability to know the algorithm with the higher effectiveness.
\end{itemize}

\section{Objective}
In order to achieve the above goals, the following objectives must be satisfied:
\begin{itemize}
\item Simplicity of structure.
\item Functional Completeness.
\end{itemize}


\documentclass[10pt]{article}

\usepackage{amsmath}
\usepackage{algpseudocode}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{enumerate}

\title{Loop Diagonalization}
\author{Vedant Kumar \\ \texttt{vsk@berkeley.edu}}

\begin{document}
\maketitle

\section{Abstract}

The eigenvalue equation $Mv = \lambda v$ is a powerful statement about
matrix and scalar multiplications. In the context of compiler optimization,
we can use it to transform linearizable loops which run in $O(n)$ steps into
matrix operations which run in $O(\log n)$ steps.  This paper defines loop
diagonalization (the process of rewriting linearizable loops as two
matrix multiplies), discusses the implementation of this optimization
with LLVM, and discusses the advantages and limitations of this method.

\section{Overview}

Roughly speaking, a linearizable loop may equally well be represented as a
matrix $M$. Let the loop's state variables reside in a vector $v$. Each
iteration of the loop effects the update $v' = Mv$.

It follows inductively that if one iteration of a linearizable loop gives
$Mv$, $n$ iterations of the loop can be simulated by computing $M^n v$.
Performing this matrix exponentiation naively requires up to $O(nm^3)$
scalar multiplictaions.

Linear algebra comes to the rescue. Recall the equation $M = PDP^{-1}$.  The
column vectors of $P$ are the eigenvectors of $M$, and the diagonal matrix
$D$ contains the corresponding eigenvalues. We can find this
eigendecomposition for any square $M$ so long as $P$ is invertible.

It is known that $M^n = PD^nP^{-1}$, and that $D^n$ can be computed in
$O(m\log n)$ steps using the repeated squaring algorithm. With these facts
we can replace linearizable loops with fast matrix operations.

\section{Diagonalization}

The following definitions are useful: \\

\noindent
\textbf{Def. Linearizable Loop}: A tuple $(L, M, v)$ where $L$ is a list of
basic blocks with a preheader and one backedge, $M$ is a $m$ by $m$ square
matrix with $m$ linearly independent eigenvectors, and $v$ is a vector of
state variables. Let $v' = Mv$: the \emph{only} permissible instructions in
$L$ are the arithmetic operations which contribute to the goal of updating
$v$ to $v'$ with no other side effects, excepting the branch and jump
instructions required to construct a loop. \\

\noindent
\textbf{Def. Loop Diagonalization}: Given a program containing a linearizable
loop $(L, M, v)$, replace $L$ with a single basic block that computes $M^n
v$ using $PD^nP^{-1}v$. \\

Loop diagonalization transforms functions in $O(nm^3)$ into two matrix
multiplications and a diagonal matrix exponentiation, which is in $O(m^3 +
m\log n)$. This can result in an appreciable increase in program efficiency,
as will be shown.

\section{The Fibonacci Example}

Consider the iterative procedure for computing the $n$-th Fibonacci number:

\begin{center}
    \parbox{4cm}{
        \begin{algorithmic}
            \Function{fib}{n}
                \State $a\gets 1$
                \State $b\gets 1$
                \For{$i \in [2 ... n]$}
                    \State $tmp\gets a$
                    \State $a\gets b$
                    \State $b\gets tmp + b$
                \EndFor
                \State \Return b
            \EndFunction
        \end{algorithmic}
    }
\end{center}

The state vector is $v = (a, b)^T$. Initially, $v_0 = (1, 1)^T$. After each
iteration of the loop, $a' = b$ and $b' = a + b$. These linear combinations
are encoded by:

\begin{displaymath}
M = \begin{bmatrix}
        0 & 1 \\
        1 & 1
    \end{bmatrix}
\end{displaymath}

Let $\phi = \frac{1 + \sqrt{5}}{2}$. The eigenvalues of $M$ are $\phi$ and
$1-\phi$. The corresponding eigenvectors are $(1, \phi)^T$ and $(\phi,
-1)^T$. $M$ is diagonalizable because its eigenvalues are distinct and its
eigenvectors are linearly independent:

\begin{displaymath}
M = \begin{bmatrix}
    1 & \phi \\
    \phi & -1
    \end{bmatrix}
    \begin{bmatrix}
    \phi & 0 \\
    0 & 1-\phi
    \end{bmatrix}
    \begin{bmatrix}
    1 & \phi \\
    \phi & -1
    \end{bmatrix}^{-1}
\end{displaymath}

Given $M = PDP^{-1}$, $M^n = PD^nP^{-1}$:

\begin{align*}
M^n &= \begin{bmatrix}
    1 & \phi \\
    \phi & -1
    \end{bmatrix}
    \begin{bmatrix}
    \phi & 0 \\
    0 & 1-\phi
    \end{bmatrix}^n
    \begin{bmatrix}
    1 & \phi \\
    \phi & -1
    \end{bmatrix}^{-1} \\
    &= \begin{bmatrix}
    1 & \phi \\
    \phi & -1
    \end{bmatrix}
    \begin{bmatrix}
    \phi^n & 0 \\
    0 & (1-\phi)^n
    \end{bmatrix}
    \begin{bmatrix}
    1 & \phi \\
    \phi & -1
    \end{bmatrix}^{-1}
\end{align*}

Now the Fibonacci loop has been diagonalized:

\begin{center}
    \parbox{6cm}{
        \begin{algorithmic}
            \Function{fib}{n}
                \State $(a, b)^T \gets PD^{n-1}P^{-1}(1, 1)^T$
                \State \Return b
            \EndFunction
        \end{algorithmic}
    }
\end{center}

\section{Implementation}

I implemented automatic loop diagonalization using the LLVM compiler
infrastructure. This involved creating an instance of
\texttt{llvm::LoopPass} (\texttt{ADPass}) which can be loaded from a dynamic
library. \texttt{ADPass} determines if a loop is linearizable by filtering
out unsupported instructions. It then builds the loop matrix $M$ by
performing a depth-first search on the phi-nodes in the loop. This
search allows the pass to determine the coefficients of every linear
combination of $v$ in the loop, which are exactly the entries of $M$. Next,
the pass uses the \texttt{Eigen} library to find the eigendecomposition of
$M$. Finally the pass deletes the original loop, inserts newly generated
bitcode corresponding to the decomposition into the program, and rewires the
values flowing into the exit phi-nodes. The source code is available under a
non-restrictive free software license
\href{https://github.com/vedantk/auto-diagonalize}{here}.

\section{Testing}

To test the loop diagonalization algorithm, I created a test suite of
iterative processes written in C++ and compiled them at the highest
optimization levels available in LLVM 3.2 and clang++ on Linux x86-64. I
then created alternate versions of the binaries which were post-processed
with \texttt{ADPass}. \texttt{ADPass} was able to diagonalize basic loops
but depended upon pre-processing by the compiler to eliminate stack-spills
and canonicalize loops. It produced good results for all tested programs. I
did not attempt to analyze loops which I knew the pass cannot optimize.

Consider the iterative Fibonacci procedure discussed in a previous section.
I measured the time it took to call $fib(0)$, ..., $fib(200)$ 50,000 times
each. I first tried this with the normal binary and then compared its
performance to the auto-diagonalized binary. As expected, the normal program
(left) scales linearly with the size of the input, whereas the diagonalized
program (right) exhibits $O(\log n)$ performance.

\begin{center}
\includegraphics[scale=0.42]{"fib-O3"}
\includegraphics[scale=0.42]{"fib-O3-diag"}
\end{center}

\section{Discussion}

Matrix decompositions have applications in many fields, but their use in
compiler loop optimizations appears to be novel \footnote{This might have
been true when it was written in 2012, but papers on abstract acceleration
of general linear loops have been published since then.}. It isn't clear if
this optimization pays for its implementation complexity.

Leaving the issue of practicality aside, there are important technical
limitations to \texttt{ADPass}. The most severe restriction is that function
calls, branches, and other instructions with unpredictable side effects
cannot occur within a linearizable loop. In general it is not possible to
lift this draconian restriction without solving undecidable problems such as
`when exactly is this branch taken?' and `does this function call return?'.
Another problem is that there is inherent numerical instability in IEEE 754
floating point numbers, which means that the effects of auto-diagonalization
are not always completely transparent.

With that said, we're left with a somewhat academic optimization that
produces some interesting results. It allows linearizable $O(nm^3)$
processes to run in $O(m^3 + m\log n)$, using the simple magic trick $Mv =
\lambda v$.

\section{References}

\begin{enumerate}[1.]
    \item LLVM, \url{http://llvm.org/}.
    \item Eigen, v3. Ga\"{e}l Guennebaud and Beno\^{i}t Jacob and others.
        \url{http://eigen.tuxfamily.org}.
\end{enumerate}

\end{document}


\newpage
\section{Jinchao: XN}
XN is independent with the SGD training process.

After an update of the neural network, XN can be applied to each hidden layer successively, and won't change the output of it. In this example, we have only one hidden layer. The XN algorithm can be written as follows.
\begin{enumerate}
\item Denote
$$\Theta=(W_1\ b_1)\in\mathbb{R}^{n_1\times(n+1)}$$
and
$$
\hat X=(X;\mathbf{1})\in\mathbb{R}^{(n+1)\times m},
Y=\text{ReLU1}(W_1X+b_1).
$$
	Split the columns of $\hat X$ to $X_I$, $X_J$, $X_K$, s.t. for every row $\theta_i$ of $\Theta$, $i=1,...,n_1$,
	\begin{equation}
	\begin{aligned}
	\theta_i X_I\leq 0,\\
	\theta_i X_J\geq 1,\\
	\theta_i X_K\in[0,1].
	\end{aligned}
	\end{equation}
	Denote IJ as the union of I and J. For every $\theta_i$, the optimization problem can be rewritten as
	\begin{equation}\label{standardv2opt}
	\begin{aligned}
	\min_\theta&\quad\|\theta  X_{IJ}-Y_{IJ}\|_2\\
	\text{subject to}&\quad\theta X_{I}\leq 0,\\
	&\quad\theta X_{J}\geq 1,\\
	&\quad\theta X_{K}=Y_{K}.
	\end{aligned}
	\end{equation}
\end{enumerate}


\section{Linear smoothers}
Most of the smoother presented here are linear smoothers which means
that the fit at any point $x_0$ can be written as
\[
s(x) = \sum_{j=1}^n S_{j}(x) y_j.
\]

In practice we usually have the model
\[
Y_i = f(X_i) + \epsilon_i
\]
and we have observations $\{(x_i, y_i)\}$. Many times it is the vector
${\mathbf f} = \{f(x_1),\dots,f(x_n)\}'$ we are after. In this case
the vector of estimates $\hat{\mathbf f} =
\{\hat{f}(x_1),\dots,\hat{f}(x_n)\}'$ can be written as
\[
\hat{\mathbf f} = \bS \by
\]
with $\bS$ a matrix with the i,j-th entry $S_{j}(x_i)$. We will call
$\hat{\mathbf f}$ the {\it smooth}.

This makes it easy to figure out things like the variance of
$\hat{\f}$ since
\[
\var[\bS \by] = \bS \var[\by] \bS'
\]
which in the case of IID data is $\sigma^2 \bS\bS'$.


\section{Graphs}

\begin{definition}[Arborescence]
    an arborescence is a directed graph in which, for a vertex u called the
    root and any other vertex v, there is exactly one directed path from u to v.
 \end{definition}

\begin{definition}[Clique]
    A subset of vertices $C \subset V$, such that in this subgraph all nodes
    are connected, i.e.\ there is an edge from every pair of nodes.
\end{definition}

\begin{definition}[Connected graph (component)]\label{connectedcomp}
    A graph in which, from any $v \in V$, you can reach any other $u \neq v$
\end{definition}

\begin{definition}[Directed graph]
    Properties:
    \begin{itemize}
        \item $\sum\limits_{v \in V}{d(v)} = 2|E|$
        \item $\sum\limits_{v \in V}{indegree(v)} =
            \sum\limits_{v \in V}{outdegree(v)}$
    \end{itemize}
\end{definition}

\begin{definition}[Eularian]\label{eularian}
    Visit each \textit{edge} once.
\end{definition}

\begin{definition}[Hamiltonian]
    Visit each \textit{vertex} once.
\end{definition}

\begin{definition}[Metric spaces]\label{metric}
    See also~\nameref{semimetric}
    Properties:
    \begin{itemize}
        \item $d_{u,v} = 0 \iff u = v$
        \item $d_{u,v} = d_{v,u}$
        \item $\forall k, d_{u,v} \leq d_{u,k} + d_{k,v}$
    \end{itemize}
\end{definition}

\begin{definition}[Minimum mean cost]
    minimize ratio of cost of arcs (directed edges) to number of arcs
\end{definition}

\begin{definition}[MST]
    A mininum~\nameref{spantree}, such the weight of this tree is less than
    or equal to all other possible~\nameref{spantree}.
\end{definition}

\begin{definition}[Offline Graph Algorithm]
    Algorithms on a graph that requires knowing the entire structure
    For example PageRank, Clustering, so on.
\end{definition}

\begin{definition}[Online Graph Algorithm]
    Process the graph in a serial fashion, e.g. operate on the graph without knowing
    the entire structure. For example shortest path, pattern matching, etc.
\end{definition}

\begin{definition}[Planar graph]
    No edges need to cross each other.
\end{definition}

\begin{definition}[Semi-metric]\label{semimetric}
    Like a~\nameref{metric}, without the property that
    $d_{u,v} = 0 \iff u = v$
\end{definition}

\begin{definition}[Spanning tree]\label{spantree}
    Connected, undriected graph that has all vertices and some subset
    of edges to form a tree.
\end{definition}

\begin{definition}[Strongly connected component]
    Similar to~\nameref{connectedcomp}, but in a directed graph.
\end{definition}

\begin{definition}[Topological sort]\label{topsort}
    Runs in O(V + E), same as DFS, although it can be supered to $O(\log_{2} n)$
\end{definition}

\begin{definition}[Tree]\label{tree}
    Properties of a tree:
    \begin{itemize}
        \item in a tree, there will always be an even amount of nodes that has
            an uneven degree.
        \item Degree of nodes in a tree is at most twice the number of nodes
    \end{itemize}
\end{definition}

\begin{definition}[Vertex-cover]
    A selection of vertices such that each edge is incident to at least one
    of them.
\end{definition}


\subsection{The Ramsey RESET test}

The Ramsey Regression Equation Specification Error Test (RESET)

Once we have done our OLS we have \(\hat y\).

The Ramsey RESET test is an additional stage, which takes these predictions and estimates:

\(y=\theta x+\sum_{i=1}^3\alpha_i \hat {y^i}\)

We then run an F-test on \(\alpha\), with the null that \(\alpha = 0\).


\chapter{The Definition of a Graph}
\label{chapter:graph-theory-basics}
In this chapter we start a very important topic in discrete mathematics, which
became even more important with the rise of computers, we start the discussion
of graph theory. Graphs are used in mathematics and computer science to describe
networks, maps, and dependencies of objects.

\begin{definition}
  A \emph{graph} $G$ is a pair $(V, E)$ such that $E \subseteq V^2$ is a multiset.

  We say that $G$ is \emph{unoriented} iff $(u, v) \in E$ iff $(v, u) \in E$ for
  any $u, v \in V$. Otherwise the graph is \emph{oriented}.
  We say that a graph does not have \emph{loops} iff $(u, u) \notin E$ for any
  $u \in V$. Finally, we say that the graph has \emph{parallel edges} if
  $E$ is not a set.
\end{definition}
A graph is \emph{simple} iff it has no loops, it has not parallel edges, and
it is unoriented.

From now on we will follow a standard convention and think about the set of
edges of unoriented graphs as sets of \emph{unordered} pairs.

It is very convenient to draw graphs using pictures like this.
\begin{center}
  \begin{tikzpicture}
    \node[shape=circle,draw=black] (v1) at (0,0) {\tiny A};
    \node[shape=circle,draw=black] (v2) at (0,1) {\tiny B};
    \node[shape=circle,draw=black] (v3) at (1,1) {\tiny C};
    \node[shape=circle,draw=black] (v4) at (-1,2) {\tiny D};

    \draw (v1) -- (v2);
    \draw (v1) -- (v3);
    \draw (v2) -- (v4);
    \draw (v2) -- (v3);

  \end{tikzpicture}
\end{center}
In this picture, each circle corresponds to a vertice and each line corresponds
to a an edge; i.e. this diagram describes the graph
\[
  (\underbrace{\set{A, B, C, D}}_V,
  \underbrace{\set{(A, B), (A, C), (B, C), (B, D)}}_E).
\]
Note that we already use the convention that in an unoriented graph the pairs
are unordered and we have not listed $(B, A)$, $(C, A)$ etc.

To talk about graphs we need to fix the vocubulary.
An edge is said to \emph{connect} its endpoints; two vertices that are
connected by an edge are called \emph{adjacent}; and a vertex that is an
endpoint of a loop is said to be \emph{adjacent to itself}. An edge is said to
be \emph{incident} on each of its endpoints, and two edges incident
on the same end point are called \emph{adjacent}. A vertex on which no edges
are incident is called \emph{isolated}.

One of the most important examples of graphs are complete graphs defined as
follows.
\begin{definition}
  Let $n$ be a natural number. A complete graph on $n$ vertices, denoted
  $K_n$,\footnote[][-2cm]{%
    Some sources claim that the letter K in this notation stands for the German
    word komplett, but the German name for a complete graph, vollst\"{a}ndiger
    Graph, does not contain the letter K, and other sources state that the
    notation honors the contributions of Kazimierz Kuratowski to graph theory.
  }
  is a simple graph with $n$ vertices and exactly one edge connecting
  each pair of distinct vertices.
\end{definition}
\nomenclature[G]{$K_n$}{denotes the complete graph on $n$ vertices}

\begin{exercise}
\label{exercise:maximal-number-of-edges}
  Show that for all natural numbers $n$, the number of edges of $K_n$
  is $\frac{n(n - 1)}{2}$.
\end{exercise}

\section{Operations on Graphs}
Quite often in order to prove a theorem we need to modify a graph. The most
often operations are the following four. Let $G = (V, E)$ be a graph,
$F \subseteq E$ be a set of edges, $U \subseteq V$ be a set of edges,
$e \in E$ be an edge, and $v \in V$ be a vertex.
\begin{enumerate}
  \item $G[U]$ denotes the graph $(U, \set[e \in U^2]{e \in E})$,
    $G[U]$ is called the induced subgraph of $G$ on the vertices $U$;
    \nomenclature[G]{$G[U]$}{denotes the induced subgraph of $G$ on the vertices $U$
    (i.e. $(U, \set[e \in U^2]{e \in E})$)}
  \item $G[F]$ denotes the graph $(V, F)$, $G[F]$ is called the induced
    subgraph of $G$ on the edges $F$;
    \nomenclature[G]{$G[F]$}{denotes the induced subgraph of $G$ on the edges $F$
    (i.e. $(V, F)$)}
  \item $G - e$ denotes the graph $(V, E \setminus \set{e})$, i.e., the graph $G$
    without the edge $e$.
    \nomenclature[G]{$G - e$}{denotes the graph $(V, E \setminus \set{e})$}
  \item $G - v$ denotes the graph
    $(V \setminus \set{v}, E \cap (V \setminus \set{v})^2)$, i.e., the graph $G$
    without the vertex $v$.
    \nomenclature[G]{$G - v$}{denotes the graph
      $(V \setminus \set{v}, E \cap (V \setminus \set{v})^2)$}
\end{enumerate}

Note that we used the word ``subgraph'', in fact we can define formally the
meaning of this word.
\begin{definition}
  We say that a graph $H = (U, F)$ is a \emph{subgraph} of $G = (V, E)$ iff
  $U \subseteq V$ and $F \subseteq E$.
\end{definition}

\section{Degrees of Vertices}
The degree of a vertex is the number of endsegments of edges that ``stick out
of'' the vertex.
\begin{definition}
  Let $G = (V, E)$ be a graph, and $v$ be a vertex. Then
  $\deg_G(v) = |\set[{e \text{is connected to } v}]{e \in E}|$.
\end{definition}

\begin{exercise}
  Let $G = (V, E)$ be a graph and $v \in V$ be a vertex.
  What are the possible values of $\deg_G(v)$?
\end{exercise}

Note that Lemma~\ref{lemma:handshaking} shows that in any simple graph the
number of vertices with an odd degree is even.
The essence of the proof of this lemma is the following statement.
\begin{theorem}
\label{theorem:handshaking-graphs}
  Let $G = (V, E)$ be a simple graph. Then $\sum_{v \in V} \deg_G(v) = 2|E|$.
\end{theorem}

It is intresting to note that this result allows to give another proof
\Cref{corollary:binommials-equality}. Let us consider a graph $H_n =
(\set{0, 1}^n, E_n)$ such that $(x, y) \in E_n$ iff $x$ and $y$ are
different in exactly one position. Note that $\deg_{H_n} x = n$ for all
$x \in \set{0, 1}^n$; hence, $|E_n| = n 2^n / 2 = n 2^{n - 1}$. In addition,
note that there are $\binom{n}{k}$ vertices with $k$ many 1's and there are
$k$ edges that turn $1$ into $0$; i.e., there are
$\sum_{k = 0}^n k \binom{n}{k}$ edges in $H_n$. Therefore,
$\sum_{k = 0}^n k \binom{n}{k} = n 2^{n - 1}$.

\begin{chapterendexercises}
  \exercise Either draw a graph with the specified properties or explain why
    no such graph exists:
    \begin{enumerate}[nolistsep]
      \item simple graph with five vertices of degrees $1$, $2$, $3$, $3$,
        and $5$;
      \item simple graph with four vertices of degrees $1$, $2$, $3$, and $3$;
      \item simple graph with four vertices of degrees $1$, $1$, $1$, and $5$;
      \item simple graph with four vertices of degrees $1$, $2$, $3$, and $4$;
      \item simple graph with four vertices of degrees $1$, $2$, $3$, and $5$.
    \end{enumerate}
  \exercise Let $G$ be a \emph{simple} graph. Show that there are two different
    vertices $u$ and $v$ such that they have the same degree.
    \begin{solution}
      Assume $G$ is a simple graph with $n$ vertices. Suppose all vertices in
      $G$ have different degrees. The set of degrees of the vertices is then
      $\set{0, 1, \dots, n - 1}$. However this means that one vertex is
      connected to all other vertices and another vertex is connected to no
      other vertices, it is a contradiction. Therefore there are at least two
      different vertices $u$ and $v$ such that they have the same degree.
    \end{solution}
  \exercise Let $G = ([5] \times [5], E)$ be a graph such that
    $((x_1, y_1), (x_2, y_2)) \in E$ iff
    \begin{itemize}
      \item either $x_1 = x_2$ and $|y_1 - y_2| = 1$, or
      \item $|x_1 - x_2| = 1$ and $y_1 = y_2$.
    \end{itemize}

    \begin{enumerate}
      \item Is it possible to cover the graph $G$ with $8$ paths of length $5$
        (these paths should not visit any vertex twice)? Note that by covering we
        mean that every edge of $G$ is an edge of one of the paths.
      \item Is it possible to cover the graph $G$ with $5$ paths of length $8$
        (these paths should not visit any vertex twice)? Note that by covering
        we mean that every edge of $G$ is an edge of one of the paths.
    \end{enumerate}
  \exercise Let $G$ be a graph on $n$ vertices such that every vertex has odd
    degree. Show that $n$ is even.
    \begin{solution}
      One may notice that for any odd and even numbers the following identities
      hold:
      $\mathrm{odd} + \mathrm{odd} = \mathrm{even}$ and $\mathrm{even} +
      \mathrm{odd} = \mathrm{odd}$.

      Assume there are $k$ edges in $G$ where $k \in \Z$ and let $v_i$ denote
      the $i$th vertex of $G$ and $d_i$ denote degree of $v_i$, where $i \in
      \range{n}$. Then the total degree of $G$ (i.e. $\sum_{i = 1}^n d_i$)
      is $2k$ by \Cref{theorem:handshaking-graphs}. Since every vertex has odd
      degree $2k = \sum_{i = 1}^{n} d_i = \mathrm{odd} + \mathrm{odd} + \dots +
      \mathrm{odd}$.

      Notice if $n$ is odd then there is an odd number of vertices and
      $\sum_{i = 1}^n d(v_i) = \mathrm{odd} \neq 2k$ as $2k$ is even. Thus $n$
      must be even.
    \end{solution}
  \exercise In a group of $25$ people, is it possible for each to shake hands
    with exactly $3$ other people?
  \exercise Suppose that $G$ is a graph with $v$ vertices and $e$ edges and
    that the degree of each vertex is at least $d_\mathrm{min}$ and at most
    $d_\mathrm{max}$. Show that
    \[
      \frac{1}{2} v d_\mathrm{min} \le e \le \frac{1}{2} v d_\mathrm{max}.
    \]
\end{chapterendexercises}


\subsection{Axiom of regularity}

The axiom of regularity states that:

\(\forall x[x\ne \varnothing \rightarrow \exists y \in x (y\land x )=\varnothing]\)

That is, for all non-empty sets, there is an element of the set which is disjoint from the set itself.

This means that no set can be a member of itself.


\lab{Anisotropic Diffusion}{Anisotropic Diffusion}
\label{lab:AnisotropicDiffusion}

\objective{Demonstrate the use of finite difference schemes in image analysis.}

A common task in image processing is to remove extra static from an image.
This is most easily done by simply blurring the image, which can be accomplished by  treating the image as a rectangular domain and applying the diffusion (heat) equation:
\[u_t = c \Delta u\]
where $c$ is some diffusion constant and $\Delta$ is the Laplace operator.
Unfortunately, this also blurs the boundary lines between distinct elements of the image.

A more general form of the diffusion equation in two dimensions is:
\[u_t = \nabla\cdot \left( c(x,y,t) \nabla u \right)\]
where $c$ is a function representing the diffusion coefficient at each given point and time.
In this case, $\nabla \cdot$ is the divergence operator and $\nabla$ is the gradient.

To blur a picture uniformly, choose $c$ to be a constant function.
% , but we are often interested in \textit{preserving the edges} between features of the image.
Since $c$ controls how much diffusion is allowed at each point, it can be modified so that diffusion is minimized across edges in the image.
In this way we attempt to limit diffusion near the boundaries between different features of the image, and allow smaller details of the image (such as static) to blur away.
This method for image denoising is especially useful for denoising low quality images, and was first introduced by Pietro Perona and Jitendra Malik in 1987.
It is known as Anisotropic Diffusion or Perona-Malik Diffusion.

\section*{A Finite Difference Scheme}
Suppose we have some estimate $E$ of the rate of change at a given point in an image.
$E$ will be largest at the boundaries in the image.
We will then let $c(x,y,t) = g(E(x,y,t))$ where $g$ is some function such that $g(0)=1$ and $\displaystyle{\lim_{x \to \infty} g(x) = 0}$.
Thus $c$ will be small where $E$ is large, so that little diffusion occurs near the boundaries of different portions of the image.

We will model this system using a finite differencing scheme with an array of values at a 2D grid of points, and iterate through time.
Let $U_{l,m}^n$ be the discretized approximation of the function $u$, $n$ be the index in time, $l$ be the index along the $x$-axis, and $m$ be the index along the $y$-axis.

The Laplace operator can be approximated with the finite difference scheme
\[\Delta u = u_{xx}+u_{yy} \approx \frac{U_{l-1,m}^n - 2 U_{l,m}^n + U_{l+1,m}^n}{(\Delta x)^2} + \frac{U_{l,m-1}^n-2 U_{l,m}^n + U_{l,m+1}^n}{(\Delta y)^2}.\]
A good metric to use with images is to let the distance between each pixel be equal to one, so $\Delta x = \Delta y = 1$.
Rearranging terms, we obtain
\[\Delta u \approx (U_{l-1,m}^n - U_{l,m}^n) + (U_{l+1,m}^n - U_{l,m}^n) + (U_{l,m-1}^n - U_{l,m}^n) + (U_{l,m+1}^n - U_{l,m}^n).\]
Again, since we are working with images and not some time based problem, we can without loss of generality let $\Delta t = 1$, so we obtain the finite difference scheme
\[U_{l,m}^{n+1} = U_{l,m}^n + (U_{l-1,m}^n - U_{l,m}^n) + (U_{l+1,m}^n - U_{l,m}^n) + (U_{l,m-1}^n - U_{l,m}^n) + (U_{l,m+1}^n - U_{l,m}^n).\]
We will now limit the diffusion near the edges of objects by making the modification
\begin{equation}\label{eq:anisotropic}
\begin{aligned}
U_{l,m}^{n+1} = U_{l,m}^n + & \lambda \Big(g(|U_{l-1,m}^n - U_{l,m}^n|)(U_{l-1,m}^n - U_{l,m}^n) \\
			&+ g(|U_{l+1,m}^n - U_{l,m}^n|)(U_{l+1,m}^n - U_{l,m}^n) \\
			&+ g(|U_{l,m-1}^n - U_{l,m}^n|)(U_{l,m-1}^n - U_{l,m}^n) \\
			&+ g(|U_{l,m+1}^n - U_{l,m}^n|)(U_{l,m+1}^n - U_{l,m}^n)\Big),
\end{aligned}
\end{equation}
where $\lambda \leq \frac{1}{4}$ is the stability condition.

In this difference scheme, each term is affected most by nearby terms that are most similar to it, so less diffusion will happen anywhere there is a sharp difference between pixels.
This scheme also has the useful property that it does not increase or decrease the total brightness of the image.
Intuitively, this is because the effect of each point on its neighbors is exactly the opposite effect its neighbors have on it.

Two commonly used functions for $g$ are $g(x) = e^{-\left(\frac{x}{\sigma}\right)^2}$ and $g(x) = \frac{1}{1+\left(\frac{x}{\sigma}\right)^2}$.
The parameter $\sigma$ allows us to control how much diffusion decreases across boundaries, with larger $\sigma$ values allowing more diffusion.
Note that $g(0) = 1$ and $\displaystyle{\lim_{x\to \infty} g(x) = 0}$ for both functions.
In this lab we use $g(x)=e^{-\left(\frac{x}{\sigma}\right)^2}$.

It is worth noting that this particular difference scheme is \textit{not} an accurate finite difference scheme for the version of the diffusion equation we discussed before, but it \textit{does} accomplish the same thing in the same way.
As it turns out, this particular scheme is the solution to a slightly different diffusion PDE, but can still be used the same way.

For this lab's examples we read in the image using the \li{imageio.imread} function, and normalized it so that the colors are represented as floating point values between 0 and 1.
An image can converted to black and white when it is read by including the argument \li{as_gray=True}.

\begin{lstlisting}
from matplotlib import cm, pyplot as plt
from imageio import imread

# To read in an image, convert it to grayscale, and rescale it.
picture = imread('balloon.png', as_gray=True) * 1./255

# To display the picture as grayscale
plt.imshow(picture, cmap=cm.gray)
plt.show()
\end{lstlisting}

\subsection*{Simplifying Calculations}\label{sec:simp}

You will notice that the algorithm given in \ref{eq:anisotropic} does not describe what to do for the edges and corners of $U^{n+1}$.
In these cases we will simply eliminate the undefined terms in the algorithm.
For example, the top edge equation becomes
\begin{align*}
U_{l,m}^{n+1} = U_{l,m}^n + & \lambda (g(|U_{l+1,m}^n - U_{l,m}^n|)(U_{l+1,m}^n - U_{l,m}^n) \\
                    & + g(|U_{l,m+1}^n - U_{l,m}^n|)(U_{l,m+1}^n - U_{l,m}^n)) \\
					& + g(|U_{l,m-1}^n - U_{l,m}^n|)(U_{l,m-1}^n - U_{l,m}^n)),
\end{align*}
and top left corner equation becomes
\begin{align*}
U_{l,m}^{n+1} = U_{l,m}^n + & \lambda (g(|U_{l+1,m}^n - U_{l,m}^n|)(U_{l+1,m}^n - U_{l,m}^n) \\
					& + g(|U_{l,m+1}^n - U_{l,m}^n|)(U_{l,m+1}^n - U_{l,m}^n)).
\end{align*}
Essentially we are only using the terms of the difference scheme that are actually defined.

To help facilitate this we can create a larger "padded" matrix that will make these calculations easy to do.
This padded matrix will have an extra row on the top and bottom, and an extra column on either side of the original matrix.
These extra rows and columns will duplicate the outer edge of the original matrix.

So if our original array \li{X} has shape \li{m,n}, then our padded array \li{Y} has shape \li{m+2,n+2}.
The top edge of \li{Y} will be defined so that \li{Y[0,1:-1] == X[0,:]} is true, and the rest of the edges of \li{Y} follow the same pattern.

Notice that this allows us to simply implement the algorithm found in \ref{eq:anisotropic} without having to make special cases for the edges and corners, since those previously undefined terms become zero when using the padded matrix.

\begin{problem}
\label{prob:anisdiff_bw}
Complete the following function, by implementing the anisotropic diffusion algorithm found in \ref{eq:anisotropic} for black and white images.
Use the padded array technique found in the Simplifying Calculations section.

In your function, use
\[g(x) = e^{-\left(\frac{x}{\sigma}\right)^2}\]

\begin{lstlisting}
def anisdiff_bw(U, N, lambda_, g):
    """ Run the Anisotropic Diffusion differencing scheme
    on the array U of grayscale values for an image.
    Perform N iterations, use the function g
    to limit diffusion across boundaries in the image.
    Operate on U inplace to optimize performance. """
    pass
\end{lstlisting}
Run the function on \li{balloon.jpg}.
Show the original image and the diffused image for $\sigma = .1$, $\lambda = .25$, $N = 5, 20, 100$.
\end{problem}

\newpage
\vfill
\begin{figure}[ht]
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{balloon_grayscale.jpg}
\caption*{original image}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{balloon_grayscale_5.jpg}
\caption*{5 iterations with $\sigma = .1$ and $\lambda = .25$}
\end{minipage}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{balloon_grayscale_20.jpg}
\caption*{20 iterations}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{balloon_grayscale_100.jpg}
\caption*{100 iterations}
\end{minipage}
\end{figure}
\vfill
\clearpage

\section*{Color Schemes}

Colored images can be processed in a similar manner.
Instead of being represented as a two-dimensional array, colored images are represented as three dimensional arrays.
The third dimension is used to store the intensities of each of the standard 3 colors.
This diffusion process can be carried out in the exact same way, on each of the arrays of intensities for each color, but instead of detecting edges just in one color, we need to detect edges in any color, so instead of using something of the form $g(|U_{l+1,m}^n - U_{l,m}^n|)$ as before, we will now use something of the form $g(||U_{l+1,m}^n - U_{l,m}^n||)$, where $U_{l+1,m}^n$ and $U_{l,m}^n$ are vectors now instead of scalars.
The difference scheme can be treated as an equation on vectors in 3-space and now reads:
\begin{align*}
U_{l,m}^{n+1} = U_{l,m}^n + & \lambda (g(||U_{l-1,m}^n - U_{l,m}^n||)(U_{l-1,m}^n - U_{l,m}^n) \\
					& + g(||U_{l+1,m}^n - U_{l,m}^n||)(U_{l+1,m}^n - U_{l,m}^n) \\
					& + g(||U_{l,m-1}^n - U_{l,m}^n||)(U_{l,m-1}^n - U_{l,m}^n) \\
					& + g(||U_{l,m+1}^n - U_{l,m}^n||)(U_{l,m+1}^n - U_{l,m}^n))
\end{align*}

When implementing this scheme for colored images, use the $2$-norm on 3-space, i.e $||x||=\sqrt{x_1^2+x_2^2+x_3^2}$ where $x_1$, $x_2$, and $x_3$ are the different coordinates of $x$.

\begin{problem}
Complete the following function to process a colored image.
You may modify your code from the previous problem.
Measure the difference between pixels using the $2$-norm.
Use the corresponding vector versions of the boundary conditions given in Problem \ref{prob:anisdiff_bw}.

\begin{lstlisting}
def anisdiff_color(U, N, lambda_, sigma):
    """ Run the Anisotropic Diffusion differencing scheme
    on the array U of grayscale values for an image.
    Perform N iterations, use the function g = e^{-x^2/sigma^2}
    to limit diffusion across boundaries in the image.
    Operate on U inplace to optimize performance. """
    pass
\end{lstlisting}
Run the function on \li{balloons_color.jpg}.
Show the original image and the diffused image for $\sigma = .1$, $\lambda = .25$, $N = 5, 20, 100$.

Hint: If you have an $m \times n \times 3$ matrix representing the RGB differences of each pixel, then to find a matrix representing the norm of the differences, you can use the following code.
This code squares each value and sums along the last axis, and takes the square root.
In order to keep the dimension size of the matrix and aid in broadcasting, you must use \li{keepdims=True}.

\begin{lstlisting}
# x is mxnx3 matrix of pixel color values
norm = np.sqrt(np.sum(x**2, axis=2, keepdims=True))
\end{lstlisting}

\end{problem}

\newpage
\section*{Noisy Images}

\begin{problem}
Use the following code to add noise to your grayscale image.

\begin{lstlisting}
from numpy.random import randint

image = imread('balloon.jpg', as_gray=True)
x, y = image.shape
for i in xrange(x*y//100):
	image[randint(x),randint(y)] = 127 + randint(127)
\end{lstlisting}

Run \li{anisdiff_bw()} on the noisy image with $\sigma=.1$, $\lambda=.25$, $N=20$.
Display the original image and the noisy image.
Explain why anisotropic diffusion does not smooth out the noise.

Hint: Don't forget to rescale.
\end{problem}

\section*{Minimum Bias (Optional)}

This sort of anisotropic diffusion can be very effective, but, depending on the image, it may also smear out edges that do not have large differences between them.
An example of this limitation can be seen in Figure \ref{fig:anisdif_smearing}

\begin{figure}
\begin{minipage}[b]{.45\linewidth}
\centering
\includegraphics[width=\textwidth]{baloons_resized_bw.jpg}
\caption*{original image}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{baloons_resized_bw_50.jpg}
\caption*{after 50 iterations}
\end{minipage}
\caption{Smearing of similar colors when using an anisotropic diffusion filter.}
\label{fig:anisdif_smearing}
\end{figure}

As we can see, after 100 iterations, some of the boundaries between similar shades of grey have smeared unevenly. You may still have to look closely to see it.
This can be counteracted somewhat by further decreasing the $\sigma$ value, but if we have random noise throughout the image, this will not remove it.
If we have random static in the image, we can remove this using a modified version of the filter.
Instead of measuring the rate of change in the picture in each direction, we change each point according to whether or not any of its adjacent points have roughly the same value it has.
This is called a minimum-biased filter.
This sort of trick is especially good for removing isolated pixels that are different from those around them.
A very simple way to do this is by taking the average of the two smallest differences between each pixel and its eight neighbors and using that in place of $g$ in the difference scheme above.
Along the boundaries, we do not have 8 neighbors for each pixel, but we can get by by just using the pixels we have and eliminating the other terms in the difference scheme, just as we did before.
This will make it so that points that neighbor points of similar value will not be changed, while points that do not match their surroundings will be faded to become more like the points surrounding them.
This does not have the same symmetrical diffusion as the other scheme, i.e. if one pixel changes, it does not necessarily change its neighboring pixels by the same amount.
As long as you leave $\lambda \leq \frac{1}{4}$ and you have scaled the pixels to have floating point values between 0 and 1, the scheme will still remain within its minimum and maximum bounds, since the tendency is always to move points closer to the values of their neighbors.
To demonstrate the action of such a filter, we make changes to random pixels in the color version of the same photo and use both filters to remove the noise we have added.
Below, we include an example where we have added noise to the color version of that same picture, then used a minimum-biased filter to diminish the noise and the original filter to smooth what remains.

\begin{problem*}
(Optional)

Implement the minimum-biased finite difference scheme described above.
Add noise to \li{balloons_color.jpg} using the provided code below, and clean it using your implementation.
Show the original image, the noised image, and the cleaned image.

\begin{lstlisting}
image = imread('balloons_color.jpg')
x,y,z = image.shape
for dim in xrange(z):
    for i in xrange(x*y//100):
        # Assign a random value to a random place
        image[randint(x),randint(y),dim] = 127 + randint(127)
\end{lstlisting}
Hint: Don't forget to rescale.

\end{problem*}

\newpage
\vfill
\begin{figure}[ht]
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{baloons_resized_color.jpg}
\caption*{original image}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{baloons_resized_noisy.png}
\caption*{randomly changed 100000 color values}
\end{minipage}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{baloons_resized_minbias.jpg}
\caption*{300 iterations of a min-biased scheme}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[width=\textwidth]{baloons_resized_both.jpg}
\caption*{after 8 additional iterations of the first filter with $\lambda=.25$ and $\sigma=.04$.}
\end{minipage}
\end{figure}
\vfill
\clearpage

\nocite{Perona1988,Kim2009}


\subsection{Real numbers aren't closed}

Define as an ordered pair of reals

We have a complete set of real numbers. Do we need any more?

For the real numbers, we showed there were functions on the rational numbers which did not have rational solutions. We can similarly show that there are functions on real numbers which do not have real solutions.

Consider:

\(f(x)=\sqrt x\)

This has no real solution for \(x<0\).

We define:

\(i:=\sqrt {-1}\)

\(i\) and \(-i\) can be used interchangeably.

\((-i)^2=(-1)^2i^2=i^2=-1\)

Complex numbers can be shown more generally as:

\(a+bi\)

We define the complex conjugate of

\(x=a+bi\)

As

\(\bar x=a-bi\)

Note that

\(x\bar x=(a+bi)(a-bi)=a^2-b^2\)

We can take exponents of imaginary numbers

\(c^{i\theta}=a+bi\)

We know the opposite is true.

\(c^{-i\theta}=a-bi\)

So

\(c^{i\theta}c^{-i\theta}=(a+bi)(a-bi)\)

\(1=a^2-b^2\)

The case where \(c=e\) is of particular note. We explore this later.


\subsection{The score}

The score is defined as the differential of the log-likelihood function with respect to \(\theta \).

\(V(\theta, X)=\dfrac{\delta }{\delta \theta }l(\theta ; X) \)

\(V(\theta, X)=\dfrac{1 }{\prod_{i=1}^nP(X_i|\theta )}\dfrac{\delta }{\delta \theta}L(\theta; X) \)

\subsection{Expectation of the score}

The expectation of the score, given the true value of \(\theta \) is:

\(E[V(X|\theta)]=\int V(X|\theta) dX\)

\(E[V(X|\theta)]=E[\dfrac{1 }{\prod_{i=1}^nP(X_i|\theta )}\dfrac{\delta }{\delta \theta}L(\theta; X) ]\)

\(E[V(X|\theta)]=\int \dfrac{1 }{\prod_{i=1}^nP(X_i|\theta )}\dfrac{\delta }{\delta \theta}L(\theta; X) \)

\(E[\dfrac{1 }{\prod_{i=1}^nP(X_i|\theta )}]\)

\(\int \dfrac{1 }{\prod_{i=1}^nP(X_i|\theta )}P(\theta )d\theta \)

We can show that the expected value of this is \(0\).

\subsection{Variance of the score}

The variance of the score is:

\(var [\dfrac{\delta }{\delta \theta }l(\theta ; X) ]\)

\(var [\dfrac{1 }{\prod_{i=1}^nP(X_i|\theta )}]\)


\section*{Dimension Reduction}
\subsection*{Principal component analysis (PCA)}
Given: $D={x_1,...,x_n} \subset \mathbb{R}^d$, $1\leq k \leq d$\\
$\Sigma_{d \times d} = \frac{1}{n}\sum_{i=1}^n x_i x_i^T$, $\mu =\frac{1}{n}\sum_{i = 1}^n x_i = 0$\\
Sol.:
$(W,z_1,...,z_n) = argmin \sum_{i=1}^n||W z_i - x_i||_2^2$,\\
where $W \in \mathbb{R}^{d \times k}$ is orthogonal, $z_1,...,z_n\in\mathbb{R}^k$ is given by $W = (v_1|...|v_k)$ and $z_i = W^T x_i$ where $\Sigma = \sum_{i=1}^d \lambda_i v_i v_i^T$, $\lambda_1 \geq ... \geq \lambda_d \geq 0$

\subsection*{Kernel PCA}
For general $k\geq1$, the Kernel PC are given by $\alpha^{(1)},...,\alpha^{(k)}\in \mathbb{R}^n$, where $\alpha^{(i)} = \frac{1}{\sqrt{\lambda_i}}v_i$ is obtained from: $K = \sum_{i=1}^n \lambda_i v_i v_i^T$, $\lambda_1 \geq ... \geq \lambda_d \geq 0$\\
Given this, a new point $x$ is projected as $z \in \mathbb{R}^k$:\\
$z_i = \sum_{j=1}^n\alpha_j^{(i)}k(x,x_j)$

\subsection*{Autoencoders}
Try to learn identity function: $x \approx f(x;\theta)$\\
$f(x;\theta) = f_2(f_1(x;\theta_1);\theta_2)$; $f_1:$ en-, $f_2:$ decoder\\
Training: $ \underset{w}{\operatorname{min}}\sum_{i=1}^n||x_i-f(x_i;W)||_2^2$