\section{Application: Error correcting codes}

\begin{outcome}
  \begin{enumerate}
  \item Determine the block length, message length, Hamming distance,
    and rate of a code.
  \item Determine whether a code is $m$-error detecting and $m$-error
    correcting.
  \item Find generator and check matrices for a linear code.
  \item Use the syndrome method to correct errors in code blocks.
  \item Construct and use Hamming codes.
  \end{enumerate}
\end{outcome}

% ----------------------------------------------------------------------
\subsection*{Binary codes}

When transmitting or storing information on digital media, the
information is usually encoded as a sequence of bits, i.e., 0s and 1s.
For example, in the ASCII code, each symbol is encoded as a sequence
of 8 bits. The letter ``A'' is encoded as $01000001$, the letter ``B''
is encoded as $01000010$, and so on. We can think of a sequence of $n$
bits as an $n$-dimensional column vector over the field $\Z_2$, i.e.,
as an element of $\Z_2^n$. For our purposes, it is convenient to
continue writing bit sequences horizontally, but we will consider this
to be merely an alternate notation for a column vector.

One issue with digital data is that the data can sometimes be
corrupted. DVDs may be scratched, magnetic storage may depolarize, and
data sent by radio transmission may be subject to interference. This
can result in errors in the data, such as some 0s being changed to 1s
or vice versa. One of the ways to deal with such errors is to
introduce \textbf{redundancy}%
\index{error correcting code!redundancy}%
\index{redundancy} in the way the data is encoded.

A very simple example of redundant encoding is the so-called
\textbf{3-repetition code}%
\index{error correcting code!repetition code}%
\index{repetition code}. This simply means to repeat each bit 3
times. Thus, the bit $0$ is encoded as $000$ and the bit $1$ as
$111$. For example, the bit string $100101$ is encoded as
$111000000111000111$. With this encoding, single bit errors are easy
to detect and correct. Assuming that at most one error occurs within
each 3-bit block, the blocks can be decoded by ``majority
decision''. Namely, if the bits in a block are not the same, we assume
that the error occurred in the bit that is in the minority. The
following table shows the decoding scheme:
\begin{center}
  \begin{tabular}{|c|c|c|c|}
    \hline
    Received & Likely error & Corrected & Decoded \\\hline
    $000$    & none         & $000$     & $0$     \\
    $100$    & bit 1        & $000$     & $0$     \\
    $010$    & bit 2        & $000$     & $0$     \\
    $001$    & bit 3        & $000$     & $0$     \\
    $011$    & bit 1        & $111$     & $1$     \\
    $101$    & bit 2        & $111$     & $1$     \\
    $110$    & bit 3        & $111$     & $1$     \\
    $111$    & none         & $111$     & $1$     \\\hline
  \end{tabular}
\end{center}

\begin{example}{Decoding the 3-repetition code}{decode-repetition}
  A message was encoded with the 3-repetition code. The receiver
  receives the following data: $001011000100111110$. What was the
  likely message?
\end{example}

\begin{solution}
  We start by dividing the received data into blocks of 3 bits:
  \begin{equation*}
    001~011~000~100~111~110.
  \end{equation*}
  We then decode each block separately, using the majority rule.
  The first block $001$ has a majority of zeros, so the likely error is
  in the third bit and the likely decoding is $0$. The second block $011$
  has a majority of ones, so the likely error is in the first bit and
  the likely decoding is $1$. Continuing this way, we find that the
  likely original message was $01~00~11$.
\end{solution}

Since the 3-repetition code can be used to correct some errors, it is
called an \textbf{error correcting code}%
\index{error correcting code}%
\index{code!error correcting|see{error correcting code}}. This
particular code only works if there are not too many errors; namely,
it can correct at most one error per code block. The main drawback of
the 3-repetition code is that it increases the message length by a
factor of 3. In this section, we will use linear algebra over the
field $\Z_2$ to construct better error correcting codes.

Most error correcting codes do not work by encoding individual
bits. Rather, the codes work by dividing the message into
\textit{message blocks} of length $k$, and encoding each such message
block by a \textit{code block} of length $n$.  This leads us to the
following definition:

\begin{definition}{Code}{code}
  Let $n$ and $k$ be positive integers with $k\leq n$. A
  \textbf{code}%
  \index{code} with \textbf{message length}%
  \index{message length}%
  \index{error correcting code!message length} $k$ and \textbf{block
    length}%
  \index{block!block length}%
  \index{error correcting code!block length} $n$ is a set $C$ of
  $2^k$ different vectors in $\Z_2^n$. We call the elements of $C$ the
  \textbf{code blocks}%
  \index{code block}%
  \index{error correcting code!code block}%
  \index{block!code block}. Since there are $2^k$ different code
  blocks, they can be used to encode \textbf{message blocks}%
  \index{message block}%
  \index{error correcting code!message block}%
  \index{block!message block} of length $k$. We say that the
  \textbf{rate}%
  \index{rate of a code}%
  \index{error correcting code!rate} of the code is $\frac{k}{n}$,
  because for every $n$ bits of encoded data transmitted, $k$
  bits of decoded data are obtained.
\end{definition}

\begin{example}{A simple code}{code-simple}
  Consider the following code with message length $k=2$ and block
  length $n=5$:
  \begin{center}
    \begin{tabular}{|c|c|}
      \hline
      Message block & Code block \\\hline
      $00$ & $00000$ \\
      $01$ & $00111$ \\
      $10$ & $11100$ \\
      $11$ & $11011$ \\\hline
    \end{tabular}
  \end{center}
  \begin{enumialphparenastyle}
    \begin{enumerate}
    \item Encode the message\/ $011110$.
    \item Decode the message\/ $111000011111011$.
    \item What is the rate of this code? Is it better or worse than
      the rate of the 3-repetition code?
    \end{enumerate}
  \end{enumialphparenastyle}
\end{example}

\begin{solution}
  (a) We divide the message into blocks of length $2$:
  $01~11~10$. Then we encode each block separately:
  $00111~11011~11100$. (b) We divide the message into blocks of length
  $5$: $11100~00111~11011$. Then we decode each block separately:
  $10~01~11$. (c) The rate is $\frac{2}{5}=0.4$. It is slightly
  higher, and therefore better, than the rate of the 3-repetition code,
  which is $\frac{1}{3}\approx 0.33$.
\end{solution}

% ----------------------------------------------------------------------
\subsection*{Hamming distance and error correction}

The error correction capabilities of a code depend on a property
called the \textit{Hamming distance} of the code, which we now define.

\begin{definition}{Hamming weight and Hamming distance}{hamming-distance}
  \begin{itemize}
  \item Let $\vect{v}$ be a vector in $\Z_2^n$. The \textbf{Hamming
      weight}%
    \index{Hamming weight}%
    \index{error correcting code!Hamming weight} of\/ $\vect{v}$,
    denoted $\Hw(\vect{v})$, is the number of components of\/ $\vect{v}$
    that are equal to 1.
  \item Let $\vect{v},\vect{w}$ be two vectors in $\Z_2^n$. The
    \textbf{Hamming distance}%
    \index{Hamming distance}%
    \index{error correcting code!Hamming distance} between $\vect{v}$
    and $\vect{w}$, denoted $\Hd(\vect{v},\vect{w})$, is the number of
    components where $\vect{v}$ and $\vect{w}$ differ. We can also
    express this as the Hamming weight of\/ $\vect{v}-\vect{w}$, i.e.,
    $\Hd(\vect{v},\vect{w})=\Hw(\vect{v}-\vect{w})$.
  \item Finally, we say that the \textbf{Hamming distance of a code}
    is equal to the smallest Hamming distance between any two code
    blocks.
  \end{itemize}
  A code with block length $n$, message length $k$, and Hamming
  distance $d$ is also called an \textbf{$(n,k,d)$-code}%
  \index{nkd-code@$(n,k,d)$-code}%
  \index{error correcting code!nkd-code@$(n,k,d)$-code}.
\end{definition}

\begin{example}{Hamming distance}{hamming-distance}
  Calculate the Hamming distance between $00111$ and $11100$. What is
  the Hamming distance of the 3-repetition code? What is the
  Hamming distance of the code in Example~\ref{exa:code-simple}?
\end{example}

\begin{solution}
  The vectors $00111$ and $11100$ differ in 4 places, so their Hamming
  distance is $\Hd(00111,11100)=4$. This is also equal to the Hamming
  weight of $00111-11100 = 11011$.

  The 3-repetition code has only two code blocks: $000$ and
  $111$. Since their Hamming distance is 3, the Hamming distance of
  the code is also 3.

  To calculate the Hamming distance of the code from
  Example~\ref{exa:code-simple}, we calculate the Hamming distance
  between all pairs of code blocks:
  \begin{equation*}
    \begin{array}{rcl}
      \Hd(00000,00111) &=& 3, \\
      \Hd(00000,11100) &=& 3, \\
      \Hd(00000,11011) &=& 4, \\
      \Hd(00111,11100) &=& 4, \\
      \Hd(00111,11011) &=& 3, \\
      \Hd(11100,11011) &=& 3. \\
    \end{array}
  \end{equation*}
  Since the smallest distance between any two code blocks is $3$, the
  Hamming distance of the code is $3$.
\end{solution}

The significance of a code's Hamming distance is explained by the
following definition and proposition.

\begin{definition}{Error detection and error correction}{detection-correction}
  \begin{itemize}
  \item We say that a code $C$ is \textbf{$m$-error detecting}%
    \index{error correcting code!m-error detecting@$m$-error
      detecting} if for all valid code blocks $\vect{v}$, whenever\/
    $\vect{w}$ is obtained from $\vect{v}$ by introducing up to $m$
    bit errors, then $\vect{w}$ is not a valid code block.
  \item We say that a code $C$ is \textbf{$m$-error correcting}%
    \index{error correcting code!m-error correcting@$m$-error
      correcting} if for all valid code blocks $\vect{v}$, whenever\/
    $\vect{w}$ is obtained from $\vect{v}$ by introducing up to $m$
    bit errors, then $\vect{v}$ is the only valid code block within
    Hamming distance $m$ of $\vect{w}$.
  \end{itemize}
\end{definition}

\begin{proposition}{Error detection and error correction}{detection-correction}
  Consider a code with Hamming distance $d$. Then the code is:
  \begin{itemize}
  \item $m$-error detecting if $m\leq d-1$;
  \item $m$-error correcting if $2m\leq d-1$.
  \end{itemize}
\end{proposition}

\begin{proof}
  Assume up to $m$ errors have happened. In other words, let
  $\vect{v}$ be a valid code block, and let $\vect{w}$ be the code
  block obtained from $\vect{v}$ by introducing up to $m$ errors.  To
  prove the first claim, assume $m\leq d-1$.  Then
  $\Hd(\vect{v},\vect{w})\leq m\leq d-1$. Since $d$ is the minimum
  distance between any two valid code blocks, $\vect{w}$ cannot be a
  valid code block. Hence, the errors can be detected.  To prove the
  second claim, assume $2m\leq d-1$. Then
  $\Hd(\vect{v},\vect{w})\leq m$. Assume that there exists another
  valid code block $\vect{u}$ within Hamming distance $m$ of
  $\vect{w}$, i.e., assume $\Hd(\vect{w},\vect{u})\leq m$. Then
  \begin{equation*}
    \Hd(\vect{v},\vect{u}) \leq \Hd(\vect{v},\vect{w}) +
    \Hd(\vect{w},\vect{u}) \leq m+m \leq d-1.
  \end{equation*}
  Therefore, the Hamming distance of $\vect{v}$ and $\vect{u}$ is at
  most $d-1$, contradicting the assumption that the code has Hamming
  distance $d$. Hence, there is no such code block $\vect{u}$, and the
  errors can be corrected.
\end{proof}

\begin{example}{Error detection and error correction}{detection-correction}
  Consider a code with Hamming distance $3$. How many errors per code
  block can the code detect? How many can it correct? Also answer this
  question for Hamming distance $2$, $4$ and $5$.
\end{example}

\begin{solution}
  By Proposition~\ref{prop:detection-correction}, a code with Hamming
  distance $3$ can detect up to $2$ errors and correct up to $1$
  error. The answers for other Hamming distances are summarized in the
  following table:
  \begin{equation*}
    \begin{array}{|c|c|c|}
      \hline
      \mbox{Hamming distance} & \mbox{Errors detected} & \mbox{Errors corrected} \\\hline
      2 & 1 & 0 \\
      3 & 2 & 1 \\
      4 & 3 & 1 \\
      5 & 4 & 2 \\
      \hline
    \end{array}
  \end{equation*}
\end{solution}

\begin{example}{Decoding}{simple-code-decoding}
  The following message has been encoded using the code of
  Example~\ref{exa:code-simple}. It contains some errors, but no more
  than 1 error per code block. Can you decode the message?
  \begin{equation*}
    11101~01000~11011~11111~00011
  \end{equation*}
\end{example}

\begin{solution}
  Since the code of Example~\ref{exa:code-simple} has Hamming distance
  $3$, it can correct up to 1 error per code block, so we are able to
  decode the message uniquely. For each code block, we must find the
  unique valid code block that is within Hamming distance $1$ or less.
  \begin{center}
    \begin{tabular}{|c|c|c|c|}
      \hline
      Received & Error & Corrected & Decoded \\\hline
      $11101$  & bit 5 & $11100$   & $10$    \\
      $01000$  & bit 2 & $00000$   & $00$    \\
      $11011$  & none  & $11011$   & $11$    \\
      $11111$  & bit 3 & $11011$   & $11$    \\
      $00011$  & bit 3 & $00111$   & $01$    \\
      \hline
    \end{tabular}
  \end{center}
  The decoded message is $10~00~11~11~01$.
\end{solution}

We can say that a ``good'' error correcting code is one that has a
high rate (of message length divided by block length) and a large
Hamming distance.

% ----------------------------------------------------------------------
\subsection*{Linear codes}

To construct a code of message length $n$, we need to specify a set of
$2^n$ code blocks. If $n$ is large, it is not really feasible to write
down a list of all the code blocks, and to check all their Hamming
distances by hand. For example, a code of block length $n=10$ requires
$2^{10}=1024$ code blocks, and we need to check more than half a
million Hamming distances. Instead, we will focus on a particular
class of codes that is much easier to describe. These are the linear
codes.

\begin{definition}{Linear code}{linear-code}
  A \textbf{linear code}%
  \index{linear code}%
  \index{error correcting code!linear code} is a subspace of
  $\Z_2^n$. If the subspace is $k$-dimensional, then the code has
  message length $k$ and block length $n$.
\end{definition}

The advantage of a linear code is that to specify the code blocks, we
only need to list $k$ basis elements, rather than all $2^k$ elements
of the code.

\begin{example}{Linear code}{linear-code}
  Show that the code from Example~\ref{exa:code-simple} is
  linear. What is a basis for the code?
\end{example}

\begin{solution}
  Consider all linear combinations of $11100$ and $00111$ (with
  scalars in $\Z_2$):
  \begin{equation*}
    \begin{array}{rcl}
      0(11100) + 0(00111) &=& 00000, \\
      0(11100) + 1(00111) &=& 00111, \\
      1(11100) + 0(00111) &=& 11100, \\
      1(11100) + 1(00111) &=& 11011. \\
    \end{array}
  \end{equation*}
  This shows that the code of Example~\ref{exa:code-simple} is
  $\sspan\set{11100, 00111}$, and hence a subspace of $\Z_2^5$. A
  basis for the code is $\set{11100, 00111}$
\end{solution}

From Section~\ref{sec:null-space}, we know that every subspace of
$\Z_2^n$, and therefore every linear code, is the column space of some
matrix $G$, and also the null space of some matrix $H$. Such matrices
are called a {\em generator matrix} and a {\em check matrix} for the
code, respectively.

\begin{definition}{Generator matrix and check matrix}{generator-check-matrix}
  Consider a linear code $C$ with block length $n$ and message length
  $k$, i.e., a $k$-dimensional subspace of $\Z_2^n$.
  \begin{itemize}
  \item A \textbf{generator matrix}%
    \index{generator matrix}%
    \index{linear code!generator matrix}%
    \index{error correcting code!generator matrix} for the code is an
    $n\times k$-matrix $G$ such that $C$ is the column space of $G$.
  \item A \textbf{check matrix}%
    \index{check matrix}%
    \index{linear code!check matrix}%
    \index{error correcting code!check matrix} for the code is an
    $(n-k)\times n$-matrix $H$ such that $C$ is the null space of $H$.
  \end{itemize}
\end{definition}

\begin{example}{Generator matrix and check matrix}{generator-check-matrix}
  Find a generator matrix and check matrix for the linear code from
  Example~\ref{exa:code-simple}.
\end{example}

\begin{solution}
  We already found in Example~\ref{exa:linear-code} that
  $\set{11100, 00111}$ is a basis for this code. We can obtain a
  generator matrix by using the basis vectors as columns:
  \begin{equation*}
    G = \begin{mymatrix}{cc}
      1 & 0 \\
      1 & 0 \\
      1 & 1 \\
      0 & 1 \\
      0 & 1 \\
    \end{mymatrix}.
  \end{equation*}
  To find a check matrix, assume that $\mat{a,b,c,d,e}$ is a row of
  the check matrix. Then for every code block $\vect{v}\in C$, we must
  have $\mat{a,b,c,d,e}\vect{v}=0$. Since the code blocks are spanned
  by $11100$ and $00111$, it suffices to consider the two equations
  \begin{equation*}
    \begin{mymatrix}{ccccc} a & b & c & d & e \end{mymatrix}
    \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \\ 0 \end{mymatrix}
    = 1a+1b+1c+0d+0e = 0
  \end{equation*}
  and
  \begin{equation*}
    \begin{mymatrix}{ccccc} a & b & c & d & e \end{mymatrix}
    \begin{mymatrix}{c} 0 \\ 0 \\ 1 \\ 1 \\ 1 \end{mymatrix}
    = 0a+0b+1c+1d+1e = 0.
  \end{equation*}
  Solving this system of equations, we find that the following is a
  basis for the solution space:
  \begin{equation*}
    \set{
      \begin{mymatrix}{ccccc} 1 & 1 & 0 & 0 & 0 \end{mymatrix},
      \begin{mymatrix}{ccccc} 1 & 0 & 1 & 1 & 0 \end{mymatrix},
      \begin{mymatrix}{ccccc} 1 & 0 & 1 & 0 & 1 \end{mymatrix}
    }
  \end{equation*}
  We can use these basic solutions as the rows of the check matrix:
  \begin{equation*}
    H = \begin{mymatrix}{ccccc}
      1 & 1 & 0 & 0 & 0 \\
      1 & 0 & 1 & 1 & 0 \\
      1 & 0 & 1 & 0 & 1 \\
    \end{mymatrix}
  \end{equation*}
  \vspace{-6ex}\par
\end{solution}

% ----------------------------------------------------------------------
\subsection*{Encoding and the generator matrix}

Let $C$ be a linear code with generator matrix $G$. Since $C$ is the
column space of the generator matrix, there is a simple encoding
method: we can simply multiply the generator matrix $G$ by a message
block to obtain the corresponding code block. In other words, if
$\vect{u}\in\Z_2^k$ is a message block, we can use
$\vect{v}=G\vect{u}\in\Z_2^n$ as the code block.

\begin{example}{Using the generator matrix for encoding}{generator-matrix-encoding}
  Use the generator matrix
  \begin{equation*}
    G = \begin{mymatrix}{cc}
      1 & 0 \\
      1 & 0 \\
      1 & 1 \\
      0 & 1 \\
      0 & 1 \\
    \end{mymatrix}
  \end{equation*}
  to encode the message\/ $01~11~10$.
\end{example}

\begin{solution}
  The message blocks are
  $\vect{u}_1 = \begin{mysmallmatrix}{c} 0 \\ 1 \end{mysmallmatrix}$,
  $\vect{u}_2 = \begin{mysmallmatrix}{c} 1 \\ 1 \end{mysmallmatrix}$,
  and
  $\vect{u}_3 = \begin{mysmallmatrix}{c} 1 \\ 0 \end{mysmallmatrix}$.
  We obtain the corresponding code blocks by multiplication with the
  generator matrix:
  \begin{equation*}
    \vect{v}_1
    = G\vect{u}_1
    = \begin{mymatrix}{c} 0 \\ 0 \\ 1 \\ 1 \\ 1 \end{mymatrix}
    \quad
    \vect{v}_2
    = G\vect{u}_2
    = \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 1 \\ 1 \end{mymatrix}
    \quad
    \vect{v}_3
    = G\vect{u}_3
    = \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \\ 0 \end{mymatrix}
  \end{equation*}
  Therefore the encoded message is $00111~11011~11100$. Note that
  this is the same answer as in Example~\ref{exa:code-simple}(a).
\end{solution}

% ----------------------------------------------------------------------
\subsection*{Decoding and the check matrix}

Let $C$ be a linear code with check matrix $H$.  By definition, $C$ is
the null space of the check matrix. This means that a vector
$\vect{v}\in\Z_2^n$ is a valid code block (i.e., a code block without
errors) if and only if $H\vect{v}=\vect{0}$. So we can easily use the
check matrix to determine whether a code block contains errors or not.

However, something even better is true. The value of $H\vect{v}$, when
it is not zero, tell us not only {\em that} an error has occurred, but
also {\em which} error has occurred! To see why, consider a code block
$\vect{v}'$ possibly containing errors. Then
$\vect{v}' = \vect{v} + \vect{e}$, where $\vect{v}$ is a valid code
block and $\vect{e}$ is an \textbf{error pattern}%
\index{error pattern}%
\index{error correcting code!error pattern}%
\index{linear code!error pattern}. The error pattern is the vector
$\vect{e}$ that has a $1$ in every component in which an error
occurred, and a $0$ everywhere else.  Then we have
\begin{equation*}
  H\vect{v}' = H(\vect{v} + \vect{e}) = H\vect{v} + H\vect{e} =
  \vect{0} + H\vect{e} = H\vect{e}.
\end{equation*}
Therefore, the value of $H\vect{v}'$ only depends on the error
pattern, and not on $\vect{v}$. The value
$\vect{s} = H\vect{v'} = H\vect{e}$ is called the \textbf{syndrome}%
\index{syndrome}%
\index{error correcting code!syndrome}%
\index{linear code!syndrome} of the error. A code is error correcting,
for a class of error patterns, if and only if each such error pattern
has a different syndrome. In that case, we can simply make a table of
all the syndromes and corresponding error patterns, as an efficient
method for correcting errors. Such a table is called a
\textbf{syndrome table}%
\index{syndrome table}%
\index{error correcting code!syndrome table}%
\index{linear code!syndrome table} and the corresponding decoding
method is called \textbf{syndrome decoding}%
\index{syndrome decoding}%
\index{error correcting code!syndrome decoding}%
\index{linear code!syndrome decoding}.

\begin{example}{Syndrome decoding}{syndrome-decoding}
  Consider the linear code of Example~\ref{exa:code-simple} with check
  matrix
  \begin{equation*}
    H = \begin{mymatrix}{ccccc}
      1 & 1 & 0 & 0 & 0 \\
      1 & 0 & 1 & 1 & 0 \\
      1 & 0 & 1 & 0 & 1 \\
    \end{mymatrix}.
  \end{equation*}
  Make a syndrome table for all single-bit errors.  Then use your
  syndrome table to decode the message
  $11101~01000~11011~11111~00011$.
\end{example}

\begin{solution}
  Since we are only interested in single-bit errors, there are six
  error patterns to consider: $00000$ (no error), $10000$, $01000$,
  $00100$, $00010$, and $00001$. For each of these error patterns
  $\vect{e}$, we compute the corresponding syndrome $H\vect{e}$. For
  example,
  \begin{equation*}
    H \begin{mymatrix}{c} 0 \\ 0 \\ 0 \\ 0 \\ 0 \end{mymatrix}
    = \begin{mymatrix}{c} 0 \\ 0 \\ 0 \end{mymatrix},\quad
    H \begin{mymatrix}{c} 1 \\ 0 \\ 0 \\ 0 \\ 0 \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 1 \\ 1 \end{mymatrix},\quad
    H \begin{mymatrix}{c} 0 \\ 1 \\ 0 \\ 0 \\ 0 \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 0 \\ 0 \end{mymatrix},
  \end{equation*}
  and so on. We put this information in a table (writing the vectors
  horizontally as usual in this section). This is the syndrome table:
  \begin{equation*}
    \begin{array}{|c|c|}
      \hline
      \mbox{Error pattern $\vect{e}$} & \mbox{Syndrome $H\vect{e}$} \\\hline
      00000 & 000 \\
      10000 & 111 \\
      01000 & 100 \\
      00100 & 011 \\
      00010 & 010 \\
      00001 & 001 \\
      \hline
    \end{array}
  \end{equation*}
  To decode the message $11101~01000~11011~11111~00011$, we first
  calculate the syndrome of each code block by multiplying it by the
  check matrix. The syndrome table then tells us the corresponding
  error pattern, which we can use to correct the code block.

  For example, consider the first code block $11101$. Multiplying by
  $H$, we get the syndrome $001$:
  \begin{equation*}
    H \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 0 \\ 0 \\ 1 \end{mymatrix}
  \end{equation*}
  The syndrome table then tells us that the corresponding error pattern
  is $00001$. Therefore, the corrected code block is $11101 + 00001 =
  11100$. The decoded block is $10$. We proceed in the same way for
  all the code blocks:
  \begin{equation*}
    \begin{array}{|c|c|c|c|c|}
      \hline
      \mbox{Received code block $\vect{v}'$} &
      \mbox{Syndrome $H\vect{v}'$} &
      \mbox{Error pattern $\vect{e}$} &
      \mbox{Corrected code block $\vect{v}=\vect{v}'+\vect{e}$} &
      \mbox{Decoded} \\
      \hline
      11101 & 001 & 00001 & 11100 & 10 \\
      01000 & 100 & 01000 & 00000 & 00 \\
      11011 & 000 & 00000 & 11011 & 11 \\
      11111 & 011 & 00100 & 11011 & 11 \\
      00011 & 011 & 00100 & 00111 & 01 \\
      \hline
    \end{array}
  \end{equation*}
  Therefore, the decoded message is $10~00~11~11~01$. Note that this
  is the same answer we got in Example~\ref{exa:simple-code-decoding}.
  But the syndrome table gives a more systematic method of finding the
  error patterns, which we previously had to do by guessing, or by
  comparing to all possible code blocks.
\end{solution}

\begin{example}{An invalid code block}{invalid-code-block}
  Using the code of the previous example, suppose you have received
  the code block $10010$. Can you decode it?
\end{example}

\begin{solution}
  The syndrome of this code block is
  \begin{equation*}
    H \begin{mymatrix}{c} 1 \\ 0 \\ 0 \\ 1 \\ 0 \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 0 \\ 1 \end{mymatrix}.
  \end{equation*}
  Since the syndrome $101$ is not in our syndrome table, it is not the
  syndrome of any single-bit error. Therefore, the code block $10010$
  must contain more than one error. Since our code is only 1-error
  correcting, this error cannot be corrected. (In fact, there are two
  valid code words within Hamming distance $2$, namely $00000$ and
  $11011$).
\end{solution}

% ----------------------------------------------------------------------
\subsection*{Hamming codes}

The syndrome decoding method immediately gives us the following
theorem:

\begin{theorem}{Check matrices of 1-error correcting codes}{1-error-check}
  Consider a linear code with check matrix $H$. Then the code is
  1-error correcting if and only if all columns of the check matrix
  are non-zero and distinct.
\end{theorem}

\begin{proof}
  The syndrome of the zero error pattern is always
  $H\vect{0}=\vect{0}$. Let $\vect{e}_i$ be the error pattern
  containing a single bit error in the $i\th$ component. Then its
  syndrome is $H\vect{e}_i$, which is the $i\th$ column of $H$.

  If $H$ has distinct, non-zero columns, then each possible single bit
  error has a different syndrome. Therefore, the errors can be
  corrected.

  Conversely, if $H$ has a column that is zero, or two columns that
  are equal, then the corresponding error patterns have the same
  syndrome, and therefore cannot be corrected.
\end{proof}

This theorem was discovered by Richard Hamming%
\index{Richard Hamming}%
\index{Hamming, Richard} in 1950. Hamming then realized that the best
single-error correcting binary codes can be constructed by letting the
check matrix have {\em all} possible non-zero columns. The resulting
codes are called Hamming codes.

\begin{definition}{Hamming code}{hamming-code}
  Let $r\geq 2$ The $r\th$ \textbf{Hamming code}%
  \index{Hamming code}%
  \index{error correcting code!Hamming code}%
  \index{linear code!Hamming code} is a linear code whose check matrix
  $H$ is an $r\times (2^r-1)$-matrix that has all possible non-zero
  column vectors as its columns.
\end{definition}

From the size of the check matrix, we know that the code length is
$n=2^r-1$. Since the check matrix has rank $r$, its null space has
dimension $k=n-r$. Thus, the $r\th$ Hamming code is an $(n,k,3)$-code,
where $n=2^r-1$ and $k=2^r-1-r$.

It is customary to order the columns of the check matrix so that the
last $r$ columns are the standard basis vectors. In other words, the
check matrix is usually taken to be of the form
\begin{equation*}
  H=\begin{mymatrix}{c|c}A&I\end{mymatrix}
\end{equation*}
where $I$ is the $r\times r$-identity matrix and $A$ is the
$r\times k$-matrix consisting of the remaining columns of $H$. In that
case, we can take the $n\times k$-matrix
\begin{equation*}
  G = \begin{mymatrix}{c} I \\\hline A \end{mymatrix}
\end{equation*}
as the generator matrix, where $I$ is the $k\times k$-identity
matrix. Specifically, for this choice of $G$, we have $HG=0$, ensuring
that the column space of $G$ is contained in the null space of
$H$. Moreover, since $\rank G = k$, the column space of $G$ is
$k$-dimensional, and therefore equal to the null space of $H$, so that
$G$ is indeed a correct generator matrix for the code.

\begin{example}{Hamming code for $r=3$}{hamming-code-3}
  Construct a check matrix and generator matrix for the Hamming code
  with $r=3$. Then encode the message $0101~1110~0111$.
\end{example}

\begin{solution}
  The check matrix must be a $3\times 7$-matrix whose columns are all
  the possible non-zero vectors of length 3. Moreover, we will follow
  the convention of using the standard basis vectors as the last 3
  columns. Such a matrix is
  \begin{equation*}
    H = \begin{mymatrix}{cccc|ccc}
      1 & 1 & 0 & 1 & 1 & 0 & 0 \\
      1 & 0 & 1 & 1 & 0 & 1 & 0 \\
      0 & 1 & 1 & 1 & 0 & 0 & 1 \\
    \end{mymatrix}.
  \end{equation*}
  The corresponding generator matrix $G$ is
  \begin{equation*}
    G = \begin{mymatrix}{cccc}
      1 & 0 & 0 & 0 \\
      0 & 1 & 0 & 0 \\
      0 & 0 & 1 & 0 \\
      0 & 0 & 0 & 1 \\\hline
      1 & 1 & 0 & 1 \\
      1 & 0 & 1 & 1 \\
      0 & 1 & 1 & 1 \\
    \end{mymatrix}.
  \end{equation*}
  To encode the message $0101~1110~0111$, we multiply the generator
  matrix by each code block:
  \begin{equation*}
    G \begin{mymatrix}{c} 0 \\ 1 \\ 0 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 0 \\ 1 \\ 0 \\ 1 \\ 0 \\ 1 \\ 0 \end{mymatrix},
    \quad
    G \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \\ 0 \\ 0 \\ 0 \end{mymatrix},
    \quad
    G \begin{mymatrix}{c} 0 \\ 1 \\ 1 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 0 \\ 1 \\ 1 \\ 1 \\ 0 \\ 0 \\ 1 \end{mymatrix}.
  \end{equation*}
  Thus, the encoded message is $0101010~1110000~0111001$. Note that
  the fact that the top part of the generator matrix $G$ is the
  identity matrix has the pleasant effect that the $k$ first bits of
  each code block are the corresponding message block. This makes
  decoding especially convenient (after any errors have been corrected
  first).
\end{solution}

\begin{example}{Decoding a Hamming code}{hamming-3-decoding}
  Using the Hamming $(7,4,3)$-code of the previous example, correct
  the errors in the message $1011101~1101001~0001101~1110000$ and
  decode it.
\end{example}

\begin{solution}
  We do not need to make a syndrome table, because the syndromes are
  exactly the columns of the check matrix $H$. More precisely, the
  $i\th$ column of the check matrix is the syndrome of the error
  pattern containing a single-bit error in the $i\th$ bit.
  We use the check matrix
  \begin{equation*}
    H = \begin{mymatrix}{cccc|ccc}
      1 & 1 & 0 & 1 & 1 & 0 & 0 \\
      1 & 0 & 1 & 1 & 0 & 1 & 0 \\
      0 & 1 & 1 & 1 & 0 & 0 & 1 \\
    \end{mymatrix}
  \end{equation*}
  to calculate the syndrome of each code block:
  \begin{equation*}
    H \begin{mymatrix}{c} 1 \\ 0 \\ 1 \\ 1 \\ 1 \\ 0 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 1 \\ 1 \end{mymatrix},
    \quad
    H \begin{mymatrix}{c} 1 \\ 1 \\ 0 \\ 1 \\ 0 \\ 0 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 0 \\ 1 \end{mymatrix},
    \quad
    H \begin{mymatrix}{c} 0 \\ 0 \\ 0 \\ 1 \\ 1 \\ 0 \\ 1 \end{mymatrix}
    = \begin{mymatrix}{c} 0 \\ 1 \\ 0 \end{mymatrix},
    \quad
    H \begin{mymatrix}{c} 1 \\ 1 \\ 1 \\ 0 \\ 0 \\ 0 \\ 0 \end{mymatrix}
    = \begin{mymatrix}{c} 0 \\ 0 \\ 0 \end{mymatrix}.
  \end{equation*}
  From these syndromes, we can read off the error locations and
  correct the errors:
  \begin{equation*}
    \begin{array}{|c|c|c|c|c|c|}
      \hline
      \mbox{Received code block} &
      \mbox{Syndrome} &
      \mbox{Error position} &
      \mbox{Error pattern} &
      \mbox{Corrected code block} &
      \mbox{Decoded} \\
      \hline
      1011101 & 111 & \mbox{bit 4} & 0001000 & 1010101 & 1010 \\
      1101001 & 101 & \mbox{bit 2} & 0100000 & 1001001 & 1001 \\
      0001101 & 010 & \mbox{bit 6} & 0000010 & 0001111 & 0001 \\
      1110000 & 000 & \mbox{none}  & 0000000 & 1110000 & 1110 \\
      \hline
    \end{array}
  \end{equation*}
  Thus, the decoded message is $1010~1001~0001~1110$.
\end{solution}

\begin{example}{Hamming code for $r=2$}{hamming-code-2}
  What is the Hamming code for $r=2$? Have you seen this code before?
\end{example}

\begin{solution}
  The check matrix and generator matrix for the Hamming code with
  $r=2$ are
  \begin{equation*}
    H = \begin{mymatrix}{c|cc}
      1 & 1 & 0 \\
      1 & 0 & 1 \\
    \end{mymatrix}
    \quad\mbox{and}\quad
    G = \begin{mymatrix}{c} 1 \\\hline 1 \\ 1 \end{mymatrix}.
  \end{equation*}
  The code has block size $n=3$ and message length $k=1$. The encoding
  function is
  \begin{equation*}
    G\begin{mymatrix}{c} 0 \end{mymatrix}
    = \begin{mymatrix}{c} 0 \\ 0 \\ 0 \end{mymatrix},\quad
    G\begin{mymatrix}{c} 1 \end{mymatrix}
    = \begin{mymatrix}{c} 1 \\ 1 \\ 1 \end{mymatrix}.
  \end{equation*}
  This is exactly the 3-repetition code.
\end{solution}

We end this section by remarking that the Hamming codes are very
efficient. The rate of the $r\th$ Hamming code is
\begin{equation*}
  \frac{k}{n}
  ~=~ \frac{2^r-1-r}{2^r-1}
  ~=~ 1 - \frac{r}{2^r-1},
\end{equation*}
which is very close to $1$ when $r$ is large. The following table
lists the block sizes, message sizes, and rates of all Hamming codes
up to $r=8$.
\begin{equation*}
  \begin{array}{|c|c|c|c|}
    \hline
    r & \mbox{Block size $n$} & \mbox{Message size $k$} & \mbox{Rate} \\\hline
    2  & 3    & 1    & 0.333 \\
    3  & 7    & 4    & 0.571 \\
    4  & 15   & 11   & 0.733 \\
    5  & 31   & 26   & 0.839 \\
    6  & 63   & 57   & 0.905 \\
    7  & 127  & 120  & 0.945 \\
    8  & 255  & 247  & 0.969 \\
    \hline
  \end{array}
\end{equation*}
Of course, since the codes are only 1-error correcting, one cannot
increase the block size indefinitely, or else the probability of
having two or more errors in a block becomes too large. There exist
more sophisticated error correcting codes with larger Hamming
distances, which can correct many errors per code block. You might
learn more about such codes in a course on applied modern algebra.


\subsection{Constant}
Let $a$ be a constant.\\
By definitions of a Laplace transform and an improper integral,
\begin{equation*}
	\Laplace{a} = \lim\limits_{n\to\infty}{\int_{0}^{n}{ae^{-st}\mathrm{d}t}}
\end{equation*}
\begin{equation*}
	= \frac{a}{s}\lim\limits_{n\to\infty}\left[-e^{-st}\right]_{0}^{n} \text{, } s \neq 0
\end{equation*}
\begin{equation*}
	= \frac{a}{s}\lim\limits_{n\to\infty}{\left(1-e^{-sn}\right)} \text{, } s \neq 0
\end{equation*}
\begin{equation*}
	= \frac{a}{s} \text{, } s > 0
\end{equation*}
So,
\begin{equation*}
	\Laplace{a} = \frac{a}{s} \text{, } s > 0
\end{equation*}


%!TEX root = /home/renaud/Documents/EPL/tfe/latex/tfe.tex
\section{Stieltjes integral} \label{app:stieltjes}
% Sources : wolfram world http://mathworld.wolfram.com/StieltjesIntegral.html
% wikipedia : https://en.wikipedia.org/wiki/Riemann%E2%80%93Stieltjes_integral
The Riemann-Stieltjes integral is a generalization of the Riemann integral. Let $f$ and $g$ be real-valued functions defined on a closed interval $[a,\,b]$. The Riemann-Stieltjes integral of $f$ with respect to $g$ is denoted
\begin{equation} \label{eq:stieltjes}
	\int_a^b f(t) \rm dg(t).
\end{equation}
Consider a partition of the interval
\begin{equation}
	a = t_0 < t_1 < \dots < t_{n-1} < t_n = b,
\end{equation}
and define
\begin{equation}
	h_n \triangleq \max_{i \in \{1,2,\dots,n\}} (t_i - t_{i-1}).
\end{equation}
Now take the Riemann sum
\begin{equation}
	\sum_{i=1}^{n} f(\tau_i)[g(t_i) - g(t_{i-1})],
\end{equation}
with $\tau_i \in [t_{i-1},\,t_i]$. If the sum tends to a fixed number $I$ as $n \rightarrow \infty$ and $h_n \rightarrow 0$, then
\begin{equation}
	\int_a^b f(t) \rm dg(t) = I.
\end{equation}
% If the function $g'$ is such that
% \begin{equation}
% 	g(t) = g(t_0) + \int_{t_0}^t g'(t') \rm dt',
% \end{equation}
% where the integral is the Lebesgue integral (i.e. if $g$ is absolutely continuous), then
% \begin{equation}
% 	\int_a^b f(t) \rm dg(t) = \int_a^b f(t) g'(t) \rm dt,
% \end{equation}
% where the second integral is a classical Riemann integral.


\section{Orthogonality and the Gram Schmidt process}

\begin{outcome}
  \begin{enumerate}
  \item Determine if a given set is orthogonal or orthonormal.
  \item Determine if a given matrix is orthogonal.
  \item Given a linearly independent set, use the Gram-Schmidt Process
    to find corresponding orthogonal and orthonormal sets.
  \item Find the orthogonal projection of a vector onto a subspace.
  \item Find the least squares approximation for a collection of
    points.
  \end{enumerate}
\end{outcome}


% !TEX root = Main.tex
\section{Neural Networks}
$F^\sigma:\mathbb{R}^n \rightarrow \mathbb{R}^m, F_j^\sigma(x) = \sigma(w_j^\top x)$ for $j=1,..,m$\\
\textbf{Activation fuctions}: logistic function $\sigma(x)=\frac{1}{1+e^{-x}}$, $\sigma'(x)=\sigma(x)(1-\sigma(x))$, ReLu $\phi(z)=max(0,z)$ \\
\textbf{Output layer}: linear regression: $\hat{\mathbf{y}} = \mathbf{W}^L\mathbf{x}^{L-1}$\\
binary classification (logistic):\\
$\hat{y_1} = \text{P}[Y=1|\mathbf{x}] = \frac{1}{1 + \exp[-\langle \mathbf{w}_1^L,\mathbf{x}^{L-1}\rangle]}$\\
multiclass (soft-max):\\
$\hat{y_k} = \text{P}[Y=k|\mathbf{x}]= \frac{\exp[\langle \mathbf{w}_k^L,\mathbf{x}^{L-1}\rangle]}{\sum_{m=1}^{K}{\exp[\langle \mathbf{w}_m^L, \mathbf{x}^{L-1}\rangle]}}$.\\
\textbf{Loss function} squared loss: $\frac{1}{2}(y - \hat{y})^2$\\
cross-entropy loss: $-y \log \hat{y} - (1-y)\log(1-\hat{y})$.\\
\textbf{Regularization}: add $l_2$-regularizer to objective or add drop-out layers. \\
\textbf{Units and Layers}: layer-to-layer fwd. prop. notation: $\mathbf{x}^{(l)} = \sigma^{(l)}\left(\mathbb{W}^{\left(l\right)}\mathbf{x}^{\left(l-1\right)}\right)$
where $y = x^{(L)}$ is the output activation vector.

\subsection*{Backpropagation}
\textbullet Use SGD to optimize over weights: $\theta \leftarrow \theta - \eta \nabla_\theta l(y_t;y(x_t;\theta))$ for $t=\{1,..,T\}$
\textbullet We want to know $\partial l / \partial w_{ij}^{(l')} $ i.e how does changing weights affect the loss.
\textbullet three steps:
\begin{inparaenum}[\color{red} 1.]
    \item how does output $y$ affect loss
    \item how do activities of units affect each other resp. $y$.
    \item how do weights affect activities of units.
\end{inparaenum}
\textbullet
\begin{inparaenum}[\color{red} 1.]
\item $\nabla_yl=\frac{\partial}{\partial y}l(y^*,y)=..$
\item $\frac{\partial\mathbf{x}^{(l)}}{\partial\mathbf{x}^{(l-n)}} = \mathbf{J}^{(l)}\cdot\frac{\partial\mathbf{x}^{(l-1)}}{\partial\mathbf{x}^{(l-n)}}=\mathbf{J}^{(l)}\cdot\mathbf{J}^{(l-1)}\cdots\mathbf{J}^{(l-n+1)}$
where $\mathbf{x}$ = prev. layer activation, $\mathbf{x^+}$ = next layer activation. Jacobian matrix $\mathbf{J}$ = $J_{ij}$ of mapping $\mathbf{x}\rightarrow\mathbf{x^+}$, $\mathbf{x_i^+} = \sigma(\mathbf{w}_i^\top\mathbf{x})$, $J_{ij} = \frac{\partial \mathbf{x_i^+}}{\partial \mathbf{x}_j} = w_{ij}\cdot\sigma'(\mathbf{w}_i^\top\mathbf{x})$.
\item $\frac{\partial x_i^+}{\partial w_{ij}} = \sigma'(w_i^\top x)x_j$
\end{inparaenum}

\textbullet Perform forward pass to compute activities for all units. Compute gradient of objective wrt output layer activites. Propagate gradient info back from output to inputs. Compute local gradients of activities wrt weights.
\subsection*{Convolutional Neural Networks}
\textbullet \textbf{Convolution step}: primary purpose is to extract features from the input image. Parameter/weight sharing = a kernel is used on multiple locations of the image with the same weights. Sparse interactions = by making kernel smaller than input. Discrete convolution operator $s[i,j]=(I*K)[i,j] = \sum_m \sum_n I[m,n]K[i-m,j-n]$ where I is the image and K the kernel. Note that arguments are commutative. \\
\textbullet \textbf{Pooling step}: reduce dim of each feature map e.g by max, sum, average over a predefined spatial neighborhood. Why? Scale invariant representation of image, less params/less overfitting.


\section{The Cayley-Hamilton Theorem}

\begin{outcome}
  \begin{enumerate}
  \item For a square matrix $A$, find a polynomial $p(x)$ such that $p(A)=0$.
  \end{enumerate}
\end{outcome}

In this section, we will consider the so-called
\textbf{Cayley-Hamilton theorem}%
\index{Cayley-Hamilton theorem}%
\index{characteristic polynomial!Cayley-Hamilton theorem}%
\index{matrix!Cayley-Hamilton theorem}. It states that every square
matrix is a root of its own characteristic polynomial. We use the
following notation. If
\begin{equation*}
  p(\eigenvar) = a_n\eigenvar^n + a_{n-1}\eigenvar^{n-1} + \ldots +
  a_1\eigenvar + a_0
\end{equation*}
is a polynomial, we denote by $p(A)$ the matrix defined by
\begin{equation*}
  p(A) = a_nA^n + a_{n-1}A^{n-1} + \ldots + a_1A + a_0I.
\end{equation*}
The explanation for the last term is that $A^0$ is interpreted as $I$,
the identity matrix.

\begin{theorem}{Cayley-Hamilton theorem}{cayley-hamilton}
  Let $A$ be a square matrix and let $p(\eigenvar)=\det(A-\eigenvar
  I)$ be its characteristic polynomial. Then $p(A)=0$.
\end{theorem}

Before we prove this theorem, we consider an example.

\begin{example}{Cayley-Hamilton theorem}{cayley-hamilton}
  Let
  \begin{equation*}
    A = \begin{mymatrix}{rr}
      3 & 4 \\
      -1 & 2 \\
    \end{mymatrix}.
  \end{equation*}
  Find the characteristic polynomial $p(\eigenvar)$, and compute
  $p(A)$.
\end{example}

\begin{solution}
  The characteristic polynomial is
  \begin{equation*}
    p(\eigenvar)
    ~=~ \det(A-\eigenvar I)
    ~=~ \begin{absmatrix}{cc}
      3-\eigenvar & 4 \\
      -1 & 2-\eigenvar \\
    \end{absmatrix}
    ~=~ (3-\eigenvar)(2-\eigenvar) - (-1)4
    ~=~ \eigenvar^2 - 5\eigenvar + 10.
  \end{equation*}
  Applying the characteristic polynomial to $A$, we get
  \begin{eqnarray*}
    p(A) ~=~ A^2 - 5A + 10I
    &=& \begin{mymatrix}{rr}
      3 & 4 \\
      -1 & 2 \\
    \end{mymatrix}^2
    - 5 \begin{mymatrix}{rr}
      3 & 4 \\
      -1 & 2 \\
    \end{mymatrix}
    + \begin{mymatrix}{rr}
      10 & 0 \\
      0 & 10 \\
    \end{mymatrix} \\
    &=& \begin{mymatrix}{rr}
      5 & 20 \\
      -5 & 0 \\
    \end{mymatrix}
    - \begin{mymatrix}{rr}
      15 & 20 \\
      -5 & 10 \\
    \end{mymatrix}
    + \begin{mymatrix}{rr}
      10 & 0 \\
      0 & 10 \\
    \end{mymatrix}
    ~=~ \begin{mymatrix}{rr}
      0 & 0 \\
      0 & 0 \\
    \end{mymatrix},
  \end{eqnarray*}
  just as predicted by the Cayley-Hamilton theorem.
\end{solution}

The remainder of this section is devoted to the proof of the
Cayley-Hamilton theorem. Readers who are not interested in the proof
can skip this material. We begin with a lemma:

\begin{lemma}{Polynomials with matrix coefficients}{polynomial-matrix-coefficient}
  Let $A_0,\ldots,A_m$ be $n\times n$-matrices and assume that for all
  scalars $\eigenvar$,
  \begin{equation*}
    A_0 + A_1\eigenvar + \ldots + A_m\eigenvar^m = 0.
  \end{equation*}
  Then each $A_i = 0$.
\end{lemma}

\begin{proof}
  Multiply by $\eigenvar^{-m}$ to obtain
  \begin{equation*}
    A_0 \eigenvar^{-m} + A_1 \eigenvar^{-m+1} + \ldots + A_{m-1}\eigenvar^{-1} + A_m = 0.
  \end{equation*}
  Now let $\abs{\eigenvar}\rightarrow\infty$ to obtain $A_m = 0$. With
  this, multiply by $\eigenvar$ to obtain
  \begin{equation*}
    A_0 \eigenvar^{-m+1} + A_1 \eigenvar^{-m+2} + \ldots + A_{m-1} = 0.
  \end{equation*}
  Now let $\abs{\eigenvar}\rightarrow\infty$ to obtain $A_{m-1} =
  0$. Continue multiplying by $\eigenvar$ and letting
  $\eigenvar\to\infty$ to obtain $A_i=0$ for all $i$.
\end{proof}

The following is a simple consequence of the lemma.

\begin{corollary}{}{polynomial-matrix-coefficient}
  Let $A_i$ and $B_i$ be $n\times n$-matrices and suppose that
  \begin{equation*}
    A_0 + A_1\eigenvar + \ldots + A_m\eigenvar^m =
    B_0 + B_1\eigenvar + \ldots + B_m\eigenvar^m
  \end{equation*}
  for all $\eigenvar$. Then for any $n\times n$-matrix $C$,
  \begin{equation*}
    A_0 + A_1C + \ldots + A_mC^m =
    B_0 + B_1C + \ldots + B_mC^m.
  \end{equation*}
\end{corollary}

\begin{proof}
  Subtracting the right-hand side from the left-hand side and using
  Lemma~\ref{lem:polynomial-matrix-coefficient}, we get that $A_i=B_i$
  for all $i$. But then the conclusion immediately follows.
\end{proof}

With this preparation, it is now relatively easy to prove the
Cayley-Hamilton theorem.

\begin{proofof}{of the Cayley-Hamilton Theorem} Let $A$ be an
  $n\times n$-matrix, and let $p(\eigenvar)=\det(A-\eigenvar I)$ be
  its characteristic polynomial.  Let $\adj(A-\eigenvar I)$ be the
  adjugate of the matrix $A-\eigenvar I$ (see
  Section~\ref{sec:adjugate} for the definition of the
  adjugate). Since each of the entries of the adjugate is a cofactor
  of $A-\eigenvar I$, the entries are polynomials in $\eigenvar$ of
  degree at most $n-1$. Therefore, the adjugate can be written in the
  form
  \begin{equation*}
    \adj(A-\eigenvar I) = C_0 + C_1\eigenvar + \ldots + C_{n-1}\eigenvar^{n-1}.
  \end{equation*}
  By Theorem~\ref{thm:inverse-and-determinant}, we have
  \begin{equation*}
    \det(A-\eigenvar I)\,I = (A-\eigenvar I) \, \adj(A-\eigenvar I),
  \end{equation*}
  or equivalently,
  \begin{equation*}
    p(\eigenvar)\,I =
    (A-\eigenvar I)\,(C_0 + C_1\eigenvar + \ldots + C_{n-1}\eigenvar^{n-1}).
  \end{equation*}
  Since this equation holds for all $\eigenvar$,
  Corollary~\ref{cor:polynomial-matrix-coefficient} may be
  used. Therefore, if $\eigenvar$ is replaced with $A$, the two sides
  will be equal. Thus
  \begin{equation*}
    p(A)\,I = (A-A)\,(C_0 + C_1A + \ldots + C_{n-1}A^{n-1}) = 0.
  \end{equation*}
  It follows that $p(A)=0$, concluding the proof of the
  Cayley-Hamilton Theorem.
\end{proofof}


\section{Principal Component Analysis}
To reduce the dimensionality and increase the variance, a Principal Component Analysis (PCA) is used. The PCA transforms the data into a new set of dimensions with a minimum amount of correlation. This new feature space contains the dimensions, which are responsible for most of the variance within the data. Hence the covariance is maximized. The covariance matrix of a vector $x$ of length $n$ is given as
\[
    \mathbf{S} = \frac{1}{N} \sum_{i=1}^{n}{(\mathbf{x}_i - \overline{\mathbf{x}}) (\mathbf{x}_i - \overline{\mathbf{x}})^\mathsf{T}}
\]
with $\overline{\mathbf{x}}$ as mean value of the vector. A PCA resulting in one dimension would always result in the eigenvector $\mathbf{u}_1$ with the largest eigenvalue $\lambda_1$ of the covariance matrix
\[
    \mathbf{S} \mathbf{u_1} = \lambda_1 \mathbf{u}_1.
\]
The eigenvector gives then the transformation of the PCA. More dimensions could be calculated by defining Iteratively the eigenvectors. For higher dimensions also more efficient algorithms are known. Details on the theory can be found in \cite[p.~561]{bib:Bishop2006}. Using the PCA\footnote{Using sklearn.decomposition.PCA from scikit-learn (http://scikit-learn.org/)}, we reduced our features to $d=5$ dimensions.\\\\
To get a data point for one sample, the PCA of all the features in the $n$ blocks can be seen as one long feature vector of length $d \cdot n$. Consequently, the features of each sample can be considered as a point in a $d \cdot n$ dimensional space.\\
All feature data and PCA-reduced feature data are saved as Pandas DataFrames. This reduces the whole data set from about 560~Mb of wave files to a 60~Mb HD5 database. Calculating this takes about 5 minutes on a modern computer\footnote{The multiprocessing module in Python seemed to have a problem with Windows, thus forcing the Windows version to use only one core, which would quadruple the runtime.}.


\section{Projection Pursuit}
One solution is projection-pursuit. It assumes a model of the form
\[
f(X_1,\dots,X_n) = \sum_{j=1}^p f_j\{\bg{\alpha}_j'\bX\}
\]
where $\bg{\alpha}_j ' \bX$ denotes a one dimensional projection of the
vector $(X_1,\dots,X_p)'$ and $f_j$ is an arbitrary function of this
projection.

The model builds up the regression surface by estimating these
univariate regressions along carefully chosen projections defined by
the $\bg{\alpha}_k$. Thus for $K=1$ and $p=2$ the regression surface
looks like a corrugated sheet and is constant in the directions
orthogonal to $\bf{\alpha}_k$.

If you don't see how this solves the problem of dimensionality, the
next section will help you understand.


\subsection{Diagonalization}\label{subsec:diagonalization}

\begin{definition}\label{def:diagonal_matrix}
  We say that the square \hyperref[def:array/matrix]{matrix} \( \{ a_{i,j} \}_{i,j=1}^{n,n} \) over a field \( \BbbK \) is \term{diagonal} if \( a_{i,j} = 0 \) whenever \( i \neq j \), that is, only the main diagonal has nonzero elements:
  \begin{equation*}
    \begin{pmatrix}
      a_{1,1} & 0       & \cdots & 0       \\
      0       & a_{2,2} & \cdots & 0       \\
      \vdots  & \vdots  & \ddots & \vdots  \\
      0       & 0       & \cdots & a_{n,n}
    \end{pmatrix}.
  \end{equation*}

  If all elements along the main diagonal are \( 1 \), we call the matrix the \term{identity matrix} of order \( n \) and denote it by
  \begin{equation*}
    E_n \coloneqq
    \begin{pmatrix}
      1      & 0      & \cdots & 0      \\
      0      & 1      & \cdots & 0      \\
      \vdots & \vdots & \ddots & \vdots \\
      0      & 0      & \cdots & 1
    \end{pmatrix}.
  \end{equation*}
\end{definition}

\begin{definition}\label{def:diagonalizable_matrix}
  A matrix over a field is \term{diagonalizable} if it is isomorphic to a diagonal matrix.
\end{definition}

\begin{theorem}[Spectral theorem for matrices]\label{def:spectral_theorem_for_matrices}\mcite[thm. 6.5]{Knapp2016BasicAlgebra}
  If \( R \) is a ring and \( 2 \) is a unit in \( R \), every symmetric matrix over \( R \) is diagonalizable.
\end{theorem}


\chapter{Compound Interest}

When you loan money to someone, you typically charge them some sort of
interest. The most common loan of this sort is what the bank calls a
``savings account''.  Any money you put in the account is loaned to
the bank. The bank then lends it to someone else, who pays interest to
the bank. And the bank gives some of that interest to you.

However, what if you leave the interest in your account? And you start
making \textit{interest on the interest}? This is known as
\textit{compound interest}.\index{compound interest}

\section{An example with annual interest payments}

Lets say that you put \$1000 in a savings account that pays 6\%
interest every year. How much money would you have after 12 years?
Let's make a spreadsheet.

\includegraphics[width=0.4\textwidth]{StartInterest.png}

Create a new spreadsheet and edit the cells to look like this.  All
the cells in row 1 - 4 are just values: just type in what you
see.

The fifth row is all formulas:

\begin{tabular}{c | c | c}
  After year & Interest & Balance \\
  \hline
  = A4 + 1 & = B\$1 * C4 & = C4 + B5 \\
\end{tabular}

The interest rate field should be formatted as a percentage. One thing
to know when dealing with percentages in the spreadsheet: if the field
says ``600\%'', its value is 6.

The cells in the Interest and Balance column should be formatted as currency.

You are about to make a bunch of copies of the cells in the fifth row,
so make sure they look right.

Click on A5 and shift click on C5 to select all three cells. Drag the
lower-right corner down to fill the rows 6 - 15.

\includegraphics[width=0.5\textwidth]{CopiedCellsInterest.png}

Look at the numbers.  The first interest payment is \$60, but the last
is \$113.90. Your balance has more than doubled!

\section{Exponential Growth}

We figured this out numerically by repeatedly multiplying the balance
by the interest rate.  What if you wanted to know what the balance
would be $n$ years after investing $P_0$ dollars with an annual interest
rate of $r$? (Note that $r$ in our example would be 0.06, not 6.0.)

Each year, the balance is multiplied by $1 + r$, so after one year,
$P_0$ would become $P_0 \times (1 + r)$.  The next year you would multiply
this number by $(1 + r)$ again: $P_0 \times (1 + r) \times (1 + r)$. The
next year? $P_0 \times (1 + r) \times (1 + r) \times (1 + r)$ See the
pattern? If we define $P_n$ be this balance after $n$ years, then

$$P_n = P_0 (1+r)^n$$

Because $n$ is an exponent, we call this \textit{exponential growth}.
And there are few things as terrifying to a scientist as
the phrase ``The population is undergoing exponential growth''.\index{exponential growth}

\section{Sensitivity to interest rate}

For most people, the first surprising thing about compound interest is
how quickly your money grows after a few years.  The second thing that
is surprising is how much difference a small change in the percentage
rate makes.

Lets add another set of columns that shows what happens to your money
if you convince the bank to pay you 8\% instead of 6\%.

Copy everything from columns B and C:

\includegraphics[width=0.6\textwidth]{CopyForSecondInterest.png}

Now edit the second interest rate to be 8\%:

\includegraphics[width=0.6\textwidth]{AtBiggerInterestRate.png}


\section{Sandwich Theorem}
We can use the Sandwich Theorem to indirectly find limits by "sandwiching" the function in question between two functions we do know the limit of.
If these two sandwiching functions go to the same value in the limit, then so to must the function in question.
\begin{theorem}[The Sandwich Theorem]
	If $g(x) \leq f(x) \leq h(x)$ and $\lim_{x \to c}{g(x)} = \lim_{x\to c}{h(x)} = L$, then $\lim_{x \to c}{f(x)} = L$.
\end{theorem}

\begin{example}
	Evaluate the following limit
	\begin{equation*}
		\lim_{\theta \to 0}{\frac{\sin{\theta}}{\theta}}
	\end{equation*}
\end{example}
\begin{answer}
	We'll need to use some geometric ideas to solve this limit.
	Consider the following on a unit circle.

	\begin{figure}[H]
		\label{sin_limit_proof}
		\centering
		\includegraphics[width = 0.5\textwidth]{./limits_continuity/sin_limit_proof.png}
		\caption{\hyperref{}{}{}{Triangle with internal angle $\theta$ inside a unit circle.}}
	\end{figure}

	We can see that the area of the swept arc is between the two triangle with base of length 1 and heights of $\sin{\theta}$ and $\tan{\theta}$.
	So, we can write the following inequality.
	\begin{align*}
		\frac{1}{2}\sin{\theta} &\leq \frac{1}{2}\theta \leq \frac{1}{2}\frac{\sin{\theta}}{\cos{\theta}} \\
		\sin{\theta} \leq \theta &\leq \frac{\sin{\theta}}{\cos{\theta}}
	\end{align*}

	Taking the reciprocal of each part and multiplying by $\sin{\theta}$,
	\begin{equation*}
		1 \geq \frac{\sin{\theta}}{\theta} \geq \cos{\theta}.
	\end{equation*}

	Taking the limit of as $\theta$ approaches 0 of each term,
	\begin{align*}
		1 \geq & \lim_{\theta \to 0}{\frac{\sin{\theta}}{\theta}} \geq \lim_{\theta\to 0}{\cos{\theta}} \\
		1 \geq & \lim_{\theta \to 0}{\frac{\sin{\theta}}{\theta}} \geq 1.
	\end{align*}

	So, by the Sandwich Theorem,
	\begin{equation*}
		\lim_{\theta \to 0}{\frac{\sin{\theta}}{\theta}} = 1.
	\end{equation*}
\end{answer}


\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{parskip}
\usepackage{graphicx}
\usepackage{bm}

\DeclareMathOperator*{\E}{\mathbb{E}}
\newcommand{\R}{\mathbb{R}}

\begin{document}
\title{Stuff I Should Know for Bayesian Statistics}
\author{D. Aaron Hillegass \\hillegass@gatech.edu}
\maketitle

\section*{Normal Distribution}

A normal distribution has two parameters: the mean ($\theta$) and the variance ($\sigma^2$). $x$ can go from $-\infty$ to $\infty$. Its density is given by
\begin{equation*}
p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x - \theta)^2}{2\sigma^2}}
\end{equation*}

The conjugate prior of the mean of a normal distribution is another normal distribution. Specifically, if $x_1, \dots, x_n$, are identical and independent with a normal distribution $\mathcal{N}(\theta, \sigma^2)$ and the prior is $\theta \sim  \mathcal{N}(\theta_0, \sigma_0^2)$,
\begin{equation*}
\theta | x_1,\ldots,x_n \sim \mathcal{N}\left( \frac{\frac{\sigma^2}{n}\theta_0 + \sigma_0^2\bar{x}}{\frac{\sigma^2}{n}+\sigma_0^2},\left( \frac{n}{\sigma^2} + \frac{1}{\sigma_0^2} \right)^{-1} \right)
\end{equation*}

The conjugate prior of the variance of a normal distribution is an inverse gamma distribution.

\section*{Gamma Distribution}

The gamma distribution has two parameters: $\alpha$ and $\beta$. $x$ can go from $0$ to $\infty$ Its density is given by
\begin{equation*}
p(x) =   \frac{ \beta^\alpha}{\Gamma(\alpha)} x^{\alpha-1} e^{-\beta x} \quad \text{ for } x > 0 \text{ and } \alpha, \beta > 0,
\end{equation*}

The mean of the distribution is
\begin{equation*}
E[x] = \frac{ \alpha}{\beta}
\end{equation*}

If $\alpha \geq 1$, the mode is
\begin{equation*}
\frac{ \alpha - 1}{\beta}
\end{equation*}

\section*{Inverse Gamma Distribution}

An inverse gamma distribution has two parameters: $\alpha$ and $\beta$. $x$ can go from $0$ to $\infty$. Its density is given by
\begin{equation*}
p(x) =
\frac{\beta^\alpha}{\Gamma(\alpha)}(1/x)^{\alpha + 1}e^{\left(-\frac{\beta}{x}\right)}
\end{equation*}

If $\alpha > 1$, the mean of the distribution is
\begin{equation*}
E[x] = \frac{ \beta}{\alpha - 1}
\end{equation*}

The mode is
\begin{equation*}
\frac{ \beta}{\alpha + 1}
\end{equation*}

\section*{Binomial Distribution}

The binomial distribution is a discrete distribution that takes one parameter: $\pi$ is the probability of one success in one try. The probability of getting exactly $k$ successes in $n$ trials is given by
\begin{equation*}
p(k) = \binom{n}{k}\pi^k(1-\pi)^{n-k}
\end{equation*}

The mean is $n\pi$.

The conjugate prior of $\pi$ is the Beta distribution.

\section*{Beta Distribution}

The beta distribution takes two parameters: $\alpha$ and $\beta$. Its density is given by
\begin{equation*}
p(x) = \frac{1}{Beta(\alpha,\beta)} x^{\alpha-1}(1-x)^{\beta-1}
\end{equation*}

The mode is
\begin{equation*}
\frac{\alpha - 1} {\alpha + \beta - 2}
\end{equation*}

\section*{Negative Binomial Distribution}

The negative binomial distribution is a discrete probability distribution that takes two parameters: $r$ and $p$. For any $k \in 0, 1,2,\ldots$,
\begin{equation*}
p(k) = \binom{k+r-1}{k} (1-p)^r p^k
\end{equation*}

The mean is
\begin{equation*}
\E \left( p(k) \right) = \frac{pr}{1-p}
\end{equation*}

If $r \leq 1$, the mode is 0. Otherwise it is $\lfloor \frac{p(r - 1)}{1 - p} \rfloor$

The variance is
\begin{equation*}
\frac{pr}{(1-p)^2}
\end{equation*}

\section*{Exponential Distribution}

The exponential distribution takes one parameter $\lambda$. Its probability density is
\begin{equation*}
p(x) = \begin{cases}
\lambda e^{-\lambda x} & x \ge 0, \\
0 & x < 0.
\end{cases}
\end{equation*}
Note that this is a special case of the gamma distribution.

Its cumulative distribution function is
\begin{equation*}
F(x) = \begin{cases}
1-e^{-\lambda x} & x \ge 0, \\
0 & x < 0.
\end{cases}
\end{equation*}

Its mean is $\frac{1}{\lambda}$. Its median is $\frac{\log{2}}{\lambda}$. Its mode is 0. Its variance is $\frac{1}{\lambda^2}$.

The conjugate prior for $\lambda$ in the exponential distribution is a gamma distribution.

\section*{Poisson Distribution}

The Poisson Distribution is a discrete distribution that takes one parameter: $\lambda$. Its distribution is given by
\begin{equation*}
p(x) = \frac{\lambda^x  e^{-\lambda}}{x!} \hspace{5mm}  x \in \lbrace 0, 1, \ldots\rbrace
\end{equation*}

$\lambda$ is both its mean and its variance.

The congugate prior for $\lambda$ is the gamma distribution

\section*{Pareto Distribution}

The Type 1 Pareto Distribution has two parameters $\alpha$ and $x_\mathrm{m}$, where $x_\mathrm{m}$ is the minimum possible value.  Both  $x_\mathrm{m}$ and $\alpha$ must be positive. Its density is
\begin{equation*}
p(x)= \begin{cases} \frac{\alpha x_\mathrm{m}^\alpha}{x^{\alpha+1}} & x \ge x_\mathrm{m}, \\ 0 & x < x_\mathrm{m}. \end{cases}
\end{equation*}

The mean of the distribution is
\begin{equation*}
\operatorname{E}(X)= \begin{cases} \infty & \alpha\le 1, \\
\frac{\alpha x_\mathrm{m}}{\alpha-1} & \alpha>1.
\end{cases}
\end{equation*}

Its cumulative distribution function is
\begin{equation*}
F(x) = \begin{cases}
1-\left(\frac{x_\mathrm{m}}{x}\right)^\alpha & x \ge x_\mathrm{m}, \\
0 & x < x_\mathrm{m}.\end{cases}
\end{equation*}

\section*{Gamma Function}

The gamma function is a generalization of factorial. That is, if $n$ is a positive integer
\begin{equation*}
\Gamma(n) = (n-1)!
\end{equation*}
But the gamma function is defined:
\begin{equation*}
\Gamma(z) = \int_0^\infty x^{z-1} e^{-x}\,dx
\end{equation*}

\section*{Beta Function}

The beta function is
\begin{equation*}
Beta(x,y) = \int_0^1t^{x-1}(1-t)^{y-1}\,dt = \frac{\Gamma(x)\,\Gamma(y)}{\Gamma(x+y)}
\end{equation*}

\section*{Choose Function}

The choose function is:
\begin{equation*}
\binom{n}{k} = \frac{n!}{k! (n-k)!}
\end{equation*}

\section*{Linear Regression}

If you are trying to use $p$ inputs ($x_1, \ldots, x_p$) to predict an ouput $y$, you could create a vector of coefficients $(\beta_0, \ldots, \beta_n)$ such that
\begin{equation*}
y \approx \beta_0 + \beta_1 x_1 + \ldots + \beta_p x_p
\end{equation*}

So how do you figure out what the $\beta$ is? You use the $n$ data points you have: The column vector $\boldsymbol{y} = (y_1, y_2, \ldots, y_n)'$ is the outcomes. The matrix $\boldsymbol{X}$ is the inputs.  Row $i$ of $\boldsymbol{X}$ starts with 1 and has the inputs $(x_{i,1}, x_{i,2},\ldots,x_{i, p})$ that correspond to $y_i$. Thus $X$ has $n$ rows and $p+1$ columns.

Given the data you have, the sum of the squares of the errors for a column vector $\beta$ (of length $p + 1$) is given by
\begin{equation*}
\boldsymbol{y}^T \boldsymbol{y} - 2 \beta^T \boldsymbol{X}^T \boldsymbol{y} + \beta^T \boldsymbol{X}^T \boldsymbol{X} \beta
\end{equation*}

To minimize that, you set $\beta$ to be:
\begin{equation*}
\beta = \left( \boldsymbol{X}^T \boldsymbol{X} \right)^{-1} \boldsymbol{X}^T \boldsymbol{y}
\end{equation*}

\section*{Logarithmic Identities}

\begin{equation*}
\log{\left( e^x \right)} = x \hspace{10mm} e^{\log{x}} = x
\end{equation*}

\begin{equation*}
\log{\left(  ab \right)} = \log a + \log b \hspace{10mm}  \log{\left(\frac{a}{b} \right)} = \log a - \log b
\end{equation*}

\begin{equation*}
\log{\left(  a^b \right)} = b\log a \hspace{10mm}  \log{\left(\sqrt[b]{a} \right)} = \frac{\log{a}}{b}
\end{equation*}

\begin{equation*}
a^{\log{b}} = b^{\log{a}}  \hspace{10mm}  x^{\frac{\log(a)}{\log(x)}} = a
\end{equation*}

\begin{equation*}
{d \over dx} \log x = {1 \over x } \hspace{10mm} \log x = \int_1^x \frac {1}{t} dt
\end{equation*}

\section*{Stuff Bayesians Like To Calculate}

\begin{equation*}
p(\theta | y)
=  \frac{p(y | \theta) p(\theta)}{p(y)}
=  \frac{p(y | \theta) p(\theta)}{\int p(y | \theta)p(\theta)d\theta}
\end{equation*}

\begin{equation*}
\E (\theta | y) =  \int \theta\ p(\theta| y)d\theta = \frac{\int \theta\ p(y|\theta)p(\theta)d\theta}{\int p(y|\theta)p(\theta)d\theta}
\end{equation*}

\begin{equation*}
var (\theta | y) = \E (\theta^2 | y) - \left(\E(\theta | y)\right)^2
\end{equation*}

\begin{equation*}
p(\theta \in A \ | \ y) = \int_{A} p(\theta | y)d\theta
\end{equation*}

The posterior predictive distribution:
\begin{equation*}
p(y \ | \ data) = \int p(y\ |\ \theta) p(\theta\ |\ data) d\theta
\end{equation*}

\end{document}


\subsection{Linear Approximations}
\noindent
Since $\partial z = f_x\partial x + f_y\partial y$, we can approximate $\Delta z$ (the change in any function) as $\Delta z \approx f_x\Delta x + f_y\Delta y$, since values of $f$ and the tangent plane are close. We an rewrite this approximation as a dot product: $\Delta z \approx \langle f_x, f_y\rangle \cdot \langle \Delta x, \Delta y \rangle$.\\

For example, say a cylindrical can has a radius $r=1$ and a height $h=5$. If the radius is increased by .1 and the height is increased by 1, what is the approximate $\Delta V$?
\begin{equation*}
	V(r,h) = \pi r^2 h, V_r = 2\pi rh \text{, and } V_h = \pi r^2
\end{equation*}
\begin{equation*}
	V_{r}(1,5) = 10\pi  \text{ and } V_{h}(1,5) = \pi
\end{equation*}
\begin{equation*}
	\Delta V \approx 10\pi(.1) + \pi(1) = 2\pi
\end{equation*}
Comparing this to the actual answer of $2.26\pi$, we see our approximation is decent.


\section{M/M/m/m Queues}
\label{sec:M-M-m-m-Queues}

A $M/M/m/m$ is a queue where \footnote{also called $m$-server loss system}
(i) the arrival process is Poissonian with rate $\lambda$,
(ii) the service process is Exponential with rate $\mu$,
(iii) there are $m$ servers,
(iv) there is no buffer,
(v) the scheduling policy is FCFS.

%\begin{figure}[tp]
%\label{fig:M-M-m-m-Queue}
%	\centering
%	\includegraphics{fig/M-M-m-m-Queue}
%	\caption{An M/M/m/m queue and its corresponding CTMC.}
%\end{figure}

The key question in these types of systems is determining the \textit{Blocking Probability} $P_{block}$, that is the probability that a job is dropped.

\begin{theorem}[State Probability]
\label{thm:M-M-m-m-Probability-State}

	For any $M/M/m/m$, the state probability is

	\begin{equation}
	\label{eqn:M-M-m-m-Probability-State}
	\pi_{i} = \Big( \frac{\lambda}{\mu} \Big)^{i} \frac{1}{i!} \pi_{0}
	\end{equation}

	with

	\begin{equation}
	\label{eqn:M-M-m-m-Queue-Probability-State-Zero}
	\pi_{0} = \Big[ \sum_{i=0}^{m} \Big( \frac{\lambda}{\mu} \Big)^{i} \frac{1}{i!} \Big]^{-1}
	\end{equation}

	\begin{proof}
		For a formal demonstration, see \cite{harchol2013performance} on page 256-257.
	\end{proof}
\end{theorem}

\begin{theorem}[Blocking Probability]
\label{thm:M-M-m-m-Probability-Block}
	For any $M/M/m/m$, the blocking probability is

	\begin{equation}
	\label{eqn:M-M-m-m-Probability-Block}
		P_{block} = \pi_{m} = \frac{\Big(\frac{\lambda}{\mu}\Big)^{m} \frac{1}{m!}}{\sum_{i=0}^{m} \Big( \frac{\lambda}{\mu} \Big)^{i} \frac{1}{i!}}
	\end{equation}

	\begin{proof}
		Follows from the fact that $P_{block} = \pi_{m}$.
	\end{proof}
\end{theorem}

\Cref{thm:M-M-m-m-Probability-Block} is called \textit{Erlang-B Formula}.

This formula exposes the \textit{Insensitivity Property}, that is it is independent of arrivals and service distribution and depends only on the mean quantities of distributions.
Insensitivity often occurs where there is no queue.

An easy way to remember \Cref{eqn:M-M-m-m-Probability-Block} is as follows.

\begin{equation*}
\begin{split}
	P_{block} & = \pi_{m} \cdot \frac{e^{-\frac{\lambda}{\mu}}}{e^{-\frac{\lambda}{\mu}}} \\
			  & = \frac{e^{-\frac{\lambda}{\mu}} \cdot \Big(\frac{\lambda}{\mu}\Big)^{m} \frac{1}{m!}}{\sum_{i=0}^{m} e^{-\frac{\lambda}{\mu}} \cdot \Big( \frac{\lambda}{\mu} \Big)^{i} \frac{1}{i!}} \\
			  & = \frac{\probability{X = m}}{\probability{X \leq m}}
\end{split}
\end{equation*}

where $X \sim Poisson(\frac{\lambda}{\mu})$.


\section{Recursion} % (fold)
\label{sec:recursion}

\begin{questions}
\titledquestion{Power} % (fold)
\label{sub:power}

Write a recursive function that computes $a^b$ for given $a$ and $b$, where
$b$ is an integer. Do not use \texttt{**}.

% titledquestion power (end)

\titledquestion{Recursive map and filter} % (fold)
\label{sub:recursive_map}

Write a recursive functions \texttt{myrecmap} and \texttt{myrecfilter}
that implement the \texttt{map} and \texttt{filter} functions using recursion.

% titledquestion Recursive map (end)

\titledquestion{Purify}
\label{sub:purify}

Write two functions, one that uses iteration (say a for loop), and the other using recursion, that achieve the following:
The input of the function is a list with integers. The functions return a (new) list with only the even integers in the list.

\titledquestion{Product}
\label{sub:product}

Write two functions, one that uses iteration, and the other using recursion, that achieve the following:
The input of the function is a list with numbers. The functions return the product of the numbers in the list.

\titledquestion{Factorial} % (fold)
\label{sub:factorial}

Write a recursive function to compute $n! = n \times (n-1) \times \ldots \times 1$.
Note that $0!$ is defined to equal $1$.

% titledquestion factorial (end)

\titledquestion{Recursive root finding} % (fold)
\label{sub:rec_root_finding}

In Exercise~\ref{sec:functions}.\ref{sub:root_finding} you wrote a function to find a root of a function $f$.
Now write a recursive function that finds the root of a function.

% titledquestion rec_root_finding (end)

\titledquestion{Collatz sequence} % (fold)
\label{sub:collatz_sequence}

Write a recursive implementation of a function that returns a list with
the Collatz sequence started at an arbitrary starting value.

Recall:
A Collatz sequence is formed as follows:
We start with some number $x_0$, and we find the next number in the sequence by
\[
    x_{i+1} = \begin{cases}
        x_i / 2 & \text{ if $x_i$ is even}\\
        3x_i + 1 & \text{ if $x_i$ is odd}
    \end{cases}
\]
If $x_i = 1$, we stop iterating and have found the full sequence.

% titledquestion collatz_sequence (end)

\titledquestion{Fibonacci sequence} % (fold)
\label{sub:fibonacci_sequence}

The Fibonacci sequence $\{F_i\}_i=0^\infty$ starts with $F_0=0, F_1=1$.
Every subsequent value in the sequence is the sum of the last elements in the
sequence:
\[
    F_{n} = F_{n-1} + F_{n-2}
\]

\begin{parts}
    \part Implement a non-recursive function that computes the $F_n$
    \part Implement a recursive function that computes $F_n$
    \part Compare the runtime for computing $F_{35}$ recursively versus
    non-recursively, and explain the difference.
    \part This does not mean a recursion is not feasible for this problem, only
    that the naive implementation isn't the best. We can get a better version
    using either of the following
    \begin{parts}
        \part Store values already calculated, say in a dictionary, so you can look them up instead of redoing the calculation.
        \part Generalizing the Fibonacci sequence to an additive
        sequence with arbitrary starting points $t_0$ and $t_1$, and finding
        a recursive algorithm to find the $n$th term in such a sequence.
        Note that finding the $n$th term in a sequence started from $t_0$ and $t_1$
        is the same as finding the $n-1$th term in a sequence started from $t_1$ and $t_0 + t_1$.
    \end{parts}
    Implement one of the above (or both).

\end{parts}

% titledquestion fibonacci_sequence (end)

\titledquestion{Palindromes} % (fold)
\label{sub:palindromes}

Given a string \texttt{t}, we are interested in finding the largest palindrome in \texttt{t},
where we are allowed to remove characters from \texttt{t}.
For example, consider the string \texttt{abcdba}, then the function should return \texttt{abcba}. Before you start coding, figure out the recursion on paper first.

Extra: you will notice that if you are not careful, this will take a very long time to compute for longer inputs. However, a simple modification can speed up your code so that it runs in $\mathcal{O}(n^2)$, where $n$ is the length of the input string. Find and implement this modification.\footnote{This exercise is inspired by an exercise in cs221.}

% titledquestion palindromes (end)

\titledquestion{Quicksort} % (fold)
\label{sub:quicksort}

There are many sorting algorithms, see for example \url{http://www.sorting-algorithms.com/}.
Quicksort is a well known (and quick) sorting algorithm that works as follows:

\begin{parts}
    \part Test whether the list is sorted, if not:
    \part Select one part of the list as \emph{pivot}, any element works.
    \part Create a new list, \texttt{left} and \texttt{right}, and put all
        elements smaller than the pivot in the \texttt{left} list, and all
        elements larger than the pivot in the \texttt{right} list.
    \part Recursively sort the left and right list, and return
        \texttt{sorted left + pivot + sorted right}.
\end{parts}

Implement the quicksort algorithm. However, first think about the following caveats
and how to get around them:
\begin{parts}
    \part Testing whether a list is sorted takes quite a bit of work, so we
        do not want to do this. Instead, we want to perform a much simpler check:
        what list is trivially sorted?
    \part What happens when the pivot element occurs multiple times in the list?
        How can we get around this?
\end{parts}

% titledquestion quicksort (end)
\end{questions}
% section recursion (end)


\chapter{Introduction to Neural Network}

\section{Biological Neural Networks}

\begin{description}
\item[Soma] cell body which processes incoming activations and converts input into output activation
\item[Dendrites] receptive zones that receive activation signals from other neurons
\item[Axon] transmission lines that send activation signals to other neurons
\item[Synapses] allow weighted signal transmission between the dendrites and axons
\end{description}

\section{Artificial Neuron}
Input signal vector:
$$\mathbf{x}=(x_1x_2\ldots x_p)^{T}$$
The weight vector:
$$\mathbf{w}=(w_1w_2\ldots w_p)^{T}$$
Total synaptic input:
\begin{equation*}
\begin{split}
u &= \sum_{i=1}^{p} w_ix_i - \theta \\
\mathbf{u} &= \mathbf{w}^{T} \mathbf{x} - \pmb{\theta}
\end{split}
\end{equation*}
Output Activation:
$$\mathbf{y} = \Phi(\mathbf{u})$$
\begin{center}where $\Phi$ is activation function\end{center}

\section{Typical ANN Activation Functions}
Threshold Activation Function:
$$\Phi (u)=
\begin{cases}
    1 & u>0 \\
    0 & otherwise
\end{cases}
$$
Linear Activation Function:
$$\Phi (u) = u$$
Ramp Activation Function:
$$\Phi (u) = max\{ 0, min\{ 1, u + 0.5 \} \}$$
Unipolar Sigmoid Activation Function:
$$\Phi (u) = \frac{a}{1+exp(-bu)}$$
Bipolar Sigmoid Activation Function:
\begin{equation*}
\begin{split}
\Phi (u) &= a\Bigg(\frac{1-exp(-bu)}{1+exp(-bu)}\Bigg) \\
&= a\Bigg(-1 + \frac{2}{1+exp(-bu)}\Bigg)
\end{split}
\end{equation*}

\section{ANN Architecture}
\begin{description}
\item[Two-Layer Feed-forward Network] input layer + output layer
\item[Multilayer Feed-forward Network] input layer + hidden layers + output layer
\item[Recurrent networks without hidden neurons] single layer of neurons with each neuron feeding its output signal back to the input layer
\end{description}

\section{Feed-forward ANN Analysis}
Weight matrix to hidden layer:
$$\mathbf{W = [w_1w_2 \ldots w_J]^{T}}$$
Synaptic inputs to hidden layer:
$$\mathbf{u=Wx}$$
\clearpage
\noindent Output of hidden layer:
$$\mathbf{y}=f(\mathbf{u})$$
\begin{center}where $f$ is activation function of hidden layer neurons \end{center}
Weight matrix to output layer:
$$\mathbf{V = [v_1v_2 \ldots v_J]^{T}}$$
Synaptic inputs to output layer:
$$s=Vy$$
Output:
$$\mathbf{z} = g(\mathbf{s})$$
\begin{center}where $g$ is activation function of output layer neurons \end{center}

\section{ANN Learning}
\begin{description}
\item[Supervised Learning] For each training input pattern, the network is presented with the correct target answer by a teacher
\item[Unsupervised Learning] For each training inputs, the network adjusts weights without knowing the correct target
\end{description}

\section{Characteristics of ANN}
\begin{itemize}
\item Parallel and distributed processing
\item Adaptiveness
\item Generalization
\item Fault-tolerance
\item Ease of construction
\end{itemize}

\section{ANN Limitations}
\begin{itemize}
\item Operational problems, computational time increase
\item Intractable system, black boxes
\end{itemize}


\documentclass[../algorithms.tex]{subfiles}
\begin{document}
% \chapter{Introduction}
In this chapter, I will mainly discuss some problems so that we can see how different approaches and algorithms can make difference on the time and space complexity.
\section{Maximum Subarray}
LeetCode 53. Maximum Subarray

Find the contiguous subarray within an array (containing at least one number) which has the largest sum. For example, given the array $[-2,1,-3,4,-1,2,1,-5,4]$, the contiguous subarray $[4,-1,2,1]$ has the largest sum = 6.
\subsection{Brute Force}
The brute force solution of this problem is to use two for loops, one pointer at the start position of the subarray, the other point at the end position of the subarray. Then we get the maximum sum of these subarries. The time complexity is $O(n^3)$, where we spent $O(n)$ to the sum of each subarray. The code is writen as:
\begin{lstlisting}[language=Python]
for i in range(n):
    for j in range(i+1,n):
\end{lstlisting}
However, if we can get the sum of each subarray with $O(1)$. Then we can lower the complexity to $O(n^2)$. Here one solution is to trade space for efficiency. the sum of subarray from index $i$ to $j$ is $sum(i,j)=sum(0,j)-sum(0,i)$. We can pre compute the accumulated sum to each index and save it in an array of the same size, which gives us $O(n^2)$ time complexity and $O(n)$ space complexity.
\subsection{Divide and Conquer}
To further improve the efficiency, we use divide and conquer, where we divide one array into two halves: the maximum subarray might located on the left size, or the right side, or some in the left side and some in the right size, which crossed the bound. $T(n) = max(T(left),T(right), T(cross))$, max is for merging and the T(cross) is for the case that the potential subarray across the mid point. For the complexity, $T(n)=2T(n/2)+n$, if we use the master method, it would give us $O(nlgn)$. With this solution, we use $O(lgn)$ space for the recursive function stack space.
\begin{lstlisting}[language=Python]
def maxSubArray(self, nums):
        """
        :type nums: List[int]
        :rtype: int
        """
        def getCrossMax(low,mid,high):
            left_sum,right_sum =0,0
            left_max,  right_max = -maxint, -maxint
            left_i,right_j=-1,-1
            for i in xrange(mid,low-1,-1): #[)
                left_sum+=nums[i]
                if left_sum>left_max:
                    left_max= left_sum
                    left_i = i
            for j in xrange(mid+1,high+1):
                right_sum+=nums[j]
                if right_sum>right_max:
                    right_max= right_sum
                    right_j = j
            return (left_i,right_j,left_max+right_max)

        def maxSubarray(low,high):
            if low==high:
                return (low,high, nums[low])
            mid = (low+high)//2
            rslt=[]
            #left_low, left_high, left_sum = maxSubarray(low,mid) #[low,mid]
            rslt.append(maxSubarray(low,mid)) #[low,mid]
            #right_low,right_high,right_sum = maxSubarray(mid+1,high)#[mid+1,high]
            rslt.append(maxSubarray(mid+1,high))
            #cross_low,cross_high,cross_sum = getCrossMax(low, mid, high)
            rslt.append(getCrossMax(low, mid, high))
            return max(rslt, key=lambda x: x[2])
        return maxSubarray(0,len(nums)-1)[2]
\end{lstlisting}
\subsection{Dynamic Programming}
Using dynamic programming: the $f$ memorize the maximum subarray value till $j$, $f[ j ] = max f [ j- 1] + S [ j ] , S [ j ]$. This would gave us $O(n)$ time complexity and $O(n)$ space complexity.
\subsection{Greedy Algorithm}
Because $sum(i,j)=sum(0,j)-sum(0,i)$, to till index $j$, we use $f(j)$ represents the maximum subarray value. which gives us relation $f(j) = sum(0,j)-min(Sum(0,i))_{i in [0,j] }, j>=1$
\subsection{Prefix Sum}
convert this problem to best time to buy and sell stock problem. $[0, -2, -1, -4, 0, -1, 1, 2, -3, 1]$, which is to find the maximum benefit, => O(n), use prefix$\_$sum, the difference is we set prefix$\_$sum to 0 when it is smaller than 0, $O(n)$. Or we can try two pointers.
\begin{lstlisting}
from   sys import maxint
class Solution(object):
    def maxSubArray(self, nums):
        """
        :type nums: List[int]
        :rtype: int
        """
        max_so_far = -maxint - 1
        prefix_sum= 0
        for i in range(0, len(nums)):
            prefix_sum+= nums[i]
            if (max_so_far < prefix_sum):
                max_so_far = prefix_sum

            if prefix_sum< 0:
                prefix_sum= 0
        return max_so_far
\end{lstlisting}
From this problem, we get a peek how using different methods can gradually improve the algorithms' performance.

\end{document}


%!TEX root=report.tex
\subsection{OLS}
OLS (Ordinary Least Squares) regression is used for finding the best linear unbiased estimator(BLUE) of $Y$ base on one or more exogenous variables $X$.
This is done by a matrix-vector product between a $\beta$ vector and $X$.
Since $X$ can be custom tailored for various purposes it is often referred to as the design matrix. The model is
\begin{align}
Y=X \beta +\epsilon && \text{, where } \mathrm{E}[\epsilon] = 0 \text{ and } \mathrm{D}[\epsilon] = \mathrm{D}[Y] = \sigma^2 I .
\end{align}

In the equation above the $\beta$-vector is the unknown parameter which needs to be determined while  $\epsilon$ is a vector containing the residuals which the model cannot account for.
Often the purpose of OLS is to predict $Y$, but in this case we want to analyze the individual $\beta_i$ elements.

\subsubsection{The solution to the OLS problem}
As the name sugest, the OLS method minimizes the sum of squared residuals ($\epsilon^T \epsilon$).
It is seen directly that $\epsilon = Y - X \beta$ and thus the following is obtained
\begin{equation}
\begin{split}
\epsilon^T\epsilon&=(Y-X\beta)^T (Y-X\beta)\\
&=(Y^T-\beta^T X^T) (Y-X\beta) \\
&=Y^T Y-\beta^T X^T Y-Y^T X \beta + \beta^T X^T X \beta \\
&=Y^T Y- 2\beta^T X^T Y+ \beta^T X^T X \beta.
\end{split}
\end{equation}

One can now differentiate with respect to the $\beta$ vector
\begin{equation}
\begin{split}
\frac{\partial \epsilon^T\epsilon}{\partial \beta}&=-2 X^T Y+2X^T X \beta=2(-X^T Y+X^T X \beta).
\end{split}
\end{equation}

Now  $\epsilon^T \epsilon$'s minimum is given by solving for $\frac{\partial \epsilon^T\epsilon}{\partial \beta} = 0$
\begin{equation}
\begin{split}
\frac{\partial \epsilon^T\epsilon}{\partial \beta} = 0 \Rightarrow 2(-X^T Y+X^T X \hat{\beta}) &= 0 \\
X^T X \hat{\beta}&=X^T Y \\
\hat{\beta}&=(X^T X)^{-1} X^T Y.
\end{split}
\end{equation}

The solution above is formally correct \cite[p.~12]{statistical-learning}, but if $X$ is badly conditioned $(X^T X)^{-1}$ might not be numerically stable (i.e., if the columns in $X$ are highly correlated leading to a near singular $X^T X$) \cite[p.~8]{aasbjerg-ls}.
To avoid this problem $X$ should be factorised and the solution reformulated; in this report we will exclusively do this using SVD
\begin{equation}
\begin{split}
\left( U \Sigma V^T\right)^T \left(U \Sigma V^T\right) \hat{\beta} &= \left(U \Sigma V^T\right)^T Y \\
V \Sigma^2 V^T \hat{\beta} &= V \Sigma U^T Y \\
\left(V \Sigma^{-2} V^T\right) V \Sigma^2 V^T \hat{\beta} &= \left(V \Sigma^{-2} V^T\right) V \Sigma U^T Y \\
\hat{\beta} &= V \Sigma^{-1} U^T Y.
\end{split}
\end{equation}

It should be noted, that when multiple $\hat{\beta}$ vectors need to be calculated (one vector for each spatial location on the surface) optimization is possible, by arranging $Y$ as a matrix with each column corresponding to a location.
$\hat{\beta}$ will then be a matrix containing all solutions instead of a vector containing only one solution.

\subsubsection{The ``Hat" matrix}
In the special case of the GRACE data, the $X$ matrix is identical for every position. This can be exploited by constructing a hat matrix $H$ which only depends on $X$ and projects $Y$ onto $\hat{Y}$ (puts the hat on $Y$).
\begin{equation}
\begin{split}
\hat{Y} &= X \hat{\beta} \Rightarrow \hat{Y} = X V \Sigma^{-1} U^T Y \\
\hat{Y} &= H Y \quad \text{, where } H =X(X^{T}X)^{-1}X^{T}= X V \Sigma^{-1} U^T.
\end{split}
\end{equation}

As earlier with $\beta$, the hat matrix can be calculated for all $Y$ vectors by vertically stacking $Y$ to form a matrix.

An important property of the $H$ matrix is that it is idempotent ($H^2 = H$) and symmetrical ($H^T = H$).
This is because $H$ is a projection matrix which projects $Y$ onto $\hat{Y}$.
Projecting $Y$ onto $\hat{Y}$ and then projecting again onto $\hat{Y}$, will obviously not change anything, because one is already in the $\hat{Y}$-plane, spanned by the columns of $X$.

\subsubsection{Root Mean Squared Error}

The ``Root Mean Squared Error'' (RMSE) is an indicator of how good an $Y$ estimate is.
It can be calculated as
\begin{equation}
\hat{\sigma} = \sqrt{\frac{\left(Y - \hat{Y}\right)^T \left(Y - \hat{Y}\right)}{n-m}},
\end{equation}

where $m$ is the number of parameters (elements in $\beta$) and $n$ is the number of observations (elements in $Y$).
RMSE is an estimate for the standard deviation of $Y$ \cite[theorem~3.4]{time-series-analysis} thus the symbol $\hat{\sigma}$.

\subsubsection{The variance of $\hat{Y}$}

The dispersion (variance-covariance) of $\hat{Y}$ can be calculated as
\begin{equation}
\mathrm{D}[\hat{Y}] = \mathrm{D}[H Y] = H \mathrm{D}[Y] H^T = \sigma^2 H^2 = \sigma^2 H.
\end{equation}

The variance of $\hat{Y_i}$ is given by the diagonal elements in $\mathrm{D}[\hat{Y}]$
\begin{equation}
\mathrm{Var}[\hat{Y_i}] = \sigma^2 H_{ii}.
\end{equation}

Because $\sigma^2$ is a scalar, the diagonal in $H$ is important to examine, since it can reveal potential elements (in our case points of time) with high variance in the predictions.

\subsubsection{The dispersion of $\hat{\boldsymbol\beta}$}

The dispersion of $\hat{\beta}$ is calculated as \cite[theorem~3.2]{time-series-analysis}:
\begin{equation}
\mathrm{D}[\hat{\beta}] = \sigma^2 (X^T X)^{-1} = \sigma^2 V \Sigma^{-2} V^T
\end{equation}

Since $\sigma^2$ is a scalar and dependent of $Y$, a similar spatial independent expression can be made, by removing the scalar $\sigma^2$  factor, thus looking exclusively at $V \Sigma^{-2} V^T$.

\subsubsection{p-values for $\hat{\boldsymbol\beta}$}

Assuming the residuals are normally distributed, the p-values for OLS parameters can be calculated using the student's t-distribution, with the t-score \cite[p.~172]{time-series-analysis}
\begin{align}
\mathrm{t} = \frac{\hat{\beta_i}}{\mathrm{SD}[\hat{\beta_i}]} && \text{where: } \mathrm{SD}[\hat{\beta_i}] = \sqrt{\mathrm{Cov}[\hat{\beta}]_{ii}} = \hat{\sigma} \sqrt{ (V \Sigma^{-2} V^T)_{ii} }.
\end{align}

Now by plugging the t-score into the Student's Cumulative distribution function ($\Phi_t$) with $N - p$ degrees of freedom, we get
\begin{equation}
p = 2 \cdot \Phi_t\left(\mathrm{abs}(t), N-p\right)
\end{equation}

The null-hypothesis is that $\beta_i = 0$ and the alternative hypothesis is $\beta_i \not = 0$.


\documentclass{article}
\title{Dual Derivations of ALO for Elastic Net}
\author{Yuze Zhou}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{algorithm}
\usepackage{subfigure}
\begin{document}
\maketitle
\section{Dual Problem of Elastic Net}
\paragraph{}The original problem for elastic net is to solve for $\hat{\beta}$ such that:
\begin{center}
$\hat{\beta} = \arg \min\limits_{\beta} \frac{1}{2}||y-X\beta||_{2}^{2} + \lambda_{1}||\beta||_{1}+\lambda_{2}||\beta||_{2}^{2}$
\end{center}
\paragraph{}By adding the Lagrangian, we get the formulation of $L$:
\begin{center}
$L = \frac{1}{2}||y-z||_{2}^{2} + \lambda_{1}||\beta||_{1}+\lambda_{2}||\beta||_{2}^{2}+u^{\tau}(z-X\beta)$
\end{center}
\paragraph{}The original problem is solving the primal of the Lagrangian such that $p^{*} = \min\limits_{\beta,z}\max\limits_{u}L$ and the dual formulation $d^{*} = \max\limits_{u}\min\limits_{\beta,z}L$, to minimize over $z$:
\begin{center}
$\frac{\partial L}{\partial z} = z -y +u = 0$\\
$y = u + z$\\
\end{center}
\paragraph{}Since $\beta$ is penalized element-wisely, we can minimize over $\beta$ by minimizing over each $\beta_{i}$, that is, we have to minimize $\lambda_{1}|\beta_{i}| + \lambda_{2}\beta_{i}^{2} - u^{\tau}X_{i}\beta$ for each dimension of $\beta$, where $X_{i}$ denotes the $i$th column of $X$, therefore:
\begin{center}
$\min\limits_{\beta}\lambda_{1}|\beta_{i}| + \lambda_{2}\beta_{i}^{2} - u^{\tau}X_{i}\beta = $\\
$ $\\
$= \left\{
\begin{aligned}
0 \quad if \quad |u^{\tau}X_{i}| \leq \lambda_{1}\\
-\frac{(\lambda_{1}-|u^{\tau}X_{i}|)^{2}}{4\lambda_{2}} \quad if \quad |u^{\tau}X_{i}| > \lambda_{1}\\
\end{aligned}
\right.
$
\end{center}
\paragraph{}By taking all the above to the Lagrangian, we could obtain the dual problem $d^{*}$ as:
\begin{center}
$d^{*} = \min\limits_{u} \frac{1}{2}||y-u||_{2}^{2} + \sum\limits_{j: |X_{j}^{\tau}u| > \lambda_{1}}\frac{(\lambda_{1}-|u^{\tau}X_{i}|)^{2}}{4\lambda_{2}}$
\end{center}
\paragraph{}The minimizer $\hat{u}$ could also be obtained from the dual problem through a proximal approach:
\begin{center}
$\hat{u} = \textbf{prox}_{R}(y) \quad where \quad R(u) = \sum\limits_{j: |X_{j}^{\tau}u| > \lambda_{1}}\frac{(\lambda_{1}-|u^{\tau}X_{i}|)^{2}}{4\lambda_{2}}$
\end{center}
\section{ALO Estimation for Elastic Net}
\paragraph{}By replacing the full data problem $y$ with $y_{\alpha} = y + (y_{i}^{/i}-y_{i})e_{i}$, where $y_{i}^{/i}$ is the true loo estimator and $e_{i}$ is the $i$th standard vector, and let $u^{/i} = \textbf{prox}_{R}(y_{\alpha})$, therefore:
\begin{align*}
0 &= e_{i}^{\tau}u^{/i}\\
& = e_{i}^{\tau}\textbf{prox}_{R}(y_{\alpha})\\
& \approx e_{i}^{\tau}(\textbf{prox}_{R}(y)+J_{R}(y)(y_{\alpha}-y))\\
& \approx \hat{u}_{i} + J_{ii}(y_{i}^{/i}-y_{i})
\end{align*}
\paragraph{}Here $J_{R}(y)$ denotes the Jacobian matric of the proximal operator at $y$, thus the alo estimator $\tilde{y}_{i}$ is obtained as
\begin{center}
$\tilde{y}_{i} = y_{i} - \frac{\hat{u}_{i}}{J_{ii}}$
\end{center}
\paragraph{}The Jacobian could locally be obtained as:
\begin{align*}
J_{R}(y) &= (I+\nabla^{2}R(\textbf{prox}_{R}(y)))^{-1}\\
&= (I + \nabla^{2}R(\hat{u}))^{-1}\\
&= (I + \frac{1}{2\lambda_{2}}X_{E}X_{E}^{\tau})^{-1}\\
\end{align*}
\paragraph{}Here $E = \{j:|X_{j}^{\tau}u|>\lambda_{1}\}$.
\end{document}


\chapter{Introduction}

\section{Metric space}

A metric $d$ on a set $X$, also called a distance function, is a function
that defines a distance between each pair of elements of the set. A
set with a metric is called a \textbf{metric space}.\\

Formally, $d : X \times X \rightarrow R$ is a metric if it is a function satisfying the following properties $\forall x, y, z \epsilon X$:
\begin{enumerate}
\item{Non-negativity : $d(x, y) \geq 0$}
\item{Indiscernability : $d(x, y) = 0$ iff $x = y$}
\item{Symmetry : $d(x, y) = d(y, x)$}
\item{Subadditivity : $d(x, y) + d(y, z) \geq d(x, z)$}
\end{enumerate}
\section{The Christofides' algorithm}
The Christofides' algorithm is an approximation algorithm for Metric TSP with an approximation ratio of $1.5$. Let G be a graph with $n$ points in the euclidean metric space. Below is a description of the steps involved in the Christofides' algorithm.
\vspace{0.7in}
\subsection{Pseudo code}
\textbf{Algorithm}
\begin{enumerate}
    \item{
        Find an MST of $G$, say $T$.
    }
    \item {Compute a minimum cost perfect matching, $M$, on the set of odd-degree vertices of $T$.}
    \item {
        Add $M$ to $T$ and obtain an Eulerian multi-graph $H$.
    }
    \item {Find an Euler tour, $E$ of this graph.
    }
    \item {Output the tour that visits vertices of $G$ in order of their first appearance in $E$.}
\end{enumerate}
\begin{figure}[h]
    \centering
    \caption{MST $T$ of $G$}
    \includegraphics[scale=0.3]{1.jpg}
\end{figure}
\begin{figure}[h]
    \centering
    \caption{Multi-graph $H$ of $G$}
    \includegraphics[scale=0.3]{2.jpg}
\end{figure}
\begin{figure}[h]
    \centering
    \caption{Euler tour $E$ of $H$}
    \includegraphics[scale=0.3]{3.jpg}
\end{figure}
\subsection{Time complexity}
\begin{enumerate}
    \item {Creating MST $T$ of $G$ : $O(nlogn)$}
    \item {Finding the minimum cost perfect matching $M$ : $O(n^3)$}
    \item {Creating multi-graph H : $O(n)$}
    \item {Finding Euler tour $E$ in $H$ : $O(n)$}
\end{enumerate}
% some text\cite{citation-1-name-here}, some more texts
% even more text\footnote{<footnote here>}, and even more.
% \section{Motivation}


\subsection{Gibb's sampling}

\subsubsection{Introduction}

As with Metropolis-Hastings, we want to generate samples for \(P(X)\) and use this to approximate its form.

We do this by using the conditional distribution. If \(X\) is a vector then we also have:

\(P(x_j|x_0,...,x_{j-1},x_{j+1},...,x_n)\)

We use our knowledge of this distribution.

Start with vector \(x_0\).

This has components \(x_{0,j}\)

To form the next vector \(x_1\) we loop through each component.

\(P(x_{1,0}|x_{0,0},x_{0,1},...,x_{0,n})\)

We use this to form \(x_{1,0}\)

However after th the first component we update this so it uses the updated variables.

\(P(x_{1,k}|x_{1,0},...,x_{1,k-1},x_{0,k},...,x_{0,n}\)

This means we only need to know the conditional distributions.