\lab{Markov Chains}{Markov Chains}
\label{lab:Markov}
\objective{
A Markov chain is a collection of states with specified probabilities for transitioning from one state to another.
They are characterized by the fact that the future behavior of the system depends only on its current state.
In this lab we learn to construct, analyze, and interact with Markov chains, then use a Markov-based approach to simulate natural language.}

\section*{State Space Models} % ===============================================

Many systems can be described by a finite number of \emph{states}.
For example, a board game where players move around the board based on dice rolls can be modeled by a Markov chain.
Each space represents a state, and a player is said to be in a state if their piece is currently on the corresponding space.
In this case, the probability of moving from one space to another only depends on the player's current location; where the player was on a previous turn does not affect their current turn.

Markov chains with a finite number of states have an associated \emph{transition matrix} that stores the information about the possible transitions between the states in the chain.
The $(i,j)$th entry of the matrix gives the probability of moving \textbf{from state $j$ to state $i$}.
Thus, each of the columns of the transition matrix sum to $1$.

\begin{info} % Column stochastic versus row stochastic.
A transition matrix where the columns sum to $1$ is called \emph{column stochastic} (or \emph{left stochastic}).
The rows of a \emph{row stochastic} (or \emph{right stochastic}) transition matrix each sum to $1$ and the $(i,j)$th entry of the matrix is the probability of moving from state $i$ to state $j$.
Both representations are common, but in this lab we exclusively use column stochastic transition matrices for consistency.
\end{info}

Consider a very simple weather model in which the weather tomorrow depends only on the weather today.
For now, we consider only two possible weather states: hot and cold.
Suppose that if today is hot, then the probability that tomorrow is also hot is 0.7, and that if today is cold, the probability that tomorrow is also cold is 0.4.
By assigning ``hot'' to the $0$th row and column, and ``cold'' to the $1$st row and column, this Markov chain has the following transition matrix.

\begin{align*}
\begin{blockarray}{ccc}
& \text{\textcolor{red}{hot today}} & \text{\textcolor{blue}{cold today}} \\
\begin{block}{c[cc]}
\text{\textcolor{red}{hot tomorrow}}   & 0.7 & 0.6 \topstrut \\
\text{\textcolor{blue}{cold tomorrow}} & 0.3 & 0.4 \botstrut \\
\end{block}\end{blockarray}
\end{align*}
%
The $0$th column of the matrix says that if it is hot today, there is a $70\%$ chance that tomorrow will be hot ($0$th row) and a $30\%$ chance that tomorrow will be cold ($1$st row).
The $1$st column says if it is cold today, then there is a $60\%$ chance of heat and a $40\%$ chance of cold tomorrow.

Markov chains can be represented by a \emph{state diagram}, a type of directed graph.
The nodes in the graph are the states, and the edges indicate the state transition probabilities.
The Markov chain described above has the following state diagram.

\begin{figure}[H] % 2-state chain.
\centering
\begin{tikzpicture}[normalcircle/.style={draw, circle, minimum size=1.5cm, thick, node distance=2.5cm}]
    % Place the circles
    \node[normalcircle] (Hot) {};
    \node[red] at (Hot) {hot};

    \node[normalcircle, right=of Hot] (Cold) {};
    \node[blue] at (Cold) {cold};

    % Draw loop, place number, draw line
    \draw[thick,->,>=stealth',red!70!black] (Hot)+(-.52,.52) arc (325:40:.35 and -.85);
    \node[left,red!70!black] at (Hot.west) [shift={+(-.5,0)}] {0.7};
    \draw[thick,->,>=stealth',red!70!black] (Hot.north east) -- node[above] {0.3} (Cold.north west);

    \draw[thick,<-,>=stealth',blue!70!black] (Cold)+(.52,.52) arc (325:40:-.35 and -.85);
    \node[right,blue!70!black] at (Cold.east) [shift={+(.5,0)}] {0.4};
    \draw[thick,->,>=stealth',blue!70!black] (Cold.south west) -- node[below] {0.6} (Hot.south east);
\end{tikzpicture}
\end{figure}

\begin{problem} % MarkovChain.__init__().
\label{prob:markov-chain-class-constructor}
Define a \li{MarkovChain} class whose constructor accepts an $n\times n$ transition matrix $A$ and, optionally, a list of state labels.
If $A$ is not column stochastic, raise a \li{ValueError}.
Construct a dictionary mapping the state labels to the row/column index that they correspond to in $A$ (given by order of the labels in the list), and save $A$, the list of labels, and this dictionary as attributes.
If there are no state labels given, use the labels $\begin{bmatrix}0 & 1 & \ldots & n-1\end{bmatrix}$.

For example, for the weather model described above, the transition matrix is
\begin{align*}
    A = \left[\begin{array}{cc}0.7 & 0.6 \\ 0.3 & 0.4\end{array}\right],
\end{align*}
the list of state labels is \li{["hot", "cold"]}, and the dictionary mapping labels to indices is \li{\{"hot":0, "cold":1\}}.
This Markov chain could be also represented by the transition matrix
\begin{align*}
    \widetilde{A}
    = \left[\begin{array}{cc}
        0.4 & 0.3 \\ 0.6 & 0.7
    \end{array}\right],
\end{align*}
the labels \li{["cold", "hot"]}, and the resulting dictionary \li{\{"cold":0, "hot":1\}}.
\end{problem}

\subsection*{Simulating State Transitions} % ----------------------------------

Simulating the weather model described above requires a programmatic way of choosing between the outgoing transition probabilities of each state.
For example, if it is cold today, we could flip a weighted coin that lands on tails $60\%$ of the time (guess tomorrow is hot) and heads $40\%$ of the time (guess tomorrow is cold) to predict the weather tomorrow.
The \emph{Bernoulli distribution} with parameter $p = 0.4$ simulates this behavior: $60\%$ of draws are $0$, and $40\%$ of draws are a $1$.

A \emph{binomial distribution} is the sum several Bernoulli draws: one binomial draw with parameters $n$ and $p$ indicates the number of successes out of $n$ independent experiments, each with probability $p$ of success.
In other words, $n$ is the number of times to flip the coin, and $p$ is the probability that the coin lands on heads.
Thus, a binomial draw with $n=1$ is a Bernoulli draw.

NumPy does not have a function dedicated to drawing from a Bernoulli distribution; instead, use the more general \li{np.random.binomial()} with $n=1$ to make a Bernoulli draw.

\begin{lstlisting}
>>> import numpy as np

# Draw from the Bernoulli distribution with p = .5 (flip one fair coin).
>>> np.random.binomial(n=1, p=.5)
0                             # The coin flip resulted in tails.

# Draw from the Bernoulli distribution with p = .3 (flip one weighted coin).
>>> np.random.binomial(n=1, p=.3)
0                             # Also tails.
\end{lstlisting}

For the weather model, if the ``cold'' state corresponds to row and column 1 in the transition matrix, $p$ should be the probability that tomorrow is cold.
So, if today is cold, select $p=0.4$; if today is hot, set $p=0.3$.
Then draw from the binomial distribution with $n=1$ and the selected $p$.
If the result is $0$, transition to the ``hot'' state; if the result is $1$, stay in the ``cold'' state.

% Consider again the simple weather model and suppose that today is hot.
% The column that corresponds to ``hot''in the transition matrix is $[0.7, 0.3]$.
% To determine whether tomorrow is hot or cold, draw from the binomial distribution with $n = 1$ and $p = 0.3$.
% If the draw is 1, which has $30\%$ likelihood, then tomorrow is cold.
% If the draw is 0, which has $70\%$ likelihood, then tomorrow is hot.

Using Bernoulli draws to determine state transitions works for Markov chains with two states, but larger Markov chains require draws from a \emph{categorical distribution}, a multivariate generalization of the Bernoulli distribution.
A draw from a categorical distribution with parameters $(p_1,p_2,\ldots,p_k)$ satisfying $\sum_{i=1}^k p_i = 1$ indicates which of $k$ outcomes occurs.
If $k=2$, a draw simulates a coin flip (a Bernoulli draw); if $k=6$, a draw simulates rolling a six-sided die. %, like rolling a weighted $k$-sided die.
Just as the Bernoulli distribution is a special case of the binomial distribution, the categorical distribution is a special case of the \emph{multinomial distribution} which indicates how many times each of the $k$ outcomes occurs in $n$ repeated experiments.
Use \li{np.random.multinomial()} with $n = 1$ to make a categorical draw.

\begin{lstlisting}
# Draw from the categorical distribution (roll a fair four-sided die).
>>> np.random.multinomial(1, np.array([1./4, 1./4, 1./4, 1./4]))
array([0, 0, 0, 1])     # The roll resulted in a 3.

# Draw from another categorical distribution (roll a weighted four-sided die).
>>> np.random.multinomial(1, np.array([.5, .3, .2, 0]))
array([0, 1, 0, 0])     # The roll resulted in a 1.
\end{lstlisting}

Consider a four-state weather model with the transition matrix
\begin{comment} % The beginnings of the 4-state chain diagram.
\begin{tikzpicture}[normalcircle/.style={draw,circle,minimum size=1.5cm,fill=none,thick,node distance=3.5cm}]
% Nodes
\node[normalcircle] (A) {\textcolor{red}{hot}};
\node[normalcircle] (B) [above of=A] {\textcolor{green}{mild}};
\node[normalcircle] (C) [right of=B] {\textcolor{blue}{cold}};
\node[normalcircle] (D) [below of=C] {\textcolor{cyan}{freezing}};
% Edges
\draw[bend right=60,thick,->,>=stealth'] (D) edge (C);
\draw[bend left=60,thick,->,>=stealth',blue] (B) edge (C);
\draw[bend right=60,thick,->,>=stealth',blue] (B) edge (A);
\foreach \a/\b in {A/B,A/C,A/D,C/B,C/D} \draw[thick,->,>=stealth'] (\a) edge (\b);
\draw[thick,->,>=stealth',blue] (B) edge (D);
\draw[thick,->,>=stealth',shorten >=1pt,blue] (B) to [out=110,in=170,loop,looseness=4.5] (B);
\end{tikzpicture}
\end{comment}
\begin{align*}
\begin{blockarray}{ccccc}
& \text{\textcolor{red}{hot}} & \text{\textcolor[rgb]{0,.6,0}{mild}} & \text{\textcolor{blue}{cold}} & \text{\textcolor{cyan}{freezing}} \\
\begin{block}{c[cccc]}
\text{\textcolor{red}{hot}}              & 0.5 & 0.3 & 0.1 & 0   \topstrut \\
\text{\textcolor[rgb]{0,.6,0}{mild}}     & 0.3 & 0.3 & 0.3 & 0.3 \\
\text{\textcolor{blue}{cold}}            & 0.2 & 0.3 & 0.4 & 0.5 \\
\text{\textcolor{cyan}{freezing}}        &   0 & 0.1 & 0.2 & 0.2 \botstrut \\
\end{block}\end{blockarray}.
\end{align*}
If today is hot, the probabilities of transitioning to each state are given by the ``hot'' column of the transition matrix.
Therefore, to choose a new state, draw from the categorical distribution with parameters ($0.5$, $0.3$, $0.2$, $0$).
The result $\begin{bmatrix}0 & 1 & 0 & 0\end{bmatrix}$ indicates a transition to the state corresponding to the $1$st row and column (tomorrow is mild), while the result $\begin{bmatrix}0 & 0 & 1 & 0\end{bmatrix}$ indicates a transition to the state corresponding to the $2$nd row and column (tomorrow is cold).
In other words, the position of the $1$ tells which column of the matrix to use as the parameters for the next categorical draw.

\begin{problem} % MarkovChain.transition()
Write a method for the \li{MarkovChain} class that accepts a single state label.
Use the label-to-index dictionary to determine the column of $A$ that corresponds to the provided state label, then draw from the corresponding categorical distribution to choose a state to transition to.
Return the corresponding label of the new state (not its index).
\\(Hint: \li{np.argmax()} may be useful.)
\label{prob:markov-class-transition}
\end{problem}

\begin{problem} % MarkovChain.walk() and MarkovChain.path().
\label{prob:markov-class-walkers}
Add the following methods to the \li{MarkovChain} class.
\begin{itemize}
\item \li{walk()}: Accept a state label and an interger $N$.
Starting at the specified state, use your method from Problem \ref{prob:markov-class-transition} to transition from state to state $N-1$ times, recording the state label at each step.
Return the list of $N$ state labels, including the initial state.
\item \li{path()}: Accept labels for an initial state and an end state.
Beginning at the initial state, transition from state to state until arriving at the specified end state, recording the state label at each step.
Return the list of state labels, including the initial and final states.
\end{itemize}
Test your methods on the two-state and four-state weather models described previously.
\end{problem}

\section*{General State Distributions} % ======================================

\begin{comment} % The 1-Norm. Might be able to get away without this.
For an $n\times 1$ vector $\x$ with entries $x_i$ and an $n\times n$ matrix $A$ with entries $a_{ij}$, the \emph{1-norm} is defined as follows.
\begin{align*}
\|\x\|_1 = \sum_{i=1}^n|x_i| && \|A\|_1 = \sup_j \sum_{i=1}^n |a_{ij}|
\end{align*}
In other words, the $1$-norm for both vectors and matrices is the maximum absolute column sum.
Then if $A$ is a transition matrix, $\|A\|_1 = 1$, since each of the entries of the matrix are positive and each of the columns sum to $1$ by definition.
The power method with the 1-norm can be used to find the unique stable steady state distribution of $A$.
\end{comment}

For a Markov chain with $n$ states, the probability of being in each state can be encoded by a $n$-vector $\x$, called a \emph{state distribution vector}.
The entries of $\x$ must be nonnegative and sum to 1, and the $i$th entry $x_i$ of $\x$ is the probability of being in state $i$.
For example, the state distribution vector $\x = \begin{bmatrix}0.8 & 0.2\end{bmatrix}\trp$ corresponding to the 2-state weather model indicates an $80\%$ chance that today is hot and a $20\%$ chance that today is cold.
On the other hand, the vector $\x = \begin{bmatrix}0 & 1\end{bmatrix}\trp$ implies that today is, with $100\%$ certainty, cold.

If $A$ is a transition matrix for a Markov chain with $n$ states and $\x$ is a corresponding state distribution vector, then $A\x$ is also a state distribution vector.
% To verify this fact, let $a_{ij}$ be the entries of $A$ and $x_i$ the entries of $\x$.
% The columns of $A$ sum to $1$, so $\sum_{j=1}^n a_{ij} = 1$ for $i = 1,\ 2,\ \ldots,\ n$.
% In addition, $\sum_{j=1}^n x_j = 1$ since the entries of $\x$ also sum to $1$.
% From matrix multiplication, the $i$th entry of $A\x$ is given by $\sum_{j=1}^n a_{ij}x_j$, so the sum of the entries of $A\x$ is
% \[\sum_{i=1}^n\sum_{j=1}^n a_{ij}x_j
% = \sum_{j=1}^n\left(x_j\left(\sum_{j=1}^n a_{ij}\right)\right)
% = \sum_{j=1}^n x_j = 1.\]
In fact, if $\x_k$ is the state distribution vector corresponding to a certain time $k$, then $\x_{k+1} = A\x_k$ contains the probabilities of being in each state after allowing the system to transition again.
For the weather model, this means that if there is an $80\%$ chance that it will be hot 5 days from now, written $\x_{5} = \begin{bmatrix}0.8 & 0.2\end{bmatrix}\trp$, then since
\[
\x_{6} = A\x_{5} =
\left[\begin{array}{cc}
0.7 & 0.6 \\
0.3 & 0.4 \\
\end{array}\right]
\left[\begin{array}{c}0.8 \\ 0.2\end{array}\right]
=
\left[\begin{array}{c}0.68 \\ 0.32\end{array}\right],
\]
there is a $68\%$ chance that 6 days from now will be a hot day.

\subsection*{Convergent Transition Matrices} % --------------------------------

Given an initial state distribution vector $\x_{0}$, defining $\x_{k+1} = A\x_k$ yields the significant relation
\[
\x_k = A\x_{k-1} = A(A\x_{k-2}) = A(A(A\x_{x-3})) = \cdots = A^k\x_{0}.
\]

This indicates that the $(i,j)$th entry of $A^k$ is the probability of transition from state $j$ to state $i$ in $k$ steps.
For the transition matrix of the 2-state weather model, a pattern emerges in $A^k$ for even small values of $k$:
\[
A = \left[\begin{array}{cc}
0.7 & 0.6 \\
0.3 & 0.4 \\
\end{array}\right],
\quad
A^2 = \left[\begin{array}{cc}
0.67 & 0.66 \\
0.33 & 0.34 \\
\end{array}\right],
\quad
A^3 = \left[\begin{array}{cc}
0.667 & 0.666 \\
0.333 & 0.334 \\
\end{array}\right].
\]
As $k\rightarrow\infty$, the entries of $A^k$ converge, written
\begin{equation}
\lim_{k \rightarrow \infty} A^k = \left[\begin{array}{ccc}
2/3 & 2/3 \\
1/3 & 1/3 \\
\end{array}\right].
\label{eq:markov-steady-transition}
\end{equation}
In addition, for any initial state distribution vector $\x_{0} = [a,\ b]\trp$ (meaning $a,b\ge 0$ and $a + b = 1$),
\[
\lim_{k \rightarrow \infty} \x_k = \lim_{k \rightarrow \infty}A^k\x_{0}
=
\left[\begin{array}{ccc}
2/3 & 2/3 \\
1/3 & 1/3 \\
\end{array}\right]
\left[\begin{array}{c}a\\b\end{array}\right]
=
\left[\begin{array}{c}2(a+b)/3\\(a+b)/3\end{array}\right]
=
\left[\begin{array}{c}2/3\\1/3\end{array}\right].
\]

Thus, $\x_k \rightarrow \x = \begin{bmatrix}2/3 & 1/3\end{bmatrix}\trp$ as $k\rightarrow\infty$,  regardless of the initial state distribution $\x_{0}$.
So, according to this model, no matter the weather today, the probability that it is hot a week from now is approximately $66.67\%$.
In fact, approximately 2 out of 3 days in the year should be hot.

\subsection*{Steady State Distributions} % ------------------------------------

The state distribution $\x = \begin{bmatrix}2/3 & 1/3\end{bmatrix}\trp$ has another important property:
\[
A\x =
\left[\begin{array}{cc}
7/10 & 3/5 \\
3/10 & 2/5 \\
\end{array}\right]
\left[\begin{array}{c}2/3 \\ 1/3\end{array}\right]
=
\left[\begin{array}{c}14/30 + 3/15 \\ 6/30 + 2/15\end{array}\right]
=
\left[\begin{array}{c}2/3 \\ 1/3\end{array}\right]
= \x.
\]
Any $\x$ satisfying $A\x = \x$ is called a \emph{steady state distribution} or a \emph{stable fixed point} of $A$.
In other words, a steady state distribution is an eigenvector of $A$ corresponding to the eigenvalue $\lambda = 1$.

% TODO: Verify this paragraph.
Every finite Markov chain has at least one steady state distribution.
If some power $A^k$ of $A$ has all positive (nonzero) entries, then the steady state distribution is unique.%
\footnote{This is a consequence of the \emph{Perron-Frobenius theorem}, which is presented in detail in Volume 1.}
In this case, $\lim_{k\rightarrow\infty}A^k$ is the matrix whose columns are all equal to the unique steady state distribution, as in (\ref{eq:markov-steady-transition}).
Under these circumstances, the steady state distribution $\x$ can be found by iteratively calculating $\x_{k+1} = A\x_k$, as long as the initial vector $\x_{0}$ is a state distribution vector.

\begin{warn}
Though every Markov chain has at least one steady state distribution, the procedure described above fails if $A^k$ fails to converge.
For instance, consider the transition matrix
\[
A = \left[\begin{array}{ccc}
0 & 0 & 1 \\
0 & 1 & 0 \\
1 & 0 & 0
\end{array}\right]
,\quad A^k = \begin{cases}
A\quad\text{if } k\ \text{is odd}
\\
I\quad\text{if } k\ \text{is even.}
\end{cases}
\]
In this case as $k\rightarrow\infty$, $A^k$ oscillates between two different matrices.

Furthermore, the steady state distribution is not always unique; the transition matrix defined above, for example, has infinitely many.
\end{warn}

\begin{problem} % Use the power method (simple) to get the steady state.
Write a method for the \li{MarkovChain} class that accepts a convergence tolerance \li{tol} and a maximum number of iterations \li{maxiter}.
Generate a random state distribution vector $\x_{0}$ and calculate $\x_{k+1} = A\x_k$ until $\|\x_{k-1} - \x_k\|_1 <$ \li{tol}, where $A$ is the transition matrix saved in the constructor.
If $k$ exceeds \li{maxiter}, raise a \li{ValueError} to indicate that $A^k$ does not converge.
Return the approximate steady state distribution $\x$ of $A$.

To test your function, generate a random transition matrix $A$.
Verify that $A\x = \x$ and that the columns of $A^k$ approach $\x$ as $k\rightarrow\infty$.
To compute $A^k$, use NumPy's (very efficient) algorithm for computing matrix powers. % (which is not part of \li{scipy.linalg}).

\begin{lstlisting}
>>> A = np.array([[.7, .6],[.3, .4]])
>>> np.linalg.matrix_power(A, 10)       # Compute A^10.
array([[ 0.66666667,  0.66666667],
       [ 0.33333333,  0.33333333]])
\end{lstlisting}
%
Finally, use your method to validate the results of Problem \ref{prob:markov-class-walkers}:
for the two-state and four-state weather models,
\begin{enumerate}
    \item Calculate the steady state distribution corresponding to the transition matrix.
    \item Run a weather simulation for a large number of days using \li{walk()} and verify that the results match the steady state distribution (for example, approximately 2/3 of the days should be hot for the two-state model).
\end{enumerate}
\label{prob:markov-power-method}
\end{problem}

\begin{info}
Problem \ref{prob:markov-power-method} is a special case of the \emph{power method}, an algorithm for calculating an eigenvector of a matrix corresponding to the eigenvalue of largest magnitude.
The general power method, together with a discussion of its convergence conditions, is discussed in Volume 1.
\end{info}

\section*{Using Markov Chains to Simulate English} % ==========================

One of the original applications of Markov chains was to study \emph{natural languages}, meaning spoken or written languages like English or Russian \cite{von2006five}.
In the early $20$th century, Markov used his chains to model how Russian switched from vowels to consonants.
By mid-century, they had been used as an attempt to model English.
It turns out that plain Markov chains are, by themselves, insufficient to model or produce very good English.
However, they can approach a fairly good model of bad English, with sometimes amusing results.

By nature, a Markov chain is only concerned with its current state, not with previous states.
A Markov chain simulating transitions between English words is therefore completely unaware of context or even of previous words in a sentence.
For example, if a chain's current state is the word ``continuous,'' the chain may say that the next word in a sentence is more likely to be ``function'' rather than ``raccoon.''
However the phrase ``continuous function'' may be gibberish in the context of the rest of the sentence.

% The transition probabilities of the resulting Markov chain should reflect the sort of English that the source authors speak.
% Thus, the Markov chain built from \emph{The Complete Works of William Shakespeare} differs greatly from the Markov chain built from a collection of academic journals.
% The source collection of works in the next problems is called the \emph{training set}.

\subsection*{Generating Random Sentences} % -----------------------------------

Consider the problem of generating English sentences that are similar to the text contained in a specific file, called the \emph{training set}.
The goal is to construct a Markov chain whose states and transition probabilities represent the vocabulary and---hopefully---the style of the source material.
There are several ways to approach this problem, but one simple strategy is to assign each unique word in the training set to a state, then construct the transition probabilities between the states based on the ordering of the words in the training set.
To indicate the beginning and end of a sentence requires two extra states: a \emph{start state}, \textcolor[rgb]{0,.6,0}{\$tart}, marking the beginning of a sentence; and a \emph{stop state}, \textcolor{red}{\$top}, marking the end.
The start state should only transitions to words that appear at the beginning of a sentence in the training set, and only words that appear at the end a sentence in the training set should transition to the stop state.

% English sentences are of varying length, so it is unknown beforehand how many words to choose (how many state transitions to make) before ending the sentence.
% To capture this feature, include two extra states in the chain: a \emph{start state}, \textcolor[rgb]{0,.6,0}{\$tart}, marking the beginning of a sentence; and a \emph{stop state}, \textcolor{red}{\$top}, marking the end.
% Thus, a training set with $n$ unique words has an $(n+2)\times (n+2)$ transition matrix.

Consider the following small training set, paraphrased from Dr. Seuss \cite{geisel1960green}.

\begin{lstlisting}
<<I am Sam Sam I am.
Do you like green eggs and ham?
I do not like them, Sam I am.
I do not like green eggs and ham.>>
\end{lstlisting}

There are 15 unique words in this training set, including punctuation (so ``ham?'' and ``ham.'' are counted as distinct words) and capitalization (so ``Do'' and ``do'' are also different):
%
\begin{align*}
\text{I\quad am\quad Sam\quad am.\quad Do\quad you\quad like\quad green}
\\
\text{eggs\quad and\quad ham?\quad do\quad not\quad them,\quad ham.}
\end{align*}
%
With start and stop states, the transition matrix should be $17 \times 17$.
Each state must be assigned a row and column index in the transition matrix, for example,
% An easy way to do this is to assign the states an index based on the order that they appear in the training set.
%
\begin{align*}
\begin{array}{ccccccc}
\text{\textcolor[rgb]{0,.6,0}{\$tart}} & \text{I} & \text{am} & \text{Sam} & \ldots & \text{ham.} & \text{\textcolor{red}{\$top}}
\\
0 & 1 & 2 & 3 & \ldots & 15 & 16
\end{array}
\end{align*}
%
The $(i,j)$th entry of the transition matrix $A$ should be the probability that word $j$ is followed by word $i$.
For instance, the word ``Sam'' is followed by the words ``Sam'' once and ``I'' twice in the training set, so the state corresponding to ``Sam'' (index 3) should transition to the state for ``Sam'' with probability $1/3$, and to the state for ``I'' (index 1) with probability $2/3$.
That is, $A_{3,3} = 1/3$, $A_{1,3} = 2/3$, and $A_{i,3} = 0$ for $i \notin\{1,3\}$.
Similarly, the start state should transition to the state for ``I'' with probability $3/4$, and to the state for ``Do'' with probability $1/4$; the states for ``am.'', ``ham?'', and ``ham.'' should each transition to the stop state.

To construct the transition matrix, parse the training set and add $1$ to $A_{i,j}$ every time word $j$ is followed by word $i$, in this case arriving at the matrix
%
\begin{align*}
\begin{blockarray}{cccccccc}
& \text{\textcolor[rgb]{.3,.6,.1}{\$tart}} & \textcolor{gray}{\text{I}} & \textcolor{gray}{\text{am}} & \textcolor{gray}{\text{Sam}} & & \textcolor{gray}{\text{ham.}} & \text{\textcolor[rgb]{1,0,0}{\$top}} \\
\begin{block}{c[ccccccc]}
\text{\textcolor[rgb]{.3,.6,.1}{\$tart}}    & 0 & 0 & 0 & 0 & \ldots & 0 & 0 \topstrut \\
\textcolor{gray}{\text{I}}        & 3 & 0 & 0 & 2 & \ldots & 0 & 0\\
\textcolor{gray}{\text{am}}       & 0 & 1 & 0 & 0 & \ldots & 0 & 0\\
\textcolor{gray}{\text{Sam}}      & 0 & 0 & 1 & 1 & \ldots & 0 & 0\\
& \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots\\
\textcolor{gray}{\text{ham.}}     & 0 & 0 & 0 & 0 & \ldots & 0 & 0\\
\text{\textcolor[rgb]{1,0,0}{\$top}}        & 0 & 0 & 0 & 0 & \ldots & 1 & 0 \botstrut \\
\end{block}\end{blockarray}.
\end{align*}
To avoid a column of zeros, set $A_{j,j} = 1$ where j is the index of the stop state (so the stop state always transitions to itself).
Next, divide each column by its sum so that each column sums to 1:
\begin{align*}
\begin{blockarray}{cccccccc}
& \text{\textcolor[rgb]{.3,.6,.1}{\$tart}} & \textcolor{gray}{\text{I}} & \textcolor{gray}{\text{am}} & \textcolor{gray}{\text{Sam}} & & \textcolor{gray}{\text{ham.}} & \text{\textcolor[rgb]{1,0,0}{\$top}} \\
\begin{block}{c[ccccccc]}
\text{\textcolor[rgb]{.3,.6,.1}{\$tart}} & 0 & 0 & 0 & 0 & \ldots & 0 & 0 \topstrut\\
\textcolor{gray}{\text{I}}        & 3/4 & 0 & 0 & 2/3 & \ldots & 0 & 0\\
\textcolor{gray}{\text{am}}       & 0 & 1/5 & 0 & 0 & \ldots & 0 & 0\\
\textcolor{gray}{\text{Sam}}      & 0 & 0 & 1 & 1/3 & \ldots & 0 & 0\\
& \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots\\
\textcolor{gray}{\text{ham.}}     & 0 & 0 & 0 & 0 & \ldots & 0 & 0\\
\text{\textcolor[rgb]{1,0,0}{\$top}}    & 0 & 0 & 0 & 0 & \ldots & 1 & 1 \botstrut\\
\end{block}\end{blockarray}.
\end{align*}

The $3/4$ indicates that 3 out of 4 times, the sentences in the training set start with the word ``I''.
Similarly, the $2/3$ and $1/3$ says that ``Sam'' is followed by ``I'' twice and by ``Sam'' once in the training set.
Note that ``am'' (without a period) always transitions to ``Sam'' and that ``ham.'' (with a period) always transitions the stop state.

The entire procedure of creating the transition matrix for the Markov chain with words from a file as states is summarized below.

\begin{algorithm}[H] % Read a file and convert it into a Markov chain.
\begin{algorithmic}[1]
\Procedure{MakeTransitionMatrix}{\texttt{filename}}
\State Read the training set from the file \texttt{filename}.
\State Get the set of unique words in the training set (the state labels).
\State Add labels \li{"\$tart"} and \li{"\$top"} to the set of states labels.
\State Initialize an appropriately sized square array of zeros to be the transition matrix.
\For {each sentence in the training set}
    \State Split the sentence into a list of words.
    \State Prepend \li{"\$tart"} and append \li{"\$top"} to the list of words.
    \For {each consecutive pair $(x, y)$ of words in the list of words}
    \State Add 1 to the entry of the transition matrix that corresponds to \par\qquad\qquad transitioning from state $x$ to state $y$.
    \EndFor
\EndFor
\State Make sure the stop state transitions to itself.
\State Normalize each column by dividing by the column sums.
\EndProcedure
\end{algorithmic}
\caption{Convert a training set of sentences into a Markov chain.}
\label{alg:MarkovSentencesTransitionMatrix}
\end{algorithm}

\begin{problem} % SentenceGenerator.__init__()
\label{prob:markov-random-sentences-init}
Write a class called \li{SentenceGenerator} that inherits from the \li{MarkovChain} class.
The constructor should accept a filename (the training set).
Read the file and build a transition matrix from its contents as described in Algorithm \ref{alg:MarkovSentencesTransitionMatrix}.
Save the same attributes as the constructor of \li{MarkovChain} does so that inherited methods work correctly.
Assume that the training set has one complete sentence written on each line.
\\(Hint: if the contents of the file are in the string \li{s}, then \li{s.split()} is the list of words and \li{s.split('\n')} is the list of sentences.)
\end{problem}

\begin{info} % The steady state is the stop state.
The Markov chains that result from the procedure in Problem \ref{prob:markov-random-sentences-init} have a few interesting structural characteristics.
The stop state is a \emph{sink}, meaning it only transitions to itself.
Because of this, and since every node has a path to the stop state, any traversal of the chain will end up in the stop state forever.
The stop state is therefore called an \emph{absorbing state}, and the chain as a whole is called an \emph{absorbing Markov chain}.
Furthermore, the steady state is the vector with a $1$ in the entry corresponding to the stop state and $0$s everywhere else.
\end{info}

\begin{problem} % Create random sentences.
\label{prob:markov-random-sentences-babble}
Add a method to the \li{SentenceGenerator} class called \li{babble()}.
Use the \li{path()} method from Problem \ref{prob:markov-class-walkers} to generate a random sentence based on the training document.
That is, generate a path from the start state to the stop state, remove the \li{"\$tart"} and \li{"\$top"} labels from the path, and join the resulting list together into a single, space-separated string.

For example, your \li{SentenceGenerator} class should be able to create random sentences that sound somewhat like Yoda speaking.

\begin{lstlisting}
>>> yoda = SentenceGenerator("yoda.txt")
>>> for _ in range(3):
... 	print(yoda.babble())
...
<<Impossible to my size, do not!
For eight hundred years old to enter the dark side of Congress there is.
But beware of the Wookiees, I have.>>
\end{lstlisting}
\end{problem}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Other Applications of Markov Chains} % ---------------------------

Markov chains are a useful way to study many probabilistic phenomena, so they have a wide variety of applications.
The following are just a few that are covered in other parts of this lab manual series.
\begin{itemize}
\item \textbf{PageRank}: Google's PageRank algorithm uses a Markov chain-based approach to rank web pages.
The main idea is to use the entries of the steady state vector as a measure of importance for the corresponding states.
For example, the steady state $\x = \begin{bmatrix}2/3 & 1/3\end{bmatrix}\trp$ for the two-state weather model means that the hot state is ``more important'' (occurs more frequently) than the cold state.
See the PageRank lab in Volume 1.

\item \textbf{MCMC Sampling}: A \emph{Monte Carlo Markov Chain} (MCMC) method constructs a Markov chain whose steady state is a probability distribution that is difficult to sample from directly.
This provides a way to sample from nontrivial or abstract distributions.
Many MCMC methods are used in various fields, from machine learning to physics.
See the Volume 3 lab on the Metropolis-Hastings algorithm.

\item \textbf{Hidden Markov Models}: The Markov chain simulations in this lab use an initial condition (a state distribution vector $\x_0$) and known transition probabilities to make predictions forward in time.
Converseyl, a \emph{hidden Markov model} (HMM) assumes that a given set of observations are the result of a Markov process, then uses those observations to infer the corresponding transition probabilities.
Hidden Markov models are used extensively in modern machine learning, especially for speech and language processing.
See the Volume 3 lab on Speech Recognition.
\end{itemize}

\subsection*{Large Training Sets} % -------------------------------------------

The approach in Problems \ref{prob:markov-random-sentences-init} and \ref{prob:markov-random-sentences-babble} begins to fail as the training set grows larger.
For example, a single Shakespearean play may not be large enough to cause memory problems, but \emph{The Complete Works of William Shakespeare} certainly will.

To accommodate larger data sets, consider use a sparse matrix from \li{scipy.sparse} for the transition matrix instead of a regular NumPy array.
Specifically, construct the transition matrix as a \li{lil_matrix} (which is easy to build incrementally), then convert it to the \li{csc_matrix} format (which supports fast column operations).
Ensure that the process still works on small training sets, then proceed to larger training sets.
How are the resulting sentences different if a very large training set is used instead of a small training set?

\subsection*{Variations on the English Model} % -------------------------------

Choosing a different state space for the English Markov model produces different results.
Consider modifying the \li{SentenceGenerator} class so that it can determine the state space in a few different ways.
The following ideas are just a few possibilities.

\begin{itemize}
\item Let each punctuation mark have its own state.
In the Dr. Seuss training set, instead of having two states for the words ``ham?'' and ``ham.'', there would be three states: ``ham'', ``?'', and ``.'', with ``ham'' transitioning to both punctuation states.

\item Model paragraphs instead of sentences.
Add a \textcolor[rgb]{0,.6,0}{\$tartParagraph} state that always transitions to \textcolor[rgb]{0,.6,0}{\$tartSentence} and a \textcolor{red}{\$topParagraph} state that is sometimes transitioned to by \textcolor{red}{\$topSentence}.

\item Let the states be individual letters instead of individual words.
Be sure to include a state for the spaces between words.
% We will explore this particular state space choice more in Volume 3 with hidden Markov models.

\item Construct the state space so that the next state depends on both the current and previous states.
This kind of Markov chain is called a \emph{Markov chain of order 2}.
This way, every set of three consecutive words in a randomly generated sentence should be part of the training set, as opposed to only every consecutive pair of words coming from the set.

\item Instead of generating random sentences from a single source, simulate a random conversation between $n$ people.
Construct a Markov chain $M_i,$ for each person, $i=1,\ \ldots,\ n$, then create a Markov chain $C$ describing the conversation transitions from person to person; in other words, the states of $C$ are the $M_i$.
To create the conversation, generate a random sentence from the first person using $M_1$.
Then use $C$ to determine the next speaker, generate a random sentence using their Markov chain, and so on.
\end{itemize}

\subsection*{Natural Language Processing Tools} % -----------------------------

The Markov model of Problems \ref{prob:markov-random-sentences-init} and \ref{prob:markov-random-sentences-babble} is a \emph{natural language processing} application.
Python's \li{nltk} module (natural language toolkit) has many tools for parsing and analyzing text for these kinds of problems \cite{bird2004nltk}.
For example, \li{nltk.sent_tokenize()} reads a single string and splits it up into sentences.
This could be useful, for example, in making the \li{SentenceGenerator} class compatible with files that do not have one sentence per line.

\begin{lstlisting}
>>> from nltk import sent_tokenize
>>> with open("yoda.txt", 'r') as yoda:
...     sentences = sent_tokenize(yoda.read())
...
>>> print(sentences)
<<['Away with your weapon!',
 'I mean you no harm.',
 'I am wondering - why are you here?',
 ...>>
\end{lstlisting}

The \li{nltk} module is \textbf{not} part of the Python standard library.
For instructions on downloading, installing, and using \li{nltk}, visit \url{http://www.nltk.org/}.


\chapter{Summations}

\section{Summation formulas and properties}

\begin{enumerate}

\item[A.1{-}1] {Find a simple formula for $\sum_{k = 1}^{n} (2k - 1)$.}

\begin{framed}
\begin{equation*}
\begin{aligned}
  \sum_{k = 1}^{n} (2k - 1) &= \sum_{k = 1}^{n} 2k - \sum_{k = 1}^{n} 1\\
                            &= 2 \sum_{k = 1}^{n} k - n\\
                            &= 2 \cdot \frac{1}{2} n (n + 2) - n\\
                            &= n^2 + n - n\\
                            &= n^2.
\end{aligned}
\end{equation*}
\end{framed}

\item[A.1{-}2] {($\star$) Show that
$\sum_{k = 1}^{n} 1/(2k - 1) = \ln(\sqrt n) + O(1)$ by manipulating the harmonic
series.}

\begin{framed}
\begin{equation*}
\begin{aligned}
  \sum_{k = 1}^{n} 1/(2k - 1) &= \frac{1}{1} + \frac{1}{3} + \frac{1}{5} + \dots + \frac{1}{2n - 3} + \frac{1}{2n - 1}\\
                              &= \left(\frac{1}{1} + \frac{1}{2} + \frac{1}{3} + \dots + \frac{1}{2n}\right)
                              - \frac{1}{2} \left(1 + \frac{1}{2} + \frac{1}{3} + \dots + \frac{1}{n}\right)\\
                              &= \sum_{k = 1}^{2n} \frac{1}{k} - \frac{1}{2} \sum_{k - 1}^{n} \frac{1}{k}\\
                              &= \ln{2n} + O(1) - \frac{1}{2} (\ln n + O(1))\\
                              &= \ln n + \ln 2 + O(1) - \frac{1}{2} \ln n - \frac{1}{2} O(1)\\
                              &= \frac{1}{2} \ln n + O(1)\\
                              &= \ln(\sqrt n) + O(1).
\end{aligned}
\end{equation*}
\end{framed}

\item[A.1{-}3] {Show that $\sum_{k = 0}^{\infty} k^2 x^k = x(1 + x)/(1 - x)^3$
for $0 < |x| < 1$.}

\begin{framed}
From Equation A.8, we have
\[
  \sum_{k = 0}^{\infty} k x^k = \frac{x}{(1 - x)^2}.
\]
differentiating both sides and multiplying by $x$, we have
\begin{equation*}
\begin{aligned}
  \sum_{k = 0}^{\infty} k^2 x^k &= x \cdot \frac{1 \cdot (1 - x)^2 - (2 \cdot (1 - x) \cdot (-1) \cdot x)}{(1 - x)^4}\\
                                &= x \cdot \frac{(1 - x)(1 - x) + (1 - x) \cdot 2x}{(1 - x)^4}\\
                                &= x \cdot \frac{(1 - x) + 2x}{(1 - x)^3}\\
                                &= \frac{x(1 + x)}{(1 - x)^3}.
\end{aligned}
\end{equation*}
\end{framed}

\newpage

\item[A.1{-}4] {($\star$) Show that $\sum_{k = 0}^{\infty} (k - 1)/2^k = 0$.}

\begin{framed}
\begin{equation*}
\begin{aligned}
  \sum_{k = 0}^{\infty} (k - 1)/2^k
  &= \sum_{k = 0}^{\infty} \left( \frac{k}{2^k} - \frac{1}{2^k} \right)\\
  &= \sum_{k = 0}^{\infty} k \frac{1}{2^k} - \sum_{k = 0}^{\infty} \frac{1}{2^k}\\
  &= \sum_{k = 0}^{\infty} k \left( \frac{1}{2} \right)^k - \sum_{k = 0}^{\infty} \left( \frac{1}{2} \right)^k\\
  &= \frac{(1/2)}{(1 - (1/2))^2} - \frac{1}{1 - (1/2)}\\
  &= \frac{(1/2)}{1 - 1 - (1/4)} - 2\\
  &= 4/2 - 2\\
  &= 0.
\end{aligned}
\end{equation*}
\end{framed}

\item[A.1{-}5] {($\star$) Evaluate the sum
$\sum_{k = 1}^{\infty} (2k + 1) x^{2k}$ for $|x| < 1$.}

\begin{framed}
\begin{equation*}
\begin{aligned}
  \sum_{k = 1}^{\infty} (2k + 1) x^{2k}
  &= \frac{d}{dx} \cdot \sum_{k = 1}^{\infty} x^{2k + 1}\\
  &= \frac{d}{dx} \cdot x \cdot \sum_{k = 1}^{\infty} x^{2k}\\
  &= \frac{d}{dx} \cdot x \cdot \sum_{k = 0}^{\infty} (x^2)^k - x\\
  &= \frac{d}{dx} \cdot x \cdot \frac{1}{1 - x^2} - x\\
  &= \frac{d}{dx} \cdot \frac{x - x (1 - x^2)}{1 - x^2}\\
  &= \frac{d}{dx} \cdot \frac{x^3}{1 - x^2}\\
  &= \frac{3 x^2 (1 - x^2) - (-2x) x^3}{(1 - x^2)^2}\\
  &= \frac{3 x^2 - 3 x^4 + 2 x^4}{(1 - x^2)^2}\\
  &= \frac{(3 - x^2) \cdot x^2}{(1 - x^2)^2}.
\end{aligned}
\end{equation*}
\end{framed}

\item[A.1{-}6] {Prove that
$\sum_{k=1}^{n} O(f_k(i)) = O(\sum_{k = 1}^{n} f_k(i))$ by using the linearity
property of summations.}

\begin{framed}
Skipped.
\end{framed}

\newpage

\item[A.1{-}7] {Evaluate the product $\prod_{k = 1}^{n} 2 \cdot 4^k$.}

\begin{framed}
We have
\[
  \prod_{k = 1}^n (2 \cdot 4^k) = 2^{\lg{\left(\prod_{k = 1}^n (2 \cdot 4^k)\right)}},
\]
and
\begin{equation*}
\begin{aligned}
  \lg{\left(\prod_{k = 1}^n (2 \cdot 4^k)\right)}
  &= \sum_{k = 1}^{n} \lg (2 \cdot 2^{2k})\\
  &= \sum_{k = 1}^{n} \lg 2^{2k + 1}\\
  &= \sum_{k = 1}^{n} (2k + 1)\\
  &= 2 \sum_{k = 1}^n k + \sum_{k = 1}^n 1\\
  &= n (n + 1) + n\\
  &= n (n + 2).
\end{aligned}
\end{equation*}

Thus,
\[
  \prod_{k = 1}^n (2 \cdot 4^k) = 2^{n (n + 2)}.
\]
\end{framed}

\item[A.1{-}8] {($\star$) Evalute the product $\prod_{k = 2}^{n} (1 - 1/k^2)$.}

\begin{framed}
We have
\[
  \prod_{k = 2}^{n} \left(1 - \frac{1}{k^2}\right) = 2^{\lg{\left( \sum_{k = 2}^{n} \lg{(1 - 1/k^2)} \right)}},
\]
and
\begin{equation*}
\begin{aligned}
  \sum_{k = 2}^{n} \lg{\left(1 - \frac{1}{k^2}\right)}
  &= \sum_{k = 2}^{n} \lg{\left( \frac{k^2 - 1}{k^2} \right)}\\
  &= \sum_{k = 2}^{n} \lg{\left( \frac{(k - 1)}{k} \cdot \frac{(k + 1)}{k} \right)}\\
  &= \sum_{k = 2}^{n} \left( \lg \left( \frac{k - 1}{k} \right) + \lg \left( \frac{k + 1}{k} \right) \right)\\
  &= \lg{\frac{1}{2}} + \lg{\frac{3}{2}} +
     \lg{\frac{2}{3}} + \lg{\frac{4}{3}} +
     \lg{\frac{3}{4}} + \lg{\frac{5}{4}} +
     \dots +
     \lg{\frac{n - 2}{n - 1}} + \lg{\frac{n}{n - 1}} +
     \lg{\frac{n - 1}{n}} + \lg{\frac{n + 1}{n}}\\
  &= \lg 1 - \lg 2 + \cancel{\lg 3} - \cancel{\lg 2} + \cancel{\lg 2} - \cancel{\lg 3}
           + \cancel{\lg 4} - \cancel{\lg 3} + \cancel{\lg 3} - \cancel{\lg 4} + \cancel{\lg 5} - \cancel{\lg 4} + \dots\\
  & \qquad + \cancel{\lg (n - 2)} - \cancel{\lg (n - 1)} + \cancel{\lg n} - \cancel{\lg (n - 1)}
           + \cancel{\lg (n - 1)} - \cancel{\lg n} + \lg (n + 1) - \lg n\\
  &= 0 - 1 + \lg (n + 1) - \lg n\\
  &= \lg (n + 1) - \lg (n) - 1.
\end{aligned}
\end{equation*}

Thus,
\[
  \prod_{k = 2}^{n} \left(1 - \frac{1}{k^2}\right) = 2^{(\lg (n + 1) - (\lg (n) + 1))}
  = \frac{2^{\lg{(n + 1)}}}{2^{\lg{(n) + 1}}}
  = \frac{n + 1}{2^{\lg n} \cdot 2}
  = \frac{n + 1}{2n}.
\]
\end{framed}

\end{enumerate}

\newpage

\section{Bounding summations}

\begin{enumerate}

\item[A.2{-}1] {Show that $\sum_{k = 1}^{n} 1/k^2$ is bounded above by
a constant.}

\begin{framed}
\begin{equation*}
\begin{aligned}
  \sum_{k = 1}^{n} &=   1 + \sum_{k = 2}^{n} \frac{1}{k^2}\\
                   &\le 1 + \int_{1}^{n} \frac{dx}{x^2}\\
                   &=   1 + \left( \Eval{- \frac{1}{x}}{1}{n} \right)\\
                   &=   1 + \left( - \frac{1}{n} - \left( - \frac{1}{1} \right) \right)\\
                   &= 2 - \frac{1}{n}\\
                   &\le 2.
\end{aligned}
\end{equation*}
\end{framed}

\item[A.2{-}2] {Find an asymptotic upper bound on the summation
\[
  \sum_{k = 0}^{\floor{\lg n}} \ceil{n/2^k}.
\]
}

\begin{framed}
\begin{equation*}
\begin{aligned}
  \sum_{k = 0}^{\floor{\lg n}} \Bigl\lceil \frac{n}{2^k} \Bigr\rceil
  &=   n \cdot \sum_{k = 0}^{\floor{\lg n}}  \Bigl\lceil \frac{1}{2^k} \Bigr\rceil\\
  &\le n \cdot \sum_{k = 0}^{\lg n}  \left( \frac{1}{2^k} + 1 \right)\\
  &=   n \cdot \sum_{k = 0}^{\lg n}  \left( \frac{1}{2^k} \right) + \sum_{k = 0}^{\lg n} 1\\
  &=   n \cdot \frac{1}{1 - (1/2)} + \lg n + 1\\
  &=   2n + \lg n + 1\\
  &=   O(n).
\end{aligned}
\end{equation*}
\end{framed}

\item[A.2{-}3] {Show that the $n$th harmonic number is $\Omega(\lg n)$ by
splitting the summation.}

\begin{framed}
\begin{equation*}
\begin{aligned}
  \sum_{k = 1}^n \frac{1}{k} &\ge \sum_{i = 0}^{\floor{\lg n} - 1} \sum_{j = 0}^{2^i - 1} \frac{1}{2^i + j}\\
                             &\ge \sum_{i = 0}^{\floor{\lg n} - 1} \sum_{j = 0}^{2^i - 1} \frac{1}{2^{i + 1}}\\
                             &=   \sum_{i = 0}^{\floor{\lg n} - 1} \frac{1}{2} \cdot \sum_{j = 0}^{2^i - 1} \frac{1}{2^{i}}\\
                             &=   \sum_{i = 0}^{\floor{\lg n} - 1} \frac{1}{2}\\
                             &\ge \sum_{i = 0}^{\lg n - 2} \frac{1}{2}\\
                             &=   \frac{1}{2} (\lg (n) - 1)\\
                             &=   \Omega(\lg n).
\end{aligned}
\end{equation*}
\end{framed}

\newpage

\item[A.2{-}4] {Approximate $\sum_{k = 1}^{n} k^3$ with an integral.}

\begin{framed}
We have
\[
  \int_{0}^{n} x^3 dx \le \sum_{k = 1}^{n} k^3 \le \int_{1}^{n + 1} x^3 dx.
\]

For a lower bound, we obtain
\[
  \sum_{k = 1}^{n} k^3 \ge \int_{0}^{n} x^3 dx = \Eval{\frac{x^4}{4}}{0}{n} = \frac{n^4}{4} = \Omega(n^4).
\]

For the upper bound, we obtain
\[
  \sum_{k = 1}^{n} k^3 \le \int_{1}^{n + 1} x^3 dx = \Eval{\frac{x^4}{4}}{1}{n + 1} = \frac{(n + 1)^4 - 1}{4} = O(n^4).
\]

Thus,
\[
  \sum_{k = 1}^{n} k^3 = \Theta(n^4).
\]

\end{framed}

\item[A.2{-}5] {Why didn't we use the integral approximation (A.12) directly on
$\sum_{k = 1}^{n} 1/k$ to obtain an upper bound on the $n$th harmonic number?}

\begin{framed}
Applying (A.12) directly, we obtain
\[
  \sum_{k = 1}^{n} \frac{1}{k} \le \int_{0}^{n} \frac{1}{x} dx,
\]
but the function $1/x$ is undefined for $x = 0$ (because of the division by
zero).
\end{framed}

\end{enumerate}

\newpage

\section*{Problems}
\addcontentsline{toc}{section}{\protect\numberline{}Problems}%

\begin{enumerate}

\item[A{-}1]{\textbf{\emph{Bounding summations}}\\
Give asymptotically tight bounds on the following summations. Assume that
$r \ge 0$ and $s \ge 0$ are constants.
\begin{enumerate}
  \item[a.] $\sum_{k = 1}^{n} k^r$.
  \item[b.] $\sum_{k = 1}^{n} \lg^s k$.
  \item[c.] $\sum_{k = 1}^{n} k^r \lg^s k$.
\end{enumerate}
}

\begin{framed}
\begin{enumerate}
\item[(a)] For a lower bound, we have
\begin{equation*}
\begin{aligned}
  \sum_{k = 1}^{n} k^r &\ge \int_{0}^{n} x^r dx\\
                       &=   \Eval{\frac{x^{(r + 1)}}{r + 1}}{0}{n}\\
                       &=   \frac{n^{(r + 1)}}{r + 1} - \frac{0^{(r + 1)}}{r + 1}\\
                       &\ge n^{(r + 1)}\\
                       &=   \Omega(n^{(r + 1)}),
\end{aligned}
\end{equation*}
and for the upper bound, we have
\[
  \sum_{k = 1}^{n} k^r \le \sum_{k = 1}^{n} n^r = n^{(r + 1)} = O(n^{(r + 1)}).
\]

Thus,
\[
  \sum_{k = 1}^{n} = \Theta(n^{(r + 1)}).
\]

\item[(b)] For a lower bound, we have
\begin{equation*}
\begin{aligned}
  \sum_{k = 1}^{n} \lg^{s} k &=   \sum_{k = 1}^{n/2} \lg^{s} k + \sum_{k = n/2 + 1}^{n} \lg^{s} k\\
                             &\ge \sum_{k = 1}^{n/2} 0 + \sum_{k = n/2 + 1}^{n} \lg^{s}{\left(\frac{n}{2}\right)}\\
                             &=   \frac{n}{2} \lg^{s}{\left(\frac{n}{2}\right)}\\
                             &=   \frac{n}{2} \lg^{s}{n} - \frac{n}{2} \lg^{s}{2}\\
                             &\ge \frac{1}{2} n \lg^{s} n - \frac{1}{2} n\\
                             &=   \Omega(n \lg^s n),
\end{aligned}
\end{equation*}
and for the upper bound, we have
\[
  \sum_{k = 1}^{n} \lg^{s} k \le \sum_{k = 1}^{n} \lg^{s} n = n \lg^s n = O(n \lg^s n).
\]
Thus,
\[
  \sum_{k = 1}^{n} \lg^{s} k = \Theta(n \lg^s n).
\]

\item[(c)] It is easy to see that this summation is greater than the one from
item (a). Thus, it is $\Omega(n^{(r + 1)})$. Also, we have
\[
  \sum_{k = 1}^{n} k^r \lg^{s} k \le \sum_{k = 1}^{n} n^r \lg^{s} n = O(n^{(r + 1)} \lg^s n).
\]

Thus, I guess it is $\Theta(n^{(r + 1)} \lg^s n)$.

\end{enumerate}
\end{framed}

\end{enumerate}


\section{The fundamental theorem of algebra}

\begin{outcome}
  \begin{enumerate}
  \item Find the complex roots of a quadratic polynomial.
  \item In special cases, find the complex roots of a polynomial of
    degree 3 or more.
  \item Factor a polynomial into linear factors.
  \end{enumerate}
\end{outcome}

The complex numbers were invented so that equations such as $z^2+1=0$
would have solutions. In fact, this equation has two complex
solutions, namely $z=i$ and $z=-i$. However, something much more
general (and surprising) is true: {\em every} non-trivial polynomial
equation has a solution in the complex numbers. To understand this
statement, recall that a \textbf{polynomial}%
\index{polynomial} is an expression of the form
\begin{equation*}
  p(z) = a_nz^n + a_{n-1}z^{n-1} + \ldots + a_1z + a_0.
\end{equation*}
The constants $a_0,\ldots,a_n$ are called the \textbf{coefficients}%
\index{coefficient!of a polynomial}%
\index{polynomial!coefficient} of the polynomial. If $a_n$ is the
largest non-zero coefficient, we say that the polynomial has
\textbf{degree}%
\index{degree!of a polynomial}%
\index{polynomial!degree} $n$. A polynomial of degree $0$ is of the
form $p(z) = a_0$, and is also called a \textbf{constant polynomial}%
\index{polynomial!constant}%
\index{constant polynomial}. Recall that a \textbf{root}%
\index{root!of a polynomial}%
\index{polynomial!root} of a polynomial is a number $z$ such that
$p(z)=0$.  The fundamental theorem of algebra is the following:

\begin{theorem}{Fundamental theorem of algebra}{fundamental-algebra}
  Every non-constant polynomial $p(z)$ with real or complex
  coefficients has a complex root.
\end{theorem}

The proof of this theorem is beyond the scope of this book. Note that
the theorem does not say that the roots are always easy to find. To
find the roots of a polynomial of degree 2, we can use the quadratic
formula. However, if the degree is greater than 2, we may sometimes
have to use fancier methods, such as Newton's method from calculus, or
even a computer algebra system, to locate the roots. We give some
examples.

\begin{example}{Roots of a quadratic polynomial}{complex-root}
  Find the roots of the polynomial $p(z) = z^2 - 2z + 2$.
\end{example}

\begin{solution}
  The quadratic formula gives
  \begin{equation*}
    z = \frac{2 \pm \sqrt{-4}}{2}.
  \end{equation*}
  Of course, in the real numbers, the square root of $-4$ does not
  exist, so $p(z)$ has no roots in the real numbers. However, in the
  complex numbers, the square root of $-4$ exists and is equal to
  $\pm2i$. Thus, the roots of $p(z)$ are:
  \begin{equation*}
    z = \frac{2 \pm 2i}{2} = 1\pm i.
  \end{equation*}
  Indeed, we can double-check that $1+i$ and $1-i$ are in fact roots:
  \begin{equation*}
    \begin{array}{ll}
      p(1+i) = (1+i)^2 - 2(1+i) + 2 = (1 + 2i + (-1)) - 2 - 2i + 2 = 0, \\
      p(1-i) = (1-i)^2 - 2(1-i) + 2 = (1 - 2i + (-1)) - 2 + 2i + 2 = 0. \\
    \end{array}
  \end{equation*}
  \vspace{-2ex}
\end{solution}

\begin{example}{Roots of a cubic polynomial}{complex-root2}
  Find the roots of the polynomial $p(z) = z^3 - 4z^2 + 9z - 10$.
\end{example}

\begin{solution}
  By the intermediate value theorem of calculus, we know that a cubic
  polynomial with real coefficients always has at least one real
  root. This is because $p(z)$ goes to $-\infty$ when $z\to-\infty$
  and to $\infty$ when $z\to\infty$. By trial and error, we find that
  $z=2$ is a root of this polynomial. We can therefore factor out
  $(z-2)$ from this polynomial:
  \begin{equation*}
    p(z) = z^3 - 4z^2 + 9z - 10 = (z-2)(z^2 - 2z + 5).
  \end{equation*}
  Now we can use the quadratic formula to find the roots of $z^2 - 2z
  + 5$. We find
  \begin{equation*}
    z = \frac{2\pm\sqrt{-16}}{2} = \frac{2\pm 4i}{2} = 1\pm 2i.
  \end{equation*}
  Thus, the three complex roots of $p(z)$ are $z=2$, $z=1+2i$, and $z=1-2i$.
\end{solution}

The following proposition is an important and useful consequence of
the fundamental theorem of algebra:

\begin{proposition}{Factoring a polynomial}{complex-factoring}
  Let $p(z)$ be a polynomial of degree $n$ with real or complex
  coefficients. Then $p(z)$ can be factored into $n$ linear factors
  over the complex numbers, i.e., $p(z)$ can be written in the form
  \begin{equation*}
    p(z) = a(z-b_1)(z-b_2)\cdots(z-b_n),
  \end{equation*}
  where $b_1,\ldots,b_n$ are (not necessarily distinct) roots of
  $p(z)$.
\end{proposition}

\begin{proof}
  If $n=0$, then $p(z)=a$ and there is nothing to show. Otherwise, by
  the fundamental theorem of algebra, $p(z)$ has at least one complex
  root, say $b_1$. From calculus, we know that we can factor out
  $(z-b_1)$ from $p(z)$, i.e., we can find a polynomial $q(z)$ of
  degree $n-1$ such that
  \begin{equation*}
    p(z) = (z-b_1) q(z),
  \end{equation*}
  We can repeatedly apply the same procedure to $q(z)$ until $p(z)$
  has been factored into linear factors.
\end{proof}

\begin{example}{Factoring a polynomial}{complex-factoring}
  Factor $p(z) = z^3 - 4z^2 + 9z - 10$ into linear factors.
\end{example}

\begin{solution}
  From Example~\ref{exa:complex-root2}, we know that $p(z)$ has three
  distinct roots $b_1=2$, $b_2=1+2i$, and $b_3=1-2i$. We can therefore
  write
  \begin{equation*}
    p(z) = a(z-b_1)(z-b_2)(z-b_3).
  \end{equation*}
  Since the leading term is $z^3$, we find that $a=1$. Therefore
  \begin{equation*}
    p(z) = (z-2)\,(z-1-2i)\,(z-1+2i).
  \end{equation*}
  \vspace{-2ex}
\end{solution}


\section{Secret Sharing}\label{s:secret-sharing}
Secret sharing is a method for distributing a secret between $N$ different parties, where a secret message $m$ is divided into parts, giving each participant its own unique part $(s_1, s_2, \dots, s_N)$.
Combining some of the parts or all of them are needed in order to reconstruct the original secret.
The most common type of secret sharing is a scheme with one dealer and $N$ different players.
Initially, the dealer splits the secret to shares and gives each player a split.
Only with all -- or most of the shares $k \geq T$  ($T$ is a threshold) -- someone is able to reconstruct the original secret.
The dealer accomplishes this by giving each player a share in such a way that any group of $T$ or more players can together reconstruct the secret but no group of fewer than $T$ players can.
Such a system is called a $(T, N)$ - threshold scheme.

A naive secret sharing scheme between a dealer and two parties is to apply bitwise \texttt{XOR} to the secret and a random string of the same length.
For the sake of simplicity, let us assume that the secret message m is a string of 5 bits (either 0 or 1), \textit{e.g.} 01101.
Then, the dealer generates a random string of 5 bits $s_1$ (\textit{e.g.} $10011$) and performs the \texttt{XOR} operation between $s_1$ and $m$; a string $s_2 = 11110$ arises.
Finally, the dealer gives the $s_1$ and $s_2$ to the two parties.
Neither one of them is able to retrieve the secret message $m$ without the other half share, and also nor is able to retrieve any information about the original message.

There exist more complex and elaborate systems for more than two parties, such as Shamir’s \cite{shamir1979share}, Blakley’s \cite{blakley1994linear}, or Additive \cite{kim2003designs} secret sharing schemes.

\subsection{Shamir Secret Sharing}\label{ss:shamir-secret-sharing}
Secret sharing was proposed in \cite{shamir1979share} by Shamir, and it was the first method to enable distribution of a secret to $N$ parties.
A secret message $m$ is divided into $N$ parts, giving each participant its own unique part, where any $T$ subset of $N$ can recover the secret, but no $T-1$ element subset can.

Adi Shamir's threshold scheme is based on the idea of polynomial interpolation -- the interpolation of a given dataset by the polynomial of lowest possible degree that passes through the points of the dataset --, using Lagrange coefficients.
Shamir's secret sharing scheme constructs a polynomial $P$ of degree $N-1$, where $N$ is the number of players, as follows.

\subsubsection{Algorithm}\label{sss:shamir-algorithm}
\begin{itemize}
  \item Dealer chooses a random polynomial of degree $N - 1$ so that $P(0) = m$, where $m$ is the secret message, and also the number $T$ of sufficient subsets that can reconstruct the secret.

  \textit{For example\footnote{This example is based on a similar example in \url{https://en.wikipedia.org/wiki/Shamir's_Secret_Sharing} Wikipedia link.}}, let us suppose our secret message $m = 1234$ and $N = 6$ parties, but any subset of $T = 3$ is sufficient to reconstruct the secret.
  Then our polynomial should have degree 2, $f(x) = ax^2 + bx + m$.
  The dealer chooses $N - 1$ random numbers, \textit{i.e.} $a = 94$ and $b = 166$, thus our polynomial is $f(x) = 94x^2 + 166x + 1234$.

  \item Dealer distributes $N$ pairs $(x_i , P(x_i)),$ $x_i \neq 0$

  First the dealer creates the $N$ pairs.
  \textit{For example}:\\
  $D_0 = (1, 1494)$, $D_1 = (2, 1942)$, $D_2 = (3, 2578)$, $D_3 = (4, 3402)$, $D_4 = (5, 4414)$, $D_5 = (6, 5614)$ and consecutively he\myslash she distributes them to the $N$ parties.

  \item $N$ players can reconstruct the polynomial $P$ with their pairs, however $N - 1$ cannot

  In order to reconstruct the secret any $T$ points will be enough, \textit{for instance}, $D_1, D_3$ and $D_4$.
  Computing the Lagrange polynomials:\\
  $l_0 = \cfrac{x - x_1}{x_0 - x_1} \times \cfrac{x - x_2}{x_0 - x_2} = \cfrac{x - 4}{2 - 4} \times \cfrac{x - 5}{2 - 5} = \cfrac{1}{6}x^2 - \cfrac{3}{2}x + \cfrac{10}{3}$

  $l_1 = \cfrac{x - x_0}{x_1 - x_0} \times \cfrac{x - x_2}{x_1 - x_2} = \cfrac{x - 2}{4 - 2} \times \cfrac{x - 5}{4 - 5} = \cfrac{1}{2}x^2 + \cfrac{7}{2}x - 5$

  $l_2 = \cfrac{x - x_0}{x_2 - x_0} \times \cfrac{x - x_1}{x_2 - x_1} = \cfrac{x - 2}{5 - 2} \times \cfrac{x - 4}{5 - 4} = \cfrac{1}{3}x^2 - 2x + \cfrac{8}{3}$

  Therefore:

  $f(x) = \sum_{i=0}^{2} y_i \times l_i(x) = 94x^2 + 166x + 1234$

  Each party can compute $f(0)$ in order to obtain the secret, in this case $f(0) = 1234$.
\end{itemize}

It is evident, that in Shamir's scheme two points are sufficient to define a line $f(x) = ax + b$, three points are sufficient to define a parabola $f(x) = ax^2 + bx + c$, four points to define a cubic curve $f(x) = ax^3 + bx^2 + cx + d$ and so forth.
That is, it takes $T$ points to define a polynomial of degree $T-1$, where $T - 1$ will be the sufficient number of parties that can reconstructed the secret message.

\subsubsection{Threat Model}\label{sss:shamir-threat-model}
The Shamir secret sharing scheme can tolerate both passive and active adversaries, however in the latter case it has more strict bounds.
Shamir's sharing algorithm is secure against a passive adversary when $T < \cfrac{N}{2}$, while it achieves information-theoretic security for $T < \cfrac{N}{3}$ with an active adversary.
This means that even if the adversary has unbounded computational power, they cannot learn any information about the secret underlying a share.
The BGW \cite{brakerski2014leveled} protocol, which defines how to compute addition and multiplication on secret shares, is often used to compute functions with Shamir secret shares.

\subsection{Additive Secret Sharing}\label{ss:additive-secret-sharing}
Another form of secret sharing, and probably the simplest one, is additive sharing.
Again, a secret message $m$ is divided into $N$ parts, where each participant has its own unique part.
In contrast with Shamir's sharing algorithm, all parts are necessary in order to recover the original secret.

\subsubsection{Algorithm}\label{sss:additive-algorithm}
\begin{itemize}

  \item Dealer chooses a randomly $N - 1$ numbers such that $x_i \in \mathbb{Z}, i \in [1, N-1]$

  \item Computing $x_N$ from the random numbers is pretty trivial; $x_N = m - x_1 - x_2 - \dots - x_{N-1}$

  \item Finally, the dealer distributes the $N$ secrets ($x_1, \dots x_N$)

  \item In order to reconstruct the original message $m$, the parties have to combine their secrets: $m = x_1 + x_2 + \dots + x_N$
\end{itemize}

Seeing any $N-1$ values (\textit{i.e.} $x_1, \dots x_{N-1}$) does not give any clue about what $m$ could be, since the remaining $x_i$ may change the final result to any element of $\mathbb{Z}$ with equal probability.
More elaborate schemes than the $(\mathbb{Z}, +)$ have been introduced, in the form of $(A, \oplus)$ (which means in group $A$ using the operation $\oplus$), such as linear secret sharing.

There are many systems that have implemented various forms of SMPC with secret sharing schemes.
The most popular is SPDZ \cite{damgaard2012multiparty, damgaard2013practical}, which implements MPC with additive secret shares and is secure against active adversaries.
Another popular system that implements SMPC using additive secret shares is the Sharemind Secure Computing Platform, which we survey in section \ref{c:sharemind}.

\subsubsection{Threat Model}\label{sss:additive-threat-model}
In contrast to Shamir's secret sharing scheme, that can tolerate up to $T < \cfrac{N}{2}$ passive adversaries or $T < \cfrac{N}{3}$ active, as mentioned in \ref{sss:shamir-threat-model}, additive secret sharing schemes can tolerate the adversary controlling all but one party, that is $T < N$.
As we already mentioned, in order to reconstruct the original message $m$, all parties have to add their secrets ($m = x_1 + x_2 + \dots + x_N$).
It has become evident, that if one $x_i$ is missing, the initial message $m$ cannot be reconstructed.
% Additive schemes maintain security against a passive and active adversary with unbounded computational power.

\subsection{Secret Sharing Homomorphism}\label{ss:secret-sharing-homomorphism}
In \cite{benaloh1986secret}, Benaloh observed that many secret sharing schemes have a homomorphic property which allows multiple secrets to be combined by direct computation on shares.

Suppose we have a secret $m$ that can be computed from $T$ sub-secrets $\{m_i\}^T_{i=1}$ using a functions $f$ as follows $m = f(m_1, \dots, m_T)$.
Let $\oplus$ and $\otimes$ be binary functions on elements of the secret domain $S$ and of the share domain $T$, respectively.
We say that a $(k, N)$ threshold scheme has the $(\oplus, \otimes)$-homomorphism property (or is $(\oplus, \otimes)$-homomorphic) if for all $f: T^k \rightarrow S$, whenever a message

\hfil $m = f(m_1, \dots, m_k)$

and another message

\hfil  $m' = f(m'_1, \dots, m'_k)$

then the composition of the shares are the shares of the composition:

\hfil $m \oplus m' = f(m_1 \otimes m'_1, \dots, m_k \otimes m'_k)$

Shamir’s polynomial based secret sharing \ref{ss:shamir-secret-sharing} scheme is $(+, +)$-homomorphic since the sum of the polynomials use to share the sub-secrets is itself a polynomial that can be used to deal shares of the super secret.


\subsection{Axiom schema of specification}

\subsubsection{The axiom schema of unrestricted comprehension}

We want to formalise the relationship between the preterite and the set. An obvious way of doing this is to add an axiom for each preterite in our structure that:

\(\forall x \exists s[P(x)\leftrightarrow (x\in s)]\)

This is known as "unrestricted comprehension" and there are problems with this approach.

Consider a predicate for all terms which are not members of themselves. That is:

\(\neg (x\in x)\)

This implies the following is true:

\(\forall x\exists s[\neg (x\in x) \leftrightarrow (x\in s)]\)

As this is true for all \(x\), it is true for \(x=s\). So:

\(\exists s[\neg (s\in s) \leftrightarrow (s  \in s)]\)

This statement is false. As we have inferred a false formula, the axiom of unrestricted comprehension does not work. This result is known as Russel's Paradox.

This is an axiom schema rather than an axiom. That is, there is a new axiom for each preterite.

\subsubsection{Axiom schema of specification}

To resolve Russel's paradox, we amend the axiom schema to:

\(\forall x \forall a \exists s[(P(x)\land x\in a )\leftrightarrow (x\in s)]\)

That is, for every set \(a\), we can define a subset \(s\) for each predicate.

This resolves Russel's Paradox. Let's take the same steps on the above formula as in unrestricted comprehension;

\(\forall x \forall a \exists s[(\neg (x\in x)\land x\in a )\leftrightarrow (x\in s)]\)

\(\exists s[(\neg (s\in s)\land s\in s )\leftrightarrow (s\in s)]\)

So long as the subsets \(s\) are not members of themselves, this holds.


\subsection{Ordered sets}

\subsubsection{Totally ordered sets}

A totally ordered set is one where the relation is defined on all pairs:

\(\forall a \forall b	(a\le b)\lor (b\le a)\)

Note that totality implies reflexivity.

\subsubsection{Partially ordered sets (poset)}

A partially ordered set, or poset, is one where the relation is defined between each element and itself.

\(\forall a (a\le a)\)

That is, every element is related to itself.

These are also called posets.

\subsubsection{Well-ordering}

A well-ordering on a set is a total order on the set where the set contains a minimum number. For example the relation \(\le \) on the natural numbers is a well-ordering because \(0\) is the minimum.

The relation \(\le \) on the integers however is not a well-ordering, as there is no minimum number in the set.


\section{Simple Benchmark}
To better illustrate the benefits of GPUs we performed a few simple benchmarks:
we timed the MC calculation of $\pi$ on a GPU using the Tensorflow (v2.5.0-2) package and compared it
to an equivalent implementation using the NumPy (v1.20.3-1) package.
All benchmarks were performed on a Linux platform using a Ryzen 3700X CPU and an NVIDIA GTX 1070 GPU.
The CPU and GPU are roughly equal in terms of price and release date.
\subsection{Hit-And-Miss MC}
The calculation of $\pi$ using hit-and-miss MC can be implemented with the following Python code:
\begin{lstlisting}
@tf.function
def mc_tf(sample_size):
    rand_xy = tf.random.uniform(
	(sample_size, 2))

    # Map:
    in_circle = tf.square(rand_xy[:, 0]) \
	+ tf.square(rand_xy[:, 1]) < 1.0

    # Reduce:
    mc_estimate = tf.math.count_nonzero(
	in_circle) / sample_size

    return 4 * mc_estimate
\end{lstlisting}
Notably the entire calculation can be expressed as a combination of map and reduce operations,
implying that it can be efficiently parallelized on GPUs.
\begin{figure*}
	\centering
	\includegraphics[width=\linewidth]{pi_hit_and_miss_benchmark.png}
	\caption{
		Benchmark of hit-and-miss MC.
		Runtime with NumPy is proportional to sample size.
		Runtime with Tensorflow is constant for $N \le 10^7$.
		Runtime with Tensorflow is proportional to sample size for $N \ge 10^9$
		(assuming Tensorflow is already initialized).
	}
	\label{fig:pi_hit_and_miss_benchmark}
\end{figure*}
Figure \ref{fig:pi_hit_and_miss_benchmark} shows the result of the benchmark.
The runtime with NumPy (single CPU core) is proportional to sample size, indicating negligible overhead.
The runtime with Tensorflow (GPU) is flat for small sample sizes and proportional to sample size for large sample sizes,
indicating significant overhead.
For small sample sizes NumPy is much faster than Tensorflow, for large sample sizes Tensorflow is much faster than NumPy.

A large part of the GPU overhead is due to the initialization of the Tensorflow library which is several gigabytes in size.
However, some overhead still remains when this is accounted for.
The reason for this overhead is the memory management described in section \ref{sec:programming}.
Compared to computation memory accesses are very slow.
Memory transfers between system memory and GPU memory are even slower.
To execute the above code on a GPU the generated kernel needs to first be transferred to GPU memory.
After executing code the result needs to be transferred back to system memory.
For small sample sizes the runtime is entirely dominated by the memory transfers,
resulting in a runtime that is essentially constant.
For large sample sizes the time needed for memory transfers becomes negligible,
resulting in a runtime that is proportional to sample size.
\subsection{Crude MC}
The calculation of $\pi$ using crude MC can be implemented with the following Python code:
\begin{lstlisting}
@tf.function
def mc_tf(sample_size):
    rand_x = tf.random.uniform(
	(sample_size,))

    # Map:
    function_values = 1.0 - tf.math.sqrt(
	1.0 - tf.math.square(rand_x))

    # Reduce:
    mc_estimate = tf.reduce_mean(
	function_values)

    return 4 * mc_estimate
\end{lstlisting}
Again, the entire calculation can be expressed as a combination of map and reduce operations.
\begin{figure*}
	\includegraphics[width=\linewidth]{pi_crude_benchmark.png}
	\caption{
		Benchmark of crude MC.
		Runtime with NumPy is proportional to sample size.
		Runtime with Tensorflow (GPU) is constant for $N <= 10^7$.
		Runtime with Tensorflow (CPU and GPU) is proportional to sample size for $N >= 10^9$
		(assuming Tensorflow is already initialized).
	}
	\label{fig:pi_crude_benchmark}
\end{figure*}
Figure \ref{fig:pi_crude_benchmark} shows the result of the benchmark.
The results for NumPy and Tensorflow (GPU) are essentially the same as for the hit-and-miss implementation.
Additionally to Tensorflow (GPU) we also benchmarked Tensorflow (CPU) to investigate the runtime
of a multithreaded CPU implementation.
Just like with Tensorflow (GPU) there is significant overhead that becomes negligible for large sample sizes.
The overhead is due to the cost associated with thread creation.
At least in this benchmark Tensorflow (GPU) was always faster than Tensorflow (CPU), particularly for large sample sizes.


\chapter{Group actions overkill AIME problems}
Consider this problem from the 1996 AIME:
\begin{quote}
	(AIME 1996) Two of the squares of a $7 \times 7$ checkerboard are painted yellow, and the rest are painted green. Two color schemes are equivalent if one can be obtained from the other by applying a rotation in the plane of the board. How many inequivalent color schemes are possible?
\end{quote}

What's happening here? Let $X$ be the set of the $\binom{49}{2}$ possible colorings of the board.
What's the natural interpretation of ``rotation''?
Answer: the group $\Zc 4 = \left<r \mid r^4=1\right>$ somehow ``acts'' on this set $X$ by sending one state $x \in X$ to another state $r \cdot x$, which is just $x$ rotated by $90\dg$.
Intuitively we're just saying that two configurations are the same if they can be reached from one another by this ``action''.

We can make all of this precise using the idea of a group action.

\section{Definition of a group action}
\prototype{The AIME problem.}

\begin{definition}
	Let $X$ be a set and $G$ a group.
	A \vocab{group action} is a binary operation $\cdot : G \times X \to X$
	which lets a $g \in G$ send an $x \in X$ to $g \cdot x$.
	It satisfies the axioms
	\begin{itemize}
		\ii $(g_1g_2) \cdot x = g_1 \cdot (g_2 \cdot x)$ for any $g_1, g_2 \in G$
		for all $x \in X$.
		\ii $1_G \cdot x = x$ for any $x \in X$.
	\end{itemize}
\end{definition}

\begin{example}[Examples of group actions]
	Let $G=(G,\star)$ be a group.
	\begin{enumerate}[(a)]
		\ii The group $\Zc 4$ can act on the set of ways to color a $7 \times 7$
		board either yellow or green.
		\ii The group $\Zc4 = \left<r \mid r^4=1\right>$ acts on the $xy$-plane $\RR^2$ as follows: $r \cdot (x,y) = (y,-x)$.
		In other words, it's a rotation by $90\dg$.
		\ii The dihedral group $D_{2n}$ acts on the set of ways to color the vertices of an $n$-gon.
		\ii The group $S_n$ acts on $X = \left\{ 1,2,\dots,n \right\}$
		by applying the permutation $\sigma$: $\sigma \cdot x \defeq \sigma(x)$.
		\ii The group $G$ can act on itself (i.e.\ $X=G$) by left multiplication: put $g \cdot g' \defeq g \star g'$.
	\end{enumerate}
\end{example}

\section{Stabilizers and orbits}
\prototype{Again the AIME problem.}

Given a group action $G$ on $X$,
we can define an equivalence relation $\sim$ on $X$ as follows:
$x \sim y$ if $x = g \cdot y$ for some $g \in G$.
For example, in the AIME problem, $\sim$ means ``one can be obtained from the other by a rotation''.
\begin{ques}
	Why is this an equivalence relation?
\end{ques}
In that case, the AIME problem wants the number of equivalence classes under $\sim$.
So let's give these equivalence classes a name: \vocab{orbits}.
We usually denote orbits by $\OO$.

As usual, orbits carve out $X$ into equivalence classes.
\begin{center}
	\begin{asy}
		bigbox("$X$");
		draw(ellipse(origin,0.8,2.5));
		draw(ellipse((-2,0),0.8,1.5));
		draw(ellipse(( 2,0),0.8,2.25));
		for (int i=-3; i<=3; ++i) {
			dot( (0, 0.7*i) );
		}
		dot( (-2,0) );
		dot( (-2,-1) );
		dot( (-2,1) );
		dot( ( 2,0) );
		dot( ( 2,-1) );
		dot( ( 2,1) );
		dot( ( 2,-2) );
		dot( ( 2,2) );

		MP("\mathcal O_1", (-2,-1.5), dir(225));
		MP("\mathcal O_2", ( 0,-2.5), dir(180));
		MP("\mathcal O_3", ( 2,-2.25), dir(225));
	\end{asy}
\end{center}

It turns out that a very closely related concept is:
\begin{definition}
	The \vocab{stabilizer} of a point $x \in X$,
	denoted $\Stab_G(x)$, is the set of $g \in G$ which fix $x$; in other words
	\[ \Stab_G(x) \defeq \left\{ g \in G \mid g \cdot x = x \right\}. \]
\end{definition}
\begin{example}
	Consider the AIME problem again, with $X$ the possible set of states
	(again $G = \Zc4$).
	Let $x$ be the configuration where two opposite corners are colored yellow.
	Evidently $1_G$ fixes $x$, but so does the $180\dg$ rotation $r^2$.
	But $r$ and $r^3$ do not preserve $x$, so
	$\Stab_G(x) = \{1,r^2\} \cong \Zc2$.
\end{example}
\begin{ques}
	Why is $\Stab_G(x)$ a subgroup of $G$?
\end{ques}

Once we realize the stabilizer is a group, this leads us to what I privately call the ``fundamental theorem of how big an orbit is''.
\begin{theorem}[Orbit-stabilizer theorem]
	Let $\OO$ be an orbit, and pick any $x \in \OO$.
	Let $S = \Stab_G(x)$ be a subgroup of $G$.
	There is a natural bijection between $\OO$ and left cosets.
	In particular,
	\[ \left\lvert \OO \right\rvert \left\lvert S \right\rvert = \left\lvert G \right\rvert. \]
	In particular, the stabilizers of each $x \in \OO$ have the same size.
\end{theorem}
\begin{proof}
	The point is that every coset $gS$ just specifies an element of $\OO$,
	namely $g \cdot x$. The fact that $S$ is a stabilizer implies
	that it is irrelevant which representative we pick.

	\begin{center}
		\begin{asy}
			size(6cm);
			draw(ellipse(origin, 0.5, 2));
			label("$\mathcal O \subseteq X$", (0,2), dir(90));
			for (real i=-1.5; i<=1.5; ++i) {
				dot( (0,i) );
			}
			draw( (0.3,1.8)--(0.3,1.2)--(4,1.2)--(4,1.8)--cycle );
			label("$S \subseteq G$", (4,1.5), dir(0));
			for (real i=0.7; i < 4; i+=0.7) {
				label("$\circ$", (i, 1.5), origin);
			}
			draw( (-0.2,1.5)..(-1,0.5)..(-0.2,-0.5), EndArrow);
			label("$g$", (-1,0.5), dir(180));
		\end{asy}
	\end{center}

	Since the $\left\lvert \mathcal O \right\rvert$ cosets partition $G$,
	each of size $\left\lvert S \right\rvert$, we obtain the second result.
\end{proof}

\section{Burnside's lemma}
Now for the crux of this chapter: a way to count the number of orbits.
\begin{theorem}
	[Burnside's lemma]
	Let $G$ act on a set $X$.
	The number of orbits of the action is equal to
	\[ \frac{1}{\left\lvert G \right\rvert}
		\sum_{g \in G} \left\lvert \FixPt g \right\rvert \]
	where $\FixPt g$ is the set of points $x \in X$
	such that $g \cdot x = x$.
\end{theorem}
The proof is deferred as a bonus problem,
since it has a very olympiad-flavored solution.
As usual, this lemma was not actually proven by Burnside;
Cauchy got there first, and thus it is sometimes called
\emph{the lemma that is not Burnside's}.
Example application:
\begin{example}
	[AIME 1996]
	{ \footnotesize Two of the squares of a $7 \times 7$ checkerboard are painted yellow, and the rest are painted green. Two color schemes are equivalent if one can be obtained from the other by applying a rotation in the plane of the board. How many inequivalent color schemes are possible?  }

	We know that $G = \Zc4$ acts on the set $X$ of $\binom{49}{2}$ possible coloring schemes.
	Now we can compute $\FixPt g$ explicitly for each $g \in \Zc4$.
	\begin{itemize}
		\ii If $g = 1_G$, then every coloring is fixed, for a count of $\binom{49}{2} = 1176$.
		\ii If $g = r^2$ there are exactly $24$ coloring schemes fixed by $g$:
		this occurs when the two squares are reflections across the center,
		which means they are preserved under a $180\dg$ rotation.
		\ii If $g = r$ or $g=r^3$, then there are no fixed coloring schemes.
	\end{itemize}
	As $\left\lvert G \right\rvert = 4$, the average is
	\[ \frac{1176 + 24 + 0 + 0}{4} = 300. \]
\end{example}

\begin{exercise}[MathCounts Chapter Target Round]
	A circular spinner has seven sections of equal size,
	each of which is colored either red or blue.
	Two colorings are considered the same if one can be rotated to yield the other.
	In how many ways can the spinner be colored? (Answer: 20)
\end{exercise}
% The group in question is $\Zc7$; the answer should be $20$.

Consult \cite{ref:aops_burnside}
for some more examples of ``hands-on'' applications.

\section{Conjugation of elements}
\prototype{In $S_n$, conjugacy classes are ``cycle types''.}
A particularly common type of action is the so-called \vocab{conjugation}.
We let $G$ act on itself as follows:
\[ g : h \mapsto ghg\inv. \]
You might think this definition is a little artificial.
Who cares about the element $ghg\inv$?
Let me try to convince you this definition is not so unnatural.
\begin{example}
	[Conjugacy in $S_n$]
	Let $G = S_5$, and fix a $\pi \in S_5$.
	Here's the question: is $\pi \sigma \pi \inv$ related to $\sigma$?
	To illustrate this,
	I'll write out a completely random example of a permutation $\sigma \in S_5$.
	\[
		\text{If }
		\sigma = \;
		\begin{array}{ccc}
		1 & \mapsto & 3 \\
		2 & \mapsto & 1 \\
		3 & \mapsto & 5 \\
		4 & \mapsto & 2 \\
		5 & \mapsto & 4
		\end{array}
		\qquad
		\text{then}
		\qquad
		\pi \sigma \pi\inv =
		\begin{array}{ccc}
		\pi(1) & \mapsto & \pi(3) \\
		\pi(2) & \mapsto & \pi(1) \\
		\pi(3) & \mapsto & \pi(5) \\
		\pi(4) & \mapsto & \pi(2) \\
		\pi(5) & \mapsto & \pi(4)
		\end{array}
	\]
	Thus our fixed $\pi$ doesn't really change the structure of $\sigma$ at all:
	it just ``renames'' each of the elements $1$, $2$, $3$, $4$, $5$
	to $\pi(1)$, $\pi(2)$, $\pi(3)$, $\pi(4)$, $\pi(5)$.
\end{example}
But wait, you say.
That's just a very particular type of group behaving nicely under conjugation.
Why does this mean anything more generally?
All I have to say is: remember Cayley's theorem!
(This was \Cref{thm:cayley_theorem}.)

In any case, we may now define:
\begin{definition}
	The \vocab{conjugacy classes} of a group $G$ are the orbits of $G$ under
	the conjugacy action.
\end{definition}

Let's see what the conjugacy classes of $S_n$ are, for example.
\begin{example}
	[Conjugacy classes of $S_n$ correspond to cycle types]
	Intuitively, the discussion above says that two elements of $S_n$ should
	be conjugate if they have the same ``shape'', regardless of what the elements are named.
	The right way to make the notion of ``shape'' rigorous is cycle notation.
	For example, consider the permutation
	\[ \sigma_1 = (1 \; 3 \; 5)(2 \; 4) \]
	in cycle notation, meaning $1 \mapsto 3 \mapsto 5 \mapsto 1$ and $2 \mapsto 4 \mapsto 2$.
	It is conjugate to the permutation
	\[ \sigma_2 = (1 \; 2 \; 3)(4 \; 5) \]
	or any other way of relabeling the elements.
	So, we could think of $\sigma$ as having conjugacy class
	\[ (- \; - \; -)(- \; -). \]
	More generally, you can show that two elements of $S_n$ are conjugate
	if and only if they have the same ``shape'' under cycle decomposition.
\end{example}
\begin{ques}
	Show that the number of conjugacy classes of $S_n$
	equals the number of \emph{partitions} of $n$.
\end{ques}

As long as I've put the above picture, I may as well also define:
\begin{definition}
	Let $G$ be a group.
	The \vocab{center} of $G$, denoted $Z(G)$, is the set of elements $x \in G$
	such that $xg = gx$ for every $g \in G$.
	More succinctly,
	\[ Z(G) \defeq \left\{ x \in G \mid gx=xg \; \forall g \in G \right\}. \]
\end{definition}
You can check this is indeed a subgroup of $G$.
\begin{ques}
	Why is $Z(G)$ normal in $G$?
\end{ques}
\begin{ques}
	What are the conjugacy classes of elements in the center?
\end{ques}

A trivial result that gets used enough that I should explicitly call it out:
\begin{corollary}[Conjugacy in abelian groups is trivial]
	If $G$ is abelian, then the conjugacy classes all have size one.
\end{corollary}

\section\problemhead
\begin{problem}
	[PUMaC 2009 C8]
	Taotao wants to buy a bracelet consisting of seven beads,
	each of which is orange, white or black.
	(The bracelet can be rotated and reflected in space.)
	Find the number of possible bracelets.
	\begin{hint}
		Just apply Burnside's lemma directly to get the answer of $198$
		(the relevant group is $D_{14}$).
	\end{hint}
\end{problem}

\begin{problem}
	Show that two elements in the same conjugacy class
	have the same order.
	\begin{hint}
		There are multiple ways to see this.
		One is to just do the algebraic manipulation.
		Another is to use Cayley's theorem to embed $G$ into a symmetric group.
	\end{hint}
\end{problem}

\begin{problem}
	\gim
	Prove Burnside's lemma.
	\begin{hint}
		Double-count pairs $(g,x)$ with $g \cdot x = x$.
	\end{hint}
\end{problem}

\begin{sproblem}
	[The ``class equation'']
	\label{prob:class_eq}
	Let $G$ be a finite group.
	We define the \vocab{centralizer} $C_G(g) = \{ x \in G \mid xg = gx \}$
	for each $g \in G$.
	Show that
	\[ \left\lvert G \right\rvert = \left\lvert Z(G) \right\rvert + \sum_{s \in S}
	\frac{\left\lvert G \right\rvert}{\left\lvert C_G(s) \right\rvert} \]
	where $S \subseteq G$ is defined as follows:
	for each conjugacy class $C \subseteq G$ with $|C| > 1$,
	we pick a representative of $C$ and add it to $S$.
\end{sproblem}

\begin{dproblem}
	[Classical]
	\gim
	Assume $G$ is a finite group and $p$ is the smallest prime dividing its order.
	Let $H$ be a subgroup of $G$ with $\left\lvert G \right\rvert / \left\lvert H \right\rvert = p$.
	Show that $H$ is normal in $G$.
	\begin{hint}
		Let $G$ act on the left cosets $\{gH \mid g \in G\}$
		by left multiplication: $g' \cdot gH = g'gH$.
		Consider the orbit $\OO$ of the coset $H$.
		By the orbit-stabilizer theorem, $\left\lvert \OO \right\rvert$ divides $\left\lvert G \right\rvert$.
		But $\left\lvert \OO \right\rvert \le p$ also.
		So either $\OO = \{H\}$ or $\OO$ contains all cosets.
		The first case is impossible.
	\end{hint}
\end{dproblem}


\chapter{Multi-Layer Perceptron}
The network is trained in a supervised manner with the error back-propagation algorithm based on the error correction learning rule.

\section{Delta Learning Rule of the Output Layer}
\begin{figure}[h]
\centering
\includegraphics[width=10cm]{chapter3_mlp}
\end{figure}
\noindent Total synaptic input to the output layer:
$$\mathbf{u = Wy}$$
Output of the network:
$$\mathbf{o} = f(\mathbf{u})$$
Desired output for pattern $p$:
$$\mathbf{d_p}=(d_{p1}, d_{p2} \ldots d_{pK})^{T}$$
The error at output layer:
$$E_p=\frac{1}{2}\sum_{k=1}^{K}(d_{pk}-o_{pk})^2$$
For gradient descent update algorithm:
\begin{equation}
\triangle w_{kj} = - \alpha \frac{\partial E}{\partial w_{kj}}
\label{weight_change}
\end{equation}
By using chain rule, the gradient of error function:
\begin{equation}
\frac{\partial E}{\partial w_{kj}} = \frac{\partial E}{\partial u_k} \frac{\partial u_k}{\partial w_{kj}}
\label{error_derive_chain_rule}
\end{equation}
The \emph{error signal term} produce by the $k$ th neuron is defined as:
\begin{equation}
\delta_{ok} \equiv - \frac{\partial E}{\partial u_k}
\label{signal_term}
\end{equation}
Since
\begin{equation}
\begin{split}
u_k &= \sum_{j=1}^{J} w_{kj} y_j\\
\frac{\partial u_k}{\partial w_{kj}} &= y_j
\end{split}
\label{input_derive_mlp}
\end{equation}
Substituting \ref{signal_term} and \ref{input_derive_mlp} to \ref{error_derive_chain_rule}:
\begin{equation}
\frac{\partial E}{\partial w_{kj}} = - \delta_{ok} y_j
\label{error_derive_2}
\end{equation}
The weight adjustment in \ref{weight_change} can be rewritten:
\begin{equation}
\triangle w_{kj} = \alpha \delta_{ok} y_j
\label{weight_change_2}
\end{equation}
The error signal term can be written as:
\begin{equation}
\delta_{ok}=-\frac{\partial E}{\partial o_k}\frac{\partial o_k}{\partial u_k}
\label{signal_term_chain_rule}
\end{equation}
Since
\begin{equation}
\begin{split}
E = \frac{1}{2} \sum_{k=1}^{K} (d_k - o_k)^{2} \\
\frac{\partial E}{\partial o_k} = - (d_k - o_k)
\end{split}
\label{error_derive_mlp}
\end{equation}
And since
\begin{equation}
\begin{split}
o_k &= f(u_k) \\
\frac{\partial o_k}{\partial u_k} &= f_{k}'(u_k)
\end{split}
\label{output_derive_mlp}
\end{equation}
Subtituting \ref{output_derive_mlp} and \ref{error_derive_mlp} to \ref{error_derive_mlp}:
\begin{equation}
\begin{split}
\delta_{ok} = (d_k -o_k)f_{k}'(u_k)
\end{split}
\label{signal_term_2}
\end{equation}
Final equation of weight adjustment:
$$\triangle w_{kj} = \alpha (d_k - o_k)f_{k}'(u_k) y_j$$
The learning equation:
\begin{equation*}
\begin{split}
w_{kj}^{new} &= w_{kj}^{old} + \triangle w_{kj} \\
w_{kj}^{new} &= w_{kj}^{old} + \alpha \delta_{ok} y_j
\end{split}
\end{equation*}
Rewritten the learning equation in matrix form:
$$\mathbf{W^{new} = W^{old} + \alpha \delta_o y^{T}}$$
$$\mathbf{\delta_o = (d-o) .* f'(u)}$$

\section{Generalized Delta Learning Rule}
\begin{figure}[h]
\includegraphics[width=10cm]{chapter3_mlp_2}
\end{figure}
\noindent The weight adjustment of hidden layer neurons:
\begin{equation}
\triangle v_{ji} = -\beta \frac{\partial E}{\partial v_{ji}}
\label{weight_adjust_hidden}
\end{equation}
Using chain rule:
\begin{equation}
\frac{\partial E}{\partial v_{ji}} = \frac{\partial E}{\partial s_j} \frac{\partial s_j}{\partial v_{ji}}
\label{error_derive_hidden}
\end{equation}
Since the net synaptic input to hidden layer neuron is:
\begin{equation}
\begin{split}
s_j &= \sum_{i} v_{ji} z_i\\
\frac{\partial s_j}{\partial v_{ji} } &= z_i
\end{split}
\label{input_derive_hidden}
\end{equation}
The error signal term at hidden layer is defined as:
\begin{equation}
\delta_{yj} \equiv - \frac{\partial E}{\partial s_j}
\label{error_term_hidden}
\end{equation}
Substituting \ref{input_derive_hidden} and \ref{error_term_hidden} into \ref{error_derive_hidden}:
$$\frac{\partial E}{\partial v_{ji}} = - \delta_{yj} z_i$$
Equation \ref{weight_adjust_hidden} can be written as:
$$ \triangle v_{ji} = \beta \delta_{yj} z_i $$
The error signal term in equation \ref{error_term_hidden} can be computed as:
\begin{equation}
\delta_{yj} = - \frac{\partial E}{\partial y_j} \frac{\partial y_j}{\partial s_j}
\label{error_term_derive_hidden}
\end{equation}
Where:
\begin{equation}
\begin{split}
E &= \frac{1}{2} \sum_{k=1}^{K} (d_k - f_k (u_k))^{2} \\
\frac{\partial E}{\partial y_j} &= - \sum_{k=1}^{K} (d_k - o_k) \frac{\partial}{\partial y_j} (f_k (u_k)) \\
&= - \sum_{k=1}^{K} (d_k - o_k) f_k' (u_k) \frac{\partial u_k}{\partial y_j} \\
&= - \sum_{k=1}^{K} \delta_{ok} \frac{\partial u_k}{\partial y_j}
\end{split}
\label{temp1}
\end{equation}
Since:
\begin{equation}
\begin{split}
u_k &= \sum_{j=1}^{J} w_{kj} y_j \\
\frac{\partial u_k}{\partial y_j} &= w_{kj}
\end{split}
\label{temp2}
\end{equation}
And since:
\begin{equation}
\begin{split}
y &= f(s) \\
\frac{\partial y}{\partial s_j} &= f_{j}'(s_j)
\end{split}
\label{temp3}
\end{equation}
Substituting \ref{temp1} \ref{temp2} \ref{temp3} into \ref{error_term_derive_hidden}:
$$\delta_{yj} = f_j'(s_j)\sum_{k=1}^{K} \delta_{ok} w_{kj}$$
Final equation of weight adjustment:
$$\triangle v_{ji} = \beta \delta_{yj} z_i$$
The learning equations:
\begin{equation*}
\begin{split}
v_{ji}^{new} &= v_{ji}^{old} + \triangle v_{ji} \\
v_{ji}^{new} &= v_{ji}^{old} + \beta \delta_{yj} z_i
\end{split}
\end{equation*}
Rewritten the learning equation in matrix form:
$$\mathbf{V^{new} = V^{old}} + \beta \mathbf{\delta_y z^{T}}$$
$$\mathbf{\delta_y = f'(s) .* W^{T} \delta_o}$$

\section{Application to Function Approximation}
Kolmogorov Theorem: Given any function $\mathbf{\varphi: I^{n} \rightarrow R^{m}, \varphi(x) = y}$ where $I$ is the closed unit interval [0, 1], $\phi$ can be implemented exactly by a three layer neural network with $n$ input node, $2n+1$ hidden layer neurons and $m$ output layer neurons


\subsection{Adjacency matrix}

We can represent a finite graph as a square matrix. \(m_{ij}\) is the number of edges connecting vertex \(i\) to vertex \(j\).


\subsubsection{Self-Similar Models --- Theory}

The pressure normalization of SZ models can also be related directly
to the mass of the cluster, through considerations of self-similarity.

We start from the characteristic temperature for an isothermal sphere
of mass \mathM500\

\begin{equation}
k\T500 = \mu m_p {{G \M500}\over{2\R500}}
\label{eq:t500}
\end{equation}

where \mathM500\ is given by

\begin{equation}
\M500 = 500  \rho_c(z) {{4\pi}\over{3}}{\R500}^3
\label{eq:r500}
\end{equation}

and

\begin{equation}
\rho_c(z) = {{3 H(z)^2}\over{8\pi G}}.
\end{equation}

The characteristic gas density is

\begin{equation}
\rho_{g,500} = 500 f_B\rho_c(z)
\end{equation}

and the electron density is

\begin{equation}
n_{e,500} = {\rho_{g,500}\over{\mu_e m_p}}
\end{equation}

Putting it all together, we have

\begin{eqnarray}\nonumber
\P500 = n_{e,500} \times k\T500 &=& {\rho_{g,500}\over{\mu_e m_p}} \times \mu m_p {{G \M500}\over{2\R500}}\\\nonumber
&=& {{\mu}\over{\mu_e}}\rho_{g,500} \times {{G \M500}\over{2\R500}}\\
&=& {{\mu}\over{\mu_e}} 500 f_B\rho_c(z) \times {{G \M500}\over{2\R500}}
\end{eqnarray}

From Eq.\ref{eq:r500} we have

\begin{equation}
\R500 = \left({3\over{4\pi}}{\M500\over{500 \rho_c(z)}}\right)^{1/3}
\end{equation}

whence

\begin{eqnarray}\nonumber
\P500 &=& {{\mu}\over{\mu_e}} 500 f_B\rho_c(z) \times {{G \M500}\over{2\R500}}\\\nonumber
      &=& {{\mu}\over{\mu_e}} 500 f_B\rho_c(z) \times {{G\M500}\over{2}} \left({{4\pi}\over{3}}{500\rho_c(z)\over{\M500}}\right)^{1/3}\\\nonumber
      &=& {{\mu}\over{\mu_e}} f_B G {{3}\over{8\pi}} \left( {{4\pi}\over{3}} 500\rho_c(z) \right)^{4/3}\M500^{2/3}\\\nonumber
      &=& {{\mu}\over{\mu_e}} f_B G {{3}\over{8\pi}} \left( {{4\pi}\over{3}} 500 {{3 H(z)^2}\over{8\pi G}} \right)^{4/3}\M500^{2/3}\\
      &=& {{\mu}\over{\mu_e}} f_B {{3}\over{8\pi}} \left( {{500 G^{-1/4} H(z)^2}\over{2}} \right)^{4/3}\M500^{2/3}
\label{eq:p500}
\end{eqnarray}


\subsection{Dot product}

\subsubsection{Dot product is a bilinear form}

This is a bilinear form, a mapping from two vectors in the same vector space to the underlying field.

\(V\times V \rightarrow F\)

\subsubsection{Calculating the dot product}

This is calculated by multiplying each matching element, and summing the results.

\(u\cdot v =\sum_{i=1}^nu_iv_i\)

\subsubsection{Dot product on the complex numbers}

Properties don't hold. Can get zero vectors from non-zero inputs. Get complex numbers from dot product on itself.

Inner products better deal with complex number fields. However they are not bilinear maps.


\section{Introduction}
This paper attempts to summarise some important defintions and theorems in theory of dynamical systems (DS). We only presume basic knowledge of topology and analysis. When we say 'time interval' we simply refer to a connected open subset $U$ of $\rz$ and 'time' to an element in $U$.
\subsection{Background}
Let $(X,d)$ be a metric space, that is there exists a map
$$d : X \times X \longrightarrow \rz,$$
such that $d(x,y) \geq 0$, for all $x, y \in X$, $d(x,x) = 0$ and
$$d(x,z) \leq d(x,y) + d(y,z),\ \forall x, y, z \in X.$$
\subsubsection{Dynamical systems}
Given a metric space $(X,d)$ and map $f : X \longrightarrow X$, a time discrete dynamical system (TD) is a map
$$\phi : \nz \times X \longrightarrow X, (n,x) \longmapsto f^n (x) := \underbrace{f \circ \ldots \circ f}_{n-\mathrm{times}}(x)$$
inducing a sequence $(f^n(x))_{n \in \nz} \subset X$ for all $x \in X$ and $n \geq 0$. A time-continuous dynamical system (TC) is a map
$$\varphi : \rz \times X \longrightarrow X,$$
such that $\varphi(0,x) = x$ and $\varphi(t + s,x) = \varphi(s,\varphi(t,x)) = \varphi(t,\varphi(s,x))$ for all $t, s \in U \subset \rz$ open and $x \in X$. We denote a TC as
$$(X, d, \varphi, U).$$
We remark that a given a time-continous dynamical system $(X,d,\varphi)$ we get a family of maps $\{f_t : X \longrightarrow X\}_{t \in U}$ indexed by time such that $f_0 = id_X$ and $\varphi(t,x) = f_t(x)$ for all $x \in X$ and $t \in \rz$, as well as
$$f_s(f_t(x)) = f_t(f_s(x)).$$
Furthermore, we get a time-discrete DS when we restrict the domain accordingly. What this entails will be later discussed.\\
\indent In addition, we restrict the time domain to an open subset as there are examples of smooth TCs which do not exist for all times:
$$\partial(x) = x^2 \Rightarrow \varphi(t,x_0) = \frac{1}{1 - x_0(t - t_0)}, \forall t \geq t_0$$
has no smooth solution $x \in C^\infty(\rz)$ for all initial conditions $(t_0,x_0) \in U \times X$ (only locally smooth solution).
\begin{defi}
Let $(X, d, \varphi, U)$ be a TC and $x_0 \in X$. We call a map $x : U \longrightarrow X$ a solution if $x(t) = \varphi(t,x_0)$ for all $t \in U$. If $U = \rz$ we call $x \in X^U$ a global solution.
\end{defi}
\subsubsection{Attractors and basin of attraction}
Until now, we did not need the metric $d$. However, the next definition requires this map. Moreover, let $X$ be complete wrt. to the metric induced topology.
\begin{defi}[Metric attractors]
Let $(X, d, \varphi, U)$ be a TC. A subset $A \subset X$ is called an attractor if there exists an open neighborhood $V \supset A$ such that
$$\forall \varepsilon > 0\ \exists T \in U\ \forall t \geq T:\ d(\varphi(t,x),A) < \varepsilon,$$
for all $x \in V$. This property is denoted as usual as a limit:
$$A := \{x \in V : \lim_{t \rightarrow \infty} \varphi(t,x) \in A\}$$
%We remark that not all metric spaces do have a limit, i.e. $\varphi
We call the union of all neighborhoods $V \supset A$, such that
$$\lim_{t \rightarrow \infty}\varphi(t,x) \in A$$
the basin of attraction, denoted by $B(A)$.
\end{defi}
Again, strictly speaking we only need a (complete) topological space $(X,\tau)$ with a neighborhood basis $\beta(x)$ wrt. $x \in X$:
\begin{defi}[Topological attractor]
Let $(X, \tau, \varphi, U)$ be a TC. A subset $A \subset X$ is called an attractor if there exists an open neighborhood $V \supset A$ such that
$$\forall W(A) \in \beta(A)\ \exists T \in U\ \forall t \geq T:\ \varphi(t,x) \in W(A),$$
for all $x \in V$. We call the union of all neighborhoods $V \supset A$, such that
$$\lim_{t \rightarrow \infty}\varphi(t,x) \in A$$
the basin of attraction, denoted by $B(A)$.
\end{defi}
Here,
$$W(A) = \bigcup_{x \in A} W(x),\ \beta(A) = \bigcup_{x \in A} \beta(x)\ \mathrm{where}\ W(x) \in \beta(x)\ \forall x \in X.$$