\chapter{Precision issues and epsilons}

Computational geometry very often means working with floating-point values. Even when the input points are all integers, as soon as intermediate steps require things like line intersections, orthogonal projections or circle tangents, we have no choice but to use floating-point numbers to represent coordinates.

Using floating-point numbers comes at a cost: loss of precision. The number of distinct values that can be represented by a data type is limited by its number of bits, and therefore many ``simple'' values like $0.1$ or $\sqrt{2}$ cannot be exactly represented. Worse, even if $a$ and $b$ are exact, there is no guarantee that simple operations like $a+b$, $a-b$ or $ab$ will give an exact result.

Though many people are well aware that those issues exist, they will most often argue that they only cause small imprecisions in the answer in the end, and do not have any major consequence or the behavior of algorithms. In the rest of this chapter, we will show how both those assumptions can sometimes be false, then present some ways in which we \emph{can} make accurate statements about how precision loss affects algorithms, go on with a few practical examples, and finally give some general advice to problem solvers and setters.

\input{precision/small-big}
\input{precision/small-break}
\input{precision/modelling}
%\subsection{Notations}
%\todo{}
\input{precision/cases}
\input{precision/advice}


\section*{Introduction}
In this chapter, generalities about systems of Partial Differential Equations (PDEs) are first introduced. More specifically, the notions of \textit{characteristics} and \textit{hyperbolicity} for PDEs are introduced in section \ref{sec:PDEs} for first-order systems, which will be of particular interest in the remainder of the manuscript.
Then, introduction of balance laws of solid dynamics and derivation of constitutive equations from the thermodynamics in section \ref{sec:solidMech_equations}, lead to first-order hyperbolic PDEs.
By using tools introduced in section \ref{sec:PDEs}, the characteristic analysis of these systems is carried out in section \ref{sec:characteristic_analysis} in order to derive exact solutions of particular problems in section \ref{sec:riemann_problems}. These solutions allow the highlighting of different types of waves: (i) discontinuous waves, governed by the \textit{Rankine-Hugoniot} jump condition, within one-dimensional linear elastic and elastic-plastic media (ii) shock waves, also following the Rankine-Hugoniot condition, and simple waves, within a non-linear problem (one-dimensional strain state in a \textit{Saint-Venant-Kirchhoff} hyperelastic medium).
At last, strategies enabling the computation of approximate solutions of non-linear problems are reviewed in section \ref{sec:riemann_solvers}.

\section{Generalities -- Hyperbolic partial differential equations}
\label{sec:PDEs}
\input{chapter2/PDEs}

\section{Governing equations of solid mechanics}
\label{sec:solidMech_equations}
\input{chapter2/conservationLaws}

\section{Characteristic analysis -- Structure of solutions}
\label{sec:characteristic_analysis}
\input{chapter2/characteristicAnalysis}

\section{Some solutions of Riemann problems}
\label{sec:riemann_problems}
\input{chapter2/exact_solutions}
\section{Approximate--State Riemann solvers}
\label{sec:riemann_solvers}
\input{chapter2/riemann_solvers}

\section{Conclusion}
It has been seen in this chapter that solid dynamics balance equations can be written as a first order hyperbolic system whose theory has been recalled in section \ref{sec:PDEs}.
Indeed, the thermodynamics framework assuming generalized standard materials combined with conservation laws allowed in section \ref{sec:solidMech_equations} the building of conservative or quasi-linear forms.
Those systems of partial differential equations admit non-complex eigenvalues and independent eigenvectors provided that some requirements on the stored energy function are satisfied (positive definiteness of the acoustic tensor).
Then, the characteristic analysis of the quasi-linear form in section \ref{sec:characteristic_analysis} enabled the highlighting of specific wave types involved in the solutions of dynamic problems, that is: discontinuous, shock and simple waves.
Even though exact solutions of linear and non-linear problems have been developed in section \ref{sec:riemann_problems}, it is not possible in general, hence the introduction of approximate-state Riemann solvers in section \ref{sec:riemann_solvers}.
This solution strategy will be used in what follows as an element of the \textit{Discontinuous Galerkin Material Point Method}, which is the object of the next chapter.

%%% Local Variables:
%%% mode: latex
%%% ispell-local-dictionary: "american"
%%% TeX-master: "../mainManuscript"
%%% End:


An instance of an inference system begins by defining the set of recognized keywords and operators. Keywords and operators all have an ariety, a class to which they belong, and a priority level to define how lack of parentheses should be interpreted. As an example we will consider the unary notation for defining natural numbers.

\subsection{Inductive definition of natural numbers}
Natural numbers (\texttt{Num}) in the \textit{unary notation}, also known as \textit{Peano numbers}, are defined in terms of one zero value (\texttt{z}), which does not take anything as input and one successor value (\texttt{s}), which takes as input another number\footnote{The successor of a natural number is that number plus one, thus \texttt{s(a)} can be seen as intuitively equivalent to \texttt{a+1}. A word of warning: we are implying a distinction here between the operation \texttt{+1}, which is an elementary operation that we always know how to perform in a single step, and arbitrary addition of potentially large numbers: although the two operations both use the same symbol \texttt{+}, \textbf{they are not the same!}}

\begin{lstlisting}
Data [] "z" []    Priority 2  Type Num
Data [] "s" [Num] Priority 3  Type Num
\end{lstlisting}

With the keywords above it becomes possible to express some natural numbers. A few examples are:
\begin{description}
\item[0] \texttt{z}
\item[1] \texttt{s(z)}
\item[2] \texttt{s(s(z))}
\item[3] \texttt{s(s(s(z)))}
\end{description}

All the examples above are \textit{expressions}, because they only use keywords and composition of keywords. We could also define \textit{patterns}, which can be informally considered as ``expressions with holes'', where the holes are symbols that may be replaced with any valid expression. Consider pattern \texttt{s(s(a))}, which is the pattern that describes the ``successor of the successor of \texttt{a}'', whatever \texttt{a} will be. Possible expressions that may generated from this pattern are:

\begin{itemize}
\item \texttt{s(s(z))}, for \texttt{a = z}
\item \texttt{s(s(s(z)))}, for \texttt{a = s(z)}
\end{itemize}

We call expressions that may generated from a pattern \textit{instances}. We can consider a pattern as a function that takes one or more expressions as parameters and returns a resulting expression.

\subsection{Expressions of natural numbers}
Natural numbers may be composed together to form expressions. An expression is defined recursively as the sum of two expressions, \texttt{+}, or the product of two expressions \texttt{*}:

\begin{lstlisting}
Func [Expr] "+" [Expr] Priority 0  Type Expr => Num
Func [Expr] "*" [Expr] Priority 1  Type Expr => Num
\end{lstlisting}

Notice that numbers were defined in the above as belonging to class \texttt{Num}, whereas addition and multiplication belong to the class \texttt{Expr}, and both expect an \texttt{Expr} to the right and an \texttt{Expr} to the left. So far thus it would be impossible to use numbers as built with \texttt{s} and \texttt{z} as left and right parameters of \texttt{+} and \texttt{*}. In order to connect these two sets of keywords we can specify a relationship between \texttt{Num} and \texttt{Expr}, namely that a \texttt{Num} can be used where an \texttt{Expr} is expected. We can do so with the following code:

\begin{lstlisting}
Num is Expr
\end{lstlisting}

The \texttt{is} operator plays a role that is akin to that of the inheritance operators found in object-oriented languages.\footnote{To be precise, this is an example of \textit{subtype polymorphism}.}

Multiplication takes syntactic precedence over addition, thus we can be slightly less verbose with parentheses just like with the usual definitions of addition and multiplication.

Valid expressions written in this notation could be:

\begin{itemize}
\item \texttt{z + z}
\item \texttt{s(z) + s(z)}
\item \texttt{z + s(z)}
\item \texttt{s(s(z)) * s(s(z))}
\end{itemize}

We can also define patterns such as:

\begin{itemize}
\item \texttt{z + a}, the addition of \texttt{z} and an arbitrary expression \texttt{a}
\item \texttt{s(z) + s(a)}, the addition of \texttt{s(z)} and the successor of an arbitrary expression \texttt{a}
\end{itemize}

\subsection{Addition as repeated succession}
There is no intrinsic difference between keywords such as \texttt{s} and \texttt{z}, and keywords such as \texttt{+} and \texttt{*}, even though one might be tempted to think that only because we used symbols commonly known in arithmetics then they will have their usual meaning here as well. It is very important to realize that this is not the case. We could have just as easily used other symbols such as \texttt{++} for multiplication and \texttt{?} for addition, or anything else we might have fancied. The inference system assumes no prior meaning of symbols, and indeed we are not forced in any way to follow the usual rules of engagement known for numbers. Nevertheless, to reduce the confusion, we will indeed follow such conventions.

We will now assign meaning to the various symbols we just defined. This will allow us to perform transformations on our expressions, in order to perform computations on numbers. We do so by specifiying a \textit{set of rules}. Let us begin with addition.

We can very easily state that adding zero (\texttt{z}) to an arbitrary number \texttt{a} will result in \texttt{a} itself, without any further steps. We express this as a rule without premises as follows:

\begin{lstlisting}
-----------
z + a => a
\end{lstlisting}

The lack of premises above the horizontal bar means that as soon as the inference system recognizes a pattern of the form \texttt{z+a}, then it will immediately yield \texttt{a} as the output result.

Suppose now that we add a number which is not zero to another arbitrary number. Since we are dealing with natural numbers, this means that a non-zero number will always be at least one, thus it will be at least one application of \texttt{s} to some arbitrary number. The input pattern that describes this set of circumstances would be \texttt{s(a) + b}. Given this pattern it is not immediately possible to derive a result: rather, we must decompose the determination of the result into a series of intermediate steps. As a first step, we add \texttt{a} and \texttt{b} together. Then we can return the result by appliying the successor operation once to the result. We can express this as a single premise:

\begin{lstlisting}
a + b => c
-----------------
s(a) + b => s(c)
\end{lstlisting}

Notice that the above is exactly equivalent to the following very simple equation:

$$(a+1) + b = (a+b)+1$$

where $a+b$ is an intermediate value called \texttt{c}, and $a+1$ is the successor of \texttt{a}, thus \texttt{s(a)}.

\subsubsection{Termination proof (informal sketch)}
One might be tempted to wonder how such a blatantly cyclical definition would ever be able to reach any useful conclusion. After all we are defining addition in terms of another addition: does this not equate to ``looping forever''?

We can show, inductively, that we have no infinite looping. Let us define the \texttt{height} of an addition \texttt{a + b} as the number of applications of \texttt{s} within \texttt{a}. For example:

\begin{itemize}
\item \texttt{height(z + z)} is $0$
\item \texttt{height(z + s(z))} is $0$
\item \texttt{height(s(z) + z)} is $1$
\end{itemize}

The \textbf{base case} of our proof is that of height equal to $0$. This means that our addition takes \texttt{z} as a first parameter, and thus rule

\begin{lstlisting}
-----------
z + a => a
\end{lstlisting}

can be applied. Addition terminates in this case, yielding the second term as a result.

The \textbf{inductive case} of our proof is that of height equal to $n>1$. This means that our addition takes \texttt{s(a)} as a first parameter. We cannot apply the previous rule, but we can apply the second:

\begin{lstlisting}
a + b => c
-----------------
s(a) + b => s(c)
\end{lstlisting}

This rule will terminate only if \texttt{a + b => c} terminates. Given that \texttt{height(a + b)} is $n-1$, because of the induction hypothesis we can assume its evaluation will terminate as well. Evaluation of \texttt{s(c)} is just a trivial step which does not pose any risk of recursive behaviour, and thus we can conclude that indeed the process will terminate.

\subsubsection{Example of addition execution}
Consider the following addition: \texttt{s(s(z)) + s(z)}. Let us see how it is evaluated. First we see which rules we can apply. Clearly the first term is not \texttt{z}, thus we must apply

\begin{lstlisting}
a + b => c
-----------------
s(a) + b => s(c)
\end{lstlisting}

where \texttt{a = s(z)} and \texttt{b = s(z)}. Let us perform this replacement:

\begin{lstlisting}
s(z) + s(z) => c
-----------------
s(s(z)) + s(z) => s(c)
\end{lstlisting}

We cannot directly determine \texttt{c}, but we can use \texttt{s(z) + s(z)} as the current expression to evaluate.  Since the first operand of the intermediate addition is not \texttt{z}, then we have to apply the same rule again, this time with \texttt{a = z} and \texttt{b = s(z)}:

\begin{lstlisting}
z + s(z) => c'
--------------------
s(z) + s(z) => s(c')
---------------------- c = s(c')
s(s(z)) + s(z) => s(c)
\end{lstlisting}

Notice that the result of the second application of the rule will yield its own result, which we called \texttt{c'} to disambiguate it with the result of the first application. The successor of the result of the second application will be the result of the intermediate application of the first rule, thus we could also write:

\begin{lstlisting}
z + s(z) => c'
--------------------
s(z) + s(z) => s(c')
---------------------------
s(s(z)) + s(z) => s(s(c'))
\end{lstlisting}

Now we can apply rule

\begin{lstlisting}
-----------
z + a => a
\end{lstlisting}

to the current expression to evaluate, which is \texttt{z + s(z)}, for \texttt{a = s(z)}. This leads us to finding out that \texttt{c' = a = s(z)}, thus yielding:

\begin{lstlisting}
z + s(z) => s(z)
--------------------
s(z) + s(z) => s(s(z))
---------------------------
s(s(z)) + s(z) => s(s(s(z)))
\end{lstlisting}

Let us take a quick step back and realize that \texttt{s(s(z))} is the successor of the succcessor of zero, that is $2$, and \texttt{s(z)} is the successor of zero, that is $1$. Their sum, as computed by our system, is \texttt{s(s(s(z)))}, that is $3$, precisely as we would have expected given common arithmetic sense.

\subsection{Multiplication as repeated addition}
Multiplication is defined in a way that is very similar to addition. We start with a base case of multiplication that we know how to solve immediately. From basic arithmetic we know that multiplication of zero and an arbitrary number always yields zero, thus:

\begin{lstlisting}
-----------
z * a => z
\end{lstlisting}

For the intermediate step, we will use the same decomposition technique that we used for addition. The input pattern will thus be multiplication of a number greater than zero, \texttt{s(a)}, and an arbitrary number \texttt{b}: \texttt{s(a) * b}. To solve this multiplication, first we will multiply \texttt{a} and \texttt{b} together (remember the termination proof that we saw for addition: \texttt{height(a) < height(s(a))}, therefore we expect this smaller multiplication to terminate), and then we add \texttt{b} to the result:

\begin{lstlisting}
a * b => c
c + b => d
--------------
s(a) * b => d
\end{lstlisting}

Notice that the above is exactly equivalent to the following very simple equation:

$$(a+1) \times b = (a \times b) + b$$

where $a \times b$ is an intermediate value called \texttt{c}.

\subsection{Resolving nested operations}
So far we have only defined operators between numbers. As a further level of sophistication, we could define a stronger version of our operations that do not only work with numbers, but rather with arbitrarily nested operations. This means that we wish to be able to support expressions of the form \texttt{a+b+c}, which we currently have not specified how to solve.

We begin with the definition of a new operator which will be responsible for the recursive exploration of an expression in order to resolve its inner operations:

\begin{lstlisting}
Func = "!" LeftArguments = [] RightArguments = [Expr] Priority = 1 Type = Expr => Num
\end{lstlisting}

As a base case we can say that if we encounter a number, then \texttt{!} simply returns it as it was:

\begin{lstlisting}
--------
!z => z

---------------
!(s a) => s a
\end{lstlisting}

The inductive step, on the other hand, works with operations and ensures that both operands have been simplified to numbers before performing the actual operation:

\begin{lstlisting}
!a => a'
!b => b'
a' + b' => c
--------------
!(a + b) => c

!a => a'
!b => b'
a' * b' => c
--------------
!(a * b) => c
\end{lstlisting}

The two rules above are relatively straightforward, and require no change in the definition of addition and multiplication. Thanks to these rules, we can now solve a complex input expression such as \texttt{!(((s(s(z))) * (s(s(z)))) * (s(s(z)) + s(z)))}, equivalent to $2 * 2 * (2 + 1)$), and obtaining as a result the expected \texttt{(s(s(s(s(s(s(s(s(s(s(s(sz))))))))))))}, which is $12$.

\subsection{Closing considerations}
The natural numbers example that we have just seen is very important. Its importance is not of a practical nature, since natural numbers are well known and moreover unary encoding is one of the least efficient ways to manipulate numbers. The importance of the example above is rather conceptual: armed with only the most basic constructs of logical derivation, a highly abstract concept, we have built something that is very concrete and apparently unrelated.

The process of logical derivation is indeed very powerful. In the following we will use it to define increasingly complex systems. We will begin by reformulating integer numbers with the well-known \textit{binary encoding}. Then we will show how to manipulate sequences of things, in order to dispose over more complex data structures. Finally, we will build a small imperative language interpreter and its type system. Finally, we will explore the consequences of having implemented a full-blown programming language within our logical system.


\chapter{\label{ch:fsa}Finite state automata and regular languages}

The regular languages, the simplest class of languages
we defined in Chapter~\ref{ch:formal-languages},
and the corresponding abstract machines, finite state automata (FSA),
have many practical uses ranging from circuit design to string search.
They are also an important tool for computational linguistics.
In this chapter we will study FSA and regular languages.

%TODO: applications - dictionaries, shallow parsing ...

%TODO: link to regular expressions

\section{Finite state automata}

A finite state automaton is an abstract machine with a finite number of states.
The machine can be in one of the states in a given time,
and it changes its sates as it processes an input string
from a finite alphabet.
If the machine is in one of final or accepting states at the end of the input,
it accepts the string.
As noted earlier, finite state automaton
recognize and generate regular languages.
A finite state automaton accepts all and the only strings
that are in the corresponding regular language.

\begin{marginfigure}
  \begin{center}
    \tikzset{external/export next=false}
    \begin{tikzpicture}[every initial by arrow/.style={thick},
      ]
      \node[draw,state,initial,initial text={}] at (0,0) (s0) {0};
      \node[draw,above right=of s0,state] (s1) {1};
      \node[draw,below right=of s0,state,accepting] (s2) {2};
      \draw[->,thick,>=stealth]
                (s0) edge[above,bend left] node (nl) {b} (s1)
                (s0) edge[below,bend right] node{a} (s2)
                (s1) edge[loop above] node{b} (q1);
      \draw[->,thick,>=stealth]
                (s1) edge[right,bend left=90] node{c} (s2);
      \node[blue] at (-1, -1) (ilab) {initial state};
      \draw[blue,->,shorten >=1pt]  (ilab) to[bend right] (s0);
      \node[blue] at (-1, 1) (tlab) {transition};
      \draw[blue,->,shorten >=1pt]  (tlab) to[bend left] (nl.south);
%      \draw[blue,->,shorten >=1pt]  (tlab) to[bend left] ($(s0)!0.5!(s1)$);
      \node[blue] at (0, 2) (slab) {state};
      \draw[blue,->,shorten >=1pt]  (slab.east) to[bend left] (s1.north west);
      \node[blue] at (0, -2) (flab) {accepting state};
      \draw[blue,->,shorten >=1pt]  (flab) to[bend right] (s2);
    \end{tikzpicture}
  \end{center}
    \caption{\label{fig:example-fsa}%
      An example finite-state machine.
    }
\end{marginfigure}
A common way to think about (and represent) an FSA is a directed graph,
where the nodes correspond to the states,
and edges are labeled with the symbols from the alphabet.
Figure~\ref{fig:example-fsa} presents an FSA represented as
a directed graph.
The machine starts at the start state (marked with an incoming arrow)
and moves between the states based on the given input.
If the machine is in a final state
(represented as a double circle)at the end of the input,
the input string is accepted.

\subsection{Deterministic finite automata}
Formally, a finite state automaton, $M$,
is a tuple $(\Sigma,Q, q_{0}, F, \Delta)$ with

\begin{itemize}[nosep]
  \item[$\Sigma$] is the alphabet, a finite set of symbols
  \item[$Q$] a finite set of states
  \item[$q_{0}$] is the start state, $q_{0} \in Q$
  \item[$F$] is the set of final states, $F \subseteq Q$
  \item[$\Delta$] is a function that takes a state and a symbol in the alphabet,
    and returns another state ($\Delta: Q\times\Sigma \ra Q$)
\end{itemize}

The important part of the this definition is the last item,
translates to the requirement that given a state and a symbol,
there is only one possible state to go.
As a result,
a deterministic automata requires
exactly one edge labeled by each symbol in the alphabet leaving every state.


\lab{Algorithms}{Recursion and Persistent Variables}{Recursion and Persistent Variables}

\objective{We will briefly discuss an important technique in programming known as recursion. Then we will discuss the use of persistent variables in MATLAB.}

A recursive function is one that calls itself. For example we could write a factorial function as follows:

\begin{lstlisting}[style=matlab]
function out = myFactorial(n)
if n ~= 1
     out = n*factorial(n-1);
else
     out = 1;
end
\end{lstlisting}

This function calculates factorial recursively. It calls \li{factorial(n-1)} until $n = 1$, at which point each function finally terminates.

One critical aspect of this code is the fact that if n == 1, out = 1. This is known as the base case. The base case is the case where the recursion breaks down, or in other words where the function stops calling itself. Every recursive algorithm must have some sort of base case, or else it will call itself indefinitely.

In fact this function only works on positive integers. On any other input the function never terminates. Thus we should modify the function as follows to avoid problems:

\begin{lstlisting}[style=matlab]
function out = myFactorial(n)
if (n <= 0 || rem(1,n) ~= 0)
	error('input must be a positive integer');
end

if n ~= 1
     out = n*factorial(n-1);
else
     out = 1;
end
\end{lstlisting}

This practice (of writing code to handle bad user input) is known as defensive programming. It is good to get into this habit.

Here's a simple combinatoric example. Suppose our goal is to enumerate every possible way to select six numbers out of a list of ten (note that we are talking combinations not permutations). We could use some sort of complicated \li{for} loop structure like this:

\begin{lstlisting}[style=matlab]
for i = ...
    for j = ...
        for k = ...
            for l = ...
            .
            .
            .
\end{lstlisting}

However, we wish to stress emphatically, that this is not the right way to do any problem ever. Once you write more than three for loops you are probably skirting on the edges of intractability (in fact, any time you attack a problem that suggests this approach you are likely attacking a problem that is inherently intractable). However, oftentimes such a problem can at least be programmed using recursion.

For example in our combinatorics problem above we could write something like the following:

\begin{lstlisting}[style=matlab]
function out = Combinations(values,k)
%This functions outputs all possible combinations of k elements from the
%vector values.
if min(size(values)) ~= 1
    error('values must be a vector');
end
if k > length(values)
    error('k must be smaller than length(values)');
end
if (k <=0 || mod(k,1) ~= 0)
    error('k must be a positive integer');
end

%Make input vectors column vectors
if size(values,2) > size(values,1)
    values = values';
end

out = [];
n = length(values);
if k == 1
    out = values;
else

%This loop iterates through all of elements of the vector values that have at least k
%elements after them (inclusive). For each element it then calls
%Combinations(values(i+1:end),k-1), which returns combinations of size k-1
%for the elements succeeding the current element. This is so that we do not
%get repeats of combinations.
for i = 1:n-(k-1)
    % Calculate the number of possible combinations (to allow proper
    % concatenation in the recursive call.
    numCombs = factorial(n-i)/((factorial(k-1))*(factorial(n-i-(k-1))));
    %This is the recursive call.
    out = [out;[values(i)*ones(numCombs,1), Combinations(values(i+1:end),k-1)]];
end
end
\end{lstlisting}

This solution utilizes recursion to make the coding much more simple. Write this code up and understand why it does what it does, and how the recursion works.

One note of importance: just because a recursive algorithm exists does not mean that it is the correct way to solve the problem. The factorial example makes this readily evident. Generally speaking recursion slows things down, so it should be avoided when possible. However, it can provide elegant solutions to difficult problems, and is a good arrow to have in your quiver.

\begin{problem}
Use recursion to calculate the determinant of a matrix using cofactor expansions (this is known as Laplace's formula). This can be done in four lines. You can see how simple recursive programs can be. Now time your function and compare its performance to finding the determinant using the LU decomposition that you wrote earlier. How does it fare? You will probably notice that the recursive method is much slower for even moderate values of n. Why is this? Laplace's formula is $O(n!)$, so even though it's very easy to code it's terribly ineffective in general.
\end{problem}

\section*{Persistent Variables}

One of the crucial concepts in computer programming is known as scoping. A variable's scope is the function that it lives in. When we write functions the only variables that we have to worry about are the ones that are passed into it. This is because my function has its own scope. A variable that only lives in a specific function (as most variables should) is known as a local variable.

Persistent variables are a tool that MATLAB uses to ease program complexity. A local variable that is declared to be persistent will have the same value the next time the parent function is called. We will illustrate using a function that calculates the fibonacci numbers. In this example we will also practice using recursive functions.

Write the following function:

\begin{lstlisting}[style=matlab]
function out = fib1(a,b,n)
%% calculate the nth fibonacci number using the seeds a,b
if n ==1
	out = a;
else if n == 2
        out = b;
else
        out = fib1(a,b,n-1) + fib1(a,b,n-2);
end
\end{lstlisting}

This function is very easy to read, and directly applies the recursive definition of the fibonacci sequence. However, there is a problem. Consider the function calls made for $n=5, a = 1, b = 1$:

\begin{align*}
fib(5) &= fib(4) + fib(3) = (fib(3) + fib(2)) + (fib(2) + fib(1)) \\
       &= ((fib(2) + fib(1)) + fib(2)) + (fib(2) + fib(1))
\end{align*}

It is clear that many calculations are being repeated unnecessarily. One possible way to solve this problem is to use a \li{for} loop instead. Try writing the following function:

\begin{lstlisting}[style=matlab]
function out = fib2(a,b,n)
%% calculate the nth fibonacci number using the seeds a,b
if n == 1
	out = a;
else if n == 2
        out = b;
else
        x = [a b];
        for i = 1:n-2
            x = [b sum(x)];
        end
        out = x(2);
end
\end{lstlisting}

This version has the advantage that there are no repeated calculations. However, suppose that we are expecting a and b to usually remain the same. Each time \li{fib} was called all of the calculations would have to be redone. This is where persistent variable can be useful. If a variable is declared persistent its value will be stored for the next time the function is called. In this example we can store the entire fibonacci sequence as a persistent variable and only recalculate as necessary. The following code implements this approach:

\begin{lstlisting}[style=matlab]
function out = fib3(a,b,n)
%% calculate the nth fibonacci number using the seeds a,b
persistent f
if length(f) < 2 || f(1) ~= a || f(2) ~= b
    f = [a b];
end
for k = (length(f) + 1):n
    f(k) = f(k-2) + f(k-1);
end
out = f(n);
\end{lstlisting}

Test this code out. Try testing its performance against the other versions (you could try calculating the same fibonacci number several thousand times). You should see a significant difference.

We finish this section with a word of caution. Persistent variables require significant overhead, and accessing them is much slower than accessing normal variables. They should accordingly be used sparingly.

\begin{problem}
Use persistent variables to speed up the calculations of the \li{expss} function that you wrote in chapter one (hint: certain operations, such as factorial, are costly). Test your new function's performance against the old one. How does it perform?
\end{problem}

\begin{problem}
Compare the performance of the \li{fib3} versus \li{fib2}, when the seeds are random every time. You should run it several thousand times in your test. What do you notice? \li{fib3} is much slower because accessing a persistent variable is very costly. This reminds us that using persistent variables should be avoided in most situations.
\end{problem}


\lab{Algorithms}{Value Function Iteration}{Value Function Iteration}

\objective{This section teaches the fundamentals of Dynamic Programming using value function iteration.}

Often it is of interest to optimize decision making in some sequential process.  For example, an oil company may need to decide how much oil to excavate and sell each month as prices change, a person entering retirement may need to decide how much of their savings to spend each year, or a model of economic growth may require a decision about how much to invest in capital versus how much to spend each year.  In this lab we will formulate a general dynamic optimization problem.  We will explore techniques for solving such a problem with both finite and infinite time horizons.

\section*{The Sequential Problem, Finite Horizon}
\label{SecRecProbFinHor}
Suppose there are time periods $t=0,1,\ldots, T$ and at each time period we take an action $c_t$. Furthermore, at the beginning of each time period $t$ we are in some state $W_t$.  In many cases $W_t$ might represent an available resource, such as money.  At each time we receive some reward, $u(W_t,c_t)$, for taking action $c_t$ given state $W_t$.  We assume that rewards are worth more now than later. We let $\beta\in (0,1)$ represent what is called the discount factor and gives the ratio of preference for rewards today versus rewards tomorrow.  For example, receiving a dollar today is preferable to receiving a dollar in a year because taking a dollar today and putting it in a  savings account results in having more than a dollar in a year.  Lastly, over time our state variable $W_t$ changes according to some rule depending on the previous state and our actions,
\begin{equation}
\label{motion}
W_{t+1} = g(W_t,c_t).
\end{equation}
Equation \eqref{motion} is sometimes referred to as the law of motion, as it describes how we move from state to state.
Mathematically such a problem can be represented as follows:
\begin{equation*}
\max \sum_{t=0}^T \beta^t u(W_t,c_t) \quad \text{s.t.} \quad W_{t+1} = g(W_t,c_t)
\end{equation*}
where our initial state, $W_0$ is given.  There may also be restrictions on our choices $c_t$.  For example, in many applications the state $W_t$ represents the amount of some resource available, and $c_t$ represents the amount we use up in time period $t$.  In this case we would require $c_t \in [0,W_t]$.

For simplicity, lets assume that $u$ is a function of $c_t$ only (this is often, though not always, the case in practice).  First let's consider the case that $T=0$.  So we maximize $u(c_0)$ over $c_0 \in [0,W_0]$.  In most cases $u$ is increasing, which we will assume here.  In this case it will be optimal to choose the largest value of $c_0$ possible, that is, $c_0 = W_0$.  Thinking of $W_0$ as our available resources, this simply means, that if we don't have future periods to consider, we will use all of it.

In fact this is always true in the last period.  In the problem with $T$-periods we know that we will use all of our resources remaining in period $T$.  Consider the two period problem:
\begin{equation}
\label{2perprob}
\max \, u(c_0) + \beta u(c_1)
\end{equation}
where $c_0 \in [0,W_0]$, $c_1 \in [0,W_1]$ and $W_1 = g(W_0,c_0)$.  We know that in the last period we will use all of our remaining resources so that $c_1 = W_1$.  Substituting gives
\begin{equation*}
\max \, u(c_0) + \beta u(g(W_0,c_0)).
\end{equation*}
Then we need only determine $c_0$.  Taking the derivative of \eqref{2perprob} with respect to $c_0$ and setting equal to zero gives the first order condition
\begin{equation}
\label{FOC}
u'(c_0) = -\beta u'(g(W_0,c_0))g_c(W_0,c_0)
\end{equation}
where $g_c$ is the partial derivative of $g$ with respect to $c$.

Given a specific form for $u$ we could solve for $W_1$ and obtain the optimal solution.  In fact, we can solve a problem of any length $T$ in this manner by starting at the last time period and working backward.  We know that $W_{T+1} = 0$.  Working backward in time we obtain an equation at each time step $t<T$ by taking the derivative with respect to $c_t$ and setting equal to zero.  This process is called backward induction.  The equations at each time step, such as equation \eqref{FOC} are sometimes called the inter-temporal Euler equations.  These equations, along with $c_T = W_T$ make $T+1$ equations to go with our $T+1$ unknowns $c_0,c_1,c_2,\ldots,c_T$ where we can use the law of motion to relate the $c_t$ and $W_t$.

\section*{The Recursive Problem, Finite Horizon}
\label{SecRecProbFinHor}
Approaching the problem sequentially like this can be somewhat messy.  The dynamic programming approach we consider now is more easily adaptable to many situations.  The key to the dynamic programming approach is to define our optimization problem in terms of subproblems.  Notice that if we are in time period $t$, we face a problem of exactly the same form as the problem at time $0$.  We are in some state $W_t$, and want to maximize the sum from $t$ to $T$.  With this idea in mind, we define a function $V_t(W_t)$ called the value function.  The function $V_t$ gives the value of entering time $t$ in state $W_t$ and making optimal decisions moving forward.  So
\begin{equation*}
V_{t-1}(W_{t-1}) = \max u(c_{t-1}) + \beta V_t(g(W_{t-1},c_{t-1})).
\end{equation*}
This is called the Bellman Equation.  The key to this formulation is that we decide what to do in period $t-1$ with the assumption that our actions in the remaining periods will be optimal.  This is called the principal of optimality.

Let us consider a specific example from economics called The Cake Eating Problem.  Suppose $W_t$ represents the size of a cake.  At each time period we can choose how much to consume.  What we eat,$c_t$, gives us rewards.  What we save,$W_t-c_t$, does not give rewards (until it is eaten in a later period).  The law of motion \eqref{motion} becomes
\begin{equation}\label{LOM_EX}
W_{t+1} = g(W_t,c_t) = W_t-c_t
\end{equation}
Now we have completely defined the problem.  The Bellman Equation is
\begin{equation*}
V_{t-1}(W_{t-1}) = \max u(c_{t-1}) + \beta V_t(W_{t-1}-c_{t-1}).
\end{equation*}
As before, we know that in the last time period, we should not save anything.  So $V_{T+1}(W_{T+1}) = 0$.  There is no value in leaving wealth for period $T+1$.  So $c_T = W_T$.  Then $V_T(W_T) = u(c_T)$.  Consider the value function equation for period $T-1$.
\begin{align*}
V_{T-1}(W_{T-1}) &= \max u(c_{T-1}) + \beta V_T(W_{T-1}-c_{T-1}) \\
                 &= \max u(c_{T-1}) + \beta (W_{T-1}-c_{T-1})
\end{align*}

Thus given $W_{T-1}$, we can determine $V_{T-1}$ by maximizing over $c_{T-1}$.  Notice that by the law of motion, it is enough to determine the values of the $W_t$ since these determine the $c_t$.  In fact rearranging \eqref{LOM_EX} we have
\begin{equation*}
c_t = W_t - W_{t+1}.
\end{equation*}
Thus we can rewrite the value function as
\begin{equation}
\label{cake_valfn}
V_{t-1}(W_{t-1}) = \max_{W_t} u(W_{t-1} - W_{t}) + \beta V_t(W_t).
\end{equation}

The solution to this problem is often called a policy function.  A policy function determines an action based on the current state.  Denoting the policy function by $\psi$, this can be written
\begin{equation*}
W_{t+1}=\psi_t \left(W_t\right)
\end{equation*}
The policy function gives the optimal amount of cake to leave for next period (equivalent to the amount of consumption) given the amount of cake at the start of the period.  In other words, it determines the choice of $W_t$ that satisfies the $\max$ condition in equation \eqref{cake_valfn}.

\begin{problem}
\label{prob:cake_prob}
Follow the steps below to solve the problem described above.  Take $u(c_t) = \sqrt{c_t}$, $\beta = 0.9$, and $T=10$.
\begin{enumerate}
\item Let the maximum size of the cake be $W_{max} = 1$. Approximate the continuum of possible cake sizes by a column vector called $W$ that ranges from to 0 to 1.  Let the number of possible cake values be $N=100$.

\item Note that in order to compute the value function we need $u(W_{t-1}-W_t)$.  Create an $N$ by $N$ matrix that contains all possible values of $W_{t-1} - W_t$ (where $W_{t-1}$ corresponds to rows and $W_{t}$ to columns).  Make sure that $c_t \geq 0$ is satisfied by replacing negative entries in the matrix with zero.  Then take the square root to get a matrix of $u(W_{t-1}-W_t)$.  To make sure we do not choose $W_{t-1} - W_t < 0$ when maximizing, replace the corresponding entries of the $u(W_{t-1}-W_t)$ matrix with a large negative number (e.g. $-10^{10}$).

\item Next create an $N$ by $T+2$ (corresponding to $t=0,1,\ldots, T+1$) matrix representing the value function for a given time $t$ and state $W_t$.  We can initialize it to zeros and begin filling in the columns starting with the last (which we know is zeros).

\item Now we are ready to iterate backward and find the value function for each time period.  To find $V_T$, we compute $u(W_T - W_{T+1}) + \beta V_{T+1}(W_{T+1})$ for all values of $W_{T-1}$ and $W_T$.  This will result in an $N$ by $N$ matrix where the rows correspond to values of $W_{T}$ and the columns correspond to values of $W_{T+1}$.  Note that to compute this we need a matrix representing $\beta V_{T+1}(W_{T+1})$.  Because this quantity does not depend on $W_T$, its rows should be equal.  To do this, we want to take $\beta V_{T+1}(W_{T+1})$ as a row vector and stack this vector to create a matrix with equal rows.  There are multiple ways to do this.  One is the \li{np.repeat} function.  For example, if \li{b} is a row vector it could be used like the following.
\begin{lstlisting}
b = [[1, 2, 3]]
np.repeat(b, 3, axis = 0)
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])
\end{lstlisting}
In general, be careful about having the correct rows, columns, transposes, etc throughout your code.

Now we maximize over choices of $W_T$ (choosing how much to save for next period).  Then we will have a row vector representing the value function for period $T$ across all possible $W_T$.  Iterate this procedure to fill in the value function for all $t=0,1,\ldots, T+1$.

\item In each iteration you maximize to find the value function at time $t$.  Save the values of $W_{t+1}$ that achieve the maximum.  The result is an $N$ by $T+1$ matrix whose $n,t$ entry gives the optimal amount of cake to leave for for period $t+1$ given that we start period $t$ with the the $n$th value of our $W$ vector of cake.  This is the policy function.

\item Plot the surface of the Value and Policy functions.  This can be done by including the following import lines
\begin{lstlisting}
from matplotlib import pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
\end{lstlisting}
and using the following code
\begin{lstlisting}
x = np.arange(0, N)
y = np.arange(0, T+2)
X, Y = np.meshgrid(x, y)
fig1 = plt.figure()
ax1 = Axes3D(fig1)
ax1.plot_surface(W[X], Y, np.transpose(V), cmap=cm.coolwarm)
plt.show()

fig2 = plt.figure()
ax2 = Axes3D(fig2)
y = np.arange(0,T+1)
X, Y = np.meshgrid(x, y)
ax2.plot_surface(W[X], Y, np.transpose(psi), cmap = cm.coolwarm)
plt.show()
\end{lstlisting}
where \li{W} is the vector of wealth levels, \li{V} is the value function, and \li{psi} is the policy function.

You should also try plotting the value and policy functions for fixed time periods across $W_t$, or for fixed $W_t$ across time and make sure that these plots fit your intuition.
\end{enumerate}
\end{problem}

\section*{The Recursive Problem, Infinite Horizon}
\label{SecRecProbInFHor}
Next we consider an infinite horizon problem.  For simplicity, we continue with the example from the previous section.  Suppose that rather than optimizing over $t = 0,1,\ldots,T$, we wish to optimize over an infinite time horizon:
\begin{equation*}
\max \sum_{t=0}^\infty \beta^t u(W_t,c_t) \quad \text{s.t.} \quad W_{t+1} = g(W_t,c_t).
\end{equation*}
Since at any time $t$, there are an infinite number of periods remaining, one might suspect that the optimal policy will not depend on the current time $t$.

\begin{problem}
\label{prob:cake_prob2}
Compute the solution to Problem \ref{prob:cake_prob} with $T = 1000$.  Plot the policy function across time for fixed $W_t = 1$.  Notice that it is the same for all time periods, except those near the end time $T$.
\end{problem}

As suggested by the results of Problem \ref{prob:cake_prob2},  the policy function for the infinite horizon problem does not depend on the time $t$ (this can be proved).  That is, at any time $t$, the optimal decision depends only on the amount of cake at the beginning of the period, not the value of $t$.  So everything can now be written in terms of variables today and variables tomorrow. We will denote variables tomorrow with a ``$\:'\:$".
\begin{equation}
\label{EqBellman}
V\left(W\right) = \max_{W'\in[0,W]}\:\: u\left(W - W'\right) + \beta V\left(W'\right)
\end{equation}
Note that the value function $V$ on the left-hand-side of \eqref{EqBellman} and on the right-hand-side are the same function.

Because the problem now has an infinite horizon, the nature of the solution is a little different. The solution to \eqref{EqBellman} is a policy function $W'=\psi(W)$ that creates a fixed point in $V$. In other words, the solution is a policy function $\psi(W)$ that makes the function $V$ on the left-hand-side of \eqref{EqBellman} equal the function $V$ on the right-hand-side.

Define $C$ as an operator on any value function $V_k\left(W\right)$. Let $C$ perform the following operation.
\begin{equation}
\label{EqContraction}
C\Bigl(V_k\left(W\right)\Bigr) \equiv \max_{W'\in[0,W]}\: u\left(W-W'\right) + \beta V_k\left(W'\right)
\end{equation}
Note that the value function on the right-hand-side of \eqref{EqContraction} and on the left-hand-side are the same function $V_k$, but have a different value of the size of the cake--$W$ versus $W'$. The operator $C$ takes in a function $V_k$, and gives a new function which we will call $V_{k+1}$:
\begin{equation*}
V_{k+1}\left(W\right) \equiv C\Bigl(V_k\left(W\right)\Bigr).
\end{equation*}
The value function $V_{k+1}$ that results from the operation $C$ is not necessarily the same as the value function that the system began with $V_k$. However, according to equation \eqref{EqBellman} we seek a $V$ such that $C(V) = V$.  The solution, then, is the fixed point in $V$.
\begin{equation*}
C\Bigl(V_k\left(W\right)\Bigr) = V_{k+1}\left(W\right) = V_k\left(W\right) = V\left(W\right)
\end{equation*}

The operator $C(\cdot)$ is called a contraction mapping if applying it over and over again to an arbitrary value function $V_k$ converges to a fixed point. One way to characterize a contraction mapping is:
\begin{equation*}
\lim_{k\rightarrow\infty}\: C^k\Bigl(V_0\left(W\right)\Bigr) = V_k(W) =  V\left(W\right)
\end{equation*}
for any $V_0$.  It can be shown that if $u(\cdot)$ is real-valued, continuous, and bounded, $\beta\in(0,1)$, and that the constraint set $W'\in[0,W]$ is nonempty, compact-valued, and continuous, then the operator $C$ is a contraction and thus we can obtain a solution by iteration.

Remember, in the infinite horizon problem both the value and policy functions do not depend on time.  Computationally, this means that the value and policy functions in the infinite horizon problem are one dimensional.

\begin{problem}
Solve the Cake Eating Problem with an infinite time horizon by following the steps below.  As in Problem \ref{prob:cake_prob}, take $u(c_t) = \sqrt{c_t}$, $\beta = 0.9$.
\begin{enumerate}
\item As in Problem \ref{prob:cake_prob}, let the maximum size of the cake be $W_{max} = 1$. Approximate the continuum of possible cake sizes by a column vector called $W$ that ranges 0 to 1.  Let the number of possible cake values be $N=100$.

\item \label{item:step2} Initialize the value function, V to a vector of zeros of length $N$.  This is $V_0$  Perform one iteration of the contraction operation given in equation \eqref{EqContraction} to get a new value function $V_1$ (this should be very similar to Problem 1).  Determine the resulting policy function $W' = \psi_1\left(W\right)$.  [HINT: The policy function should be a vector of length $N$ of optimal future values of the cake $W'$ given the current value of the cake $W$, and $V_T$ should be an $N$-length vector representing the value of entering a period with cake size $W$.]

\item \label{item:step3} Generate a norm $\delta_0 = \norm{V_0\left(W\right) - V_1\left(W'\right)}$ that measures the distance between the two value functions. Define the distance metric as the sum of the squared differences,
\begin{equation}
\label{EqDist}
\delta_1\equiv \norm{V_1\left(W\right) - V_0\left(W'\right)} = \left(V_1 - V_0\right)'*\left(V_1 - V_0\right)
\end{equation}
where $\left(V_1-V_0\right)'$ is the transpose of the difference of the two vectors. Defined in this way, $\delta_1\in [0,\infty)$.

\item \label{item:step4} Take the resulting $V_1$ from \ref{item:step2}, and perform the same contraction on it to generate $V_2$ and $\psi_2$. That is, generate,
\begin{equation*}
  V_2\left(W\right) = C\Bigl(V_1\left(W\right)\Bigr) = \max_{W'\in[0,W]}\: u\left(W - W'\right) + \beta V_1\left(W'\right)
\end{equation*}
and the accompanying policy function $W'=\psi_2\left(W\right)$. Calculate the accompanying distance measure for $\delta_2$ using the formula from \eqref{EqDist} with the updated period subscripts. Compare $\delta_2$ with $\delta_1$ from \ref{item:step3}.

\item \label{item:step5} Repeat \ref{item:step4} and generate $V_3$ and $\psi_2$ by performing the contraction on $V_2$. Compare $\delta_3$ to $\delta_2$ and $\delta_1$.

\item Write a loop that performs the contraction operation from \ref{item:step2}, \ref{item:step4}, and \ref{item:step5} iteratively until the distance measure is very small $\delta_k < 10^{-9}$.  The distance measure $\delta_k$ being arbitrarily close to zero means you have converged to the fixed point $V_k = V_{k+1} = V$. (For fun, you can show that the policy function converges to the same function regardless of what you put in for your initial policy function value.)

\item Plot the policy function for the converged problem $W' = \psi\left(W\right)$ which gives the value of the cake tomorrow ($y$-axis) as a function of the cake today ($x$-axis).
\end{enumerate}
\end{problem}


%!TEX root = ../../dissertation.tex

\subsection{L-Systems} % (fold)
\label{sub:l_systems}

Lindenmayer Systems (L-Systems) are a class of string rewriting mechanisms, originally developed by Lindenmayer as a mathematical theory for plant
development. It is capable of describe the behavior of plant cells and model the growth processes of plant development.

An L-Systems consists of two different parts, one axiom and a set of production rules. The axiom is the starting point of the system, acting as a seed. Then it is applied in this seed the set of production rules, that change the initial string, producing other strings.
This is an iterative process, so after the production of a larger set of strings, the rules can be applied to each one of them which grows the size of
the set even more.

L-Systems are used to model the natural growth of vegetation (Figure~\ref{fig:trees}), and the generation of Fractals.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.65\textwidth]{images/Theory/L_Systems/Dragon_trees.jpg}
    \caption{Trees with L-Systems}
    \label{fig:trees}
\end{figure}

In this process, each symbol is associated with a production rule. For instance having $\{F, +, -\}$ for the alphabet and \emph{production} $\{F \rightarrow
 F+F--F+F\}$. From a starting axiom \emph{aba}, and the application of the rules we have:\\
\begin{equation} \label{eq:seed}
F\\
\end{equation}
\begin{equation} \label{eq:step1}
F+F--F+F\\
\end{equation}
\begin{equation} \label{eq:step2}
F+F--F+F \; + \; F+F--F+F \;- \;- \;F+F--F+F \;+ \;F+F--F+F\\
\end{equation}

%\begin{align}
%\begin{split}
%F\\
%F+F--F+F\\
%F+F--F+F \; + \; F+F--F+F \;- \;- \;F+F--F+F \;+ \;F+F--F+F\\
%\end{split}
%\end{align}
%\\

This is an example of the evolution of one system where the production is applied  in (\ref{eq:seed}) that turns into $F+F--F+F$. Note that the space
between the symbols are just for readability.

All the symbols are assigned with a geometric meaning. The notion of a turtle with a pen, as proposed in \cite{abelson1982aa}, with the symbols being
interpreted as moving instructions to the turtle, is a simple way to understand, where ``F'' means forward and the symbols ``+'' and ``-'' are interpreted as rotations counter-clockwise and clockwise respectively by a predefined angle. By applying this method to the last example and setting the angle for the rotation to $60^{\circ}$ the result is Figure~\ref{fig:kockLS}.

\begin{figure}[htbp]
   \centering
   \includegraphics[width=0.55\textwidth]{images/Theory/L_Systems/koch.png}
   \caption{Result of the ``turtle walk'' with the given example}
   \label{fig:kockLS}
\end{figure}

%\begin{wrapfigure}{r}{0.5\textwidth}
%	\vspace{-15pt}
%    \centering
%    \includegraphics[width=0.55\textwidth]{images/Theory/L_Systems/koch.png}
%    \caption{}
%    \label{fig:kockLS}
%	\vspace{-25pt}
%\end{wrapfigure}

%$\bigodot \; \bigodot$

% This concept of the turtle can be considered also in 3D.

% subsection l_systems (end)


\newsection{Searching}

\subsection{Motivation}
Now that we know how to make arrays, we want to use them efficiently. This chapter and the next will be a more ``high-level" approach to problems in programming and Computer Science. Searching is a well-studied and important problem. We will cover 2 basic searching algorithms that will find elements in an array quickly and efficiently: linear and binary searching.

\subsection{Linear Search}
Linear Search is searching from one end of the array until the end, and stopping if we find the element that we desire. Pseudocode to match a linear search is:
\begin{verbatim}
function LinearSearch(array a, variable v):
     for(item i in a):
          if i == v:
               return i; // or any non-"error" value (in array)
     return -1; // some "error" value (not in array)
\end{verbatim}
For example, if we have an array of \verb|int| variables and we are searching for one, we can construct code such as this:
\begin{lstlisting}
public boolean linearSearch(int[] array, int search) {
     for (int i : array) {
          if (i == search) {
               return true;
          }
     }
     return false;
}
\end{lstlisting}
However, this loop can be inefficient, because we have no guarantee what the ordering or contents of \verb|array| are. We have to search (on average) $\frac{n}{2}$ elements. Therefore, we cannot make a ``guess" as to improvements can be done.

\par However, if we can have a guarantee on the ordering of elements in the array, we can get better performance, as shown by binary search.

\subsection{Binary Search}
We can improve our algorithm by using binary search, on one condition: we \textbf{must} guarantee that the input array \textbf{is already sorted}. We will cover how to sort an unsorted array in the next chapter. Pseudocode of the algorithm is the following:
\begin{verbatim}
function BinarySearch(array a, int key):
     low = 0, high = a.length;
     while high >= low:
          mid = average(low, high);
          if a[mid] == key:
               return mid; // found
          else if a[mid] < key:
               low = mid + 1;
          else
               high = mid - 1;
     return -1; // not found
\end{verbatim}
The algorithm works like this: since the array is sorted, we check the middle value. If the value to search is larger than that middle value, we ``cut off" the lower half of the array, since we can guarantee it is not in that part (and likewise for less than). Then we check the middle half of the top half of the array, and keep adjusting what is the ``low" and ``high" ranges of what part of the array we are searching. We stop when we keep adjusting ``low" to the point that it is higher than ``high" (it does not make sense to search in a range that is backwards). At that point, we have not found our element, and we return.

\par For example, we can create a similar procedure in Java with a \verb|int| array (the concept works with Strings also, just comparing their lexicographic ordering instead):
\begin{lstlisting}
public int binarySearch(int[] array, int k) {
     int min = 0, max = array.length-1;
     while (max >= min) {
          int mid = (min+max)/2;
          if (a[mid] == k) {
               return mid;
          } else if (a[mid] < k) {
               min = mid+1;
          } else {
               max = mid-1;
          }
     }
}
\end{lstlisting}
Now for an example. Suppose I have an array consisting of $\{-3, -1, 2, 4, 7\}$. Clearly, the array is sorted, so we can use binary search. I want to search for the element 4. Here is the execution of binary search:
\begin{verbatim}
// Start binary search
min = 0, max = 5
// 7 >= -3, so enter while loop
mid = (0+5)/2 = 2 (integer division)
a[mid] = 2
// check #1: 2 != 4, so skip if statement
// check #2: 2 < 4, so enter else if
min = 2+1 = 3

// 2nd Execution:
mid = (3+5)/2 = 4
a[mid] = 7
// check #1: 7 != 4, so skip if statement
// check #2: 7 > 4, so skip else if statement
max = 4-1 = 3

// 3rd Execution:
mid = (3+3)/2 = 3
a[mid] = 4
// check #1: 4 == 4, so return 4, success!
\end{verbatim}
\subsection{Written Exercises}

\begin{enumerate}

\item Use the sorted list below and use binary search to look for Mike in the list. Show all the names that will be compared before Mike is found. Then, repeat the same process for Cathy (note: Cathy is not in the list).
\begin{table}[h]
\begin{tabular}{lllllll}
Aaron & Betsy & Doug & Elise & Mike & Pat & Steven
\end{tabular}
\end{table}

\item What is the benefit of using binary search over linear search?

\item (Warning: this question is difficult) The line:
\begin{lstlisting}
int mid = (min+max)/2;
\end{lstlisting}
in the code for binary search above has a bug, but is very specific. It works in the vast majority of cases, though. Figure out what the bug is, and what can be done to fix it (hint: think about the bounds on the values for \verb|int|).

\end{enumerate}


\lab{Naive Bayes}{Naive Bayes}
\label{Ch:NaiveBayes}

\objective{Implement Naive Bayes Classification Models}

%Outline
%
%Brief overview
%	With/Without independence assumptions
%
%Types of Naive Bayes models
%	Multinomial
%	Gaussian
%
%Simple Examples
%	Multinomial - Fair vs Loaded Dice
%		Dataset - generated by me
%
%	Gaussian - Wheat Kernels
%		UCI Machine Learning
%		Sample test from training
%
%Application: spam Filtering
%	Dataset - Emails
%	Build wordcounts and labels
%	Sample test from training

\subsection*{Introduction}

Naive Bayes classification methods are a good introduction to machine learning techniques.
They are relatively straightforward to understand and implement, yet they are very effective for certain applications.
They are limited by their strong independence assumptions.

Recall that the classification problem tries to assign the correct label to a set of features
(called a \emph{feature vector}).
For example, suppose we wish to label two pieces of fruit with their correct names.
The first fruit is red and round, and the second fruit is long and yellow.
These attributes of the fruit are the features.
If we assign the names apple and banana respectively, then these are our labels.

It is common in classification problems to start with a set of labeled feature vectors called a training set.
We use the training set to train our algorithm to make predictions.
It is also common to have another smaller set of labeled feature vectors called a test set.
To verify the effectiveness of our algorithm, we predict the labels of the test set, and then compare the predicted labels to the true labels.

Now recall Bayes rule for random variables:
\[
P(Y|X) = \frac{P(X|Y)P(Y)}{\int P(X|Y)P(Y) dy}
\]
Where $P(Y)$ is our prior distribution and $P(X|Y)$ is our likelihood function.

Suppose that we have a set of features that we wish to label.
Let $x = (x_1, \ldots, x_n)$ be a feature vector and let $C = {c_1, \ldots, c_k}$ be our set of labels.
We may apply Bayes rule to this problem as follows:
\[
P(c_i|x) = P(c_i | x_1, \ldots, x_n)  = \frac{P(x_1, \ldots, x_n|c_i)P(c_i)}{P(x_1, \ldots, x_n)}
\]
If we make no further assumptions, this problem is intractable.
To effectively estimate even a case where each feature is a boolean value would require us to estimate around $k2^{n}$ parameters.
The problem gets exponentially worse if we were to consider non-boolean features.

However, if we make the assumption that features are conditionally independent from each other, the problem simplifies dramatically.
If we make this assumption and apply Bayes rule, we have
\[
P(c_i | x_1, \ldots, x_n) = \frac{P(x_1|c_i)P(x_2|c_i)\ldots P(x_n|c_i)P(c_i)}{P(x_1,\ldots,x_n)}
\]
In this case, we only need to estimate $kn$ parameters.
The naive Bayes classification algorithm chooses the label with the highest probability.
Since this is independent of the denominator in Bayes rule, we can simplify the problem further.
Given an unlabeled feature vector $x=(x_1,\ldots,x_n)$, we assign the label
\[
c = \underset{i \in \{1, \ldots, k\}}{\argmax}\, P(c_i)\prod_{j=1}^n P(x_j|c_i)
\]

To assign a label to a set of features, we calculate each $P(x_j|c_i)$ and choose a prior $P(c_i)$.
We then choose the argmax as above.
The calculation of conditional probabilities and choice of prior will depend on the type of problem that we are solving.

\subsection*{Gaussian Classifiers}

Gaussian classifiers are commonly used when dealing with continuous data.
We assume that each feature is normally distributed and conditionally independent.
We may then calculate $\mu_{j,i}$ and $\sigma^2_{j,i}$ corresponding to each feature of each class.
Then $P(x_j|c_i)$ can be calculated using the gaussian pdf
\[
P(x_j|c_i) = \frac{1}{\sqrt{2\pi\sigma_{j,i}^2}}\exp{-\frac{(x_j - \mu_{j,i})^2}{2\sigma^2_{j,i}}}
\]

For example, suppose we have a training set with labels indicating the sex of a person (1 for female, 2 for male), and  features consisting of hair length and height (features 1 and 2, respectively).
Further suppose that the mean height of the men in the training set is 1.75 meters with standard deviation 7 cm, and mean hair length is 5 centimeters with standard deviation 2.5 cm.
Similarly, suppose that the mean height of the women in the training set is 1.25 meters with standard deviation 6 cm, and the mean hair length is 15 centimeters with standard deviation 1.5 cm.
Under this setup, we have
\begin{align*}
\mu_{1,1} &= 15, \quad &&\sigma_{1,1} = 1.5\\
\mu_{2,1} &= 1.25, &&\sigma_{2,1} = .06\\
\mu_{1,2} &= 5, &&\sigma_{1,2} = 2.5\\
\mu_{2,2} &= 1.75, &&\sigma_{2,2} = .07.
\end{align*}
If we wish to classify a person that is 1.4 meters tall and a hair length of 17 centimeters, we calculate the probability of each label using the parameters given above and a uniform prior ($P(F) = P(M) = \frac{1}{2}$):
\begin{align*}
P(F\,|\,17,1.4) &= P(F)\left(\frac{1}{\sqrt{2\pi\sigma_{1,1}^2}}\exp{-\frac{(17-\mu_{1,1})^2}{2\sigma^2_{1,1}}}\right)
\left(\frac{1}{\sqrt{2\pi\sigma_{2,1}^2}}\exp{-\frac{(1.4-\mu_{2,1})^2}{2\sigma^2_{2,1}}}\right)\\
&= .016\\
P(M\,|\,17,1.4) &= P(M)\left(\frac{1}{\sqrt{2\pi\sigma_{1,2}^2}}\exp{-\frac{(17-\mu_{1,2})^2}{2\sigma^2_{1,2}}}\right)
\left(\frac{1}{\sqrt{2\pi\sigma_{2,2}^2}}\exp{-\frac{(1.4-\mu_{2,2})^2}{2\sigma^2_{2,2}}}\right)\\
&= 1.7\times 10^{-11}
\end{align*}
The Female label has a greater probability given the feature vector, and so we classify the person as Female.

A nice way to visualize how a classifier works is to plot the decision boundaries for two-dimensional subspaces of the
feature vector space.
For example, the decision boundaries for a Gaussian Naive Bayes classifier trained on a dataset consisting of the
sepal widths and lengths of three different types of flowers is shown in Figure \ref{fig:decision_boundary}.

\begin{figure}
	\centering
	\includegraphics[width=.90\textwidth]{decision_boundary}
	\caption{Decision boundaries for a Gaussian Naive Bayes classifier on the iris flower dataset, together with the means
for each flower. Each point in the plane represents a 2-dimensional feature vector, and the color associated with each point indicates which class label was assigned to that feature vector.}
	\label{fig:decision_boundary}
\end{figure}

\subsection*{Working in Log Space}
In the example presented above, notice that the value of $P(M\,|\,17,1.4)$ is very small.
This is often the case in classification problems; certain classes may be very unlikely, and so calculating these probabilities may lead to numerical underflow.
This is especially pronounced in the naive Bayes model, which involves taking the product of several numbers between
0 and 1.
A useful technique to avoid underflow is to convert the computations in logarithmic space.
When we do so, the Naive Bayes label assignment is
\[
c = \underset{i \in \{1, \ldots, k\}}{\argmax}\, \log P(c_i) + \sum_{j=1}^n \log P(x_j|c_i).
\]
Since the logarithm is a monotone increasing function, the argmax is the same whether in log space or
in the original formulation.

\begin{problem}
Download the {\tt seeds\_dataset.txt} file.
This file contains 7 features describing 3 species of wheat.
\begin{enumerate}
\item Area
\item Perimeter
\item Compactness
\item Length
\item Width
\item Asymmetry Coefficient
\item Groove length
\end{enumerate}

The species of wheat are
\begin{enumerate}
\item Kama
\item Rosa
\item Canadian
\end{enumerate}

The measurements of the kernels are real valued, making this a good example to try our Gaussian classifier.

\begin{enumerate}
\item Randomly select a test set of 40 vectors from the data.  Make sure you separate this data from your training data.
\item Calculate the mean and variance for each feature of each label using the training set.
\item Using a uniform prior, predict the labels of your test set.
\item Compare your predictions to the labels of the test set. In particular, report the accuracy of the prediction,
which is the number of correctly predicted instances divided by the total number of instances.
\end{enumerate}

\end{problem}

We may also use SciPy's {\tt sklearn} library to implement a Gaussian classifier.
After importing the library, we can create a new classifier and train it with just a couple of lines of code.
To create a Gaussian classifier, use the following code.

\begin{lstlisting}
from sklearn.naive_bayes import GaussianNB
nb_classifier = GaussianNB()
\end{lstlisting}

Given a training set, we can also quickly train the classifier to a certain problem.
This requires the training set and labels as two arguments.

\begin{lstlisting}
nb_classifier.fit(training_set, labels)
\end{lstlisting}

Once the classifier has been trained, we can predict the labels for a test set.

\begin{lstlisting}
pred_labels = nb_classifier.predict(test_set)
\end{lstlisting}

The {\tt predict} method returns an array of labels for the test set.

\begin{problem}

Repeat the previous problem using {\tt sklearn}'s naive Bayes classifier.
Check that your implementation predicts the same labels as the {\tt sklearn}
implementation.

\end{problem}

\subsection*{Document Classification and spam Filters}

Naive Bayes classifiers are often used in document classification, a major example being spam detection.
When it comes to document classification, a common choice for the feature vector
is simply a count for the number of times each word in the specified vocabulary occurs in the document.
For example, suppose we are trying to classify a document, and the vocabulary of relevant words is the ordered set
\[
\text{(bank, tree, wealth, money, river, water)}.
\]
Suppose that the document is the sentence
\begin{quotation}
``The woman deposited her money in the bank, and then made her way down to the bank of the river, contemplating her wealth."
\end{quotation}
Then the feature vector for this document is
\[
(2, 0, 1, 1, 1, 0).
\]
Notice in particular that the $i$-th entry of the feature vector indicates the number of occurrences of the $i$-th
vocabulary word in the document.
Such a feature vector is often called a \emph{word-count vector}.
Notice that the count vector ignores words in the document that are not part of the vocabulary, and
it also disregards the order of words.
This simple representation of text documents is known as the \emph{bag-of-words model} or the \emph{vector space model}.
The hypothesis that drives spam filters is that spam messages will use words with different frequencies.
For example, a spam message will often have a sales pitch, so the word ``buy'' and ``cheap'' will appear often.
On the other hand, legitimate messages will probably use different language.
Thus, it is reasonable to use word-count vectors as our feature vectors when attempting to distinguish between spam and
legitimate email.

We now introduce formalisms to derive the Naive Bayes model for document classification.
Let $V = (v_1,v_2,\ldots,v_n)$ be an ordered list of words, called the vocabulary, and let
$x = (x_1,x_2,\ldots,x_n)$ be a word-count vector.
Let $\{c_1,c_2,\ldots,c_k\}$ be the set of classification labels.
In the case of continuous data and Gaussian Classifiers, each class label was associated with corresponding
mean and variance parameters, and these determined the likelihood of the feature vector given the class label.
In the case of document classification, each class label $c_i$ has a corresponding probability vector
$(p_{i,1}, p_{i,2}, \ldots, p_{i,n})$ whose entries are nonnegative and sum to 1.
This probability vector defines a categorical probability distribution over the vocabulary $V$, where $p_{i,j}$
represents the probability of seeing word $v_j$ given class label $c_i$.
With this notation in place, our Naive Bayes model takes the form
\[
c = \underset{i \in \{1, \ldots, k\}}{\argmax}\, P(c_i)\prod_{j=1}^n p_{i,j}^{x_j}.
\]

Given a training set of labeled documents, we can calculate the prior probabilities $P(c_i)$ and word probabilities
$p_{i,j}$ as follows.
Each prior probability $P(c_i)$ is simply the proportion of training documents that have the label $c_i$.
Next, let $count(c_i,v_j)$ denote the number of occurrences of word $v_j$ among all training documents that have label $c_i$.
Then we have
\[
p_{i,j}  = \frac{count(c_i,v_j)+1}{\sum_{j=1}^n(count(c_i,v_j)+1)}.
\]
(Note that adding 1 to the number of occurrences of each word is known as \emph{add-one smoothing}, and is a
common technique to prevent over-fitting.)

Once we have calculated these parameters (the ``fitting" stage), we are ready to classify new documents
(the ``prediction" stage) using the argmax equation given above.
Remember to perform calculations in log-space to prevent numerical underflow.

\begin{problem}
Implement a Naive Bayes model for document classification.
We provide an interface below.

\begin{lstlisting}
class naiveBayes(object):
    """
    This class performs naive bayes classification for word-count document features.
    """
    def __init__(self):
        """
        Initialize a naive Bayes classifier.
        """
        pass

    def fit(self,X,Y):
        """
        Fit the parameters according to the labeled training data (X,Y).

        Parameters
        ----------
        X : ndarray of shape (n_samples, n_features)
            Each row is the word-count vector for one of the documents
        Y : ndarray of shape (n_samples,)
            Gives the class label for each instance of training data. Assume class labels
            are in {0,1,...,k-1} where k is the number of classes.
        """
        # get prior class probabilities P(c_i)
        # (you may wish to store these as a length k vector as a class attribute)

        # get (smoothed) word-class probabilities
        # (you may wish to store these in a (k, n_features) matrix as a class attribute)

        pass

    def predict(self, X):
        """
        Predict the class labels of a set of test data.

        Parameters
        ----------
        X : ndarray of shape (n_samples, n_features)

        Returns
        -------
        Y : ndarray of shape (n_samples,)
            Gives the classification of each row in X
        """
        pass
\end{lstlisting}
\end{problem}

\begin{problem}
In this problem, you will train a Naive Bayes classifier using a corpus of emails extracted from the Enron dataset.

Load in the data from {\tt SpamFeatures.txt}. This is a text file containing a whitespace-delimited
numerical array with several thousand columns and several thousand rows, each row representing an email
as a count vector.
Also load in the data from {\tt SpamLabels.txt}, which is a text file containing a 1 (for legitimate email) or 0
(for spam email) on each line, in correspondence with the rows of the count vector array.
Using your document classification implementation, do the following:
\begin{enumerate}
\item Randomly create a test set from the data (500 documents), leaving the remaining documents as the training set.
\item Create a naive Bayes classifier and fit it using the training set.
\item Predict the labels of the test set and compare to the true labels (by reporting the classification accuracy).
\end{enumerate}

Next, perform the same task using {\tt sklearn}'s implementation and the same train and testing sets:
\begin{lstlisting}
>>> # assume train_vectors, train_labels, and test_vectors are defined
>>> from sklearn.naive_bayes import MultinomialNB
>>> mnb = MultinomialNB()
>>> mnb.fit(train_vectors, train_labels)
>>> predicted = mnb.predict(test_vectors)
\end{lstlisting}
Again report the accuracy of the predicted labels. The result should be on par with those produced by your
own implementation.

If you wish, you may use your Naive Bayes classifier on your own email.
To do so, you need to load in the words in the vocabulary, contained in the file {\tt SpamVocab.txt}, as follows:
\begin{lstlisting}
>>> with open("SpamVocab.txt", 'r') as f:
>>>     vocab = [s.strip() for s in f]
\end{lstlisting}
Next, load the email you wish to classify into memory, stored as a string object (either copy and paste the email
directly into the interpreter, or load it from a file). Once you have this string, convert it to a count vector using
the following code:
\begin{lstlisting}
from collections import Counter
def getCountVector(document, vocab):
    """
    Return the count vector for the given document using the given vocabulary.

    Parameters
    ----------
    document : string, words separated by whitespace
    vocab : list of strings of length n

    Returns
    -------
    counts : ndarray of shape (1,n)
    """
    tf = Counter(document.lower().split()) # get frequencies of each word
    counts = np.array([[tf[t] if t in tf else 0 for t in vocab]])
    return counts
\end{lstlisting}
Feed this output into the \li{predict} method of your classifier, and see how well it performs.
\end{problem}

\begin{comment}
I didn't find this section to be necessary. I wanted to jump right to full document classification.

A good analogy to help us understand is the problem of classifying dice as fair or loaded.
Suppose that we had 100 rolls from 1000 different dice.
Some dice are fair, but other dice favor rolls of 3 or 4.
Our task is to identify unlabeled dice as either fair or weighted.
We could generate such an experiment in python as follows.

\begin{lstlisting}

#Generate 100 rolls from 1000 dice, some fair and some weighted
#All weighted dice favor 3 and 4

import numpy as np
import random

prob_fair = 0.7
num_dice = 1000
num_rolls = 100

fair_die = np.array([1, 2, 3, 4, 5, 6])
weighted_die = np.array([1,2,3,3,3,3,4,4,4,4,5,6])

rolls = np.zeros((num_dice, num_rolls))
label = np.zeros((num_dice,1))

for i in xrange(num_dice):
    if np.random.random() < prob_fair:
        for j in xrange(num_rolls):
            rolls[i,j] = random.choice(fair_die)
        label[i] = 0
    else:
        for j in xrange(num_rolls):
            rolls[i,j] = random.choice(weighted_die)
        label[i] = 1

\end{lstlisting}

In this case, {\tt rolls} is a matrix whose rows are rolls from either a fair or a loaded die.
Our labels are stored in {\tt label}.
We can build a multinomial naive Bayes classifier using {\tt sklearn}.

\begin{lstlisting}
from sklearn.naive_bayes import MultinomailNB
mnb_classifier = MultinomialNB
mnb_classifier.fit(rolls,label)
\end{lstlisting}

The multinomial classifier trains on the roll data on each line.
We can now predict whether a new set of die rolls is either fair or loaded.

\begin{lstlisting}
roll = np.random.randint(1,7,size=100)
mnb.predict(roll)
\end{lstlisting}

\begin{problem}
The die roll problem is simplification of a spam filter.
The different kinds of die produce different kinds of rolls.
Thus, the probability of getting one kind of histogram of rolls is different depending on the label of the die.
Write an explanation of how you could extend these ideas to a spam filter, and how you would use {sklearn} to implement it.
\end{problem}
\end{comment}


\setchapterpreamble[u]{\margintoc}
\chapter{Mathematical Background}

The character of quantum mechanics is to a large extent dependent on the mathematics used to describe it. At the end of the day, all theory needs to be supported by experimental evidence. That is, by actual measurements. We quantify our measurements with numbers to speak in a standardized, precise way. When we ask questions like, ``How long?'' or ``How much?'', we typically use numbers to give an answer.

In quantum mechanics, we will use increasingly sophisicated mathematical constructs to describe the world around us, including experiments physicists perform. Some of properties of numbers that we take for granted will remain, while others will disappear. Here we begin with a exploration of two particular useful number systems: the \emph{real} and the \emph{complex} numbers.

\section{Real Numbers}\marginnote{\textbf{TODO:} Real numbers. Communitivity, distributivity. (Figures showing geometric equivalence.) \textbf{Exercises.} Field properties. Example, the real numbers. Binary. Finite fields. $\mathbf{Z}_2$. $V_4$. \textbf{Exercises.} Cartesian plane. Absolute value. Distance. Circles. \textbf{Exercises.}}

We begin with an old friend, the \emph{real line}. When we talk about the real line, we're really just talking about an ordinary geometric line. To define a sense of order, we label the points on the line with the set $\mathbf{R}$ of all \emph{real numbers}. That is, we identify each point on the line with a unique real number. And  we often speak as if the numbers were actually on a line, and as if the points on the line are actually numbers.

\begin{figure}[h]
\input{figures/mathematical-background/real-line.tex}
\caption{The real line}
\end{figure}

You may be wondering what makes the real numbers ``real''? The term real is unfortunate. The existence of real numbers is no more or less certain than any other kind of number---including the so-called imaginary numbers that we will meet later in this chapter. The philosopher Ren\'e Descartes came up with the term while investigating roots of polynomials in the 17th century. He was trying to distinguish different kinds of solutions. Since then, the name stuck. And it has confused students of mathematics ever since.

In some sense, the complex numbers, which include the combination of real and imaginary numbers, are much more real---or at least more complete---than the real numbers alone. The \emph{Fundamental Theorem of Algebra} guarantees that all polynomials of a single variable have a solution which is complex number. For example, the equation

\begin{equation}\label{eqn:mb-imaginary} x^2+ 1 = 0 \end{equation}

has no real number solution. We will see later that there is a complex number solution to that equation. Because complex solutions always exist for single-variable polynomials with complex coefficients, we say that the complex numbers are \emph{algebraically closed}.\sidenote{We will see a more examples of numbers that are closed under other operations soon.} The real numbers are not. The fact that equation \ref{eqn:mb-imaginary} has no real number solution is witness to the fact that the real numbers are not algebraically closed.

While the real line may look simple, do not be fooled. Its internal structure is incredibly complicated. For now we'll call out a few subsets of $\mathbf{R}$ that are so commonly used that they get their own symbols as well. They are the natural numbers $\mathbf{N}$, the integers $\mathbf{Z}$, and the rational numbers $\mathbf{Q}$.\sidenote{The \emph{Zahlen} means ``number'' in German. German mathematicians used $\mathbf{Z}$ to refer to integers in their papers. The rest is history.}\sidenote{Rational numbers are ratios of integers, AKA \emph{quotients}.}

\subsection{Natural Numbers}
The natural numbers $\textbf{N}$ are also sometimes called the counting numbers. They are the ones we first learn. They start 1, 2, 3, and continue that way one at a time.\sidenote{Sometimes people start the natural numbers at zero instead of one. It doesn't make much difference.} There is no largest natural number. They are infinite, which means that even these seemingly simple numbers are already quite complicated.

In ancient times, the Greeks restricted themselves to the numbers that could be constructed using a compass and an unmarked straight edge. They began with the natural numbers. Figure \ref{fig:mb-natural-numbers-construction} shows how you can produce any natural number, provided you have the time and paper.

First, starting a point, draw line segment with the straight edge. This line segment defines our unit length. The problem is how to produce another line segment of the same length as one you just drew. The straight edge, remember, is unmarked. So it doesn't help us out too much. The trick is to use the compass. For convenience, mark the starting point of line with the suggestive label \textsf{0} and the point at the other end with the label \textsf{1}.\sidenote{In Book \textsc{vii}, definition 1 of the \emph{Elements}, Euclid tells us that the first number is one, not zero. But let's follow E.W.~Djikstra's advice and start counting at zero anyway. \cite{Fitzpatrick2008, Djikstra1982}} We know that radii of a circle have the same length. So if we draw a circle centered at \textsf{1} that passes through \textsf{0}, it will contain all points that are all equally far away from \textsf{1} as the point \textsf{0} is. Breakthrough!

Now we can extend the original line segment to meet the circle. The point of their intersection is exactly one unit length away from the point \textsf{1}. Mark this point with the suggestive label \textsf{2}. We can continue this way as many times as we like---geometric line segments stretch as far as we care to imagine. After repeating the process $n$ times, we have constructed a line segment $n$ units long. Do it once again, and we're at $n+1$.

\begin{marginfigure}
  \input{figures/mathematical-background/natural-numbers-construction.tex}
  \caption{\label{fig:mb-natural-numbers-construction} Construction of the natural numbers}
\end{marginfigure}

There is something curious about the construction of the natural numbers given above: we got to pick the unit length in the first step. It didn't matter if we chose an inch, 19 centimeters, a dozen poronkusemas\sidenote{\emph{Poronkusema} is Finnish for the distance a reindeer can generally travel before stopping to urinate. It's not a precise measure of distance, perhaps 7.5 kilometers. But it's surely not more.}, or anything other length; the contstruction continues just same. Each choice gives a different \emph{coordinatization} of the line. \emph{Coordinates} are labels we give to points in geometric objects. In this case, the points on a line. Once we marked our zero point and chose a unit length, the positions of all of the coordinates of all the other points on the line were locked into place. Because our construction of the natural numbers did not make use of any particular coordinate system---remember, we got to choose the unit length---we say that the construction is \emph{coordinate-free} or \emph{coordinate independent}.

\begin{question}
  Figure \ref{fig:mb-3-real-lines} shows three real lines, each of which has been coordinatized by a different unit length. How are the coordinate systems related to one another?
\end{question}

\begin{marginfigure}
  \input{figures/mathematical-background/3-real-lines.tex}
  \caption{\label{fig:mb-3-real-lines} Three different coordinatizations by three different unit lengths}
\end{marginfigure}

\begin{question}
  Does Figure \ref{fig:mb-3-real-lines} prove that there are multiple, different real number lines? What do you think?
\end{question}

\begin{question}
  Consider a piece of string. Alice uses an English ruler to measure its length and finds the string to be exactly 25 inches long. Bob uses a metric ruler and finds the string to be exactly 63.5 centimeters long. Who is correct? What is the actual length of the string?
\end{question}

\begin{question}
  In what ways is the previous question about Alice and Bob like the geometric onstruction of the natural numbers?
\end{question}

You can do more than just construct the natural numbers. You can combine them! Two natural numbers $n$ and $m \in\mathbf{N}$ can be \emph{added} together to form their sum $n+m$ which is another natural number. To see the effects of addition geometrically, let's look at the specific of adding 1 to another number $n$. We can write this rule as

\begin{equation}
  \label{eqn:add-one}
  f: n \mapsto n + 1.
\end{equation}

We read equation \label{eqn:add-one} as the ``function $f$ that maps $n$ to $n + 1$.''\sidenote{Mathematical jargon is full of references to cartography. We have already bumped into maps and coordinates. Geometers deal with charts and altases, too. Maps capture relationships between the objects the represent, e.g., ``You are here. The nearest bathroom is there.'' Mathematics is the study of relationships between objects and transformations that preserve those relationships. So maps show up all the time.} We applied a function like $f$ when we first constructed the natural numbers, starting at the zero. When we apply $f$ to the whole real line, it looks like figure \ref{fig:add-one}.

\begin{question}
  Describe the effect of $f: n \mapsto n + 1$ on the real number line geometrically.
\end{question}

The two number lines in \ref{fig:add-one} are related to each other through the mapping $f$. The bottom lie is the \emph{image} of $f$. The image of a map is the set of output values $f(n)$ generated by applying the map to input values $n$. In this case, the image is a translation back one unit length by each coordinate. Informally, the whole number line got shifted back by one. Backward? That's a little confusing. Didn't we add to each point? Indeed we did.

\begin{marginfigure}
  \input{figures/mathematical-background/add-one.tex}
  \caption{\label{fig:add-one} The image of $f: n \mapsto n + 1$}
\end{marginfigure}

By adding 1 to each coordinate, we sped up the labeling of each point on the line. Thanks to $f$, coordinates appear one unit earlier than before, making them appear to have moved backwards. In this way, we can think of $f$ acting not on points on the line directly, but simply transforming their coordinate labels from one to another. The points on the line stay in place. Their labels move. To emphasis this viewpoint, we call $f$ a \emph{coordinate transformation} of the real line.

In general, coordinate transformations of the form $t_m: n \mapsto n + m$ translate the line $m$ units backwards. There is a special translation $t_0: n \mapsto n + 0$ called the \emph{identity} translation.\sidenote{The identity translation is so important, we'll give it its own notation $id_+$.} The identity translation is laziest translation you can imagine. It accepts an input $n$ and produces an output which is identically $n$. Because zero is the unique number such that $n + 0 = n$ for any $n \in \mathbf{N}$, we say that zero is the \emph{additive identity element} of the natural numbers.

\begin{question}
  Draw the image of $id_+$.
\end{question}

Notice that when $m \in \mathbf{N}$, the translation $t_m$ maps the coordinates of one natural number to the coordinate of another natural number. That is, the image $t_m(\mathbf{N})$ is the again the natural numbers $\mathbf{N}$. That is, the natural numbers are \emph{closed} under addition.\sidenote{I told you we'd see more examples of closure!}

\begin{question}
  What are some other operations that the natural numbers are closed under?
\end{question}

\begin{question}
  What is an operation that the natural numbers are not closed under?
\end{question}

% Closure under multiplication. (Scales things!) One as multiplicative identity. But $x + 1 = 0$ has no solution in the natural numbers. I.e., not closed under substraction. No, additive inverse. Enter the integers.

%
% \textbf{STREAM OF CONSCIOUSNESS}
%
% Then fractions. Pythagoras' cult discovered the irrationality of square root of two. Are there any holes? Answer to this question can be found in the study of real analysis. Basically all the numbers we would ever want are there. But special note: the reals are not the only completion of $\mathbf{Q}$. See, for example, the $p$-adics. For now, the real line will suffice.  We will review a few properties of the real numbers.
%
% Can we say \emph{the} real line with certainty? Field-preserving homomorphisms. Affine group. But real proof requires a proper construction, e.g., Dedekind cuts.
%
%
% \section{Complex Numbers}

% \marginnote{\textsf{Jordan Chapter 2: Imaginary Numbers}}


\section{Topic modeling}

\paragraph{Vector space models problems}
The approaches seen so far did not take into
consideration semantics of words.
We understood that the vector space models suffer
from high dimensionality, the terms also could be synonyms,
leading to ambiguity, in fact two words that means
the same thing are two different dimensions in the model.
\subsection{Linear algebra recap}

\paragraph{Eigenvalues and eigenvectors}
Given a square matrix of size $n$, if we find a
 $n$ dimensional vector such that
$$A\vec{x} = \lambda\vec{x}$$
$\lambda$ is a eigenvalue and $\vec{x}$ is an eigenvector.
It also holds that
$$(A - \lambda I )\vec{x} = 0 \implies |A - \lambda I| = 0$$
where $I$ is the identity matrix, and the implication holds as $\vec{x}$ is non-zero.
In general the second part of the implication has $n$ solutions, so
$n$ eigenvalues associated with a certain eigenvector.

After finding the solutions we can write that
\begin{equation}
    \begin{aligned}
        AX &= A[\vec{x_1}, \dots, \vec{x_n}] = A\vec{x_1} + \dots + A\vec{x_n}\\
        & = \lambda_1\vec{x_1} + \dots + \lambda_n\vec{x_n} = \varLambda X
    \end{aligned}
\end{equation}
Where $\varLambda$ is the diagonal matrix of the eigenvalues, and $X$ the matrix
with eigenvectors.

\paragraph{Diagonalization}
If the $n$ eigenvector found at the previous step are independent,
and the matrix $A$ is invertible, which is not a problem as its square,
we can write
$$AX = X\varLambda \rightarrow AXX^{-1} = XAX^{-1} \rightarrow A = X\varLambda X^{-1}$$

This representation is called diagonalization of $A$.
Also, if $A$ is symmetric, we can write
$$A = X\varLambda X^{\top}$$

\subsection{Latent Semantic Indexing}
The goal is to discover topics that motivate data
by using matrix factorization techniques, while
taking in consideration the possibility of expressing
the same topic with different words.

\paragraph{Diagonalization}
We learned that if a matrix is symmetric we can write the
diagonalization as $A = X\varLambda X^{\top}$.
The question is, what happens if we multiply the
document-term matrix by itself?

We find a symmetric matrix that somehow encapsulates a notion of similarity
of documents because if two documents share the same words the
product will be really high for that two documents.

If we multiply the term-document matrix we obtain a similar result.

\paragraph{Singular Value decomposition}
The idea behind the approach is to define the term document matrix
$A$ in terms of $$A = U \Sigma V^\top$$
this means finding something that holds together documents and terms.
The terms in the diagonalization represent:
\begin{itemize}
    \item$V$ is the matrix with the eigenvectors of $AA^\top$
    \item$U$ is the matrix with the eigenvectors of $A^\top A$
    \item $\Sigma$ is the diagonal matrix composed starting from the eigenvalues
    of $A A^\top$ and $A^\top A$, which are the same
\end{itemize}
Note that the eigenvalues in $\Sigma$ are sorted on the diagonal.

\paragraph{Defining topics}
After applying singular value decomposition
to the document term matrix we have the latent topic matrix
$\Sigma$.
We can consider a new matrix $\Sigma_{k \times k}$, that
somehow encapsulate the top $k$ relevant topics for
the documents.
Note that after this cut of $\Sigma$ we get a diagonalization
that is an approximation of the original matrix.
$$A \approx U \Sigma_{k \times k} V^\top$$

To make topics human understandable
we assign each a set of words, and also, we can assign to each
document a topic, in a soft clustering fashion.

Basically, given the three matrixes that SVD outputs, after
cutting $\Sigma$ as discussed above,
we can discover what topics have the documents by multiplying
$U$ and $\Sigma_{k \times k}$. The output will be a matrix with document
on rows and topics on columns, where the absolute value of
a cell represent how strong a topics is present in a document.

In the same way we can find a relation between topics and words
by multiplying $\Sigma_{k \times k}$ and $V^\top$.

\subsection{Latent Dirichlet Allocation}
This approach relies on probability theory to find topics
in documents.
In the LSI approach we tried to find a \emph{glue}, formally $\Sigma$,
between document and terms.

In LDA the goal is to estimate the probabilistic relation
between topics and documents, basically the probability that
a given document contains a given topic, and the one that
a topics contains a certain word.

\paragraph{Elements}
The basic elements of the approach are:
\begin{itemize}
    \item $\phi^{(k)}$ probability distribution over the vocabulary
    for the kth topic
    \item $\theta_d$ document distribution over topics
    \item $z_i$ topic index for the word $w_i$
    \item $\alpha, \beta$ hyperparameters that govern the first
    two distributions
\end{itemize}

\paragraph{Estimating the generative model}
To find the topics and document distributions we
assume that they were generated by a given distribution
and try to estimate them.

\dots\\
\dots\\
\dots