%!TEX root =  ../main.tex

\section{Matrices}

\paragraph{Matrix}
A \textbf{matrix} (plural matrices) is a rectangular array of numbers, symbols, or expressions, arranged in meaningful rows and columns.  The individual items in a matrix are called its elements or entries.

\paragraph{Size, Entries}
A matrix's size is described by the number of rows, by the number of columns.  If a matrix is given
a name, an entry may be referred to by a subscript of row and column on that letter.  For example,
on matrix \textbf{[A]}, one might refer to the entry in the second row and third column as
\textbf{A}$_{2,3}$

\begin{example}
\exProblem
Given that A is 1234 what is $A_{1,4}$?

\exSolution
4
\end{example}

\subsection{Addition and Scalars}
Matrices may be added if and only if they are the exact same size.  A matrix maybe multiplied by
a number (called a \gls{scalar}), which is simply multiplied against every element in the matrix.
Two matrices are added just by adding the for corresponding entries, i.e. $A_{i,j}+B_{i,j}$
produces the new entry at $i,j$.

\begin{example}
\exProblem
If A is 1234, what is 2A?

\exSolution
2468?
\end{example}

\begin{example}
\exProblem
If A is 1234 and B is 0102 what is A+B?

\exSolution
1336
\end{example}

\subsection{Matrix Multiplication}
The product of two matrices is the coming together of rows of the first, with columns of the second.  For example,
to compute the top left entry in the product of two matrices, one multiplies each entry in the first row of the first matrix,
against the corresponding entry in the first column of the second matrix.  (See below for a helpful visual.)  Naturally,
this means that the rows and columns must match up.

\subsection{Square Matrices}
identity, inverses, determinants
\subsection{Gaussian Elimination}
augmented matrices


\chapter{Artificial Neural Networks}
\label{chap:neuralnetworks}

\emph{Artificial neural networks} are a class of machine learning algorithms which are loosely inspired by neuroscientific models of how biological neuronal networks operate.
Early work in neuroscience by Santiago Ram{\'o}n y Cajal indicated that the nervous system is composed of complex networks of individual cells, later called neurons~\cite{cajal1955}.
These neurons consist of a soma and axon.
The soma is the main part of the cell, and the axon extends from the soma, attaching to one or more somata from other neurons.
A neuron creates a so called "action potential" if it is sufficiently excited by the axons attached to it, which propagates down its own axon to help excite or inhibit other neurons to which it is attached~\cite{mcculloch1943}.
In this way, electrical signals are propagated through a neronal network.
Work by Hodgkin \& Huxley described mathematically how action potentials are generated in neurons and propagate from one neuron to another~\cite{hodgkin1952}.

Artificial neural networks mimic this behavior with a mathematical network of \emph{artificial} neurons.
Mathematical signals propagate through this network, whereby each neuron performs a simple mathematical operation on its inputs to determine its output.
\emph{Feed forward artificial neural networks} are a subclass of artificial neural networks which contain no cycles, that is, there is no sequence of input/output connections that start and end at the same neuron.
This class of artificial neural network always contains at least one input neuron with nothing feeding into it, and at least one output neuron which feeds into nothing.
They therefore constitute a function, where the input is applied to the input neurons, the signal propagates through the network, and the output is extracted from the output neurons.
If the neuron operations are parameterized, then it may be possible to approximate various other functions through the appropriate selection of parameters.
Let the function defined by a feed forward artificial neural network be denoted by $f(x|\Theta)$, where $x$ is the input vector and $\Theta$ is the complete set of parameters.

For feed forward neural networks, $f$ is computed in a sequence of layers, each consisting of some number of neurons, which perform comparatively simple mathematical operations on their inputs to produce an output.
A layer's output is either used as input to the subsequent layer, or taken as the network output if there are no subsequent layers.
By convention, the input vector $x$ is considered the first layer of a network, with one neuron per input, where each neuron takes the value of the respective input.
The last layer in the network produces the function's output, either a scalar or vector of values.
All intermediate layers are termed \emph{hidden layers}, each of which accepts the output of the preceding layer and produces one output per neuron in that layer.
The number of layers in a network (besides the first layer) is its \emph{depth}.
Mathematically, we can express the output for layer $i$ as $h_i(x_i| \theta_i)$, where $x_i$ is the input to the layer and $\theta_i$ is the set of parameters for that layer.

Some important notes on terminology: artificial neurons are also called \emph{units}, which will be the preferred term hereafter to avoid confusion with biological neurons. Also, omitting the \emph{artificial} from artificial neural networks is common parlance in the literature, and will be done in this thesis for convenience. To avoid possible confusion, biological neuronal networks will always be explicitly referred to as such.

A common form of neural network layer is a \emph{dense layer}, in which each unit calculates a weighted sum of the layer inputs, where the set of weights is unique to each unit.
It is also common to add a scalar bias term and apply a nonlinear activation function to the weighted sum.
Dense layers have a convenient mathematical representation:

\begin{equation}
h_i(x_i|W, b)=\sigma(W x_i + b),
\label{eq:denselayer}
\end{equation}

\noindent
where $i$ indexes the layer, $W$ is a matrix of weights, $x_i$ is the layer input, $b$ is a vector of biases, and $\sigma(\cdot)$ is a nonlinear activation function.
If there are $n$ inputs to a layer and $m$ units in the layer, then $W\in\mathbb{R}^{n \times m}$ and $b\in\mathbb{R}^{m}$.
The expression $W x + b$ is the \emph{signal}, and $h$ is the \emph{activation}.
For hidden layers, activations have traditionally taken the form of a sigmoidal, or "s-shaped", function such as the logistic function, $\sigma(x) = 1/(1+e^{-x})$, or hyperbolic tangent, $\sigma(x) = tanh(x)$.
These sigmoidal functions switch from "off" (low value) to "on" (high value) if the weighted sum in the argument is sufficiently high.
This loosely mimics the way that signals from attached axons combine to trigger an action potential in a neuron.
Figure \ref{fig:twolayernetwork} shows a graphical depiction of a parameterized, feed forward artificial neural network.

\begin{figure}
	\centering
	%\begin{center}
	\includegraphics[width=0.8\textwidth]{simpletwolayernetwork.pdf}
	%\end{center}
	\caption{A two layer neural network with two inputs, three hidden units, and a single output. The hidden layer and output are both dense layers, and arrows depict the weighted sum computed for each unit. Weights for each layer are numbered according to their index in the weight matrix W}
	\label{fig:twolayernetwork}
\end{figure}

Despite being conceptually simple, this formulation of neural networks is capable of approximating any continuous function on a compact subset in $\mathbb{R}$ with arbitrarily small error, given enough hidden units~\cite{cybenko1989}.
The challenge of using neural networks for function approximation is in finding the appropriate set of parameters to approximate the true function.
For simple feed forward neural networks, this is accomplished by quantifying the error between the true function and the approximation for a given training data set $X=(x_1, x_2, ..., x_N)$, using a differentiable loss function $L(\Theta | X)$.
The weights are updated by differentiating $L$ with respect to $\Theta$ and "taking a step" in parameter space opposite the direction of the gradient.
This update step can be repeated iteratively in a process called \emph{gradient descent}, which was first proposed by Augustin-Louis Cauchy~\cite{cauchy1847}.
Mathematically, parameters are updated according to:
\begin{equation}
\Theta_{k+1} = \Theta_k - \eta \nabla L(\Theta_k | X) = \Theta_k - \eta \sum_{n=1}^{N} \nabla L(\Theta_k | x_n),
\label{eq:batch_gd}
\end{equation}

\noindent
where $k$ indicates the iteration and $\eta$ is a tunable step size.
The gradient can be efficiently calculated using an algorithm called \emph{backpropagation}~\cite{rumelhart1988}.
Weights are usually initialized by drawing from random distributions that have some empirical or theoretical justification~\cite{glorot2010}.
A variant of this algorithm, called \emph{stochastic gradient descent} (SCG) performs an update using a gradient computed from a random sample of the training data at each iteration.
There is also a batched version where the training data are randomly shuffled and divided into a fixed number of equal sized \emph{mini-batches}, and an update is performed on each one.
When all mini-batches have been used in an update step, this constitutes an \emph{epoch}.
Training can be repeated for a fixed number of epochs or until the parameters sufficiently converge.
Gradient descent is a first order method because it uses only the gradient of the loss function, however various second order methods which incorporate the Hessian have also been used to train neural networks~\cite{fletcher1964, polak1969, moller1993, marquardt1963}.

Like many machine learning algorithms, neural networks risk \emph{overfitting}, in which the network learns to approximate the training data (which usually contain noise), rather than the underlying function from which the training data are assumed to be drawn.
This hinders the ability of neural networks to generalize to unseen data.
In neural networks, overfitting is commonly the result of an overcomplete parameterization coupled with overtraining~\cite{reed1993, dalianis1993}.
Many regularization techniques have been introduced to prevent overfitting of neural networks, including early stopping~\cite{morgan1990}, model pruning~\cite{reed1993}, weight decay\cite{krogh1992}, lateral inhibition~\cite{krizhevsky2012}, induced sparsity~\cite{ng2011, makhzani2015}, and dropout~\cite{srivastava2014}, with dropout being one of the most common.

Dropout consists of randomly dropping units from the network during the training phase.
Each time a training example is presented to the network, each unit in the network is dropped with probability $p$, commonly 0.5.
Each application of dropout can be seen as generating a new network whose units are a subset of the units in the original network.
Given $n$ units in a network, there are $2^n$ possible dropout networks which all share weights with each other.
During testing, no dropout is performed, and weights are rescaled by $p$.
This is equivalent to averaging the prediction of each dropout network.
Dropout helps prevent overfitting by limiting the effective number of parameters in each dropout network and reducing the amount of co-adaptation among units in the network~\cite{srivastava2014}.

In recent years, more advanced forms of neural networks under the moniker \emph{deep learning} have demonstrated success in sophisticated tasks, particularly for image and speech recognition~\cite{krizhevsky2012, lecun2015, masci2011, hinton2012, he2016}.
A common task is labeling an image or audio sample according to its content.
Labeling tasks are commonly decomposed into many binary tasks, where a neural network must indicate whether a particular label is appropriate.
Positive examples are those where the label is appropriate, and negative examples are those where it is not.
Deep learning has also been applied in a variety of other applications such as quantitative structure activity relationship (QSAR) prediction~\cite{ma2015}, particle detection~\cite{ciodaro2012}, and reinforcement learning\cite{mnih2015}.
These advances were catalyzed by the availability of large volumes of labeled data\cite{deng2009, krizhevsky2009} and the improvements in computing power from general purpose graphical processing units (GP-GPUs) and distributed systems\cite{chetlur2014, chu2007}.
Such factors allow experimentation with larger networks and more complicated layer operations such as convolution, which are described in the following section.

\section{Convolutional Neural Networks}

Accurately labeling an image using a neural network often relies on the network's ability to detect low level features in the image such as edges and textures, which collectively indicate the image's content~\cite{ng2011}.
This is accomplished by the appropriate parameter settings, which maximize a unit's activation when presented with a feature of interest, and can be learned through the standard backpropagation algorithm using training images.
The challenge, however, is that the relevant features may only occur in certain regions of the training images, so the network may not be able to recognize that feature in a new region.
Convolutional layers explicitly allow detection of features in a translation invariant way, through a clever weight sharing strategy.

To illustrate the limitation of dense layers in translation invariant feature detection, consider the following example.
Suppose the labeling task is to determine if an image does or does not contain a cat.
Suppose further that no training image contains a cat in the upper left corner of the image.
Therefore it is possible, even if the network has learned which low level features indicate the presence of a cat, that it will not recognize a cat contained in the upper left corner of the image, since it has not been explicitly shown examples of this.
This may seem unlikely, but consider that the network does not incorporate any semantic meaning of the word "cat", only that a certain set of example images are positive examples and a certain set are negative examples.
Indeed, consider the slightly different labeling task where the classes being considered are "contains a cat anywhere but the top left corner" and its negation.
In this case, the same set of training images are appropriate and the hope is that the network would \emph{not} classify images with cats in the upper left corner as positive examples.
In most cases, the labeling task is more like the former example than the latter, in which case it would be beneficial for the network to learn to detect objects in a \emph{translation invariant} fashion.
This is the primary purpose of convolutional layers.

The term convolution is inherited from the field of functional analysis, where two functions, $f$ and $g$, are combined in the following way to generate a third function, $(f*g)$:

\begin{equation}
(f*g)(t) = \int_{-\infty}^{\infty}f(\tau)~g(t-\tau)~d\tau.
\label{eq:math_conv}
\end{equation}

\noindent
The convolution involves reflecting one function about the y axis, shifting it with respect to the other function, and integrating the product.
The value of $t$ indicates the magnitude of shift.
If $f, g \colon \mathbb{Z} \rightarrow \mathbb{R}$, then the discrete convolution is analogously defined as:

\begin{equation}
(f*g)(t) = \sum_{\tau = -\infty}^{\infty}f(\tau)~g(t-\tau).
\label{eq:math_conv}
\end{equation}

\noindent
Where $t, \tau \in \mathbb{Z}$.
If we imagine $f$ and $g$ to be infinite dimensional vectors indexed by $\tau$, then convolution amounts to reversing $g$, shifting it by $t$ units relative to $f$, and taking the dot product of the two vectors.
The same operation can be defined for two finite vectors by making them infinite by appending infinitely many zeros before and after them.
The zero portions contribute nothing to the convolution and allow the original finite vectors to be partially overlapping.
A finite vector can be extracted from the result, since the convolution function is nonzero only when the convolved vectors overlap.
Figure \ref{fig:cont_disc_conv} illustrates the analogy between continuous convolution and discrete convolution of finite vectors.

\begin{figure}
	\centering
	%\begin{center}
	\includegraphics[width=0.8\textwidth]{discrete_continuous_conv.png}
	%\end{center}
	\caption{The relative position of two functions at four different values of t, for both continuous (left) and discrete (right) convolution. The functions are distinguished by color. In the continuous case, the functions are multiplied and integrated whereas in the discrete case, the dot product of the two arrays is taken (extending either array with zeros when necessary).}
	\label{fig:cont_disc_conv}
\end{figure}

Convolution can be extended for functions of two variables:

\begin{equation}
(f*g)(t_1, t_2) = \sum_{\tau_1 = -\infty}^{\infty} ~\sum_{\tau_2 = -\infty}^{\infty}f(\tau_1, \tau_2)~g(t_1-\tau_1, t_2-\tau_2),
\label{eq:math_conv}
\end{equation}

\noindent
where $t_1, t_2, \tau_1, \tau_2 \in \mathbb{Z}$.
Here, $f$ and $g$ can be interpreted as infinite matrices, where convolution involves reversing $g$ in both indices, shifting it in both indices relative to $f$, and taking the dot product with $f$.
Again, this may be adapted for two finite matrices using the same infinite extension of zeros as for vectors.
This discrete, two dimensional, finite analog to convolution forms the basis of convolutional layers in neural networks.

To understand convolution of a monochrome pixelated image, consider it as a finite matrix where the matrix entries contain pixel values.
Further, consider a smaller matrix called a \emph{filter} which entries are filter \emph{weights}.
By convolving the filter over the image (as previously described), a new matrix is created where each value is the dot product of the matrices for a given shift of the filter relative to the image.
Technically the filter can be assumed reflected already, since the filter weights are trainable parameters anyways.
The region of the image covered by the filter is called the \emph{receptive field}, and the dot product can be understood as taking a weighted sum of values in the receptive field, where the weights are precisely the filter weights.
As with dense layers, each weighted sum is typically passed through some nonlinear activation function.
Figure \ref{fig:convolutionallayer} depicts the application of a filter to the input of an image in three distinct positions.
The dot product between filter and image naturally extends to images with multiple \emph{channels} (colors), as long as the filter has the same number of channels as the image.

\begin{figure}
	\centering
	%\begin{center}
	\includegraphics[width=0.8\textwidth]{conv_grid.png}
	%\end{center}
	\caption{Application of a 3x3 convolution filter to an input image. The dot product is taken between the filter and the receptive field (green) to produce a scalar output (red). Three positions of the filter are shown, each with a unique receptive field and output position.}
	\label{fig:convolutionallayer}
\end{figure}

The result of a convolution in a particular position depends on the values of the input image and the weights, and indeed there exist weights which produce larger outputs only in the presence of particular local features in the input.
Hence convolution can be used in a neural network to detect patterns in the input.
Furthermore, since a filter is convolved across the whole image, feature detection can be performed in a translation invariant way as desired.
A filter may only detect a particular local feature, so multiple filters can be used to detect multiple features simultaneously, each producing a different channel in the output.
Note that the output of a convolution layer resembles the structure of the input, so convolutional layers may be stacked.

Convolutional neural networks often incorporate \emph{pooling layers}, which reduce a region of the input (e.g. 3x3 pixels for an image) to a single grid element that takes either the maximum or average value (per channel) of the region.
For simplicity in this thesis, grid elements will be called pixels, regardless of grid dimensionality.
Pooling allows downsampling of the grid, and is often performed between convolutional layers.
It's also common to include a series of dense layers after all convolutional layers which combine the results of convolution in a way relevant to the learning task.
In order to do this, the multichannel input must first be flattened into a one dimensional vector.
Figure \ref{fig:conv_network} shows an example of a complete convolutional neural network.

\begin{figure}
	\centering
	%\begin{center}
	\includegraphics[width=0.8\textwidth]{conv_network.png}
	%\end{center}
	\caption{A convolutional neural network, consisting of alternating convolution and pooling layers, followed by two dense layers, producing a scalar output. Stacked rectangles indicate multiple channels. These convolutional layers increase the number of channels, which is typical but not necessary. Pooling downsamples the image but retains the same number of channels.}
	\label{fig:conv_network}
\end{figure}

The design of convolutional layers gives rise to certain properties which prove useful in image related tasks:
\begin{itemize}
	\item \emph{Structural awareness.}
	Convolutional filters are constructed so that they exploit the structural characteristics of the input.
 	Rather than treat each pixel as independent of all others, convolution operates on local regions of pixels, allowing filters to detect spatial correlations in a straightforward manner.
	\item \emph{Locality}.
	Filters are typically significantly smaller than the size of the input, so for a given position, they are operating on a local neighborhood (receptive field) of the input image.
	Any features learned by a filter will necessarily be local as well.
 	The advantage of local features is that they are simpler and more universal than global features, since they can be combined to construct a wide variety of complex patterns (see \emph{abstraction} below).
	For example, filters which detect edges or textures will be more useful to subsequent layers in the network than a filter which recognizes only a single object or animal.
 	\item \emph{Translation invariance}.
 	As discussed, a filter "inspects" each part of the image for a particular pattern, regardless of where that pattern occurred in the training images.
 	This improves generalizability because during training certain features may be presented in limited regions of the image, but detection of those features can still operate on a global scale.
 	\item \emph{Abstraction}. Whereas a network's first convolutional layer detects patterns in the input, subsequent layers detect patterns in the output of previous layers.
 	This promotes a hierarchical abstraction of the input, where early layers detect local features, and subsequent layers combine features into more complex patterns.
 	For example, early filters may detect edges of varying orientations, middle filters may detect combinations of edges which indicate curves in space, and latter filters may detect combinations of curves which indicate a particular handwritten letter or digit.
 	\item \emph{Resistence to overfitting}.
 	One interpretation of a convolutional layer is that each filter is a series of units, each responsible for a distinct receptive field on the input.
 	A unit has nonzero weights only for the regions of the input in its receptive field, and all units in the filters share the same weights.
 	This combination of weight sharing and sparsity make a convolutional layer less prone to overfitting compared to a dense layer with the same number of units.
 	Because of the weight sharing in convolutional layers, the final dense layers often contain the majority of the parameters in the network.
\end{itemize}

The above properties make convolutional neural networks well suited for structured data with a natural hierarchy of abstraction.
The structural hierarchy of proteins suggests that convolution might be useful in interface prediction.
Unfortunately, neither protein residues, nor their constituent atoms are aligned in a regular grid, so the above definition of convolution cannot be directly applied.
In Chapter \ref{chap:methods}, convolutions are introduced which operate on irregular structures, namely graphs, and a graphical representation of proteins is presented which allows use of graph convolution for protein interface prediction.


\subsection{Row and Column Swapping}

Can define elementary row operations in the identity matrix.

\subsubsection{Swapping rows}

\[
    \begin{bmatrix}
        0&1\\
        1&0
    \end{bmatrix}
    \begin{bmatrix}
        a&b\\
        c&d
    \end{bmatrix}
    =
    \begin{bmatrix}
        c&d\\
        a&b
    \end{bmatrix}
\]

Modifier $B$ is always on \textbf{left}.

\subsubsection{Swapping columns}

\[
    \begin{bmatrix}
        a&b\\
        c&d
    \end{bmatrix}
    \begin{bmatrix}
        0&1\\
        1&0
    \end{bmatrix}
    =
    \begin{bmatrix}
        b&a\\
        d&c
    \end{bmatrix}
\]

Modifier $B$ is always on \textbf{right}.

\subsection{Elimination steps}

Performing elimination:

\[E_{2,1}A+E_{3,2}A=(E_{2,1}E_{3,2})A=U\]

Elimination algorithm:
\begin{itemize}
    \item $E_{2,1}$ is the pivot. Swap with $R_2$ if 0 (and $(2,1)$ is nonzero).
    \item $E_{3,2}$ involves getting $(2,2)$ as a pivot assuming nonzero to get $(3,2)$ as 0
    \item Result is invertible and non-singular, where $U$ is upper-triangular
\end{itemize}

Matrix multiplication is not necessarily commutative but always associative.

\subsection{Matrix Multiplication Facts}

If $A$ is an $m\times n$ matrix and $B$ is $n\times p$, then $AB=C$ must be $m\times p$.
Standard method would be to take dot products by row and column. By column: Columns of $C$ are combinations of columns
of $A$. By row: Rows of $C$ are combinations of rows of $B$.

\subsection{Example (Row)}

\[
    \left[\begin{array}{ll}
        2 & 7 \\
        3 & 8 \\
        4 & 9
        \end{array}\right]\left[\begin{array}{ll}
        1 & 6 \\
        0 & 0
        \end{array}\right]=\left[\begin{array}{l}
        2 \\
        3 \\
        4
        \end{array}\right]\left[\begin{array}{ll}
        1 & 6
        \end{array}\right]+\left[\begin{array}{l}
        7 \\
        8 \\
        9
        \end{array}\right]\left[\begin{array}{ll}
        0 & 0
    \end{array}\right]
\]


%!TEX root = index.tex
\section{Symmetry groups}

\epigraph{Mathematics is the art of giving the same name to different things.}{Henri Poincare}

\subsection{Multiplying permutations}
Let $ [n]$ denote the set $ \{1, 2, \ldots, n \} $. The \textbf{symmetry group of $ n$ elements}, $ S_n$, is defined to be the set of all permutations of $ n$ elements. We think of the permutations as \emph{functions} $ [n] \rightarrow [n]$.  For example, the element $ \sigma = (1 \: 2 \: 4 \: 3)$ denotes the function $ \sigma: [4] \rightarrow [4]$ which sends $ \sigma(1) = 1, \sigma(2) = 2, \sigma(3) = 4, \sigma(4) = 3$,
  \begin{align*}
    1 & \mapsto 1 \\
    2 & \mapsto 2 \\
    3 & \mapsto 4 \\
    4 & \mapsto 3
  \end{align*}
This simple change in perspective now allows us to \textbf{multiply} two permutations, by simply composing the corresponding functions. For example, the element $ \sigma = (1 \: 2 \: 4 \: 3)$ and $ \tau = (4 \: 1 \: 2 \: 3)$ the \emph{product} $ \sigma \cdot \tau = (4 \: 1 \: 3 \: 2)$
  \begin{align*}
    1 & \mapsto^\sigma 1  \mapsto^\tau 4 \\
    2 & \mapsto^\sigma 2  \mapsto^\tau 1 \\
    3 & \mapsto^\sigma 4  \mapsto^\tau 3 \\
    4 & \mapsto^\sigma 3  \mapsto^\tau 2
  \end{align*}
Note that this is like \emph{applying} $ \sigma$ to $ \tau$.

\begin{questions}
  \item For $ e = (1 \: 2), \tau = (2 \: 1) \in S_2$ compute: $ e \cdot e, e \cdot \tau, \tau \cdot e, \tau \cdot \tau$
\end{questions}

\begin{questions}[resume]
  \item For $ \sigma = (2 \: 3 \: 1), \tau = (2 \: 1 \: 3) \in S_3$ compute
  \begin{enumerate}
    \item $ \sigma \cdot \sigma$, $ \sigma \cdot \sigma \cdot \sigma$
    \item $ \sigma \cdot \tau$, $ (\sigma \cdot \tau) \cdot (\sigma \cdot \tau)$
    \item $ \tau \cdot \sigma$
    \item Write all the elements of $ S_3$ in terms of $ \sigma$ and $ \tau$.
  \end{enumerate}
  Because $\sigma \cdot \tau \neq \tau \cdot \sigma$ we say that $ S_3$ is \textbf{non-abelian}.
\end{questions}

\newpage
\subsection{Subgroups}
\begin{questions}[resume]
  \item Let $ e = (1 \: 2 \: 3 \: \cdots \: n) \in S_n$. Let $ \sigma $ be any permutation in $ S_n$.
  \begin{enumerate}
    \item Compute $ e \cdot \sigma$ and $ \sigma \cdot e$.
    \item Find an element $ \tau$ such that $ \tau \cdot \sigma = e$ and $ \sigma \cdot \tau = e$.
  \end{enumerate}
  Hence $ e$ is called the (group) \textbf{identity} in $ S_n$ and $ \tau = \sigma^{-1}$ is called the \textbf{inverse} of $ \sigma$.
  \begin{enumerate}[resume]
    \item Explicitly find the inverses of all the elements of $ S_3$.
  \end{enumerate}
\end{questions}
A set with (associative)  multiplication, an identity, and inverses is called a \textbf{group}.

A subset $ G \subseteq S_n$ which is itself a group is called a \textbf{subgroup} of $ S_n$ i.e. $ G$ is a subgroup of $ S_n$ if
\begin{itemize}
  \item (contains identity) $ e \in G$
  \item (closed under inverses) $ g \in G \implies g^{-1} \in G$
  \item (closed under multiplication) $ g,h \in G \implies g \cdot h \in G$
\end{itemize}

\begin{questions}[resume]
  \item Show that $\{e\}$ is a subgroup of any $ S_n$. This is called the \textbf{trivial group}.
  \item  Let $ \sigma = (3 \: 1 \: 2), \tau = (2 \: 1 \: 3) \in S_3$.
  \begin{enumerate}
    \item Show that $\{e, \tau\}$ is a subgroup of $ S_3$.
    \item What element(s) do you need to add to $ \{ e, \sigma\}$ to make it a subgroup  of $ S_3$?
    \item What element(s) do you need to add to $ \{ e, \tau, \sigma\}$ to make it a subgroup of $ S_3$?
  \end{enumerate}
  \item List all the subgroups of $ S_3$.

  \item For $ \sigma = (2 \: 3 \: \cdots \: n \: 1) \in S_n$,
  \begin{enumerate}
    \item Find $ \sigma^k$.
    \item Find $ \sigma^{-1}$.
    \item What are the elements that needs to be added to the set $ \{ e , \sigma \} \subseteq S_n$ to make it a subgroup?
  \end{enumerate}
\end{questions}

\newpage
\subsection{Group Actions}
Groups naturally occur as symmetries of mathematical objects, $ S_n$ is the symmetry group of a set of $ n$ elements. But the same group can show up as the symmetries of multiple objects. We think of the group as \textbf{acting} on the object via some self-transformations. \\

For example, $ S_3$ \emph{acts} an equilateral triangle with vertices labelled $ \{ 1,2,3 \}$ via geometrical transformations (rotations and reflections): $ \sigma = (2 \: 1 \: 3)$ \emph{acts} via rotating the triangle counterclockwise by $ 2\pi/3$ and $ \tau = (2 \: 1 \: 3)$ \emph{acts} via reflecting along one of the bisectors. On the other hand $ S_3$ does not act naturally on, say, an isosceles triangle, in this case the natural symmetry group is $ S_2$ which acts by reflecting along \emph{the} bisector. For a scalene triangle, the natural symmetry group is the trivial group.

\begin{questions}[resume]
  \item \begin{enumerate}
    \item How many geometrical transformations are there of a square?
    \item $ S_4$ does not act naturally on a square. What is an example of an element of $ S_4$ which does not correspond to any geometrical transformation?
    \item What is the \emph{subgroup} of $ S_4$ that acts via geometrical transformations?
  \end{enumerate}

  \item Generalize the above problem to a regular $ n$-gon. These groups are called the \textbf{dihedral groups}, denoted $ D_{2n}$. (why $ 2n$?)

  \item If we restrict only to rotations and do not allow reflections, then $ S_4$ does not naturally act on a tetrahedron.
  \begin{enumerate}
    \item What is an example of an element of $ S_4$ which does not correspond to any rotation of the tetrahedron?
    \item Describe how the other elements of $ S_4$ act on the tetrahedron.
    \item How many rotational symmetries does the tetrahedron have? What is the rotational symmetry group?
  \end{enumerate}
\end{questions}

The rotational symmetry group of a cube is also extremely interesting and easy to understand, try reading about it online.


\section*{Problem 4}

\begin{problem}

The following problem has two parts, writing the algorithm and then programming it.
\end{problem}

\begin{subpr}{\hfill 10 pts.} Using the expression for the force between a charge and a metal sphere. First, write how one can compute the electrostatic force between two metal balls of radius $r_1$ and $r_2$, charged with charges $q_1$ and $q_2$ respectively, in free space, with the distance between their centres being $d$. (The actual computation may be tedious.)

Say this force is $F$ Consider the following dimensionless quantities:
$$f=\frac{4 \pi \epsilon_0 d^2}{{q_1}^2}\cdot F$$
$$u=\frac{r_1}{d}$$
$$v=\frac{r_2}{d}$$
$$p=\frac{q_1}{q_2}$$
Express all equations in the above procedure in terms of these dimensionless quantities.

Write a program in any language (Python, C, etc.) to compute $f$ given $u,v,p.$ The maximum uncertainty in $f$ from the calculation must not be more than $\approx 1\%$ for $p \approx 1$ $u  \approx v \approx 0.5.$
\end{subpr}
\clearpage


%!TEX root = ../Thesis.tex
\chapter{Non-Dimensionalization of Hamilton's Equations} \label{app:non-dimensionalization}
We set:
\begin{align}
t &= k_t T \\
x &= k_x X \\
y &= k_y Y \\
p_x &= k_{px} P_X \\
p_y &= k_{py} P_Y ,
\end{align}
with
\begin{align}
k_t &= \dfrac{1}{\omega} = \sqrt{\dfrac{(R+r)^3}{G(M+m)}} \\[0.5cm]
k_x &= k_y = R+r \\[0.5cm]
k_{px} &= k_{py} = m_s \sqrt{\dfrac{G(M+m)}{(R+r)}} .
\end{align}

So we get for $\dot{X}$
\begin{align}
\dfrac{k_x}{k_t} \dod{X}{T} &= \dfrac{k_{px} P_X}{m_s} + \omega k_y Y \\[0.3cm]
\Leftrightarrow \qquad \dod{X}{T} &= \dfrac{k_t k_{px}}{k_x} \dfrac{P_X}{m_s} + \omega k_t \dfrac{k_y}{k_x} Y \\[0.3cm]
\Leftrightarrow \qquad \dot{X} &= P_X + Y .
\end{align}

And for $\dot{Y}$ by symmetry
\begin{align}
\dot{Y} = P_Y - X \ ,
\end{align}

For $\dot{P}_X$ we get
\begin{align}
\dod{P_X}{T} &= \dfrac{k_{py}}{k_{px}} k_t\omega P_Y - \dfrac{k_t}{k_{px}}G m_s \left[\dfrac{M(k_x X + R)}{\left(k_x X + R)^2 + k_y^2 Y^2\right)^{3/2}} + \dfrac{m (k_x X - r)}{\left((k_x X - r)^2 + k_y^2 Y^2\right)^{3/2}} \right].
\end{align}

Now letting some of the $k$s cancel and expanding the rest of the $k$s, we get
\begin{align}
\notag &= P_Y - \left(\sqrt{\dfrac{(R+r)^3}{G(M+m)}}\right) \left(\dfrac{1}{m_s} \sqrt{\dfrac{(R+r)}{G(M+m)}}\right) G m_s \cdot \\
&\left[\dfrac{M[(R+r) X + R]}{[((R+r) X + R)^2 + (R+r)^2 Y^2]^{3/2}} + \dfrac{m[(R+r) X - r]}{[((R+r) X - r)^2 + (R+r)^2 Y^2]^{3/2}} \right].
\end{align}
First simplifying and pulling out $(R+r)$, then multiplying everything into the big parenthesis, we get:
\begin{align}
\notag &= P_Y - \dfrac{(R+r)^2}{M+m} \cdot \\
&\left[\dfrac{M(R+r)\left(X + \dfrac{R}{R+r}\right)}{(R+r)^3\left[\left(X + \dfrac{R}{R+r}\right)^2 + Y^2\right]^{3/2}} + \dfrac{m (R+r)\left(X - \dfrac{r}{R+r}\right)}{(R+r)^3\left[\left(X - \dfrac{r}{R+r}\right)^2 + Y^2\right]^{3/2}} \right] \\[1cm]
&= P_Y - \left[\dfrac{\dfrac{M}{M+m}\left(X + \dfrac{R}{R+r}\right)}{\left(\left(X + \dfrac{R}{R+r}\right)^2 + Y^2\right)^{3/2}} + \dfrac{\dfrac{m}{M+m} \left(X - \dfrac{r}{R+r}\right)}{\left(\left(X - \dfrac{r}{R+r}\right)^2 + Y^2\right)^{3/2}} \right] ,
\end{align}
where $\dfrac{M}{M+m}$ and $\dfrac{R}{R+r}$ are dimensionless parameters. There is a simple relation between the two given by the equation for the center of mass:
\begin{align}
M\vec{R} + m\vec{r} &= \vec{0} \\[0.5cm]
\Rightarrow \qquad \dfrac{m}{M+m} &= \dfrac{R}{R+r} = k \\[0.5cm]
\Leftrightarrow \qquad \dfrac{M}{M+m} &= \dfrac{r}{R+r} = 1-k
\end{align}
With that in mind we finally get
\begin{align}
\dot{P}_X = P_Y - \dfrac{(1-k)(X+k)}{((X+k)^2+Y^2)^{3/2}} - \dfrac{k(X-1+k))}{((X-1+k))^2+Y^2)^{3/2}}
\end{align}
by symmetry for $P_Y$ we get
\begin{align}
\dot{P}_Y = -P_X - \dfrac{(1-k)Y}{((X+k)^2+Y^2)^{3/2}} - \dfrac{k Y}{((X-1+k))^2+Y^2)^{3/2}}
\end{align}

The same procedure applies for $H$, which takes us from this:
\begin{align}
    H = \dfrac{p_x^2 + p_y^2}{2 m_s} + p_x\omega y - p_y\omega x - G m_s \left(\dfrac{M}{\sqrt{(x+R)^2+y^2}} + \dfrac{m}{\sqrt{(x-r)^2+y^2}}\right).
\end{align}

to this:

\begin{align}
    H = \frac{P_X^2 + P_Y^2}{2} + Y P_X - X P_Y - \frac{1 -k}{\sqrt{(X+k)^2 + Y^2 }} - \frac{k}{\sqrt{(X-1+k))^2 + Y^2}}
\end{align}


\section{Factor Graph}
Our SLAM is formulated as a bipartite factor graph with two types of nodes:
the variables (poses) which are elements of the set $\Theta$, and the
factors, which constrain the variables via the set of measurements  $\mathcal{Z}$. The
factor graph defines a probability $P(\Theta|\mathcal{Z})$ that
assumes its maximum a posteriori (MAP) value for the optimal variable set
$\Theta^*$, given the measurements:
\begin{equation}
\Theta^* = \argmax_\Theta P(\Theta|\mathcal{Z})\ .
\end{equation}
The set of variables $\mathcal{Z}$ contains:
\begin{itemize}
\item The camera poses $\ctrans{\mathrm{body}(\mathrm{cam}\ j)}{T}{\mathrm{cam}\ j}$, with respect to the bodies they are attached to.
\item The tag poses $\ctrans{\mathrm{body}(\mathrm{tag}\ k)}{T}{\mathrm{tag}\ k}$,  relative to their respective bodies.
\item The world poses $\ctrans{\mathrm{w}}{T}{\mathrm{body}\ l}$, of static bodies.
\item The time-dependent world poses $\ctrans{\mathrm{w}}{T}{\mathrm{body}\ m}(t)$, $t = 1\dots N_{\mathrm{t}}$ of dynamic bodies.
\end{itemize}
The likelihood is expressed \cite{kaess2011} as a product of factors $p^{(i)}$
that connects the variables with each other via measurements to form the desired graph structure:
\begin{equation}
  \label{eq:totalprob}
  P(\Theta|\mathcal{Z}) = \prod_ip^{(i)}(\Theta|\mathcal{Z})\ .
\end{equation}
To make the factors $p^{(i)}$ computationally tractable, we follow the standard
approach \cite{kaess2011} and model them as Gaussians:
\begin{equation}
  g(x;\mu,\Sigma) = \exp(-\frac{1}{2}||x\ominus\mu||_\Sigma^2)\ .
\end{equation}
Here, $x$ is the variable, $\mu$ the center of the Gaussian, and $\Sigma$ defines the Mahalanobis distance.
Note the use of the $\ominus$ operator, which reduces to straight subtraction for elements of a vector space,
but produces 6-dimensional Lie algebra coordinates when applied to elements on the SE(3) manifold:
\begin{equation}
  \label{eq:ominus}
  \ctrans{}{T}{A} \ominus \ctrans{}{T}{B} =
  \left[[\log(\mathrm{Rot}(\ctransi{}{T}{B}\ctrans{}{T}{A}))]_\vee^\top,\mathrm{Trans}(\ctransi{}{T}{B}\ctrans{}{T}{A})^\top\right]^\top\ .
\end{equation}
In (\ref{eq:ominus}) Rot() and Trans() refer to the rotational and translational part of the SE(3) transform,
respectively, log() is the matrix logarithm, and $\vee$ denotes the
{\em vee} map operator. Equipped with the definition of a Gaussian
on SE(3), we can now introduce the basic factors $p^{(i)}$ from Eq.\ (\ref{eq:totalprob}).

{\em Absolute Pose Prior}. This unary factor can be used to specify a prior pose
$\ctrans{}{T}{0}$ with noise $\Sigma$ for e.g. a tag or a camera:
\begin{equation}
  p_A(\ctrans{}{T}{}|\ctrans{}{T}{0},\Sigma) = g(\ctrans{}{T}{}; \ctrans{}{T}{0}, \Sigma)\ .
\end{equation}

{\em Relative Pose Prior}. With this binary factor, a known transform $\Delta \ctrans{}{T}{}$
between two pose variables can be specified, with noise $\Sigma$:
\begin{equation}
  p_R(\ctrans{}{T}{A}, \ctrans{}{T}{B}|\Delta\ctrans{}{T}{},\Sigma) = g(\ctransi{}{T}{B}\ctrans{}{T}{A}; \Delta\ctrans{}{T}{}, \Sigma)\ .
\end{equation}
If odometry body pose differences $\Delta \ctrans{}{T}{\mathrm{odom}}(t)$ with noise $\sigma$
are available from e.g. a VIO algorithm running
alongside TagSLAM, a relative pose prior of
$p_R(\ctrans{\mathrm{w}}{T}{\mathrm{body}}(t),
\ctrans{\mathrm{w}}{T}{\mathrm{body}}(t-1)|
\Delta \ctrans{}{T}{\mathrm{odom}}(t),\sigma)$ can be used to insert
the odometry updates into the pose graph.

{\em Tag Projection Factor}. The output of the tag detection
library is a list of tag IDs and the corresponding image
coordinates $\mv{u}_c$ (in units of pixels) of the corners
$c=1\dots 4$ of every tag. This gives rise to one quaternary
tag projection factor per tag:
\begin{equation}
  \begin{split}
    p_T(\ctrans{\mathrm{w}}{T}{\mathrm{body}},
    \ctrans{\mathrm{rig}}{T}{\mathrm{cam}},
    \ctrans{\mathrm{body}}{T}{\mathrm{tag}},
    \ctrans{\mathrm{w}}{T}{\mathrm{rig}}|\{\mv{u}_c\},\sigma_p) =\\
    \prod_{c=1\dots 4}g(\Pi(\ctrans{\mathrm{cam}}{T}{\mathrm{rig}}
    \ctrans{\mathrm{rig}}{T}{\mathrm{w}}
    \ctrans{\mathrm{w}}{T}{\mathrm{body}}
    \ctrans{\mathrm{body}}{T}{\mathrm{tag}}\mv{s}_c);
    \mv{u}_c,\sigma_p)\ .
  \end{split}
\end{equation}
Here, $\mv{s}$ refers to the corner coordinates in the tag reference frame,
i.e. $\mv{s}_1 = [-l/2, -l/2, 0]^\top$,
$\mv{s}_2 = [l/2, -l/2, 0]^\top$,
$\mv{s}_3 = [l/2,  l/2, 0]^\top$,
$\mv{s}_4 = [-l/2, l/2, 0]^\top$
for a tag of side length $l$. A sequence of transforms expresses $\mv{s}$ in
camera coordinates, after which the function $\Pi$ projects \cite{ma2003} the point onto the
sensor plane and converts it to pixel coordinates. The noise parameter $\sigma_p$
is a diagonal matrix that reflects the accuracy of the tag library's corner detector,
which is usually assumed to be about one pixel.

We can visualize the factor structure of Eq.\ (\ref{eq:totalprob}) by means of
a graph as shown in Fig.\ \ref{fig:sample_graph} for the scene from
Fig.\ \ref{fig:scene_with_block}. In Fig.\ \ref{fig:sample_graph},
black squares represent factors, whereas circles denote pose variables
to be optimized.
The prior factors $p_A$ constrain
the static poses  $\ctrans{l}{T}{2}$, $\ctrans{\mathrm{w}}{T}{l}$,
$\ctrans{\mathrm{r}}{T}{\mathrm{c}}$, and
$\ctrans{\mathrm{b}}{T}{105}$. A tag projection factors $p_T$ arising
from an observation of Tag 2 determines the dynamic rig pose
$\ctrans{\mathrm{w}}{T}{\mathrm{r}}(t)$, whereas an observation of Tag
105 likewise yields the block pose
$\ctrans{\mathrm{w}}{T}{\mathrm{b}}(t)$. Assuming that odometry for
the rig is provided by some external algorithm, there is a relative
pose factor $p_R$ connecting the rig poses for $t$ and $t+1$. Two more
tag observations at $t+1$ generate additional factors that further constrain rig and block poses at $t+1$.

\begin{figure}[ht]
  \vspace{0.25cm}
  \newcommand{\relfacgraphsize}{0.75}
  \begin{center}
    \begin{tabular}{cc}
      \resizebox{\relfacgraphsize\columnwidth}{!}{
      \input{sections/sample_graph.tex}
      }
    \end{tabular}
  \end{center}
  \caption{Factor graph of Eq.\ \ref{eq:totalprob} for the scene shown in Fig.\ \ref{fig:scene_with_block}}
  \label{fig:sample_graph}
\end{figure}


documentclass[../PHYS306Notes.tex]{subfiles}

\begin{document}
\subsection{Worksheet - Shortest Distance and Variational Principle}
\begin{p}
We wish to find the shortest path between two points $(x_1,y_1)$ and $(x_2,y_2)$ in the plane. First, write out the distance between these two points in terms of a general function $y(x)$ connecting points $(x_1,y_1)$ and $(x_2,y_2)$, and possibly it’s derivative $y'(x)$, in terms of an independent variable $x$.
\begin{center}
    \includegraphics[scale=0.6]{Lecture-1/W1-img1.png}
\end{center}
\end{p}
\begin{s}
We can define an infinitesimal path element of the trajectory from $(x_1, y_1)$ to $(x_2, y_2)$ as:
\[ ds = \sqrt{dx^2 + dy^2}\]
We can use the identity:
\[ dy \equiv \dod{y}{x}dx = y'(x)dx\]
Substituting this into the path element equation above, we obtain:
\[ds = \sqrt{dx^2 + \left(y'(x)dx\right)^2}\]
We may integrate the path element from the initial state to the final state to obtain the length of the path:
\[L(y(x), y'(x)) = \int_{t_1}^{t_2} ds = \int_{x_1}^{x_2} \sqrt{dx^2 + \left(y'(x)dx\right)^2} = \int_{x_1}^{x_2} \sqrt{1 + y'(x)^2}dx\]
Notice here we have the functional $L[y(x), y'(x)]$ (here only depends on $y'(x)$).

\end{s}

\begin{p}
If light were travelling between points $(x_1,y_1)$ and $(x_2,y_2)$, we would expect it to follow a straight line, but if the index of refraction $n = n(x,y)$ is not constant, the speed of light in the medium is $v = c/n$ and the path is not a straight line but follows the trajectory of minimum time (Fermat, 1662).Find an integral expression for the time taken by a trajectory $y(x)$ through a medium $n(x,y)$.
\end{p}
\begin{s}
The infinitesimal time element $dt$ to travel a distance $ds$ is given by:
\[dt = \frac{ds}{v} = \frac{n(x,y) ds}{c} = \frac{1}{c} n(x,y)\sqrt{1 + y'(x)^2}dx \]
Where in the second inequality we use the identity $v = c/n$. Again we can integrate from an initial time to a final time to obtain the total time the light takes to travel along the path:
\[T = \int_{t_1}^{t_2} dt = \int_{x_1}^{x_2} \frac{1}{c} n(x,y)\sqrt{1 + y'(x)^2}dx = \frac{1}{c}\int_{x_1}^{x_2} n(x,y) \sqrt{1+y'(x)^2}dx\]
Where again we have a functional $T[y(x), y'(x)]$.
\end{s}

\begin{p}
In our variational treatment, if the “wrong” or “varied” curves from the minimum pass through the points 1 and 2, write the condition on the deviations from the correct curve, $\eta(x)$.
\end{p}
\begin{s}
$\eta(x_1) = \eta(x_2) = 0$ as the endpoints of the wrong path must match that of the correct path.
\end{s}

\begin{p}
If $S(\alpha)=\int_{x_{1}}^{x_{2}} f\left(y+\alpha \eta, y^{\prime}+\alpha \eta^{\prime}, x\right) d x,$ use the chain rule to write $d S(\alpha) / d \alpha$ in terms of an integral of a function that contains derivatives on $y$ and $y^{\prime},$ and $\eta$ and $\eta^{\prime}$.
\end{p}
\begin{s}
As the integral is with respect to x, we may interchange the order of integration and differentiation:
\[\left. \dod{S(\alpha)}{\alpha}\right|_{\alpha = 0} = \left.\dod{}{\alpha}\int_{x_{1}}^{x_{2}} f\left(y+\alpha \eta, y^{\prime}+\alpha \eta^{\prime}, x\right) dx\right|_{\alpha = 0} = \int_{x_1}^{x_2} \left(\left.\dpd{}{\alpha}f\left(y+\alpha \eta, y^{\prime}+\alpha \eta^{\prime}, x\right)\right|_{\alpha = 0}\right) dx \]
Then by the chain rule, we have:
\[\left.\dod{S(\alpha)}{\alpha}\right|_{\alpha = 0} = \int_{x_{1}}^{x_{2}} \left(\eta\dpd{f}{y} + \eta'\dpd{f}{y'}\right)dx = \int_{x_1}^{x_2} \eta\dpd{f}{y} dx + \int_{x_1}^{x_2} \eta'\dpd{f}{y'}dx\]
\end{s}

\begin{p}
Rewrite the $2^{\text {nd }}$ term $\int_{x 1}^{x 2} \eta^{\prime} \frac{\partial f}{\partial y^{\prime}} d x$ by integrating by parts. Note $\int v d u=[u v]-\int u d v$ is equivalent to $\int u^{\prime} v d x=[u v]-\int u v^{\prime} d x$. After simplifying, write the integral expression for $\frac{\partial S}{\partial \alpha}=0 .$ l.e. $\frac{\partial S}{\partial \alpha}= \int_{x 1}^{x 2} \eta(x)[\ldots] d x=0 .$ This condition ensures that $S(\alpha)$ has a minimum at $\alpha=0$ and $y$ is the curve that extremizes S.
\end{p}
\begin{s}
Carrying out integration by parts on the second term, we have:
\[\left. \dod{S(\alpha)}{\alpha}\right|_{\alpha = 0} = \int_{x_1}^{x_2} \eta \dpd{f}{y} dx + \left. \eta\dpd{f}{y'}\right|_{x_1}^{x_2} - \int_{x_1}^{x_2}\eta\left(\dod{}{x}\dpd{f}{y'}\right)dx \]
From problem 3, we know that $\eta(x_1) = \eta(x_2) = 0$ and so the second term evaluates to zero, leaving us with:
\[\left. \dod{S(\alpha)}{\alpha}\right|_{\alpha = 0} = \int_{x_1}^{x_2} \left(\eta\dpd{f}{y} - \eta\dod{}{x}\dpd{f}{y'}\right)dx\]
and we set this to zero:
\[\left. \dod{S(\alpha)}{\alpha}\right|_{\alpha = 0} = \int_{x_1}^{x_2}\eta\left(\dpd{f}{y} - \dod{}{x}\dpd{f}{y'}\right)dx = 0\]
We require that this holds for \textbf{all} possible deviations $\eta(x)$. The only way this could hold is if the term in brackets is zero; this is exactly the Euler-Lagrange equation!
\end{s}

\begin{p}
Write the general form of the Euler-Lagrange Equation. What is the function $f$ for the distance between two points? What does this say about $\pd{f}{y}$ in the E-L equation?
\end{p}
\begin{s}
The general form of the Euler-Lagrange equation is given by:
\[\dpd{f}{y}  - \dod{}{x}\dpd{f}{y'} = 0\]
The function for the distance between two points is $f = \sqrt{1+ y'^2}$ as derived above. Hence, $\dpd{f}{y} = 0$, and therefore by the Euler Lagrange equation:
\[\dpd{f}{y'} = \text{constant}\]
\end{s}

\begin{p}
Solve the above equation for $y'^2$ and hence $y'$ and $y$.
\end{p}
\begin{s}
We have that:
\[\dpd{f}{y'} = \frac{y'}{\sqrt{1+y'^2}} = \text{Constant}\]
Therefore:
\[y'^2 = c^2(1+y'^2)\]
Or alternatively:
\[y'^2 = \frac{c^2}{1-c^2} = c' \implies y'(x) = \sqrt{c'} = m\]
So integrating, we have that:
\[y(x) = mx + b\]
as we knew already!
\end{s}
\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% formulation.tex
% Formulation to allocation tech report in master.tex
% This section: David Albrecht
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Formulation}
\label{sec:form}

In this section we show how our allocation problem can be formulated
as a Convex Programming problem. To do this we will need to make the
following assumptions:
\begin{enumerate}
\item The object we are searching for is stationary during the time of
the search.
\item The probability of detecting the object in a given area is
proportional to the time searching, $\delta t$, when $\delta t$ is
small.
\item The probability of detecting the object in a given region for a
given resource is independent of the probability of detecting the
object in that region with any other resource.
\item The probability of detecting the object at time $t=0$ is zero.
\end{enumerate}

Let $D_{a,r}(t)$ be the probability of detecting the object in an area
$a$ using a resource $r$ by time $t$. Then, by (2), for small $\delta
t$ we have:
$$D_{a,r}(t + \delta t) = D_{a,r}(t) + w_{a,r}\delta t(1-D_{a,r}(t)),$$
where $w_{a,r}$ is a measure of the effectiveness of resource $r$ in
area $a$. Letting $\delta t\to 0$, we obtain
$$D_{a,r}^\prime(t) = w_{a,r}(1-D_{a,r}(t)),\quad\mbox{ and by (4) }\quad
D_{a,r}(0) = 0.$$
Solving this equation we find:
$$D_{a,r}(t) = 1-\exp(-w_{a,r}t), \quad t\ge 0.$$

Now, let $\bar{D}$ be the probability of \emph{not} detecting the
object, $p_a$ the probability the object is in area $a$, $T_r$ the
total amount of time available for searching with resource $r$, and
$t_{a,r}$ the time spent searching in area $a$ with resource $r$. Then
by (3),
\begin{eqnarray*}
\bar{D} &=& \sum_{a = 1}^A p_a\prod_{r = 1}^R \bar{D}_{a,r}(t_{a,r})\\
&=& \sum_{a = 1}^A p_a \exp\left(-\sum_{r = 1}^R w_{a,r}t_{a,r}\right),
\end{eqnarray*}
where
$$
\sum_{a = 1}^A t_{a,r} \le T_r, \qquad r = 1,\ldots,R.
$$
\newpage

Therefore, since the problem of maximize the probability of detection
is the same as the problem of minimizing the probability of \emph{not}
detecting we obtain the following Convex Programming {\bf (CP)} problem

$$
\min_{t_{a,r}\ge 0}\quad \sum_{a = 1}^A p_a \exp\left(-\sum_{r = 1}^R
w_{a,r}t_{a,r}\right),
$$
such that
$$
\sum_{a = 1}^A t_{a,r} \le T_r, \qquad r = 1,\ldots,R.
$$

The following theorem from Nonlinear Programming gives conditions for
solving this problem.

\begin{theorem} Suppose $f$, $g_1\ldots,g_m$ are differentiable convex
functions on ${\cal R}^n$, and there exists $x^*$ and $\lambda^{*}$
which satisfy:
\begin{eqnarray}
\frac{\partial f}{\partial x_i}(x^{*}) + \sum_{j=1}^m \lambda^*_j\frac{\partial
g_j}{\partial x_i}(x^{*}) &=& 0, \\
\lambda^{*}_j g_j(x^{*}) &=& 0, \\
g_j(x^{*}) &\le& 0, \\
\lambda^{*}_j &\ge& 0,
\end{eqnarray}
where $i = 1,\ldots, n$ and $j = 1,\ldots, m$. Then $x^{*}$ is a solution of
$$
\min_{x} f(x)\qquad\mbox{s.t.}\qquad g_j(x)\le 0,\quad j=1,\ldots,m.
$$

\end{theorem}

The conditions (1) - (4) in the above theorem are known as the Karush
Kuhn Tucker (KKT) conditions and for the {\bf CP} problem these conditions
are as follows:
\begin{eqnarray}
-p_aw_{a,r}\exp(-\sum_{r=1}^R w_{a,r}t_{a,r}) + \beta_r - \alpha_{a,r} &=& 0\\
\beta_r(\sum_{a = 1}^A t_{a,r} - T_r) &=& 0\\
\sum_{a = 1}^A t_{a,r} &\le& T_r\\
\alpha_{a,r}t_{a,r} &=& 0, \\
\alpha_{a,r} \ge 0, \qquad t_{a,r} \ge 0, \qquad \beta_r &\ge& 0,
\end{eqnarray}
where $a = 1,\ldots,A$ and $r =1,\ldots,R$.

Therefore, $\{t_{a,r}\}$ is a solution of the problem {\bf CP} if
there exists $\{t_{a,r}, \beta_r, \alpha_{a,r}\}$ which satisfy
conditions (5) - (9).

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "master"
%%% End:


\chapter{Basics and foundations}
\section{Complex numbers}
\subsection{Operations}

\begin{gather}
\frac{z_1}{z_2}
= \frac{z_1\overline z_2}{z_2\overline z_2}
= \frac{z_1\overline z_2}{|z_2|^2},\\
\frac{1}{z} = \frac{\overline z}{z\overline z}
= \frac{\overline z}{|z|^2}.
\end{gather}

\subsection{Absolut value}
For all $z_1,z_2\in\mathbb C$:
\begin{gather}
|z_1z_2| = |z_1|\,|z_2|,\\
z_2\ne 0\implies \Big|\frac{z_1}{z_2}\Big|
= \frac{|z_1|}{|z_2|},\\
z\,\overline z = |z|^2.
\end{gather}

\subsection{Conjugation}
For all $z_1,z_2\in\mathbb C$:
\begin{gather}
\overline{z_1+z_2} = \overline z_1+\overline z_2,\qquad
\overline{z_1-z_2} = \overline z_1-\overline z_2,\\
\overline{z_1 z_2} = \overline z_1\,\overline z_2,\qquad
z_2\ne 0 \implies \overline{\Big(\frac{z_1}{z_2}\Big)}
= \frac{\overline z_1}{\overline z_2},\\
\overline{\overline z}=z,\qquad
|\overline{z}| = |z|,\qquad
z\,\overline z = |z|^2,\\
\operatorname{Re}(z) = \frac{z+\overline z}{2},\qquad
\operatorname{Im}(z) = \frac{z-\overline z}{2\ui},\\
\overline{\cos(z)} = \cos(\overline z),\qquad
\overline{\sin(z)} = \sin(\overline z),\\
\overline{\exp(z)} = \exp(\overline z).
\end{gather}

\begin{table*}[t]
\caption{Operations}
\bgroup
\def\arraystretch{1.4}
\begin{tabular}{|l|r|l|l|}
\hline
  \thbf{Name}
& \thbf{Operation}
& \thbf{Polar form}
& \thbf{Cartesian form}\\
\hline
  Identity
& $z$ & $=r\ee^{\ui\varphi}$
& $= a+b\ui$\\
\hline
  Addition
& $z_1+z_2$ &
& $= (a_1+a_2)+(b_1+b_2)\ui$\\
\hline
  Subtraction
& $z_1-z_2$ &
& $= (a_1-a_2)+(b_1-b_2)\ui$\\
\hline
  Multiplication
& $z_1 z_2$
& $= r_1 r_2 \ee^{\ui(\varphi_1+\varphi_2)}$
& $= (a_1 a_2 - b_1 b_2)+(a_1 b_2+a_2 b_1)\ui$\\
\hline
  Division
& $\displaystyle\frac{z_1}{z_2}$
& $\displaystyle =\frac{r_1}{r_2}\ee^{\ui(\varphi_1-\varphi_2)}$
& $\displaystyle =\frac{a_1 a_2 + b_1 b_2}{a_2^2+b_2^2}
   + \frac{a_2 b_1 - a_1 b_2}{a_2^2+b_2^2}\ui$\\
\hline
  Reciprocal
& $\displaystyle\frac{1}{z}$
& $\displaystyle =\frac{1}{r}\ee^{-\ui\varphi}$
& $\displaystyle =\frac{a}{a^2+b^2}-\frac{b}{a^2+b^2}\ui$\\
\hline
  Real part
& $\operatorname{Re}(z)$
& $=\cos\varphi$
& $=a$\\
\hline
  Imaginary part
& $\operatorname{Im}(z)$
& $=\sin\varphi$
& $=b$\\
\hline
  Conjugation
& $\overline{z}$
& $=r\ee^{-\varphi\ui}$
& $=a-b\ui$\\
\hline
  Absolut value
& $|z|$
& $=r$
& $=\sqrt{a^2+b^2}$\\
\hline
  Argument
& $\arg(z)$
& $=\varphi$
& $\displaystyle =s(b)\arccos\Big(\frac{a}{r}\Big)$\\
\hline
\end{tabular}
\egroup\\
\\
$s(b):=\begin{cases}
+1 & \text{if}\;b\ge 0,\\
-1 & \text{if}\;b<0
\end{cases}$
\end{table*}

\section{Logic}
\subsection{Propositional logic}
\subsubsection{Boolean algebra}
\begin{table*}[t]
\caption{Boolean algebra}
\begin{tabular}{l|l|l}
\thbf{Disjunction} & \thbf{Conjunction} &\\
  $A\lor A \Leftrightarrow A$
& $A\land A \Leftrightarrow A$
& Laws of idempotence\\
  $A\lor 0 \Leftrightarrow A$
& $A\land 1 \Leftrightarrow A$
& Laws of neutrality\\
  $A\lor 1 \Leftrightarrow 1$
& $A\land 0 = 0$
& Laws of annihilation\\
  $A\lor \overline A \Leftrightarrow 1$
& $A\land \overline A \Leftrightarrow 0$
& Laws of complementation\\
\noalign{\vspace{1em}}
  $A\lor B \Leftrightarrow B\lor A$
& $A\land B \Leftrightarrow B\land A$
& Laws of commutativity\\
  $(A\lor B)\lor C \Leftrightarrow A\lor (B\lor C)$
& $(A\land B)\land C \Leftrightarrow A\land (B\land C)$
& Laws of associativity\\
  $\overline{A\lor B} \Leftrightarrow \overline A\land\overline B$
& $\overline{A\land B} \Leftrightarrow \overline A\lor\overline B$
& De Morgan's laws\\
  $A\lor (A\land B) \Leftrightarrow A$
& $A\land (A\lor B) \Leftrightarrow A$
& Laws of absorption\\
\end{tabular}
\end{table*}

\noindent
\strong{Laws of distributivity}:
\begin{gather}
A\lor (B\land C) = (A\lor B)\land (A\lor C),\\
A\land (B\lor C) = (A\land B)\lor (A\land C).
\end{gather}

\subsubsection{Functions in two arguments}
There are 16 boolean functions in two arguments.

\begin{tabular}{r|l}
\textbf{\texttt{AB}} & \thbf{value}\\
\texttt{00} & \texttt{a}\\
\texttt{01} & \texttt{b}\\
\texttt{10} & \texttt{c}\\
\texttt{11} & \texttt{d}
\end{tabular}

\begin{tabular}{r|l|l|l}
\thbf{No.}& \textbf{\texttt{dcba}} & \thbf{Function} & \thbf{Name}\\
 0 & \texttt{0000} & 0 & Contradiction\\
 1 & \texttt{0001} & $\overline{A\lor B}$ & NOR\\
 2 & \texttt{0010} & $\overline{B\Rightarrow A}$\\
 3 & \texttt{0011} & $\overline A$\\
 4 & \texttt{0100} & $\overline{A\Rightarrow B}$\\
 5 & \texttt{0101} & $\overline{B}$\\
 6 & \texttt{0110} & $A\oplus B$ & Contravalence\\
 7 & \texttt{0111} & $\overline{A\land B}$ & NAND\\
 8 & \texttt{1000} & $A\land B$ & Conjunction\\
 9 & \texttt{1001} & $A\Leftrightarrow B$ & Equivalence\\
10 & \texttt{1010} & $B$ & Projection\\
11 & \texttt{1011} & $A\Rightarrow B$ & Implication\\
12 & \texttt{1100} & $A$ & Projection\\
13 & \texttt{1101} & $B\Rightarrow A$ & Implication\\
14 & \texttt{1110} & $A\lor B$ & Disjunction\\
15 & \texttt{1111} & $1$ & Tautology
\end{tabular}

\subsection{Predicate logic}
\subsubsection{Basic laws}
Negation (De Morgan's laws):
\begin{gather}
\overline{\forall x[P(x)]}\iff \exists x[\overline{P(x)}],\\
\overline{\exists x[P(x)]}\iff \forall x[\overline{P(x)}].
\end{gather}
Generalized laws of distributivity:
\begin{gather}
P\lor\forall x[Q(x)] \iff \forall x[P\lor Q(x)],\\
P\land\exists x[Q(x)] \iff \exists x[P\land Q(x)].
\end{gather}
Generalized laws of idempotence:
\begin{gather}
\begin{split}
\exists x{\in}M\,[P] & \iff
(M\ne\{\})\land P\\
& \iff\begin{cases}
P & \text{if}\; M\ne\{\},\\
0 & \text{if}\; M=\{\}.
\end{cases}
\end{split}\\
\begin{split}
\forall x{\in}M\,[P]& \iff
(M=\{\})\lor P\\
&\iff\begin{cases}
P & \text{if}\; M\ne\{\},\\
1 & \text{if}\; M=\{\}.
\end{cases}
\end{split}
\end{gather}
Equivalences:
\begin{gather}
\hspace{-2em}\forall x\forall y[P(x,y)] \iff \forall y\forall x[P(x,y)],\\
\hspace{-2em}\exists x\exists y[P(x,y)] \iff \exists y\exists x[P(x,y)],\\
\hspace{-2em}\forall x[P(x)\land Q(x)] \iff \forall x[P(x)]\land\forall x[Q(x)],\\
\hspace{-2em}\exists x[P(x)\lor Q(x)] \iff \forall x[P(x)]\lor\forall x[Q(x)],\\
\hspace{-2em}\forall x[P(x)\Rightarrow Q] \iff \exists x[P(x)]\Rightarrow Q,\\
\hspace{-2em}\forall x[P\Rightarrow Q(x)] \iff P\Rightarrow\forall x[Q(x)],\\
\hspace{-2em}\exists x[P(x)\Rightarrow Q(x)]
  \iff\forall x[P(x)]\Rightarrow\exists x[Q(x)].
\end{gather}
Implications:
\begin{gather}
\hspace{-2em}\exists x\forall y[P(x,y)]\implies \forall y\exists x[P(x,y)],\\
\hspace{-2em}\forall x[P(x)]\lor\forall x[Q(x)]\implies\forall x[P(x)\lor Q(x)],\\
\hspace{-2em}\exists x[P(x)\land Q(x)]\implies
  \exists x[P(x)]\land \exists x[Q(x)],\\
\hspace{-2em}\forall x[P(x)\Rightarrow Q(x)]\implies
  (\forall x[P(x)]\Rightarrow\forall x[Q(x)]),\\
\hspace{-2em}\forall x[P(x)\Leftrightarrow Q(x)]\implies
  (\forall x[P(x)]\Leftrightarrow\forall x[Q(x)]).
\end{gather}

\subsubsection{Finite sets}
Let $M=\{x_1,\ldots,x_n\}$. One has:
\begin{gather}
\forall x{\in}M\,[P(x)]\iff P(x_1)\land\ldots\land P(x_n),\\
\exists x{\in}M\,[P(x)]\iff P(x_1)\lor\ldots\lor P(x_n).
\end{gather}

\subsubsection{Restricted quantification}
\begin{gather}
\begin{split}
& \forall x{\in}M\,[P(x)]\;:\Longleftrightarrow\;\forall x[x\notin M\lor P(x)]\\
& \quad\iff\forall x[x\in M\Rightarrow P(x)],
\end{split}\\
\exists x{\in}M\,[P(x)]\;:\Longleftrightarrow\;\exists x[x\in M\land P(x)],\\
\forall x{\in}M{\setminus}N\,[P(x)]\iff \forall x[x\notin N\Rightarrow P(x)].
\end{gather}

\subsubsection{Product sets as\\
domains of discourse}
\begin{gather}
\forall(x,y)\,[P(x,y)]\iff \forall x\forall y[P(x,y)],\\
\exists(x,y)\,[P(x,y)]\iff \exists x\exists y[P(x,y)].
\end{gather}
By analogy:
\begin{gather}
\forall(x,y,z)\,\iff \forall x\forall y\forall z,\\
\exists(x,y,z)\,\iff \exists x\exists y\exists z
\end{gather}
etc.

\subsubsection{Alternative representation}
Let $P\colon G\to\{0,1\}$ and $M\subseteq G$.
Let $P(M)$ be the image of $M$ under $P$. One has
\begin{equation}
\begin{split}
&\forall x{\in}M\,[P(x)] \iff P(M)=\{1\}\\
& \iff M\subseteq\{x{\in}G\mid P(x)\}
\end{split}
\end{equation}
and
\begin{equation}
\begin{split}
& \exists x{\in}M\,[P(x)] \iff \{1\}\subseteq P(M)\\
& \iff M\cap\{x{\in}G\mid P(x)\}\ne\{\}.
\end{split}
\end{equation}

\subsubsection{Uniqueness}
Quantifier of unique existence:
\begin{equation}
\begin{split}
&\exists!x\,[P(x)]\\
&:\Longleftrightarrow\; \exists x\,[P(x)\land \forall y\,[P(y)\Rightarrow x=y]]\\
&\iff \exists x\,[P(x)]\land \forall x\forall y[P(x)\land P(y)\Rightarrow x=y].
\end{split}
\end{equation}

\section{Set theory}
\subsection{Definitions}
Subset relation:
\begin{equation}
A\subseteq B\;:\Longleftrightarrow\; \forall x\,[x\in A\implies x\in B].
\end{equation}
Equality:
\begin{equation}
A=B\;:\Longleftrightarrow\; \forall x\,[x\in A\iff x\in B].
\end{equation}
Union:
\begin{equation}
A\cup B:=\{x\mid x\in A\lor x\in B\}.
\end{equation}
Intersection:
\begin{equation}
A\cap B:=\{x\mid x\in A\land x\in B\}.
\end{equation}
Difference set:
\begin{equation}
A\setminus B:=\{x\mid x\in A\land x\not\in B\}.
\end{equation}
Symmetric difference:
\begin{equation}
A\triangle B:=\{x\mid x\in A\oplus x\in B\}.
\end{equation}

\subsection{Boolean algebra}
\begin{table*}[t]
\caption{Boolean algebra}
\begin{tabular}{l|l|l}
\thbf{Union} & \thbf{Intersection} &\\
  $A\cup A = A$
& $A\cap A = A$
& Laws of idempotence\\
  $A\cup \{\} = A$
& $A\cap G = A$
& Laws of neutrality\\
  $A\cup G = G$
& $A\cap \{\} = \{\}$
& Laws of annihilation\\
  $A\cup \overline A = G$
& $A\cap \overline A = \{\}$
& Laws of complementation\\
\noalign{\vspace{1em}}
  $A\cup B = B\cup A$
& $A\cap B = B\cap A$
& Laws of commutativity\\
  $(A\cup B)\cup C = A\cup (B\cup C)$
& $(A\cap B)\cap C = A\cap (B\cap C)$
& Laws of associativity\\
  $\overline{A\cup B} = \overline A\cap\overline B$
& $\overline{A\cap B} = \overline A\cup\overline B$
& De Morgan's laws\\
  $A\cup (A\cap B) = A$
& $A\cap (A\cup B) = A$
& Laws of absorption\\
\end{tabular}\\
\\
$G$: Universe
\end{table*}

\noindent
\strong{Laws of distributivity}:
\begin{gather}
M\cup (A\cap B) = (M\cup A)\cap (M\cup B),\\
M\cap (A\cup B) = (M\cap A)\cup (M\cap B).
\end{gather}

\subsection{Subset relation}
Decomposition of equality:
\begin{equation}
A=B \iff A\subseteq B \land B\subseteq A.
\end{equation}
Pharaphrasing of subset relations:
\begin{equation}
\begin{split}
A\subseteq B &\iff A\cap B=A\\
& \iff A\cup B=B\\
& \iff A\setminus B=\{\}.
\end{split}
\end{equation}
Law of contraposition:
\begin{equation}
A\subseteq B = \overline B\subseteq \overline A.
\end{equation}

\subsection{Inductive sets}
Set theoretical model of the natural numbers:
\begin{equation}
\begin{split}
& 0:=\{\},\quad 1:=\{0\},\quad 2:=\{0,1\},\\
& 3:=\{0,1,2\},\quad \text{usw.}
\end{split}
\end{equation}
Successor function:
\begin{equation}
x' := x\cup\{x\}.
\end{equation}
Proof by induction: For a predicate $A(n)$ with $n\in\mathbb N$
one has:
\begin{equation}
\begin{split}
& A(n_0)\land \forall n\ge n_0\,[A(n)\Rightarrow A(n+1)]\\
& \implies \forall n\ge n_0\,[A(n)].
\end{split}
\end{equation}


% !TEX root = index.tex

\section{Final Remarks}

\setlength{\epigraphwidth}{0.6\textwidth}
\epigraph{\it Don't just read it; fight it!\\
Ask your own questions, look for your own examples, discover your own proofs. Is the hypothesis necessary? Is the converse true? What happens in the classical special case? What about the degenerate cases? Where does the proof use the hypothesis? }{Paul Halmos}

On the last day, we will just tie up some loose ends.

\subsection{Rank-Nullity theorem}
The following is first of the many simple and yet remarkably useful theorems in linear algebra.
\begin{theorem}[Rank-Nullity theorem]
  \label{theorem:rankNullityTheorem}
  Let $A$ be a matrix of size $\ell \times k$ so that $A$ defines a linear transformation $A: \bbr^k \rightarrow \bbr^\ell $. Then,
  \begin{align*}
    \nul A + \rank A =  k.
  \end{align*}
\end{theorem}

\begin{qbox}
  Rank-nullity can be used to detect isomorphisms.
  Let $A$ be a square matrix of size $k \times k$ representing a linear operator $A:\bbr^k \rightarrow \bbr^k$.\\

  Using the Rank-Nullity theorem, show that the following are equivalent
      \begin{enumerate}
        \item $A$ is an isomorphism,
        \item $\nul A = 0$,
        \item $\rank A = k$.\\
      \end{enumerate}
\end{qbox}

\begin{remark}
  There is one more criterion for detecting isomorphisms

    \begin{enumerate}
      \item[4.] $\det A \neq 0$,
    \end{enumerate}

  where $\det A$ is the ``determinant of $A$''. Determinant is the signed volume of the $k$-dimensional cube formed by the columns of $A$, and can be computed very efficiently using Gaussian elimination.
\end{remark}

\begin{qbox}
  Let $a$, $b$, $c$ be scalars. Assume that at least one of $a$, $b$, $c$ is non-zero.
  \begin{enumerate}
    \item Show that the matrix $A = \begin{bmatrix} a & b & c \end{bmatrix}$ has rank 1.
    \item Using the Rank-Nullity Theorem, conclude that $ax + by + cz = 0$ defines a plane in $\bbr^3$.
  \end{enumerate}
  The same proof can be used to show that a single non-trivial linear equation always defines an $n-1$ dimensional hyperplane in $\bbr^n$.
\end{qbox}

\begin{proof}[Proof of the Rank-Nullity theorem]
  Let $A: \bbr^k \rightarrow \bbr^\ell$. Let $$n = \nul A.$$
  To prove the theorem, we will construct a basis of $\im A$ of size $k - n$.

  Let $$\calb' = \set{\vec{v}_1, \dots, \vec{v}_n}$$ be a basis of $\ker A$.

  \textbf{Case 1:} $n = k$. In this case, we have $\ker A = \bbr^k$. By the definition of kernel, $A \vec{v} = 0$ for all $\vec{v} \in \bbr^k$. Hence, $\im A = \{ \vec{0} \}$ which implies that $\rank A = 0$.

  \textbf{Case 2:} $n < k$. In this case, by repeatedly adding linearly independent vectors as in the proof of Theorem \ref{theorem:existenceOfBasis}, we can extend $\calb'$ to a basis $\calb$ of $\bbr^k$.
  \begin{align*}
    \calb=\{\underbrace{\vec{v}_1, \dots, \vec{v}_n}_{\calb'}, \vec{w}_{n+1}, \dots, \vec{w}_k\}.
  \end{align*}

  \textbf{Claim:} The set $\calb''  = \{A\vec{w}_{n+1}, \dots, A\vec{w}_k \}$ is a basis for $\im A$.

  We need to show that $\spn(\calb'') = \im A$ and $\calb''$ is linearly independent.
  To prove the first part, note that any vector in $V$ can be written as
  \begin{align}
    \tag{*}
    \label{eq:eq1}
    \vec{v} = c_1 \vec{v}_1 + \dots + c_n \vec{v}_n + c_{n+1} \vec{w}_{n+1} + \dots + c_k \vec{w}_k
  \end{align}
  for some scalars $c_1, \dots, c_k$.
  \begin{qbox}
    Apply $A$ to both sides of \eqref{eq:eq1} to conclude that
      \begin{align*}
        \im A = \spn(A\vec{w}_{n+1}, \dots, A\vec{w}_k)
      \end{align*}
  \end{qbox}
  It remains to show that $\calb''$ is linearly independent. Suppose there are scalars $d_{n+1}, \dots, d_k$ such that
  \begin{align*}
      d_{n+1} A\vec{w}_{n+1} + \dots + d_kA \vec{w}_k &= 0 \\
      \implies A(d_{n+1} \vec{w}_{n+1} + \dots + d_k \vec{w}_k) &= 0 && \mbox{by linearity} \\
      \implies d_{n+1} \vec{w}_{n+1} + \dots + d_k \vec{w}_k &\in \ker A && \mbox{by definition of }\ker \\
      \implies d_{n+1} \vec{w}_{n+1} + \dots + d_k \vec{w}_k &\in \spn({\vec{v}_1, \dots, \vec{v}_n}) \\
      \implies d_{n+1} \vec{w}_{n+1} + \dots + d_k \vec{w}_k &= d_1\vec{v}_1 + \dots + d_n \vec{v}_n && \mbox{by definition of }\spn
  \end{align*}
  for some scalars $d_1, \dots, d_n$.

  But the vectors $\{\vec{v}_1, \dots, \vec{v}_n, \vec{w}_{n+1}, \dots, \vec{w}_k\}$ form a basis for $V$ and hence are linearly independent. Hence, the only possible $d_i$'s satisfying the last equation are $d_1 = \dots = d_k = 0$ which proves the linear independence of $\calb''$.
\end{proof}

\begin{remark}
  Rank-Nullity theorem is closely related to theorems from other areas of mathematics. For example, vector spaces are abelian groups under the vector addition operation and subspaces are the same as (normal) subgroups. The first isomorphism theorem in group theory then gives us $$\bbr^k / \ker A \cong \im A.$$
  The Rank-Nullity theorem is then a version of Lagrange's theorem.
  \begin{qbox}[If you know what an exact sequence means]
    Using the Rank-Nullity theorem, show that if
    \begin{align*}
      0 \rightarrow V_0 \rightarrow V_1 \rightarrow \dots \rightarrow V_n \rightarrow 0
    \end{align*}
    is an exact sequence of vector spaces then
    \begin{align*}
      \dim V_0 - \dim V_1 + \dots + (-1)^i \dim V_i + \dots + (-1)^n \dim V_n = 0
    \end{align*}
    This is saying that the Euler characteristic of an exact sequence is trivial.
  \end{qbox}

\end{remark}

\subsection{Composition of linear transformations}
We saw in Section \ref{section:linearTransformation} that composition of linear transformations is itself a linear transformation. For Euclidean spaces, this gives rise to products of matrices.

\begin{equation*}
  \begin{tikzcd}
    \bbr^k \ar[rr,"B"] \ar[rrrr,"AB"', bend right] && \bbr^\ell \ar[rr,"A"] && \bbr^m
  \end{tikzcd}
\end{equation*}

By carefully expanding out $A(B( \vec{v}))$ one can find the general formula for $AB$, this is quite tedious to derive but easy to use. You should look it up.

Here is the formula for multiplying two $2 \times 2$ matrices.

\begin{equation*}
  \begin{tikzcd}
    \bbr^2 \ar[rr,"B"] \ar[rrrr,"AB"', bend right] && \bbr^2 \ar[rr,"A"] && \bbr^2
  \end{tikzcd}
\end{equation*}
\begin{align}
  \label{eq:eq2}
  \begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{bmatrix}
  \begin{bmatrix} B_{11} & B_{12} \\ B_{21} & B_{22} \end{bmatrix}
    &=
  \begin{bmatrix} A_{11} B_{11} + A_{12} B_{21} & A_{11} B_{12} + A_{12} B_{22} \\ A_{21} B_{11} + A_{22} B_{21} & A_{21} B_{12} + A_{22} B_{22} \end{bmatrix}
\end{align}

\begin{qbox}
  In Section \ref{section:linearTransformation}, we computed the identity, rotation, and reflection matrices
  \begin{align*}
    [\id_{\bbr^2}] &= \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \\\\
    [\rot] &= \begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{bmatrix}\\\\
    [\refl] &= \begin{bmatrix} \cos 2\theta & \sin 2\theta \\ \sin 2\theta & -\cos 2\theta \end{bmatrix}
  \end{align*}
  Using \eqref{eq:eq2} compute
  \begin{multicols}{2}
    \begin{enumerate}
      \item $[\id_{\bbr^2}] \begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{bmatrix}$
      \item $ \begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{bmatrix} [\id_{\bbr^2}]$
      \item $ [\rot] [\rot[\varphi]] $
      \item $[\refl] [\refl]$
      \item $[\refl] [\refl[\varphi]]$
    \end{enumerate}
  \end{multicols}
  See any old friends? Interpret these geometrically.
\end{qbox}

\begin{qbox}[Optional] Derive the formula for matrix multiplication of $2 \times 2$ matrices using compositions of linear transformations.
\end{qbox}

\hrule
\hrule
\hrule

Congratulations on making it this far!!!(Yay!) What we have ``covered'' in the past week is more than what you would do in a month of regular class.
There is hardly an area of mathematics that does not use linear algebra, the more math you do the better you'll be able to understand and appreciate it.
Here are some suggested topics to read from here:
\begin{multicols}{2}
  \begin{enumerate}
    \item Change of basis theorem,
    \item Eigenvalues and eigenvectors,
    \item Determinants,
    \item Gaussian elimination,
    \item Inner product spaces,
    \item Spectral theorem,
    \item Jordan canonical forms,
    \item Matrix groups.
  \end{enumerate}
\end{multicols}


\section{Adaptive Noise Cancellation}

\begin{enumerate}[label=\alph*), leftmargin=*]

%% a)
\item
%

Let the pure sine wave $x(n)$, the noisy signal $s(n)$, the noise term $\eta(n)$ and the ALE filter output $\hat{x}(n; \Delta)$, parametrised by the delay parameter $\Delta$.
Then the Mean Squared Error (MSE) is given by:

\begin{align}
    \mathtt{MSE} =
    \E \bigg[ \big( s(n) - \hat{x}(n; \Delta) \big)^{2} \bigg]  &=  \E \bigg[ \big( x(n) + \eta(n) - \hat{x}(n; \Delta) \big)^{2} \bigg] \\
                                                                &=  \E \bigg[ \big( \eta(n) + (x(n) - \hat{x}(n; \Delta)) \big)^{2} \bigg] \\
                                                                &=  \E \bigg[ \eta^{2}(n) \bigg] +
                                                                    \E \bigg[ \big( x(n) - \hat{x}(n; \Delta) \big)^{2} \bigg] +
                                                                   2\E \bigg[ \eta(n) \big(x(n) - \hat{x}(n; \Delta) \big) \bigg]
\label{eq:ale_mse}
\end{align}

The first term, noise power $\E [ \eta^{2}(n) ]$, is independent of $\Delta$, while the second term, Mean Squared Prediction Error $\E [ (x(n) - \hat{x}(n; \Delta))^{2} ]$ is not
a function of noise $\eta(n)$. Hence, the last term only involves both the delay $\Delta$ (through $\hat{x}(n)$) and the noise term, so we will minimise it:

\begin{equation}
    \underset{\Delta \in \sN}{min}\ \E \bigg[ \eta(n) \big(x(n) - \hat{x}(n; \Delta) \big) \bigg]
\end{equation}

Using the fact that $x(n)$ and $\eta(n)$ are uncorrelated the term $\E [ \eta(n) x(n) ]$ vanished:

\begin{align}
    \underset{\Delta \in \sN}{min}\ \E \bigg[ \eta(n) \hat{x}(n; \Delta) \bigg] &\rightarrow
        \underset{\Delta \in \sN}{min}\ \E \bigg[ \big( u(n) + 0.5u(n-2) \big) \vw^{T} \vu(n; \Delta) \bigg] \\
                                                                                &\rightarrow
        \underset{\Delta \in \sN}{min}\ \E \bigg[ \big( u(n) + 0.5u(n-2) \big) \sum_{i=0}^{M-1} w_{i} s(n - \Delta - i) \bigg] \\
                                                                                &\rightarrow
        \underset{\Delta \in \sN}{min}\ \E \bigg[ \big( u(n) + 0.5u(n-2) \big) \sum_{i=0}^{M-1} w_{i} \big( x(n - \Delta - i) + \eta(n - \Delta - i) \big) \bigg] \\
                                                                                &\rightarrow
        \underset{\Delta \in \sN}{min}\ \E \bigg[ \big( u(n) + 0.5u(n-2) \big) \sum_{i=0}^{M-1} w_{i} \big( \eta(n - \Delta - i) \big) \bigg] \\
                                                                                &\rightarrow
        \underset{\Delta \in \sN}{min}\ \E \bigg[ \big( u(n) + 0.5u(n-2) \big) \sum_{i=0}^{M-1} w_{i} \big( u(n - \Delta - i) + 0.5 u(n - 2 - \Delta - i) \big) \bigg]
\label{con:delta} \\
                                                                                &\rightarrow
        0, \quad \Delta > 2
\end{align}

Since $u(n)$ is identically and \textbf{independently} distributed white noise:

\begin{equation}
    \E \bigg[ u(n) u(n - j) \bigg] = 0, \quad \forall j \neq 0
\end{equation}

therefore the expectation in (\ref{con:delta}) is zero and thus minimised for $\Delta > 2$, since the terms are non time-overlapping.
This is an expected result, since the colored noise signal $\eta(n)$ is a second order MA process.

The theoretical optimal delay range, $\Delta > 2$ is also verified empirically. In figure \ref{fig:3_3_a_1} the clean signal $x(n)$ against, $s(n)$ and filter output $\hat{x}(n)$
are illustrated for different delay $\Delta$ values. Moreover, the MPSE as a function of $\Delta$ is also plotted in figure \ref{fig:3_3_a_2}, verifying the improved performance $\Delta > 2$.
All experiments are conducted using a fixed model order $M = 5$ LMS filter.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/a/ale_overlay-Delta_1}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/a/ale_overlay-Delta_2}
    \end{subfigure}
    ~
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/a/ale_overlay-Delta_3}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/a/ale_overlay-Delta_4}
    \end{subfigure}
    \caption{ALE: overlay plots for various $\Delta$ delays, for fixed $M=5$.}
    \label{fig:3_3_a_1}
\end{figure}

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/a/ale_mpse}
    \end{subfigure}
    \caption{ALE: MPSE against $\Delta$, for fixed $M=5$.}
    \label{fig:3_3_a_2}
\end{figure}

%% b)
\item
%

The experiments are repeated varying now both the delay parameter, $\Delta$, and the model order, $M$, obtaining figures \ref{fig:3_3_b_1}, \ref{fig:3_3_b_2}.
We notice that the mean squared prediction error (MPSE) is minimised for the hyperparmeters pair $(\Delta, M) = (3, 6)$.

Over-modelling (large $M$) results in excess degrees of freedom that increase computational complexity and over-fit noise, degrading
model performance. For model order $M=6$ MPSE is minimised, while the model complutational load is still not prohibitive.

In the previous part we showed theoretically that for $\Delta > 2$ the noise and the filter output are uncorrelated thus MSE is minimised.
Nonetheless, the second term in (\ref{eq:ale_mse}) was ignored. The impact of this term on the MPSE is illustrated in figure \ref{fig:3_3_b_2},
where very large $\Delta$ (i.e $\Delta = 25$) inevitably cause a time-shift between the filter output $\hat{x}(n)$ and the true sine wave $x(n)$.
Hence, $\Delta = 3$ is the optimal parameter, minimising delay effects between $x(n)$ and $\hat{x}(n)$, as well as guaranteeing uncorrelation between
$\hat{x}(n)$ and $\eta(n)$.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/b/ale_mpse_vs_Delta}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/b/ale_mpse_vs_M}
    \end{subfigure}
    \caption{ALE: MPSE against delay $\Delta$ and model order $M$.}
    \label{fig:3_3_b_1}
\end{figure}

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/b/ale_overlay-Delta_1}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/b/ale_overlay-Delta_3}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/b/ale_overlay-Delta_25}
    \end{subfigure}
    \caption{ALE: model deterioration for large delays $\Delta$.}
    \label{fig:3_3_b_2}
\end{figure}

%% c)
\item
%

We compare the performance of the Adaptive Noise Cancellation (ANC) configuration to the Adaptive Line Enchancer (ALE) configuration, used in previous parts.
The colored noise $\epsilon(n)$ is used as input to the LMS filter to perform ANC, such that:

\begin{equation}
    \epsilon(n) = 0.9 \eta(n) + 0.05
\end{equation}

and as a result $\epsilon(n)$ is the secondary noise signal, correlated to the $\eta(n)$, primary noise signal. Note that the relationship between the two signals
is unknown to the ANC algorithm.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/c/ALE_overlay}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/c/ANC_overlay}
    \end{subfigure}
    \caption{ALE vs ANC: overlay plots and mean prediction squared error.}
    \label{fig:3_3_c_1}
\end{figure}

In figure \ref{fig:3_3_c_1} the overlay plots for 100 realisations of the process $x(n)$ are provided, along with the denoised versions of both the ALE and ANC configurations.
ANC performs overall better, with MPSE = 0.1374, than the ALE configuration, which scores MPSE = 0.2520. Nonetheless, we highlight the fact that the ANC algorithm
does poorly in the first timesteps, but for $t > 400$, it tracks the true signal $x(n)$ much better. An ensemble of realisations of the process is also simulated and its mean are illustrated in figure \ref{fig:3_3_c_2}.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/c/comparison}
    \end{subfigure}
    \caption{ALE vs ANC: ensemble mean comparison.}
    \label{fig:3_3_c_2}
\end{figure}

%% d)
\item
%

Let a synthetic reference input, $\epsilon(n)$, composed of a sinusoid of $50\ Hz$ corrupted by white Gaussian noise. The ANC configuration is used with inputs
$\epsilon(n)$ and the \texttt{POz} EEG time-series, in order to remove the strong $50 Hz$ frequency component due to power-line interference (mains).

For illustration purposes \texttt{spectrogram}s are plotted using a rectangular window of length $L=4096$ and $80\%$ overlap. The obtained spectrograms are provided
in figures \ref{fig:3_3_d_1} and \ref{fig:3_3_d_2}. As expected the original \texttt{POz} signal has a strong $50\ Hz$ frequency component.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/original}
    \end{subfigure}
    \caption{EEG: original, reference spectrogram.}
    \label{fig:3_3_d_1}
\end{figure}

The LMS filter order $M$ and the step-size $\mu$ is varied, and the impact on the spectrogram is shown in figure \ref{fig:3_3_d_2}.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/M_1-mu_0.001}.pdf}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/M_1-mu_0.005}.pdf}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/M_1-mu_0.100}.pdf}
    \end{subfigure}
    ~
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/M_10-mu_0.001}.pdf}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/M_10-mu_0.005}.pdf}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/M_10-mu_0.100}.pdf}
    \end{subfigure}
    ~
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/M_25-mu_0.001}.pdf}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/M_25-mu_0.005}.pdf}
    \end{subfigure}
    ~
    \begin{subfigure}{0.32\textwidth}
        \centering
        \includegraphics[height=1in]{{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/M_25-mu_0.100}.pdf}
    \end{subfigure}
    \caption{EEG: ANC denoised spectrogram for different model order, $M$, and step-sizes, $\mu$.}
    \label{fig:3_3_d_2}
\end{figure}

Large step-sizes (i.e $\mu = 0.1$) affect significantly the spectral components around $50\ Hz$, degrading ANC performance.
On the other hand, small step-sizes (i.e $\mu = 0.001$) take more time to reach steady-state, however provide successful denoising, without disrupting the frequencies close to $50\ Hz$.

On the other hand, under-modelling (i.e $M = 1$) leads to poor noise cancellation, since the $50\ Hz$ power-line interface component has not been attenuated.
However, over-modelling (i.e $M=25$) degrades quality of neighbour frequencies, while a medium size model, such as $M=10$, achieves satisfying performance, by eliminating the
$50\ Hz$ component of interest, without affecting any other compoenents.

Overall, an ANC configuration with $(M, \mu) = (10, 0.001)$ is selected. The corresponding denoised periodogram is also provided at figure \ref{fig:3_3_d_3}.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/periodogram}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-noise-cancellation/assets/d/periodogram-error}
    \end{subfigure}
    \caption{EGG: periodograms of original and denoised signals.}
    \label{fig:3_3_d_3}
\end{figure}

%
\end{enumerate}


\lab{Hysteresis}{Bifurcations}
\label{lab:Bifurcations}

Recall that any ordinary differential equation can be written as a first order system of DEs,
\begin{align}
\dot{x} = F(x), \quad \dot{x} := \frac{d}{dt}x(t).\label{fos}
\end{align}
Many interesting applications and physical phenomena can be modelled using ODEs.
Given a mathematical model of the form \eqref{fos}, it is important to understand geometrically how its solutions behave.
This information can then be conveyed in a phase portrait, a graph describing solutions of \eqref{fos} with differential initial conditions.
The first step in constructing a phase portrait is to find the equilibrium solutions of the equation, i.e., the zeros of $F(x)$, and to determine their stability.

It is often the case that the mathematical model we study depends on some parameter or set of parameters $\lambda$.
Thus the ODE becomes
\begin{align}
\dot{x} = F(x,\lambda).\label{fos2}
\end{align}
The parameter $\lambda$ can then be tuned to better fit the physical application.
As $\lambda$ varies, the equilibrium solutions and other geometric features of \eqref{fos2} may suddenly change.
A value of $\lambda$ where the phase portrait changes is called a \emph{bifurcation point}; the study of how these changes occur is called \emph{bifurcation theory}.
The parameter values and corresponding equilibrium solutions are often graphed together in a bifurcation diagram.

As an example, consider the scalar differential equation
\begin{eqnarray}
\dot{x} &=& x^2 + \lambda. \label{snbifurcation}
\end{eqnarray}
For $\lambda > 0$ equation (\ref{snbifurcation}) has no equilibrium solutions.
At $\lambda = 0$ the equilibrium point $x=0$ appears, and for $\lambda < 0$ it splits into two equilibrium points.
For this system, a bifurcation occurs at $\lambda = 0$.
This is an example of a saddle-node bifurcation.
The bifurcation diagram is shown in Figure \ref{bifurcation:sn}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{SaddleNBifurcation.pdf}
\caption{Bifurcation diagram for the equation $\dot{x} = \lambda + x^2$.}
\label{bifurcation:sn}
\end{figure}

% The flow of a differential equation \[\dot{x} = f(x)\] is the family of all possible solutions $\phi(t,x_0)$, where $x_0$ represents the arbitrary initial value, $t \in \mathbb{R}.$
% Here we will mainly consider scalar differential equations (so-called because $x$ is one dimensional).
% It is often necessary to study a family of differential equations. These will have the form
% \[\dot{x}= F(x,c),\]
% where $c$ may be a single parameter or a vector of parameters.
%
% Bifurcation theory is the study of how the qualitative structure of the flow of a differential equation varies as parameters in the differential equation are varied.
% A differential equation has a stable orbit structure if sufficiently small changes in the parameter value do not change the qualitative structure of the flow.
% A parameter value for which the flow does not have a stable orbit structure is called a bifurcation value.
%
% Terminology
% bifurcation theory: 'the study of possible changes in the structure of orbits of a differential equation depending on variable parameters'
% 'the study of changes in the qualitative structure of the flow of a differential equation as parameters are varied'
% phase portrait:
% bifurcation diagram:
% number of orbits and their direction of flow of a differential equation = 'orbit structure of the differential equation' or 'the qualitiative structure of the flow'.
%
% Hyperbolic equilibrium points
% $F(x_0,c_0) = 0$ where $\frac{df}{dx}(x_0,c_0) \not = 0.$
% In this case the stability of the equilibrium point $x_0$ for values of $c$ near $c_0$ is determined by the derivative $\frac{df}{dx}(x_0,c_0)$.
% Example: $\dot{x} = x-c.$ Plot $x$
%
% Hyperbolic equilibrium - mainly to be used when discussing stability
% Saddle-Node Bifurcation - \[\dot{x} = c + x^2.\]
% Transcritical Bifurcation - \[\dot{x} = cx + x^2.\]
% Hysteresis Loop - \[\dot{x} = c + x-x^3.\]
% Pitchfork Bifurcation (Supercritical) - \[\dot{x} = cx-x^3.\]
% For an exercise, do a variation of \[\dot{x} = 1+cx-x^3.\]

Suppose that $F(x_0,\lambda_0) = 0.$
We use a method called natural embedding to find zeros $(x,\lambda)$ of $F$ for nearby values of $\lambda$.
Specifically, we step forward in $\lambda$ by letting $\lambda_1 = \lambda_0 + \triangle \lambda$, and use Newton's method to find the value $x_1$ that satisfies $F(x_1,\lambda_1) = 0.$
This method works well except when $\lambda$ is near a bifurcation point $\lambda^*$.

The following code implements the natural embedding algorithm, and then uses that algorithm to find the curves in the bifurcation diagram for (\ref{snbifurcation}).
Notice that this algorithm needs a good initial guess for $x_0$ to get started.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{SaddleNPhasePortrait.pdf}
\caption{Phase Portrait for the equation $\dot{x} = -2 + x^2$.}
\label{phaseportrait:sn}
\end{figure}

\begin{lstlisting}
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import newton

def EmbeddingAlg(param_list, guess, F):
    X = []
    for param in param_list:
        try:
            # Solve for x_value making F(x_value, param) = 0.
            x_value = newton(F, guess, fprime=None, args=(param,), tol=1E-7, maxiter=50)
            # Record the solution and update guess for the next iteration.
            X.append(x_value)
            guess = x_value
        except RuntimeError:
            # If Newton's method fails, return a truncated list of parameters
            # with the corresponding x values.
            return param_list[:len(X)], X
    # Return the list of parameters and the corresponding x values.
    return param_list, X

def F(x, lmbda):
    return x**2 + lmbda

# Top curve shown in the bifurcation diagram
C1, X1 = EmbeddingAlg(np.linspace(-5, 0, 200), np.sqrt(5), F)
# The bottom curve
C2, X2 = EmbeddingAlg(np.linspace(-5, 0, 200), -np.sqrt(5), F)
\end{lstlisting}

\begin{problem}
Use the natural embedding algorithm to create a bifurcation diagram for the differential equation
\[\dot{x} = \lambda x-x^3.\]
This type of bifurcation is called a pitchfork bifurcation (you should see a pitchfork in your diagram).

Hints: Essentially this amounts to running the same code as the example, but with different parameters and function calls so that you are tracing through the right curves for this problem.
To make this first problem work, you will want to have your `linspace` run from high to low instead of from low to high.
There will be three different lines in this image.
\end{problem}

\begin{problem}
Create bifurcation diagrams for the differential equation
\[\dot{x} = \eta + \lambda x-x^3,\]
where $\eta = -1, -.2, .2$ and $1.$  Notice that when $\eta = 0$ you can see the pitchfork bifurcation of the previous problem.
There should be four different lines in this image.
There will be one line for each value of $\eta$.
\end{problem}

An interesting ODE is given by
\begin{align*}
	x' &= \lambda + x - x^3.
\end{align*}
This system has a bifurcation diagram containing what is known as a hysteresis loop, shown in Figure \ref{bifurcation:hysteresis}.
In the hysteresis loop, when the parameter $\lambda$ moves beyond the bifurcation point the equilibrium solution makes a sudden jump to the other stable branch.
When this occurs the system cannot reach its previous equilibrium by simply rewinding the parameter slightly.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{HysteresisBifurcation.pdf}
\caption{Bifurcation diagram for the ODE $x' = \lambda + x - x^3$. }
\label{bifurcation:hysteresis}
\end{figure}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{BudwormEquilibria.pdf}
\caption{Graphical demonstration of nonzero equilibrium solutions for the budworm population (here $r = .56$).
Note that as $k$ increases, the number of solutions goes from one to three, and then back to one. }
\label{equilibria:budworm}
\end{figure}

\section*{Budworm Population Dynamics}
Here we study a mathematical model describing the population dynamics of an insect called the spruce budworm.
In eastern Canada, an outbreak in the budworm population can destroy most of the trees in a forest of balsam fir trees in about 4 years.
The mathematical model is given by
\begin{align}
\dot{N} = RN\left(1 - \frac{N}{K}\right) - p(N). \label{budworm1}
\end{align}
This model was studied by Ludwig et al (1978), and is described well in Strogatz's text \emph{Nonlinear Dynamics and Chaos}.
Here $N(t)$ represents the budworm population at time $t$, $R$ is the growth rate of the budworm population and $K$ represents the carrying capacity of the environment.
We could interpret $K$ to represent the amount of food available to the budworms.
$p(N)$ represents the death rate of budworms due to predators (birds); we assume specifically that $p(N)$ has the form $P(N) = \frac{BN^2}{A^2 + N^2}$.

Before studying the equilibrium points of \eqref{budworm1} it is important to reduce the number of parameters in the system by nondimensionalizing.
Thus, we make the coordinate change $x = N/A$, $\tau = Bt/A$, $r = RA/B$, and $k = K/A$, obtaining finally the system
\begin{align}
	\frac{dx}{d \tau} &= rx(1-x/k) - \frac{x^2}{1+x^2}.
\end{align}

Note that $x = 0$ is always an equilibrium solution.
To find other equilibrium solutions we study the equation $r(1-x/k)-x/(1+x^2) = 0$.
Fix $r = .56$, and consider Figure \eqref{equilibria:budworm}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{BudwormPopulation.pdf}
\caption{Bifurcation diagram for the budworm population model.
The parameter $r$ is fixed at $0.56.$
The lower stable branch is known as the refuge level of the bundworm population, while the upper stable branch is known as the outbreak level.
Once the budworm population reaches an outbreak level, the available food (foliage of the balsam fir trees) in the system must be reduced drastically to jump back down to refuge level.
Thus many of the balsam fir trees die before the budworm population returns to refuge level.}
\label{bifurcation:budworm}
\end{figure}

\begin{problem}[Budworm Population]
Reproduce the bifurcation diagram for the differential equation
\begin{align*}
	\frac{dx}{d \tau} &= rx(1-x/k) - \frac{x^2}{1+x^2},
\end{align*}
where $r = 0.56$.

Hint: Find a value for $k$ that you know is in the middle of the plot (i.e. where there are three possible solutions), then use the code above to expand along each contour till you obtain the desired curve.
Now find the proper initial guesses that give you the right bifurcation curve.
The final plot will look like the one in Figure \ref{bifurcation:budworm}, but you will probably have to run the embedding algorithm 6 times to get every part of the plot.
\end{problem}