\chapter{Neural Networks}
\label{cha:neural_networks}
\epigraph{In the recent years Neural Networks (NN) have solved more and more
  tasks that have previously been too difficult or simply too tedious to solve
  with traditionally coded algorithms. They have been successfully applied to a
  variety of problems such as pattern recognition, image classification and
  prediction.  As the NN algorithms learn from data, they seem to be good
  candidates for finding anomalies without any prior knowledge about the
  given dataset, as long as it is big enough. By applying online learning
  algorithms that learn and adapt continuously it should in theory be possible
  to create an automated, adaptive outlier detection.\\ This chapter describes
  the mechanics of NNs from the ground up. Starting with the widely used
  \emph{feedforward networks} (FNN), we go on to \emph{recurrent neural nets}
  (RNN). Similar to biological neural networks, they have cyclic connections,
  and are capable of processing sequences. After that, a special kind of RNN is
  introduced, that dramatically cuts computational costs and solves some
  notorious problems that arise during RNN training. In addition to the general
  difficulty of training NNs, they often rely on hyper-parameters, that are
  typically set by manually tuning the network.  This chapter is closed with a
  brief description of hyper-parameter optimization techniques which were used
  to automate this task.
}

\section{Feedforward Neural Networks}
\label{sec:feedforward_neural_networks}

The idea for artificial neural networks is based on the human brain,
which is a highly complex, non-linear, and parallel computer.  Just like the
brain, neural networks are constructed from small units that are connected to each other
with weights. In analogy to its biological counterpart these units are also
called \emph{neurons}. Older literature also often refers to them as
\emph{perceptrons}.  They are capable of processing and passing on incoming
information.
Traditional \emph{feedforward neural networks} (FNN) implement static
input-output mappings, which mathematically makes them pure functions of the
input signals. Figure \ref{fig:perceptron} shows a simple schematic of a single
neuron.  It generally has $n$ inputs $u_j$ and one output.  Each of the inputs
has an assigned weight $w_j$, which determines the contribution of a given
input to the output.  The output of a neuron will further be referred to as
\emph{activation} $y$:
\begin{equation}
  \label{eq:ffn_eq}
  y = \varphi \left( \sum_j (w_j u_j) + b  \right)
       = \varphi (\vec{w} \cdot \vt{u} + b).
\end{equation}
The sum over all the inputs multiplied by their weights can conveniently be
represented by a dot product.  Often a bias term $b$ is included, which can be
taken as a measure of how easy it is to activate the neuron.  The function
$\varphi$ is called activation function and can have various different forms,
from a simple binary function to an arbitrary (non-linear), monotonically
increasing function.  A frequently used activation function in FNNs is the
sigmoid function
\begin{equation}
  \sigma (z) = \frac{1}{1 + e^{-z}},
\end{equation}
which is shown in Fig.~\ref{fig:sigmoid}. To create a \emph{layer} of neurons
they are simply stacked on top of each other:
\begin{equation}
  \vt{y} = \varphi (\wmatr{} \vt{u} + \vec{b}).
\end{equation}

\begin{figure}
  \begin{minipage}{.42\textwidth}
    \centering
    \Perceptron
    \vspace{4.5mm}
    \caption{Schematic of a neuron [\cite{Nielsen2015}].
    The activation function is represented by the circle.}
    \label{fig:perceptron}
  \end{minipage}
  \hspace{.02\textwidth}
  \begin{minipage}{.54\textwidth}
    \centering
    \includegraphics[width=\linewidth]{sigmoid_tanh.pdf}
    \caption{Sigmoid function $\varphi$. Typically used as an activation function
    in feedforward neural nets.}
    \label{fig:sigmoid}
  \end{minipage}
\end{figure}

The final FNN with multiple layers (such as in
Fig.~\ref{fig:feedforward_network}) is created by feeding the output of one
layer to another one until the last layer of the network is reached.  The
\emph{universal approximation theorem} [\cite{uni_approx_theorem}] states the
ability of FNNs to approximate an arbitrary non-linear function $f$
\begin{equation}
   \vt{d} = f(\vt{u})
\end{equation}
with arbitrary precision. This means that given any $\epsilon > 0$ we can find
an FNN $F$, such that
\begin{equation}
  ||F(\Theta, \vt{u}) - f(\vt{u})||_2 < \epsilon
\end{equation}

for all possible inputs $\vt{u}$. The exact form of the approximation $F$
depends on the network architecture, but generally it depends on a set of
parameters $\Theta$ called weights (and biases), which represent several layers
of the FNN.  Of course, the universal approximation theorem does not say
anything about the practical learnability of a task, which will be discussed in
Sec.~\ref{sub:gradient_descent}.

Each layer of an FNN can be represented by a matrix, and therefore only has the
computational expressibility of linear functions. This linearity is only broken
by the activation function. Without the non-linear activation an arbitrary
number of layers would not be more effective than a single one.  Through the
organization in layers FNNs are able to model not only arbitrary functions but
also to separate datasets that are not linearly separable.
Fig.~\ref{fig:feedforward_network} shows an FNN where the first (input) layer
represents the data that is fed into the network, followed by one or more
hidden layers, and an output layer.  Hidden layers act as a non-linear
transform that distorts the input in such a way that its classes become
linearly separable by the output layer.  An interactive explanation of how this
works can be found in a blog post by [\cite{colah_topology}].

\begin{figure}
  \centering
  \FeedForwardNet{2}
  \caption{Fully connected feedforward neural network with a single hidden
    layer and two output neurons.}
  \label{fig:feedforward_network}
\end{figure}

Every network that has more than two or three hidden layers is typically called
a \emph{deep neural network}. It is fundamentally not different from the basic
architecture of the described feedforward networks but holds the potential of
more powerful transformations of the input. The type of neural network that is
shown in Fig.~\ref{fig:feedforward_network} is called \emph{feedforward
network}, because the input data is entering the network at the input layer and
then passed through the network towards the output layer.  Feedforward nets are
often applied to classification tasks, such as the recognition of a certain
shape in an image.

The goal of the machine learning approach is to find a weight configuration
that captures the \emph{essence} of the presented dataset, meaning that it
generalizes well, including over inputs that it has not been trained with.
This is typically approached by some variation of Gradient Descent (GD). GD
algorithms try to minimize a certain loss or cost function with respect to a
given weight configuration (more detailed description in
Sec.~\ref{sub:gradient_descent}).  However, the pursued generalization of the
network is highly dependent on the training dataset. It should ideally include
the total variability of possible inputs.  This is very hard to achieve in
practice and the available datasets are typically split randomly into three
categories: \emph{training}, \emph{validation}, and \emph{test} set.  The
training set is used to optimize the network. Its loss is minimized by a GD
algorithm. The validation set is used to determine the performance of the
network on inputs that are not included in the training set.  The optimization
should be stopped as soon as the error on the validation set is not decreasing
any more in order to reduce the risk of overfitting. This is one of many
regularization methods that are applied in ML in order to achieve a
generalization over previously unseen datapoints. Because the information of
the validation set is leaking into the network via the early stopping
criterion, a third dataset is needed to evaluate the actual performance and
generalization of the network.  This third set is called the test set.

Feedforward networks outperform conventional algorithms at tasks such as image
recognition and other classification and pattern recognition.  They are,
however, not very suitable to model time series, as they cannot model correlations
of previous inputs. This means that they are not designed for the prediction of
sequences.  To be able to process sequences and make predictions,
\emph{recurrent} weight connections are introduced in
Sec.~\ref{sec:recurrent_neural_networks}.  Despite the compelling results that
can be achieved by FNNs, they have a few drawbacks.  Backpropagation algorithms
typically need very large amounts of data to train the network. The gradient
descent steps have to be small enough to not jump over the desired minima,
which leads to very long training times.  Additionally the nature of the
training data can lead to biases of the resulting network if it does not
properly represent the possible parameter space.  It is hard to infer
afterwards how the network came to a specific result, as the weight matrices do
not represent a traceable way of reasoning or logic.  A trained network is like
a black box, which does not come with an obvious way of determining how or why
a certain classification was made.

\subsection{Convolutional Neural Nets}%
\label{sub:convolutional_neural_nets}

A breakthrough in image classification performance was achieved by the
introduction of a subtype of FNNs, the \emph{convolutional neural networks}
(CNN), which can leverage the spatial structure of the input data.  They
combine the concept of filters from signal processing with the ML approach by
learning their own filters (often referred to as kernels as well) for feature
detection in images.  The convolutional operation of a given filter $F$ and an
image $H$ is defined by:
\begin{align}
  G &= H * F \text{, where}\\
  G[i,j] &= \sum_{u=-k}^k \sum_{v=-k}^k H[i-u, j-v]F[u,v]
\end{align}

A well known example of a filter for edge detection looks like this:
\begin{equation}
  F = \begin{bmatrix}
    -1 & -1 & -1 \\
    -1 & 8  & -1 \\
    -1 & -1 & -1
  \end{bmatrix}
\end{equation}

Applying it to a (black and white) image $H$
\begin{equation}
  H = \begin{bmatrix}
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	  1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0\\
	\end{bmatrix}
\end{equation}

and padding the convoluted image $G$ to obtain the original dimensions results
in a filtered image $R$, which is zero everywhere except at the location of the
edge:

\begin{equation}
  \newcommand{\nan}{-}
  \newcommand{\minus}{\text{-}}
	R = \begin{bmatrix}
    \nan & \nan & \nan & \nan & \nan & \nan  & \nan & \nan & \nan\\
    \nan & 0 & 0 & 0 & 3 & \minus 3 & 0 & 0 & \nan\\
    \nan & 0 & 0 & 0 & 3 & \minus 3 & 0 & 0 & \nan\\
    \nan & 0 & 0 & 0 & 3 & \minus 3 & 0 & 0 & \nan\\
    \nan & 0 & 0 & 0 & 3 & \minus 3 & 0 & 0 & \nan\\
    \nan & \nan & \nan & \nan & \nan & \nan  & \nan & \nan & \nan\\
  \end{bmatrix}
\end{equation}

A convolutional layer consists of multiple kernels, which are learned via GD.
Each kernel can be thought of as a neuron that sees only a small part of the
input image at a time and slides over the whole image. Its output therefore
represents the local structure of said part of the image, which makes
a CNN capable of exploiting spatial structures in images (see schematic in
Fig.~\ref{fig:conv_layer}).  The kernels which are sliding over the input
images naturally lead to a translation invariance of the learned patterns
[\cite{lecun1995convolutional}]. A pattern that is recognized by a filter in one
region of the image can just as well be detected somewhere else, which stands
in contrast to fully connected layers for which the same pattern in different
parts of an image looks completely different.  Scale invariance can be achieved
by using multiple kernels of different size.  Feedforward nets, which are fully
connected to the input cannot leverage the spatial information as efficiently.
Additionally, CNNs decrease the number of parameters in comparison to a normal
FNN, which reduces the risk of overfitting and thus leads to a better
generalization of NNs.
\begin{figure}
  \centering
  \includegraphics[width=0.4\linewidth]{conv_layer.png}
  \caption{The 2D input image on the left is transformed into a 3D output
  volume by applying multiple convolutions to the input [\cite{conv_layer_wiki}].}
  \label{fig:conv_layer}
\end{figure}

An interesting example of a CNN outside the realm of image classification is
AtomNet [\cite{dzamba1510atomnet}], which is used to predict bioactivity of
molecules for drug discovery by exploiting the local structure of biochemical
interactions.

\subsection{Gradient Descent}
\label{sub:gradient_descent}

The most common technique to train neural networks is Gradient Descent.
It is based on minimizing a \emph{loss} (cost or penalty) function $\mathcal{L}$
\begin{equation}
  \label{eq:batch_loss}
  \mathcal{L}(\Theta) = \sum_{\vt{u} \in \mathcal{U}}
                        ||F(\Theta, \vt{u}) - \vt{d}||_2,
\end{equation}
which defines how close the network is to producing the
desired results.  The loss function that is used throughout this thesis is
given by Eq.~\ref{eq:batch_loss}, where the target outputs, given a certain
input $\vec{u}$ out of all training examples $\mathcal{U}$, are denoted by
$\vt{d}$.

Initially, the weights of the network are set randomly and are to be adjusted
in the optimization phase, also called learning or training.  The loss is, of
course, dependent on all the weights and biases, which is where the gradient
descent comes into play.  The partial derivatives of the loss function are
taken with respect to all the weight and biases of the network to find the
gradient which points in the direction of steepest descent.  With this
calculated direction we can step the weights towards the nearest local minimum
and like that gradually increase the performance of the network.
\begin{equation}
  \label{eq:gradient_descent}
  \Theta' = \Theta + \eta \dd{\mathcal{L}}{\Theta}
\end{equation}
The size of the steps is defined by the \emph{learning rate} $\eta$, which has
to be chosen carefully.  If it is too large, the algorithm will oscillate
around the desired minimum, but if chosen too small, the training times will
become too long.  Several adaptive gradient computation algorithms address this
issue. One promising algorithm is called \emph{Adam} (Adaptive Moment
Estimation). Adam combines adaptive gradient descent methods with
momentum based algorithms [\cite{ADAM}]. Momentum-based optimizers add a fraction
of the previously used weight update to simulate an acceleration of the
gradient descent.  The method of applying the gradient descent algorithm to
multi-layer (deep) neural networks is called \emph{backpropagation} (BP),
because the gradient calculation is started at the last layer and iteratively
propagated back through the whole network by applying the chain rule. A more
detailed description of BP is given in Sec.~\ref{sub:backpropagation}.

There are three different variations of gradient descent, which only differ in
the way the sum in Eq.~\ref{eq:batch_loss} is interpreted.  Summing over all
available training examples $\vt{u}$, namely the whole \emph{batch}, is called
{\em batch GD}. Calculating the loss only for a single randomly chosen example,
is called \emph{stochastic gradient descent} (SGD). The compromise of the two,
mini-batch GD, uses a subset of the training examples, performs a weight
update, and then goes on to the next mini-batch.  Both stochastic and
mini-batch GD end up calculating an approximate loss $L$ from a subset $U
\subset \mathcal{U}$:
\begin{equation}
  \mathcal{L}(\Theta) \approx L(\Theta) =
    \sum_{\vt{u} \in U} ||F(\Theta, \vt{u}) - \vt{d}||_2,
\end{equation}

which also leads to an approximated gradient. The updates that are performed
with the approximate gradient are hoped to enable the optimizer to jump out of
shallow local minima and saddle points, which is discussed further in the next
paragraph.

\subsubsection{Local Minima of the Error Surface}%
\label{ssub:local_minima_of_the_error_surface}

As BP is a gradient-based method, there is no guarantee at all, that the
algorithm will find the global minimum of the error surface.  To illustrate
this, Fig.~\ref{fig:error_surface_bgd} depicts a two dimensional error surface
with a global and a local minimum.  Depending on where on the surface the
optimization is started, it ends up in the local or global minimum.  It was
shown that for linear activation functions, the error surface contains only a
single minimum, the global one, with all other locations of zero gradient being
saddle points [\cite{BALDI}]. Momentum-based optimizers can find the global
minimum quite efficiently in such cases. However, this cannot be generalized to
the practically used NNs that almost exclusively use non-linear activations.
Optimizers that include momentum like the Adam optimizer can still sometimes
yield better results, as depicted in Fig.~\ref{fig:error_surface_bgd}.  The two
plots both show a two dimensional error surface, where the coordinates $x$ and
$y$ are regarded as the weights that have to be optimized.  The black, dotted
lines show different runs with varying initial values of $x$ and $y$.  The left
plot shows the convergence paths of plain batch GD algorithm as described
above, which converges towards the global minimum in three out of five cases.
As GD is a purely gradient based method, the path always advances in the
direction of steepest descent.  With the more advanced Adam optimizer, the
global minimum is found in four out of five cases. The momentum towards the
global minimum that is gained in the beginning carries the optimization away
from the local minimum.  Of course there are numerous cases where this approach
will not yield a global minimum. Additionally the Adam algorithm is much more
computationally expensive than plain GD and needs more steps, as can be seen
from the denser dots on the black lines.
\begin{figure}
  \begin{minipage}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{gradient_descent.pdf}
    GD
  \end{minipage}
  \hspace{.02\textwidth}
  \begin{minipage}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{adam.pdf}
    Adam
  \end{minipage}
  \caption{On the left we can see a plain GD optimizer while on the right the
    Adam optimizer was used.  Depending on the starting values of the two
    variables, there are cases in which the optimization ends in the local
    minimum on the right. The GD optimizer needs fewer steps, while Adam finds
    the global minimum in one more case.}
  \label{fig:error_surface_bgd}
\end{figure}

\begin{figure}
  \begin{minipage}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{stochastic_gradient_descent.pdf}
    stochastic GD
  \end{minipage}
  \hspace{.02\textwidth}
  \begin{minipage}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{stochastic_adam.pdf}
    stochastic Adam
  \end{minipage}
  \caption{Convergence paths of stochastic GD algorithms. The noisy paths are
    due to the approximation of the gradient through mini-batches.}
  \label{fig:error_surface_sgd}
\end{figure}
To date, the most commonly used technique to encourage GD to converge to the
global minimum is mini-batch gradient descent.  Instead of evaluating the
\emph{true} loss of the whole training set as indicated by
Eq.~\ref{eq:batch_loss}, SGD adapts the weights after every mini-batch $X$.
This leads to an approximation of the gradient and causes oscillations in the
convergence path, which should make it more probable to escape local minima.
Fig.~\ref{fig:error_surface_sgd} again compares plain GD and Adam, but now with
the perturbing effect of evaluating an approximate loss. The number of training
steps that have to be taken increase significantly, but plain SGD is able to
find the global minimum in four out of five cases. Stochastic Adam almost
reaches the minimum in all five cases, but seems to get stuck right before
the global minimum in the last case.  The conclusion we can draw from the above
examples is that care has to be taken with respect to the convergence of NNs.
Different optimizers, gradient descent variations, and initializations can
yield vastly different results. However, experience has shown that in most
pattern recognition tasks, SGD algorithms can find a local minimum that yields
satisfactory performance, even though it is not guaranteed that this minimum is
global.  In most applications, an increase in the network connections is
supposed to create paths around suboptimal local minima [\cite{rumelhart1986}].
Additionally, in most applications the training is actually stopped early,
meaning at the time the validation error does not decrease further.  This is
done because a network that is fitted perfectly to the training data would most
probably not generalize well over new inputs.  In most cases it is therefore
acceptable or even desirable to remain in a \emph{good} local minimum that
results in a good generalization of the network.

\subsection{Backpropagation}%
\label{sub:backpropagation}

Backpropagation is the basic algorithm that carries out the GD minimizations
that were introduced previously. It was invented in the 1970s but not widely
used until the paper by~[\cite{rumelhart1986}], which marks a breakthrough in the
field of machine learning.  For an easy to read but in depth description of the
subtleties of BP the reader is referred to the book \emph{Neural Networks and
Deep Learning} by [\cite{Nielsen2015}].

Fundamentally, BP is nothing more than the application of the chain rule to the
cost function of a neural network, which can be solved by an \emph{automatic
differentiation} (AD) algorithm. In fact, BP is a special case of AD called
\emph{reverse mode} AD~[\cite{autodiff}].  To illustrate the BP algorithm we will
consider a network with $L$ feedforward layers, where the activations of the
first layer are just the inputs to the network ($\vec{u} = \vec{y}_1$) and the
last layer contains the network outputs $\vec{y}_L$. The components $y^j_l$ of
the vector $\vec{y}_l$ that contains all activations of a layer $l$ are
calculated based on the previous layer:
\begin{equation}
\begin{aligned}
  z^j_l &= \sum_k w^{jk}_l y^k_{l-1} + b^k_L \\
  y^j_l &= \varphi(z^k_l) \label{eq:forward_pass},
\end{aligned}
\end{equation}
where $\vec{z}_l$ are called the \emph{weighted inputs}.  The system above may
be altered such that a given activation depends on any \emph{previous} layer
activation, but \emph{not} on activations of any \emph{next} layer. This step is
named \emph{forward pass}.  During the forward pass, all activations are
calculated starting from the first layer. The goal of the \emph{backward pass}
is to adjust all the weights $w^{jk}_l$ (and biases $b^k_L$) such that the loss $L$
\begin{equation}
  L = \frac{1}{2}\sum_k (d^k - y^k_L)^2
\end{equation}
of a single input example $\vec{u}$ is minimized. BP solves this by iteratively
calculating the gradients needed for Eq.~\ref{eq:gradient_descent} starting
from the last layer.  Before directly calculating the necessary gradients it is
easier to compute the \emph{error} $\delta_L$ of the weighted inputs for each
unit $j$ of the last layer:
\begin{equation}
  \delta^j_L = \dd{L}{z^j_L} = \frac{1}{2}\sum_k \dd{}{z^j_L}(d^k - y^k_L)^2
             = \dd{L}{y^j_L}\dd{y^j_L}{z^j_L}
             = (d^j - y^j_L) \varphi'(z^j_L) \label{eq:delta_last_layer},
\end{equation}
where the sum vanishes because $y^k_L$ only depends on $z^j_L$ if $j = k$.
Now the error of any layer $l$ can be expressed in terms of the error of the
next layer $l+1$:
\begin{equation}
  \delta^j_{l} = \dd{L}{z^j_l} = \sum_k \dd{L}{z^j_{l+1}}\dd{z^j_{l+1}}{z^j_l}
  = \sum_k \delta^j_{l+1} \dd{z^j_{l+1}}{z^j_l}
  = \sum_k \delta^j_{l+1} w^{kj}_{l+1} \varphi'(z^j_l). \label{eq:delta_other_layer}
\end{equation}

The desired error gradients can be found to be:
\begin{equation}
\begin{aligned}
  \dd{L}{b^j_l} &= \delta^j_l, \\
  \dd{L}{w^{jk}_l} &= y^j_{l-1} \delta^j_l. \label{eq:error_grads}
\end{aligned}
\end{equation}
They can be used to incrementally update the weights and biases according
to Eq.~\ref{eq:gradient_descent}.

\subsubsection{The Backpropagation Algorithm}%
\label{ssub:the_backpropagation_algorithm}

\begin{enumerate}
  \item \emph{Input}: $\vec{u}$ is the activation vector of the first layer
    $\vec{y}_1$
  \item \emph{Forward pass}: Compute activations $\vec{y}_l$ for each
    $l=2,3,...,L$ (Eq.~\ref{eq:forward_pass})
  \item \emph{Output error}: Compute error $\delta_L$ of last layer
    (Eq.~\ref{eq:delta_last_layer})
  \item \emph{Backpropagate}: Compute $\delta_{l-1}$ based on error of layer
    $l$ (Eq.~\ref{eq:delta_other_layer})
  \item \emph{Output}: Obtain the gradients of the cost function
    (Eq.~\ref{eq:error_grads})
\end{enumerate}


\documentclass[letterpaper,10pt]{article}
\usepackage{epsfig,comment,amssymb,listings,amsmath}
\setlength{\parskip}{1.8mm}
\setlength{\parindent}{0mm}
\begin{document}
%\date{}
\title{\Large \bf Commutation relations in Fock space}

\author{
{\rm Daniel Tang}\\
}

\maketitle

% Use the following at camera-ready time to suppress page numbers.
% Comment it out when you first submit the paper for review.
%\thispagestyle{empty}

\section{Commutation identities}
%##########################################################

\begin{equation}
AB = BA + [A,B]
\end{equation}

\begin{equation}
[A, BC] = [A,B]C + B[A,C]
\end{equation}

\begin{equation}
[AB, C] =  A[B,C] + [A,C]B
\end{equation}

\begin{equation}
[A,B + C] = [A,B] + [A,C]
\label{sumcommute}
\end{equation}
\begin{equation}
[A,mB] = m[A,B]
\end{equation}
where $m$ is a scalar multiplicative constant.
\begin{equation}
[A,B] = -[B,A]
\end{equation}

\begin{equation}
[ab,AB] = a[b,A]B + [a,A]bB + Aa[b,B] + A[a,B]b
\end{equation}
\begin{equation}
 = a[b,A]B  + [a,A]Bb + aA[b,B] + A[a,B]b
\end{equation}

\begin{equation}
[A,B^n] = \sum_{q=1}^n B^{q-1}[A,B]B^{n-q}
\end{equation}

\begin{equation}
[A,B^{n+1}] = [A,B]B^n + [A,B^n]B - [[A,B^n],B]
\end{equation}

\begin{multline}
[[A,C],[B,D]]=[[[A,B],C],D]+[[[B,C],D],A]
\\+[[[C,D],A],B]+[[[D,A],B],C]
\end{multline}
(source Wikipedia "Commutator")

\begin{equation}
[[A,H],[B,H]]=[[[A,B],H],H]+[[[B,H],H],A]-[[[A,H],B],H]
\end{equation}
so
\begin{multline}
[[^nA,H],[^mB,H]]=[^2[[^{n-1}A,H],[^{m-1}B,H]],H]
\\+[[^{m+1}B,H],[^{n-1}A,H]]-[[[^nA,H],[^{m-1}B,H]],H]
\end{multline}

If we define $[^nA,H] = [\dots [[A,H],H] \dots ,H]$ to be the $n$-fold commutation:

\begin{equation}
[^n AB,H] = \sum_{m=0}^n {n \choose m}[^mA,H][^{n-m}B,H]
\end{equation}

\begin{equation}
[^n ABC,H] = \sum_{a+b+c = n} \frac{n!}{a!b!c!}[^aA,H][^bB,H][^cC,H]
\end{equation}

\section{Annihilation and creation operators}
%##########################################################

Beginning with

\[
[a_\psi,a_\phi^\dag] = a_\psi a_\phi^\dag - a_\phi^\dag a_\psi =
\begin{cases}
 1 & \text{if }\psi=\phi \\
 0 & \text{otherwise} \\
 \end{cases}
\]

\section{$a^\dag$ and $a$ operators}
%##########################################################

\begin{equation}
[a^m, a^\dag] = ma^{m-1}
\end{equation}

\begin{equation}
[a,a^{\dag m}] = ma^{\dag(m-1)}
\end{equation}
the above holds for all m, even -ve.
\begin{equation}
[a^-,a^m] = ((a^- + a)^m - a^m)a^-
\end{equation}

\begin{equation}
[a^n,a^{\dag m}] = \sum_{q=1}^{\min(m,n)} \frac{m!n!}{q!(m-q)!(n-q)!} a^{\dag m-q}a^{n-q}
\end{equation}

\begin{equation}
[a^{\dag p}a^m, a^{\dag q}a^n] =
a^{\dag p}[a^m, a^{\dag q}] a^n -
a^{\dag q} [a^n, a^{\dag p}]a^m
\end{equation}

\section{$a^-$ operator}
%##########################################################

Define the $a^-$ operator such that
\[
a^-a^\dag = I
\]
where $I$ is the identity operator. Given this we can see immediately that $[a^\dag,a^-a^\dag] = 0$, so
\[
a^\dag a^-a^\dag -a^-a^\dag a^\dag = [a^\dag,a^-]a^\dag = 0
\]
So, for all states, $S$ other than the ground state,
\[
[a^\dag,a^-]S = 0
\]
For the ground state, $\emptyset$, in order to ensure $[a^\dag,a^-]\emptyset = 0$ we define
\[
a^\dag (a^-\emptyset) = \emptyset
\]
However, such terms as $a^-\emptyset$ will never arise through annihilation operators as they will always be multipllied by zero.

\section{$L_{ir}$ operator}
%##########################################################

Changes $\lambda_i$ to $(1-r)\lambda_i$

\begin{equation}
[L_{ir}, a_i^\dag] = 0
\end{equation}

\begin{equation}
[L_{ir}, a_i] = r\lambda_i L_{ir}
\end{equation}

\begin{equation}
[L_{ir}, a_i^n] =  L_{ir}(a_i^n - (a_i-r\lambda_i)^n)
\end{equation}

\section{$g_{ir}$ operator}
%##########################################################

Multiplies each basis by $(1-r)^{\Delta_i}$

\[
g_{ir}D_0 = 1
\]

\begin{equation}
[a_i^\dag, g_{ir}] = ra_i^\dag g_{ir}
\end{equation}

\begin{equation}
[a_i, g_{ir}] = \frac{r(\lambda - a_i)}{1-r}g_{ir}
\end{equation}
so
\begin{equation}
g_{ir}a_i^{\dag n} = (1-r)^na_i^{\dag n}g_{ir}
\end{equation}
and
\begin{equation}
[g_{ir}, a_i^{\dag n}] = ((1-r)^n - 1)a_i^{\dag n}g_{ir}
\end{equation}
also
\begin{equation}
g_{ir}a_i = \frac{a_i-r\lambda_i}{1-r}g_{ir}
\end{equation}
so
\begin{equation}
g_{ir}a_i^n = \left(\frac{a_i-r\lambda_i}{1-r}\right)^ng_{ir}
\end{equation}
\begin{equation}
g_{ir}a_i^n = (1-r)^{-n}\sum_{m=0}^n{n \choose m} a_i^m(-r\lambda_i)^{n-m}g_{ir}
\end{equation}

and
\begin{equation}
[g_{ir},a_i^n] = \left((1-r)^{-n}\sum_{m=0}^n{n \choose m} a_i^m(-r\lambda_i)^{n-m} - a_i^n\right) g_{ir}
\end{equation}
so
\begin{equation}
[g_{ir},a_i^{\dag n}a_i^m] = \left((1-r)^{n-m}\sum_{l=0}^m{m \choose l} a_i^{\dag n}a_i^l(-r\lambda_i)^{m-l} - a_i^{\dag n}a_i^m\right) g_{ir}
\end{equation}

\section{$Lg_{ir}$ operator}
%##########################################################

We can join the $L$ and $g$ operators into a compound operator that multiplies bases by $(1-r)^\Delta_i$ and multiplies $\lambda_i$ by $(1-r)$. It has the properties:

\begin{equation}
[Lg_{ir}, a_i^\dag] = -ra_i^\dag Lg_{ir}
\end{equation}

\begin{equation}
[Lg_{ir}, a_i] = \frac{ra_i}{1-r} Lg_{ir}
\end{equation}

So
\begin{equation}
Lg_{ir}a_i^{\dag n}a_i^m = (1-r)^{n-m}a_i^{\dag n}a_i^mLg_{ir}
\end{equation}
and
\begin{equation}
[Lg_{ir},a_i^{\dag n}a_i^m] = ((1-r)^{n-m} - 1)a_i^{\dag n}a_i^mLg_{ir}
\end{equation}

%{\footnotesize \bibliographystyle{acm}
%\bibliography{sample}}

%\theendnotes

\end{document}


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 12.1 One-Dimensional Table Look Up, Interpolation, \hbox{and Differentiation}

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

Given a table, $(x_i$, $y_i)$, of independent variable values and the
corresponding dependent variable values, this subroutine finds the
points in the table closest to a given value of the independent
variable and uses these points to interpolate for the corresponding
value of the dependent variable.  Error estimates, Hermite
interpolation (i.e.\ using $(x_i$, $y_i$, $y_i^\prime)$), different
look up methods, and the computation of derivative information are
available.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision}

\begin{description}
\item[INTEGER]  \ {\bf NTAB, NDEG, LUP, IOPT}$(\geq k_1)$\newline
[$k_1 \geq 2$ depends on options used.]

\item[REAL]  \ {\bf X, Y, XT}$(\geq $NTAB){\bf , YT}$(\geq $NTAB){\bf ,\newline
EOPT}$(\geq k_2$) [$k_2 \geq 1$ depends on options used .]
\end{description}
\begin{center}
\fbox{\begin{tabular}{@{\bf }c}
CALL SILUP (X, Y, NTAB, XT, YT,\\
NDEG, LUP, IOPT, EOPT)\\
\end{tabular}}
\end{center}

\subsubsection{Argument Definitions}

\begin{description}
\item[X]  [in] Independent variable where value of interpolant is desired.

\item[Y]  [out] Value of interpolant.

\item[NTAB]  [in] Number of points in the table, $\geq \text{NDEG}+k$,
  where $k=1$ if NDEG is odd and 2 if NDEG is even, and must be one
  bigger than this if an error estimate is requested.

\item[XT()]  [in] Array of independent variable values, XT($i)=x_i$. Must be
monotone increasing or monotone decreasing, but this is not checked
for. (XT(I$)=$ XT(I+1) is permitted, but has a special meaning, see
Section~C.) If the $x_i$ are equally spaced one should set LUP (below) to~3,
and provide $x_1$ in XT(1) and $x_2-x_1$ in XT(2).

\item[YT()]  [in] Array of dependent variable values. YT($i)=y_i.$

\item[NDEG]  [in] Nominal degree of the polynomial to be used in the
interpolation. Require $0\leq $ NDEG $\leq 15.$

For NDEG $=0$, Y is set to YT($k)$, where $k$ minimizes $|\text{X} -x_k|.$

For NDEG odd, Y is the result of standard polynomial interpolation of degree
NDEG using NDEG + 1 points. The interpolating function is continuous, but
the first derivative is usually discontinuous at tabular points.

For NDEG $>$ 0 and even, Y is obtained from a function that is a linear
combination of two polynomials of degree NDEG. This function is a polynomial
of degree NDEG + 1 using NDEG + 2 points that interpolates all but the two
outer points. It has a continuous first derivative but the second derivative
is usually discontinuous at tabular points.

Usually $|error|$ in the interpolant will tend to decrease in a fairly
regular way for increasing values of NDEG until either errors due to a lack
of precision in YT, or the inherent instability of high degree polynomial
interpolation causes errors to get worse. If one wants a continuous first
derivative, even values of NDEG $>$ 0 should be used.

\item[LUP]  [inout] Defines the type of look up method. (Changed only if LUP
$\leq 0$ on input.)

\begin{itemize}
\item[$\leq 0$]  \ If LUP $=0$, start with a binary search, else do a
sequential search starting with an index of $-$LUP. On exit, LUP is set to $%
-k$, where $k$ minimizes $|$X $-$ XT($k)|.$
\item[$= 1$]  Binary search. Use this when accesses are not sequential, and
XT values are not close to being equally spaced.
\item[$= 2$]  Start a sequential search with an index $=[1.5+($NTAB$-$1)
(X $-$ XT(1))/(XT(NTAB) $-$ XT(1))]. Use this when points are almost equally
spaced and accesses are not sequential.
\item[$= 3$]  YT(I) corresponds to XT(1) + (I$-$1)*XT(2), no search is
required to do the look up and XT can have dimension XT(2). This is
recommended when the XT values are equally spaced since it takes the least
space and should usually be fastest.
\item[$= 4$]  Internal information connected with the $x_k$ values used in
the last interpolation is reused. (Don't use this value if there is any
chance for an intervening call to this subroutine.) Only YT should be
changed; IOPT is not examined. This option saves time when interpolating
components after the first of a vector valued function.
\end{itemize}

\item[IOPT()]  [inout] IOPT(1) is used to return a status as follows:

\begin{itemize}
\item[$-$10 ]  \ An option index is out of range.
\item[$-$9 ]  \ NTAB is outside allowed limits.
\item[$-$8 ]  \ NDEG is outside allowed limits.
\item[$-$7 ]  \  LUP $>$ 3 when program was not ready for it.
\item[$-$6 ]  \ Option 3 (compute derivatives), has requested more than
15~derivatives.
\item[$-$5 ]  \ LUP = 3, and XT(2) = 0.
\item[$-$4 ]  \ XT(1) = XT(NTAB), and NTAB is not~1.
\item[$-$3 ]  \ Bad points (see option 6) mean only 0 or 1 points were
available for interpolation..
\item[$-$2 ]  \ There is only one table entry; the estimated error that was
requested was not computed.
\item[$-$1 ]  \ The accuracy requested was not obtained.
\item[0 ]  \ Normal return, no exceptional conditions.
\item[1 ]  \ X was outside the domain of the table, extrapolation used.
\item[2 ]  \ Available table values were so few that this restricted the
degree of the polynomial.  A valid error estimate is not returned in this
case.
\end{itemize}

Starting with IOPT(2) options are specified by integers in the range 0 to~7,
followed in some cases by integers providing argument(s) for the option.
Each option, with its arguments if any, is followed in IOPT by the next
option or by a~0.  If an option index is specified more than once,
only the last specification is used.

\begin{itemize}
\item[0]  End of the option list; this must always be the last option
specified in IOPT.

\item[1]  An error estimate is to be returned in EOPT(1).

\item[2]  (Argument: K2) K2 gives the polynomial degree to use when
extrapolating. The default for K2 is NDEG if NDEG $\leq $ 2 or if NDEG is
even, else it is NDEG~$-$~1.

\item[3]  (Arguments: K3, L3) Save $(k^{th}$ derivative of interpolating
polynomial$)/k!$ in EOPT(K3+$k-$1) for $k=1$, 2, ..., L3. These values are
the coefficients of the polynomial in the monomial basis expanded about X.
Require $0\leq $ L3 $\leq 15.$

\item[4]  (Argument K4) The absolute and relative errors expected in YT
entries are specified in EOPT(K4) and EOPT(K4+1) respectively. The values
provided here are used in estimating the error in the interpolation. An
error estimate is returned in EOPT(1).

\item[5]  (Argument K5, L5) Do the interpolation to the accuracy requested
by the absolute error tolerance specified in EOPT(K5) and the relative error
tolerance in EOPT(K5+1) respectively. An attempt is made to keep the final
error $<$ EOPT(K5) + EOPT(K5+1)\thinspace ($|$YT($k)|$ + $|$YT($k^{\prime
})|)$, where $k$ and $k^{\prime }$ are indices for table values close to X.
Standard polynomial interpolation is done, but here NDEG gives the maximal
degree polynomial to use in the interpolation. The actual degree used in
doing the interpolation is stored in the space for the argument L5. If both
EOPT's specified are $\leq $ 0, IOPT(1) is not set to $-$1, and no error
message is generated due to an unsatisfied accuracy request. An error
estimate is returned in EOPT(1).

\item[6]  (Argument K6) Do not use point (XT$(k)$, YT($k))$ in the
interpolation if YT($k$) = EOPT(K6). This option is useful if one has a table
with equally spaced points, but with some bad data points, and may also be
used by the multiple dimensional interpolation subroutine. Points are
selected for use in the interpolation as if points flagged with YT($k$) =
EOPT(K6) were not present.

\item[7] (Argument K7) YT(K7+$k$) gives the first derivative
corresponding to the function value in YT($k$).  These derivatives
are to be used in doing the interpolation.  One gets a continuous
interpolant only for NDEG = 3, 7, 11, and 15; these cases also give a
continuous first derivative in the interpolant.  The interpolating
polynomial satisfies $p$(XT($k$)) = YT($k$), $p^\prime $(XT($k$)) =
YT(K7+$k$) for values of $k$ that give values of XT close to X.
(Section~D describes how points are selected.) If NDEG is even, a
value of YT($k$) is used without using the corresponding value of
YT(K7+$k$).

\end{itemize}

\item[EOPT]  [inout] Array used to return an error estimate and used for
options.

\begin{description}
\item[EOPT$(1)$]  [out] contains an estimate of the error in the
interpolation if an error estimate has been requested by setting
option 1, 4 or~5.

\item[EOPT$(>1)$]  [in or out] for use by options~3--6.
\end{description}
\end{description}

\subsubsection{Modifications for Double Precision}

Change SILUP to DILUP, and the REAL type statement to DOUBLE PRECISION.

\subsection{Examples and Remarks}

DRSILUP below is a sample program that interpolates in a table of $\sin
(x)$ given with a spacing of~0.5 for NDEG = 2 to~10, and obtains an error
estimate. ODSILUP below gives the results of running DRSILUP on an IBM PC,
which uses IEEE 32~bit floating point arithmetic.

The user is reminded that polynomial interpolation of high degree is
hazardous, where ``high" depends strongly on the kind of data being
interpolated. The error estimates provided by the program are usually
greater than the actual error, but on any given interpolation may be much
too small. Similarly, the order selected by the program to satisfy a given
error criterion will usually do a good job, but will (infrequently) use too
low a degree due to an overly optimistic error estimate or an overly
pessimistic conclusion that the corrections are starting to diverge. When
many interpolations are going to be done in a table we recommend trying
several degrees at enough points to cover the kinds of functional behavior
in different areas of the table and examining the error estimates. Then
later interpolations can be done using the fixed degree that appears best
for the accuracy desired.

If one has a table with discontinuities, one can set XT(I) = XT(I+1) =
the point of the discontinuity. The $x_j$'s that are used in the
interpolation will be selected so that all $j$ satisfy $j \leq $ I, or
$j \geq$ I + 1.  If the discontinuity lies between two points, use
option 6 to define the value at the point of discontinuity.

\subsection{Functional Description}

The look up process identifies the point in the table, XT($k)$, nearest to
the input value X.  To obtain continuity in the interpolant, remaining
points are selected one at a time, keeping the number of points on either
side of X balanced as long as this is possible.  Thus XT($k)$ and
XT($k^{\prime})$, the second entry selected, will bracket X if
extrapolation is not required.  When NDEG $>0$ is even (and Hermite
interpolation is not being used), a linear combination of the polynomials
of degree NDEG interpolating the left NDEG + 1 and the right NDEG + 1
points is used.  (A total of NDEG + 2 points is used.) If it is not
possible to select $(\text{NDEG}+2) / 2$ points on either side of X, then
just NDEG + 1 points are used to obtain a polynomial of degree NDEG as in
the odd case.  When extrapolating, the default action is to use standard
polynomial interpolation of degree $2\times \max (1,\lfloor
$NDEG$/2\rfloor )$.  This default can be changed using an option value
of~2 in IOPT().  Interpolations are done using the Newton divided
difference form of the interpolating polynomial as described for
Algorithms I (NDEG odd) and IV (NDEG$>0$, even) by Krogh in
\cite{Krogh:1970:EAP}.  The derivatives are computed as described for
Algorithm V in \cite{Krogh:1970:EAP}.

Define $E_{Min}$ by the following:
\begin{equation*}
E_{Min}=E_{Abs}+E_{Rel\,}(|\text{YT}(i)|+|\text{YT}(i^{\prime })|),
\end{equation*}
where $E_{Abs}$ = EOPT(K4) if option~4 is used, and is otherwise~0; $E_{Rel}$
= EOPT(K4+1) if option~4 is used and is otherwise the smallest positive
floating point number that gives a number different from~1 when added to~1;
and $i$ and $i^{\prime}$ are the indices for the first two table entries
selected for use in the interpolation.

Then the error is estimated by

\begin{equation*}
\hspace{-5pt}\text{error est.}=1.5\left( |y_c-P_{k+1}|+
\frac{|P_k-P_{k-1}|}{32}\right)+E_{Min},
\end{equation*}
where $y_c$ is the value being returned for Y, $P_k$ is the result obtained
by standard polynomial interpolation of degree $k$, and $k$ is NDEG unless
the degree has been reduced because of extrapolation in which case $k$ is
the degree actually used. If there are not sufficient points available to
compute $P_{k+1}$, then the error estimate is given by~1.5 $%
|P_k-P_{k-1}|+E_{Min}.$

The choice of degree for option~5 makes use of the following quantities.
\begin{align*}
e_0 &= |P_0|\\
\overline{e}_1&=\max (.75,\ e_1/(e_1+e_0)\\
e_j &= |P_j - P_{j-1}|,\quad j = 1,\ 2,\ ...\\
\overline{e}_j&=\frac 12\overline{e}_{j-1}+e_j/(e_j+e_{j-1}),\quad j=2,
 3,\ ...\\
\widehat{e}_j&=\frac 12e_{j-1}\overline{e}_{j-1}/j,\quad j=2,\ 3,\ ...
\end{align*}
The value of $j$ is allowed to increase until either $e_j+e_{j-1}\leq $
error request, or $e_j\geq \widehat{e}_j$ for two successive steps with $j>1$%
. The quantity, $\overline{e}_j$ indicates the rate of decrease in $e_j$ for
successive values of $j$. A good rate of decrease in the past causes a
better rate to be required in the future. Also, better convergence is
required as the degree gets larger. The final error estimate is given by~1.5
$(e_j+.0625e_{j-1})+E_{Min\text{.}}$ When derivatives are being computed,
the error request is decreased by the factor $|(x-x_i)/(x_i-x_{i^{\prime }})|
$, where $i$ and $i^{\prime }$ are the indices for the first two table
entries selected.

\bibliography{math77}
\bibliographystyle{math77}

\subsection{Error Procedures and Restrictions}

Values of IOPT$(1) < 0$ ordinarily cause an error message to be printed, and
those $< -3$ do not ordinarily result in a return to the user. One can
change the action on errors by calling the message/error routine MESS
of Chapter~19.3 before calling this routine.

\subsection{Supporting Information}

The source language is ANSI Fortran~77.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DILUP & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, DILUP, DMESS, MESS\rule[-5pt]{0pt}{8pt}}\\
SILUP & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, MESS, SILUP, SMESS}\\
\end{tabular}

Subroutine designed and written by: Fred T. Krogh, JPL, May~1991.
Hermite interpolation added December~1994.

\begcodenp
\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSILUP}\vspace{10pt}
\lstinputlisting{\codeloc{silup}}
\newpage

\vspace{30pt}\centerline{\bf \large ODSILUP}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{silup}}
\end{document}


\section{08/24}
This course will involve only a few techniques with a variety of applications.
Many domains, such as Distributed Computing, Networks, etc., all depend on
randomized algorithms.

We will focus on a few techniques that lead to understanding.
\begin{enumerate}
    \item Union Bound
    \item Linearity of Expectation
    \item Markov's Inequality
    \item Chernoff Bounds
\end{enumerate}
These four things prove very useful in the design and understanding of
algorithms.

\subsection{Basic Definitions}
\begin{definition}{}{}
    A \emph{Sample Space} is a set $S$ whose elements consist of \emph{simple
    events} (also called \emph{elementary} events). When $S$ is finite or
    countably infinite, we say it is a \emph{Discrete} sample space.
\end{definition}

\begin{definition}{}{}
    An \emph{event} in a sample space $S$ is a subset of $S$.
\end{definition}

\begin{definition}{}{}
    A \emph{Probability Distribution} on $S$ is a function $\Pr: 2^S \to [0, 1]$
    that satisfies
    \begin{enumerate}
        \item $\prob{S} = 1$
        \item If $E_1$, $E_2$, $\dots$ are pairwise disjoint events (i.e., $E_i
        \cap E_j = \emptyset$ for all pairs $i, j$, also called mutually
        exclusive), ed by some finite or countably infinite set $I$, then
        \[\prob{\bigcup_{i \in I} E_i} = \sum_{i \in I} \prob{E_i}\]
    \end{enumerate}
\end{definition}

\subsection{Conditional Probability}
The expression $\prob{E_2 \given E_1}$ is read ``the probability of $E_2$ given
$E_1$.'' For example, if we randomly select a person from Texas, we might write
\begin{align*}
    E_1 &= \hbox{person chosen is in Houston}\\
    E_2 &= \hbox{person chosen is a UH student}
\end{align*}
in which case $\prob{E_2 \given E_1}$ is simply the probability that a randomly
selected person from Texas is a UH student \emph{given that they are in
Houston}.

\begin{definition}{Conditional Probability}{}
    The conditional probability $\prob{E_2 \given E_1}$ is defined
\[\prob{E_2 \given E_1} = \frac{\prob{E_2 \cap E_1}}{\prob{E_1}}\]
\end{definition}

Intuitively, this can be thought of as taking the probability that $E_2$ and
$E_1$ occur and ``normalizing it'' by dividing by the probability that $E_1$
occurs.

\subsection{Independence}
\begin{definition}{Independent Events}{}
Two events, $E_1$ and $E_2$, are \emph{independent} if
\[\prob{E_1 \cap E_2} = \prob{E_1}\prob{E_2}\]
or, equivalently, if
\[\prob{E_1 \given E_2} = \prob{E_1}\]
\end{definition}
For example, suppose $C_1$ and $C_2$ are the outcomes of two fair coin tosses.
These are independent, since
\[\prob{C_1 = H \cap C_2 = T} = \prob{C_1 = H}\prob{C_2 = T} = \frac{1}{4}\]
Independence is not always related to physical independence. For example, say we
are given a fair die and let
\begin{align*}
    E_1 &= \hbox{roll is even}\\
    E_2 &= \hbox{roll is less than or equal to 4}
\end{align*}
In this case, we can enumerate the sample space and explicitly determine
$\prob{E_1}$, $\prob{E_2}$, $\prob{E_1 \cap E_2}$, and $\prob{E_1}\prob{E_2}$,
to see if the events are independent:
\begin{align*}
    E_1 &= \set{2, 4, 6}\\
    E_2 &= \set{1, 2, 3, 4}\\
    E_1 \cap E_2 &= \set{2, 4}
\end{align*}
Then
\begin{align*}
    \prob{E_1} &= \frac{3}{6} = \frac{1}{2}\\
    \prob{E_2} &= \frac{4}{6} = \frac{2}{3}\\
    \prob{E_1 \cap E_2} &= \frac{2}{6} = \frac{1}{3}\\
    \prob{E_1}\prob{E_2} &= \frac{1}{2} \cdot \frac{2}{3} = \frac{1}{3}
\end{align*}
Thus, we see the events are independent. On the other hand, if $E_2$ is the
event that the roll is \emph{strictly less} than 4, we have
\begin{align*}
    E_1 &= \set{2, 4, 6}\\
    E_2 &= \set{1, 2, 3}\\
    E_1 \cap E_2 &= \set{2}
\end{align*}
Then
\begin{align*}
    \prob{E_1} &= \frac{3}{6} = \frac{1}{2}\\
    \prob{E_2} &= \frac{3}{6} = \frac{1}{2}\\
    \prob{E_1 \cap E_2} &= \frac{1}{6}\\
    \prob{E_1}\prob{E_2} &= \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}
\end{align*}
and we see that the events are \emph{not independent}.

\subsection{The Inclusion-Exclusion Principle}
%TODO insert venn diagram
A basic result in set theory is that
\begin{align*}
    \card{A \cup B} &= \card{A} + \card{B} - \card{A \cap B}\\
    \card{A \cup B \cup C} &= \card{A} + \card{B} + \card{C} - \card{A \cap B} - \card{A \cap C} - \card{B \cap C} + \card{A \cap B \cap C}
\end{align*}
which can be generalized to an arbitrary finite union by
\[\bigcup_{i=1}^n A_i = \sum_{k=1}^n (-1)^{k+1} \left(\sum_{1 \leq i_1 < \dots < i_k \leq n} \card{A_{i_1} \cap \cdots \cap A_{i_k}} \right)\]
or equivalently
\[\card{\bigcup_{i=1}^n A_i} = \sum_{\emptyset \neq J \subseteq \set{1,\dots,n}}
(-1)^{\card{J}+1} \card{\bigcap_{j\in J} A_j}\]
This yields the probability formulas
\begin{theorem}{Inclusion-Exclusion Principle}{}
    For any events $E_1$, $E_2$, \dots, $E_n$,
\begin{align*}
    \prob{E_1 \cup E_2} &= \prob{E_1} + \prob{E_2} - \prob{E_1 \cap E_2}\\
    \prob{\bigcup_{i=1}^n E_i} &= \sum_{\emptyset \neq J \subseteq \set{1,\dots,n}}
(-1)^{\card{J}+1} \prob{\bigcap_{j\in J} E_j}
\end{align*}
\end{theorem}

\subsection{Union Bound}
\begin{theorem}{Union Bound}{}
    Let $E_1$, $E_2$, \dots, be any countable set of events. Then
\[\prob{\bigcup E_i} \leq \sum\prob{E_i}\]
\end{theorem}
While this bound is often not very precise, it is useful in many cases where the
events $E_i$ are ``bad'' and we can bound the likelihood of a single $E_i$. This
allows us to bound the likelihood of \emph{any} $E_i$.

\subsection{Conditioning on Multiple Events (Chain Rule)}
By repeatedly applying the definition of conditional probability, we have
\begin{align*}\prob{E_1 \cap E_2 \cap \dots \cap E_n}
    &= \prod_{i=1}^n \prob{E_i \given E_1 \cap E_2 \cap \dots \cap E_{i-1}}\\
    &= \prob{E_1}\prob{E_2 \given E_1}\cdots \prob{E_n \given E_1 \cap E_2 \cap \dots \cap E_{n - 1}}\\
\end{align*}

\subsection{Birthday Paradox}
\begin{problem}{Birthday Paradox}{}
Suppose $n$ people are in a room and they have birthdays chosen uniformly at
random from the 365 calendar days. What is the probability that two people share
a birthday?
\end{problem}

The probability that \emph{no two} people share a birthday can be calculated by
considering the following events:

\begin{align*}
    E_1 &= \hbox{person 1 has a birthday}\\
    E_2 &= \hbox{person 2 has a different birthday than person 1}\\
    \phantom{..}&\vdots\\
    E_i &= \hbox{person $i$ has a different birthday than people 1 through $i - 1$}
\end{align*}

Specifically, the probability that no two people share a birthday is
\begin{align*}\prob{E_1 \cap E_2 \cap \dots \cap E_n}
    &= \prob{E_1}\prob{E_2 \given E_1}\cdots \prob{E_n \given E_1 \cap E_2 \cap \dots \cap E_{n - 1}}\\
    &= 1 \times \left(1 - \frac{1}{365}\right) \times \left(1 - \frac{2}{365}\right) \times \cdots \times \left(1 - \frac{365 - n + 1}{365}\right)
\end{align*}

For what value of $n$ is the above probability less than $\sfrac{1}{2}$?


\section{Mathematics components}
\comp{const}
{Output a constant}
{}{out:float}
Outputs a constant value.
\startparams
val & float & value to output\\
\end{tabularx}

\comp{add}
{Adds the inputs together}
{in1:float,in2:float}{$mul1\times(in1+add1) + mul2\times(in2+add2)$}
Adds together the inputs, after they have been processed through
some parameters.
\startparams
add1 & float & added to input 1 before multiplication by mul1\\
add2 & float & added to input 2 before multiplication by mul2\\
mul1 & float & multiplies $in1+add1$\\
mul2 & float & multiplies $in2+add2$\\
\end{tabularx}

\comp{mul}
{Multiplies the inputs together.}
{in1:float,in2:float}{$mul1\times(in1+add1) \times mul2\times(in2+add2)$}
Multiplies together the inputs, after they have been processed through
some parameters. Yes, the two multiplies are redundant.
\startparams
add1 & float & added to input 1 before multiplication by mul1\\
add2 & float & added to input 2 before multiplication by mul2\\
mul1 & float & multiplies $in1+add1$\\
mul2 & float & multiplies $in2+add2$\\
\end{tabularx}

\comp{mul}
{Clamps the input to a range}
{in:input}{output clamped to range}
Clamps the output to ensure it is within a given range.
\startparams
min & float & minimum value\\
max & float & maximum value\\
\end{tabularx}

\clearpage
\comp{func}
{Performs one of a number of functions on the sum of the inputs.}
{in1:float,in2:float}{$addout+mulout\times f \big(mul1(in1+add1)+ mul2(in2+add2)\big)$}
Takes the two inputs and processes them through multiply and add
parameters, puts the result through a function, and then puts the
output through another multiply and add stage. If an input is
not connected, it is zero. Thus, the output is:
\[
p_{addout} + p_{mulout} \times f\Big((p_{add1}+in_1)\times p_{mul1} +
(p_{add2}+in_2)\times p_{mul2} \Big)
\]
Functions supported include: sine, cosine, gaussian(x+y), x mod y, abs(x+y).
\startparams
add1 & float & added to input 1 before multiplication by mul1\\
mul1 & float & multiplies $in1+add1$\\
add2 & float & added to input 2 before multiplication by mul2\\
mul2 & float & multiplies $in2+add2$\\
func & enum & function to perform\\
mulout & float & multiplies function output\\
addout & float & added to output
\end{tabularx}

\clearpage
\comp{perlin}{Perlin coherent noise generator (input is values or current time)}
{x:float,y:float}{out:perlin(x+y),zerocrosstrig:int}
Calculates
\[
p_{addout} + p_{mulout} \times \operatorname{perlin}\Big((p_{add1}+in_1)\times p_{mul1} +
(p_{add2}+in_2)\times p_{mul2} \Big)
\]
where \emph{perlin} is the Perlin noise function with range [-1,1].
Thus, it generates
smoothly varying coherent random noise. If neither input is connected,
$in_1$ is the current time. If the output crosses zero, i.e. the previous
output was one side of zero and the new output is the other side,
the \emph{zerocrosstrig} output is set high. This is typically used
with no inputs to generate randomly timed events, with \emph{mul1} set
to control the frequency.
\startparams
octaves & int & number of octaves in noise function -- the higher, the
more ``fractal'' the noise and the ``busier'' it becomes\\
persistence & float & the noise persistence -- how much the amplitudes
decrease for each successive octave\\
add1 & float & added to input 1 (or time if no inputs are connected)
before multiplication by mul1\\
mul1 & float & multiplies $in1+add1$\\
add2 & float & added to input 2 before multiplication by mul2\\
mul2 & float & multiplies $in2+add2$\\
mulout & float & multiplies function output\\
addout & float & added to output
\end{tabularx}
For more information on \emph{octaves} and \emph{persistence}, and Perlin
noise is general, see the documentation for \texttt{libnoise}.


\chapter[Synchrotron radiation]{Synchrotron radiation}

\section{Method}

The spectral distribution of the mean photon number of synchrotron radiation
$d\bar{N}/d\omega$ produced by an ultrarelativistic electron moving in a
constant uniform magnetic field along a trajectory with length $L$ can be
expressed by definition in terms of the spectrum of the mean energy loss of
the radiation $d\bar{\Delta}/d\omega$  by the following expression \cite{maier}:

\begin{equation}
\frac{d\bar{N}}{d\omega} = \frac{1}{\omega}\frac{d\bar{\Delta}}{d\omega} =
\frac{\sqrt{3}}{2\pi}\alpha\left(\frac{L\gamma}{R}\right)\frac{1}{\omega_{c}}
\int_{\omega/\omega_{c}}^{\infty}K_{5/3}(\eta)d\eta   .
\end{equation}

Here:
\begin{eqnarray*}
\omega    &  &  \mbox{photon energy}\\
\alpha    &  &  \mbox{fine structure constant}\\
R         &  &  \mbox{instantaneous radius of curvature of the trajectory}\\
K         &  &  \mbox{Macdonald function}\\
\omega_c  &  &  \mbox{characteristic energy of synchrotron radiation}\\
\end{eqnarray*}
$\omega_c = 1.5\beta(\hbar c/R)\gamma^3$, $\hbar$ is the Planck constant,
and $\beta = v/c$  is the ratio of the electron velocity $v$  to the speed of
light $c$). In the SI system of units: $R(m) = P(GeV/c)/0.3B_{\bot}(T)$ ,
where  $B_{\bot}$ is the component of magnetic flux density perpendicular to
the electron velocity, and $P$  is the electron momentum (for the case
considered $R \approx const$   along the whole trajectory) . For the
simulation of the energy spectrum of synchrotron radiation by the Monte-Carlo
method we need to calculate the integral distribution of the mean number of
photons with the energy more than the given one: $\bar{N}_{>\omega}$  .
Transforming  $d\bar{N}/d\omega$ by the integral representation of
the Macdonald function \cite{abram}, we have :

\begin{eqnarray}
\bar{N}_{>\omega}& = &
\int_{\omega}^{\infty}\frac{d\bar{N}}{d\omega'}d\omega' \nonumber\\
& = &
\frac{\sqrt{3}}{2\pi}\alpha\left(\frac{L\gamma}{R}\right)
\int_0^{\infty}\frac{\cosh\left(\frac{5}{3}t\right)}{\cosh^2(t)}
\exp\left[-\frac{\omega}{\omega_c}\cosh(t)\right]dt .
\end{eqnarray}

The latter integral is calculated numerically by the quadrature Laguerre
formula \cite{korn} . The calculations show that taking into account
about 50 roots of
the Laguerre polynomials results in an accuracy of the integral estimation
better than $10^{-4}$ [Bag98] . The mean number of synchrotron radiation photons
$\bar{N}$  (= $\bar{N}_{>0}$) produced by an ultrarelativistic electron
moving in a constant magnetic field along a trajectory with the length $L$
  , is given by :

\begin{eqnarray}
\bar{N} = \bar{N}_{>0}& = &
\frac{\sqrt{3}}{2\pi}\alpha\left(\frac{L\gamma}{R}\right)
\int_0^{\infty}\frac{\cosh\left(\frac{5}{3}t\right)}{\cosh^2(t)}dt \nonumber\\
& = &
\frac{5}{2\sqrt{3}}\alpha\left(\frac{L\gamma}{R}\right) \approx
10^{-2}\left(\frac{L\gamma}{R}\right) .
\end{eqnarray}

Qualitatively this result can be manipulated using the fact that the mean
number of photons produced along the formation zone length
$z \approx R/\gamma$  is proportional to $\alpha$  .
 Then for the length $L$ , $\bar{N} \approx \alpha L/(R/\gamma)$ . Note, that
for the ultrarelativistic case $\gamma\gg1$   , when $R \sim \gamma$ ,
$\bar{N}$ does not depend on the electron energy but is defined by the
values of $L$ and $B_{\bot}$ only. The mean energy loss for synchrotron
radiation $\bar\Delta$ , corresponding to a trajectory with the length $L$,
experiences the essential relativistic rise instead :

\begin{equation}
\bar{\Delta} = \int_0^{\infty}\omega\frac{d\bar{N}}{d\omega}d\omega =
\frac{2}{3}\alpha\hbar c\left(\frac{L\gamma^2}{R^2}\right)\beta\gamma^2 =
\frac{8\bar{N}}{15\sqrt{3}}\omega_{c}
\approx 0.31\bar{N}\omega_{c} \sim \gamma^2
\end{equation}

The angular distribution of synchrotron radiation produced by ultrarelativistic
electron shows the clear 'searchlight' effect. The main number of photons is
radiated in the angle limit of the order of $1/\gamma$   around the electron
trajectory direction. In the interesting region of $\gamma > 10^3$
the angular resolution of
X-ray and gamma detectors usually does not allow us to measure the details
of the angular distribution of synchrotron radiation. Therefore, the angular
distribution is roughly set to be flat in the range $0 - 1/\gamma$ .

\section{Simulation of synchrotron radiation}

The distance $x$ along electron/positron trajectory to the next point of
creation of synchrotron radiation
photon is simulated according to exponential distribution,
$exp(-x\bar{N}/L)$. The energy $\omega$ of synchrotron radiation photon
is simulated
according the distribution $\bar{N}_{>\omega}/\bar{N}$. The angles $\theta$
and $\varphi$ counted from the electron trajectory (the electron momentum
is considered to be parallel to the local z-axis) are rouhgly set to be
distributed randomly in the ranges
$0 - 1/\gamma$ and $0 - 2\pi$ respectively

\section{Status of this document}
15.10.98 created by V.Grichine

\begin{thebibliography}{99}

\bibitem[Abram64]{abram} Edited by M. Abramovwitz and I.A. Stegan.
{\em Handbook of Mathemaatical Functions,
 NBS Applied Mathematics Series 55} (1964)

\bibitem[Bag98]{bag} Bagulya A.V. and Grichine V.M.
{Bulletin of Lebedev Institute, no.9-10, 7 } (1998)

\bibitem[Korn61]{korn}G.A. Korn and T.M. Korn.
{\em Mathematical Handbook for scientists and
    engineers, McGRAW-HILL BOOK COMPANY, INC} (1961)

\bibitem[Maier91]{maier}  R. Maier.
{\em Synchrotron Radiation, CERN Report 91-04, 97-115} (1991)

\end{thebibliography}


\section{The Reals}
\subsection{The Need of the Reals}
We already have $\mathbb N, \mathbb Z, \mathbb Q$, then why are we introducing the reals?
Why don't we stop at $\mathbb Q$?
\begin{proposition}
    There is no rational $x$ with $x^2=2$.
\end{proposition}
we can assume $x$ is positive since $(-x)^2=x^2$.
\begin{proof}
    Suppose we have some rational $x$ with $x^2=2$, then suppose $x=a/b$ for some positive integers $a,b$.
    So $2b^2=a^2$, but the power of $2$ in the prime factorization of $a^2$ is even but that in $2b^2$ is odd.
    This is a contradiction.
\end{proof}
\begin{remark}
    Same argument shows that if $k\in\mathbb N$ is $x^2$ for some $x\in\mathbb N$, then $k$ is a perfect square.
\end{remark}
\begin{proof}[Alternative proof]
    Again suppose that $x=a/b$ where $a,b\in\mathbb N$ has $x^2=2$.
    So every $cx+d$ where $c,d\in\mathbb Z$ is of the form $e/b$ for some $e\in\mathbb Z$.
    So $cx+d>0\implies cx+d\ge 1/d$, but then $0<x-1<1$, so $0<(x-1)^n<1/b$ for $n$ large, which is a contradiction since all $(x-1)^n$ is of the form $cx+d$ for some integer $c,d$ using $x^2=2$.
\end{proof}
How, in $\mathbb Q$, could we say $\mathbb Q$ has a gap?
Consider the set of rationals whose squares are less than 2.
Now all of these rationals are less than $2$, so $2$ is an upper bound.
$1.5,1.42,1.415,\ldots$ are all upper bounds of this set, but there is no least upper bound.
So that establishes the notion of there being a ``gap'', which is the thing we try to eliminate in our dream model of $\mathbb R$.
\begin{definition}
    The real numbers $\mathbb R$ is an ordered field with the least upper bound property.\\
    Put it into axiom, the reals consists of a set $\mathbb R$, the binary operations $+,\times$, elements $1\neq 0$, and an order $<$ where\\
    1. $(\mathbb R,+,0)$ is an abelian group.\\
    2. $(\mathbb R,\times,1)$ is an abelian group.\\
    3. $\times$ is distributive over $+$, so $\forall a,b,c\in\mathbb R,a\times(b+c)=a\times b+a\times c$.\\
    4. For any $a,b\in\mathbb R$, exactly one of $a<b,a=b,a>b$ is true.
    Also $a<b\land b<c\implies a<c$.\\
    5. $\forall a,b,c\in\mathbb R$, $a<b\implies a+c<b+c$, $c>0\implies(a<b\implies ac<bc)$.\\
    6. (Least-upper-bound property) For any nonempty $S\subset\mathbb R$ such that it is bounded from above (i.e. $\exists M\in\mathbb R, \forall x\in S,x<M$), there is a least upper bound of $S$.
\end{definition}
\begin{remark}
    1. We can already conclude $0<1$ from axioms 1 to 5.
    Indeed, if not, then $1<0$, so $0<-1$, so $0=0(-1)<(-1)(-1)=1$, contradiction.\\
    2. We can embed $\mathbb Q$ into $\mathbb R$, but the Least-upper-bound property is false in $\mathbb Q$.\\
    3. We do need the ``nonempty'' and ``bounded above'' conditions.
    We need the former to ensure that there is indeed some element in $S$ to talk about order, and we need the latter to ensure that there is at least one upper bound.\\
    4. We could constuct $\mathbb R$ from $\mathbb Q$ and we can check that axioms 1 to 6 hold.
\end{remark}
We write $\sup S$ to denote the least upper bound of $S$.
\begin{example}
    1. $S=\{x\in\mathbb R:0\le x\le 1\}$ be the closed interval $[0,1]$.
    $2$ is an upper buond for $S$ since $\forall x\in S, x\le 2$, but $3/4$ is not since $1\in S$ but $1>3/4$.
    The least upper bound is $1$, since $1$ is an upper bound for $S$ and for every upper bound $s$ of $S$ must be at least $1$ since $1\in S$.\\
    2. $S=\{x\in\mathbb R:0<x<1\}$ be the open interval $(0,1)$.
    Again, $2$ is an upper bound but $3/4$ is not since $S\ni 5/6>3/4$.
    The least upper bound is $1$ since $1$ is an upper bound and for any upper bound $s$, if $s<1$ (note that $s<0$), we must have $S\ni (1+s)/2>s$ which is a contradiction.\\
    3. $S=\{1-1/n:n\in\mathbb N\}$, then $\sup S=1$, since $1$ is clearly an upper bound and if there is an upper bound $s\in S$ such that $0<s<1$, then we take a natural number $n>1/(1-s)$, then $1-1/n>1-(1-s)=s$.
\end{example}
Note that in example 3, we have assumed the following proposition which we shall prove now.
\begin{proposition}[Axiom of Archimedes]
    $\forall x\in\mathbb R,\exists n\in\mathbb N,n>x$.
\end{proposition}
\begin{proof}
    Suppose not, the $\mathbb N$ is bounded above, so let $c=\sup\mathbb N$, then $c-1$ is not an upper bound of $\mathbb N$, so $\exists n\in\mathbb N, n>c-1\implies\mathbb N\ni n+1>c$, contradiction.
\end{proof}
\begin{corollary}
    Let $t\in\mathbb R_{>0}$, then there is some $n\in\mathbb N$ such that $1/n<t$.
\end{corollary}
\begin{proof}
    Take $n>1/t$.
\end{proof}
So the reals do not contain infinity or infinitesimal.
\begin{remark}
    1. $\sup X$ might not be in $X$.
    E.g. $\sup(0,1)=1\notin(0,1)$\\
    2. The least-upper-bound property gives as well the existence of greatest lower bound for bounded-below subsets of $\mathbb R$.
    Indeed, suppose $S\neq\varnothing$ is bounded below, then $-S=\{-s:s\in\mathbb S\}$ is bounded above, so there is some $x=\sup-S$.
    Immediately $-x$ would be the greatest lower bound of $S$.
    We denote it by $\inf S$.
\end{remark}
\begin{theorem}
    There is $x\in\mathbb R$ such that $x^2=2$.
\end{theorem}
\begin{proof}
    Consider the set $S=\{r\in\mathbb R:r^2<2\}$ which is nonempty ($1\in S$) and bounded above ($\forall s\in S,s<2$).
    So there is some $c=\sup S$, also $1\le c<2$.\\
    Consider $c^2$, if $c^2<2$, then for any $0<t<1$, we have $(c+t)^2=c^2+2ct+t^2\le c^2+5t$, so we choose any $0<t<(2-c^2)/5$, then $(c+t)^2<2$, so $c+t\in S$ but $c+t>c$, contradicting the assumption that $c$ is an upper bound.\\
    If $c^2>2$, then for any $0<t<1$, we have $(c-t)^2=c^2-2ct+t^2\ge c^2-4t$, so we can choose $t$ such that $0<t<(c^2-2)/4$, therefore $(c-t)^2>2$, thus $c-t$ is an upper bound of $S$ as well, contradicting the fact that $c$ is the least upper bound.\\
    Therefore $c^2=2$.
\end{proof}
Similarly, for any $x>0,n\in\mathbb N$, $\sqrt[n]{x}$ exists.
\begin{definition}
    A real $x$ that is not rational is called irrational.
\end{definition}
\begin{example}
    $\sqrt 2,\sqrt 3,\sqrt 5,3+5\sqrt 2$ are irrational.
\end{example}
\begin{proposition}
    For any $a,b\in\mathbb R,a<b,\exists q\in\mathbb Q, a<q<b$.
\end{proposition}
\begin{proof}
    Suppose $a<b$ and WLOG $a,b\ge 0$.
    Choose $n\in\mathbb N$ with $1/n<|b-a|=b-a$.
    So we can find $k\in\mathbb N$, $k/n\le a$ and $(k+1)/n>a$.
    Now if $(k+1)/n\ge b$, then $1/n<|b-a|\le (k+1)/n-k/n=1/n$, contradiction.
    So $a<(k+1)/n<b$.
\end{proof}
\begin{corollary}
    For any $a,b\in\mathbb R,a<b,\exists i\in\mathbb R\setminus\mathbb Q, a<i<b$.
\end{corollary}
\begin{proof}
    $\exists q\in\mathbb Q, a/\sqrt 2<q<b/\sqrt 2$, so $i=q\sqrt 2$ works.
\end{proof}
\subsection{Sequences and Their Limits}
What does $1+1/2+1/4+1/8+\ldots=2$ mean?
Why does $0.33333\ldots=1/3$?
When we come to think about it, we mean that, for example in the first case, $1,1+1/2,1+1/2+1/4,\ldots$ ``$\to$'' $2$.
But what do we mean by that?
We do not mean that the sequence will be eventually $x$, but can be ``arbitrarily close'' to that.
\begin{definition}
    The absolute value function $|x|$ is defined by
    $$\forall x\in\mathbb R,|x|=
    \begin{cases}
        x\text{, if $x\ge 0$}\\
        -x\text{, otherwise}
    \end{cases}$$
\end{definition}
Regarding absolute value, we also have the triangle inequality
\begin{proposition}
    $|x-y|\le|x-z|+|z-y|$
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
\begin{definition}
    For a sequence $(x_n)$ of reals and $x\in\mathbb R$, we say $x_n\to x$ (as $n\to\infty$) or
    $$\lim_{n\to\infty}x_n=x$$
    if $\forall\epsilon>0,\exists N\in\mathbb N,\forall n>N,|x_n-x|<\epsilon$.
\end{definition}
So it means $x_n$ will ``eventually'' go $\epsilon$-close to $x$.
\begin{example}
    1. Consider the sequence $1/2,1/2+1/4,1/2+1/4+1/8,\ldots$, so $x_n=1-1/2^n$ inductively.
    $\forall\epsilon>0$, we can choose $N\in\mathbb N$ such that $1/N<\epsilon$, for any $n\ge N$, $|1-x_n|=|1/2^n|\le |1/n|\le 1/N<\epsilon$, so $x_n\to 1$.\\
    2. Any constant sequence converges to that constant.
    Indeed, for any $\epsilon>0$, choose $N=1$, so for every $n>N$, the sequence is $c$ hence is $\epsilon$-close to the constant.\\
    3. Consider $x_n=(-1)^n$.
    This sequence does not converge to a limit.
    Suppose it does, then suppose the limit is $c$, then we can choose $\epsilon=1$, then there is some $N\in\mathbb N$, such that $\forall k>N, |x_n-c|<\epsilon=1$, but we can choose $n$ such that $2n>N$, but $2=|x_{2n}-x_{2n+1}|\le |x_{2n}-c|+|x_{2n+1}-c|<2\epsilon=2$, contradiction.\\
    3. Sequence needs not have a closed form, we can take
    $$x_n=
    \begin{cases}
        1/n\text{, if $n$ is odd}\\
        0\text{, otherwise}
    \end{cases}$$
    Then it is trivial that $x_n\to 0$.
    $\forall\epsilon>0$, we choose $N$ such that $N>1/\epsilon$, then $\forall n\ge N,|x_n-0|\le 1/N<\epsilon$.
\end{example}
\begin{remark}
    1. If $x_n\to c$ for some $c$, we say the sequence $(x_n)$ or $(x_n)_{n=1}^\infty$ is convergent.
    If it is not the case, then we say it is divergent.
    Note that the example $(-1)^n$ shows that a divergent sequence needs not to go to infinity.\\
    2. Limits, if exist, are unique.
    If $x_n\to c$ and $x_n\to d$, then $c=d$.
    Indeed, if $c\neq d$, then $|c-d|>0$, so we choose $\epsilon=|c-d|/2$, so $\exists N_1\in\mathbb N,\forall n\ge N, |x_n-c|<\epsilon,\exists N_2\in\mathbb N,\forall n\ge N_2, |x_n-d|<\epsilon$, so let $N=\max\{N_1,N_2\}$, so $|c-d|=2\epsilon >|x_N-c|+|x_N-d|\ge |c-d|$, contradiction.
\end{remark}
A sequence given in the form $x_1,x_1+x_2,x_1+x_2+x_3,\ldots$ is called a series.
We can write
$$\sum_{n=1}^\infty x_n$$
for the series, and the $k^{th}$ term of it is
$$\sum_{n=1}^k x_n$$
This is called a partial sum of the series.
If a series is convergent, one can also write the infinite sum as its limit, so
$$\sum_{n=1}^\infty \frac{1}{2^n}=1$$
We should not write and cannot write something like ``let $c$ be the limit of $(x_n)$ and blah blah blah'' before ensuring that the sequence does converge since such a number may not exist.\\
Limits do behave nicely.
\begin{example}
    1. If $x_n\le d$ always, and $x_n\to c$, then $c\le d$.
    Indeed, if $c>d$, then we can take $\epsilon=c-d$, so $|c-x_n|=c-x_n=(c-d)+(d-x_n)\ge c-d=\epsilon$, contradiction.\\
    2. If $x_n<d$ always, and $x_n\to c$, then we need not have $c<d$, since we can take for example $x_n=-1/n\to 0$ and $d=0$.
\end{example}
\begin{proposition}
    If $x_n\to x$, $y_n\to y$, then $z_n=x_n+y_n\to x+y$.
\end{proposition}
\begin{proof}
    $\forall\epsilon>0$, choose $N_1$ with $\forall n>N_1, |x_n-x|<\epsilon/2$ and $N_2$ with $\forall n>N_2, |y_n-y|<\epsilon/2$, then choose $N=\max\{N_1,N_2\}$, then $\forall n>N,|z_n-(x+y)|\le |x_n-x|+|y_n-y|<2\epsilon/2=\epsilon$, so $z_n\to x+y$.
\end{proof}
We say a sequence $x_1,x_2,\ldots$ is increasing if $x_n\le x_{n+1}$ for every $n\in\mathbb N$.
It is called bounded above if $\{x_n:n\in\mathbb N\}$ is bounded above.
\begin{theorem}
    An incresing sequence that is bounded above is convergent.
\end{theorem}
Note that this is false in $\mathbb Q$.
\begin{proof}
    Suppose $(x_n)$ is such a sequence.
    We claim that it converges to $c=\sup\{x_n:n\in\mathbb N\}$.
    For any $\epsilon>0$, $c-\epsilon$ cannot be an upper bound, so there is some $N\in\mathbb N$ such that $c\ge x_N>c-\epsilon$, then for any $n>N$, $c\ge x_n\ge x_N>c-\epsilon$, so $|c-x_n|<\epsilon$.
\end{proof}
\begin{remark}
    1. It is equivalent to say that a decreasing sequence (i.e. $x_n\ge x_{n+1}$ for all $n\in\mathbb N$) bounded below is convergent.
    So as a corollary, a bounded monotone sequence is convergent.\\
    2. In series, the equivalent form would be that if the partial sums of the series is bounded above and all terms are nonnegative, then it converges by corollary just stated.
\end{remark}
There are a few applications of the theorem (and its corollaries).
\begin{proposition}
    $\sum_{n=1}^\infty n^{-1}$ diverges, and $\sum_{n=1}^\infty n^{-2}$ converges.
\end{proposition}
Note that as sequences, neither of the sums has a (nice) closed form.
Our main idea would be doing comparisons with other series whose sums we are more familiar with.
\begin{proof}
    For the first part of the proposition, choose any partial sum
    $$P=\sum_{n=1}^k n^{-1}$$
    then consider the least $r$ such that $2^{r}>k$, so
    $$\sum_{n=1}^{2^{r+1}} \frac{1}{n}\ge P+\sum_{n=2^r}^{2^{r+1}} \frac{1}{n}\ge P+\frac{2^{r}}{2^{r+1}}=P+\frac{1}{2}$$
    So it is unbounded, hence does not converge.
    For the second part,
    $$\sum_{n=2^r}^{2^{r+1}-1} \frac{1}{n^2}\le \frac{2^r}{2^{2r}}=\frac{1}{2^r}\implies \sum_{n=1}^\infty\frac{1}{n^2}\le \sum_{r=0}^\infty\frac{1}{2^r}=2$$
    So it is bounded, hence it converges.
\end{proof}
The last sum actually tends to $\pi^2/6$, the proof of this will be covered somewhere else.\\
Secondly, decimal expansion.
When we are thinking of $0.a_1a_2a_3\ldots$, how do we know the limit $0.a_1,0.a_1a_2,\ldots$ always exists?
Obviously, we want to analyze the infinite sum
$$\sum_{n=1}^\infty \frac{a_n}{10^n}$$
This does converge, since
$$\sum_{n=1}^\infty \frac{a_n}{10^n}\le \sum_{n=1}^\infty \frac{10}{10^n}<\sum_{k=0}^\infty\frac{1}{10^k}<2$$
Conversely, given any real number $0<x<1$, we first choose $a_1\in\{0,1,\ldots,9\}$ such that $a_1/10\le x<(a_1+1)/10$, and when we have chosen $a_k$, we choose $a_{k+1}\in\{0,1,\ldots,9\}$ by choosing it to be such that
$$\sum_{n=1}^k\frac{a_n}{10^n}+\frac{a_{k+1}}{10^{k+1}}\le x <\sum_{n=1}^k\frac{a_n}{10^n}+\frac{a_{k+1}+1}{10^{k+1}}$$
such $a_{k+1}$ always exists due to the definitions of $a_1,a_2,\ldots,a_k$.
\begin{remark}
    1. The decimal (or binary or other bases) expansion of any rational number is periodic.
    Conversely, a real number whose decimal (or binary or other bases) expansion is periodic is rational.\\
    2. Decimal expansion may not be unique (in a way) since $0.500\ldots =0.499\ldots$, but this is kind of the only way that this would happen.
\end{remark}
\begin{definition}
    We define $e$ to be $1+1/1!+1/2!+1/3!+1/4!+\cdots$
\end{definition}
\begin{proposition}
    $e$ is well defined.
\end{proposition}
\begin{proof}
    The partial sum is bounded since
    $$\sum_{i=0}^\infty\frac{1}{i!}\le 1+\sum_{k=0}^\infty\frac{1}{2^k}=3$$
    Also each term is positive, therefore the sequence of partial sums is increasing, hence it converges.
\end{proof}
\subsection{Transcendental Numbers}
\begin{definition}
    A real number $a$ is called algebraic if it is the root of some nonzero polynomial with integer coefficients.
\end{definition}
\begin{example}
    1. Every rational number is algebraic.\\
    2. $\sqrt[q]{p}+r$ is algebraic for any $p,q,r\in\mathbb Q$ (given that it is well-defined).
\end{example}
However, there can be real numbers that is not algebraic.
To start with, we shall show that $e$ is irrational.
\begin{proposition}
    $e$ is irrational.
\end{proposition}
\begin{proof}
    Suppose for the sake of contradiction that $e=p/q$ such that $p\in\mathbb Z,q\in\mathbb N$.
    Obviously $q>1$.
    So
    $$\frac{p}{q}=1+\frac{1}{1!}+\frac{1}{2!}+\cdots\implies p(q-1)!=q!\left(\sum_{n=0}^q\frac{1}{n!}\right)+\sum_{n=q+1}^\infty\frac{q!}{n!}$$
    Note that the first term in the right hand side is integral, but for the second term,
    $$0<\sum_{n=q+1}^\infty\frac{q!}{n!}=\frac{1}{q+1}+\frac{1}{(q+1)(q+2)}+\cdots\le\sum_{k=1}^\infty\frac{1}{(q+1)^k}=\frac{1}{q}<1$$
    Thus the right hand side is not an integer, but the left hand side is.
    This is a contradiction.
\end{proof}
\begin{definition}
    If a real number $x$ is not algebraic, we say it is transcendental.
\end{definition}
$e$ is actually transcendental, but we will not prove it here.
However, we will give an example of a transcendental number.
\begin{proposition}
    Take
    $$c=\sum_{k=1}^\infty \frac{1}{10^{k!}}$$
    then $c$ is transcendental.
\end{proposition}
It is trivial that $c$ is well defined.
We will need the following facts about polynomials.
\begin{proposition}
    1. For any polynomial $P$, we have some constant $K>0$ such that $|P(x)-P(y)|\le K|x-y|$ for each $x,y\in [0,1]$.\\
    2. A polynomial of degree $d$ has at most $d$ roots.
\end{proposition}
\begin{proof}
    1. Suppose $P(x)=\sum_{i=0}^da_ix^i$, then for $x,y\in[0,1]$,
    $$|P(x)-P(y)|=\left|\sum_{i=1}^da_i(x^i-y^i)\right|\le d\sum_{i=1}^d|a_i||x-y|=K|x-y|,K=d\sum_{i=1}^d|a_i|$$
    2. Polynomial division.
\end{proof}
Now we are ready for the proof.
\begin{proof}[Proof that $c$ is transcendental]
    Suppose for the sake of contradiction that there is some polynomial
    $$P(x)=\sum_{k=0}^da_kx^k$$
    such that $a_k\in\mathbb Z$ and $P(c)=0$.
    Let
    $$c_n=\sum_{k=1}^n \frac{1}{10^{k!}}$$
    So $c_n\to c$.
    Note that $|c_n-c|\le 2/10^{(n+1)!}$.
    Since $P$ has at most $d$ roots, $\exists N\in\mathbb N,\forall n>N,P(c_n)\neq 0$.
    As $P$ has integer coefficients, for $n>N$,
    $$\frac{1}{10^{n!d}}\le|P(c_n)|=|P(c_n)-P(c)|\le K|c_n-c|\le\frac{2K}{10^{(n+1)!}}$$
    by the preceding proposition.
    This fails when $n$ is large, contradiction.
\end{proof}
\begin{definition}
    We say $x\in\mathbb R\setminus\mathbb Q$ is a Liouville number if $\forall n\in\mathbb N,\exists p\in\mathbb Z,q\in\mathbb N,$
    $$\left|x-\frac{p}{q}\right|<\frac{1}{q^n}$$
    That is, $x$ has a very good rational approximations.
\end{definition}
\begin{theorem}
    Every Liouville number is transcendental.
\end{theorem}
\begin{proof}
    Similar as above.
\end{proof}
\begin{remark}
    $e$ is not a Liouville number.
\end{remark}


\section*{Ex.2.2}
\subsection*{Size of hash function $h$ output for collision probability is 0}

If such a function exists, then $m$ must be at least as large as $\abs{U}$ otherwise you would with probability 1 get a collision if all keys from $U$ was hashed.