\part{Case One: Simple Object Creation, Manipulation, Summarizing
  Objects, Plotting, and Bookkeeping}

This part of the short course is a guided walk-through some fictional
analyses. We will create some data. Recode and summarize this
data. Then, we will make some simple plots. Lastly, we learn how to
save an \R{} object for later use, and discuss object bookkeeping.

\section{Simple Object Creation and Manipulation}
\subsection{Vectors}
We will start this walk-through by creating a vector of names:

\verb=vPeople <- c("Tyson", "Jonathan1", "Gary", "Peter", "Jonathan2")=.
\texttt{vPeople} is a vector of character strings. Create three more
vectors as follows:
\begin{verbatim}
vEoM <- c(FALSE, TRUE, FALSE, TRUE, NA)

vAge <- c(28, 25, 25, 30, 25)

vState <- c("TX", "CT", "CA", "DE", "MO")
\end{verbatim}
Each of these vectors is of a different mode. You can verify this with
\texttt{is()}. The \texttt{c()} function lets us concantenate elements
of the same mode and form a vector.

One technical thing to note is that ``constants'' like 1 or 4 are
vectors of length 1.

Two commands you may use from time to time are \texttt{seq()} and
\texttt{rep()}. Lastly, don't forget about the use of \texttt{:} to
create consecutive integer sequences.

\subsection{Pseudo-Random Numbers and Using Functions}

To create several more ``variable'' vectors we will use \R{}'s
pseudo-RNG capabilities. Consider \texttt{?rnorm}. The most common
probability distributions will have a page like this. So, for the
Normal distribution we see that:
\begin{description}
  \item[\texttt{rnorm}] generates random values
  \item[\texttt{dnorm}] is the density function (i.e.\ $f(x)$)
  \item[\texttt{pnorm}] is the distribution function (i.e.\ $F(x)$)
  \item[\texttt{qnorm}] is the quantile function
\end{description}
Because these are so closely related they appear on the same help
page. In the case of \texttt{rnorm}, we see that the function takes
three named arguments. The names are \texttt{n}, \texttt{mean}, and
\texttt{sd}. The last two of these have default values (0 and 1,
respectively). This means that you can evaluate a call to
\texttt{rnorm} without explicitly setting a value for \texttt{mean} or
\texttt{sd}.

Run \texttt{rnorm(4)} three times at the shell prompt. Then run
\verb=a <- rnorm(4)=. Now, reference \texttt{a} at the shell prompt
four times. Why are these different?

Using the RNG capabilities we create three more vectors:
\begin{verbatim}
vVar1 <- rbinom(5, 2, .5)
vVar2 <- rnorm(length(vPeople), vAge)
vVar3 <- rnorm(sd = vAge, mean = vAge, n = length(vPeople))
\end{verbatim}
Notice that when we call the function, the arguments do not need to
follow the order they are listed in the help file exactly. How does
\R{} match arguments if they aren't always necessary and if the order
can change? The \R{} manuals explain the precise set of rules, but
there is a shortcut to not having to worry about this. If an argument
is a named argument in the help file, explicitly name it. So, run
\texttt{rnorm(n = 4)} and not \texttt{rnorm(4)}.

\subsection{Basic Mathematical Operators}
We can perform arithmatic in the natural way. The command \texttt{4 +
  6} evaluates as we would expect. Other operators like \texttt{-},
\texttt{/}, \texttt{*}, \texttt{\%\%}, \texttt{exp()}, \texttt{log()},
\texttt{\^}, and \texttt{abs()} work, too. Create this relatively
complicated object using the arithmatic operators, the objects we have
already created, and another call to the RNG.
\begin{verbatim}
vVarY <- (4 * exp(vVar1) +
          abs(vVar2) + (vVar3) ^ (2) +
          runif(vPeople, -1, 1)
          )
\end{verbatim}
We know that \texttt{vVar1} is a vector. What is \texttt{exp(vVar1)}?
It too is a vector. How can you check this? \R{} evaluates the
\texttt{exp} function element by element and then concantenates the
result. Similarly, \texttt{4 * exp(vVar1)} is formed by multiplying
each element in the vector we just created by 4 and then
concantenating the results. \R{} adds vectors the same way, element by
element.

\R{} has trigonometric functions like \texttt{sin()} and
\texttt{cos()}, too.

Scientific notation is achieved by typing \texttt{1e3} and
\texttt{1e-1}.

\subsection{Matrices}

Recall that matrices must be of the same mode. How are the following
two objects different?
\begin{verbatim}
cbind(vVar1, vVar2, vVar3)
cbind(vVar1, vVar2, vVar3, vState)
\end{verbatim}

\R{} trys hard to make the code you give it work. Often times, this
means it will coerce objects into different modes so that the code can
run. Here, numerical elements are coerced to character strings.

Create the object \verb=mNums <- cbind(vVar1, vVar2, vVar3)=. We can
get information about its dimensionality with \texttt{dim()},
\texttt{nrow()}, and \texttt{ncol()}. We can refer to just the elemnts on
the main diagonal with \texttt{diag(mNums)}. Interestingly enough, we
can create a square, diagonal matrix with of size 6, for example, with
\texttt{diag(6)}. The arithmatic operators work on matrices just like
they do on vectors (i.e.\ element-wise). Try a few on \texttt{mNums}.

Now, there are also matrix-specific operations. We can transpose the
the matrix with \texttt{t()}. We perform matrix multiplication with
\texttt{\%*\%}. We can calculate the determinent of a square matrix
with \texttt{det()}. We can invert a square matrix with \texttt{solve()}

Sometimes the rows and columns have labels which makes life easier for
users. These are stored as a vector of character strings and this is,
not surprisingly, an object itself. See \texttt{?rownames} and
\texttt{?colnames}.

\subsection{Dataframes}
Create a data frame with the following code:
\begin{verbatim}
dfData <- data.frame(vPeople, vEoM, vAge, vState,
                     vVar1, vVar2, vVar3, vVarY
                     )
\end{verbatim}
Print out the object \texttt{dfData}. Notice that character strings
become factors. The numbers remain numerical elements and are not
coerced. Lastly, we can see that the logical values have remained as
logicals. This happens because dataframes allow us to mix modes in a
way matrices do not. See \texttt{?data.frame} for more information on
creating them.

\subsection{Indexing and Subsetting}
We can refer to subsets of vector, matrix, and dataframe
objects. Vectors have a single dimension (i.e.\ length). So, we simply
specify the positions corresponding to the elements we wish to extract
from the vector. Try:

\begin{verbatim}
vPeople[1]
vPeople[-1]
vPeople[c(1,4,2)]
\end{verbatim}

Matrices have two dimensions. So, to refer to particular elements,
specify the exact row and column as in \texttt{mNums[2, 3]} or
\texttt{mNums[c(2,1), 3]}. You can refer to entire columns with
\texttt{mNums[2, ]} and entire rows with \texttt{mNums[, 3]}.

Elements in dataframes can be referred to just like they can in
matrices. But, there is some added functionality in this case. We can
still do:
\begin{verbatim}
dfData[1,1]
dfData[-1,-1]
dfData[-1,-1]
\end{verbatim}
However, the columns can be referred to in a special way. Try:
\begin{verbatim}
dfData$vPeople
dfData[, "vAge"]
\end{verbatim}
To see the names of the columns returns as character vector use
\texttt{names(dfData)}. Even though they appear to be similar,
\texttt{dfData\$vPeople} and \texttt{vPeople} are two different
objects.

We can subset dataframes based on logical tests. Typically, the
logical test we use will be dependent on the data, itself. See
\texttt{?subset}. Then, consider \verb=subset(dfData, vAge > 25)=. One
very important point to note is that the \texttt{vAge} term in the
logical test refers to the column name in \texttt{dfData} and not the
object by the same name. Confirm this by replacing the object
\texttt{vAge} with a vector that equals the old vector plus 100. Rerun
the subset command.

Moreover, we can use any arbitrary logical vector to subset any
object. Consider:
\begin{verbatim}
vPeople[c(TRUE, TRUE, FALSE, FALSE, TRUE)]
vPeople[vAge > 25]
vPeople[((1:5) * 3) < 10]
\end{verbatim}

\subsection{Logical Operators}
We just saw some logical vectors used in indexing and subsetting. In
general, they will be very useful. The common logical operators are
\texttt{<}, \texttt{>}, \texttt{|}, \texttt{\&}, \texttt{is.*()},
\texttt{!}, \texttt{<=}, \texttt{>=} and \texttt{xor()}.

Two particularly useful functions are \texttt{any()} and
\texttt{each()}. They test a vector of logicals for whether or not at
least one and every element, respectively, evaluates to \texttt{TRUE}.

\subsection{Simple Recoding}
Combining the ways to index particular elements in an object and the
set of logical operators, we are equipped to do some simple recoding
of values within objects that we have already created.

If we find out that Tyson is not from Texas, but from France (hence,
Chatagnier), we will want to change the entry for him in the state
variable. We will leave alone the state variable in the
dataframe. Instead, consider the original vector we created,
\texttt{vState}. If we want Tyson's entry to display that he is from
\texttt{FR} we can use \verb=vState[1] <- "FR"= because we know the
first entry corresponds to him. Confirm that this worked by
considering \texttt{vState}. However, this required we know his entry
was in the first row. If we didn't know this, we could use
\begin{verbatim}
vState[vPeople == "Tyson"] <- "FR"
\end{verbatim}
However, we must be careful. This will change the state to \texttt{FR}
for every Tyson there is. If we have more than one, we'd need to go
about this another way.

Alternatively, we could use the \texttt{ifelse()} function. Suppose we
have to censor the state of origin for all individuals under the age
of 26. We could use
\begin{verbatim}
vState <- ifelse(vAge < 26, NA, vState)
\end{verbatim}
To see why, consider \texttt{?ifelse}.

\section{Summary Functions}
Many times, when using \R{}, we will want to describe many numbers or
observations with some sort of summary value like the mean. We will
use different summaries for categorical, ordinal, and interval level
data. At the same time, we have functions for both univariate and
multivariate summaries.

The functions \texttt{mean()}, \texttt{median()}, \texttt{mode()},
\texttt{sd()}, \texttt{sum()}, \texttt{max()}, and \texttt{min}
describe the central tendencies and other aspects of the data. How
these functions handle missing data (i.e. \texttt{NA}) matters
greatly. Consider the following:
\begin{verbatim}
mean(vVarY)
min(vVarY)
max(vVarY)
sd(vVarY)

mean(vEoM)
mean(vEoM, na.rm = TRUE)
\end{verbatim}

However, instead of evaluating several functions for each vector we
can use \texttt{summary(dfData)} and this will give us most of the
information we might have wanted. \R{} wisely customize the summary
for each variable based on the kind of data it is. For factors and
logicals, we find out about frequencies. For numerical data, we get a
description of the distribution.

Beyond these univariate measures, we can use \texttt{cor()} to
calculate the correlation coefficient between two numerical vectors or
\texttt{table()} to count the number of observations occurring within
each unique classification between multiple factor vectors. Consider
the following:
\begin{verbatim}
cor(vVarY, vVar1)
table(dfData$vAge, dfData$vState)
\end{verbatim}

If we are interested in conditional summaries, we need to do a little
more work. There are two natural ways to approach questions like, what
is the average value of \texttt{vVarY} within people of the same
age. We know there are three different groups because of
\texttt{unique(dfData\$vAge)}. What does \texttt{unique()} do?

We could subset the dataframe according to age, then run
\texttt{summary} or \texttt{mean}. Or, instead, we could use a very
convenient function called \texttt{tapply()}. Consider
\texttt{?tapply} and then the code below.
\begin{verbatim}
mean(subset(dfData, vAge == 25)$vVarY)
tapply(dfData$vVarY, dfData$vAge, mean)
\end{verbatim}

\section{Basic Plots}
Although the numerical/textual summaries of our data will be helpful,
we often look to make figures to communicate some aspect of our date
or some result to others. For now, we will simply go through different
kinds of plots without concerning ourselves with customization.

If we have univariate data, we could use a boxplot or a histogram. Consider
the following:
\begin{verbatim}
boxplot(rnorm(n = 50))
hist(rnorm(n = 50))
\end{verbatim}

If we were so inclined, we could simply plot the values of interest
against an index. Try:
\begin{verbatim}
plot(rnorm(n = 50))
\end{verbatim}

If we have bivariate data, we might want a scatterplot with lines
connecting observations in sequence. Consider:
\begin{verbatim}
plot(rnorm(n = 50), rnorm(n = 50), type = "b")
\end{verbatim}

For three dimensional data, we can generate a surface plot. Use:
\begin{verbatim}
persp(z = matrix(data = sort(rnorm(100, sd = 3)), ncol = 10))
\end{verbatim}

The options are many and
\url{http://cran.r-project.org/web/views/Graphics.html} is a good link
to read. If you know the kind of plot you would like to create, it
will most likely either exist already or you can create it. We will
cover building plots from the ground up and customization later.

\section{Object Bookkeeping}

Recall, just about everything in \R{} is an object. And, by now, we've
created a number of objects. To display all of the objects type
\texttt{ls()}. What is the class of the output of this function call?
If you would like to remove an object named \texttt{foo}, try
\texttt{rm(foo)}. If you would like to remove every object, use
\texttt{rm(list = ls())}. Some folks use this at the top of each
script to make sure their \R{} session is fresh. I would recommend
this.

Now, let's create an object that will contain all of our work. A list
is the best choice here. First, we initialize a list. Second, we will
save the dataframe as one element in this list. Third, and last we
will leave a note for ourselves as a character string.
\begin{verbatim}
lOutput <- list()
lOutput$data <- dfData
lOutput$comment <- "today's work was very rewarding"
lOutput
\end{verbatim}
Now, we can reference the elements in the list by their names and the
\texttt{\$} operator.

To save the object \texttt{lOutput}, first look at
\texttt{?save}. Then, run:
\begin{verbatim}
save(lOutput, file = "Z:/mydata.Rdata"}
\end{verbatim}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../full_course"
%%% End:


\subsubsection{Defective Matrix}
If $A$ has an eigenvalue $\lambda$ with multiplicity $k$ that does not generate $k$ corresponding linearly independent eigenvectors, the the matrix $A$ is defective. To generate enough vectors, we need to extend eigenvectors.
\begin{definition}
	Let $A$ be a square matrix. A nonzero vector $\vec{v}$ satisfying
	\begin{equation*}
		(A-\lambda I)^n \vec{v} = \vec{0}
	\end{equation*}
	for some eigenvalue $\lambda$ and some positive integer $n$ is a generalized eigenvector or rank $n$.
\end{definition}
These generalized eigenvectors also make up solutions.
\begin{theorem}
	Let $\vec{v}$ be an generalized eigenvector of a square matrix $A$ corresponding to an eigenvalue $\lambda$ with multiplicity $k$. Then
	\begin{equation*}
		e^{At}\vec{v} = e^{\lambda t}\left(\vec{v} + t(A-\lambda I)\vec{v} + \frac{t^2}{2!}(A-\lambda I)^2\vec{v} + \ldots\right)
	\end{equation*}
	is the corresponding fundamental solution. Further, the above sequence will terminate after $k$ or fewer terms (all subsequent terms are 0).
\end{theorem}

\ifodd\includeLinearSystemsExamples\input{./linearSystems/homogeneousSystems/defectiveMatrices_example.tex}\fi


\documentclass[
12pt, %font size
letterpaper, %for printing on American sized paper
fleqn, % for ligning up our equations
notitlepage % we don't want a title page
]{article}

\usepackage{amsmath,amsfonts,amssymb} % Math packages
\usepackage[letterpaper,margin=1in]{geometry} % Margins
\usepackage{setspace} % let's us choose between single or double spacing

\usepackage{sectsty} % Allows customizing section commands
\allsectionsfont{\normalfont\scshape} % Make all sections centered, the default font and small caps
\numberwithin{equation}{section} % Number equations within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)
\numberwithin{figure}{section} % Number figures within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)
\numberwithin{table}{section} % Number tables within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)

\setlength\parindent{0pt} % Removes all indentation from paragraphs - comment this line for an assignment with lots of text

\singlespacing
\frenchspacing % No extra spaces after sentances

\begin{document}

%------------------------------------------------
% title information
\title{\vspace{-2cm}Precalculus: Chapter 5 Notes}
\author{Amin Mesbah}
\date{}
\maketitle

%------------------------------------------------

\section{Exponents}

%------------------------------------------------

\subsection*{Laws of Exponents} % the asterisk ommits the section number
If $s$, $t$, $a$, and $b$ are real numbers with $a>0$ and $b>0$, then

% the align environment aligns things on multiple lines based on where I put an '&'
\begin{align}
	a^s \cdot a^t &= a^{s+t} &(a^s)^t &= a^{st} & (ab)^s &= a^s \cdot b^s \\
	1^s &= 1 & a^{-s} &= \frac{1}{a^s} = \left(\frac{1}{a}\right)^s & a^0 &= 1
\end{align}
% fractions are a bit tricky. You use them like this:
% \frac{numerator}{denominator}

%------------------------------------------------

\subsection*{Exponential Functions}
An \textbf{exponential function} is a function of the form

\begin{align}
	f(x) = Ca^x
\end{align}

where $a \in \mathbb{R}$, $a>0$, $a\neq1$, and $C\neq0$ is a real number. The domain of $f$ is $\mathbb{R}$. The base $a$ is the \textbf{growth factor}, and because $f(0) = Ca^0 = C$, we call $C$ the \textbf{initial value}.\\

For $f(x)=Ca^x$, where $a>0$ and $a\neq1$, if $x\in\mathbb{R}$, then

\begin{align}
	\frac{f(x+1)}{f(x)} &= a & \text{or} && f(x+1)&=af(x)
\end{align}

%------------------------------------------------

\subsection*{The number $e$}

\begin{align}
	e&=\lim_{n\to\infty}\left(1+\frac{1}{n}\right)^n
\end{align}

%------------------------------------------------

\subsection*{Solving Exponential Equations}

Use this property, expressing each side of the equation using the same base:

\begin{align}
	\text{If }a^u=a^v\text{, then }u=v
\end{align}

%------------------------------------------------

\section{Logarithms}

%------------------------------------------------

$\log_{a}x$ represents the exponent to which $a$ must be raised to obtain $x$.

\begin{align}
	y &=\log_{a}x\text{ if and only if }x=a^y \\
	y &=\ln{x}\text{ if and only if }x=e^y \\
	y &=\log{x}\text{ if and only if }x=10^y
\end{align}

The domain of $y=\log_{a}x$ is $\{x|x>0\}$.

%------------------------------------------------

\subsection*{Logarithmic Functions}

The logarithmic function is the inverse of the exponential function.

\begin{align}
	\text{If }f(x)=a^x\text{, then }f^{-1}(x)=\log_{a}x
\end{align}

The domain of $f^{-1}$ is the range of $f$. The range of $f^{-1}$ is the domain of $f$.

%------------------------------------------------

\subsection*{Properties of Logarithms}

In the following properties, $M$, $N$, and $a$ are positive real numbers, $a\neq1$, and $r \in \mathbb{R}$.

\begin{align}
	\log_{a}1 &= 0 & log_{a}a &= 1 \\
	a^{\log_{a}M} &= M & \log_{a}a^r &= r \\
	\log_{a}M^r &= r\log_{a}M & a^x &= e^{x\ln{a}}
\end{align}

\begin{align}
	\log_{a}(MN) &= \log_{a}M + \log_{a}N &
	\log_{a}\left(\frac{M}{N}\right) &= \log_{a}M - \log_{a}N
\end{align}

\begin{align}
	\text{If }M &= N\text{, then }\log_{a}M = \log_{a}N &
	\text{If }\log_{a}M &= \log_{a}N\text{, then }M = N
\end{align}

%------------------------------------------------

\subsection*{Change of Base Formula}

If $a\neq1$, $b\neq1$, and $M$ are positive real numbers, then

\begin{align}
	\log_{a}M &= \frac{\log_{b}M}{\log_{b}a} & \text{so,} &&
	\log_{a}M &= \frac{\log{M}}{\log{a}} & \text{and} && \log_{a}M &= \frac{\ln{M}}{\ln{a}}
\end{align}

%------------------------------------------------

\end{document}


\section*{Variables and constraints}
\subsection*{Input data}
$ |E| $ is the total number of exams.
\newline
$ |S| $ is the total number of enrolled students.
\newline
$ |T| $ is the total number of available timeslots.
\newline
$ \vec{ns} $ is a $ n \times n $ matrix, being $ n = |E| $, containing the number of students in conflit among two exams. More precisely, $ ns_{ij} $ contains the number of students in conflit among the exam $i$ and the exam $j$.
\newline
$ ns_{ij} \in \mathbb{N} \quad i = 1, \dots , |E| \quad j = 1, \dots , |E| $.
\newline
$ \vec{ns} $ is built directly from the input data.

\subsection*{Variables}
$ \vec{p} $ is a vector containing the penalty value associated to exams. More precisely, $ p_{k} $ contains the penalty weight associated to exams which are $k$ timeslots far, being $ k \in \mathbb{N}, 1 \le k \le 5 $. Hence $p_{k} = 2^{5-k}$.
\newline
$ \vec{y} $ is a 3d matrix containing the information about conflicts at a certain distance. More precisely,
\[
y_{ijk} =
\begin{cases}
1	&	\text{if exams $ i $ and $ j $ are $k$ timeslot far}	\\
0	&	\text{otherwise}
\end{cases}
\]

\subsection*{Output data}
$ \vec{x} $ is a vector containing the number of the timeslot assigned to an exam. More precisely, $ x_{i} $ contains the number of the timeslot assigned to the exam $i$. Hence $ 1 \le x_{i} \le |T| $.

\subsection*{Constraints}
$ ns_{ij} > 0 $ means that there are some students that can not substain the same exam in the same timeslot. Hence, it is needed to force $ x_{i} \ne x_{j} $.
In order to express this condition, it is necessary to introduce two new variables
\[
l_{ij} =
\begin{cases}
1	&	ns_{ij} > 0	\\
0	&	\text{otherwise}
\end{cases}
\]
\[
q_{ij} =
\begin{cases}
1	&	x_{i} \ne x_{j}	\\
0	&	\text{otherwise}
\end{cases}
\]
and to generate the following constraints:
\begin{gather*}
ns_{ij} \le M l_{ij} \qquad ns_{ij} \ge m l_{ij} \\
x_{i} - x_{j} \ge m q_{ij} \qquad x_{i} - x_{j} \le M q_{ij} \\
l_{ij} \le q_{ij}
\end{gather*}
\newline
It is needed to activate the proper $ y_{ijk} $ in order to represent the correct couple of exams in conflict at a certain distance.
\newline
$ x_{i} - x_{j} = k $ means that exam $i$ and exam $j$ are $k$ slots far. Hence, it is needed to force $ y_{ijk} = 1 $ being $ k \in \mathbb{N}, 1 \le k \le 5 $.
In order to express this condition, it is necessary to introduce two different blocks of constraints.

The first one forces $ y_{ijk} = 0 $ when $ x_{i} - x_{j} \ne k $ in order to "facilitate" the minimization of the objective function.
\begin{gather*}
k - (x_{i} - x_{j}) \le M (1-y_{ijk}) \\
(x_{i} - x_{j}) - k \le M (1-y_{ijk})
\end{gather*}

The second one forces $ y_{ijk} = 1 $ when $ x_{i} - x_{j} = k $.
In order to express this constrains it is needed to introduce two more boolean variables, $ y_{ijk, I} $ and $ y_{ijk, II} $.
\begin{gather}
\label{eqn:1}
(x_{i} - x_{j}) - k - \epsilon \ge (m - \epsilon) y_{ijk, I} \\
\label{eqn:2}
(x_{i} - x_{j}) - k + \epsilon \le (M + \epsilon) y_{ijk, II} \\
\label{eqn:3}
y_{ijk, I} + y_{ijk, II} - 1 \le y_{ijk}
\end{gather}
being $ \epsilon < 1 $ an arbitrarily small positive quantity.

With the relation \ref{eqn:1} the following constraint is expressed:
\[
y_{ijk, I} =
\begin{cases}
1	&	\text{if $ (x_{i} - x_{j}) \ge k $}	\\
0	&	\text{otherwise}
\end{cases}
\]
Analagously, with the relation \ref{eqn:2} the following constraint is expressed:
\[
y_{ijk, II} =
\begin{cases}
1	&	\text{if $ (x_{i} - x_{j}) \le k $}	\\
0	&	\text{otherwise}
\end{cases}
\]
Finally, the relation \ref{eqn:3} forces $ y_{ijk} = 1 $ only if $ y_{ijk, I} = 1 $ and $ y_{ijk, II} = 1 $, that represents the situation in which $ x_{i} - x_{j} = k $.


\section{Deep Learning and Neural Networks}
Consider the next model:
\begin{equation}
    \begin{cases}
        z^* &= Wx \\
        z &= \rho(z) \\
        y &= \tilde{W}z,
    \end{cases}
\end{equation}
this is a simple neural network model with one hidden layer. So the multi-layer neural network can be expressed as:
\begin{itemize}
    \item {\bf Input:}
        $$
        z^{(0)} = x,
        $$
    \item
        \begin{equation}
            \begin{cases}
                z_*^{(k)} &= W^{(k)}z^{(k)} \\
                z^{(k+1)} &= \rho(z_*^{(k)}) \\
            \end{cases}
        \end{equation}
    \item {\bf Output:}
        $$
        y = W^{t+1}z^{(T)}.
        $$
\end{itemize}
This can be seen as a discrete dynamics system. So the problem is:
\begin{equation}\label{opti-DL}
    \min_{W^{(1)}, \cdots, W^{(T+1)}} \sum_{j=1}^N (y_j - W^{(T+1)}z(x_j)^{(T)})^2.
\end{equation}
%\begin{remark}
    \begin{itemize}
        \item {\bf E:} Generally speaking: the number of parameters is larger than the number of data($N$), but there are not so many overfit  phenomenon. {\bf This is unclear!}
        \item {\bf E:} If one can solve the exact solution for the above optimization problem \eqref{opti-DL}, there is a high probability to see overfit. (Is there any experiments to show this?)
    \end{itemize}
    %\end{remark}
{\bf Application of DL:}
\begin{itemize}
    \item Dimension reduction: Encoder $+$ Decoder $\to$ Auto-encoder, a kind of unspervised learning. In mathematically, it can be expressed as:
        \begin{itemize}
            \item {\bf Input:}
                $$
                z^{(0)} = x,
                $$
            \item
                \begin{equation}
                    \begin{cases}
                        z_*^{(k)} &= W^{(k)}z^{(k-1)} \\
                        z^{(k+1)} &= \rho(z_*^{(k)}) \quad k=1:T-1\\
                    \end{cases}
                \end{equation}
            \item {\bf Mid-output:}
                \begin{align}
                    \hat{y} &= W^{t+1}z^{(T)} \\
                    \tilde{y}    &= \rho(\hat{y})
                \end{align}

            \item $$z^{T+1} = \tilde{y}$$

            \item
                \begin{equation}
                    \begin{cases}
                        z_*^{(k)} &= W^{(k)}z^{(k-1)} \\
                        z^{(k+1)} &= \rho(z_*^{(k)}) \quad k =T + 2 : T + K+1 \\
                    \end{cases}
                \end{equation}

            \item {\bf Output:}
                \begin{equation}
                    y = W^{T+K + 3}z^{(T+K+2)}.
                \end{equation}
        \end{itemize}

        So the problem is:
        \begin{equation}
            \min_{W} (x_j - W^{T+K + 3}z^{(T+K+2)(x_j)})^2.
        \end{equation}
        But in fact, the real thing we wanted is $\hat{y}$ or $\tilde{y}$, because in this way, we can use  $\hat{y}$ or $\tilde{y}$ to denote or encoder the $x$, generally have the next relation:
        \begin{equation}
            \rm{Dim}(\hat{y}) = \rm{Dim}(\tilde{y}) < \rm{Dim}(x).
        \end{equation}

    \item There is no one have researched about deep learning for cluster, but E think, one can use DL for cluster, also for density approximation.

    \item Generative model: GAN, do sampling directly without doing density approximation first. This cannot be used to write paper, but now ``NLP'' is very hot.

    \item Text analysis, time sequence, RNN and LSTM.

\end{itemize}

How to solve the problem \eqref{opti-DL}. BP! We assume that the loss function as:
$$
J(W^{(0)}, \cdots, W^{(T+1)}).
$$
So BP is:
\begin{align}
    \nabla_{W^{k}} J = \nabla_{Z^{(k)}} J\nabla_{W^{(k)}} Z^{(k)}.
\end{align}

To state the BP process better, we change the expression for DL model before as:

\subsection{Mathematical expression of neural network}
The network would be like:
\begin{figure}[!ht]
    \center{\includegraphics[width=12cm,height=6cm] {ANN.png}}
    \caption{ANN}
\end{figure}

This is called fully connected feedforward neural network. Now we want to give a more compressive expression, we collect all the output in $k$-th level $f_{k,l}$ $l = 1,\cdots, L_k$ with $f_{k,L_k+1} = 1$ as a extended vector $\bm{f}_k$.
So we can have the output in $k+1$-th level under this setting with
\begin{equation}
    \bm{f}_{k+1}(W_{k+1}) =
    \begin{pmatrix}
        \bm{g}_{k+1}(\omega_{k+1} \cdot \bm{f}_k(W_k)) \\
        1,
    \end{pmatrix}
\end{equation}
with
\begin{equation}
    W_{k+1} = \{ \omega_{k+1}, W_k\} \quad \omega_{k+1} \in \mathbb{R}^{(L_{k+1}) \times (L_{k} + 1)},
\end{equation}
and
\begin{equation}
    \bm{g}_{k+1}(x_1,\cdots,x_{L_{k+1}}) =
    \begin{pmatrix}
        g_{k+1}(x_1) \\
        \cdots  \\
        g_{k+1}(x_{L{k+1}})
    \end{pmatrix}
\end{equation}
So at last, we get the final iterative definition with the last output and initial input of this network like:
\begin{equation}
    \begin{cases}
        &\bm{f}_K(W_{K}) =
        \begin{pmatrix}
            \bm{g}_{K}(\omega_{K} \cdot \bm{f}_{K-1}(W_{K-1})) \\
            1,
        \end{pmatrix} \\
        &\bm{f}_0 =
        \begin{pmatrix}
            \bm{x} \\
            1
        \end{pmatrix}.
    \end{cases}
\end{equation}
All in all, we can define the tensor similar space like:
\begin{equation}
    T_{3}(\mathcal{L}_K) := \{ \omega_{k,li} \in \mathbb{R}~|~ 1 \le k \le K, 1\le l \le L_k, 1 \le i \le L_{k-1} + 1\},
\end{equation}
with
\begin{equation}
    \mathcal{N}_{K} = (m, L_1, L_2, \cdots, L_{K-1}, c).
\end{equation}
Here we need to mention that, for this kind of deep feedforward neural network, once $\mathcal{N}_K$ is defined we know the whole structure of this network. So we may call $\mathcal{N}_K$ as the network structure parameters. By the way, we have $c = Dim(Y^i) = n_K$ and $m = Dim(X^i) = n_0 $ which are defined by the data and problem properties.

\subsection{Interpolation by optimization}
So the  interpolation process can be seen as a optimization problem in the special function class if the activation function for every elements and  the network structure parameter $\mathcal{N}_K$ are given:
\begin{equation}
    \mathcal{F} = \{\bm{f}_K(W_K;\bm{x}) ~|~  \bm{f}_0 = (\bm{x}^{T},1)^{T}\},
\end{equation}
and the optimization problem can be seen as
\begin{equation}
    %\mathop{\min}_{f \in \mathcal{F}}  \frac{1}{N}\sum_{i=1}^N \|f(X^i) - Y^i\|^2 =
    \mathop{\min}_{W_K \in T_3(\mathcal{N}_K)}  L(W_K) := \frac{1}{N}\sum_{i=1}^N \|\bm{f}_K(W_K; X^i) - Y^i\|^2.
\end{equation}
But $\mathcal{F}$ this is neither a linear space nor a convex set.

\subsection{Back-Propagation}
Here we will talk about how to compute $\nabla_{W_K} L(W_K)$ by using chain rule which is called BP algorithm in deep learning. By the definition we have:
\begin{equation}
    \nabla_{W_K} L(W_K) = \frac{2}{N} \sum_{i=1}^{N} \nabla_{W_K}\bm{f}_K(W_K; X^i) \cdot (\bm{f}_K(W_K;X^i) - Y^i) .
\end{equation}
So, the only thing we need to do is to compute $\nabla_{W_K}\bm{f}_K(W_K; X^i)$, if we state this element by element, that is:
\begin{equation}
    \frac{\partial (\bm{f}_K)_j}{ \partial \omega_{k,l,i}} = \frac{\partial (\bm{f}_K)_j}{\partial \bm{f}_{K-1}} \cdot \frac{ \partial \bm{f}_{K-1}}{\partial \bm{f}_{K-2}} \cdots \frac{ \partial \bm{f}_k}{\partial \omega_{k,l,i}}.
\end{equation}
We can see from the above that, we only need to compute those terms like:
\begin{equation}
    \frac{ \partial \bm{f}_k}{\partial \omega_{k}}  \quad \text{and} \quad \frac{\partial \bm{f}_k}{\partial \bm{f}_{k-1}}  \quad k = 1,2,\cdots,K.
\end{equation}

Now we are going to derive the expression for every terms, first of all we have $(\bm{f}_k)_{L_k + 1} = 1$, so we have
$$
\frac{\partial(\bm f_k)_{L_{k} + 1}}{\partial \bm f_{k-1}} = (0,0,\cdots,0) \in \mathbb{R}^{L_{k-1} + 1},
$$
and
$$
\frac{\partial(\bm f_k)}{\partial (\bm f_{k-1})_{L_{k-1}+1}} = (0,0,\cdots,0)^{t} \in \mathbb{R}^{L_{k} + 1}.
$$
To compute $\frac{\partial(\bm f_k)_j}{\partial (\bm f_{k-1})_{l}}$ with $ 1 \le j \le L_k$ and $1 \le l \le L_{k-1}$ we have
$$
\frac{\partial(\bm f_k)_j}{\partial (\bm f_{k-1})_{l}} = \frac{ g_k( (\omega_k \cdot \bm{f}_{k-1})_j)}{\partial (\bm f_{k-1})_{l}} = g_k'((\omega_k \cdot \bm{f}_{k-1})_j)\omega_{k,jl} .
$$
Combine those results we have:
\begin{equation}\label{eq:partial-f}
    \frac{\partial \bm{f}_k}{\partial \bm{f}_{k-1}} =
    \begin{pmatrix}
        \omega_{k, (1:n_{k} )\times(1:n_{k-1})} & 0 \\
        0 & 0
    \end{pmatrix}
\end{equation}
The similar results can be derived for $\frac{ \partial \bm{f}_k}{\partial \omega_{k}}$ and we get:
\begin{equation}\label{eq:partial-omega}
    \frac{ \partial (\bm{f}_k)_l}{\partial \omega_{k,ij}} = \begin{cases}
        0 &\text{if} \quad l = L_k + 1\\
        g_k'( \sum_{l,s} \omega_{k,l,s}(\bm{f}_{k-1})_s)\delta_{li}\delta_{js}(\bm{f}_{k-1})_{s} \quad &\text{others}.
    \end{cases}
\end{equation}

In short, BP algorithm can be expressed as:
\begin{algorithm}[H]
    \begin{algorithmic}[1]
        \STATE {\bf{Input:}}  $X^i$ and $W_K$;
        \FOR{$k = K:-1:1$}
        \STATE Compute and save
        $$\frac{ \partial \bm{f}_k}{\partial \omega_{k}}  \quad \text{and} \quad \frac{\partial \bm{f}_k}{\partial \bm{f}_{k-1}}$$ by \ref{eq:partial-f} and \ref{eq:partial-omega}
        \STATE Compute
        $$\frac{ \partial \bm{f}_K}{\partial \omega_{k}} = \frac{\partial \bm{f}_K}{\partial \bm{f}_{K-1}} \cdot \frac{ \partial \bm{f}_{K-1}}{\partial \bm{f}_{K-2}} \cdots \frac{ \partial \bm{f}_k}{\partial \omega_{k}}$$
        \ENDFOR
    \end{algorithmic}
    \caption{Back-Propagation Algorithm}
\end{algorithm}


\chapter{Spectral Methods}

Following Durran~\cite{durran2010numerical}, we will discuss spectral
methods to solve differential equations. The intuition guiding these
methods should be thinking of a truncated Fourier series
\begin{equation}
  \phi_{N}(x) = \sum^{N}_{k=-N}c_{k}\E^{\I kx}
\end{equation}
then trying to find the coefficients $c_{k}$ such that they minimize
some error, or satisfy some system of equations. More generally, instead
of working with the basis functions $\exp(\I kx)$ we could use some
other basis: this is what spectral methods study.


\Extrachap{Notations}\label{sec:Notation}

\section*{General Math Notations}

\begin{longtable}{cl}
\hline\noalign{\smallskip}
\textbf{Symbol} & \textbf{Meaning} \\
\noalign{\smallskip}\hline\noalign{\smallskip}
$\lfloor x \rfloor$ & Floor of $x$, i.e. round down to nearest integer\\
$\lceil x \rceil$ & Ceiling of $x$, i.e. round up to nearest integer\\
% floor(key) & The largest key $\leq$ the given key \\
% ceil(key) & The smallest key $\geq$ the given key \\
$\log x$ & The base of logarithm is 2 unless otherwise stated\\
$a \wedge b$ & Logical AND\\
$a \vee b$ & Logical OR\\
$\neg a $ & Logical NOT\\
$a\AND b$ & Bit AND\\
$a \OR b$ & Bit OR\\
$a \XOR  b$ & Bit XOR\\
$\NOT a$ & Bit NOT\\
$\SHIFTL a$ & Bit shift left\\
$\SHIFTR a$ & Bit shift right\\
$\infty$ & Infinity\\
$\rightarrow$ & Tends towards, e.g., $n \rightarrow \infty$\\
$\propto$ &Proportional to; $y = ax$ can be written as $y \propto x$\\
$\abs{x}$ & Absolute value\\
$||\vec{a}||$ & $L_2$ distance (Euclidean distance) of a vector; norm-2 \\
$\abs{\mathcal{S}}$ & Size (cardinality) of a set\\
$n!$ & Factorial function\\
$\triangleq$ & Defined as\\
$O(\cdot)$ & Big-O notation, complexity upper bound\\
$\mathbb{R}$ & The real numbers\\
$0:n$ & Range (Python convention): $0:n = {0, 1, 2,...,n-1}$\\
$\approx$ & Approximately equal to\\
$\sim$ & Tilde, the leading term of mathematical expressions \\
$\arg\max\limits_x f(x)$ & Argmax: the value $x$ that maximizes $f$\\
$\binom{n}{k}$ & $n$ choose $k$ , equal to $\frac{n!}{k!(n-k)!}$\\
$\text{range}(i,j)$ & Range of number from i (inclusive) to j (exclusive) \\
$A[i:j]$ & Subarray consist of $A_i, A_{i+1}, ..., A_{j-1}$.
\noalign{\smallskip}\hline\noalign{\smallskip}
\end{longtable}

\twocolumn


%----------------------------------------------------------------------------------------
% Why Does RBI Work
%----------------------------------------------------------------------------------------

\section{Why Does RBI Work?}

\subsection{Layman's Terms}

Let’s assume we're conducting a simple linear regression. Suppose that our model failed the the homoscedasticity and normality of residuals conditions, our data is random, and we have n > 30 observations in our dataset. By the central limit theorem, we can then assume our slope coefficient, $\beta_1$ is approximately normally distributed. However, we can’t wholly trust the p-value associated with that slope. This is because for statistical inference, we need certain conditions to be met depending on the test. In this example of linear regression, we need a independence, a linear relationship between our response and predictor, and homoscedasticity of residuals. If it’s the case where these conditions cannot be met but we do have a sample representative of the population, we can create our randomization model. This randomization model is the distribution of n test statistics that we get from scrambling the data n times and calculating the given test statistic. Our randomization model also assumes the null hypothesis of the test statistic being zero or no statistically significant effect being present, creating what is called a null distribution. This is because the model has no parameters and that each time the data is shuffled, the observations become independent of their outcomes. Our original model, and typically most models, are tested at a significance level of $\alpha=.05$. Because our randomization model assumes observations are independent of their outcomes, a constant significance level, or type 1 error rate, is held constant for the randomization model. Furthermore, by creating this randomization model, our randomization distribution is approximately the null distribution of whatever statistic we're calculating. Now, we calculate the p-value from the proportion of test statistics calculated from our randomly permuted dataset that are as or more extreme than our original test statistic. We can trust this p-value because the p-value comes from a now uniform distribution.

\subsection{Sharp Null Hypothesis \& Asymptotic Normality and p-value}

As with many concepts in statistics and probability, the basis for randomization-based inference lies in the Central Limit Theorem. Let's stick with our simple linear regression example from above. As we saw, when we permute our data in simple linear regression, we are basically saying that there is independence between $x$ and $y$. This is equivalent to the null hypothesis, $H_0:\hat{\beta_1}=0$. In fact, for many permutations, this is equivalent to Fischer's sharp null hypothesis which says that there is no effect for each pairwise set of $x$ and $y$ values. Traditionally Fischer's exact test is defined as
$$H_0:Y_i(1)=Y_i(0)\quad\forall i=1,2,\dots,N \text{ for } N \text{ many permutations.}$$ As our example is dealing with linear regression we can't have this specific null hypothesis, but instead we can interpret it as $$H_0:\hat{\beta_1}_{,i}=0\quad\forall i = 1,2,\dots,N.$$ Now, let's recall the Central Limit Theorem. The CLT says that given a population with mean $\mu$ and standard deviation $\sigma$ and we take sufficiently large random samples from the population with replacement, then our sampling distribution will be approximately normally distributed about $\mu$. What does this mean for our case of simple linear regression? Well, we can think about the population of least squares estimators as every $\hat{\beta_1}$ we would get if were to permute the response column. A critical part of this is that our data from which we are calculating the least squares estimator is a representative random sample from the population. If so, then we can validate this claim that our distribution of least squares estimators is representative of the population as required by the CLT. This is crucial if we are to conduct statistical inference as randomization-based inference spells out. This is the reason we are able to generalize and make inferences about the population, even when model conditions aren't met. Now, with this in mind, each time we take a permutation of our data, we are taking a random sample from the population of $\hat{\beta_1}\text{'s}$. As we saw above, the permutations we make lead to the assumption of the null hypothesis, $H_0:\hat{\beta_1}=0$. Therefore, by the central limit theorem, our reference distribution will be centered around zero because under the null hypothesis we have $E(\beta_1)=0$ and for a normal distribution we have $E(X)=\mu$. If we take all $n!$ permutations of an $n$-length response vector we would have an approximately normal distribution. As we take more permutations and include each newly calculated test statistic in the randomization distribution, we will only further approach normality to the point where more permutations with neither help nor hurt the normality of our reference distribution. This type of distribution is called an asymptotic distribution, thus, we have asymptotic normality in our reference distribution.
\newline\\
Since our distribution in asymptotically normal, our p-value is not only valid, but it will be approximately exact, depending on the number of permutations that we run. If we reach the point where we have approached asymptotic normality, we would not just be approximating a p-value through statistical theory, but actually calculating it on an asymptotic normal distribution. By the same virtue as we showed asymptotic normality, we have an asymptotic p-value since we are calculating a p-value from this asymptotic distribution. We cannot take any more permutations that will influence the p-value since the distribution we are calculating the p-value on is nearly fixed for large numbers of permutations.

\subsection{In General}

This process that we took is true for many test statistics that we can choose. If we know that our sample is truly an approximation of the population, we are able to construct a roughly normal randomization centered around the null hypothesis of our test. Examples of other test statistics that we can use this approach for are means, proportions, difference in means, difference in proportions, and correlations. The same is true for $\chi^2$- and $F$-distributions, however there is a bit more legwork that needs to happen before we can see this clearly. The formula for a $\chi^2$ test statistic is $$\sum\frac{(O_i-E_i)}{E_i}$$ where $O_i$ is each observed data point and $E_i$ is the expected value for that group. Assuming that our data has been randomly sampled from the population, one of the assumptions that we hold for randomization based inference, then $O_i$ will be normal random variable meaning the difference $O_i-E_i$ is also a normal random variable. Assuming the null hypothesis  of no relationship (as we do when we shuffle the data), $E(O_i-E_i)=0$. Thus, the $\chi^2$-distribution will be a sum of squared normal random variables taken from a standard normal distribution. As we shuffle the data and  sample more normal random variables from the standard normal distribution, the sum of their squares will only further approximate a $\chi^2$-distribution. From this randomization distribution we can calculate a p-value by taking the proportion of $\chi^2$ statistics that are greater than what we observed in our original sample over the total number of permutations.
\newline\\
Similarly, we can show that our randomized $F$-distribution will in fact be $F$-distributed using the same reasoning as we did previously to show that random permutations of our data will lead to a $\chi^2$-distribution. Again, we're assuming the null in this case when we randomize our data. For an $F$-test, the null hypothesis is that both models or groups are the same. When we calculate our proportion that we do for an $F$-statistic, we will be taking a proportion of $\chi^2$ random variables (multiplied by a factor of the inverse of each $\chi^2$ random variable's degrees of freedom). By definition, this will be an $F$ random variable. Thus with many permutation we will have the null distribution of $F$-statistics which is centered at 1. Since we've satisfied that we can approximate our necessary distribution, we can calculate a p-value just as above.


\subsubsection{Simulation}
\paragraph{Algorithm 1: Force Calculation Between Two Bodies}
This equation can be used to calculate the gravitational force exerted by two masses on x and y components without the need to use the trigonometric functions.

$$\begin{bmatrix} F_x \\ F_y \end{bmatrix} = \frac{GmM}{r^3} \begin{bmatrix} r_x \\ r_y \end{bmatrix}$$

\paragraph{}
The proof for this equation is fairly straightforward, by equating the original full equation including the trigonometric component finding to the vector equation.

$$F_x=\frac{GmM}{r^2}\cos\theta \hspace{25pt} F_x=\frac{GmM}{r^3}\hat{x} \hspace{25pt} : GmM = 1 $$
\vspace{10pt}
$$F_x=\frac{1}{r^2}\cos\theta \hspace{25pt} F_x=\frac{1}{r^3}\hat{x} \hspace{25pt} : r=\sqrt{\hat{x}^2+\hat{y}^2}$$
\vspace{10pt}
$$F_x=\frac{1}{(\sqrt{\hat{x}^2+\hat{y}^2})^2}\cos\theta = \frac{1}{(\sqrt{\hat{x}^2+\hat{y}^2})^3}\hat{x} \hspace{25pt} : \frac{1}{x^2} \div \frac{1}{x^3}  = x$$
\vspace{10pt}
$$\sqrt{\hat{x}^2+\hat{y}^2}\cos\theta=\hat{x} \hspace{25pt} : \cos\theta=\frac{adj}{hyp} \hspace{25pt} adj=\hat{x} \hspace{25pt} hyp=\sqrt{\hat{x}^2+\hat{y}^2}$$
\vspace{10pt}
$$\sqrt{\hat{x}^2+\hat{y}^2}\frac{\hat{x}}{\sqrt{\hat{x}^2+\hat{y}^2}}=\hat{x} \rightarrow \hat{x} = \hat{x}$$

\paragraph{}
The benefit of not requiring the use of the trigonometric functions is that the management of signs is a lot simpler, as well as a reduced computational requirement and much simpler code, this algorithm is also much easier to use for 3D, as the forces can just be calculated on the z axis based on z distance. \\

Required Parameters:
\begin{itemize}
\item Gravitational Constant ($G$)
\item Mass of Bodies ($m$ $M$)
\item Distance Between Bodies ($r$, $r_x$ and $r_y$)
\end{itemize}

\paragraph{Algorithm 2: Calculation of all Forces}
In order to simulate the interaction of multiple bodies, all relationships between bodies must be calculated, this can be presented as an undirected graph in an abstract sense. This also means that it can be represented as a matrix.

\paragraph{}
When showing this organisation structure as a matrix, it becomes clear as to how many calculations need to be done in order to calculate all the required relationships.

\paragraph{}
Because the 'graph' is undirected, the matrix would be mirrored, the same force applies to both bodies in a relationship.
In order to reduce wasted memory footprint and increase ease of passing the variable, the x and y forces can be packed into the same matrix. (Will require slightly more complex programming when accessing the forces.)

\paragraph{}
Sign of the force must be retained, meaning that the way round that the force is calculated must be constant throughout all (x to y).
The number of individual calculations that need to be done becomes $\frac{1}{2}n^2-n$ for the main force calculation and $n^2-n$ for the directional vector, this is still $O(n^2)$ time complexity per body. \\\

Required Parameters:
\begin{itemize}
\item Current Number of Bodies
\item Scenario Body Structure
\end{itemize}

\begin{figure}[!ht]
  \centering
  $$\begin{array}{c|c|c|c|c|c}
  ~ & \textbf{0} & \textbf{1} & \textbf{2} & \textbf{3} & \textbf{4} \\
  \hline
  \textbf{0} & -      & y_{01} & y_{02} & y_{03} & y_{04} \\
  \hline
  \textbf{1} & x_{01} & -      & y_{12} & y_{13} & y_{14} \\
  \hline
  \textbf{2} & x_{02} & x_{12} & -      & y_{23} & y_{24} \\
  \hline
  \textbf{3} & x_{03} & x_{13} & x_{23} & -      & y_{34} \\
  \hline
  \textbf{4} & x_{04} & x_{14} & x_{24} & x_{34} & -
 \end{array}$$
  \caption{Example Force Matrix Representation (Body Count = 5)}
\end{figure}

\paragraph{Algorithm 3: Summation of Forces on Bodies}
The forces on each body must be summed up so the total force on each body can be resolved to two numbers for x and y respectively.

\paragraph{}
Because the force is only stored in the matrix an implied direction, in certain cases it may be necessary to flip the sign of some of the results to correctly sum the answers, Results will be stored in an individual bodies dataset.

\paragraph{}
The force is calculated x to y on the matrix, meaning that all forces will initially be calculated with a sign that represents this direction (The sign of the unit vector distance).
This function will need to 'traverse' the matrix in a similar way to the previous function, but only add a force if one of the loop axis is equal to the current body ID.

\paragraph{} If the body ID is found in the first array access column, the original sign is kept. If it is found in the second, the sign must be flipped to preserve the correct direction of force relative to the body because of the way round that force has been calculated. \\\

Required Parameters
\begin{itemize}
\item Current Number of Bodies
\item Scenario Body Structure
\item Scenario Component Force Matrix
\end{itemize}

\pagebreak
\paragraph{Algorithm 4: Integration of New Position}
Using the second order leapfrog integration method, calculates change in velocity and change in position/acceleration out of step with each other in order to reduce deviation.

\paragraph{}
Each iteration will be calculated in the order $\frac{1}{2}v \rightarrow r \rightarrow F \rightarrow a \rightarrow \frac{1}{2}v$, but initial acceleration should be calculated should any new bodies be added into the simulation as velocity is calculated at the start of each iteration.

\paragraph{}
In order to calculate the acceleration of each body the force gets calculated for every body and given to the individual body objects, the instantaneous force for that iteration is then used to calculate instantaneous acceleration based on Newton's Second Law or $a=\frac{F}{m}$ after which all other operations are done in the body object methods instead.

\paragraph{}
Velocity and Position are both accumulative properties, meaning that they are conserved in the body objects iteration to iteration. Velocity is $\Delta{v}=a\Delta{t}$, but this will be calculated as $\Delta{v}=\frac{1}{2}a\Delta{t}$ and will be done at two points in the iteration. Position is calculated as $\Delta{r}=v\Delta{t}$. (Velocity and Acceleration are first and second derivatives of position respectively.) \\\

References: \cite{leapfrog} \\\

Required Parameters
\begin{itemize}
\item Current Number of Bodies
\item Scenario Body Structure
\end{itemize}

\paragraph{Algorithm 5: Body Collision Detection and Calculation}
The simulation will have a simple body collision detection algorithm implemented that makes use of basic inelastic collisions, no fragments will be created and colliding bodies will simply merge together in mass and circular area.

\paragraph{}
Detecting if bodies are colliding can be done using the radius property of bodies, if two bodies vector distance is smaller than the sum of their radii, the bodies can be considered to have collided and the result can be calculated. Because this method checks every single body, the time order complexity is $O(n^2)$.

\paragraph{}
If a collision is detected, the momentum for individual bodies can be calculated using the equation $p=mv$, momentum will be conserved in an inelastic collision, meaning that the momentum for both bodies can be added together and then divided by the total mass of both bodies combined to get the new velocity for the result. This will need to be done for both X and Y vectors.

Required Parameters
\begin{itemize}
\item Current Number of Bodies
\item Scenario Body Structure
\end{itemize}

\pagebreak
\subsubsection{Graphics}
\paragraph{Algorithm 6: Circle Drawing}
Due to the fact that OpenGL has no inbuilt support for drawing circle primitives, they must be constructed using vertices and line segments in order to form a polygon (GL\_POLYGON) and fill it in to create a approximated circle.

\paragraph{}
To do this, the center coordinates will be used in conjunction with the radius to calculate trigonometric factors in order to speed up the calculation of the circle. Traditionally, you would calculate the position of the next point in respect to radius by calling the trigonometric functions for every vertex, however with this method they only need to be called once per object as they will be the same for every vertex.

\paragraph{}
The benefit of this is that it still allows for different numbers of segments to be used, higher numbers will make a much smoother looking circle, however it will be dependant somewhat on the radius of the drawn circle, as being able to zoom in will make the straight edges a lot more visible, even when anti-aliasing is used. Due to performance constraints, a lower number here would be better, 16 or 32 is sensible.

$$ \theta=2\pi/Segments \hspace{20pt} k_{\tan}=\tan\theta \hspace{20pt} k_{\cos}=\cos\theta $$
$$ x_{pos} = r \hspace{10pt} \& \hspace{10pt} y_{pos} = 0 $$
$$ x_{pos}+(-y_{pos}k_{\tan})k_{rad} \hspace{10pt} \& \hspace{10pt} y_{pos}+(x_{pos}k_{\tan})k_{rad}$$

References: \cite{circle} \\\

Required Parameters:
\begin{itemize}
\item Center Point
\item Radius
\item Segments
\end{itemize}

\pagebreak
\subsubsection{Interface}
\paragraph{Algorithm 7: Body Selection}
This algorithm will work in a similar way to the collision detection code, but on the end of a single body instead of two. By getting the current position of the cursor in the world by using the function gluUnProject() and checking to see if the cursor is inside the radius of any body when a click occurs. Mouse input is handled by GLFW.

\paragraph{}
This function is used as it will use the current matrices that are used for manipulating what is displayed on screen on the cursor coordinates, providing a perfect mapping of what is happening on screen to the window system coordinates.

\paragraph{}
If a body is selected, a GUI will open which will allow the properties of that body to be modified, and then applied with a button on that GUI.

\paragraph{Algorithm 8: Pan and Zoom}
By holding down middle click and dragging, the user will be able to pan the view around the scene to get a better view of different parts. By scrolling, they will also be able to zoom into the position currently at the center of their screen, mouse coordinates and clicks are provided by GLFW.

\paragraph{}
The panning will be calculated by first waiting to see if the mouse has been held down for at least several hundred milliseconds using the GLFW timer, after which the change in cursor positions can be calculated per frame. Zoom will will modify a scale variable that will have set limits, preventing the user from zooming too far in and out.

\paragraph{}
These variables will then be passed into OpenGL specific functions that will perform transformation and scales on the OpenGL ModelView Matrix, giving the illusion of movement and zooming.

\paragraph{Algorithm 9: Scenario Saving and Loading}
The scenario will be able to be saved by the user using elements provided on the UI. The user will be required to enter a file-name in order to save the scenario. The file extension (.sav) will be automatically added.

\paragraph{}
The file will contain a structured and human readable/editable version of the scenario data used by the program, it contains any scenario control variables and the properties of any bodies present in the simulation. There will also be some utility for auto-generation features, such as providing bodies with orbital velocity relative to any other body in the simulation or the generation of large super-structures.

\paragraph{}
In order to load a save, the user can either drag and drop a .sav file into the window or they can enter the file name into the UI element and click on the 'load' button.

\subsubsection{Management}
\paragraph{Algorithm 10: Thread Synchronisation and Shared Data Management}
When shared data is accessed, other threads must be prevented from accessing (Read or Write) the data at the same time as it could cause corruption or deadlocks due to race conditions. Once the thread is done accessing the data it will unlock it to allow another thread to access it.

\paragraph{}
The simulation thread will get initially get the body storage and control variables from the shared storage area (Previously set up and sent to shared by the main thread). It will then perform a number of iterations based on iterations per frame and place the data in the shared area buffer as well as setting a variable to show that new data is available.

\paragraph{}
If the simulation completes another frames worth of iteration before the renderer has taken the data and unset the new data available flag the simulation thread will wait until that data has been taken.

\paragraph{}
If any changes are made to the input scenario, the main thread will update the shared area and notify the simulation thread that new data is available, the Sim thread will discard any data it is currently carrying and take on the new user changes.


\subsection{Second and Other Derivatives}
If $f$ is a differentiable function then its derivative $f'$ is also a function and so we can take the derivative of $f'$.
The new function, denoted by $f''$, is called the \dfont{second derivative} of $f$, since it is the derivative of the derivative of $f$.

The following symbols represent the second derivative:
$$f''(x)=y''=\frac{d^2y}{dx^2}=\frac{d}{dx}\left(\frac{dy}{dx}\right).$$

We can continue this process to get the third derivative of $f$.

In general, the \dfont{$n$th derivative} of $f$ is denoted by $f^{(n)}$ and is obtained from $f$ by differentiating $n$ times.
If $y=f(x)$, then we write:
$$y^{(n)}=f^{(n)}(x)=\frac{d^ny}{dx^n}.$$


\chapter{RELEVANT BACKGROUND IN LINEAR PROGRAMMING} \label{app:LP}% Must have a blank line after every section label

This appendix presents an operational description of linear programming (LP), a method of constrained optimization for linear systems of inequalities. Linear programs are the common method for solving the maximum concurrent flow problem (MCFP) and hierarchical MCFP (HMCFP). For greater detail, the reader is directed to Luenberger and Ye~\cite{luenberger2008linear} or Chvatal~\cite{chvatal1983linear}.

\section{Linear Programming}

Linear programming optimizes an objective function that is constrained by linear equalities and inequalities. For some linear objective function $z = c \transpose x$, we want to find the $x$ that maximizes $z$ while constrained by a system of linear inequalities $Ax \leq b$.

A linear program in standard form comprises three parts:
\begin{enumerate}
\item A linear \say{cost} function to be maximized, e.g. $ z(x_{1},x_{2}) = c_1 x_1 + c_2 x_2$
\item Problem constraints, e.g.
\begin{equation}
\begin{matrix}
  a_{11} x_1 + a_{12} x_2 &\leq b_1 \\
  a_{21} x_1 + a_{22} x_2 &\leq b_2 \\
  a_{31} x_1 + a_{32} x_2 &\leq b_3 \\
\end{matrix}\end{equation}
\item Non-negative variables, e.g.
\begin{equation}\begin{matrix}
 x_1 \geq 0 \\
 x_2 \geq 0
\end{matrix}\end{equation}
\end{enumerate}
Problems with equality constraints or bounds other than non-negativity can be converted into these three parts; the reader is directed toLuenberger~and Ye~\cite{luenberger2008linear}. The problem is usually expressed in matrix form, and then becomes:
\begin{equation}
\max \{ %
\mathbf{c} \transpose
\mathbf{x} \;|\;
 \mathbf{A} \mathbf{x} \leq \mathbf{b} \land \mathbf{x} \geq 0 \}
\end{equation}

These linear constraints form half-planes that bound a convex polytope (a bounded polyhedron)~\cite{nemhauser1988integer}. Any point inside this polytope is a \emph{feasible solution}.

Solving a linear program (when a solution exists) gives an optimal basis~$\mathbf{B}$. This basis is a matrix of  For a given basis, there is a corresponding \emph{basic solution}~$(\mathbf{x_B}, \mathbf{0})$. Here, $\mathbf{0}$ is the zero vector. The solution only has values for each $x_i$ corresponding to a column in $\mathbf{B}$. At most, $m$ of its entries are nonzero, where $m$ is the number of columns in A. (Here, we assume as for our problem that the number of rows $n \geq m$.

\section{Shadow Prices and Sensitivity}

Every linear program has a corresponding dual program in the form:

\begin{equation}
\min \{ %
\bm{\lambda} \transpose
\mathbf{b} \;|\;
\bm{\lambda} \transpose \mathbf{A} \geq \mathbf{c} \transpose \land \bm{\lambda} \geq 0 \}
\end{equation}

The solutions to the dual program provide \emph{shadow prices}~$\bm \lambda$. The shadow prices reflect the marginal utility of relaxing the corresponding constraint in the primal, or the marginal cost of tightening the constraint:

\begin{align}
\lambda_i = \frac{\partial z}{\partial b_i}, \quad \forall i \in \{1, 2, \ldots, |\lambda|\}
\end{align}

These shadow prices are piecewise linear, and the particular range over which some shadow price~$\lambda_i$ is valid can be computed~\cite{luenberger2008linear}: when a value in $\mathbf{b}$ is altered, the values of the right-hand side in the optimal tableau are altered. The stable range is the interval along which the right-hand side of the tableau remains nonnegative.

\section{Practical Considerations}

LPs can be quickly solved using the simplex method~\cite{puterman2014markov}. Although the method scales exponentially with input size in the worst case, average-case performance is far better. Additional methods have been proposed with polynomial worst-case behavior, such as interior point methods.


\section{The kernel and image of a linear map}

\begin{outcome}
  \begin{enumerate}
  \item Describe the kernel and image of a linear transformation, and
    find a basis for each.
  \end{enumerate}
\end{outcome}

In this section we will consider the case where the linear transformation is not necessarily an
isomorphism. First consider the following important definition.

\begin{definition}{Kernel and image}{kernel-image}
Let $V$ and $W$ be subspaces of $\R^n$ and let $T:V\to W$ be a linear transformation. Then the image of $T$
denoted as $\func{im}(T) $ is defined to be the set
\index{linear map!image}
\index{linear map!kernel}
\begin{equation*}
\func{im}(T) = \set{T (\vect{v}):\vect{v}\in V}
\end{equation*}
In words, it consists of all vectors in $W$ which equal $T(\vect{v})$ for some $
\vect{v}\in V$.

The kernel of $T$, written $\ker (T)$, consists of all $\vect{v}\in V$ such that $T(\vect{v})=\vect{0}$. That is,
\begin{equation*}
\ker (T) =\set{\vect{v}\in V:T(\vect{v})=\vect{0}}
\end{equation*}
\end{definition}

It follows that $\func{im}(T) $ and $\ker (T) $
are subspaces of $W$ and $V$ respectively.

\begin{proposition}{Kernel and image as subspaces}{kernel-image-subspaces}
Let $V, W$ be subspaces of $\R^n$ and let $T:V\rightarrow W$ be a linear transformation. Then $\ker (
T) $ is a subspace of $V$ and $\func{im}(T) $ is a
subspace of $W$.
\end{proposition}

\begin{proof}
First consider $\ker (T)$. It is necessary to
show that if $\vect{v}_{1},\vect{v}_{2}$ are vectors in $\ker (T) $
and if $a,b$ are scalars, then $a\vect{v}_{1}+b\vect{v}_{2}$ is also in $\ker
(T)$. But
\begin{equation*}
T(a\vect{v}_{1}+b\vect{v}_{2}) =aT(\vect{v}_{1})+bT(\vect{v}_{2})=a\vect{0}+b\vect{0}=\vect{0}
\end{equation*}
Thus $\ker (T) $ is a subspace of $V$.

Next suppose $T(\vect{v}_{1}),T(\vect{v}_{2})$ are two vectors in $\func{im}(
T)$. Then if $a,b$ are scalars,
\begin{equation*}
aT(\vect{v}_{2})+bT(\vect{v}_{2})=T(a\vect{v}_{1}+b\vect{v}_{2})
\end{equation*}
and this last vector is in $\func{im}(T) $ by definition.
\end{proof}

We will now examine how to find the kernel and image of a linear transformation and describe the basis of each.

\begin{example}{Kernel and image of a linear transformation}{kernel-image}
Let $T: \R^4 \to \R^2$ be defined by
\[
T \begin{mymatrix}{c}
a \\
b \\
c \\
d
\end{mymatrix} =
\begin{mymatrix}{c}
a - b \\
c + d
\end{mymatrix}
\]
Then $T$ is a linear transformation. Find a basis for $\func{ker}(T)$ and $\func{im}(T)$.
\end{example}

\begin{solution}
You can verify that $T$ is a linear transformation.

First we will find a basis for $\func{ker}(T)$. To do so, we want to find a way to describe all vectors $\vect{x} \in \R^4$ such that $T(\vect{x}) = \vect{0}$. Let $\vect{x} =  \begin{mymatrix}{c}
a \\
b \\
c \\
d
\end{mymatrix}$ be such a vector. Then
\[
T \begin{mymatrix}{c}
a \\
b \\
c \\
d
\end{mymatrix} =
\begin{mymatrix}{c}
a - b \\
c + d
\end{mymatrix} =
\begin{mymatrix}{c}
0 \\
0
\end{mymatrix}
\]

The values of $a, b, c, d$ that make this true are given by solutions to the system
\begin{eqnarray*}
a - b &=& 0 \\
c + d &=& 0
\end{eqnarray*}
The solution to this system is $ a = s, b = s, c = t, d = -t$ where $s, t$ are scalars. We can describe $\func{ker}(T)$ as follows.
\[
\func{ker}(T) = \set{\begin{mymatrix}{r}
s \\
s \\
t \\
-t
\end{mymatrix} }
=
\sspan \set{\begin{mymatrix}{r}
1 \\
1 \\
0 \\
0
\end{mymatrix}, \begin{mymatrix}{r}
0 \\
0 \\
1 \\
-1
\end{mymatrix} }
\]
Notice that this set is linearly independent and therefore forms a basis for $\func{ker}(T)$.

We move on to finding a basis for $\func{im}(T)$. We can write the image of $T$ as
\[
\func{im}(T) = \set{\begin{mymatrix}{c}
a - b \\
c + d
\end{mymatrix}
}
\]
We can write this in the form
\[
\sspan = \set{
\begin{mymatrix}{r}
1 \\
0
\end{mymatrix},
\begin{mymatrix}{r}
-1 \\
0
\end{mymatrix},
\begin{mymatrix}{r}
0 \\
1
\end{mymatrix},
\begin{mymatrix}{r}
0 \\
1
\end{mymatrix} }
\]
This set is clearly not linearly independent. By removing unnecessary vectors from the set we can create a linearly independent set with the same span. This gives a basis for $\func{im}(T)$ as
\[
\func{im}(T) = \sspan \set{
\begin{mymatrix}{r}
1 \\
0
\end{mymatrix},
\begin{mymatrix}{r}
0 \\
1
\end{mymatrix}
}
\]
\end{solution}

Recall that a linear transformation $T$ is called one to one if and only if $T(\vect{x}) = \vect{0}$ implies $\vect{x} = \vect{0}$. Using the concept of kernel, we can state this theorem in another way.

\begin{theorem}{One to one and kernel}{one-to-one-kernel}
Let $T$ be a linear transformation where $\func{ker}(T)$ is the kernel of $T$. Then $T$ is one to one if and only if $\func{ker}(T)$ consists of \textbf{only} the zero vector.
\end{theorem}

A major result is the relation between the dimension of the kernel and
dimension of the image of a linear transformation. In the previous example $\func{ker}(T)$ had dimension $2$, and $\func{im}(T)$ also had dimension of $2$. Is it a coincidence that the dimension of $\Mat_{2,2}$ is $4 = 2 + 2$? Consider the following theorem.

\begin{theorem}{Dimension of kernel and image}{dimension-kernel-image}
Let $T:V\rightarrow W$ be a linear transformation where $V,W$ are subspaces of $\R^n$. Suppose the dimension of $V$ is $m$. Then
\[
m=\dim (\ker (T)) +\dim (\func{im}(
T))
\]
\index{rank added to nullity}
\index{nullity}
\index{rank}
\end{theorem}

\begin{proof}
From Proposition~\ref{prop:kernel-image-subspaces}, $\func{im}(T) $ is a subspace of $W$. We know that there exists a basis for $\func{im}(T)$, $\set{T(\vect{v}
_{1}),\ldots,T(\vect{v}_{r})} . $ Similarly, there is a basis for $\ker
(T) ,\set{\vect{u}_{1},\ldots,\vect{u}_{s}}$. Then if $
\vect{v}\in V$, there exist scalars $c_{i}$ such that
\begin{equation*}
T(\vect{v})=\sum_{i=1}^{r}c_{i}T(\vect{v}_{i})
\end{equation*}
Hence $T\paren{\vect{v}-\sum_{i=1}^{r}c_{i}\vect{v}_{i}} =0$. It follows
that $\vect{v}-\sum_{i=1}^{r}c_{i}\vect{v}_{i}$ is in $\ker (T)$.
Hence there are scalars $a_{i}$ such that
\begin{equation*}
\vect{v}-\sum_{i=1}^{r}c_{i}\vect{v}_{i}=\sum_{j=1}^{s}a_{j}\vect{u}_{j}
\end{equation*}
Hence $\vect{v}=\sum_{i=1}^{r}c_{i}\vect{v}_{i}+\sum_{j=1}^{s}a_{j}\vect{u}
_{j}. $ Since $\vect{v}$ is arbitrary, it follows that
\begin{equation*}
V=\sspan\set{\vect{u}_{1},\ldots,\vect{u}_{s},\vect{v}_{1},\ldots,
\vect{v}_{r}}
\end{equation*}
If the vectors $\set{\vect{u}_{1},\ldots,\vect{u}_{s},\vect{v}_{1},\ldots,
\vect{v}_{r}} $ are linearly independent, then it will follow that
this set is a basis. Suppose then that
\begin{equation*}
\sum_{i=1}^{r}c_{i}\vect{v}_{i}+\sum_{j=1}^{s}a_{j}\vect{u}_{j}=0
\end{equation*}
Apply $T$ to both sides to obtain
\begin{equation*}
\sum_{i=1}^{r}c_{i}T(\vect{v}_{i})+\sum_{j=1}^{s}a_{j}T(\vect{u})
_{j}=\sum_{i=1}^{r}c_{i}T(\vect{v}_{i})=0
\end{equation*}
Since $\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{r})} $ is linearly
independent, it follows that each $c_{i}=0$. Hence $\sum_{j=1}^{s}a_{j}\vect{u
}_{j}=0$ and so, since the $\set{\vect{u}_{1},\ldots,\vect{u}_{s}} $
are linearly independent, it follows that each $a_{j}=0$ also. Therefore $\set{\vect{u}_{1},\ldots,\vect{u}_{s},\vect{v}_{1},\ldots,\vect{v}
_{r}} $ is a basis for $V$ and so
\begin{equation*}
n=s+r=\dim (\ker (T)) +\dim (\func{im}(
T))
\end{equation*}
\end{proof}

The above theorem leads to the next corollary.

\begin{corollary}{}{}
Let $T:V\rightarrow W$ be a linear transformation where $V,W$ are subspaces of $\R^n$. Suppose the dimension of $V$ is $m$. Then
\[
\dim (\ker (T)) \leq m
\]
\[
\dim (\func{im}(T)) \leq m
\]
\end{corollary}

This follows directly from the fact that $n=\dim (\ker (T)) +\dim (\func{im}(
T))$.

Consider the following example.

\begin{example}{}{}
Let $T:\R^{2}\rightarrow \R^{3}$ be defined by
\begin{equation*}
T(\vect{x})=\begin{mymatrix}{rr}
1 & 0 \\
1 & 0 \\
0 & 1
\end{mymatrix} \vect{x}
\end{equation*}
Then $\func{im}(T) =V$ is a subspace of $\R^{3}$ and $T$
is an isomorphism of $\R^{2}$ and $V$. Find a $2\times 3$-matrix $A$
such that the restriction of multiplication by $A$ to $V=\func{im}(
T) $ equals $T^{-1}$.
\end{example}

\begin{solution}
Since the two columns of the above matrix are linearly independent, we conclude that $\dim(\func{im}(T)) = 2$ and therefore $\dim(\func{ker}(T)) = 2 - \dim(\func{im}(T)) = 2-2 = 0$ by Theorem~\ref{thm:dimension-kernel-image}. Then by Theorem~\ref{thm:one-to-one-kernel} it follows that $T$ is one to one.

Thus $T$ is an isomorphism of $\R^{2}$ and the $2$-dimensional subspace of $\R^{3}$ which is the
span of the columns of the given matrix. Now in particular,
\begin{equation*}
T(\vect{e}_{1})=\begin{mymatrix}{r}
1 \\
1 \\
0
\end{mymatrix} ,\ T(\vect{e}_{2})=\begin{mymatrix}{r}
0 \\
0 \\
1
\end{mymatrix}
\end{equation*}
Thus
\begin{equation*}
T^{-1}\begin{mymatrix}{r}
1 \\
1 \\
0
\end{mymatrix} =\vect{e}_{1},\ T^{-1}\begin{mymatrix}{c}
0 \\
0 \\
1
\end{mymatrix} =\vect{e}_{2}
\end{equation*}
Extend $T^{-1}$ to all of $\R^{3}$ by defining
\begin{equation*}
T^{-1}\begin{mymatrix}{c}
0 \\
1 \\
0
\end{mymatrix} =\vect{e}_{1}
\end{equation*}
Notice that the vectors
\begin{equation*}
\set{\begin{mymatrix}{c}
1 \\
1 \\
0
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
0 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
1 \\
0
\end{mymatrix} }
\end{equation*}
are linearly independent so $T^{-1}$ can be extended linearly to yield a
linear transformation defined on $\R^{3}$. The matrix of $T^{-1}$
denoted as $A$ needs to satisfy
\begin{equation*}
A\begin{mymatrix}{rrr}
1 & 0 & 0 \\
1 & 0 & 1 \\
0 & 1 & 0
\end{mymatrix} =\begin{mymatrix}{rrr}
1 & 0 & 1 \\
0 & 1 & 0
\end{mymatrix}
\end{equation*}
and so
\begin{equation*}
A=\begin{mymatrix}{rrr}
1 & 0 & 1 \\
0 & 1 & 0
\end{mymatrix} \begin{mymatrix}{rrr}
1 & 0 & 0 \\
1 & 0 & 1 \\
0 & 1 & 0
\end{mymatrix}^{-1}=\begin{mymatrix}{rrr}
0 & 1 & 0 \\
0 & 0 & 1
\end{mymatrix}
\end{equation*}
Note that
\begin{equation*}
\begin{mymatrix}{rrr}
0 & 1 & 0 \\
0 & 0 & 1
\end{mymatrix} \begin{mymatrix}{c}
1 \\
1 \\
0
\end{mymatrix} =\begin{mymatrix}{c}
1 \\
0
\end{mymatrix}
\end{equation*}
\begin{equation*}
\begin{mymatrix}{rrr}
0 & 1 & 0 \\
0 & 0 & 1
\end{mymatrix} \begin{mymatrix}{c}
0 \\
0 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
0 \\
1
\end{mymatrix}
\end{equation*}
so the restriction to $V$ of matrix multiplication by this matrix yields $
T^{-1}$.
\end{solution}


\chapter{Countable Sets}
In the previous section we studied properties of finite sets; this section
uses developed methods to study of infinite sets.

\section{Equipotent Sets}

\Cref{theorem:bijection-to-equality} says that there is a bijection between
finite sets $X$ to $Y$ iff they have equal cardinalities. In this case we say
that $X$ and $Y$ equipotent.
This idea can be extended to infinite sets.
\begin{definition}
  Let $X$ and $Y$ be sets. We say that $X$ and $Y$ are \emph{equipotent} (this
  is written $\cardinality{X} = \cardinality{Y}$) if there is a bijection from
  $X$ to $Y$.

  We say that a set $X$ is \emph{denumerable} if $X$ and $\N$ are equipotent.
  If the set $X$ is denumerable, then it is said to have cardinality $\aleph_0$
  (this is written $\cardinality{X} = \aleph_0$ and read `aleph null').

  We also say that the set is \emph{countable} if it is finite or denumerable;
  otherwise the set is \emph{uncountable}.\footnote{%
    This notation may vary from book to book and in some of them countable is
    used where this book uses denumerable.
  }
\end{definition}

It is possible to show that $\Z$ is denumerable. Indeed, we may consider $f :
\Z$ to $\N$ such that
\[
  f(x) =
  \begin{cases}
    2x + 2 & \text{if } x \ge 0 \\
    -2x - 1 & \text{if } x < 0
  \end{cases}.
\]
It is clear that $f(-1) = 1$, $f(-2) = 3$, \dots, $f(0) = 2$, $f(1) = 4$, $f(2)
= 6$, \dots, so $f$ is a bijection.
\begin{itemize}
  \item More formally $f$ is a surjection since $f(k / 2 - 1) = k$ for any even
    $k$ and $f(-(k + 1) / 2) = k$ for all odd $k$.
  \item Assume $f(x) = f(y)$ for some $x$ and $y$. Note that if $x$ and
    $y$ have different signs, one of $f(x)$ and $f(y)$ is even and another is
    odd so they cannot be equal to each other.
    \begin{itemize}
      \item If $x, y \ge 0$, then $f(x) = 2x + 2 = f(y) = 2y + 2$ implies that
        $x = y$.
      \item If $x, y < 0$, then $f(x) = -2x - 1= f(y) = -2y - 1$ also implies
        that $x = y$.
    \end{itemize}
    Hence, $f$ is an injection.
\end{itemize}

\begin{exercise}
  Show that $\N_0$ and $\Z$ are equipotent.
\end{exercise}

\section{Properties of Denumerable Sets}

For finite sets, their cardinality determine if they are equipotent or not; the
same property can be proven for denumerable sets.
\begin{remark}
\label{remark:denumerable-equipotent-to-denumerable}
  Let $X$ and $Y$ be sets such that $X$ is denumerable. Then $Y$ is denumerable
  iff $X$ and $Y$ are equipotent.
\end{remark}

The additive principle states that union of two finite sets is also finite
though normally it is bigger; however, in case of denumerable sets only first
part is true.
\begin{theorem}
\label{theorem:union-denumerable-sets}
  Let $X$ and $Y$ be denumerable sets. Then $X \cup Y$ is also denumerable.
\end{theorem}

\begin{exercise}
  Prove \Cref{theorem:union-denumerable-sets}.
\end{exercise}

We can also prove an analogue of the multiplicative principle.
\begin{theorem}
  Let $X$ and $Y$ be denumerable sets. Then $X \times Y$ is also denumerable.
\end{theorem}
\begin{proof}
  It is clear that it is enough to prove the statement for $X = \N$ and $Y =
  \N$. The set $\N^2$ can be represented as an infinite grid:
  \[
    \begin{matrix}
      (1, 1) & (2, 1) & (3, 1) & \dots \\
      (1, 2) & (2, 2) & \dots \\
      (1, 3) & \dots \\
      \dots \\
    \end{matrix}.
  \]
  We cannot enumerate elements of this table column-wise or row-wise since they
  are infinite. However, we can enumerate them along the diagonals, as follows:
  \[
    \begin{matrix}
      1 & 3 & 6 & \dots \\
      2 & 5 & \dots \\
      4 & \dots \\
      \dots \\
    \end{matrix}.
  \]
  Note $n$th diagonal contains $n$ elements so the element $(m, n)$ appears as
  the $m$th element of the $(m + n - 1)$th diagonal and has the number
  $1 + 2 + \dots + (m + n - 2) + n = (m + n - 1) (m + n - 2) / 2 + n$.
\end{proof}

In addition we may prove the following.
\begin{theorem}
  Let $X$ and $Y$ be two sets such that $X$ is denumerable and $Y \subseteq X$.
  Then $Y$ is countable.
\end{theorem}
\begin{proof}
  It is clear that the statement is true if $Y$ is finite and $X = \N$.
  Hence, let us assume that $Y \subseteq \N$ is infinite. So we need to
  construct a bijection $f : Y \to \N$; we are going to do it inductively.

  We start with the base case.
  Let $f(1)$ be the smallest element of $Y$; it exists by the well-ordering
  principle (\Cref{theorem:well-ordering}).

  For the induction step from $k$ to $k + 1$, assume that $f(k)$ is defined.
  Let $f(k + 1)$ be the minimal element of $\set[y > f(k)]{y \in Y}$ (this set
  is not empty since $Y$ is infinite).

  This process clearly defines an increasing function from $\N$ to $Y$. So to
  finish the proof we need to prove that it is a surjection.
  Let $y \in Y$ and let $n$ be the minimal number such that $f(n) \ge y$ (such a
  number exists since it can be proven that $f(n) \ge n$). Then by the
  definition of $f$, $f(n) = y$.
\end{proof}

Using the last two results we can make the following observation.
\begin{theorem}
  The set of rationals $\Q$ is denumerable.
\end{theorem}

\begin{chapterendexercises}
    \exercise[recommended] Show that $\set[n \in \Z]{2n}$ is denumerable.
    \exercise Show that $\set[n \in \Z]{n^2}$ is denumerable.
    \exercise Let $X$ be a countable set and $n$ be a positive integer. Prove
      that $X^n$ is countable.
    \exercise[recommended] Let $X_1$, \dots, $X_n$, \dots be finite sets. Show that
      $\bigcup_{i \in \N} X_i$ is countable.
    \exercise Let $X_1$, \dots, $X_n$, \dots be denumerable sets. Show that
      $\bigcup_{i \in \N} X_i$ is denumerable.
    \exercise Show that $\set{0, 1}^\N$ is equipotent to $\set{0, 1, 2, 3}^\N$.
\end{chapterendexercises}


\subsection{Introduction to Determinant}

Can define the $2\times 2$ determinant as a function $D:\mathbb{M}_{2\times2}\rightarrow R$.
It can be observed that $2\times 2$ matrix $A$ is only invertible if $D(A)=ad-bc\neq 0$.

\subsection{Cross-Product}

Given $\begin{bmatrix}a\\ c\end{bmatrix}$ and $\begin{bmatrix}b\\ d\end{bmatrix}$, the cross
product is defined as the $\R^3$ vector $D(A)\tb{e}_3=(ad-bc)\tb{e}_3$. The direction
of this vector is the sign of $\mathrm{det}(A)$.

\noindent
Can visualize using right hand rule: if sweeping index into middle is appropriate for the vectors,
then the direction of thumb is cross-product direction (positive). Otherwise, sign is negative.

\subsubsection{Algorithm}

For $\tb{v}=\begin{bmatrix}v_x\\ v_y\\ v_z\end{bmatrix}$ and $\tb{w}=\begin{bmatrix}w_x\\ w_y\\ w_z\end{bmatrix}$:

\[\boxed{\tb{v}\times\tb{w}=c_z\tb{e}_1+c_y\tb{e}_2+c_z\tb{e}_z}\]

Following through, to calculate each component ignore the desired row and perform cross-product on remaining matrix:

\[c_x=\begin{bmatrix}v_y\\ v_z\end{bmatrix}\times \begin{bmatrix}w_y\\ w_z\end{bmatrix}=v_yw_z-v_zw_z\]

The $y$ component is done as $bc-ad$ compared to $ad-bc$.

\[c_y=\begin{bmatrix}v_x\\ v_z\end{bmatrix}\times \begin{bmatrix}w_x\\ w_z\end{bmatrix}=w_xv_z-w_zv_x\]

\[c_z=\begin{bmatrix}v_y\\ v_z\end{bmatrix}\times \begin{bmatrix}w_y\\ w_z\end{bmatrix}=v_xw_y-v_yw_x\]

\subsection{Determinant Theory}

Considering $A=\begin{bmatrix}|&|&| \\\tb v_1 & \tb v_2 &\tb v_3 \\ |&|&| \\ \end{bmatrix}$,
say it is singular such that $\tb{v}_3\in \mathrm{span}\{\tb{v}_1,\tb{v}_2\}$. Because it is assumed that
$\tb{v}_1,\tb{v}_2$ are linearly independent, then $\mathrm{span}\{\tb{v}_1,\tb{v}_2\}$ is perpendicular to
$\tb{v}_1\times \tb{v}_2$ by definition. Thus, $\boxed{(\tb{v}_1\times\tb{v}_2)\cdot \tb{v}_3=0}$.
If $\tb{v}_1,\tb{v}_2$ are not linearly independent, then this is still 0 because the cross-product (area of parallelogram made by vectors)
is still 0.

\noindent
If $\tb{v}_1,\tb{v}_2,\tb{v}_3$ are linearly independent, then $(\tb{v}_1\times\tb{v}_2)\cdot \tb{v}_3\neq0$.

\[\boxed{D(A)=\mathrm{det}(A)=(\tb{v}_1\times\tb{v}_2)\cdot \tb{v}_3}\]

\noindent
The sub-matrices used when computing cross-products are \textbf{minors}.
Can rewrite determinant:

\[\boxed{\det A= a_{1,3}\left|A_{1,3}\right|-a_{2,3}\left|A_{2,3}\right|+a_{3,3}\left|A_{3,3}\right|}\]

Must use following rule for the sign of constant terms $a_{m,n}$ (dot product):

\[\begin{bmatrix}+&-&+\\-&+&-\\+&-&+\end{bmatrix}\]

\subsection{Rules}

Determinant of $n\times n$ follows recursively:

\[\boxed{\mbox{det}A=a_{1,1}\left|A_{1,1}\right|-a_{1,2}\left|A_{1,2}\right|+a_{1,3}\left|A_{1,3}\right|+\dots\pm a_{1,n}\left|A_{1,n}\right|}\]

\noindent
Rules:
\begin{itemize}
    \item Swapping rows multiplies determinant by -1
    \item Multiplying row by $m$ scales determinant by $m$
    \item Replacing row with sum of row and multiple of another does not change determinant
    \item If $A$ and $B$ are $n\times n$, then $\mathrm{det}(AB)=\mathrm{det}(A)\mathrm{det}(B)$
    \item Cramer's rule: If $A\tb{x}=\tb{b}$ is a linear system with invertible $A$ then $\tb{x}$ components can be determined
    from $x_i=\frac{\mathrm{det}(A-_{b,i})}{\mathrm{det}(A)}$ where $A-_{b,i}$ replaces $i^{\mathrm{th}}$ column of $A$ with $\tb{b}$
\end{itemize}


\section{Recurrence and Transience of Random Walks}
\begin{example}[Simple random walk on $\mathbb Z$]
    Consider the Markov chain with $I=\mathbb Z$ with $p_{i,i+1}=p,p_{i,i-1}=q=1-p$ where $p\in (0,1)$.
    We shall try to compute $p_{00}^{(N)}$.
    Now if $N$ is odd, then obviously $p_{00}^{(N)}=0$ by a parity argument.
    If $N=2n,n\in\mathbb Z$,
    $$p_{00}^{(2n)}=\binom{2n}{n}p^nq^n=\frac{(2n)!}{(n!)^2}p^nq^n$$.
    Stirling's formula gives $n!\sim\sqrt{2\pi n}e^{-n}n^n$ for large $n$ which gives
    $$p_{00}^{(2n)}\sim\frac{\sqrt{4\pi n}}{2\pi n}\frac{(2n)^{2n}}{n^{2n}}(pq)^n=\frac{C}{\sqrt{n}}(4pq)^n$$
    where $C$ is a constant we do not really care about at the moment.
    In the symmetric case $p=q=1/2$, we have $p_{00}^{(2n)}\sim C/\sqrt{n}\ge C/(2\sqrt{n})$ for $n\ge n_0$ for some $n_0$.
    Hence
    $$\sum_{n=0}^\infty p_{00}^{(n)}\ge \sum_{n=n_0}p_{00}^{(2n)}\ge \frac{C}{2}\sum_{n=n_0}^\infty \frac{1}{\sqrt{n}}=\infty$$
    so the random walk is recurrent.
    Otherwise $p\neq q$, in which case necessarily $4pq<(2p+2q)/2=1$ by AM-GM, so $p_{00}^{(2n)}\le r^n$ for $n\ge n_0$, so $\sum_np_{00}^{(n)}$ converges, therefore this random walk is transient.
    Hence symmetric random walks on $\mathbb Z$ is recurrent while asymmetric ones are transient.
\end{example}
\begin{example}[Simple symmetric random walk on $\mathbb Z^2$]
    Take $I=\mathbb Z^2$ and
    $$p_{ij}=\begin{cases}
        1/4\text{, if $|i-j|=1$}\\
        0\text{, otherwise}
    \end{cases}$$
    We will use a nice trick, which sadly doesn't work in high dimensions in general.
    Suppose $X_0=0$ and write $X_n^{\pm}$ as the orthogonal projections of $X_n$ onto the diagonal lines $y=\pm x$.
    Now $X_n^\pm$ are independent symmetric random walks in $2^{-1/2}\mathbb Z$.
    Also $X_0=0$ iff $X_0^{\pm}=0$.
    So by Stirling's approximation.
    $$p_{00}^{(2n)}=\left( \binom{2n}{n}\left(\frac{1}{2}\right)^{2n} \right)^2\sim\frac{C}{n}$$
    for some constant $C$.
    But then by summing them up we know this random walk is also recurrent.
\end{example}
Some na\"ive kids will then conjecture that any symmetric random walks on $\mathbb Z^n$ is recurrent.
Well, they are called na\"ive for a reason.
\begin{example}[Simple symmetric random walk on $\mathbb Z^3$]
    Consider $I=\mathbb Z^3$ and
    $$p_{ij}=\begin{cases}
        1/6\text{, if $|i-j|=1$}\\
        0\text{, otherwise}
    \end{cases}$$
    Again $p_{00}^{(N)}=0$ for odd $N$.
    All walks from $0$ to $0$ must take the same number of steps in directions $\pm e_i$, so
    \begin{align*}
        p_{00}^{(2n)}&=\sum_{i,j,k\ge 0,i+j+k=n}\binom{2n}{i,i,j,j,k,k}\left( \frac{1}{6} \right)^{2n}\\
        &=\binom{2n}{n}\left(\frac{1}{2} \right)^{2n}\sum_{i,j,k\ge 0,i+j+k=n}\binom{n}{i,j,k}^2\left( \frac{1}{3} \right)^{2n}
    \end{align*}
    Now if $n=3m$, it can be easily seen that $\binom{n}{i,j,k}$ is at most $\binom{n}{m,m,m}$.
    Also, we have
    $$\sum_{i,j,k\ge 0,i+j+k=n}\binom{n}{i,j,k}\left( \frac{1}{3} \right)^n=1$$
    by the multinomial theorem.
    These facts then implies that if $n=3m$,
    $$p_{00}^{(2n)}\le\binom{2n}{n}\left( \frac{1}{2} \right)^{2n}\binom{3m}{m,m,m}\left( \frac{1}{3} \right)^{3m}\sim Cn^{-3/2}$$
    In addition, we know that $p_{00}^{(2n)}\ge (1/6)^2p_{00}^{(2n-2)}$, so we can modify our $C$ by a factor to obtain $p_{00}^{(2n)}\le Cn^{-3/2}$ for large enough $n$, which shows
    $$\sum_{n}p_{00}^{(2n)}\le C\sum_{n}n^{-3/2}<\infty$$
    Hence this random walk is transcient.
\end{example}


\chapter{Discrete Fourier Transform}

The Discrete Fourier Transform or DFT is the Fourier Transform of a finite length DT signal. As we shall see, the DFT/FFT is mathematically equivalent to the Discrete-Time Fourier Series. It can be viewed as a way to numerically approximate the CT Fourier transform. Lets first just state the transform and then derive it and see how to interpret and apply it.

Given a finite-length sequence of real or complex numbers $x[n]$, indexed from $0$ to $N-1$, the \emph{Discrete Fourier Transform} or DFT is given by
\[
X[k] = \sum_{n = 0}^{N-1}  x[n] e^{-j \frac{2\pi}{N}k n}
\]
for $k = 0, 1, 2, \cdots, N-1$. When $N$ is a power of 2, an efficient algorithm to compute this result exists and is called the \emph{Fast Fourier Transform} or FFT.

\section{Numerically Approximating the CT Fourier Transform}

Recall the CT Fourier transform pair $x(t) \stackrel{\mathcal{F}}{\longleftrightarrow} X(j\omega)$:
\[
X(j\omega) = \int\limits_{-\infty}^{\infty} x(t) e^{-j\omega t} \; dt
\]

\[
x(t) = \frac{1}{2\pi} \int\limits_{-\infty}^{\infty} X(j\omega) e^{j\omega t} \; d\omega
\]

How could we compute these when we have a physical signal, rather than just a mathematical model?

Recall from calculus the (left) Riemann sum approximation of a definite integral
\[
\int\limits_{t_1}^{t_2} f(t) \; dt = \lim_{N \rightarrow \infty} \sum_{n = 0}^{N-1} \frac{t_2-t_1}{N} f\left(t_1 + n\frac{t_2-t_1}{N}\right)
\]
\begin{center}
  \includegraphics[scale=0.9]{graphics/riemann_sum.pdf}
\end{center}
In the case of the CTFT, if the signal $x(t)$ is non-zero only over some interval $(t_1, t_2)$, then
\[
\mathcal{F}\{ x(t) \} = \int\limits_{t_1}^{t_2} x(t) e^{-j\omega t} \; dt \approx \sum_{n = 0}^{N-1} \frac{t_2-t_1}{N} x\left(t_1 + n\frac{t_2-t_1}{N}\right) e^{-j\omega \left(t_1 + n\frac{t_2-t_1}{N}\right)}
\]
for large $N$.\\[1em]

If we define the time sample spacing as $T = \frac{t_2-t_1}{N}$, then
\[
X(j\omega) \approx \sum_{n = 0}^{N-1} T \, x\left(t_1 + nT\right) e^{-j\omega \left(t_1 + nT\right)}
\]

Note $x\left(t_1 + nT\right)$ corresponds to \emph{samples} of $x(t)$ starting at $t_1$ with sampling interval $T$. This information is equivalent to the triad $t_1, T, x[n]$, where $x[n]$ is a finite length sequence of numbers, i.e. a DT signal where $x[n] = 0$ for $n < 0$ and $n \geq N$. Thus,

\[
x\left(t_1 + nT\right) = x[n]
\]

Substituting into our approximation
\[
X(j\omega) \approx \sum_{n = 0}^{N-1} T \, x\left(t_1 + nT\right) e^{-j\omega \left(t_1 + nT\right)} = T e^{-j\omega t_1} \sum_{n = 0}^{N-1} x[n] e^{-j\omega n T}
\]

Now, consider a sampling of the frequency axis $\omega = \frac{2\pi}{NT} k$. Then
\[
\omega n T = \frac{2\pi}{N T} k n T = \frac{2\pi}{N}k n
\]
and
\[
X\left(j\frac{2\pi}{NT} k\right) = T e^{-j\omega t_1} \underbrace{\sum_{n = 0}^{N-1} x[n]  e^{-j \frac{2\pi}{N}k n}}_{\text{DFT}} = T e^{-j\omega t_1} \, X[k]
\]

Thus we see the DFT corresponds to the Fourier transform of a sampled CT signal over a limited time-interval, at samples of the frequency axis.

Similarly, in the case of the Inverse CTFT, if the signal $X(j\omega)$ is non-zero only over some interval $(\omega_1, \omega_2)$, then
\begin{align*}
  \mathcal{F}^{-1}\{ X(j\omega) \} &= \frac{1}{2\pi} \int\limits_{\omega_1}^{\omega_2} X(j\omega) e^{j\omega t} \; dt\\ &\approx \frac{1}{2\pi} \sum_{k = 0}^{M-1} \frac{\omega_2-\omega_1}{M} X\left(\omega_1 + k\frac{\omega_2-\omega_1}{M}\right) e^{j \left(\omega_1 + k\frac{\omega_2-\omega_1}{M}\right)t}
\end{align*}
for large $M$. If we define the frequency sample spacing as $W = \frac{\omega_2-\omega_1}{M}$, then
\[
x(t) \approx \sum_{m = 0}^{M-1} \frac{W}{2\pi}  \, X\left(\omega_1 + kW\right) e^{j\left(\omega_1 + kW\right)t}
\]

Note $X\left(\omega_1 + mW\right)$ corresponds to samples of $X(j\omega)$ starting a $\omega_1$ with sampling interval $W$. This information is equivalent to the triad $\omega_1$, $W$, $X[k]$, where $X[k]$ is a finite length sequence of numbers where
\[
X\left(\omega_1 + mW\right) = X[k]
\]
Substituting
\[
x(t) \approx \sum_{m = 0}^{M-1} \frac{W}{2\pi}  \, X\left(\omega_1 + kW\right) e^{j\left(\omega_1 + kW\right)t} = \frac{W}{2\pi} e^{j\omega_1} \sum_{m = 0}^{M-1}  X[k] e^{j kWt}
\]

Consider the sampling of the time axis in the derivation of the DFT, $t = nT$. Let $\omega_1 = 0$ and $\omega_2 = \frac{2\pi}{T}$ and $M = N$. Then
\[
kWt = kWnT = k \frac{2\pi}{NT} n T = \frac{2\pi}{N}kn
\]
Since $W = \frac{2\pi}{NT}$
\[
x(nT) = \frac{1}{T} \underbrace{\frac{1}{N} \sum_{k = 0}^{N-1}  X[k] e^{j \frac{2\pi}{N}k n}}_{\text{Inverse DFT}} = \frac{1}{T} x[n]
\]
Thus we see the IDFT corresponds to the Inverse Fourier transform of a sampled Fourier Transform over a limited bandwidth, at samples of the time axis.

This gives us the DFT pair
\[
X[k] = \text{DFT} \{ x[n] \} = \sum_{n = 0}^{N-1}  x[n] e^{-j \frac{2\pi}{N}k n}
\]
\[
x[n] = \text{IDFT} \{ x[n] \} = \frac{1}{N} \sum_{k = 0}^{N-1}  X[k] e^{j \frac{2\pi}{N}k n}
\]
Note the similarity to the DT Fourier Series when $N_0 = 0$
\[
x[n] = \sum\limits_{k = N_0}^{N_0 + N - 1} a_k e^{j\frac{2\pi}{N}kn} =  \sum\limits_{k = 0}^{N - 1} a_k e^{j\frac{2\pi}{N}kn}
\]
\[
a_k = \frac{1}{N}\sum\limits_{n = N_0}^{N_0 + N - 1} x[n] e^{-j\frac{2\pi}{N}kn} =  \frac{1}{N}\sum\limits_{n = 0}^{N - 1} x[n] e^{-j\frac{2\pi}{N}kn}
\]

\section{Efficient Computation of DFT (FFT)}

Given the DFT pair and an input signal, it is easy to compute the DFT. For example in C++ we can define a signal as an array of complex numbers

\begin{verbatim}
#include <complex>
#include <vector>

typedef Signal std::vector<std::complex<double>>;
\end{verbatim}

and implement the DFT in a straightforward translation of the expressions above:

\begin{verbatim}
Signal dft(const Signal & in){

  Signal out = in;

  const int N = in.size();

  for(int k = 0; k < N; ++k){
    out[k] = 0;
    for(int n = 0; n < N; ++n){
      out[k] += in[n]*exp(-j*2.*PI*double(k)*double(n)/double(N));
    }
  }

  return out;
}
\end{verbatim}

Because of the nested for loops the number of multiplies and adds required to compute the DFT is proportional to the number of samples in the signal, squared. However, by expanding the complex exponential we see
\begin{align*}
  X[k] &= \sum_{n = 0}^{N-1}  x[n] e^{-j \frac{2\pi}{N}k n}\\
  &= \sum_{n = 0}^{N-1}  x[n] \left\{ \cos\left( -\frac{2\pi}{N}k n\right) + j \sin\left(-\frac{2\pi}{N}k n \right)\right\}\\
  &= \sum_{n = 0}^{N-1}  x[n] \left\{ \cos\left( \frac{2\pi}{N}k n\right) - j \sin\left(\frac{2\pi}{N}k n \right)\right\}
\end{align*}
Which can be compactly written as
\[
X = \mathcal{W} x
\]
where $x \in \mathbb{C}^N$ is the sampled signal treated as a complex-valued vector, and $\mathcal{W} \in \mathbb{C}^{N\times N}$ is a complex-valued matrix with entries
\[
\mathcal{W}_{k\,n} = \cos\left( \frac{2\pi}{N}k n\right) - j \sin\left(\frac{2\pi}{N}k n \right) = \left( e^{-j\frac{2\pi}{N}}\right)^{kn}
\]

Similarly for the inverse DFT
\begin{align*}
  x[n] &= \sum_{k = 0}^{N-1}  X[k] e^{j \frac{2\pi}{N}k n}\\
  &= \sum_{k = 0}^{N-1}  X[k] \left\{ \cos\left( \frac{2\pi}{N}k n\right) + j \sin\left(\frac{2\pi}{N}k n \right)\right\}
\end{align*}
Which can be compactly written as
\[
x = \frac{1}{N} \mathcal{W}^{*} X
\]

This implies $\frac{1}{N}\mathcal{W}\mathcal{W}^* = I$. This special structure ($\mathcal{W}$ is orthogonal) is what enables the Fast Fourier Transform algorithm to compute the DFT/IDFT in $O(N\log_2 N)$ multiply/adds. The most common algorithm for implemeting the FFT is called the Cooley–Tukey radix-2 algorithm. This algorithm can be implemented using C++ as:

\begin{verbatim}
Signal fft(const Signal & in){

  Signal out;

  std::size_t n = in.size();
  double logn = log2(n);

  for (unsigned int i = 0; i < n; ++i) {
    int rev = bitReverse(i, logn);
    out[i] = in[rev];
  }

  // make sure logn is positive integer > 1
  std::size_t temp = static_cast<std::size_t>(logn);
  assert(static_cast<double>(temp) == logn);

  for(std::size_t s = 1; s <= logn; ++s){
    std::complex<double> w(1,0);

    int m = 1 << s; // 2 power s
    int m2 = m >> 1; // m2 = m/2 -1

    std::complex<double> wm = exp(-PI*j/static_cast<double>(m2));

    for(std::size_t j = 0; j < m2; ++j){
      for(std::size_t k = j; k < n; k+=m){
        std::complex<double> t = w*out[k+m2];
        std::complex<double> u = out[k];
        out[k] = u + t;
        out[k+m2] = u - t;
      }
      w = w*wm;
    }
  }

  return out;
}
\end{verbatim}

where the function \texttt{bitReverse} reverses the bitwise representation of the index argument

\begin{verbatim}
unsigned int bitReverse(unsigned int x, int log2n){
  int n = 0;
  for (int i = 0; i < log2n; i++){
    n <<= 1;
    n |= (x & 1);
    x >>= 1;
  }
 return n;
}
\end{verbatim}

\section{DFT/FFT in Matlab}

In Matlab (and other languages/platforms) you can use the Fast Fourier Transform to compute the DFT. For example:

\begin{verbatim}
>> T = 0.001;
>> t = 0:T:100;
>> x = cos(2*pi*t);
>> X = fft(x);
>> plot(abs(X))
\end{verbatim}
To give a plot consistent with the CTFT
\begin{verbatim}
>> w = (-pi/0.001):(2*pi/100):(pi/0.001);
>> stem(w, T*fftshift(abs(X)))
\end{verbatim}
or
\begin{verbatim}
>> stem(w, T*fftshift(angle(X)))
\end{verbatim}

\section{Summary of Fourier Transforms}

\begin{itemize}
\item Discrete-time Fourier Series: periodic DT signal $x[n] \mapsto a_k$ periodic discrete frequencies
\item Discrete-time Fourier Transform: aperiodic DT signal of indefinite length $x[n] \mapsto X(e^{j\omega})$ periodic continuous frequencies
\item Continuous-time Fourier Series: periodic CT signal  $x(t) \mapsto a_k$ discrete frequencies of indefinite length
\item Continuous-time Fourier Transform: aperiodic CT signal of indefinite length  $x(t) \mapsto X(j\omega)$ continuous frequencies
\end{itemize}

And now we have the DFT
\begin{itemize}
\item Discrete Fourier Transform: aperiodic DT signal of finite length $x[n] \mapsto X[k]$ periodic discrete frequencies
\end{itemize}

\section{Applications the DFT}

Our discussion of the DFT raises some important questions:

\begin{itemize}
\item For what values of sampling interval $T$ does this hold?
\item What are the effects of time and frequency sampling on $x(t)$ and $X(j\omega)$?
\item What if $x(t)$ or $X(j\omega)$ is non-zero outside the interval?
\end{itemize}
These will be answered in the last two lectures. It also admits some important applications:
\begin{itemize}
\item Numerical computation of Fourier transform of physical signals
\item Simulation or approximation of stable CT systems
\item Implementation of CT systems using DT systems
\end{itemize}

As an example application, suppose you have a physical signal, say an audio signal from a microphone. How would you estimate it's Fourier Transform? Sample $x(t)$ at a frequency of $\frac{2\pi}{T}$ for $NT$ seconds.
\[
x[n] = x(nT)
\]
\[
X[k] = \text{DFT}\left\{ x[n] \right\}
\]
\[
X\left(j\frac{2\pi}{NT} k\right) = T X[k]
\]
Note, in practice this requires multiplication by a windowing function to get good results unless there is silence on either side of the audio.

\begin{example}
  Consider a CT signal $x(t) = \cos(2\pi t)\left[u(t) - u(t-100)\right]$ sampled at a frequency of $\frac{2\pi}{0.001}$ for $NT = 100$ seconds to obtain $x[n]$. Given the DFT of $x[n]$, $X[k]$, what values of $k$ correspond to $\omega = \pm 2\pi$?

  \[
  \omega = 2\pi = \frac{2\pi}{NT} k \implies k = 100
  \]

  \[
  \omega = - 2\pi = \frac{2\pi}{NT} k \implies k = -100
  \]
  However $k \in (0, N-1)$ where $N= 100000$. Thus $k = -100 = N - 100 = 99900$. Note, the Matlab command \texttt{fftshift} does this unwrapping for you.
\end{example}

As another application, suppose you have a CT frequency response, for example a CT filter. How could you simulate the response to a physical signal, such as an audio signal from a microphone? Again, sample $x(t)$ at a frequency of $\frac{2\pi}{T}$ for $NT$ seconds.
\[
x[n] = x(nT)
\]
\[
X[k] = \text{DFT}\left\{ x[n] \right\}
\]
Using the convolution property of the CTFT
\[
Y[k] = H\left(j\frac{2\pi}{NT} k\right) X[k]
\]
\[
y(nT) = \frac{1}{T} \text{IDFT}\left\{ Y[k] \right\}
\]

As a final application example we consider the case of filtering. DT implementations of CT systems have a number of benefits over CT implementations. The previous application hints at a method to implement a CT system using a DFT. We sample $x(t)$ at a frequency of $\frac{2\pi}{T}$ for $NT$ seconds into a buffer, called a \emph{frame}.
\[
x[n] = x(nT)
\]
\[
X[k] = \text{DFT}\left\{ x[n] \right\}
\]
\[
Y[k] = H\left(e^{j\frac{2\pi}{NT} k}\right) X[k]
\]
\[
y(t) \approx y(nT) = \frac{1}{T} \text{IDFT}\left\{ Y[k] \right\}
\]
This last step is called \emph{reconstruction}. Note this can be done in real time using three frames, one being sampled, one being processing, and one being reconstructed.

Note the DT filter in the previous application adds a two frame delay. This delay can be removed using an FIR or IIR filter implementation, as we saw in lecture 25. We can sample $x(t)$ at a frequency of $\frac{2\pi}{T}$ continuously into a \emph{ring buffer}.
\[
x[n] = x(nT)
\]
Compute $y[n]$ using a delay difference equation, e.g. for second order
\[
y[n] = a_1 y[n-2] + a_2 y[n-1] + a_3 x[n-2] + a_4 x[n-1] + a_5 x[n]
\]
Reconstruct the current output
\[
y(t) \approx y(nT) = y[n]
\]


\subsection{Rolle's theorem}

\subsubsection{Rolle’s theorem}

Take a real function \(f(x)\) on closed interval \([a,b]\), differentiable on \((a,b,)\), and \(f(a)=f(b)\).

Rolle’s theorem states that:

\(\exists c\in(a,b) (f’(c)=0)\)

Generalised Rolle’s theorem states that:

Generalised Rolle’s theorem implies Rolle’s theorem, so we only need to prove the generalised theorem.