\section{Deterministic Capital Budgeting}
\label{sec:DeterministicCapitalBudgeting}

We consider a capital budgeting problem for a nuclear generation station, with possible extension to
a larger fleet of plants.
Due to limited resources, we can only select a subset from a list of
several candidate capital projects.
Our goal is to maximize overall NPV associated with the
selected subset.
In doing so, we must respect resource limits and capture key structural and
stochastic dependencies of the system, although in this section we start with the simpler
deterministic case, ignoring randomness.
Example projects include upgrading a steam turbine,
refurbishing or replacing a set of reactor coolant pumps, and replacing a set of feed-water heaters.

\[
\begin{array}{ll}
%%%%%%%%%%%%%% INDICES AND SET %%%%%%%%%%%%%%%%
\multicolumn{2}{l}{\mbox{\em Indexes and sets:} } \\
t \in T  & \mbox{time periods (years)} \\
i \in I  & \mbox{investment candidate projects} \\
j \in J_{i}	& \mbox{options for selecting project $i$} \\%, e.g., initiate project $i$ in year $t$ or $t+2$ and in a standard (three year) or in an expedited (two year) manner} \\
% i^{'},j^{'} \in IJ_{ij} & \mbox{piggybacking situations} \\%, i.e., option $j^{'}$ for project $i^{'}$ can be selected only if option $j$ is selected for project $i$} \\
k \in K	& \mbox{types of resources} \\%, e.g., capital funds, O\&M funds, labor-hours, time during outage} \\
\\
%%%%%%%%%%%%%% DATA %%%%%%%%%%%%%%%%
\multicolumn{2}{l}{\mbox{\em Data:}} \\
a_{ij} & \mbox{reward (revenue minus financial cost) of selecting project $i$ via option $j$}  \\
b_{kt} & \mbox{available budget for a resource of type $k$ in year $t$}\\
c_{ijkt}  & \mbox{consumption of resource of type $k$ in year $t$ if project $i$ is performed via option $j$} \\
\\
%%%%%%%%%%%%%% DECISION VARS %%%%%%%%%%%%%%%%
\multicolumn{2}{l}{\mbox{\em Decision variables:}}  \\
x_{ij} & \mbox{1 if project $i$ is selected via option $j$; 0 otherwise} \hspace*{4.0in}\\
\end{array}
\]

\vst \noi {\em Formulation:}
\begin{subequations}\label{model-deter}
\begin{eqnarray}
&\dst \max_{x} &  \dst \sum_{i \in I, j \in J_{i}} a_{ij} x_{ij} \label{obj_deter} \\
& s. t.  & \sum_{j \in J_{i}} x_{ij} = 1,   i \in I \\
& & \sum_{i \in I, j \in J_{i}} c_{ijkt} x_{ij} \leq b_{kt}, k \in K, t \in T \\
& & x_{ij} \in \{0,1\}, j \in J_{i}, i \in I.
\end{eqnarray}
\end{subequations}

The decision variables, $x_{ij}$, indicate whether we choose to do project i by means j. Restated,
if $x_{ij}=1$, then we recommend doing project $i$ via option $j$; taken together, these decision
variables produce both a portfolio of selected projects and a schedule for performing those projects
over time.  The set of available options, $j \in J_i$, can explicitly include the “do-nothing” option,
and the first constraint ensures that we choose exactly one option from the available set for each
project, including the possibility of selecting the do-nothing option. Even if we select the
do-nothing option for a project, it induces an NPV, $a_{ij}$, which may be negative, representing
growing O\&M costs, losses in plant efficiency, etc. The second structural constraint ensures that
the budget of each resource $k$ is respected in each year $t$. The objective function
includes the NPV for each project-option pair, $a_{ij}$, and the correct NPV is selected by
the $0-1$ decision variable, $x_{ij}$.

% The third structural constraint
% captures piggybacking situations in which option $j^{'}$ for project $i^{'}$ (which may have cheaper
% costs) may be selected only if project-option pair $(i,j)$ is also selected.

We note that certain projects must be done (e.g., for safety and or regulatory
reasons). This can be handled within the mathematical formulation just given, without introducing
additional constructs. The set $J_i$ typically includes a do-nothing option for each project,
but when project $i$ must be done, we simply do not include the do-nothing option. Mathematically,
one alternative is to disclude an explicit do-nothing option, replace the first structural
constraint with an inequality, and add an additional set of must-do projects with an equality
constraint. Both options are mathematically equivalent and simply represent a choice to be made by
the analyst. In LOGOS, we use \xmlNode{mandatory} and \xmlNode{nonSelection} to
handle these conditions. \xmlNode{mandatory} is used to specify the must-do projects,
while \xmlNode{nonSelection} is used to activate the do-nothing option.

\nb If a project is listed under \xmlNode{mandatory}, the do-nothing option is not allowed
for this project. In addition, handling the do-nothing option implicitly leads to the NPV being
calculated relative to that of the do-nothing option.

The objective of capital budgeting is to find the right combination of binary decisions for
every investment so that overall profit is maximized. The output is a
collection of projects to be carried out, and we refer to this selected collection of projects
as a ``project portfolio''. However, as is frequently the case for capital budgeting with
NPP applications, in practice, several optional constraints, such as resources/liabilities,
dependencies/synergies, options, time windows for every investment, etc., have to be
fulfilled. This leads to a various variations of the knapsack problem.
In the following subsection, we will present several different variants of the above
capital budgeting problem (i.e., variants of the knapsack problem).

\subsection{Single Knapsack Problem Optimization}
\label{subsec:skp}

\subsubsection{Simple Knapsack Problem}
The simple knapsack problem (KP) for capital budgeting can be defined as follows:
we are given an instance of the capital budgeting problem with investment set $I$,
consisting of $I$ investments $i$ with profit $a_i$ (e.g. NPV), cost $c_i$,
and available budget $b$. The objective is to select a subset of $I$ such
that the total profit of the selected investments is maximized and the total cost does
not exceed $b$. Alternatively, KP can be formulated as a solution of the following
linear integer programming formulation:

\begin{subequations}\label{simpleKP}
\begin{eqnarray}
&\dst \max_{x} &  \dst \sum_{i \in I} a_{i} x_{i} \\
& s. t. & \sum_{i \in I} c_{i} x_{i} \leq b\\
& & x_{i} \in \{0,1\}, i \in I.
\end{eqnarray}
\end{subequations}

Example LOGOS input XML:
\begin{lstlisting}[style=XML]
<Logos>
  <Sets>
    <investments>
      1,2,3,4,5,6,7,8,9,10
    </investments>
  </Sets>

  <Parameters>
    <net_present_values index="investments">
      18,20,17,19,25,21,27,23,25,24
    </net_present_values>
    <costs index="investments">
      1,3,7,4,8,9,6,10,2,5
    </costs>
    <available_capitals>
      15
    </available_capitals>
  </Parameters>

  <Settings>
    <solver>cbc</solver>
    <sense>maximize</sense>
  </Settings>
</Logos>
\end{lstlisting}

When running this case, LOGOS would generate a CSV (comma separated values) file
containing solutions for the optimization problem (i.e. values of decision variables
and maximum profit [MaxNPV is used to describe the maximum profit]). The header of
this CSV file contains the indices listed under \xmlNode{investments}
used as indices for decision variables and the objective variable \textbf{MaxNPV}.
The data provides the values for both decision variables and the objective variable.

Example LOGOS output CSV:
\begin{lstlisting}[language=python]
1,2,3,4,5,6,7,8,9,10,MaxNPV
1.0,1.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0,1.0,106.0
\end{lstlisting}

In this case, projects \textbf{1, 2, 4, 9, and 10} are selected with a maximum
profit of 106.0.

\subsubsection{Bounded Knapsack Problem}
In the capital budgeting problem described above, it may be the case that not all
investments/projects are different from each other. For example, in practice,
there may be given a number ($n_i$) of identical pumps/valves to be replaced. In this
case, the number of decision variables is equal to the number of different
investments, rather than the total number of investments. The constraint for the
decision variables becomes:
\begin{equation}
0\leq x_i \leq n_i, i\in N
\end{equation}
The resulting problem is called the bounded knapsack problem (BKP) and is formally defined as:

\begin{subequations}\label{boundedKP}
\begin{eqnarray}
&\dst \max_{x} &  \dst \sum_{i \in I} a_{i} x_{i} \\
& s. t. & \sum_{i \in I} c_{i} x_{i} \leq b\\
& & x_{i} \in \{0,n_i\}, i \in I.
\end{eqnarray}
\end{subequations}

Example LOGOS input XML:
\begin{lstlisting}[style=XML]
<Logos>
  <Sets>
    <investments>
      1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
      14, 15, 16, 17, 18, 19, 20, 21, 22
    </investments>
  </Sets>

  <Parameters>
    <net_present_values index="investments">
      150,35,200,60,60,45,60,40,30,10,70,
      30,15,10,40,70,75,80,20,12,50,10
    </net_present_values>
    <costs index="investments">
      9,13,153,50,15,68,27,39,23,52,11,32,
      24,48,73,42,43,22,7,18,4,30
    </costs>
    <available_capitals>
      400
    </available_capitals>
  </Parameters>

  <Settings>
    <lowerBounds>
      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
    </lowerBounds>
    <upperBounds>
      1,1,2,2,2,3,3,3,1,3,1,
      1,2,2,1,1,1,1,1,2,1,2
    </upperBounds>
    <solver>glpk</solver>
    <sense>maximize</sense>
  </Settings>
</Logos>
\end{lstlisting}

Example LOGOS output CSV:
\begin{lstlisting}[language=python]
1,2,...,21,22,MaxNPV
1.0,1.0,...,1.0,0.0,1010.0
\end{lstlisting}

\subsubsection{Multi-Dimensional Knapsack Problem: DKP}
Moving in a different direction, we now take into account not only the cost constraint, but also
the limited commitment of critical resources, including: (i) capital cost, (ii) O\&M
costs, (iii) time and labor-hours during a planned outage, and (iv) personnel,
installation and maintenance equipment, workspaces, etc.. Denoting the cost of every
investment by $c_{ik}$ for each resource $k$ and introducing the corresponding limited resource
$b_k$, we can formulate the capital budgeting problem as a multi-dimensional
or D-dimensional knapsack problem formally defined by:

\begin{subequations}\label{boundedKP}
\begin{eqnarray}
&\dst \max_{x} &  \dst \sum_{i \in I} a_{i} x_{i} \\
& s. t. & \sum_{i \in I} c_{ik} x_{i} \leq b_k, k\in K\\
& & x_{i} \in \{0,1\}, i \in I.
\end{eqnarray}
\end{subequations}

Where the limited resources set is denoted by $K$, consisting of $k$ “colors” of money
within capital costs, O\&M costs, personnel availability, etc.
Another example is provided if the plant has multi-year investments. Consider a DKP problem in
which the costs of each investment and the available capitals vary according to time
period $t$. By defining $c_{it}$ as the cost of investment $i$ at time period $i$,
and $b_t$ as the available capital at time period $t$, we get:

\begin{subequations}\label{DKP}
\begin{eqnarray}
&\dst \max_{x} &  \dst \sum_{i \in I} a_{i} x_{i} \\
& s. t. & \sum_{i \in I} c_{it} x_{i} \leq b_t, t\in T\\
& & x_{i} \in \{0,1\}, i \in I.
\end{eqnarray}
\end{subequations}

Example LOGOS input XML:
\begin{lstlisting}[style=XML]
<Logos>
  <Sets>
    <investments>
      1,2,3,4,5,6,7,8,9
    </investments>
    <time_periods>
      1,2,3,4,5
    </time_periods>
  </Sets>

  <Parameters>
    <net_present_values index="investments">
      2.315,0.824,22.459,60.589,0.667,5.173,4.003,0.582,0.122
    </net_present_values>
    <costs index="investments, time_periods">
      0.219,0.257,0.085,0.0,0.0,
      0.0,0.0,0.122,0.103,0.013,
      5.044,1.839,0.0,0.0,0.0,
      6.74,6.134,10.442,0.0,0.0,
      0.425,0.0,0.0,0.0,0.0,
      2.125,2.122,0.0,0.0,0.0,
      2.387,0.19,0.012,2.383,0.192,
      0.0,0.95,0.0,0.0,0.0,
      0.03,0.03,0.688,0.0,0.0
    </costs>
    <available_capitals index="time_periods">
      0.665,4.712,9.642,3.458,1.683
    </available_capitals>
  </Parameters>

  <Settings>
    <solver>glpk</solver>
    <sense>maximize</sense>
  </Settings>
</Logos>
\end{lstlisting}

Example LOGOS output CSV:
\begin{lstlisting}[language=python]
1,2,3,4,5,6,7,8,9,MaxNPV
1.0,1.0,0.0,0.0,1.0,0.0,0.0,1.0,0.0,4.388
\end{lstlisting}

\subsection{Multiple Knapsack Problem Optimization}
\label{subsec:mkp}
Another interesting variant of the capital budgeting problem arises if we consider
maintenance for multiple units in a NPP in parallel, i.e. it has to be decided whether
to accept a particular replacement and, in the positive case, in which unit to conduct
the corresponding replacement. This can be formulated by
introducing a binary decision variable for every maintenance-unit combination.
If there are $I$ investments (investment set $I$) on the list of maintenance requests and $m$
units (unit set $M$) available, we use binary variables:

\begin{equation}
x_{im} \in \{0,1\}, i\in I, m\in M
\end{equation}
The resulting problem is called the multiple knapsack problem (MKP),
and the mathematical formulation is given by:

\begin{subequations}\label{MKP}
\begin{eqnarray}
&\dst \max_{x} &  \dst \sum_{m \in M} \sum_{i \in I} a_{i} x_{im} \\
& s. t. & \sum_{i \in I} c_{i} x_{im} \leq b_m, m \in M\\
& & \sum_{m \in M} x_{im} \leq 1 \\
& & x_{im} \in \{0,1\}, i \in I, m \in M.
\end{eqnarray}
\end{subequations}

Example LOGOS input XML:
\begin{lstlisting}[style=XML]
<Logos>
  <Sets>
    <investments>
      1,2,3,4,5,6,7,8,9,10
    </investments>
    <capitals>
      unit_1, unit_2
    </capitals>
  </Sets>

  <Parameters>
    <net_present_values index="investments">
      78, 35, 89, 36, 94, 75, 74, 79, 80, 16
    </net_present_values>
    <costs index="investments">
      18, 9, 23, 20, 59, 61, 70, 75, 76, 30
    </costs>
    <available_capitals index="capitals">
      103, 156
    </available_capitals>
  </Parameters>

  <Settings>
    <solver>cbc</solver>
    <sense>maximize</sense>
    <problem_type>MultipleKnapsack</problem_type>
  </Settings>
</Logos>
\end{lstlisting}

Example LOGOS output CSV:
\begin{lstlisting}[language=python]
1,2,3,4,5,6,7,8,9,10,capitals,MaxNPV
0.0,0.0,1.0,1.0,1.0,0.0,0.0,0.0,0.0,0.0,unit_1,452.0
1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,1.0,0.0,unit_2,452.0
\end{lstlisting}

\subsection{Multiple-Choice Multi-Dimensional Knapsack Problem Optimization}
\label{subsec:mckp}
Another quite different variant of the capital budgeting problem appears if there may
be multiple ways to carry out each investment/project. Each investment $i$, however, exists
in a number of options in which the j-th option has cost $c_{ij}$ and profit
value $a_{ij}$. This problem can be expressed as the multiple-choice knapsack problem
(MCKP). Assume $J_i$ is the set of different options for investment $i$. Using the
decision variables $x_{ij}$ to denote whether option $j$ was chosen from the set $J_i$,
the mathematical formulation of MCKP is given by:

\begin{subequations}\label{MKP}
\begin{eqnarray}
&\dst \max_{x} &  \dst \sum_{i \in I, j \in J_{i}} a_{ij} x_{ij} \\
& s. t.  & \sum_{j \in J_{i}} x_{ij} = 1,   i \in I \\
& & \sum_{i \in I, j \in J_{i}} c_{ij} x_{ij} \leq b \\
& & x_{ij} \in \{0,1\}, j \in J_{i}, i \in I.
\end{eqnarray}
\end{subequations}

Considering the limited resources and multi-year investments, the MCKP may be extended
to a D-dimensional MCKP problem (D-MCKP). For example, a project may be performed over
a three-year period--say, years `$t$, $t+1$, $t+2$'--or the start of the project could
instead be two years hence, changing the equation to `$t+2$, $t+3$, $t+4$'.
Alternatively, at increased cost and benefit, it may be possible to complete
the project in two years: `$t$, $t+1$', or `$t+2$, $t+3$'. When selecting a project to
uprate plant capacity, we may have the options of increasing it by 3\% or 6\%.
In these cases, the problem can be expressed as the D-MCKP.
This problem is formally defined as follows:

\begin{subequations}\label{model-deter}
\begin{eqnarray}
&\dst \max_{x} &  \dst \sum_{i \in I, j \in J_{i}} a_{ij} x_{ij} \label{obj_deter} \\
& s. t.  & \sum_{j \in J_{i}} x_{ij} = 1,   i \in I \\
& & \sum_{i \in I, j \in J_{i}} c_{ijt} x_{ij} \leq b_{t}, t \in T \\
& & x_{ij} \in \{0,1\}, j \in J_{i}, i \in I.
\end{eqnarray}
\end{subequations}

or in the same as the problem described at the beginning of this chapter
(see Section~\ref{sec:DeterministicCapitalBudgeting}):

\begin{subequations}\label{model-deter}
\begin{eqnarray}
&\dst \max_{x} &  \dst \sum_{i \in I, j \in J_{i}} a_{ij} x_{ij} \label{obj_deter} \\
& s. t.  & \sum_{j \in J_{i}} x_{ij} = 1,   i \in I \\
& & \sum_{i \in I, j \in J_{i}} c_{ijkt} x_{ij} \leq b_{kt}, k \in K, t \in T \\
& & x_{ij} \in \{0,1\}, j \in J_{i}, i \in I.
\end{eqnarray}
\end{subequations}

Example LOGOS input XML:
\begin{lstlisting}[style=XML]
<Logos>
  <Sets>
    <investments>
      1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17
    </investments>
    <options index='investments'>
      1;
      1;
      1;
      1,2,3;
      1,2,3,4;
      1,2,3,4,5,6,7;
      1;
      1;
      1;
      1;
      1;
      1;
      1;
      1;
      1;
      1;
      1
    </options>
  </Sets>

  <Parameters>
    <net_present_values index='options'>
      2.046 2.679 2.489 2.61 2.313 1.02 3.013 2.55 3.351 3.423 3.781 2.525
      2.169 2.267 2.747 4.309 6.452 2.849 7.945 2.538 1.761 3.002 3.449
      2.865 3.999 2.283 0.9 8.608
    </net_present_values>
    <costs index='options'>
      36538462
      83849038
      4615385
      2788461538
      2692307692
      5480769231
      1634615385
      2981730768
      7211538462
      9038461538
      649038462
      650000000
      216346154
      212500000
      3076923077
      3942307692
      1144230769
      675721154
      1442307692
      99711538
      4807692
      123076923
      138461538
      86538462
      108653846
      75092404
      6413462
      147932692
    </costs>
    <available_capitals>
      15E9
    </available_capitals>
  </Parameters>

  <Settings>
    <solver>cbc</solver>
    <sense>maximize</sense>
    <problem_type>mckp</problem_type>
  </Settings>
</Logos>
\end{lstlisting}

Example LOGOS output CSV:
\begin{lstlisting}[language=python]
1__1,2__1,3__1,4__1,4__2,4__3,...,17__1,MaxNPV
1.0,1.0,1.0,1.0,0.0,0.0,...,1.0,59.82600000000001
\end{lstlisting}

In the output file, the names \textbf{``investmentsIndex"\_\_``optionsIndex"} are used to
specify the decision variable. For example, \textbf{1\_\_1} indicates that investment \textbf{1}
with option \textbf{1} is selected.


\subsection{Orthogonal matrices}

Recall that the process to find the inverse of a matrix was often cumbersome.
In contrast, it was very easy to take the transpose of a matrix. Luckily for some special
matrices, the transpose equals the inverse. When an $n \times n$-matrix has all real
entries and its transpose equals its inverse, the matrix is called an \textbf{orthogonal matrix}.

The precise definition is as follows.

\begin{definition}{Orthogonal matrices}{orthogonal-matrices}
A real $n\times n$-matrix $U$ is called an
\index{matrix!orthogonal} \textbf{orthogonal} matrix if $UU^{T}=U^{T}U=I$.
\end{definition}

Note since $U$ is assumed to be a square matrix, it suffices to verify
only one of these equalities $UU^{T}=I$ or $U^{T}U=I$ holds to
guarantee that $U^T$ is the inverse of $U$.

Consider the following example.

\begin{example}{Orthogonal matrix}{}
Show the matrix
\begin{equation*}
U=\begin{mymatrix}{rr}
\vspace{0.05in}\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\
\vspace{0.05in}\frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}}
\end{mymatrix}
\end{equation*}
is orthogonal.
\end{example}

\begin{solution}
All we need to do is verify (one of the equations from) the requirements of Definition~\ref{def:orthogonal-matrices}.

\begin{equation*}
UU^{T}=\begin{mymatrix}{rr}
\vspace{0.05in}\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\
\vspace{0.05in}\frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}}
\end{mymatrix} \begin{mymatrix}{rr}
\vspace{0.05in}\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\
\vspace{0.05in}\frac{1}{\sqrt{2}} & -\frac{1}{\sqrt{2}}
\end{mymatrix} =\allowbreak \begin{mymatrix}{cc}
1 & 0 \\
0 & 1
\end{mymatrix}
\end{equation*}

Since $UU^{T} = I$, this matrix is orthogonal.
\end{solution}

Here is another example.

\begin{example}{Orthogonal matrix}{}
Let $U=\begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 0 & -1 \\
0 & -1 & 0
\end{mymatrix}$. Is $U$ orthogonal?
\end{example}

\begin{solution}
Again the answer is yes and this can be verified simply by showing that $U^{T}U=I$:

\begin{eqnarray*}
U^{T}U&=&\begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 0 & -1 \\
0 & -1 & 0
\end{mymatrix} ^{T}\begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 0 & -1 \\
0 & -1 & 0
\end{mymatrix} \\
&=&\begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 0 & -1 \\
0 & -1 & 0
\end{mymatrix} \begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 0 & -1 \\
0 & -1 & 0
\end{mymatrix} \\
&=&\begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{mymatrix}
\end{eqnarray*}
\end{solution}

When we say that $U$ is orthogonal, we are saying that $UU^T=I$, meaning that
\begin{equation*}
\sum_{j}u_{ij}u_{jk}^{T}=\sum_{j}u_{ij}u_{kj}=\delta _{ik}
\end{equation*}
where $\delta _{ij}$ is the \textbf{Kronecker symbol}
defined
\index{Kronecker symbol} by
\begin{equation*}
\delta _{ij}=\left\{
\begin{array}{c}
1
\text{ if }i=j \\
0\text{ if }i\neq j
\end{array}
\right.
\end{equation*}

In words, the product of the $i\th$ row of $U$ with the $k\th$ row
gives $1$ if $i=k$ and $0$ if $i\neq k$. The same is true of the columns because
$U^{T}U=I$ also. Therefore,
\begin{equation*}
\sum_{j}u_{ij}^{T}u_{jk}=\sum_{j}u_{ji}u_{jk}=\delta _{ik}
\end{equation*}
which says that the product of one column with another column gives $1$ if the two
columns are the same and $0$ if the two columns are different.

More succinctly, this states that if $\vect{u}_{1},\ldots,\vect{u}_{n}$
are the columns of $U$, an orthogonal matrix, then
\[
\vect{u}_{i}\dotprod \vect{u}_{j}=\delta _{ij} = \left\{
\begin{array}{c}
1\text{ if }i=j \\
0\text{ if }i\neq j
\end{array}
\right.
\]

We will say that the columns form an orthonormal set of vectors, and similarly for the rows. Thus a matrix is \textbf{orthogonal} if its rows (or columns) form an
\textbf{orthonormal} set of vectors. Notice that the convention is to call such a matrix orthogonal rather than orthonormal (although this may make more sense!).

\begin{proposition}{Orthonormal basis}{orthonormal-basis}
The rows of an $n \times n$ orthogonal matrix form an orthonormal
basis of $\R^n$. Further, any orthonormal basis of
$\R^n$ can be used to construct an $n \times n$ orthogonal
matrix.
\end{proposition}

\begin{proof}
Recall from Theorem~\ref{thm:orthonormal-basis} that an orthonormal set is
linearly independent and forms a basis for its span. Since the rows of
an $n \times n$ orthogonal matrix form an orthonormal set, they must
be linearly independent. Now we have $n$ linearly independent vectors,
and it follows that their span equals $\R^n$. Therefore these
vectors form an orthonormal basis for $\R^n$.

Suppose now that we have an orthonormal basis for $\R^n$. Since the
basis will contain $n$ vectors, these can be used to construct an $n
\times n$-matrix, with each vector becoming a row. Therefore the
matrix is composed of orthonormal rows, which by our above discussion,
means that the matrix is orthogonal. Note we could also have construct
a matrix with each vector becoming a column instead, and this would
again be an orthogonal matrix. In fact this is simply the transpose of
the previous matrix.
\end{proof}

Consider the following proposition.

\begin{proposition}{Determinant of orthogonal matrices}{orthogonal-determinant}
Suppose $U$ is an orthogonal matrix. Then $\det (U) = \pm 1$.
\end{proposition}

\begin{proof}
This result follows from the properties of determinants. Recall that
for any matrix $A$, $\det(A)^T = \det(A)$. Now if $U$ is orthogonal, then:
\begin{equation*}
(\det (U)) ^{2}=\det (U^{T}) \det (U)
=\det (U^{T}U) =\det (I) =1
\end{equation*}

Therefore $(\det (U))^2 = 1$ and it follows that $\det (U) = \pm 1$.
\end{proof}

Orthogonal matrices are divided into two classes, proper and improper.
\index{matrix!proper}\index{matrix!improper}
The proper orthogonal matrices are those whose determinant equals 1
and the improper ones are those whose determinant equals $-1$. The
reason for the distinction is that the improper orthogonal matrices
are sometimes considered to have no physical significance. These
matrices cause a change in orientation which would correspond to
material passing through itself in a non-physical manner. Thus in
considering which coordinate systems must be considered in certain
applications, you only need to consider those which are related by a
proper orthogonal transformation. Geometrically, the linear
transformations determined by the proper orthogonal matrices
correspond to the composition of rotations.

We conclude this section with two useful properties of orthogonal matrices.

\begin{example}{Product and inverse of orthogonal matrices}{product-inverse-orthogonal}
Suppose $A$ and $B$ are orthogonal matrices. Then $AB$ and $A^{-1}$ both exist and are orthogonal.
\end{example}

\begin{solution}
First we examine the product $AB$.
\[ (AB)(B^TA^T)=A(BB^T)A^T =AA^T=I \]
Since $AB$ is square, $B^TA^T=(AB)^T$ is the inverse of
$AB$, so $AB$ is invertible, and $(AB)^{-1}=(AB)^T$
Therefore, $AB$ is orthogonal.

Next we show that $A^{-1}=A^T$ is also orthogonal.
\[ (A^{-1})^{-1} = A = (A^T)^{T}
=(A^{-1})^{T} \]
Therefore $A^{-1}$ is also orthogonal.
\end{solution}


\documentclass[a4paper,10pt]{article}

%\usepackage[landscape]{geometry}

\usepackage{mathtools}
\usepackage{booktabs}
\usepackage{tabularx}

\title{\vspace*{-2cm}Big \(\bigO\)-notation for A\&W algorithms}
\date{}

\pagenumbering{gobble}

\newcommand{\bigO}{\mathcal{O}}

\begin{document}
    \maketitle
    \renewcommand{\arraystretch}{1.25}
    \subsection*{Definitions}
    \begin{center}
        \begin{tabularx}{\textwidth}{l >{\raggedleft\arraybackslash}X}
            \toprule
            Number of vertices in a graph & \(n\) \\
            Number of edges in a graph & \(m\) \\
            Highest capacity in network & \(U\) \\
            Number of vertices at the edge of a convex hull & \(h\) \\
            Variable chosen for desired accuracy of Monte-Carlo algorithms & \(\lambda\) \\
            \bottomrule
        \end{tabularx}
    \end{center}

    \subsection*{Overview of algorithms}

    Some of the algorithms are randomized. They thus have their average runtime noted.
    \begin{center}
    \begin{tabular}{lr}
        \toprule
        Maximal flow (Ford-Fulkerson) & \(\bigO(mnU)\)\\
        Maximal flow (Capacity Scaling) & \(\bigO(mn(1 + \log U))\) \\
        Maximal flow (Dynamic Trees) & \(\bigO(mn\log n)\) \\
        Smallest enclosing disc (naive) & \(\bigO(n^4)\) \\
        Eulerian path in connected graph & \(\bigO(m)\) \\
        Count Hamiltonian cycles in graph & \(\bigO(n^{2.81}\log n \cdot 2^n)\) \\
        2-approximation of Metric TSP & \(\bigO(n^2)\) \\
        1.5-approximation of Metric TSP & \(\bigO(n^3)\) \\
        Maximal matching & \(\bigO(m)\) \\
        Maximum matching in bipartite graph (Hopcroft-Karp) & \(\bigO(\sqrt{n}(n+m))\) \\
        Perfect matching in \(2^k\)-regular bipartite graph & \(\bigO(m)\) \\
        Greedy coloring & \(\bigO(m)\) \\
        \(\bigO(\sqrt{n})\)-coloring for 3-colorable graphs & \(\bigO(m)\) \\
        Convex hull (Jarvis-Wrap) & \(\bigO(nh)\) \\
        Convex hull with \textit{sorted} points (LocalRepair) & \(\bigO(n)\) \\
        QuickSelect & \(\bigO(n)\) \\
        Miller-Rabin prime test & \(\bigO(\ln n)\) \\
        Colorful-Path with length \(\log n\) & \(\bigO(mn \log n)\) \\
        Minimal cut (naive) & \(\bigO(\lambda n^4)\) \\
        Minimal cut (single bootstrapping) & \(\bigO(\lambda n^3)\) \\
        Minimal cut (repeated bootstrapping) & \(\bigO(n^2\text{poly}(\log n))\) \\
        Smallest enclosing disc (clever randomized) & \(\bigO(n \log n)\) \\
        \bottomrule
    \end{tabular}
    \end{center}
\end{document}


\section{Linear and Higher Order Approximations}\label{sec:Approx}
When we define the derivative $f^{\prime
}\left( x\right) $ as the rate of change of $f\left( x\right) $ with respect
to $x,$ we notice that in relation to the graph of $f,$ the derivative is
the slope of the tangent line, which (loosely speaking) is the line that just
grazes the graph. But what precisely do we mean by this? In short,
\textbf{the tangent line approximates the graph near the point of contact}. The
definition of the derivative $f^{\prime }\left( a\right) $ guarantees this
when it exists: By taking $x$ sufficiently close to $a$ but not equal to $a,$%
\begin{equation*}
\frac{f\left( x\right) -f\left( a\right) }{x-a}\approx f^{\prime }\left(
a\right) ,
\end{equation*}%
and consequently,%
\begin{equation*}
f\left( x\right) \approx f^{\prime }\left( a\right) \left( x-a\right)
+f\left( a\right) .
\end{equation*}%
The left hand side gives us the $y$-value of the function $y=f\left(
x\right) $ and the right hand side gives us the $y$-value $y=f^{\prime
}\left( a\right) \left( x-a\right) +f\left( a\right) $ for the tangent line
to the graph of $f$ at the point $\left( a,f\left( a\right) \right) .$

In this section we will explore how to apply this idea to approximate some
values of $f$, some changes in the values of $f$, and also the roots of $f.$

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Subsections to include
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\input{5-applications-of-derivatives/5-4-1-linear-approximations}
\input{5-applications-of-derivatives/5-4-2-differentials}
\input{5-applications-of-derivatives/5-4-3-taylor-polynomials}
\input{5-applications-of-derivatives/5-4-4-newtons-method}

% Exercises for each subsection are in the separate files


\subsection{The macroelement technique}

\begin{intro}
  The arguments of the previous section can be extended to the case
  where instead of individual mesh cell $\cell$ we condider small
  groups of such cells, called
  \putindex{macroelement}s~\cite{Stenberg84}. This will give us some
  more freedom for testing the inf-sup condition locally, but possibly
  not on each cell separately. In order to get uniform estimates, we
  introduce a generalization of the concept of shape regularity.
\end{intro}

\begin{Definition}{macro-equivalence}
  A \define{macroelement} $M\subset \T_h$ is a union of cell
  $\cell_i\in\T_h$. Given the mappings $\Phi_\cell\colon \widehat\cell \to
  \cell$, there is a reference macroelement $\widehat M$ and a mapping
  $\Phi_M\colon \widehat M\to M$ such that $\Phi_M(\widehat M) = M$. We say
  that $M$ is equivalent to $\widehat M$.
\end{Definition}

\begin{Definition}{macro-spaces}
  For a \putindex{macroelement} $M$, we introduce the spaces
  \begin{align}
    \label{eq:macroelement:27}
    V_M &= \bigl\{ u\in H^1_0(M;\R^d) \big\vert
            \;\exists v_h\in V_h\colon u=v_{h|M}\bigr\},
    \\
    \label{eq:macroelement:28}
    Q_M &= \bigl\{ p\in L^2_0(M) \big\vert
            \;\exists q_h\in Q_h\colon p=q_{h|M}\bigr\},
  \end{align}
  the kernel of the discrete, local gradient operator
  \begin{gather}
    \label{eq:macroelement:29}
    \ker{B^\transpose_M} = \bigl\{q\in Q_M \big\vert
    \;\forall v\in V_M\colon \form(\div v,q) = 0\bigr\}.
  \end{gather}
  For a mesh $\mesh_h$ which is subdivided into a nonoverlapping set
  of macros $M_j$, we define the space of pressure functions
  $Q_M^0\subset Q$ which are constant on each macro with the
  $L^2$-projection $\Pi_M\colon Q_h\to Q_M^0$.
\end{Definition}

\begin{Lemma}{macro-local}
  Let $\{M\}$ with $M\subset \mesh_h$ be a set of macroelements
  equivalent to the same reference macroelement $\widehat M$. Let the
  family $\{\mesh_h\}$ be shape regular and assume
  $\ker{B^\transpose_{M}} = \{0\}$. Then, there is a constant
  $\beta_M>0$ independent of $h$ such that for all $M$ there holds
  \begin{gather}
    \label{eq:macroelement:31}
    \inf_{p\in Q_M} \sup_{v\in V_M}
    \frac{\form(\div v,q)_M}{\norm{v}_{1,M}\norm{q}_{0,M}}
    \ge \beta_{\widehat M}.
  \end{gather}
\end{Lemma}

\begin{proof}
  For each $M$, the existence of such a constant $\beta_{M}$ is
  equivalent to the fact that the smallest nonzero singular value is
  strictly greater than zero in finite dimensional spaces. Next, we
  observe that $\nabla \Phi_{M}$ is bounded by the
  $\nabla\Phi_{\cell_i}$, which in turn is uniformly bounded by the
  shape regularity. Therefore, by a cell-wise scaling argument and the
  fact that $\widehat M$ has only finitely many cells we have
  \begin{gather}
    \beta_{\widehat M} = \min_M \beta_M > 0.
  \end{gather}
\end{proof}

\begin{remark}
  Depending on the technique of proof being used, we also may decide
  to impose~\eqref{eq:macroelement:31} directly for each macroelement.
\end{remark}

\begin{Lemma}{macro-projection}
  Let $\mesh_h$ be subdivided into a nonoverlapping set of
  macroelements $M$ such that there is a finite set of reference macro
  elements $\widehat M$, and let $\ker{B^\transpose_M} = \{0\}$. Then, there is
  a constant $c_1>0$ such that for all $q_h\in Q_h$ exists $v_h\in V_h$
  satisfying
  \begin{gather}
    \label{eq:macroelement:32}
    \begin{split}
      \form(\div v_h, q_h) = \form(\div v_h,{(\identity-\Pi_M)} q_h)
      &\ge c_1 \norm{(\identity-\Pi_M) q_h}_0^2\\
      \norm{v_h}_2 & \le \norm{(\identity-\Pi_M) q_h}_0.
    \end{split}
  \end{gather}
\end{Lemma}

\begin{proof}
  For any $q_h\in Q_h$, we have $(\identity-\Pi_M) q_h\in Q_M$ for each
  macroelement $M$. Applying \slideref{Lemma}{macro-local}, for each
  $M$ there exists $v_M\in V_M$ such that
  \begin{gather}
    \form(\div v_M, {(\identity-\Pi_M)} q_h)_M
    \ge c_1 \norm{v_M}_{1,M}\norm{(\identity-\Pi_M) q_h}_{0,M},
  \end{gather}
  with $c=\min \beta_{\widehat M}$. Let now $v_M$ such that
  $\norm{v_M}_{1,M} = \norm{(\identity-\Pi_M) q_h}_{0,M}$ and $v$ such that
  $v_{|M} = v_M$. Since $v_M$ is zero on $\d M$, we have $v\in
  V_h$. By construction,~\eqref{eq:macroelement:32} holds.
\end{proof}

\begin{Lemma}{macro-constant}
  Let $\{M_j\}$ be a covering of $\mesh_h$ with macroelements. For any
  two macros $M_i$ and $M_j$ let $T_{ij}$ be their common
  boundary. Then, if every $T_{ij}$ of codimension one contains at
  least one degree of freedom controlling $u\cdot\n$, there is a
  constant $c_2>0$ such that for every $q_h\in Q_h$ exists $w_h\in
  V_h$ such that
  \begin{gather}
    \label{eq:macroelement:33}
    \begin{split}
      \form(\div w_h, \Pi_M q_h) &\ge c_2 \norm{\Pi_M q_h}^2_0\\
      \norm{w_h}_1 &\le \norm{\Pi_M q_h}_0.
    \end{split}
  \end{gather}
\end{Lemma}

\begin{Problem}
  Prove \slideref{Lemma}{macro-constant}
\end{Problem}

\begin{Theorem}{macroelement}
  Let the assumptions of \slideref{Lemma}{macro-projection}
  hold. Then, the pair $V_h\times Q_h$ is inf-sup stable.
\end{Theorem}

\begin{proof}
  Let $q_h\in Q_h$. Then, applying \slideref{Lemma}{macro-projection}
  and \slideref{Lemma}{macro-constant}, we obtain functions $v_h$ and
  $w_h$ such that for $u_h = v_h+ \delta w_h$ for any $\delta>0$ there
  holds
  \begin{align}
    \form(\div u_h, q_h)
    &= \form(\div v_h, q_h) + \form(\div w_h, q_h) \\
    &\ge c_1 \norm{(\identity-\Pi_M) q_h}_0^2
      + \delta c_2 \norm{\Pi_M q_h}_0^2
      - \delta \norm{w_h}_1 \norm{(\identity-\Pi_M) q_h}_0 \\
    & = (c_1 - \delta) \norm{(\identity-\Pi_M) q_h}_0^2
      + \delta c_2 \norm{\Pi_M q_h}_0^2.
  \end{align}
  Choosing $\delta = c_1/(1+c_2)$, we obtain
  \begin{gather}
    \label{eq:macroelement:34}
    \begin{split}
      \form(\div u_h, q_h) &\ge \frac{c_1c_2}{1+c_2} \norm{q}_0^2 \\
      \norm{u}_1 &\le \norm{q}_0.
    \end{split}
  \end{gather}
\end{proof}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:


\section{Brownian Motion}

\begin{frame}{Brownian motion}
Brownian motion can be constructed as a limit of random walks.
\begin{eqnarray*}
 P(Y_i=\Delta x)=\frac{1}{2} \\
 P(Y_i=-\Delta x)=\frac{1}{2}
\end{eqnarray*}
\pause

\begin{center}
What is the distribution of
\begin{equation*}
X_n=Y_1+Y_2+...+Y_n
\end{equation*}
if $Y_1,Y_2,...,Y_n$ are i.i.d.?
\end{center}
\end{frame}

\begin{frame}
\begin{eqnarray*}
X_n&= &Y_1+Y_2+...+Y_n\\ \\
\pause
\ln (M_{X_n})(\lambda)&\approx &\frac{(\Delta x)^2}{\Delta t} \left(\frac{\lambda ^2 T}{2}+\frac{\lambda ^4 \Delta x ^2 T}{4}+... \right)\\
\end{eqnarray*}
\pause
If we assume that $\frac{(\Delta x)^2}{\Delta t}=k$ then,
\begin{equation*}
\lim_{\Delta x \to 0}{M_{X_n}(\lambda)}=\exp\left(\frac{kT\lambda ^2}{2}\right)
\end{equation*}
which means $X_n \sim N(0,kT)$
\end{frame}

\begin{frame}
\begin{block}{Standard Brownian Motion}
A random variable $B(t)$ that depends continuously on $t \in [0,T]$ and satisfies:
\begin{itemize}
\item $B(0)=0$
\item For $0 \leq s<t\leq T$: $B(t)-B(s)\sim N(0, t-s)$
\item For $0 \leq s<t<u<v\leq T$ the increments $B(t)-B(s)$ and $B(v)-B(u)$ are independent.
\end{itemize}
\end{block}
\end{frame}

\begin{frame}{Computational Purposes}
\pause
Discretized Brownian motion: $B(t)$ is specified at discrete t values.
\begin{eqnarray*}
&& W(0)=0\\
&& W(j)=W(j-1)+dW(j)
\end{eqnarray*}
$dW(j)$ is an independent random variable of the form $N(0,\Delta t)$\cite{doi:10.1137/S0036144500378302}.
\end{frame}

\begin{frame}
\begin{center}
\includegraphics[scale=0.5]{r_w_3.png}
\end{center}
\end{frame}

\begin{frame}
\begin{center}
\includegraphics[scale=0.5]{r_w_4.png}
\end{center}
\end{frame}

\begin{frame}
\begin{center}
\includegraphics[scale=0.5]{hist.png}
\end{center}
\end{frame}

\begin{frame}
\begin{center}
\includegraphics[scale=0.5]{qqplot.png}
\end{center}
\end{frame}

\section{Stochastic Integrals}

\begin{frame}{Stochastic Differential Equations}
\pause
We'll take an ordinary differential equation that contains a deterministic part and add an additional stochastic term:
\begin{eqnarray*}
dx=fdt+gdB
\end{eqnarray*}
\end{frame}

\begin{frame}{Stochastic Integrals}
\pause
\begin{center}
What does $\int_{a}^{b} f(t)dB$ mean? \bigskip \pause  $\int_{a}^{b} B(t)dB$?\pause \\

\begin{equation*}
\sum_{i=1}^{n}B(t_{j-1})\left( B(t_{j})-B(t_{j-1})\right) ?
\end{equation*}
\pause
\bigskip

In $[0,1]$ with $\Delta t=1/500$:\\
Left hand side=-0.42765\\
Mid-Point=0.088892
\end{center}
\end{frame}

\begin{frame}
\begin{block}{Definition}
Given f(t) a continuous function with bounded variation. Let $a_i=f(t_{i-1})$, then \textbf{the Wiener integral} is:
\begin{equation*}
I(f):=\lim_{n \to \infty}\sum_{i=1}^{n}a_i(B(t_i)-B(t_{i-1}))
\end{equation*}
\pause
\begin{itemize}
\item $E[I(f)]=0$
\item $Var[I(f)]=\int_{a}^{b}f^2(t)dt$
\end{itemize}
\end{block}
\end{frame}


\section{Areas in Polar Coordinates}\label{sec:Areas in polar coordinates}
We can use the equation of a curve in polar coordinates to compute
some areas bounded by such curves.
The basic approach is the same as with any application of integration:
Find an approximation that approaches the true value. For areas in
rectangular coordinates, we approximated the region using rectangles;
in polar coordinates, we use sectors of circles, as depicted in
Figure~\ref{fig:approximating area with sectors}. Recall that the
area of a sector of a circle is $\ds \alpha r^2/2$, where $\alpha$ is the
angle subtended by the sector. If the curve is given by $r=f(\theta)$,
and the angle subtended by a small sector is $\Delta\theta$,
the area is $\ds (\Delta\theta)(f(\theta))^2/2$.
Thus we approximate the total area as
$$\sum_{i=0}^{n-1} {1\over 2} f(\theta_i)^2\;\Delta\theta.$$
As a limit, this will give rise to:
$$\int_a^b {1\over 2} f(\theta)^2\;d\theta.$$

\begin{example}{Area inside a Cardioid}{cardioidarea}
Find the area inside the cardioid $r=1+\cos\theta$.
\end{example}

\begin{solution}
$$\int_0^{2\pi}{1\over 2} (1+\cos\theta)^2\;d\theta=
\ds{1\over 2}\int_0^{2\pi} 1+2\cos\theta+\cos^2\theta\;d\theta=
\ds{1\over 2}\left(\theta +2\sin\theta+
\ds{\theta\over2}+\ds{\sin2\theta\over4}\middle)\right|_0^{2\pi}={3\pi\over2}.$$
\end{solution}

As in the example above, if a function is defined over the complete region $[0,2\pi]$, then the bounds for the integral are $a=0$, $b=2\pi$. However, sometimes we need to be careful!

\begin{example}{Area inside a Loop}{areainloop}
Find the area inside the loop $r=\sqrt{\cos\theta}$.
\end{example}
\begin{solution}
We look for points of intersection.
\begin{align*}
0&=\sqrt{\cos\theta}	\\
0&=\cos\theta
\end{align*}
This has solutions $\theta=\pi/2$ and $3\pi/2$, but we must select our bounds for integration correctly. Since $\cos\theta$ is negative on $(\pi/2,3\pi/3)$, $\sqrt{\cos\theta}$ is undefined. Note that one full revolution is $2\pi$; that is, the function is $2\pi$ periodic. If we were to integrate on $[3\pi/2,\pi/2]$ we would get the same answer, and can avoid a scenario in which $\sqrt{\cos\theta}$ is undefined. Equivalently, we can integrate on $[-\pi/2,\pi/2]$.
\[\ds \int_a^b\frac{1}{2} f(\theta)^2\,d\theta=\int_{-\pi/2}^{\pi/2}\frac{1}{2}\left(\sqrt{\cos\theta}\right)^2\,d\theta=\frac{1}{2}\int_{-\pi/2}^{\pi/2}\cos\theta\,d\theta=\frac{1}{2}\Big(\sin\theta\Big)\Big|_{-\pi/2}^{\pi/2}=1\]
\end{solution}

\figure[H]
\centerline{\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <20truemm,20truemm>
\setplotarea x from 0 to 2.5, y from 0  to 1.5
\axis left shiftedto x=0 /
\axis bottom shiftedto y=0 /
\setquadratic
\plot 1.941 0.393 1.900 0.506 1.850 0.613 1.791 0.715 1.723 0.811
1.648 0.900 1.565 0.981 1.476 1.054 1.382 1.118 1.284 1.172
1.182 1.217 1.078 1.252 0.973 1.278 0.867 1.293 0.762 1.299
0.659 1.295 0.559 1.283 0.462 1.262 0.369 1.233 0.281 1.196
0.199 1.153 /
\setlinear
\plot 0 0 0.75 1.299 /
\circulararc -15 degrees from 0.75 1.299 center at 0 0
\plot 0 0 1.207 1.207 /
\circulararc -15 degrees from 1.207 1.207 center at 0 0
\plot 0 0 1.616 0.933 /
\circulararc -15 degrees from 1.616 0.933 center at 0 0
\plot 0 0 1.9 0.51 /
\endpicture}}
\caption{{Approximating area by sectors of circles.} \label{fig:approximating area with sectors}}
\endfigure

\begin{example}{Area Between Circles}{areabetweencircles}
Find the area between the circles $r=2$ and
$r=4\sin\theta$, as shown in figure~\ref{fig:polar area between curves}.
\end{example}

\begin{solution}
The two curves intersect where $2=4\sin\theta$, or $\sin\theta=1/2$,
so $\theta=\pi/6$ or $5\pi/6$. The area we want is then
$$
  {1\over2}\int_{\pi/6}^{5\pi/6}
  16\sin^2\theta-4\;d\theta={4\over3}\pi + 2\sqrt{3}.
$$
\end{solution}

%\figure[!ht]
%\centerline{
%\hbox{\hfill\tikzpicture[domain=-2:2,x=6mm,y=6mm]
%\draw[->] (-2.1,0) -- (2.1,0) ;
%\draw[->] (0,-2.1) -- (0,4.1) ;
%\gpad
%\draw[color=black] plot[parametric,id=\the\gpnum,domain=0:2*3.1416,samples=50]
%function{2*cos(t),2*sin(t)};
%\gpad
%\draw[color=black] plot[parametric,id=\the\gpnum,domain=0:3.1416,samples=50]
%function{4*sin(t)*cos(t),4*sin(t)*sin(t)};
%\gpad
%\fill[opacity=0.5,fill=red!20]
%plot[parametric,id=\the\gpnum,domain=0.5236:2.618]
%function{4*sin(t)*cos(t),4*sin(t)*sin(t)} node {\gpad}
%plot[parametric,id=\the\gpnum,domain=0.5236:2.618]
%function{2*cos(3.1416-t),2*sin(3.1416-t)} -- cycle;
%\endtikzpicture\hfill}}
%\caption{{An area between curves.} \label{fig:polar area between curves}}
%\endfigure

\figure[H]
\[
\includegraphics[width=1.75in]{images/7-area-circles}
\]
\caption{{An area between curves.} \label{fig:polar area between curves}}
\endfigure

This first example makes the process appear more straightforward than it
is. Since points have many different representations in polar
coordinates, it is not always so easy to identify points of
intersection.

\begin{example}{Shaded Area}{shadedarea}
Find the shaded area in the first graph of
figure~\ref{fig:harder area between polar curves} as the difference
of the other two shaded areas. The cardioid is $r=1+\sin\theta$ and
the circle is $r=3\sin\theta$.
\end{example}

\begin{solution}
We attempt to find points of intersection:
\begin{eqnarray*}
  1+\sin\theta&=3\sin\theta\cr
  1&=2\sin\theta\cr
  1/2&=\sin\theta.\cr
\end{eqnarray*}
This has solutions $\theta=\pi/6$ and $5\pi/6$; $\pi/6$ corresponds to
the intersection in the first quadrant that we need.  Note that no
solution of this equation corresponds to the intersection point at the
origin, but fortunately that one is obvious. The cardioid goes through
the origin when $\theta=-\pi/2$; the circle goes through the origin at
multiples of $\pi$, starting with $0$.

Now the larger region has area
$$
  {1\over2}\int_{-\pi/2}^{\pi/6} (1+\sin\theta)^2\;d\theta=
  {\pi\over2}-{9\over16}\sqrt{3},
$$
and the smaller has area
$$
  {1\over2}\int_{0}^{\pi/6} (3\sin\theta)^2\;d\theta=
  {3\pi\over8} - {9\over16}\sqrt{3},
$$
so the difference is the area we seek, which is $\pi/8$.
\end{solution}

%\figure[!ht]
%\centerline{
%\hbox to \hsize{\hfill\tikzpicture[domain=-2:2,x=9mm,y=9mm]
%\draw[->] (-1.6,0) -- (1.6,0) ;
%\draw[->] (0,-1) -- (0,3.1) ;
%\gpad
%\draw[color=black] plot[parametric,id=\the\gpnum,domain=0:2*pi,samples=50]
%function{(1+sin(t))*cos(t),(1+sin(t))*sin(t)};
%\gpad
%\draw[color=black] plot[parametric,id=\the\gpnum,domain=0:pi,samples=50]
%function{3*sin(t)*cos(t),3*sin(t)*sin(t)};
%\gpad
%\fill[opacity=0.5,fill=red!20]
%plot[parametric,id=\the\gpnum,domain=-pi/2:pi/6]
%function{(1+sin(t))*cos(t),(1+sin(t))*sin(t)} node {\gpad}
%plot[parametric,id=\the\gpnum,domain=0:pi/6]
%function{3*sin(pi/6-t)*cos(pi/6-t),3*sin(pi/6-t)*sin(pi/6-t)};
%\endtikzpicture
%\quad
%\tikzpicture[domain=-2:2,x=9mm,y=9mm]
%\draw[->] (-1.6,0) -- (1.6,0) ;
%\draw[->] (0,-1) -- (0,3.1) ;
%\gpad
%\draw[color=black] plot[parametric,id=\the\gpnum,domain=0:2*pi,samples=50]
%function{(1+sin(t))*cos(t),(1+sin(t))*sin(t)};
%\gpad
%\draw[color=black] plot[parametric,id=\the\gpnum,domain=0:pi,samples=50]
%function{3*sin(t)*cos(t),3*sin(t)*sin(t)};
%\gpad
%\fill[opacity=0.5,fill=red!20]
%plot[parametric,id=\the\gpnum,domain=-pi/2:pi/6]
%function{(1+sin(t))*cos(t),(1+sin(t))*sin(t)};
%\endtikzpicture
%\quad
%\tikzpicture[domain=-2:2,x=9mm,y=9mm]
%\draw[->] (-1.6,0) -- (1.6,0) ;
%\draw[->] (0,-1) -- (0,3.1) ;
%\gpad
%\draw[color=black] plot[parametric,id=\the\gpnum,domain=0:2*pi,samples=50]
%function{(1+sin(t))*cos(t),(1+sin(t))*sin(t)};
%\gpad
%\draw[color=black] plot[parametric,id=\the\gpnum,domain=0:pi,samples=50]
%function{3*sin(t)*cos(t),3*sin(t)*sin(t)};
%\gpad
%\fill[opacity=0.5,fill=red!20]
%plot[parametric,id=\the\gpnum,domain=0:pi/6]
%function{3*sin(t)*cos(t),3*sin(t)*sin(t)};
%\endtikzpicture
%\hfill}}
%\caption{{An area between curves.} \label{fig:harder area between polar curves}}
%\endfigure
\figure[H]
\[
\includegraphics[width=4.5in]{images/7-area-between}
\]
\caption{{An area between curves.} \label{fig:harder area between polar curves}}
\endfigure

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:Areas in polar coordinates}}

\begin{enumialphparenastyle}

%%%%%%%%%%
\begin{ex}
Find the area enclosed by the curve.
\begin{multicols}{2}
\begin{enumerate}
	\item	$\ds r=\sqrt{\sin\theta}$
	\item	$\ds r=2+\cos\theta$
	\item	$\ds r=\sec\theta, \pi/6\le\theta\le\pi/3$
	\item	$\ds r=\cos\theta, 0\le\theta\le\pi/3$
	\item	$\ds r=2a\cos\theta, a>0$
	\item	$\ds r=4+3\sin\theta$
\end{enumerate}
\end{multicols}
\begin{sol}
\begin{multicols}{2}
\begin{enumerate}
	\item	$1$
	\item	$9\pi/2$
	\item	$\ds \sqrt3/3$
	\item	$\ds \pi/12+\sqrt3/16$
	\item	$\ds \pi a^2/4$
	\item	$41\pi/2$
\end{enumerate}
\end{multicols}
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside the loop formed by
$\ds r=\tan(\theta/2)$.
\begin{sol}
 $2-\pi/2$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside one loop of $\ds r=\cos(3\theta)$.
\begin{sol}
 $\pi/12$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside one loop of $\ds r=\sin^2\theta$.
\begin{sol}
 $3\pi/16$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside the small loop of $\ds r=(1/2)+\cos\theta$.
\begin{sol}
 $\ds \pi/4-3\sqrt3/8$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside $\ds r=(1/2)+\cos\theta$, including the
area inside the small loop.
\begin{sol}
 $\ds \pi/2+3\sqrt3/8$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside one loop of $\ds r^2=\cos(2\theta)$.
\begin{sol}
 $1$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area enclosed by $r=\tan\theta$ and
$\ds r={\csc\theta\over\sqrt2}$.
\begin{sol}
 $3/2-\pi/4$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside $r=2\cos\theta$ and outside
$r=1$.
\begin{sol}
 $\ds \pi/3+\sqrt3/2$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside $r=2\sin\theta$ and above
the line $r=(3/2)\csc\theta$.
\begin{sol}
 $\ds \pi/3-\sqrt3/4$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside $r=\theta$, $0\le\theta\le2\pi$.
\begin{sol}
 $\ds 4\pi^3/3$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside $\ds r=\sqrt{\theta}$, $0\le\theta\le2\pi$.
\begin{sol}
 $\ds \pi^2$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside both $\ds r=\sqrt3\cos\theta$ and
$r=\sin\theta$.
\begin{sol}
 $\ds 5\pi/24-\sqrt3/4$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the area inside both $r=1-\cos\theta$
and $r=\cos\theta$.
\begin{sol}
 $\ds 7\pi/12-\sqrt3$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 The center of a circle of radius 1 is on the
circumference of a circle of radius 2. Find the area
of the region inside both circles.
\begin{sol}
 $\ds 4\pi-\sqrt{15}/2-7\arccos(1/4)$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Find the shaded area in figure~\ref{fig:area inside spiral}.
The curve is $r=\theta$, $0\le\theta\le3\pi$.
\begin{sol}
 $\ds 3\pi^3$
\end{sol}
\end{ex}

%\figure[!ht]
%\centerline{
%\hbox to \hsize{\hfill
%\tikzpicture[domain=-2:2,x=4mm,y=4mm]
%\draw[->] (-10,0) -- (7,0) ;
%\draw[->] (0,-5.2) -- (0,8.5) ;
%\gpad
%\draw[color=black] plot[parametric,id=\the\gpnum,domain=0:3*pi,samples=100]
%function{(t)*cos(t),(t)*sin(t)};
%\gpad
%\fill[opacity=0.5,fill=red!20] (0,0) -- (2*pi,0)
%plot[parametric,id=\the\gpnum,domain=2*pi:3*pi]
%function{(t)*cos(t),(t)*sin(t)} node {\gpad} -- (-pi,0)
%plot[parametric,id=\the\gpnum,domain=0:pi]
%function{(pi-t)*cos(pi-t),(pi-t)*sin(pi-t)};
%\endtikzpicture
%\hfill}}
%\caption{{An area bounded by the spiral of Archimedes.} \label{fig:area inside spiral}}
%\endfigure
\figure[H]
\[
\includegraphics[width=2.75in]{images/7-arch}
\]
\caption{{An area bounded by the spiral of Archimedes.} \label{fig:area inside spiral}}
\endfigure

\end{enumialphparenastyle}


\subsection{The Expectation-Maximisation algorithm}

\subsubsection{Expectation-Maximisation algorithm}

This is used to learn the parameters for a Gaussian Mixture Model

We cannot simply maximise the likelihood function, because this cannot be specified for a latent model.

The log likelihood function normally is:

\(L(\theta ; X)=p(X|\theta )\)

With hidden variables it is:

\(L(\theta ; X, Z)=p(X|\theta )=\int p(X, Z|\theta)dZ\)

\subsubsection{1: Expectation step}

We consider the expected log likelihood. We call this

\(E[\log L(\theta ; X, Z)]\)

\subsubsection{2: Maximisation step}


\section{Motivation of Group}
Groups are the mathematical notion of symmetries.
Indeed, if we go further into this topic, we will find that the symmetries of anything give a group, and any group is actually the symmetry of something.
Then why study symmetries in this way?
Why do we not just desribing the symmetries one by one instead?\\
Consider a tetrahedron.
There are $12$ rotational symmetries of it: $1$ doing nothing, $8$ rotations on an axis joining one of the vertices and the centre of the tetrahedron, and $3$ rotations on the axis joining the midpoints of opposite sides.
The interesting thing is, if you composite two of the rotations that are listed above, you get another rotation.
For example, if we label the vertices as $1,2,3,4$, then one of the rotations on the axis passing though vertex $1$ may send the vertices like
$$1\to 1, 2\to 4, 3\to 2, 4\to 3$$
and the rotation on the axis joining midpoints of opposite segments would permute them by
$$1\to 3, 2\to 4, 3\to 1, 4\to 2$$
We let $R$ be the former rotation and $S$ be the latter, then we could find $S\circ R$, the rotation given by doing $R$ first then $S$ next.
Indeed, this is the permutation
$$1\to 3, 2\to 2, 3\to 4, 4\to 1$$
which is one of the rotations on the axis passing though $2$.
Here is an interesting thing: we can do $R\circ S$ as well, but it is, as one can check, a rotation on the axis passing though $4$!
So $RS\neq SR$, as in the order of the composition of two rotations matters.
This is kind of the point of group theory.\\
Now we can look at the rotational symmetries of another soli: an isocagonal cone.
This time it is quite obvious: the rotational symmetries are precisely the rotations on the central axis of degree $n\pi/6$ where $n=0,1,2,\ldots, 11$.
Now this set of rotations has order $12$ as well.
But are the two sets of rotational symmetries, one on a tetrahedron and another on a isocagonal cone, the same?
No, our intuition said. But why?
\begin{proposition}
    The groups of rotational symmetries of a tetrahedron and a isocagonal cone are different.
\end{proposition}
\begin{proof}
    Note that every rotation in the tetrahedron group is the same as doing nothing after repeating itself 2 or 3 (so, 6) times.
    But the rotation of $\pi/6$ degrees of the isocagonal cone does not have this property.
    Therefore the two groups are different.
\end{proof}
There is another way of doing it,
\begin{proof}[Alternative proof]
    In our previous example, we have found two rotations $S$ $R$ such that they do not commute, i.e. $RS\neq SR$.
    However, every two rotations in the isocagonal cone group commmutes.
    Therefore they are different.
\end{proof}
Note that in the second proof, there is an important property of groups that was used: commutativity.
This notion, expressed in several contexts, is essential to group theory.
But to see that, you have to dive into the world of groups.


% !TEX root = main.tex

%------------------------------------------------
\chapter{Conservative Equations}
\begin{definition}
A conservative equation is an equation of the form $$\frac{d^2x}{dt^2} +V'(x) = 0.$$
\end{definition}
\begin{remark}
This means that we can consider any equation $$\frac{d^2x}{dt^2} + F(x) = 0$$ in which we can find an indefinite integral of $F.$
\end{remark}
\subsection*{Trick for Conservative Equation}
Multiply the equation by $\frac{dx}{dt}$ to obtain $$\frac{dx}{dt}\frac{d^2x}{dt^2}+V'(x)\frac{dx}{dt} = 0$$ or by the chain rule, $$\frac{d}{dt}\bigg[\frac{1}{2}\bigg(\frac{dx}{dt}\bigg)^2+V(x)\bigg] = 0.$$ Hence $$\frac{1}{2}\bigg(\frac{dx}{dt}\bigg)^2+V(x) = E$$ where $E$ is a constant. We could rearrange this as $$\frac{dx}{dt} = \pm \sqrt[]{2E-2V(x)}.$$ This is an infinite family  (parametrized by E) of separable first order equations. Unfortunately this usually does not help.
\begin{example}
Consider a pendulum of length $l$ with all the concentrated at the end. The equation governing the motion of the pendulum is $$\frac{d^2\theta}{dt^2} +\frac{g}{l}\sin(\theta) = 0$$ where $g$ is the acceleration due to gravity. Prove that there exisits $T>0$ such that $\theta (t+T) = \theta(t)$ for all $t.$ The least such $T$ is called the period of $\theta.$ So $$V'(\theta) = \frac{g}{l}\sin(\theta)$$ and without loss of generality we take $$V(\theta) = \frac{-g}{l}\cos(\theta)$$ and the energy conservation equation becomes $$\frac{1}{2}\bigg(\frac{dx}{dt}\bigg)^2- \frac{g}{l}\cos(\theta) = E.$$ In the special case where $E = \frac{g}{l}$ we get $$\frac{1}{2}\bigg(\frac{dx}{dt}\bigg)^2 = \frac{g}{l}(1+\cos(\theta) = \frac{2g}{l}\cos^2\bigg(\frac{\theta}{2}\bigg).$$ However this is the only cases where an explicit solution is possible. For the other cases we shall plot the curves in $\mathbb{R}^2$ given by $t \mapsto(\theta(t), \frac{d\theta}{dt}).$
\end{example}
\begin{definition}
Consider a conservative equation $$\frac{d^2x}{dt^2} + V'(x) = 0.$$ The phase diagram for this equation is the set of parametric curves $$t \rightarrow(x(t),x'(t))$$ where $x$ solves the differential equation.
\end{definition}
\begin{example} [Linearized Pendulum]
From the previous example $$\frac{d^2x}{dt^2}+\frac{g}{l}x = 0$$ has an auxiliary quadratic $$\lambda^2 + \frac{g}{l} = 0$$ with roots $$\lambda = \pm i \omega, \hspace{15pt} \omega = \sqrt[]{\frac{g}{l}}.$$ The general solution is $x(t) = A\sin(\omega t) + B\cos(\omega t)$ where $A$, $B$ are constants. We can also write this as $x(t) = R\cos(\omega t - \delta).$ Then $$R\cos(\omega t - \delta) = R\cos(\omega t )\cos(\delta)+R\sin(\omega t)sin(\delta)$$ and thus $$R\cos(\delta) = B, \hspace{15pt} R\sin(\delta) = A.$$ Hence $$x'(t) = -\omega R \sin(\omega t -\delta)$$ and so $$(x(t),x'(t) = (R\cos(\omega t - \delta), -\omega R\sin(\omega t -delta)).$$ These are ellipses because $$\bigg(\frac{x(t)}{R}\bigg)^2+\bigg(\frac{x'(t)}{\omega R}\bigg)^2 = 1$$ The fact that the solutions are periodic is revealed by the fact that the curves in the phase diagram are closed.
\end{example}
\begin{example}[Pendulum Phase Diagram]
For the lineraized pendulum $$\frac{d^2x}{dt^2}+\frac{g}{l}x=0.$$ We have $$V'(x)=\frac{g}{l}x = \omega^2x.$$ We can take $$V(x) = \frac{1}{2}\omega^2x^2$$ and the energy conservation equation is $$\frac{1}{2}(x'(t))^2+V(x) = E$$ where $E$ is a constant i.e $$\frac{1}{2}\omega ^"(x(t))^2+\frac{1}{2}(x'(t))^2 = E.$$ Then by multiplying with 2 and dividing by $2E$ to get $$\bigg(\frac{x(t)\omega}{\sqrt[]{2E}}\bigg)^2+\bigg(\frac{x'(t)}{\sqrt[]{2E}}\bigg)^2 = 1$$ which is the same as the previous example with $$R =\frac{\sqrt[]{2E}}{\omega}.$$ For the full pendulum equation $$V'(x) = \frac{g}{l}\sin(x) = \omega^2\sin(x)$$ and we take $$V(x) = \omega^2 - \omega^2 \cos(x).$$ The phase curves are $$\frac{1}{2}(x'(t))^2+\omega^2(1-\cos(x))=E.$$
 We want to plot the curves, and we already have the case $$V(x) = \frac{1}{2}x^2$$ or generally $$V(x) = kx^2 \hspace{25pt} k>0$$ when the phase curves are ellipses. In both of these examples, $V(x)  \geq 0$ everywhere so we need $E\leq 0$. When $E=0$ we need both $y=0$ and $v(x) = 0$. For the case $V(x) = \frac{g}{l}(1-\cos(x))$ this gives the points $(2n\pi,0), n\in \mathbb{Z}.$ For the case $V(x) = kx^2$ we just get (0,0) when $E>0$ is small. Thus we draw a line at height $E$ across the graph $V(x).$
 \smallskip
 For the pendulum equation when $E\geq \frac{2g}{l}$ then $E\geq V(x)$ for all $x$.

 \bigskip

 Suppose we now wish to approximate the phase curves, either for the pendulum equation or any other, in a small neighborhood of a point $(x_0,0)$ at which $V$ has a local maximum e.g $x_0 = \pi$ for the pendulum equation with $V(x) = \frac{g}{l}(1-\cos(x))$. We use the Taylor expansion $$V(x) = V(x_0) +(x-x_0)+V'(x_0)+\frac{1}{2!}(x-x_0)^2V''(x_0)+....$$ Since $V$ has a local maximum at $x_0$ $$V'(x_0) = 0$$ $$V''(x_0) \leq 0.$$ Assume $V''(x_0) <0$ and so the energy conservation equation is $$E=\frac{1}{2}(x')^2+V(x) = \frac{1}{2}(x')^2+V(x_0) + \frac{1}{2|}(x-x_0)^2V''(x_0) +.....$$ $$\approx \frac{1}{2}(x')^2-\frac{1}{2}\omega^2(x-x_0)^2$$ where $\omega^2 = -V''(x_0)>0.$ The curves $\frac{1}{2}y^2-\frac{1}{2}\omega^2(x-x_0)^2 = E$ are hyperbola.
\end{example}
\begin{example}
Draw the phase diagram for the equation $$\frac{d^2x}{dt^2}+2x-3x^2 = 0$$ show that there exists periodic solutions and obtain an integral expression for their periods.
\begin{solution}
Here $V'(x) = 2x-3x^2$ so $V(x) = x^2-x^3$ is a suitable choice for $v$. The energy equation is $$\frac{1}{2}(x')^2+x^2+x^3 = E$$
\end{solution}
\end{example}
\subsection*{Integral Expression for the Period of a Periodic Solution}
Observe that since $\frac{1}{2}(x')^2+V(x)$ is a constant and since $x'=0$ when $x=\alpha$ and $x=\beta$ we have $$V(\alpha)=v(\beta).$$ The period of the solution is the time it takes from $\alpha$ to $\beta$ plus the time to return. By symmetry these times are equal so the period is twice the transit from $\alpha$ to $\beta.$ Thus the period is $$T= \int_0^T dt = 2\int_\alpha^\beta\frac{dt}{dx}dx = 2\int_\alpha^\beta \frac{dx}{x'}$$ Now $$\frac{1}{2}(x')^2 +V(x) = V(\alpha)=v(\beta)$$ giving $$x' = \pm \sqrt[]{2(V(\beta)-V(x))}.$$ From $\alpha$ to $\beta$, $x$ increases so $$x' = \sqrt[]{2(V(\beta)-V(x))}.$$ Thus $$T = \sqrt[]{2}\int_\alpha^\beta\frac{dx}{\sqrt[]{V(\beta)-V(x)}}.$$

%------------------------------------------------
\endinput


\section{Introduction}

In this lab, our goal is to study the phenomenon of simple harmonic oscillation,
find the spring constant and effective mass of a spring,
and how to use the air track. We will analyze the relationship between the
oscillation period and the mass of the oscillator, check whether the oscillation
period depends on the the amplitude, and examine the relationship between the
maximum speed and the amplitude.

There are various kinds of periodic motion in nature, among which the simplest
and the most fundamental one is the simple harmonic motion, where the restoring
force is proportional to the displacement from the equilibrium position and as a
result, the position of a particle depends on time as the sine (or cosine)
function. Discussion of the simple harmonic motion is a basis for studying more
complex situations.

\subsubsection{Hooke's Law}
Within the elastic limit of deformation, the force Fx needed to be applied in
order to stretch or compress a spring by the distance x is proportional to that
distance, i.e.

\begin{equation}
\label{hookeslaw}
F_x=kx,
\end{equation}

where $k$ is a the spring constant that characterizing how easy it is to deform a
spring.
This constant will be found in the present exercise using a measurement device
called the Jolly balance.
The linear relation  between the force and the deformation, is known as the
Hooke’s Law.
According to Newton's third law of dynamics, the spring exerts a reaction force
(called the elastic force) of the same magnitude but opposite direction.

\subsection{Equation of Motion of the Simple Harmonic Oscillator}
\begin{figure}[H]
\centering
\includegraphics[width=12cm]{fig/i1}
\caption{Mass-spring system}
\label{mss}
\end{figure}

As shown in Figure~\ref{mss}, the mass with two springs is placed on an
air track, which aims at eliminating the frictional forces. Assuming that the
restoring force is the only force acting on mass M, the equation of motion od
mass M is

\begin{equation}
\label{M1}
M\frac{d^2x}{dt^2}+(k_1+k_2)x=0.
\end{equation}

Hence the general solution to Eq. \ref{M1} is

\begin{equation}
x(t)=Acos(\omega_0t+\phi_0),
\end{equation}

where $\omega_0=\sqrt{(k_1+k_2)/M}$ is the natural angular frequency of the
oscillations (determined by the parameters of the system), A is the amplitude,
and $\phi_0$ is the initial phase (determined by initial conditions). The
natural period of oscillation is

\begin{equation}
T=\frac{2\pi}{\omega_0}=2\pi\sqrt{\frac{M}{k_1+k_2}}.
\end{equation}

In this exercise, the relationship mentioned above will be studied.

\subsection{Mass of the Spring}

We take into the mass of the springs in terms of the so-called
effective mass, which is the sum of the mass of the object $M$ and
the effective mass of springs $m_0$.
The angular frequency can be expressed as

\begin{equation}
  \label{omega1}
\omega_0=\sqrt{\frac{k_1+k_2}{M+m_0}}
\end{equation}

where $m_0$ is 1/3 of the actual mass of the spring.

\subsection{Mechanical Energy in Harmonic Motion}

The elastic potential energy for a spring-mass system is $U=kx^2/2$ and the
kinetic energy of an oscillating mass is $K=mv^2/2$.
At the equilibrium position ($x=0$), the speed of the mass is maximum
$v=v_{\max}$.
At this point the total mechanical energy is equal to maximum kinetic energy
$K_{\max}$ .
On the other hand, at maximum displacement ($x=\pm A$) the mass is
instantaneously at rest, and the contribution to the total mechanical energy is
due to the potential energy only, which is at its maximum $U_{\max}$.
In the absence of non-conservative forces (such as frictional forces or drag
forces), the total mechanical energy is conserved and $K_{\max}=U_{\max}$, which
implies

\begin{equation}
k=\frac{m (v_{max})^2}{A^2}
\label{kmv2_A2}
\end{equation}


\subsection*{Models}
\label{sec:methods_models}

We modeled the data by considering only the carbon chain length of the molecules composing each stimulus.
The computation of the animal's response probability to each test stimulus was given by first computing:
\begin{equation}
f(S+_i,test_j) = f_1(S+_i) * f_2(S+_i,test_j)
\end{equation}
with each function argument the carbon chain length of an $S+$ component $i$ or a test stimulus component $j$, where:
\begin{equation}
f_1(x_i) = exp(-a*x_i) \\
f_2(x_i,x_j) = exp(-b*|x_i-x_j|)
\end{equation}
with $a$ and $b$ free parameters, obtained by maximum likelihood estimation (see below).
$f_1$ represents the salience of an $S+$ component, and the functional form of $f_1$ is inspired by the vapor pressure dependence of carbon chain length, which is one component of perceived intensity.  $f_2$ represents the generalization gradient across carbon chain length.

We then considered 3 classes of models, varying in computational complexity, i.e. the number of computations of $f$ across stimulus components.
\begin{itemize}
\item \textit{Scalar}: Each stimulus was represented by the mean carbon chain length of its components.  The probability of response ($p_r$) was a function of the difference between the mean carbon chain length of the $S+$ and that of the test stimulus, i.e.:
\begin{equation}
p_r = p_{r0} * f(S+_{mean},test_{mean})
\end{equation}
\item \textit{All-to-All}: Each stimulus was represented by an array of length N containing the carbon chain lengths of the components, e.g. an array of length 2 for a binary mixture. $p_r$ was a function of the difference between each element of the $S+$ array and each element of the test stimulus array, i.e. the difference in carbon chain lengths between each $S+$ component and each test stimulus component.
\begin{equation}
p_r = p_{r0} * \frac{1}{N^2}\sum_{i=1}^{N}\sum_{j=1}^{N}f(S+_i,test_j)
\end{equation}
\item \textit{Nearest Neighbor}: Each stimulus was represented as in \textit{All-to-All}, but $p_r$ was a function only of the difference between each element $i$ of the $S+$ array and the single nearest (most similar in carbon chain length) element of the test stimulus array, with index $i'$.
\begin{equation}
p_r = p_{r0} * \frac{1}{N}\sum_{i=1}^{N}f(S+_i,test_{i'})
\end{equation}
\end{itemize}

Each model was fit to the pooled data, expressed as response probabilities for each test stimulus in each experiment, using maximum likelihood estimation to obtain the parameters $a$, $b$, and $p_{r0}$, which applied universally to all experiments.  In an alternative fit, we allows $p_{r0}$ to vary according to the experiment type (although it was still fixed for all animals, strains, and stimuli within each experiment), to account for variable baseline response motivation across experiments.  Goodness-of-fit is reported as the mean-squared error between the model fit and the observed response probabilities, and is fundamentally bounded at the low end due to binomial variability.

All code and analysis is available at \url{http://github.com/quolf/nearest-neighbor}.


\documentclass[../research.tex]{subfile}

\subsection{Sequences}
\label{sec:1.1}

  We denote set of positive integers as $\N$.

  \begin{prop}
    Let $a_i = (a_i^1,\dots,a_i^n)\in\R^n$ be a sequence,  $\forall n\in\N$. Then
    $a_i\to a:=(a^1,\dots,a^n)\in\R^n$, as $i\to\infty$ iff $a_i^k\to a^k$ as $i\to\infty$,
    $\forall k\in\{1,\dots,n\}$.
  \end{prop}

  \begin{proof}
    We know that
    $$\max_{k=1,\dots,n}{|a^k|}\leq\|a\|\leq\sqrt{n}\max_{k=1,\dots,n}{|a^k|}.$$

    To show this, observe that:
    $$\sqrt{\max_k{((a^k)}^2)}\leq\sqrt{(a^1)^2+\dots+(a^n)^2}\leq\sqrt{n\max_k{((a^k)^2})}$$
    $$\Rightarrow \sqrt{(\max_k{|a^k|})^2}\leq\|x\|\leq\sqrt{n(\max_k{|a^k|})^2}$$
    $$\Rightarrow \max_k{|a^k|}\leq\|x\|\leq\sqrt{n}\max_k{|a^k|}$$

    First assume that $a_i\to a$ as $i\to\infty$. Then $\forall\epsilon>0$ $\exists N\in\N$ such
    that $\|a_i-a\|<\epsilon$ whenever $i\geq N$.

    Then $\forall k=1\dots n$, $|a^k_i-a^k|\leq\max_k{|a^k_i-a^k|}\leq\|a_i-a\|<\epsilon$ whenever
    $i\geq N$, so $a^k_i\to a^k$ as $i\to\infty$.

    Conversely, assume that $a_i^k\to a^k$ as $i\to\infty$, $\forall k\in\{1,\dots,n\}$.
    Then $\forall\epsilon>0$, $\exists N^k\in\N$ such that $|a_i^k-a^k|\leq\frac{\epsilon}{\sqrt{n}}$
    whenever $i\geq N^k$, $\forall k=1,\dots,n$. Let
    $$N=\max_{k=1,\dots,n}{N^k}.$$

    Then $\max_k{|a^k_i-a^k|}<\frac{\epsilon}{\sqrt{n}}$ whenever $i\geq N$, so
    $\|a_i-a\|\leq\sqrt{n}\max_k{|a^k_i-a^k|}<\epsilon$.
  \end{proof}

  \begin{prop}
    The limit of a sequence in $\R^n$ is unique.
  \end{prop}

  \begin{proof}
    Let $(x_i)_{i\in\N}$ be a sequence with $x_i\in\R^n$.
    Suppose for a contradiction that $x_i\to a$ and $x_i\to b$ as $i\to\infty$, with $a\neq b$.

    Then $\forall\epsilon>0$, $\exists N_a,N_b\in\N$ such that $\|x_i-a\|<\frac{\epsilon}{2}$ whenever
    $i\geq N_a$, and $\|x_i-b\|<\frac{\epsilon}{2}$ whenever $i\geq N_b$. Let
    $$N=\max{\{N_a,N_b\}}.$$
    Then whenever $i\geq N$, we have:
    \begin{align*}
    \|a-b\|&=\|a-x_i+x_i-b\|\\
           &\leq\|a-x_i\|+\|x_i-b\|\\
           &<\frac{\epsilon}{2}+\frac{\epsilon}{2}=\epsilon,
    \end{align*}
    And since $\epsilon$ can be made arbitrarily small, we find that $a=b$, which is a contradiction.
  \end{proof}

  \begin{theorem}[Monotone convergence]
    An increasing and bounded above sequence is convergent. Likewise, a decreasing and bounded below
    sequence is convergent.
  \end{theorem}

  \begin{proof}
    Suppose $(a_i)$ is bounded above and increasing. Then the set $A=\{a_i:i\in\N\}$ has a supremum,
    say $a=\sup{A}$. I claim that $a_i\to a$ as $i\to\infty$. \\

    \textit{Proof of claim.}
      Fix $\eps$. Then $\exists a_N$, with $N\in\N$ such that
      $$a-\epsilon<a_N\leq a<a+\epsilon.$$
      Since $(a_i)$ is increasing, $a_i\geq a_N$ when $i\geq N$, and since it is also bounded above
      by $a$, we have
      $$a-\epsilon<a_N\leq a\leq a_i<a+\epsilon,$$
      and so
      $$|a-a_i|<\epsilon.$$
    So $a_i\to a$ as $i\to\infty$. Simililary, we can show that if $(a_i)$ is bounded below and
    decreasing, $a_i\to \inf{\{a_i:i\in\N\}}$ as $i\to\infty$.
  \end{proof}

  \begin{lemma}
    Let $(a_n)_{n\in\N}$ be a bounded sequence of real numbers. Then $(a_n)_{n\in\N}$ has a convergent
    subsequence.
  \end{lemma}
%
%  \begin{proof}
%    We say $x_i$ is a peak point of $(a_n)$ if $a_n\leq x_i, \forall n>i$.\\
%
%    Then either we have infinitely many peak points, or finitely many.
%    In the first case, we have an infinite subsequence, $(x_{i_j})_{j\in\mathbb{N}}$, of peak points
%    which is non-increasing, since $a_{i_n}\leq x_{i_j}$ whenever $n>j$. Then since $(a_i)$ is
%    bounded, so is $(x_{i_j})$, so it is convergent.\\
%
%    Now consider the case of finitely many peak points, say $x_1,\dots x_k$. Choose
%    $n_1>\text{max}\{1,\dots,k\}$. Since $a_{n_1}$ is not a peak point, $\exists n_2 > n_1$ such
%    that $a_{n_2}>a_{n_1}$. Since $a_{n_2}$ is not a peak point, $\exists n_3 > n_2$ such that
%    $a_{n_3}>a_{n_2}$. Continuing inductively, with $a_{n_i}>a_{n_{i-1}}$, we obtain a decreasing
%    subsequence $(a_{n_j})_{j\in\N}$. Then since $(a_{n_j})_{j\in\N}$ is bounded, it converges.
%  \end{proof}
%
%  We now present another proof.
%
  \begin{proof}
    Since $(a_i)$ is bounded, $\exists K\geq0$ such that $a_i\in I_0:=[-K,K]\,\forall i\in\N$.
    Then one of $[-K,0]$, $[0,K]$ contains infinitely many elements of $(a_i)$. Choose the
    leftmost one, and call it $I_1$. Repeat this inductively, such that $I_j:=[c_j,d_j]$
    contains infinitely many elements of $(a_i)$. We then inductively define
    $$i_j=\min_{i>i_{j-1}}{i:a_i\in I_j}.$$
    In other words, $a_{i_j}$ is the first element of $(a_i)$ which lies in $I_j$. Now, since
    $$I_j\subset I_{j-1}\subset\dots\subset I_1\subset I_0,$$
    The sequences $(c_j)$ and $(d_j)$ are monotically increasing and decreasing, respectively,
    and since they are bounded, they both converge, so say that
    $$c_j\to c\in[-K,K]\text{ as }j\to\infty,$$
    $$d_j\to d\in[-K,K]\text{ as }j\to\infty.$$
    Since $d_j-c_j=\text{length}(I_j)=2^{1-j}K\to0$ as $j\to\infty$, we have that $c=d$, and
    since $c_j\leq a_{i_j}\leq d_j$, we also have that $a_{i_j}\to c=d$ as $j\to\infty$.
  \end{proof}

  \begin{cor}
    Let $(a_i)_{i\in\N}$ be a bounded sequence in $\R^n$. Then $(a_i)$ has a
    convergent subsequence.
  \end{cor}
  \begin{proof}
    Since $(a_i)$ is bounded, and the component sequences $(a_i^k)$ are bounded $\forall k=1,\dots,n$.

    Then by the previous lemma, $(a_i^1),\dots,(a_i^n)$ have subsequences $(a_{i_{j_1}}^1), \dots,
    (a_{i_{j_n}}^n)$ converging to $a^1,\dots,a^n$ respectively. Let
    \begin{align*}
      J_1&=\{j_{1_1},j_{1_2},\dots\},\\
      J_2&=\{j_{2_1},j_{2_2},\dots\},\\
         &\vdots \\
      J_n&=\{j_{n_1},j_{n_2},\dots\},
    \end{align*}
    and $J=J_1\cap\dots\cap J_n$.\\

    Then each component subsequence $(a_j^k)_{j\in J}$ converges to $a^k$, so $(a_j)_{j\in J}$
    converges to $a$.
  \end{proof}

  \begin{theorem}[Bolzano-Weierstrass]
    Let $E\subset\R^n$ be compact and $(a_i)_{i\in\N}$ a sequence in $E$. Then $(a_i)$ has a
    subsequence converging to a point in $E$.
  \end{theorem}

  \begin{proof}
    Since $(a_i)$ is bounded, it has a subsequence $(a_{i_j})\to a$ as $j\to\infty$.

    And since $E$ is closed, and $a$ a limit point of $E$, $a\in E$.
  \end{proof}

\subsection{Continuous Functions}
\label{sec:1.2}

  \begin{definition}[Continuity]
    Let $A\subset\R^n$ be open. We say $f:A\to\R^m$ is \textit{continuous} on $A$ if,
    $\forall a\in A$, $\forall\epsilon>0$ $\exists\delta>0$ such that
    $\|f(x)-f(a)\|<\epsilon$, whenever $\|x-a\|<\delta$.
  \end{definition}

  Generally, we can think of continuous functions are those for which points that are `near' are mapped to
  by points that are also `near', and so showing that a function is continuous amounts to finding a positive
  relationship between the definition of `nearness' in the domain and the definition of `nearness' in
  the range (these are not generally the same). In this
  case, points $x,y\in\R^n$ are near if the distance between them, $\|x-y\|$, is small.

  In a more general topological sense, we say that two points are near, in a certain capacity,
  if they are both elements of the same open set. Thus, very vaguely, the more open sets the points
  are both elements of, the nearer they are. In Euclidean space, open sets are defined as arbitary
  unions of open balls $B_\epsilon(x)$, for $x\in\R^n$ and $\epsilon>0$. Thus, $x=y$ precisely when
  $y\in B_\epsilon(x)$ (and $x\in B_\epsilon(y)$) $\forall\epsilon>0$.

  This is the Euclidean case of
  a general property called non-seperatedness, which we use to describe
  points that both lie in every open set. Since they share every open set in common, their nearness
  is maximal, and so we can think of them as essentially being indistinguishable - i.e. the same point.
  We will revist this later, when we discuss seperatedness conditions.

  \begin{prop}
    For $f:A\to\R^m$, we write $f(x)=(f_1(x),\dots,f_m(x))$.

    Then $f$ is continuous if and only if each of the component functions
    $f_i:A\to\R$ are continuous.
  \end{prop}

  \begin{proof}
    First assume $f$ is continuous at $a\in A$. Then $\forall\epsilon>0$
    $\exists\delta>0$ such that $\|f(x)-f(a)\|<\epsilon$ whenever $\|x-a\|<\delta$. Then
    \begin{align*}
      |f_i(x)-f_i(a)|&\leq\max_i{|f_i(x)-f_i(a)|}\\
                     &\leq\|f(x)-f(a)\|\\
                     &<\epsilon\quad\forall i=1,\dots,m.
    \end{align*}

    Hence each $f_i$ is continuous at $a$. Now assume that each $f_i$ is continuous at $a\in A$.

    Then $\forall\epsilon>0$, $\exists\delta_i>0$ such that $|f_i(x)-f_i(a)|<\frac{\epsilon}{m\sqrt{m}}$
    whenever $\|x-a\|<\delta_i$, $\forall i=1,\dots,m$. Now let $\delta=\min{\{\delta_1,\dots,\delta_m\}}$

    Then when $\|x-a\|<\delta$, $\|x-a\|<\delta_i$, so $|f_i(x)-f_i(a)|<\frac{\epsilon}{m\sqrt{m}}$
    $\forall i=1,\dots,m$. Then
    \begin{align*}
    \|f_i(x)-f_i(a)\|&\leq\sqrt{m}\max_i{|f_i(x)-f_i(a)|}\\
                     &\leq\sqrt{m}\sum_{i=1}^m{|f_i(x)-f_i(a)|}\\
                     &<\epsilon
    \end{align*}
  \end{proof}

  \begin{example}
    Every linear function is continuous.
  \end{example}

  \begin{prop}
    Let $A\subset\R^n$ be open. Then $f:A\to\R^m$ is continuous at $x\in A$ if and only if
    $f(x_i)\to f(x)$ as $i\to\infty$ for any sequence $(x_i)$ converging to $x$.
  \end{prop}

  \begin{proof}
    Let $(x_i)$ be a sequence in $A$ converging to $x\in A$, and assume $f:A\to\R^m$ to be continuous
    at $x$. Then $\forall\epsilon>0$, $\exists\delta>0$ such that $\|f(x)-f(y)\|<\epsilon$
    whenever $0<\|x-y\|<\delta$.\\

    Since $x_i\to x$, $\nat$ such that $\|x_i-x\|<\delta$ whenever $i\geq N$, and hence $\|f(x_i)-f(x)\|<\epsilon$,
    so we have that $f(x_i)\to f(x)$ as $i\to\infty$.\\

    Now assume the converse holds, and for a contradiction that $f$ is not continuous. Then
    $\exists\epsilon>0$ such that $\forall\delta>0$ $\|f(x)-f(x_i)\|\geq\epsilon$ whenever
    $0<\|x-x_i\|<\delta$, and in particular whenever $0<\|x-x_i\|<\frac{1}{i}$, $\forall i\in\N$.

    Then the sequence $(x_i)$ converges to $x$, so by assumption the image sequence $(f(x_i))$
    converges to $f(x)$.\\

    Thus $\exists N\in\N$ such that $\|f(x_i)-f(x)\|<\epsilon$ whenever $i\geq N$, which is a
    contradiction. Hence, $f$ is continuous at $x$.
  \end{proof}

  \begin{theorem}[Extreme Value Theorem]
    Let $E\subset\R^n$ be compact and non-empty. Let $f:E\to\R$ be continuous.
    Then $f$ achieves its maximum and minimum values on $E$.
  \end{theorem}

  \begin{proof}
    We prove that $f$ reaches its maximum on $E$. First suppose for a contradiction that
    $f$ is unbounded above on $E$. Then $\forall M\geq 0$, and in particular
    $\forall i\in\N$, $\exists x_i\in E$ such that $f(x_i)>i$.
    In other words, $f(x_i)\to\infty$ as $i\to\infty$.

    Then since $E$ is compact, it is bounded, so the sequence $(x_i)_{i\in\N}$ is bounded.
    Then by the Bolzano-Weierstrass theorem it has a subsequence $(x_{i_j})_{j\in\N}$ convergent to $x\in E$.

    Now since $f$ is continuous on $E$, it is continuous at $x$, so $f(x_{i_j})\to f(x)$ as $j\to\infty$.
    Since $E$ is compact, it is closed, so $f(x)\in E$, and therefore $f(x)$ is finite. This is a contraction since
    $f$ was assumed to diverge to infinity for any subsequence of $(x_i)$, including $(x_{i_j})$.

    So $f$ is bounded above on $E$. Now by completeness of $\R$, $f(E)$ has a supremum, say
    $$M=\sup_{x\in E}{f(x)}.$$
    We show that that $f$ reaches its maximum on $E$, i.e. that $M\in f(E)$, or
    that $\exists p_+\in E$ such that $f(p_+)=M$. Now $\forall i\in\N$, $\exists x_i\in E$ such that
    $$M-\frac{1}{i}<f(x_i)\leq M.$$
    Then clearly $f(x_i)\to M$ as $i\to\infty$, so by the Bolzano-Weierstrass theorem $(x_i)_{i\in\N}$ has
    a subsequence $(x_{i_j})_{j\in\N}$ convergent in to some $p_+\in E$. Since $f$ is continuous at $p_+$,
    $f(x_{i_j})\to f(p_+)$ as $j\to\infty$. Hence $f(p_+)=M$, so $f$ achieves its maximum on $E$.

    Now in a similar fashion we prove that $f$ reaches its minimum on $E$. Suppose that
    $f$ is unbounded below on $E$. Then $\forall i\in\N$, $\exists x_i\in E$ such that $f(x_i)<-i$.
    So $f(x_i)\to-\infty$ as $i\to\infty$.

    Now, the sequence $(x_i)_{i\in\N}$ is bounded. Then by the Bolzano-Weierstrass theorem it has a
    subsequence $(x_{i_j})_{j\in\N}$ convergent to $x\in E$. Since $f$ is continuous at
    $x$, $f(x_{i_j})\to f(x)$ as $j\to\infty$, and so $f(x)\in E$ (so $f(x)$ is finite). This is a contraction since
    $f$ was assumed to diverge to negative infinity for any subsequence of $(x_i)$, including $(x_{i_j})$.

    So $f$ is bounded below on $E$. Now by completeness of $\R$, $f(E)$ has an infimum, say
    $$m=\inf_{x\in E}{f(x)}.$$
    We show that that $f$ reaches its minimum on $E$, i.e. that $m\in f(E)$, or
    that $\exists p_-\in E$ such that $f(p_-)=m$. Now $\forall i\in\N$, $\exists x_i\in E$ such that
    $$m\leq f(x_i)<m+\frac{1}{i}.$$
    Then clearly $f(x_i)\to m$ as $i\to\infty$, so by the Bolzano-Weierstrass theorem $(x_i)_{i\in\N}$ has
    a subsequence $(x_{i_j})_{j\in\N}$ convergent in to some $p_-\in E$. Since $f$ is continuous at $p_-$,
    $f(x_{i_j})\to f(p_-)$ as $j\to\infty$. Hence $f(p_-)=m$, so $f$ achieves its minimum on $E$.
  \end{proof}

\subsection{Uniform Continuity and Convergence}
\label{sec:1.3}

  \begin{definition}[Uniformly Continuous]
    Let $A\subset\R^n$ be open. We say $f:A\to\R^m$ is \textit{uniformly continuous} on $A$ if,
    $\forall\epsilon>0$ $\exists\delta>0$ such that $\forall x,y\in A$,
    $\|f(x)-f(y)\|<\epsilon$ whenever $\|x-y\|<\delta$.
  \end{definition}

  \textbf{Remark:} Clearly uniform continuity implies continuity.

  \begin{prop}
    Let $f:\R\to\R$ be differentiable with bounded derivative. Then $f$ is uniformly continuous.
  \end{prop}

  \begin{proof}
    Fix $\eps$. Then $\forall x,y\in\R$ with $x<y$, by the Mean Value Theorem
    $\exists c\in(x,y)$ such that $f'(c)=\frac{f(y)-f(x)}{y-x}$. By assumption
    $\exists M\geq 0$ such that $|f'(c)|=\frac{|f(y)-f(x)|}{|y-x|}\leq M$.

    Set $\delta = \frac{\epsilon}{M+1}$. Then
    \begin{align*}
      |f(y)-f(x)|&\leq M|y-x|\\
                 &<\frac{M\epsilon}{M+1}\\
                 &<\frac{M\epsilon}{M}\\
                 &=\epsilon
    \end{align*}
  \end{proof}

  \begin{prop}
    A continuous function on a compact subset $E\subset\R^n$ is uniformly continuous.
  \end{prop}

  \begin{proof}
    Let $E\subset\R^n$ be compact, and $f:E\to\R^m$ continuous. We assume for a contradiction that
    $f$ is not uniformly continuous. Then $\exists\epsilon>0$ such that $\forall i\in\N$
    $\exists x_i,y_i\in E$ with $\|x_i-y_i\|<\frac{1}{i}$ such that $\|f(x_i)-f(y_i)\|\geq\epsilon$.

    Then by the Bolzano-Weierstrass theorem, the sequence $(x_i)_{i\in\N}$ has a
    subsequences $(x_{i_j})_{j\in\N}$ converging to some $x\in E$.  Furthermore, the sequence $(y_i)_{i\in\N}$
    has a convergent subsequence $(y_{i_j})_{j\in\N}$. We now show that $y_{i_j}\to x$ as $j\to\infty$.

    Choose $\epsilon'>0$. Then $\exists N\in\N$ such that $\|x_{i_j}-x\|<\frac{\epsilon'}{2}$ whenever $i_j\geq N$.
    \begin{align*}
    \|x-y_{i_j}\|&=\|x-x_{i_j}+x_{i_j}-y_{i_j}\|\\
                 &\leq\|x-x_{i_j}\|+\|x_{i_j}-y_{i_j}\|\\
                 &<\frac{\epsilon'}{2}+\frac{1}{i_j}\\
                 &\leq\frac{\epsilon'}{2}+\frac{1}{j}\\
                 &<\frac{\epsilon'}{2}+\frac{\epsilon'}{2}\\
                 &=\epsilon'
    \end{align*}
  \end{proof}

  \begin{theorem}[Heine-Borel]
    Let $A\subset\R^n$ be closed and bounded. Then $A$ is compact.
  \end{theorem}

  \begin{definition}[Uniform convergence]
    Let $A\subset\R^n$ and $(f_i)_{i\in\N}$ be a sequence of functions $f_i:A\to\R^m.$

    We say $f_i\to f$ \textit{uniformly} to $f:A\to\R^m$ if and only if $\forall\epsilon>0$,
    $\exists N\in\N$ such that $\|f_i(x)-f(x)\|<\epsilon$ whenever $i\geq N$ and $x\in A$.
  \end{definition}

  \begin{theorem}
    Let $A\subset\R^n$ and $(f_i)_{i\in\N}$ be a sequence of functions $f_i:A\to\R^m$ which
    converges uniformly to $f:A\to\R^m$ uniformly.

    Then if $f_i$ is uniformly continuous $\forall i\in\N$, $f$ is uniformly continuous.
  \end{theorem}

  \begin{proof}
    Fix $\epsilon>0$. Then since $f_i\to f$ uniformly, $\exists N\in\N$ such that
    $\|f_i(x)-f(x)\|<\epsilon/3$ whenever $i\geq N$ and $x\in A$.

    Also since each $f_i$ is uniformly continuous, $\exists\delta>0$ such that $\|f_i(x)-f_i(y)\|$
    $\forall x,y\in A$. Then using the triangle inequality:
    \begin{align*}
      \|f(x)-f(y)\| &\leq\|f(x)-f_i(x)\|+\|f_i(x)-f(y)\| \\
                    &<\epsilon/3+\|f_i(x)-f_i(y)\|+\|f_i(y)-f(y)\| \\
                    &<\epsilon/3+\epsilon/3+\epsilon/3 \\
                    &=\epsilon
    \end{align*}
    $\forall x,y\in A$ such that $\|x-y\|<\delta$. Hence, $f$ is uniformly continuous.
  \end{proof}

  \begin{cor}
    Now assume that each $f_i$ is just continuous instead. Then $f$ is continuous.
  \end{cor}

  \begin{proof}
    Fix $a\in A$, and $\epsilon>0$. Then since $f_i\to f$ uniformly, $\exists N\in\N$ such that
    $\|f_i(x)-f(x)\|<\epsilon/3$ whenever $i\geq N$ and $x\in A$.

    Also since each $f_i$ is continuous, $\exists\delta>0$ such that $\|f_i(x)-f_i(a)\|<\epsilon/3$
    $\forall x\in A$. Then using the triangle inequality:
    \begin{align*}
      \|f(x)-f(a)\| &\leq\|f(x)-f_i(x)\|+\|f_i(x)-f(a)\| \\
                    &<\epsilon/3+\|f_i(x)-f_i(a)\|+\|f_i(a)-f(a)\| \\
                    &<\epsilon/3+\epsilon/3+\epsilon/3 \\
                    &=\epsilon
    \end{align*}
    $\forall x\in A$ such that $\|x-a\|<\delta$. Hence, $f$ is continuous.
  \end{proof}

\subsection{Integration}
\label{sec:1.4}

  \begin{definition}[Partitions]
    Let $[a,b]\subset\R$. We define a \textit{partition}, $\mathcal{P}$, of $[a,b]$ by an ordered tuple
    $$\mathcal{P}=(x_0,x_1,\dots,x_{k-1},x_k),$$
    such that $a=x_0<x_1<\dots<x_{k-1}<x_k=b$.

    We call any interval of the form $[x_j,x_{j+1}]$ a \textit{subinterval} of the partition $\mathcal{P}$.
    The length of this subinterval is $\Delta x_j=x_{j+1}-x_j$, and we define
    $$\text{mesh}(\mathcal{P}):=\max_{j=0,\dots,k-1}{\Delta x_j}.$$

    Finally, for any two partitions $\mathcal{P}$ and $\mathcal{Q}$, we say $\mathcal{Q}$ \textit{refines}
    $\mathcal{P}$ iff every point in $\mathcal{P}$ lies in $\mathcal{Q}$.
  \end{definition}

  \begin{definition}[Darboux sums]
    Let $f:[a,b]\to\R$ be bounded, and fix a partition $\mathcal{P}$ of $[a,b]$. Then we define the
    \textit{upper} and \textit{lower Darboux sums} by
    $$U(f,\mathcal{P})=\sum_{j=0}^{k-1}M_j\Delta x_j,$$
    where $M_j=\sup_{x\in[x_j,x_{j+1}]}{f(x)}$, and
    $$L(f,\mathcal{P})=\sum_{j=0}^{k-1}m_j\Delta x_j,$$
    where $m_j=\inf_{x\in[x_j,x_{j+1}]}{f(x)}$.
  \end{definition}

  \begin{lemma}
    Let $\mathcal{P}$ and $\mathcal{Q}$ be partitions of $[a,b]$, with $\mathcal{Q}$ a refinement of
    $\mathcal{P}$. Then
    $$L(f,\mathcal{P})\leq L(f,\mathcal{Q})\leq U(f,\mathcal{Q})\leq U(f,\mathcal{P}).$$
  \end{lemma}

  \begin{proof}
    Let $\mathcal{Q}=(x_0=a,x_1,\dots,x_k=b)$. Let $I_j$ be the subinterval of $\mathcal{P}$
    containing $[x_j,x_{j+1}]$, for $j=0,\dots,k-1$. Then
    $$\inf_{x\in I_j}{f(x)}\leq\inf_{x\in [x_j,x_{j+1}]}{f(x)}\leq\sup_{x\in [x_j,x_{j+1}]}{f(x)}
    \leq\sup_{x\in I_j}{f(x)}.$$
    Then by multiplying by $\Delta x_j$ and summing over $j$, we get
    $$\sum_{j=0}^{k-1}{\Delta x_j\inf_{x\in I_j}{f(x)}}\leq\sum_{j=0}^{k-1}{\Delta x_j m_j}
    \leq\sum_{j=0}^{k-1}{\Delta x_j M_j}\leq\sum_{j=0}^{k-1}{\Delta x_j\sup_{x\in I_j}{f(x)}}.$$
    We can rewrite the two outer sums as
    $$\sum_{j=0}^{k-1}{\Delta x_j\inf_{x\in I_j}{f(x)}}=\sum_{\text{subints. of }\mathcal{P}
    \text{, }I}{\bigg[\inf_{x\in I}{f(x)}}\sum_{j\text{ s.t. }I=I_j}{\Delta x_j}\bigg]=L(f,\mathcal{P}),$$
    $$\sum_{j=0}^{k-1}{\Delta x_j\sup_{x\in I_j}{f(x)}}=\sum_{\text{subints. of }\mathcal{P}
    \text{, }I}{\bigg[\sup_{x\in I}{f(x)}}\sum_{j\text{ s.t. }I=I_j}{\Delta x_j}\bigg]=U(f,\mathcal{P}).$$
    Hence,
    $$L(f,\mathcal{P})\leq L(f,\mathcal{Q})\leq U(f,\mathcal{Q})\leq U(f,\mathcal{P}).$$
  \end{proof}

  \begin{cor}
    $$L(f,\mathcal{P})\leq U(f,\mathcal{P'})$$
    for any two partitions $\mathcal{P}$ and $\mathcal{P'}$.
  \end{cor}

  \begin{proof}
    Let $\mathcal{Q}$ be a common refinement of $\mathcal{P}$ and $\mathcal{P'}$. Then
    $$L(f,\mathcal{P})\leq L(f,\mathcal{Q})\leq U(f,\mathcal{Q})\leq U(f,\mathcal{P'}).$$
  \end{proof}

  \begin{definition}[Darboux integrals]
    We define the following sets:
    \begin{align*}
      U_f&=\{U(f,\mathcal{P'}):\mathcal{P'}\text{ is a partition of }[a,b]\}, \\
      \text{and }L_f&=\{L(f,\mathcal{P}):\mathcal{P}\text{ is a partition of }[a,b]\}.
    \end{align*}
    Let $\mathcal{Q}$ be any parition of $[a,b]$. Then $U_f$ is bounded below by $L(f,\mathcal{Q})$ and $L_f$ is
    bounded above by $U(f,\mathcal{Q})$. Thus we can define \textit{upper} and \textit{lower Darboux
    integrals} by:
    $$\overline{\int_a^b}f(x)\text{d}x=\inf{U_f},$$
    $$\underline{\int_a^b}f(x)\text{d}x=\sup{L_f}.$$
    If these two are equal, then we say $f$ is \textit{integrable}, and we let
    $$\int_a^bf(x)\text{d}x=\overline{\int_a^b}f(x)\text{d}x=\underline{\int_a^b}f(x)\text{d}x.$$
  \end{definition}

  \begin{prop}
    A bounded function $f:[a,b]\to\R$ is integrable if and only if $\eps$ there exists a partition
    $\calP$ of $[a,b]$ such that
    $$U(f,\calP)-L(f,\calP)<\epsilon.$$
  \end{prop}

  \begin{proof}
    We assume that $\eps$ there exists a partition $\calP$ of $[a,b]$ such that $U(f,\calP)-
    L(f,\calP)<\epsilon.$ Then
    $$L(f,\calP)\leq\underline{\int_a^b}f(x)\text{d}x\leq\overline{\int_a^b}f(x)\text{d}x\leq U(f,\calP),$$
    so
    $$\overline{\int_a^b}f(x)\text{d}x-\underline{\int_a^b}f(x)\text{d}x\leq U(f,\calP)-L(f,\calP)<\epsilon,$$
    and so
    $$\overline{\int_a^b}f(x)\text{d}x=\underline{\int_a^b}f(x)\text{d}x$$
    so $f$ is integrable.

    Conversely, assume that $f$ is integrable. Fix $\eps$. Then since $\lowint=\sup{L_f}$ and
    $\upint=\inf{U_f}$, $\exists\calP,\calP'$ as parititions of $[a,b]$, such that
    $$\lowint\geq L(f,\calP)>\lowint-\frac{\epsilon}{2},$$
    $$\upint\leq U(f,\calP')<\upint+\frac{\epsilon}{2}.$$
    Since $f$ is integrable, this simplifies to
    $$L(f,\calP)+\frac{\epsilon}{2}>0,$$
    $$U(f,\calP')-\frac{\epsilon}{2}<0.$$
    We let $\calQ$ be the common refinement of $\calP$ and $\calP'$. Then
    $$U(f,\calQ)-L(f,\calQ)\leq U(f,\calP')-L(f,\calP)<\frac{\epsilon}{2}+\frac{\epsilon}{2}
    =\epsilon.$$
  \end{proof}

  \begin{prop}
    Every continuous function is integrable.
  \end{prop}

  \begin{proof}
    Suppose $f:[a,b]\to\R$ is continuous. Since $[a,b]$ is compact, $f$ is uniformly continuous, so
    $\forall x,y\in[a,b]$, $\eps\del$ such that $|f(x)=f(y)|<\epsilon/{b-a}$ when $|x-y|<\delta$.
    Now fix $\eps$, let $calP$ be a partition of $[a,b]$ with $k+1$ uniformly spaced points, and
    such that $\text{mesh}(\calP)<\delta$.

    Then $\forall x,y\in[x_j,x_{j+1}]$, we have $|f(x)-f(y)|<\epsilon/{b-a}$, for $j=0,1,\dots,k-1$.
    In particular $M_j-m_j<\epsilon/{b-a}$, so $\Delta x_j(M_j-m_j)<\Delta x_j\epsilon/{b-a}$. We
    then sum over $j$, noting that $\Delta x_j=b-a/k$

    \begin{align*}
      \sum_{j=0}^{k-1}\Delta x_j(M_j-m_j)&<\sum_{j=0}^{k-1}\frac{\epsilon(b-a)}{k(b-a)} \\
      \sum_{j=0}^{k-1}\Delta x_j M_j-\sum_{j=0}^{k-1}\Delta x_j m_j&<\frac{k\epsilon}{k} \\
      U(f,\calP)-L(f,\calP) &<\epsilon
    \end{align*}
    And so $f$ is integrable by the previous proposition.
  \end{proof}

  \begin{prop}[Linearity of the integral]
    Let $f:[a,b]\to\R$ and $g:[a,b]\to\R$ be integrable functions, and $\lambda\in\R$. Then $f+
    \lambda g$ is inetgrable, and
    $$\int_a^bf(x)+\lambda g(x)\text{d}x=\int_a^bf(x)\text{d}x+\lambda\int_a^bg(x)\text{d}x.$$
  \end{prop}

  \begin{proof}

  \end{proof}

  \begin{prop}[Additivity of the integral]
    Let $f:[a,b]\to\R$ be bounded, and let $c\in[a,b]$. Then $f$ is integrable on $[a,b]$ if and
    only if $f$ is integrable on both $[a,c]$ and $[c,b]$. Moreover
    $$\int_a^bf(x)\text{d}x=\int_a^cf(x)\text{d}x+\int_c^bf(x)\text{d}x.$$
  \end{prop}

  \begin{proof}
    Assume $f$ is integrable on $[a,b]$. Fix $\eps$. Then we can choose a partition $\calP=
    (x_0=a,x_1,\dots,x_{k-1},x_k=b)$ such that $U(f,\calP)-L(f,\calP)<\epsilon$. We let $\calQ$ be a
    refinement of $\calP$ containing $c$, say $\calQ=(x_0=a,\dots,x_{l-1},x_l=c,x_{l+1},\dots,x_k=b)$.
    Clearly $U(f,\calQ)-L(f,\calQ)<\epsilon$.

    We now `split' $\calQ$ into two partitions $\calQ_1=(a,x_1,\dots,c)$ and $\calQ_2=(c,x_{l+1},\dots,b)$.
    Then $U(f,\calQ)=U(f,\calQ_1)+U(f,\calQ_2)$, and $L(f,\calQ)=L(f,\calQ_1)+L(f,\calQ_2)$, so
    $$U(f,\calQ_1)-L(f,\calQ_1)+U(f,\calQ_2)-L(f,\calQ_2)<\epsilon,$$
    and so
    $$U(f,\calQ_1)-L(f,\calQ_1)<\epsilon,$$
    $$U(f,\calQ_2)-L(f,\calQ_2)<\epsilon.$$
    Hence, $f$ is integrable on $[a,c]$ and $[c,b]$.

    Conversely, assume that $f$ is integrable on $[a,c]$ and $[c,b]$. Fix $\eps$. Then there exists
    partitions $\calP_1:=(a,x_1,\dots,c)$ and $\calP_2:=(c,\dots,x_{k-1},b)$ of $[a,c]$ and $[c,b]$ respectively such that
    $$U(f,\calP_1)-L(f,\calP_1)<\frac{\epsilon}{2},$$
    $$U(f,\calP_2)-L(f,\calP_2)<\frac{\epsilon}{2}.$$
    Then
    $$U(f,\calP_1)+U(f,\calP_2)-L(f,\calP_1)-L(f,\calP_2)<\epsilon.$$
    We now `join' $\calP_1$ and $\calP_2$ at $c$ to create a partition of $[a,b]$, $\calQ:=
    (a,\dots,x_{l-1},c,x_{l+1},\dots,b)$. Then
    $$U(f,\calQ)=U(f,\calP_1)+U(f,\calP_2),$$
    $$L(f,\calQ)=L(f,\calP_1)+L(f,\calP_2),$$
    so
    $$U(f,\calQ)-L(f,\calQ)=U(f,\calP_1)-L(f,\calP_1)+U(f,\calP_2)-L(f,\calP_2)<\frac{\epsilon}{2}
    +\frac{\epsilon}{2}=\epsilon.$$
    And hence, $f$ is integrable on $[a,b]$.

    Finally, since $\int_a^bf(x)\text{d}x=\inf{U_f}=\sup{L_f}$, for all $\epsilon>0$ there exists a partition $\calP$ of
    $[a,b]$ such that when we split it into two partitions $\calQ_1$ and $\calQ_2$ of $[a,c]$ and
    $[c,b]$ respectively, as illustrated above, we get
    \begin{align*}
      \int_a^bf(x)\text{d}x &\leq U(f,\calP) \\
                            &=U(f,\calQ_1)+U(f,\calQ_2) \\
                            &<\int_a^cf(x)\text{d}x+\frac{\epsilon}{2}+\int_c^bf(x)\text{d}x
                                                                      +\frac{\epsilon}{2} \\
                            &=\int_a^cf(x)\text{d}x+\int_c^bf(x)\text{d}x+\epsilon.
    \end{align*}
    Similarly,
    \begin{align*}
      \int_a^bf(x)\text{d}x &\geq L(f,\calP) \\
                            &=L(f,\calQ_1)+L(f,\calQ_2) \\
                            &>\int_a^cf(x)\text{d}x-\frac{\epsilon}{2}+\int_c^bf(x)\text{d}x
                                                                      -\frac{\epsilon}{2} \\
                            &=\int_a^cf(x)\text{d}x+\int_c^bf(x)\text{d}x-\epsilon,
    \end{align*}
    and hence,
    $$\bigg|\int_a^bf(x)\text{d}x-\bigg(\int_a^cf(x)\text{d}x+\int_c^bf(x)\text{d}x\bigg)\bigg|
                                                                                <\epsilon,$$
    so
    $$\int_a^bf(x)\text{d}x=\int_a^cf(x)\text{d}x+\int_c^bf(x)\text{d}x.$$
  \end{proof}

  \begin{lemma}
    Let $A\subset\R^n$ and $f,g:A\to\R$ be bounded. Then
    \begin{enumerate}
      \item $\sup_{x\in A}{-f(x)}=-\inf_{x\in A}{f(x)}$
      \item $\inf_{x\in A}{-f(x)}=-\sup_{x\in A}{f(x)}$
      \item $|\sup_{x\in A}f(x)|\leq\sup_{x\in A}|f(x)|$
      \item $|\inf_{x\in A}f(x)|\leq\inf_{x\in A}|f(x)|$
      \item $\sup_{x\in A}{f(x)+g(x)}\leq\sup_{x\in A}{f(x)}+\sup_{x\in A}{g(x)}$
      \item $\inf_{x\in A}{f(x)+g(x)}\geq\inf_{x\in A}{f(x)}+\inf_{x\in A}{g(x)}$
    \end{enumerate}
  \end{lemma}

  \begin{lemma}
    Let $A\subset\R^n$, and $f:A\to\R$ be bounded. Then
    $$\sup_{x\in A}{|f(x)|}-\inf_{x\in A}{|f(x)|}\leq\sup_{x\in A}{f(x)}-\inf_{x\in A}{f(x)}.$$
  \end{lemma}

  %\begin{proof}
  %  \verb%%
%
  %  \textit{Case 1.} $$\sup_{x\in A}f(x)\leq0.$$
  %    Then $f(x)\leq0\,\forall x\in A$, so $f(x)=-|f(x)|$.
  %
  %    So $\sup_{x\in A}{f(x)}-\inf_{x\in A}{f(x)}
  %    =\sup_{x\in A}{-|f(x)|}-\inf_{x\in A}{-|f(x)|}=-\inf_{x\in A}{|f(x)|}+\sup_{x\in A}{|f(x)|}$,
  %    and hence we have equality.
%
  %  \textit{Case 2.} $$\sup_{x\in A}f(x)\geq0,\,\inf_{x\in A}f(x)\geq0.$$
  %    Then again we have equality trivially.
%
  %  \textit{Case 3.} $$\sup_{x\in A}f(x)>0,\,\inf_{x\in A}f(x)<0.$$
  %    Then $\sup_{x\in A}f(x)=|\sup_{x\in A}f(x)|\leq\sup_{x\in A}|f(x)|$,
  %    $\inf_{x\in A}f(x)=-|\inf_{x\in A}f(x)|\geq-\inf_{x\in A}{|f(x)|}$.
  %\end{proof}

  \begin{lemma}
    Let $f:[a,b]\to\R$ be an integrable function. Then $\eps,\del$ such that
    $U(f,\calP)-L(f,\calP)<\epsilon$ whenever $\text{mesh}(\calP)<\delta$.
  \end{lemma}

  %\begin{proof}
  %  Fix $\epsilon>0$. Then, since $f$ is integrable, it is bounded, so $\exists M\geq0$ such that
  %  $|f(x)|\leq M,\,\forall x\in[a,b]$. Also, $\exists$ partition $\calQ:=(x_0=a,x_1,\dots,x_{k-1},x_k=b)$
  %  such that
  %  $$U(f,\calQ)-L(f,\calQ)<\frac{\epsilon}{2}.$$
%
  %  Let $\calP:=(x_0=a,x_1,\dots,x_{l-1},x_l=b)$ be a partition of $[a,b]$ with $\text{mesh}(\calP)
  %  <\delta:=\frac{\epsilon}{5kM}$.
%
  %  Then $\forall j=0,\dots,k-1$, we define the sets
  %  $$A_j:=\{0\leq i\leq l-1:[x_i]\}$$
  %\end{proof}

  \begin{definition}[Riemann sum]
    Given an integrable function $f:[a,b]\to\R$ and a \textit{tagged partition},
    $(\calP,\tau)$ where
    $$\calP=(x_0,\dots,x_k),$$
    $$\tau=(t_0,\dots,t_{k-1})$$ with
    $t_j\in[x_j,x_{j+1}]\,\forall j\in\{0,\dots,k-1\}$, of $[a,b]$, we define a \textit{Riemann sum} by
    $$R(f,\calP,\tau)=\sum_{j=0}^{k-1}{f(t_j)\Delta x_j}.$$
  \end{definition}

  \begin{theorem}
    Let $f:[a,b]\to\R$ be an integrable function, and $\big((\calP_i,\tau_i)\big)_{i\in\N}$ be sequence of
    tagged partitions of $[a,b]$ such that $\text{mesh}(\calP_i)\to 0$ as $i\to\infty$. Then
    $$R(f,\calP_i,\tau_i)\to\int_a^bf(x)\text{d}x,$$
    as $i\to\infty$.
  \end{theorem}

  \begin{proof}
    Fix $\varepsilon>0$. Then $\del$ such that $U(f,\calQ)-L(f,\calQ)<\varepsilon$ whenever
    $\text{mesh}(\calQ)<\delta$. Since $\text{mesh}(\calP_i)\to 0$ as $i\to\infty$, $\exists N\in\N$
    such that $|\text{mesh}(\calP_i)|<\delta$ whenever $i\geq N$. So
    $$U(f,\calP_i)-L(f,\calP_i)<\varepsilon\quad\forall i\geq N.$$
    Note that since $(m_j)_i\leq (f(t_j))_i\leq (M_j)_i$, we have $L(f,\calP_i)\leq R(f,\calP_i,\tau_i)\leq U(f,\calP_i)$,
    and in particular
    $$L(f,\calP_i)-\varepsilon<R(f,\calP_i,\tau_i)<U(f,\calP_i)+\varepsilon.$$
    Then since
    $$L(f,\calP_i)\leq\int_a^bf(x)\text{d}x\leq  U(f,\calP_i),$$
    we have
    $$-\varepsilon<R(f,\calP_i,\tau_i)-\int_a^bf(x)\text{d}x<\varepsilon.$$
    Hence
    $$\bigg|R(f,\calP_i,\tau_i)-\int_a^bf(x)\text{d}x\bigg|<\varepsilon.$$
  \end{proof}

  \begin{theorem}[Fundamental theorem of Calculus]
    Let $f:[a,b]\to\R$ be continuous. We define $F:[a,b]\to\R$ by
    $$F(x)=\int_a^xf(t)\text{d}t.$$
    Then $F$ is differentiable on $(a,b)$, and $F'(x)=f(x)$ $\forall x,y\in[a,b]$.
  \end{theorem}

  \begin{proof}
    Fix $\varepsilon>0$. Then since $f$ is continuous on $[a,b]$, it is uniformly continuous so
    $\exists\delta>0$ such that $|f(y)-f(z)|<\varepsilon$ whenever $|y-z|<\delta$. Now fix
    $x\in(a,b)$. Then $\forall y\in[a,b]$ such that $0<|y-x|<\delta$, we have:
    \begin{align*}
      \bigg|\frac{F(y)-F(x)}{y-x}-f(x)\bigg|&=\bigg|\frac{1}{y-x}\bigg(\int_a^yf(t)\text{d}t-
      \int_a^xf(t)\text{d}t\bigg)-f(x)\bigg| \\
                                            &=\bigg|\frac{1}{y-x}\int_x^yf(t)\text{d}t-f(x)\bigg| \\
                                            &=\bigg|\frac{1}{y-x}\int_x^yf(t)-f(x)\text{d}t\bigg| \\
                                            &=\frac{1}{|y-x|}\bigg|\int_x^yf(t)-f(x)\text{d}t\bigg| \\
                                            &\leq\sup_{t\in[x,y]}|f(t)-f(x)| \\
                                            &=\max_{t\in[x,y]}|f(t)-f(x)|\text{ by the extreme value theorem } \\
                                            &<\varepsilon,
    \end{align*}
    Since $|t-x|<\delta$.
  \end{proof}

\subsection{Differentiation}
\label{sec:1.5}