\subsection{Concavity and Inflection Points}\label{sec:Concavity}
We know that the sign of the derivative tells us whether a function is
increasing or decreasing; for example, when $f'(x)>0$,
$f(x)$ is increasing. The sign of the second derivative
$f''(x)$ tells us whether $f'$ is increasing or decreasing; we have
seen that if $f'$ is zero and increasing at a point then there is a
local minimum at the point. If $f'$ is zero and decreasing at a
point then there is a local maximum at the point. Thus, we extracted
information about $f$ from information about $f''$.

We can get information from the sign of $f''$ even when $f'$ is not
zero. Suppose that $f''(a)>0$. This means that near $x=a$, $f'$ is
increasing. If $f'(a)>0$, this means that $f$ slopes up and is getting
steeper; if $f'(a)<0$, this means that $f$ slopes down and is getting
{\it less} steep. The two situations are shown in
figure~\ref{fig:concave up}. A curve that is shaped like this is
called \dfont{concave up}.

\figure[H]
\centerline{\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <2truecm,2truecm>
\setplotarea x from 0 to 1, y from 0 to 1
\axis left shiftedto x=0 /
\axis bottom shiftedto y=0 ticks withvalues {$a$} / at 0.5 / /
\setquadratic
\plot 0.1 0.2 0.5 0.4 0.9 0.9 /
\setcoordinatesystem units <2truecm,2truecm> point at -2 0
\setplotarea x from 0 to 1, y from 0 to 1
\axis left shiftedto x=0 /
\axis bottom shiftedto y=0 ticks withvalues {$a$} / at 0.5 / /
\setquadratic
\plot 0.1 0.9 0.5 0.3 0.9 0.1 /
\endpicture}}
\caption{$f''(a)>0$: $f'(a)$ positive and increasing, $f'(a)$ negative and
  increasing. \label{fig:concave up}}
\endfigure

Now suppose that $f''(a)<0$. This means that near $x=a$, $f'$ is
decreasing. If $f'(a)>0$, this means that $f$ slopes up and is getting
less steep; if $f'(a)<0$, this means that $f$ slopes down and is getting
steeper. The two situations are shown in
figure~\ref{fig:concave down}. A curve that is shaped like this is
called \dfont{concave down}.

\figure[H]
\centerline{\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <2truecm,2truecm>
\setplotarea x from 0 to 1, y from 0 to 1
\axis left shiftedto x=0 /
\axis bottom shiftedto y=0 ticks withvalues {$a$} / at 0.5 / /
\setquadratic
\plot 0.1 0.2 0.5 0.7 0.9 0.9 /
\setcoordinatesystem units <2truecm,2truecm> point at -2 0
\setplotarea x from 0 to 1, y from 0 to 1
\axis left shiftedto x=0 /
\axis bottom shiftedto y=0 ticks withvalues {$a$} / at 0.5 / /
\setquadratic
\plot 0.1 0.9 0.5 0.6 0.9 0.1 /
\endpicture}}
\caption{$f''(a)<0$: $f'(a)$ positive and decreasing, $f'(a)$ negative and
  decreasing. \label{fig:concave down}}
\endfigure

If we are trying to understand the shape of the graph of a function,
knowing where it is concave up and concave down helps us to get a more
accurate picture. Of particular interest are points at which the
concavity changes from up to down or down to up; such points are
called \dfont{inflection points}. If the
concavity changes from up to down at $x=a$, $f''$ changes from
positive to the left of $a$ to negative to the right of $a$, and
usually $f''(a)=0$. We can identify such points by first finding where
$f''(x)$ is zero and then checking to see whether $f''(x)$ does in
fact go from positive to negative or negative to positive at these
points. Note that it is possible that $f''(a)=0$ but the concavity is
the same on both sides; $\ds f(x)=x^4$ at $x=0$ is an example.

\begin{example}{Concavity}{concavity}
Describe the concavity of $\ds f(x)=x^3-x$.
\end{example}
\begin{solution}
The derivatives are $\ds f'(x)=3x^2-1$ and $f''(x)=6x$.
Since $f''(0)=0$, there is potentially an inflection point at
zero. Since $f''(x)>0$ when $x>0$ and $f''(x)<0$ when $x<0$ the
concavity does change from concave down to concave up at zero, and the curve is
concave down for all $x<0$ and concave up for all $x>0$.
\end{solution}

Note that we need to compute and analyze the second derivative to
understand concavity, so we may as well try to use the second
derivative test for maxima and minima. If for some reason this fails
we can then try one of the other tests.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:Concavity}}

\begin{enumialphparenastyle}

Describe the concavity of the functions below.

%%%%%%%%%%
\begin{ex}
 $\ds y=x^2-x$
\begin{sol}
 concave up everywhere
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=2+3x-x^3$
\begin{sol}
 concave up when $x<0$, concave down when $x>0$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=x^3-9x^2+24x$
\begin{sol}
 concave down when $x<3$, concave up when $x>3$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=x^4-2x^2+3$
\begin{sol}
 concave up when $\ds x<-1/\sqrt3$ or $\ds x>1/\sqrt3$,
concave down when $\ds -1/\sqrt3<x<1/\sqrt3$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=3x^4-4x^3$
\begin{sol}
 concave up when $x<0$ or $x>2/3$,
concave down when $0<x<2/3$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=(x^2-1)/x$
\begin{sol}
 concave up when $x<0$, concave down when $x>0$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y=3x^2-(1/x^2)$
\begin{sol}
 concave up when $x<-1$ or $x>1$, concave down when
$-1<x<0$ or $0<x<1$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $y=\sin x + \cos x$
\begin{sol}
 concave down on $((8n-1)\pi/4,(8n+3)\pi/4)$,
concave up on $((8n+3)\pi/4,(8n+7)\pi/4)$, for integer $n$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = 4x+\sqrt{1-x}$
\begin{sol}
 concave down everywhere
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = (x+1)/\sqrt{5x^2 + 35}$
\begin{sol}
 concave up on $\ds (-\infty,(21-\sqrt{497})/4)$ and
$\ds (21+\sqrt{497})/4,\infty)$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y= x^5 - x$
\begin{sol}
 concave up on $(0,\infty)$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = 6x + \sin 3x$
\begin{sol}
 concave down on $(2n\pi/3,(2n+1)\pi/3)$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = x+ 1/x$
\begin{sol}
 concave up on $(0,\infty)$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = x^2+ 1/x$
\begin{sol}
 concave up on $(-\infty,-1)$ and $(0,\infty)$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = (x+5)^{1/4}$
\begin{sol}
 concave down everywhere
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = \tan^2 x$
\begin{sol}
 concave up everywhere
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y =\cos^2 x - \sin^2 x$
\begin{sol}
 concave up on $(\pi/4+n\pi,3\pi/4+n\pi)$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 $\ds y = \sin^3 x$
\begin{sol}
 inflection points at $n\pi$, $\ds \pm\arcsin(\sqrt{2/3})+n\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Identify the intervals on which the graph of the function
$\ds f(x) = x^4-4x^3 +10$ is of one of these four
shapes: concave up and increasing; concave up and decreasing; concave
down and increasing; concave down and decreasing.
\begin{sol}
 up/incr: $(3,\infty)$, up/decr: $(-\infty,0)$, $(2,3)$,
down/decr: $(0,2)$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
 Describe the concavity of $\ds y =  x^3 + bx^2 + cx + d$.
You will need to consider different cases, depending on the values of
the coefficients.
\end{ex}

%%%%%%%%%%
\begin{ex}
 Let $n$ be an integer greater than or equal to
two, and suppose $f$ is a polynomial of degree $n$. How many inflection points
can $f$ have?  Hint: Use the second derivative test and the
fundamental theorem of algebra.
\end{ex}

\end{enumialphparenastyle}


\subsubsection{Alternative Derivation Using Bayes' Rule}\label{sec:set_bayes}
In the measure-theoretic approach studied in~\cite{BBE11, BET+14}, Voronoi-cell discretizations of $\pspace$ are used to construct set-valued approximations of the updated measure directly, so we refer to it as the \emph{explicit} approach.
By contrast, sampling from densities is an \emph{implicit} approach, and is discussed in greater detail in \ref{sec:ch02-sample}.
Here, we provide a ``set-based'' derivation of the updated measure to more easily compare to the explicit approximation of the solution given measure in~\cite{BET+14}.

First, we start by observing that if $A, B \subset \pspace$ such that $A = \qoi^{-1}(\qoi(B))$, then we have that $B\subset A$ (the inclusion may be proper).
Therefore, for any probability measure $\PP$ on $(\pspace, \pborel)$,
\[
\PP(B) = \PP(B|A) \, \PP(A).
\]
If $\PP$ is intended to solve the inverse problem, then we are motivated to take
\[
\PP(A) = \observedP (\qoi(A)) = \observedP (B),
\]
in the above formula.

We must now determine how to properly define $\PP(B|A)$.
We leverage Bayes' Theorem~\cite{Smith} in order to utilize the prior density on contour events.
In other words, we use the prior (ansatz) measure $\initialP$ extended on $(\pspace, \pborel)$ and Bayes' Theorem to get
\begin{equation}\label{eq:bayes_full}
\PP(B|A) = \initialP(B|A) = \frac{ \initialP(A|B) \initialP(B) }{ \initialP(A) },
\end{equation}
and since $B \subset A$, $\initialP(A|B) = 1$, \eqref{eq:bayes_full} simplifies to

\begin{equation}\label{eq:bayes}
\initialP(B|A) = \frac{ \initialP(B) }{ \initialP(A) }.
\end{equation}

Recall from \eqref{eq:predicted} that $\predictedP$ is the push-forward of the initial measure, giving $\initialP(A) = \predictedP (\qoi(A)) = \predictedP \left (\qoi(B)\right )$, which then gives the following set-valued ``solution'' to the stochastic inverse problem:
\begin{equation}\label{eq:sip_sol_cont}
\updatedP(B) := \begin{cases}
\initialP(B) \frac{ \observedP(B) }{ \predictedP \left (\qoi(B)\right ) ) } & \text{ if } \initialP(B) > 0,\\
0 & \text{ otherwise}.
\end{cases}
\end{equation}

This set-valued update is only a solution on certain (sub-)$\sigma$-algebras of $\pborel$ for which $B$ must belong to apply \eqref{eq:sip_sol_cont}.
Nonetheless, we can form explicit approximations to this measure, e.g. as done in~\cite{BET+14, BES12, BBE11}.
In other words, an Ansatz is used in place of the prior; it serves the same purpose to distribute probabilities in directions not informed by the QoI map.

% However, such an explicit approach requires an approximation of \emph{events in $\pborel$}.
% This is in direct contrast to the numerical approximation of the density $\updated$ that only requires approximation of $\predicted$.
% Since we often expect the dimension of $\dspace$ to be less than the dimension of $\pspace$, this can prove to be a significant numerical advantage for the ``implicit'' approximation given by the updated probability density function.


\chapter{General tridiagonal linear systems}

We consider solving an $n\times n$ linear system
$\boldsymbol{A}\boldsymbol{x}=\boldsymbol{b}$ where the matrix~$\boldsymbol{A}$
is tridiagonal but not necessarily symmetric or positive-definite.  For concreteness,
suppose at first that $n=5$ and write
\[
\boldsymbol{A}=\begin{bmatrix}
\alpha_1&\gamma_1&        &        &\\
 \beta_1&\alpha_2&\gamma_2&        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix}
\]
We will reduce $\boldsymbol{A}$ to upper triangular form by Gaussian
elimination.

If $|\beta_1|\le|\alpha_1|$ then we define the \emph{Gauss transformation}
\[
\boldsymbol{M}_1=\begin{bmatrix}
     1& & & &\\
-\ell_1&1& & &\\
      & &1& &\\
      & & &1&\\
      & & & &1
\end{bmatrix}
\quad\text{where}\quad\ell_1=\frac{\beta_1}{\alpha_1},
\]
in order to eliminate the $21$-entry~$\beta_1$:
\[
\boldsymbol{M}_1\boldsymbol{A}=
\begin{bmatrix}
      1& & & &\\
-\ell_1&1& & &\\
       & &1& &\\
       & & &1&\\
       & & & &1
\end{bmatrix}
\begin{bmatrix}
\alpha_1&\gamma_1&        &        &\\
 \beta_1&\alpha_2&\gamma_2&        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix}
=\begin{bmatrix}
\alpha_1& \gamma_1&        &        &\\
        &\alpha_2'&\gamma_2&        &\\
        &  \beta_2&\alpha_3&\gamma_3&\\
        &         & \beta_3&\alpha_4&\gamma_4\\
        &         &        & \beta_4&\alpha_5
\end{bmatrix}
\]
where
\[
\alpha_2'=\alpha_2-\ell_1\gamma_1.
\]

However, if $|\beta_1|>|\alpha_1|$ then we swap the first two rows to move
$\beta_1$ to the pivot position.  To perform this operation, we swap the first
two rows of the $5\times5$ identity matrix to obtain a \emph{permutation matrix}
\[
\boldsymbol{P}_1=\begin{bmatrix}
 &1& & &\\
1& & & &\\
 & &1& &\\
 & & &1&\\
 & & & &1
\end{bmatrix},
\]
and observe that
\[
\boldsymbol{P}_1\boldsymbol{A}=
\begin{bmatrix}
 &1& & &\\
1& & & &\\
 & &1& &\\
 & & &1&\\
 & & & &1
\end{bmatrix}
\begin{bmatrix}
\alpha_1&\gamma_1&        &        &\\
 \beta_1&\alpha_2&\gamma_2&        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix}
=\begin{bmatrix}
 \beta_1&\alpha_2&\gamma_2&        &\\
\alpha_1&\gamma_1&        &        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix}.
\]
We let
\[
\ell_1=\frac{\alpha_1}{\beta_1},\quad
\alpha_1'=\beta_1,\quad\gamma_1'=\alpha_2,\quad\delta_1=\gamma_2,\quad
\alpha_2'=\gamma_1-\ell_1\alpha_2,\quad\gamma_2'=-\ell_1\gamma_2,
\]
so that
\[
\boldsymbol{M}_1\boldsymbol{P_1}\boldsymbol{A}
=\begin{bmatrix}
      1& & & &\\
-\ell_1&1& & &\\
       & &1& &\\
       & & &1&\\
       & & & &1
\end{bmatrix}
\begin{bmatrix}
 \beta_1&\alpha_2&\gamma_2&        &\\
\alpha_1&\gamma_1&        &        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix}
=\begin{bmatrix}
\alpha_1'&\gamma_1'& \delta_1&        &\\
         &\alpha_2'&\gamma_2'&        &\\
         & \beta_2 &\alpha_3 &\gamma_3&\\
         &         & \beta_3 &\alpha_4&\gamma_4\\
         &         &         & \beta_4&\alpha_5
\end{bmatrix}.
\]

For both of the above cases, performing the computations in place will result
in a matrix
\[
\boldsymbol{A}_1=
\begin{bmatrix}
\alpha_1&\gamma_1&\delta_1&        &\\
        &\alpha_2&\gamma_2&        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix},
\]
with~$\delta_1=0$ in the former case.  If $|\beta_2|\le|\alpha_2|$, then we put
\[
\boldsymbol{M}_2=\begin{bmatrix}
1&       & & &\\
 &      1& & &\\
 &-\ell_2&1& &\\
 &       & &1&\\
 &       & & &1
\end{bmatrix}\quad\text{where}\quad\ell_2=\frac{\beta_2}{\alpha_2}
\]
so that
\[
\boldsymbol{M}_2\boldsymbol{A}_1=
\boldsymbol{M}_2=\begin{bmatrix}
1&       & & &\\
 &      1& & &\\
 &-\ell_2&1& &\\
 &       & &1&\\
 &       & & &1
\end{bmatrix}
\begin{bmatrix}
\alpha_1&\gamma_1&\delta_1&        &\\
        &\alpha_2&\gamma_2&        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix}=
\begin{bmatrix}
\alpha_1&\gamma_1& \delta_1&        &\\
        &\alpha_2& \gamma_2&        &\\
        &        &\alpha_3'&\gamma_3&\\
        &        &  \beta_3&\alpha_4&\gamma_4\\
        &        &         & \beta_4&\alpha_5
\end{bmatrix},
\]
where
\[
\alpha_3'=\alpha_3-\ell_2\gamma_2.
\]
If $|\beta_2|>|\alpha_2|$ then we swap rows 2~and 3 to move $\beta_2$ to the
pivot position, letting
\[
\boldsymbol{P}_2=\begin{bmatrix}
1& & & &\\
 & &1& &\\
 &1& & &\\
 & & &1&\\
 & & & &1
\end{bmatrix}
\]
so that
\[
\boldsymbol{P}_2\boldsymbol{A}_1
=\begin{bmatrix}
1& & & &\\
 & &1& &\\
 &1& & &\\
 & & &1&\\
 & & & &1
\end{bmatrix}
\begin{bmatrix}
\alpha_1&\gamma_1&\delta_1&        &\\
        &\alpha_2&\gamma_2&        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix}
=\begin{bmatrix}
\alpha_1&\gamma_1&\delta_1&        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &\alpha_2&\gamma_2&        &\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix}
\]
With
\[
\ell_2=\frac{\alpha_2}{\beta_2},\quad\alpha_2'=\beta_2,\quad
\gamma_2'=\alpha_3,\quad\delta_2=\gamma_3,\quad
\alpha_3'=\gamma_2-\ell_2\alpha_3,\quad\gamma_3'=-\ell_2\gamma_3,
\]
we obtain
\[
\boldsymbol{M}_2\boldsymbol{P}_2\boldsymbol{A}_1
=\begin{bmatrix}
1&       & & &\\
 &      1& & &\\
 &-\ell_2&1& &\\
 &       & &1&\\
 &       & & &1
\end{bmatrix}
\begin{bmatrix}
\alpha_1&\gamma_1&\delta_1&        &\\
        & \beta_2&\alpha_3&\gamma_3&\\
        &\alpha_2&\gamma_2&        &\\
        &        & \beta_3&\alpha_4&\gamma_4\\
        &        &        & \beta_4&\alpha_5
\end{bmatrix}
=\begin{bmatrix}
\alpha_1& \gamma_1& \delta_1&         &\\
        &\alpha_2'&\gamma_2'&\delta_2 &\\
        &         &\alpha_3'&\gamma_3'&\\
        &         &  \beta_3&\alpha_4 &\gamma_4\\
        &         &         & \beta_4 &\alpha_5
\end{bmatrix}.
\]
Continuing in this way, we finish up with an upper triangular matrix
\[
\boldsymbol{U}=
\boldsymbol{M}_4\boldsymbol{P_4}
\boldsymbol{M}_3\boldsymbol{P_3}
\boldsymbol{M}_2\boldsymbol{P_2}
\boldsymbol{M}_1\boldsymbol{P_1}\boldsymbol{A}
=\begin{bmatrix}
\alpha_1&\gamma_1&\delta_1&        &\\
        &\alpha_2&\gamma_2&\delta_2&\\
        &        &\alpha_3&\gamma_3&\delta_3\\
        &        &        &\alpha_4&\gamma_4\\
        &        &        &        &\alpha_5
\end{bmatrix},
\]
where each $\boldsymbol{P}_k$ is either the $5\times5$ identity matrix, or else
the permutation matrix that results from swapping rows $k$~and $k+1$.

\begin{algorithm}
\caption{Compute the $LU$ factorization of a (possibly non-symmetric)
tridiagonal matrix.}
\label{alg: tridiagonal LU}
\begin{algorithmic}
\Require $\boldsymbol{\beta}=[\beta_1,\beta_2,\ldots,\beta_{n-1}]$ is the
sub-diagonal of $\boldsymbol{A}$.
\Require $\boldsymbol{\alpha}=[\alpha_1,\alpha_2,\ldots,\alpha_n]$ is the main
diagonal of $\boldsymbol{A}$.
\Require $\boldsymbol{\gamma}=[\gamma-1,\gamma_2,\ldots,\gamma_{n-1}]$ is the
super-diagonal of $\boldsymbol{A}$.
\Require $\boldsymbol{\delta}=[\delta_1,\delta_2,\ldots,\delta_{n-2}]$ is
storage to hold fill-in of the second super-diagonal.
\Require $\boldsymbol{p}=[p_1,p_2,\ldots,p_{n-1}]$ is an boolean vector to
record which steps use a row swap.
\Statex
\Function{Factorize!}{$\boldsymbol{\beta}, \boldsymbol{\alpha},
\boldsymbol{\gamma}, \boldsymbol{\delta}, \boldsymbol{p}$}
\For{$j=1:n-2$}
	\State $\delta_j\gets0$
	\If{$|\alpha_j|<|\beta_j|$}\Comment{Swap rows $j$ and $j+1$}
		\State $p_j\gets\mathbf{true}$
		\State $\alpha_j\swap\beta_j$
		\State $\gamma_j\swap\alpha_{j+1}$
		\State $\delta_j\swap\gamma_{j+1}$
	\Else
		\State $p_j\gets\mathbf{false}$
	\EndIf
	\State $\beta_j\gets\beta_j/\alpha_j$
	\State $\alpha_{j+1}\gets\alpha_{j+1}-\beta_j\gamma_j$
	\If{$p_j$}
		\State $\gamma_{j+1}\gets-\beta_j\delta_j$
	\EndIf
\EndFor
\If{$|\alpha_{n-1}|<|\beta_{n-1}|$}
	\State $p_{n-1}\gets\mathbf{true}$
	\State $\alpha_{n-1}\swap\beta_{n-1}$
	\State $\gamma_{n-1}\swap\alpha_n$
\Else
	\State $p_{n-1}\gets\mathbf{false}$
\EndIf
\State $\beta_{n-1}\gets\beta_{n-1}/\alpha_{n-1}$
\State $\alpha_n\gets\alpha_n-\beta_{n-1}\gamma_{n-1}$
\EndFunction
\end{algorithmic}
\end{algorithm}

Define the column vectors
\[
\boldsymbol{\ell}_k=\ell_k\boldsymbol{e}_k
\quad\text{for $1\le k\le 4$,}
\]
and observe that
\[
\boldsymbol{M}_k=\boldsymbol{I}-\boldsymbol{\ell}_k\boldsymbol{e}_k^T
\quad\text{and}\quad
\boldsymbol{P_k}^T=\boldsymbol{P}_k
\quad\text{for $1\le k\le 4$,}
\]
with
\[
\boldsymbol{P}_k\boldsymbol{e}_j=\boldsymbol{e}_j
\quad\text{and}\quad
\boldsymbol{e}_j^T=\boldsymbol{e}_j^T\boldsymbol{P}_k
\quad\text{for $1\le j<k\le 4$.}
\]
Thus,
\[
\boldsymbol{P}_4\boldsymbol{M}_3
	=\boldsymbol{P}_4-\boldsymbol{P}_4\boldsymbol{\ell}_3\boldsymbol{e}_3^T
	=\bigl(\boldsymbol{I}-\boldsymbol{P}_4\boldsymbol{\ell}_3\boldsymbol{e}_3^T
	\bigr)\boldsymbol{P}_4
	=\widetilde{\boldsymbol{M}}_3\boldsymbol{P_4},
\]
where
\[
\widetilde{\boldsymbol{M}}_3
	=\boldsymbol{I}-\tilde{\boldsymbol{\ell}}_3\boldsymbol{e}_3^T
\quad\text{and}\quad
\tilde{\boldsymbol{\ell}}_3=\boldsymbol{P}_4\boldsymbol{\ell}_3.
\]
In the same way,
\[
\boldsymbol{P}_4\boldsymbol{P}_3\boldsymbol{M}_2
=\widetilde{\boldsymbol{M}}_2\boldsymbol{P}_4\boldsymbol{P}_3
\quad\text{and}\quad
\boldsymbol{P}_4\boldsymbol{P}_3\boldsymbol{P}_2\boldsymbol{M}_1
=\widetilde{\boldsymbol{M}}_1\boldsymbol{P}_4\boldsymbol{P}_3\boldsymbol{P}_2
\]
where
\[
\widetilde{\boldsymbol{M}}_2
	=\boldsymbol{I}-\tilde{\boldsymbol{\ell}}_2\boldsymbol{e}_2^T
\quad\text{and}\quad
\tilde{\boldsymbol{\ell}}_2=\boldsymbol{P}_4\boldsymbol{P}_3\boldsymbol{\ell}_2,
\]
with
\[
\widetilde{\boldsymbol{M}}_1
	=\boldsymbol{I}-\tilde{\boldsymbol{\ell}}_1\boldsymbol{e}_1^T
\quad\text{and}\quad
\tilde{\boldsymbol{\ell}}_1
	=\boldsymbol{P}_4\boldsymbol{P}_3\boldsymbol{P}_2\boldsymbol{\ell}_2.
\]
Thus,
\[
\boldsymbol{U}=\boldsymbol{M}_4\widetilde{\boldsymbol{M}}_3
\widetilde{\boldsymbol{M}}_2\widetilde{\boldsymbol{M}}_1
\boldsymbol{P}_4\boldsymbol{P}_3\boldsymbol{P}_2\boldsymbol{P}_1
\boldsymbol{A},
\]
and so
\[
\boldsymbol{P}\boldsymbol{A}=\boldsymbol{L}\boldsymbol{U}
\quad\text{where}\quad
\boldsymbol{P}=\boldsymbol{P}_4\boldsymbol{P}_3\boldsymbol{P}_2\boldsymbol{P}_1
\quad\text{and}\quad
\boldsymbol{L}=\widetilde{\boldsymbol{M}}_1^{-1}
\widetilde{\boldsymbol{M}}_2^{-1}\widetilde{\boldsymbol{M}}_3^{-1}
\boldsymbol{M}_4^{-1}.
\]
We have
\[
\bigl(\boldsymbol{I}+\boldsymbol{\ell}_k\boldsymbol{e}_k^T\bigr)
\bigl(\boldsymbol{I}-\boldsymbol{\ell}_k\boldsymbol{e}_k^T\bigr)
=\boldsymbol{I}\quad\text{because $\boldsymbol{e}_k^T\boldsymbol{\ell}_k=0$,}
\]
and likewise when $\boldsymbol{\ell}_k$ is replaced
by~$\tilde{\boldsymbol{\ell}_k}$, showing that
\[
\boldsymbol{M}_k^{-1}=
\boldsymbol{I}+\boldsymbol{\ell}_k\boldsymbol{e}_k^T
\quad\text{and}\quad
\widetilde{\boldsymbol{M}}_k^{-1}=
\boldsymbol{I}+\tilde{\boldsymbol{\ell}}_k\boldsymbol{e}_k^T.
\]
It follows that
\[
\boldsymbol{L}=\boldsymbol{I}+\tilde{\boldsymbol{\ell}}_1\boldsymbol{e}_1
+\tilde{\boldsymbol{\ell}}_2\boldsymbol{e}_2
+\tilde{\boldsymbol{\ell}}_3\boldsymbol{e}_3
+\boldsymbol{\ell}_4\boldsymbol{e}_4,
\]
which is a unit lower triangular matrix.

In practice, to solve the linear system
$\boldsymbol{A}\boldsymbol{x}=\boldsymbol{b}$ we simply compute
\[
\boldsymbol{y}_1=\boldsymbol{M}_1\boldsymbol{P}_1\boldsymbol{b},\qquad
\boldsymbol{y}_2=\boldsymbol{M}_2\boldsymbol{P}_2\boldsymbol{y}_1,\qquad
\boldsymbol{y}_3=\boldsymbol{M}_3\boldsymbol{P}_3\boldsymbol{y}_2,\qquad
\boldsymbol{y}_4=\boldsymbol{M}_4\boldsymbol{P}_4\boldsymbol{y}_3,
\]
and then solve the upper triangular system
\[
\boldsymbol{U}\boldsymbol{x}=\boldsymbol{y}_4.
\]
Note that
\[
(\boldsymbol{M}_k\boldsymbol{z})_j=\begin{cases}
	z_j&\text{if $j\ne k+1$,}\\
	z_{k+1}-\ell_kz_k&\text{if $j=k+1$,}
\end{cases}
\]
so computing each successive $\boldsymbol{y}_k$ costs only one
fused-multiply-and-addition, plus possibly swapping entries $k$~and $k+1$
of~$\boldsymbol{y}$.  Algorithm~\ref{alg: tridiagonal LU} sets out the
computation of the $LU$-factorization, and
Algorithm~\ref{alg: tridiagonal solve} sets out the procedure for then solving
$\boldsymbol{A}\boldsymbol{x}=\boldsymbol{b}$.

\begin{algorithm}
\caption{Solve a tridiagonal linear system
$\boldsymbol{A}\boldsymbol{x}=\boldsymbol{b}$.}
\label{alg: tridiagonal solve}
\begin{algorithmic}
\Require $\boldsymbol{\beta}$, $\boldsymbol{\alpha}$, $\boldsymbol{\gamma}$,
$\boldsymbol{\delta}$, $\boldsymbol{p}$ as provided by
Algorithm~\ref{alg: tridiagonal LU}.
\Require $\boldsymbol{b}=[b_1,b_2,\ldots,b_n]$ is the right-hand side vector.
\Statex
\Function{Solve!}{$\boldsymbol{b}, \boldsymbol{\beta}, \boldsymbol{\alpha},
\boldsymbol{\gamma}, \boldsymbol{\delta}, \boldsymbol{p}$}
\For{$j=1:n-1$}
	\If{$p_j$}
		\State $b_j\swap b_{j+1}$
	\EndIf
	\State $b_{j+1}\gets b_{j+1}-\beta_jb_j$
\EndFor
\State $b_n\gets b_n/\alpha_n$
\State $b_{n-1}\gets(b_{n-1}-\gamma_{n-1}b_n)/\alpha_{n-1}$
\For{$j=n-2:-1:1$}
	\State $b_j\gets(b_j-\gamma_jb_{j+1}-\delta_jb_{j+2})/\alpha_j$
\EndFor
\EndFunction
\end{algorithmic}
\end{algorithm}


%---------------------------Relative Size-----------------------------
\section{Relative Size Squared\label{s:tet-rel-size-squared}}

This metric measures the size of a tetrahedron relative to an ensemble
containing it using volume.
Take $\overline{V}$ to be the average volume of the tetrahedra in the ensemble being analyzed
and define
\[
R = \frac{V}{\overline{V}}
\]
Then the quality is defined as
\begin{equation*}
q =  \left[ \min\left( R, \frac {1}{R}\right) \right]^2.
\end{equation*}

Note that if $\overline{V} < DBL\_MIN$ or if $R \leq DBL\_MIN$, we set $q = 0$.

\tetmetrictable{relative size squared}%
{$1$}%                  Dimension
{$[0.3,1]$}%            Acceptable range
{$[0,1]$}%              Normal range
{$[0,1]$}%              Full range
{N/A}%                  Equilateral tet
{\cite{knu:03}}%        Citation
{v\_tet\_relative\_size\_squared}%                            Verdict function name


\subsection{Kronecker delta}

Consider matrix multiplication \(AI\).

We have:

\(AI_{ik}=A_{ij}I_{jk}\)

We write this instead as:

\(AI_{ik}=A_{ij}\delta_{jk}\)

Where \(\delta_{jk}=0\) if \(j\ne k\) and \(\delta_{jk}=1\) if \(j=k\).


\subsection{Definition}

A product is a repeated multiplication of a sequence.

\(p_n=\prod_{i=0}^ns_i\)

\subsection{Multiplication of products}

We can take constants out of the product.

\(p_n=\prod_{i=0}^nca_i\)

\(p_n=a^{n}\sum_{i=0}^na_i\)

\subsection{Products of constants}

If \(a_i=c\) then the summation is then of the form:

\(p_n=\prod_{i=0}^nc\)

\(p_n=c^{n}\prod_{i=j}^n1\)

\(p_n=c^{n}\)

\subsection{Combining products}

If a sequence is the product of to other sequences then the product of the sequence is equal to the product of the two individual sequences.

\(p_n=\prod_{i=0}^na_i\)

\(p_n=\prod_{i=0}^nb_ic_i\)

\(p_n=\prod_{i=0}^nb_i\prod_{i=0}^nc_i\)


% This LaTeX was auto-generated from an M-file by MATLAB.
% To make changes, update the M-file and republish this document.

%%% \documentclass{article}
%%% \usepackage{graphicx}
%%% \usepackage{color}

%%% \sloppy
%%% \definecolor{lightgray}{gray}{0.5}
\setlength{\parindent}{0pt}

%%% \begin{document}

\subsection*{Windowed Discrete Fourier Transform}

\begin{par}
Example for algorithm SP-WFFT.
\end{par} \vspace{1em}
\begin{par}
Windowed Discrete Fourier Transform is method to calculate frequency and phase spectrum. Example shows the effect of coherent sampling, non-coherent sampling, windowing and zero padding on estimating amplitude and phase by means of FFT. Last part compares all windows in frequency space.
\end{par} \vspace{1em}

\subsubsection*{Contents}

\begin{itemize}
\setlength{\itemsep}{-1ex}
   \item Generate sample data
   \item Call algorithm
   \item Display results
   \item Compare window coefficients
\end{itemize}

\subsubsection*{Generate sample data}

\begin{par}
Construct 1 second of signal sampled at 50 Hz containing two harmonic components at 1 and 8 Hz and one interharmonic component at 15.5 Hz with various amplitudes and phases.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
clear DI
DI.fs.v = 50;
fnom = [1;   8; 15.5];
Anom = [1; 0.5;  0.3];
pnom = [0;   1;    2];
DI.t.v = [0 : 1/DI.fs.v : 1 - 1/DI.fs.v];
DI.y.v = zeros(size(DI.t.v));
for i = 1:length(fnom)
        DI.y.v = DI.y.v + Anom(i).*sin(2.*pi.*fnom(i).*DI.t.v + pnom(i));
end
%
\end{lstlisting}

\subsubsection*{Call algorithm}

\begin{par}
Calculate amplitude and phase spectrum and store results into \lstinline{DO}. Window function is not specified, therefore rectangle (none) window will be used.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
DO = qwtb('SP-WFFT', DI);
%
% Set window function to blackman and calculate windowed amplitude and phase spectrum and store results into |DOw|.
DI.window.v = 'blackman';
DOw = qwtb('SP-WFFT', DI);
%
% Set zero padding to 10 times the signal length and calculate zero padded windowed amplitude and
% phase spectrum and stre results into |DOwz|.
DI.fft_padding.v = 10.*length(DI.y.v);
DOwz = qwtb('SP-WFFT', DI);
%
\end{lstlisting}

        \begin{lstlisting}[style=output]
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
\end{lstlisting} \color{black}

\subsubsection*{Display results}

\begin{par}
Plot amplitude spectrum.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
figure; hold on;
plot(fnom, Anom, '+r', 'markersize', 10, 'linewidth', 2);
stem(DO.f.v, DO.A.v, '-ob', 'filled', 'markersize', 3);
plot(DOw.f.v, DOw.A.v, '-g','linewidth',2);
plot(DOwz.f.v, DOwz.A.v, '-k');
legend('nominal values','FFT', 'blackman window', 'blck. w. + zero padding')
title('Amplitude spectrum')
xlabel('frequency (Hz)'); ylabel('amplitude (V)');
hold off
% Plot phase spectrum
figure; hold on;
plot(fnom, pnom, '+r', 'markersize', 10, 'linewidth', 2);
stem(DO.f.v, DO.ph.v, '-ob', 'filled', 'markersize', 3);
plot(DOw.f.v, DOw.ph.v, '-g','linewidth',2);
plot(DOwz.f.v, DOwz.ph.v, '-k');
legend('nominal values','FFT', 'blackman window', 'blck. w. + zero padding', 'location', 'southwest')
title('Phase spectrum');
xlabel('frequency (Hz)'); ylabel('phase (rad)');
hold off
%
\end{lstlisting}

\begin{center}
\includegraphics[width=0.7\textwidth]{algs_examples_published/SP-WFFT_alg_example_01.pdf}
\end{center}

\begin{center}
\includegraphics[width=0.7\textwidth]{algs_examples_published/SP-WFFT_alg_example_02.pdf}
\end{center}

\subsubsection*{Compare window coefficients}

\begin{par}
Different windows has different peak widths, heights of side lobes and side lobes roll off ratio. To see the window differences a zero padding is used. First a simple signal of ones is prepared and zero padding is specified to 100x the length of signal. Next for every window a spectrum is calculated and plotted.
\end{par} \vspace{1em}
\begin{lstlisting}[style=mcode]
clear DI
DI.y.v = ones(1,10);
DI.fs.v = 1;
DI.fft_padding.v = 100.*length(DI.y.v);
avail_windows = {'barthann' 'bartlett' 'blackman' 'blackmanharris' 'blackmannuttall' 'bohman' 'cheb' 'flattop_matlab' 'flattop_SFT3F' 'flattop_SFT4F' 'flattop_SFT5F' 'flattop_SFT3M' 'flattop_SFT4M' 'flattop_SFT5M' 'flattop_248D' 'gaussian' 'hamming' 'hanning' 'kaiser' 'nuttall' 'parzen' 'rect' 'triang' 'tukey' 'welch'};
col = jet(length(avail_windows));
figure; hold on;
for i = 1:length(avail_windows)
        DI.window.v = avail_windows{i};
        DO = qwtb('SP-WFFT', DI);
        plot(DO.A.v, '-', 'color', col(i,:));
end
h=legend(avail_windows);
h=legend('location','eastoutside');
set(h,'FontSize',8, 'interpreter','none');
xlabel('FFT bin * 100');
ylabel('amplitude');
hold off;
\end{lstlisting}

        \begin{lstlisting}[style=output]
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
QWTB: no uncertainty calculation
\end{lstlisting} \color{black}

\begin{center}
\includegraphics[width=0.7\textwidth]{algs_examples_published/SP-WFFT_alg_example_03.pdf}
\end{center}

%%% \end{document}


\subsection{Hard-margin SVC}

\subsubsection{Linear separators}

We want to create a hyperplane to separate classes.

For classification problem (x, y)

Hyperplane is wx-b=0

\subsubsection{Hard margin}

If data is linearly separable then a hyperplane exists such that all data can be correctly classified

There are an infinit number that could work.

We select two parallel with distance between as large as possible. the region between these two is the margin

The maximum margin hyperplane is the one between the two margin planes

We can rescale the two hyperplanes to:

wx-b=1

wx-b=-1

The distance between the two parrallel hyperplanes is \(\dfrac{2}{||w||}\)

So we minimise \(||w||\) conditional an all points being correctly classified

\(y_i(wx_i-b)\ge 1\)

We select w and b to solve this.


\section{Regression}

Solve $w^* = \arg\min\limits_w \hat{R}(w)+ \lambda C(w)$, \highlight*{$y=w^\top x$}

\textbf{residual:} \highlight*{$r_i = y_i - w^\top x_i$}, \enskip
\textbf{cost:} $\hat R(w)$

\emph{Linear Regression}

%$\hat{R}(w) = \sum_{i=1}^n (y_i - w^\top x_i)^2 = ||Xw-y||^2_2$
$\hat{R}(w) = \sum_{i=1}^n r_i^2 = \norm{Xw-y}^2_2$

$\nabla\!_w \hat{R}(w) = -2 \sum_{i=1}^n r_i \cdot x_i$

\textbf{closed form:} $w^* = (X^\top X)^{-1} X^\top y$

\emph{Ridge regression}

%$\hat{R}(w) = \sum_{i=1}^n (y_i - w^\top x_i)^2 + \lambda ||w||_2^2$
$\hat{R}(w) = \sum_{i=1}^n r_i^2 + \lambda \norm{w}_2^2$

$\nabla\!_w \hat{R}(w) = -2 \sum_{i=1}^n r_i \cdot x_i + 2 \lambda w$

\textbf{closed form:} $w^*=(X^\top X + \lambda I)^{-1} X^\top y$

\emph{L1-regularized regression (Lasso)}\\
$\hat{R}(w) = \sum_{i=1}^n r_i^2 + \lambda \norm{w}_1$


\subsection{The Cauchy-Schwarz and triangle inequalities}

The \textbf{Cauchy-Schwarz inequality}%
\index{Cauchy-Schwarz inequality!for dot product} is a fundamental
inequality satisfied by the dot product.  It is given in the following
proposition.

\begin{proposition}{Cauchy-Schwarz inequality}{cauchy-schwarz-inequality}
  The dot product satisfies the inequality
  \begin{equation}\label{cauchy}
    \abs{\vect{u}\dotprod \vect{v}}\leq \norm{\vect{u}} \norm{\vect{v}}
  \end{equation}
  Furthermore equality is obtained if and only if one of $\vect{u}$ or $\vect{v}$ is a scalar multiple of the other.
\end{proposition}

\begin{proof}
  First note that if $\vect{u}=\vect{0}$, then both sides of
  \eqref{cauchy} are equal to zero, and so the inequality holds in
  this case. Therefore, we will assume in what follows that
  $\vect{u}\neq \vect{0}$.  Define a function of $t\in \R$ by
  \begin{equation*}
    f(t) =(t\vect{u}+\vect{v}) \dotprod (t\vect{u}+\vect{v}).
  \end{equation*}
  Then by Proposition~\ref{prop:properties-dot-product}, $f(t) \geq 0$
  for all $t\in \R$.  Also from
  Proposition~\ref{prop:properties-dot-product}, we have
  \begin{eqnarray*}
    f(t)
    &=& t\vect{u}\dotprod (t\vect{u}+\vect{v})
        + \vect{v}\dotprod (t\vect{u}+\vect{v}) \\
    &=& t^2\vect{u}\dotprod \vect{u}+t(\vect{u}\dotprod \vect{v})
        + t \vect{v}\dotprod \vect{u}
        + \vect{v}\dotprod \vect{v} \\
    &=& t^2\norm{\vect{u}}^2
        + 2t(\vect{u}\dotprod \vect{v})
        + \norm{\vect{v}}^2.
  \end{eqnarray*}
  This means the graph of $y=f(t)$ is a parabola which opens upwards
  and is never negative. It follows that this function has at most one
  root. From the quadratic formula, we know that a quadratic function
  $at^2+bt+c$ has one or zero roots if and only if $b^2-4ac\leq
  0$. Applying this reasoning to the function $f(t)$, we obtain
  \begin{equation*}
    (2(\vect{u}\dotprod\vect{v}))^2-4\norm{\vect{u}}^2\norm{\vect{v}}^2
    \leq 0,
  \end{equation*}
  which is equivalent to
  $\abs{\vect{u}\dotprod \vect{v}} \leq \norm{\vect{u}} \norm{\vect{v}}$.
\end{proof}

An important consequence of the Cauchy-Schwarz inequality is the
so-called \textbf{triangle inequality}%
\index{triangle inequality!in Rn@in $\R^n$}, which states that the length
of one side of a triangle is less than or equal the sum of the lengths
of the two other sides.

\begin{proposition}{Triangle inequality}{triangle-inequality-dot-product}
  For\/ $\vect{u},\vect{v}\in \R^n$, we have
  \begin{equation}\label{triangle-ineq-1}
    \norm{\vect{u} + \vect{v}} \leq \norm{\vect{u}} + \norm{\vect{v}}.
  \end{equation}

  \begin{center}
    \begin{tikzpicture}[scale=1.5]
      \draw[->, thick, purple](0,0) -- node[above left]{$\vect{u}+\vect{v}$}(1.5,1);
      \draw[->, thick, blue](0,0) -- node[below]{$\vect{u}$} (0.75,0);
      \draw[->, thick, red](0.75,0) -- node[below right]{$\vect{v}$} (1.5,1);
    \end{tikzpicture}
  \end{center}
\end{proposition}

\begin{proof}
  By properties of the dot product and the Cauchy-Schwarz inequality,
  we have
  \begin{eqnarray*}
    \norm{\vect{u}+\vect{v}}^2
    &=& (\vect{u}+\vect{v})\dotprod(\vect{u}+\vect{v}) \\
    &=& (\vect{u}\dotprod\vect{u})+(\vect{u}\dotprod\vect{v})+(\vect{v}\dotprod\vect{u})+(\vect{v}\dotprod\vect{v}) \\
    &=& \norm{\vect{u}}^2+2(\vect{u}\dotprod\vect{v})+\norm{\vect{v}}^2 \\
    &\leq& \norm{\vect{u}}^2+2\abs{\vect{u}\dotprod\vect{v}}+\norm{\vect{v}}^2 \\
    &\leq& \norm{\vect{u}}^2+2\norm{\vect{u}}\norm{\vect{v}}+\norm{\vect{v}}^2 \\
    &=& (\norm{\vect{u}}+\norm{\vect{v}})^2.
  \end{eqnarray*}
  Therefore,
  \begin{equation*}
    \norm{\vect{u}+\vect{v}}^2\leq(\norm{\vect{u}}+\norm{\vect{v}})^2.
  \end{equation*}
  Taking square roots of both sides, we obtain \eqref{triangle-ineq-1}.
\end{proof}

\begin{example}{Triangle inequality}{triangle-inequality}
  Use the triangle inequality to show
  \begin{equation*}
    \norm{\vect{u}} -\norm{\vect{v}} \leq \norm{\vect{u}-\vect{v}}
  \end{equation*}
  holds for all vectors $\vect{u},\vect{v}\in\R^n$.
\end{example}

\begin{solution}
  We have
  \begin{equation*}
    \norm{\vect{u}}
    = \norm{(\vect{u}-\vect{v})+\vect{v}} \\
    \leq \norm{\vect{u}-\vect{v}}+\norm{\vect{v}},
  \end{equation*}
  where we have used the triangle inequality in the last step. Note
  that this is an inequality between real numbers. Bringing
  $\norm{\vect{v}}$ to the other side of the equation, we have
  \begin{equation*}
    \norm{\vect{u}} - \norm{\vect{v}} \leq \norm{\vect{u}-\vect{v}}.
  \end{equation*}
\end{solution}


\section{Left \& Right Hand Limits}
Our definition of the limit requires that the function get arbitrarily close to the limit value when approaching from both the left and right hand sides.
However, we can evaluate limits by specifying that we only approach from one side.
We usually notate this with a superscript $+$ or $-$ next to the $x$ limit value.
So,
\begin{equation*}
	\lim_{x \to 0^+}{f(x)}
\end{equation*}
would mean ``the limit of $f(x)$ as $x$ approaches $0$ from the right'', while
\begin{equation*}
	\lim_{x \to 0^-}{f(x)}
\end{equation*}
would mean ``the limit of $f(x)$ as $x$ approaches $0$ from the left.''

Our definition of the limit from both sides requires the left and right sides to be the same.
If they are different, the the limit does not exist.
\begin{align*}
	\lim_{x \to c^+}{f(x)} = \lim_{x \to c^-}{f(x)} &\implies \lim_{x \to c^+}{f(x)} = \lim_{x \to c^-}{f(x)} = \lim_{x \to c}{f(x)} \\
	\lim_{x \to c^+}{f(x)} \neq \lim_{x \to c^-}{f(x)} &\implies \lim_{x \to c}{f(x)} \text{ does not exist (DNE)}.
\end{align*}


\subsection{Dimension function}

For a basis \(S\), the the dimension of the vector space is \(|S|\).

\(\dim (V)=|S|\)

\(S\subset V\)

\subsubsection{Finite and infinite vector spaces}

If \(\dim (V) \) is finite, then we say the vector space is finite.

Otherwise, we say the vector space is infinite.


\documentclass{amsart}

\usepackage{amsmath}
\usepackage{amssymb}

\title{PDE: Assignment 1}
\author{Mark Ditsworth}

\begin{document}
	\maketitle

	\section{Problem 1}
	If $B$ is a positive-definite, Hermitian matrix, show that there is a unique matrix $\sqrt{B}$ that is Hermitian positive-definite and $\left( \sqrt{B} \right) ^2 = B$.
	\\\\
	Let $M$ be the matrix of eigenvectors of $B$. Since $B$ is Hermitian and positive definite, $M$ and it's inverse is Hermitian and positive-definite. Then $B' = M^{-1}BM$, and $B'$ is a diagonal matrix of the eigenvalues of $B$.
	\\\\
	$M\sqrt{B'}M^{-1} = \sqrt{B}$, which is Hermitian and positive-definite.
	\\\\
	Even $M$ is not unique (any $\alpha M$ is a matrix of eigenvectors of $B$), $B'$ is unique since $(\alpha M)^{-1}B\alpha M = \frac{1}{\alpha}\alpha M^{-1}BM$. Therefore, $\sqrt{B}$ is unique.
	\\\\
	\section{Problem 2}
	$A$ and $B$ are Hermitian and $B$ is positive definite.
	\subsection{Part 1}
	Show that $B^{-1}A$ is similar to a Hermitian.
	\\\\
	$C$ is similar to $B^{-1}A \Rightarrow C=MB^{-1}AM^{-1}$ for any invertible $M$. From Problem 1, $\sqrt{B}$ is invertible so let $M=\sqrt{B}$.
	\\\\
	$C = \sqrt{B}B^{-1}A\sqrt{B}^{-1}=B^{-1/2}AB^{-1/2}$, which is Hermitian since $A$ and $\sqrt{B}$ are Hermitian. Therefore, $B^{-1}A$ is similar to a Hermitian.
	\subsection{Part 2}
	What does this say about the eigenvalues of $B^{-1}A$?
	\\\\
	The eigenvalues are real.
	\subsection{Part 3}
	Are the eigenvectors orthogonal?
	\\\\
	No. The eigenvectors of $C$ are orthagonal, but $B^{-1}A$ is not Hermitian. It is \textit{similar} to a Hermitian matrix. So it will not necessarily have orthagonal eigenvectors.

	\subsection{Part 4}
	Verify the answers above in Julia with $5\times 5$ matrices.
	\\\\
	\textit{see notebook.}
	\\\\
	\subsection{Part 5}
	What is special about $C=M^{T}BM$? Show that the elements of $C$ are a kind of dot product of the eigenvectors with a factor of B in the middle.
	\\\\
	\textit{see notebook.}

	\section{Problem 3}
	The solutions of ODE $y'' - 2y' - cy = 0$ take the form $y(t) = C_1 e^{(1+\sqrt{1+c})t} + C_2 e^{(1-\sqrt{1+c})t}$ for some constants $C_1$ and $C_2$ determined by the initial conditions. Suppose that $A$ is a real-symmetric $4 \times 4$ matrix with eigenvalues 3, 8, 15, and 24, which correspond to eigenvectors $\mathbf{x_1}$, $\mathbf{x_2}$, $\mathbf{x_3}$, $\mathbf{x_4}$.
	\subsection{Part 1}
	If $\mathbf{x}(t)$ solves the system of ODEs $\frac{d^2}{dx^2}\mathbf{x} - 2\frac{d}{dx}\mathbf{x} = A\mathbf{x}$, with initial conditions $\mathbf{x}(0) = \mathbf{a}_0$ and $\mathbf{x}'(0) = \mathbf{b}_0$, find the closed-form expression of $\mathbf{x}(t)$ in terms of the eigenvectors and initial conditions.
	\\\\
	Since A is Hermitian and positive-definite, the eigenvectors are orthogonal. Thus, the solution can be expressed as linear combination of the eigenvectors.
	\[\mathbf{x}(t) = \sum_{n=1}^{4} c_n(t) \mathbf{x}_n \]

	\[\frac{d^2}{dx^2}\mathbf{x}(t) - 2\frac{d}{dx}\mathbf{x}(t) - A\mathbf{x}(t) =
	\sum_{n=1}^{4}\left( \ddot{c_n} - 2\dot{c_n}- \lambda_n \right) \mathbf{x}_n = 0\]

	Since $\mathbf{x}_n$ are linearly independent, $\ddot{c_n} - 2\dot{c_n}- \lambda_n = 0$ for all $n$.

	\[ c_n(t) = \alpha_n e^{(1+\sqrt{1+\lambda_n})t} +
	            \beta_n e^{(1-\sqrt{1+\lambda_n})t} \]

	\[ \mathbf{x}(0) = \sum_{n=1}^{4} (\alpha_n + \beta_n)\mathbf{x}_n = \mathbf{a}_0\]

	\[ \alpha_n + \beta_n = \frac{\mathbf{x}_x^* \mathbf{a}_0}{||\mathbf{x}_n||^2}\]

	\[ \mathbf{x}'(0) = \sum_{n=1}^{4} \left( \alpha_n(1+\sqrt{1+\lambda_n}) + \beta_n(1-\sqrt{1+\lambda_n}) \right)\mathbf{x}_n = \mathbf{b}_0 \]

	\[ \mathbf{x}'(0) = \sum_{n=1}^{4}\left( \alpha_n + \beta_n + \sqrt{1+\lambda_n}(\alpha_n - \beta_n) \right) \mathbf{x}_n = \mathbf{b}_0 \]

	\[ \frac{\mathbf{x}_n^* \mathbf{a}_0}{||\mathbf{x}_n||^2} + \sqrt{1+\lambda_n}(\alpha_n - \beta_n) = \frac{\mathbf{x}_n^* \mathbf{b}_0}{||\mathbf{x}_n||^2} \]

	\[ \alpha_n - \beta_n = \frac{\mathbf{x}_n^* (\mathbf{b}_0 - \mathbf{a}_0)}{||\mathbf{x}_n||^2 \sqrt{1+\lambda_n}}\]

	\[ \mathbf{x}(t) = \sum_{n=1}^{4} \left( \left(\mathbf{x}_n^*\mathbf{a}_0 + \frac{\mathbf{x}_n^*(\mathbf{b}_0 - \mathbf{a}_0)}{\sqrt{1+\lambda_n}} \right)e^{(1+\sqrt{1+\lambda_n})t} +
	\left(\mathbf{x}_n^*\mathbf{a}_0 - \frac{\mathbf{x}_n^*(\mathbf{b}_0 - \mathbf{a}_0)}{\sqrt{1+\lambda_n}} \right)e^{(1-\sqrt{1+\lambda_n})t}
	\right) \frac{\mathbf{x}_n}{2||\mathbf{x}_n||^2} \]

	\subsection{Part 2}
	After a long time ($t >> 0$), what is the expected approximation of the solution?
	\\\\
	$\mathbf{x}(t)$ will be dominated by the fastest growing term in the mode with the largest eigenvalue ($\lambda_4 = 24$).

	\[ \mathbf{x}(t>>0) \cong \left( \mathbf{x}_4^* \mathbf{a}_0 + \frac{\mathbf{x}_4^* (\mathbf{b}_0 - \mathbf{a}_0)}{\sqrt{1+\lambda_4}}\right) e^{(1+\sqrt{1+\lambda_4})t} \frac{\mathbf{x}_4}{2||\mathbf{x}_4||^2} \]

	\[ \mathbf{x}(t>>0) \cong \left( \mathbf{x}_4^* \mathbf{a}_0 + \frac{\mathbf{x}_4^* (\mathbf{b}_0 - \mathbf{a}_0)}{5}\right) e^{6t} \frac{\mathbf{x}_4}{2||\mathbf{x}_4||^2} \]
	\\\\
	\section{Problem 4}
	Consider the 1d Poisson equation $\frac{d^2}{dx^2}u(x) = f(x)$ for the vector space of functions $u(x)$ on $x \in [0,L]$ with the Dirichlet boundary conditions $u(0) = u(L) = 0$.

	\subsection{Part 1}
	Suppose the boundary conditions are changed to \textit{periodic} boundary condition $u(0) = u(L)$. What are the eigenfunctions of $\frac{d^2}{dx^2}$ now? Will Poisson's equations have unique solutions? Under what conditions on $f(x)$ would a solution exist?
	\\\\
	Since the boundary is periodic, the eigenfunctions will be $\sin(kx)$, and $\cos(kx)$, where $k=\frac{2\pi n}{L}$ with $n = 1,2,\dots$ for sine and $n=0,1,2,\dots$ for cosine. 0 is ignored for sine because we do not allow the zero function as an eigenfunction, and negative $n$'s since they are linearly dependent on sine and cosine.
	\\\\
	\noindent
	$\sin(\phi + kx) = \cos(\phi)\sin(kx) + \sin(\phi)\cos(kx)$, and is therefore linearly dependent on sine and cosine. And exponential functions are only periodic for $e^{ikx} = \cos(kx)+i\sin(kx)$, which is linearly dependent on sine and cosine.
	\\\\
	\noindent
	The equation will not have unique solutions since the vector space is spanned by $u(x) = \alpha1 \forall \alpha$ Thus, it can have infinite solutions.
	\\\\
	\noindent
	To solve $\frac{d^2}{dx^2}u(x) = f(x)$, we would divide each term in the Fourier series by it's eigenvalue $(\frac{2\pi n}{L})^2$, which is only defined for $n>0$. This implies the $c_0 = 0$, or equivalently $\int_{0}^{L} f(x) = 0$. Under this condition, the equation is solvable.
	\subsection{Part 2}
	If instead we considered $\frac{d^2}{dx^2}v(x) = g(x)$ with the boundary condition $v(0) = v(L)+1$, do these functions form a vector space?
	\\\\
	A vector space must include the zero function, but if $v(x)=0$, then $v(0) \neq v(L)+1$. Thus, the functions do not form a vector space.
	\subsection{Part 3}
	How can we transform $v(x)$ from Part 2 back into the original $\frac{d^2}{dx^2}u(x) = f(x)$ problem with $u(x) = u(L)$ by writing $u(x)=v(x)+q(x)$ and $f(x)=g(x)+r(x)$ for some function $q$ and $r$?
	\\\\
	\noindent
	Let $q$ be a twice differential function with $q(L)-q(1)=-1$. Then $u(L) - u(0) = (v(L)-v(0))+(q(L)-q(0)) = -1 + 1 = 0$ and $u$ is periodic. Then plug $v = u-q$ back into $\frac{d^2}{dx^2}v(x) = g(x)$:

	\[\frac{d^2}{dx^2}(u-q) = g(x)\]

	\[ \frac{d^2}{dx^2}u(x) = g(x) + \frac{d^2}{dx^2}q \]

	Take $\frac{d^2}{dx^2}q$ as $r$ and

	\[ \frac{d^2}{dx^2}u(x) = g(x) + r(x) = f(x)\]
	\\\\
	\section{Problem 5}
	Consider a finite difference approximation as shown below.

	\[ u'(x) \approx \frac{-u(x+2\Delta x) + c\cdot u(x+\Delta x) - c\cdot u(x - \Delta x) + u(x - 2\Delta x)}{d\Delta x} \]

	\subsection{Part 1}
	Substitute the Taylor Series for $u(x+\Delta x)$ to show that an appropriate choice of $c$ and $d$ will make this approximation 4$^{th}$ order accurate (errors are proportional to $(\Delta x)^2$).
	\\\\
	\[ u(x + \Delta x) = \sum_{n=0}^{\infty}\frac{u^{(n)}(x)(\Delta x)^n}{n!}\]

	Denote $\hat{u}'(x)$ as the finite difference approximation of $u'(x)$.

	\[ \hat{u}'(x) = \frac{ -\sum_{n=0}^{\infty}\frac{u^{(n)}(x)(2\Delta x)^n}{n!}+
	c\sum_{n=0}^{\infty}\frac{u^{(n)}(x)(\Delta x)^n}{n!}-
	c\sum_{n=0}^{\infty}\frac{u^{(n)}(x)(-\Delta x)^n}{n!}+
	-\sum_{n=0}^{\infty}\frac{u^{(n)}(x)(-2\Delta x)^n}{n!}  }{d\Delta x}\]

	\[ \hat{u}'(x) = \frac{
	-2\sum_{n=0}^{\infty}\frac{u^{(2n+1)}(x)(2\Delta x)^{(2n+1)}}{(2n+1)!}+
	2c\sum_{n=0}^{\infty}\frac{u^{(2n+1)}(x)(\Delta x)^{(2n+1)}}{(2n+1)!}
    }{d\Delta x}\]

    \[ \hat{u}'(x) = \frac{
    	2\sum_{n=0}^{\infty}\frac{u^{(2n+1)}(x)(\Delta x)^{(2n+1)}}{(2n+1)!}\left( c - 2^{(2n+1)} \right)
    }{d\Delta x}\]

    \[ \hat{u}'(x) = \frac{2}{d} \sum_{n=0}^{\infty} \frac{u^{(2n+1)}(x)(\Delta x)^{2n}}{(2n+1)!}\left( c - 2^{2n+1} \right)
    \]

    \[ \frac{d}{2}\hat{u}'(x) = u'(x)(c-2) + \frac{1}{3!}u^{(3)}(x)(\Delta x)^2(c-8) + \frac{1}{5!}u^{(5)}(x)(\Delta x)^4(c-32) + \dots
    \]

    \[ \frac{d}{2}\hat{u}'(x) - u'(x)(c-2) = \frac{1}{3!}u^{(3)}(x)(\Delta x)^2(c-8) + \frac{1}{5!}u^{(5)}(x)(\Delta x)^4(c-32) + \dots
    \]
    \\\\
    Let $c=8$ and $d=12$.

    \[ 6\hat{u}'(x) - 6u'(x) = \frac{1}{5!}u^{(5)}(x)(\Delta x)^4(-24) + \dots
    \]

    \[ \hat{u}'(x) - u'(x) = \frac{-u^{(5)}(x)}{30} (\Delta x)^4\]

    \[\therefore \hat{u}'(x) - u'(x) \propto (\Delta x)^4 \]

    \subsection{Part 2}
    Check Part 1 by numerically computing $u'(1)$ for $u(x) = \sin(x)$, as a function of $\Delta x$. Verify the approximation is 4$^th$ order accurate using a loglog plot.
    \\\\
    \textit{see notebook.}

\end{document}


\section{First Order Logic}

\subsection{First Order Languages: Syntax}

\begin{itemize}

	\item A \textit{first-order language} (FOL) consists of the following symbols:

	\begin{itemize}

		\item \textit{Logical symbols} $ \land, \lor, \lnot, \implies, \iff, =, \forall, \exists $ (common to all FOLs)

		\item An infinite set of variables, $ x, y, z, \dots $ (also common to all FOLs)

		\item Punctuation symbols: parentheses $ ( \text{ and } ) $ and the comma `$ , $' (also common to all FOLs)

		\item A (possibly empty) set of constant symbols (e.g. $ 0, 1 $)

		\item A (possibly empty) set of function symbols (e.g. $ +, \times, - $)

		\item A (possibly empty) set of predicate symbols (e.g. $ < $)

	\end{itemize}

	\item Each function and predicate symbol has an associated arity $ n $

	\item Only the \textit{non-logical symbols} are specific to the particular language

	\item A FOL may be specified by giving only the constant, relation and function symbols

	\begin{itemize}
		\item E.g. the first-order language of arithmetic $ \Lang_A $ consists of the following:

		\begin{itemize}
			\item The constant symbol $ 0 $
			\item Unary function symbol $ S $ (the successor function)
			\item Two binary function symbols $ + $ and $ \cdot $
		\end{itemize}
	\end{itemize}

	\item Given an FOL $ \Lang $, an \textit{expression} of $ \Lang $ is a finite sequence of symbols; not all expressions are \textit{formulae}

	\item A \textit{term} of an FOL is defined inductively:

	\begin{itemize}

		\item Every constant symbol in $ \Lang $ is a term

		\item Every variable symbol in $ \Lang $ is a term

		\item If $ t_1, \dots, t_n $ are terms and $ f $ is an $ n $-ary function symbol in $ \Lang $, then $ f(t_1, \dots t_n) $ is a term in $ \Lang $

	\end{itemize}

	\item An \textit{atomic formula} of an FOL is defined as follows:

	\begin{itemize}
		\item If $ t_1 $ and $ t_2 $ are terms, then $ t_1 = t_2 $ is an atomic formula

		\item If $ F $ is an $ n $-ary predicate and $ t_1, \dots, t_n $ are terms, then $ F(t_1, \dots, t_n) $ is an atomic formula
	\end{itemize}

	\item A \textit{formula} of an FOL is defined inductively:

	\begin{itemize}

		\item An atomic formula is a formula

		\item If $ \phi $ and $ \psi $ are both formulae, then so are $ \lnot \phi $, $ \phi \land \psi $, $ \phi \lor \psi $, $ \phi \implies \psi $, and $ \phi \iff \psi $

		\item If $ \phi $ is a formula and $ x $ is a variable symbol, then $ \exists x \phi $ and $ \forall x \phi $ are formulae

		\item Parentheses should be used as necessary to ensure there is exactly one way of reading a formula

	\end{itemize}

	\item A variable is \textit{bound} by a quantifier $ \forall x $ or $ \exists x $ in a formula $ \phi $ if:

	\begin{itemize}
		\item $ x $ is in the scope of the quantifier; and

		\item the scope of the quantifier contains no other quantifiers over $ x $ with $ x $ in their scope
	\end{itemize}

	\item Any variable which is not bound in a formula $ \phi $ is \textit{free} in $ \phi $

	\item A \textit{sentence} of an FOL is a formula with no free variables

	\item Importantly, an FOL gives no \textit{meaning} to formulae -- they are not `true' or `false'

\end{itemize}

\subsection{Models: Semantics}

\begin{itemize}

	\item For an FOL $ \Lang $, an \textit{$ \Lang $-structure} or \textit{model} $ \Model $ consists of the following:

	\begin{itemize}
		\item A domain or universe: a non-empty set $ \abs{\Model} $

		\item Interpretation for constant symbols: for each constant symbol $ c $ of $ \Lang $, an element $ c^\Model \in \abs{\Model} $

		\item Interpretation for predicate symbols: for each $ n $-ary predicate symbol $ R $ of $ \Lang $, an $ n $-ary predicate $ R^\Model \subseteq \abs{\Model}^n $

		\item Interpretation for function symbols: for each $ n $-ary function symbol $ f $ of $ \Lang $, an $ n $-ary function $ f^\Model: \abs{\Model}^n \to \abs{\Model} $
	\end{itemize}

	\item A sentence of $ \Lang $ acquires \textit{meaning} when an $ \Lang $-structure $ \Model $ is given and the sentence is interpreted within $ \Model $

	\item We can determine the truth value of a formula $ \phi $ (possibly with free variables) in $ \Lang $-structure $ \Model $ if a \textit{variable assignment} $ \alpha: \text{ set of variable symbols} \to \abs{\Model} $ is given

	\item For given $ \alpha $, replace all free variables $ x_i $ in $ \phi $ by $ \alpha(x_i) $, so $ \phi $ becomes a statement in $ \Model $ which must either be true or false

	\item We say a formula $ \phi $ is \textit{true in $ \Model $} if $ \phi $ is true for \textbf{any} variable assignment $ \alpha $

	\item For a sentence $ \phi $ in $ \Lang $, its truth values does not depend on variable assignment (since there is no free variable). Thus $ \phi $ must be either true or false in $ \Model $, independent of variable assignment

\end{itemize}

\newpage

\subsection{Axiomatic Systems \& Proof}

\begin{itemize}

	\item A formal axiomatic system comprises:

	\begin{itemize}

		\item A first-order language

		\item Syntactic rules for constructing formulae from the symbols

		\item A collection of axioms

		\item Rules of inference

	\end{itemize}

	\item From the axioms we obtain other formulae using the rules of inference, called \textit{theorems}

	\item A \textit{proof} of a theorem is the process of applying the rules

	\item A set of \textit{Logical axioms} are common to first-order axiomatic systems

	\item We may also state theory-specific \textit{non-logical axioms}

	\item Two logical inference rules are also provided:

	\begin{itemize}
		\item Modus ponens: $ \prftree[r]{}{\phi \implies \psi, \phi}{\psi} $

		\item Generalisation: $ \prftree[r]{}{\phi}{\forall x \phi} $
	\end{itemize}

	\item Given $ T $, a `theory' or (possibly empty) set of non-logical axioms in $ \Lang $, a formula $ \psi $ is \textit{provable} in $ T $, denoted $ T \vdash \psi $ if there is a finite sequence $ \phi_1, \dots, \phi_n $ of formulae such that $ \phi_n $ is equal to $ \psi $ and for all $ i $ with $ 1 \le i \le n $ we have:

	\begin{itemize}

		\item $ \phi_i $ is a logical axiom; or

		\item $ \phi_i \in T $; or

		\item There are $ j, k < i $ such that $ \phi_j $ is equal to the formula $ \phi_k \implies \phi_i $; or

		\item There is a $ j < i $ such that $ \phi_i $ is equal to the formula $ \forall x \phi_j $

	\end{itemize}

	\item If a formula $ \psi $ is not provable in $ T $, then we write $ T \nvdash \psi $

	\item A formula $ \phi $ is a \textit{tautology} if $ \vdash \phi $ (i.e. it may be proved with no theory-specific axioms)

	\item We say two formulae $ \phi $ and $ \psi $ are \textit{equivalent}, denoted $ \phi \equiv \psi $ if $ \vdash \phi \iff \psi $; that is, if $ \phi \iff \psi $ is a tautology

	\item We say a theory $ T $ is \textit{consistent} if there is no formula $ \phi $ in $ \Lang $ such that $ T \vdash (\phi \land \lnot \phi) $

	\begin{itemize}
		\item If $ T $ is inconsistent, then for all formulae $ \psi $ in $ \Lang $ we have $ T \vdash \psi $

		\subitem \textit{``From contradiction, everything follows"}
	\end{itemize}

	\item We say a theory $ T $ is \textit{complete} if for all formulae $ \phi $ in $ \Lang $, $ T \vdash \phi $ or $ T \vdash \lnot \phi $

\end{itemize}


\chapter{Introductory Calculus}
\section{The Newton Quotient}
Consider any arbitrary curve.  At any point on that curve, there is a line which
is tangent to that curve.  How might we go about finding the slope of such a
line.  Well, if you consider two points, one some units to the left of the
point we are targeting, and the other some arbitrary amount of units to the
right, between those two points is a secant line.  If you were to take the limit
of the distance between the point we are targeting and the ones that we have
placed to the left and right of it, we would intuitively approach the slope of
the line tangent to the curve at that point.

This process is summarized in the Newton quotient, which is drawn out below:
\begin{equation}
m_{tan}=\lim_{\Delta x\to 0} \frac{f(x+\Delta x)-f(x)}{\Delta x}
\end{equation}

\section{Delta-Epsilon}
Let's say that for some function $f(x)$ that as $x\to c$, $f(x)\to L$.  That is
to say:
\begin{equation}
\lim_{x \to c} f(x) = L
\end{equation}

We can describe $\epsilon$ as a quantity that is the maximum distance the limit
$L$ can be away from the actual value (on a continuous function) at $f(c)$.

Furthemore, $\delta$ is the units away from $x$ to $c$ as we take the limit of
$f$.

Therefore, we can summarize these two statements and presume that for every
$\epsilon > 0$, there is a $\delta > 0$ such that for all $x$:
\begin{equation}
0 < |x-c| < \delta
\end{equation}

and...
\begin{equation}
|f(x)-L| < \epsilon
\end{equation}

\section{The Derivative}
The derivative of $f(x)$ is described as $\frac{d}{dx}f(x)$ and is itself a
function.  The derivative of $f$ is a function that for any $x$ produces a value
equal to the slope of the tangent line of $f$ at $x$.

One way to find the derivative of a function is to take the Newton Quotient of
function.  This is true because:
\begin{equation}
\frac{d}{dx}f(x)=\lim_{\Delta x\to 0} \frac{f(x+\Delta x)-f(x)}{\Delta x}
\end{equation}

\textit{Note}: since the derivative of a function and you can take the deriative
of any function, you can take the derivative of a function an arbitrary number
of times. Physics is the only real world application that requires you to take
the derivative of a function more than two times, but this class frequently
requires you to take the second derivative of a function.  This is often
expressed by the following notation:
\begin{equation}
\frac{d^2y}{dx^2}f(x)
\end{equation}

However, there are rules that we can apply to simplify the process of finding a
derivative.

\subsection{Derivation Rules}
\subsubsection{The Constant Rule}
\begin{equation}
\frac{d}{dx}c=0
\end{equation}

\subsubsection{The Constant Multiple Rule}
\begin{equation}
\frac{d}{dx}cf(x)=c\frac{d}{dx}f(x)
\end{equation}

\subsubsection{The Power Rule}
\begin{equation}
\frac{d}{dx}x^n=nx^{n-1}
\end{equation}

\subsubsection{The Sum and Difference Rule}
\begin{equation}
\frac{d}{dx}\big(f(x)\pm g(x)\big)=\frac{d}{dx}f(x) \pm \frac{d}{dx}g(x)
\end{equation}

\subsubsection{The Product Rule}
\begin{equation}
\frac{d}{dx}\big(f(x)g(x)\big)=f(x)\frac{d}{dx}g(x)+g(x)\frac{d}{dx}f(x)
\end{equation}

\subsubsection{The Quotient Rule}
\begin{equation}
\frac{d}{dx}\Bigg(\frac{f(x)}{g(x)}\Bigg)=\frac{f'(x)g(x)-g'(x)f(x)}{g^2(x)}
\end{equation}

\subsubsection{The Chain Rule}
\begin{equation}
\frac{d}{dx}f(g(x))=f'(g(x))g'(x)
\end{equation}

\section{Implicit Differentation}
Consider the function that defines the unit-circle: $x^2+y^2=1$.  How would you
go about differentiating it?  Sure, we know that $\frac{d}{dx}x^2=2x$, but what
does $\frac{d}{dx}y^2$ evaluate to?  Differentation is easy when the variable we
are differentating agrees with the variable that we are taking the differential
with respect to, but what happens when they don't?

To evaluate $\frac{d}{dx}y^2$, we preform the normal differentation of $a^b$, so
we're left with $2y$, but we still haven't solved the problem of the mismatched
variables.  The solution is simple: we simply multiply the quantity that we have
partially differentiated by the result of differentating the free-hanging
variable: $2y\frac{dy}{dx}$


\section{Introduction}
\label{sec:introduction}

Feed-forward neural networks are useful tools to solve regression problems.
They consist in a group of neurons organized in layers:
the neurons in one layer are connected to the ones in the following layer with weighted directed connection.
The first layer of the network is called input layer and represents the input data for the network:
the dimension of the input layer is equal to the number of dimensions of the dataset.
The output layer has one unit for each continue value to predict.
Between the input and the output layer, the network can have a variable number of hidden layers:
each layer can have a different number of neurons, usually called hidden units.
The output of a hidden unit is usually given by some non-linear function of the weighted sum of its input.

A way to train feed-forward neural networks is to define a cost function and optimize it using numerical optimization methods.
This idea is captured in the Backpropagation training algorithm \cite{backpropagation}:
given a neural network, backpropagation defines a cost (or error) function and uses Gradient Descent to minimize it with respect to the weights parameters.

In this assignment, we implement and train a feed-forward neural network with $1$ hidden layer and $2$ hidden unit for a simple regression problem using Stochastic Gradient Descent and study its learning curves for different-sized training datasets and learning rate policies.

\cref{sec:theory} introduces the Stochastic Gradient Descent training algorithm.
\cref{sec:implementation} describes the implementation of the various experiments.
\cref{sec:evaluation} and \cref{sec:conclusion} discuss and summarize the obtained results.


\section{Waves}

A periodic wave in space characterized by the distance two corresponding
values of the function and is called the wavelength $\lambda$.  The speed
of a wave is
\begin{equation*}
  v = f \lambda.
\end{equation*}
Any function of the form $f(x-vt)$ with finite support describes a
traveling wave in space and time with the forms
\begin{align*}
  y_{\text{pos-dir}} &= A_{+}\cos(k_{+}x - \omega_{+}t + \phi_{+})\\
  y_{\text{neg-dir}} &= A_{-}\cos(k_{-}x + \omega_{-}t + \phi_{-}).
\end{align*}
The angular wavenumber $k$ is defined as
\begin{equation*}
  k \equiv 2\pi / \lambda
\end{equation*}
which yields another relation for wave speed
\begin{equation*}
  \omega = kv.
\end{equation*}

Traveling waves can move in both positive and negative directions through time.
\begin{align*}
  y_{j+} &= A_{j+}\cos(k_{j+}x - \omega_{j+}t + \phi_{j+})\\
  y_{j-} &= A_{j-}\cos(k_{j-}x + \omega_{j-}t + \phi_{j-})
\end{align*}

In general, waves add at each point in space and time
\begin{align*}
  y(x,t) &= \quad\sum_{j=1}^N %
    A_{j+}\cos(k_{j+}x - \omega_{j+}t + \phi_{j+})%
    \\
  &\quad + \sum_{j=1}^N %
    A_{j-}\cos(k_{j-}x + \omega_{j-}t + \phi_{j-}).
\end{align*}
When waves add interference occurs.  For two identical waves save the phase
difference
\begin{equation*}
  y(x,t) = 2A\cos(kx-\omega t + \tfrac{1}{2}\phi) \cos(\tfrac{1}{2}\phi)
\end{equation*}
leading to the notions of completely constructive, destructive, and simply
reductive interference
\begin{align*}
  \phi_{\text{constructive}} &= n\pi & n= 0, \pm 1, \pm 2, \dotsc\\%
  \phi_{\text{destructive}} &= (2n+1)\pi & n= 0, \pm 1, \pm 2, \dotsc\\%
  \phi_{\text{reductive}} &\neq ( \phi_{\text{constr}}, \phi_{\text{destr}})
\end{align*}


%================================
\section{Interiors and Closures}
%================================

%--------------------------------
\begin{definition}
	[interiors]
	\label{def: interiors}
	The \textit{interior} of a set $A$, denoted $A^\circ$, is defined to be the union of all open subsets of $A$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[closure]
	\label{def: closure}
	The \textit{closure} of a set $A$, denoted $\overline A$, is defined to be the intersection of all closed supersets of $A$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[dense sets]
	\label{def: dense sets}
	Let $(X, \mathcal T)$ be a topological space, and let $A \subseteq X$. $A$ is said to be dense, iff $\overline A = X$.
\end{definition}
%--------------------------------

%--------------------------------
\begin{definition}
	[nowhere dense sets]
	\label{def: nowhere dense sets}
	A set $A$ is said to be \textit{nowhere dense} iff the interior of its closure is empty.
\end{definition}
%--------------------------------

%--------------------------------
\begin{proposition}
	[properties of interiors]
	\label{prop: properties of interiors}
	Let $(X, \mathcal T)$ be any topological space and $A, B \subseteq X$.
	\begin{enumerate}[(i)]
		\item
		(Intensive) $A^\circ \subseteq A$.

		\item
		$A$ is open iff $A = A^\circ$.

		\item
		(Idempotence) $(A^\circ)^\circ = A^\circ$.

		\item
		$(A \cap B)^\circ = A^\circ \cap B^\circ$.

		\item
		$A \subseteq B \implies A^\circ \subseteq B^\circ$.

		\item
		If $B$ is open, then $B \subseteq A$ iff $B \subseteq A^\circ$.

	\end{enumerate}

	\begin{proof} \
		\begin{enumerate}[(i)]
			\item
			By Definition \ref{def: interiors}, naturally, $A^\circ \subseteq A$.

			\item
			By Definition \ref{def: topological spaces}, $A^\circ$ is the union of open sets hence it is open. $A$ is open iff it is the union of all open subsets of $A$. Thus $A = A^\circ$.

			\item
			$A^\circ$ is open, thus $(A^\circ)^\circ = A^\circ$.

			\item
			By Definition \ref{def: interiors}, we have
			$$
			\begin{aligned}
				(A \cap B)^\circ &= \left\{ \bigcup U : U \in \mathcal T \land U \subseteq A \cap B \right\} \\
				&= \left\{ \bigcup U: (U \in \mathcal T \land U \subseteq A) \land (U \in \mathcal T \land U \subseteq B) \right\} \\
				&= \left\{ \bigcup U: U \in \mathcal T \land  U \subseteq A \right\} \cap \left\{ \bigcup U : U \in \mathcal T \land U \subseteq B \right\} \\
				&= A^\circ \cap B^\circ.
			\end{aligned}
			$$

			\item
			Clearly, $A^\circ \subseteq A$, thus,
			$$
			\begin{aligned}
				A \subseteq B &\implies A^\circ \subseteq B
			\end{aligned}
			$$
			Suppose $A^\circ \not \subseteq B^\circ$, then $A^\circ \setminus B^\circ$ is not empty ($\emptyset$ is the subset of any set, so $A^\circ$ is not empty).

			Then there exists $x \in A^\circ$ with $x \in \partial B$ ($x \in B$ but $x\notin B^\circ$). Then there exists neighbourhood $N_x \ni x$, and $N_x \cap \partial B \ne \emptyset.$ But this is impossible, for $A^\circ \subseteq B$ implies that $A^\circ \cap \partial B = \emptyset$ (This is a straight consequence of $A^\circ \cap \partial A = \emptyset$. See Proposition \ref{prop: properties of boundaries}), so such $N_x$ does not exist. Thus,
			$$
			A^\circ \subseteq B^\circ.
			$$

			\item
			If $B$ is open, then $B = B^\circ$. Then $B \subseteq A$ iff $B^\circ \subseteq A^\circ$.
 		\end{enumerate}
	\end{proof}
\end{proposition}
%--------------------------------

%--------------------------------
\begin{proposition}
	[properties of closures]
	\label{prop: properties of closures}
	Let $(X, \mathcal T)$ be a topological space, and let $A, B \subseteq X$.
	\begin{enumerate}[(i)]
		\item
		$\overline A$ is closed.

		\item
		$A$ is closed iff $A = \overline A$.

		\item
		$A \subseteq B$ implies $\overline A \subseteq \overline B$.

		\item
		If $A$ is closed, then $A \supseteq B$ iff $A \supseteq \overline B$
	\end{enumerate}

	\begin{proof} \
		\begin{enumerate}[(i)]
			\item
			By Definition \ref{def: closure}, $\overline A$ is the intersection of closed sets. By Proposition \ref{prop: dark side of topology}, $\overline A$ is closed.

			\item
			Proposition \ref{prop: dark side of topology} implies that any closed set is the intersection of closed sets, this is precisely what Definition \ref{def: closure} says.

			\item
			$A \subseteq B$ iff $X \setminus A \supseteq X \setminus B$. Then we have
			$$
			\begin{aligned}
				& X \setminus (X \setminus A)^\circ \subseteq X \setminus (X \setminus B)^\circ
			\end{aligned}
			$$

			Clearly, $(X \setminus A)^\circ$ is the union of all open set disjoint from $A$, then, by De Morgan's laws, $X \setminus (X \setminus A)^\circ$ is the intersection of all closed sets containing $A$. By Definition \ref{def: closure}, we have $(X\setminus A)^\circ = \overline A$. Thus
			$$
			\overline A \subseteq \overline B.
			$$

			\item
			If $A$ is closed, then $A = \overline A$. Suppose $B \subseteq A$, then we have
			$$
			\overline B \subseteq \overline A \iff \overline B \subseteq A.
			$$
		\end{enumerate}
	\end{proof}
\end{proposition}
%--------------------------------


\subsection{Matrix logarithms}

If we have \(e^A=B\) where \(A\) and \(B\) are matrices then we can say that \(A\) is matrix logarithm of \(B\).

That is:

\(\log B=A\)

The dimensions of a matrix and its logarithm are the same.


The grammar is the following:

\begin{align*}
    &S \to SS|LA|LR\\
    &A \to SR\\
    &L \to (\\
    &R \to )
\end{align*}

With this grammar it is possible to demonstrate that the top-down parser needs less recursive calls when it can `detect' that the string does not match the grammar very quickly.

\subsubsection{Number of iterations/recursive calls for each parsers}

With this grammar two interesting cases will be considered:

\begin{enumerate}
    \item Any pattern starting with a right parenthese:
    $$
    patterns =
    \begin{cases}
        )\string^ n\\
        ) \text{ } (\string^ n - 1\\
        )\string^ n - 1 \text{ } (\\
    \end{cases}
    $$
    Many other patterns corresponding to the same case could be found, as long as the first character is a right parenthese.
    This case is interesting because it is the case in which the top-down parser needs to solve the less sub-problems in order to parse the strings.

    For example if it parses a string of size $n$ based on one of the patterns above it will need to solve $2n - 1$ sub-problems to give the result.
    That information, while interesting, is not relevant to anticipate the running time of the parser since even if we know how many sub-problems the parser will solve it can need a very different amount of recursive calls.

    \item The other interesting case is the pattern `$(\string^ n$', that pattern is the worst that can be generated with this grammar for the top-down parser.

    With that pattern the top-down parser will need to solve $n^2 + \lfloor \dfrac{n}{2} \rfloor$ sub-problems to parse a string of size $n$.
\end{enumerate}

Of course as demonstrated in the section 2.4 with equation \ref{eq:bottom-up_iterations} the `boolean' bottom-up parser always needs the same amount of iterations for a string of size $n$ and a given grammar:

\begin{itemize}
    \item[$-$] $n = 500$
    \item[$-$] $gt = 2$
    \item[$-$] $gnt = 4$
\end{itemize}

$$
500 \cdot 2 + \dfrac{4}{6} \cdot (3 \cdot 500^2 \cdot (500 + 1) - 500 \cdot (500 + 1) \cdot (2 \cdot 500 + 1)) = \text{83,334,000 iterations}
$$

Which is indeed the number of iterations obtained when running the code.

The `string' bottom-up parser will also follow the same behaviour for any string pattern.

It is now possible to represent the anticiped behaviour that the `boolean' bottom-up parser will follow for both cases.

\FloatBarrier
\begin{figure}[h]
\centering
\begin{tikzpicture}
\begin{groupplot}[group style={group size=1 by 1},height=0.5\textwidth,width=0.5\textwidth, domain=0:1000]
    \nextgroupplot[title=`Boolean' bottom-up for both cases, ylabel=iterations, xlabel=string size]
    \addplot[red]{2 * x + ((4 / 6) * ((3 * (x^2) * (x + 1)) - (x * (x + 1) * ((2 * x) + 1))))};
\end{groupplot}
\end{tikzpicture}
\caption{Anticipation of the `boolean' bottom-up parser, grammar 1}
\end{figure}
\FloatBarrier

\subsubsection{Comparing the efficiency}

The first experimentation uses the case in which the strings are starting with a right patenthese: $)\string^ n$.

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
    \begin{groupplot}[group style={group size=2 by 2},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=Both bottom-up counters, ylabel=iterations, legend pos=north west]
    \addplot coordinates {
        (50, 21121)
        (100, 167246)
        (200, 1334496)
        (400, 10668996)
        (600, 36003496)
        (800, 85337996)
        (1000, 166672496)};
    \addlegendentry{String}
    \addplot coordinates {
        (50, 83400)
        (100, 666800)
        (200, 5333600)
        (400, 42667200)
        (600, 144000800)
        (800, 341334400)
        (1000, 666668000)};
    \addlegendentry{Boolean}
    \nextgroupplot[title=Both bottom-up running times, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (50, 0.000571)
        (100, 0.003931)
        (200, 0.031064)
        (400, 0.354137)
        (600, 1.25128)
        (800, 3.00095)
        (1000, 6.0186)};
    \addlegendentry{String}
    \addplot coordinates {
        (50, 0.002598)
        (100, 0.019514)
        (200, 0.142831)
        (400, 1.17313)
        (600, 4.05125)
        (800, 10.2203)
        (1000, 20.8444)};
    \addlegendentry{Boolean}
    \nextgroupplot[title=Top-down counter, xlabel=string size, ylabel=recursive calls]
    \addplot coordinates {
        (50, 3676)
        (100, 14851)
        (200, 59701)
        (400, 239401)
        (600, 539101)
        (800, 958801)
        (1000, 1498501)};
    \nextgroupplot[title=Top-down running time, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (50, 0.000117)
        (100, 0.000302)
        (200, 0.000982)
        (400, 0.003857)
        (600, 0.007999)
        (800, 0.015024)
        (1000, 0.022071)};
    \addlegendentry{Total}
    \addplot coordinates {
        (50, 0.000048)
        (100, 0.000097)
        (200, 0.000234)
        (400, 0.000962)
        (600, 0.00241)
        (800, 0.003652)
        (1000, 0.005607)};
    \addlegendentry{Initialization}
    \addplot coordinates {
        (50, 0.000046)
        (100, 0.000195)
        (200, 0.000733)
        (400, 0.002882)
        (600, 0.006131)
        (800, 0.011)
        (1000, 0.017119)};
    \addlegendentry{Parsing}
    \end{groupplot}
\end{tikzpicture}
\caption{Both bottom-up and top-down parsers behaviours, grammar 1, case 1}
\end{figure}
\FloatBarrier

The `boolean' bottom-up parser follows exactly the predicted behaviour, the number of iterations going up following the equation \ref{eq:bottom-up_iterations} and its running time follows exactly the same curve since for each iteration the parser does exactly the same amount of loops.
For this grammar it is slower than the `string' version.

Here the top-down parser is faster than both bottom-up parsers.
To get a more precise result the initialization and the parsing process have been timed apart from each other.
We can see that the initialization time scales up a power function.
The parsing time seems to follow the curve of the recursive calls counter.
\\
\\
It is very easy to check the complexity of the two bottom-up parsers here by resolving for both an equation of the form $y = a \cdot x^3$ by replacing $x$ and $y$ by the coordinates of a point of their graphs.

Those are respectively the theorical expressions of the `boolean' and `string' bottom-up parsers:

\begin{align*}
    &y = 2.0844 \cdot 10^{-8} \cdot x^3 &y = 6.0186 \cdot 10^{-9} \cdot x^3
\end{align*}

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
    \begin{groupplot}[group style={group size=2 by 1},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=`Boolean' bottom-up, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot[domain=0:1000, samples=100, line width=1.5pt, green] {2.0844*(10^(-8))*(x^3)};
    \addlegendentry{Theory}
    \addplot[only marks, mark=*, mark size=2pt] coordinates {
        (50, 0.002598)
        (100, 0.019514)
        (200, 0.142831)
        (400, 1.17313)
        (600, 4.05125)
        (800, 10.2203)
        (1000, 20.8444)};
    \addlegendentry{Real}
    \nextgroupplot[title=`String' bottom-up, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot[domain=0:1000, samples=100, line width=1.5pt, green] {6.0186*(10^(-9))*(x^3)};
    \addlegendentry{Theory}
    \addplot[only marks, mark=*, mark size=2pt] coordinates {
        (50, 0.000571)
        (100, 0.003931)
        (200, 0.031064)
        (400, 0.354137)
        (600, 1.25128)
        (800, 3.00095)
        (1000, 6.0186)};
    \addlegendentry{Real}
    \end{groupplot}
\end{tikzpicture}
\caption{Checking theorical fit, both bottom-up parsers}
\end{figure}
\FloatBarrier

The two functions fit almost perfectly to the points which confirms the $O(n^3)$ complexity for both bottom-up parsers.
\\
\\
The naive parser is very slow with that grammar, but it is possible to check its complexity too by resolving the equation $y = a \cdot 3^x$.

\begin{align*}
    &y = 2.146422 \cdot 10^{-10} \cdot 3^x
\end{align*}

\FloatBarrier
\begin{figure}[h]
\centering
\begin{tikzpicture}
    \begin{groupplot}[group style={group size=1 by 1},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=naive, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot[domain=10:23, samples=100, line width=1.5pt, green] {2.146422*(10^(-10))*(3^x)};
    \addlegendentry{Theory}
    \addplot[only marks, mark=*, mark size=2pt] coordinates {
        (10, 0.000236)
        (15, 0.017009)
        (18, 0.248498)
        (20, 1.39282)
        (21, 3.36746)
        (22, 7.98121)
        (23, 20.2071)};
    \addlegendentry{Real}
    \end{groupplot}
\end{tikzpicture}
\caption{Checking theorical fit, naive parser}
\end{figure}
\FloatBarrier

The theorical curve fits perfectly the running time of the naive parser which proves that the complexity of the naive parser is $O(3^n)$, it also shows that with that grammar this parser is very bad compared to the others.
\\
\\
The second pattern is the pattern only constituated by left parentheses: `$(\string^ n$', what makes it interesting is the fact that it is the worst case that we can generate with this grammar for the top-down parser.
The bottom-up parsers results will not be displayed in that case since they have the same behaviour as with the first case.

\FloatBarrier
\begin{figure}[h]
\begin{tikzpicture}
    \begin{groupplot}[group style={group size=2 by 1},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=Top-down counter, xlabel=string size, ylabel=recursive calls]
    \addplot coordinates {
        (50, 43526)
        (100, 340801)
        (200, 2696601)
        (400, 21453201)
        (600, 72269801)
        (800, 171146401)
        (1000, 334083001)};
    \nextgroupplot[title=Top-down running time, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot coordinates {
        (50, 0.000605)
        (100, 0.004426)
        (200, 0.031748)
        (400, 0.252762)
        (600, 0.819545)
        (800, 1.88128)
        (1000, 3.71316)};
    \addlegendentry{Total}
    \addplot coordinates {
        (50, 0.000033)
        (100, 0.000119)
        (200, 0.000244)
        (400, 0.00081)
        (600, 0.001904)
        (800, 0.003088)
        (1000, 0.005164)};
    \addlegendentry{Initialization}
    \addplot coordinates {
        (50, 0.000561)
        (100, 0.004297)
        (200, 0.031488)
        (400, 0.251933)
        (600, 0.817614)
        (800, 1.87816)
        (1000, 3.70797)};
    \addlegendentry{Parsing}
    \end{groupplot}
\end{tikzpicture}
\caption{Top-down parser behaviour, grammar 1, case 2}
\end{figure}
\FloatBarrier

As we can see on the graph above in that case the initialization time of the top-down parser is negligible, the parsing time is the one that matters.
The parsing time follows exactly the same curve as the number of recursive calls.
Despite the fact that this case is the worst one for the top-down parser its running time remains lower than the other parsers.

We can conclude that for this grammar the top-down parser is more efficient than both bottom-up algorithms.
\\
\\
Here we can verify the complexity of the top-down parser by resolving the same equation as for the two bottom-up parsers.

\begin{align*}
    &y = 3.10797 \cdot 10^{-9} \cdot x^3
\end{align*}

\FloatBarrier
\begin{figure}[h]
\centering
\begin{tikzpicture}
    \begin{groupplot}[group style={group size=1 by 1},height=0.5\textwidth,width=0.5\textwidth]
    \nextgroupplot[title=top-down, xlabel=string size, ylabel=seconds, legend pos=north west]
    \addplot[domain=0:1000, samples=100, line width=1.5pt, green] {3.70797*(10^(-9))*(x^3)};
    \addlegendentry{Theory}
    \addplot[only marks, mark=*, mark size=2pt] coordinates {
        (50, 0.000561)
        (100, 0.004297)
        (200, 0.031488)
        (400, 0.251933)
        (600, 0.817614)
        (800, 1.87816)
        (1000, 3.70797)};
    \addlegendentry{Real}
    \end{groupplot}
\end{tikzpicture}
\caption{Checking theorical fit, grammar 1, case 2}
\end{figure}
\FloatBarrier

The curve fits perfectly the points of the top-down parser which proves that the complexity of the algorithm is $O(n^3)$.