\subsection{The Split-Step Fourier method}
The Split-Step Fourier method can be used to solve nonlinear partial differential equations. The result is obtained by evolving the system in both the time and the frequency domain. Therefore the differential equation is split into a linear and a nonlinear part, which can be considered/regarded separately by using sufficiently small steps. The transformation between time and frequency domain can be achieved efficiently by a fast Fourier transform (FFT) algorithm, which makes this method desirable.

In order to solve the GPE we first take a look at the time-dependent Schr\"odinger equation (\ref{eqn:Schroedinger}).
In one spatial dimension its solution is
\begin{align}
\psi(z,t+\Delta t) = \exp\left[\alpha \left( -\frac{\hbar^2}{2m}\nabla^2 + V(z)\right) \right]\psi(z,t).
\end{align}
We defined $\displaystyle \alpha := -\frac i \hbar \Delta t $ simply for convenience.
Since the terms in the exponent do not commute, one cannot simply split the exponential function up. But if one rewrites the solution as
\begin{align}
\psi (z,t + \Delta t) = \exp\left(\frac \alpha 2 V(z)\right) \cdot
\exp\left( -\alpha\frac{\hbar^2}{2m}\nabla^2\right) \cdot
\exp\left(\frac \alpha 2 V(z) \right) \cdot \psi(z,t),
\end{align}
the error is of $\mathcal{O}(\Delta t^3)$, which is accurate enough in our case and allows us to treat the two steps separately. Due to the fact, that $V$ is diagonal in the time domain and $\nabla^2$ is diagonal in the frequency domain, they simply become multiplications by $V(z)$ and the squared wavevector $k^2$, respectively. Combined with the speed of the FFT this enables a faster and more efficient way to obtain the result than calculating everything in the time domain would be.
Putting everything together yields
\begin{align}
\psi (z,t + \Delta t) = \exp\left(\frac \alpha 2 V(z)\right) \cdot \mathcal{F}^{-1} \left[
\exp\left( -\frac{\alpha\hbar^2}{2m}k^2\right) \cdot \mathcal{F} \left[
 \exp\left(\frac \alpha 2 V(z) \right) \cdot \psi(z,t) \right] \right].
\end{align}

Although we used the linear Schr\"odinger equation with time-independent Hamiltonian, the result is also true for the GPE if one replaces $V(x)$ by $V(x) + g\norm{\psi}^2$. By using the latest result in every step of the calculation one can ensure that the error does not exceed $\mathcal{O}(\Delta t^3)$.

Another important characteristic of the Split-Step Fourier method is, that it conserves the norm of the initial density distribution. To prove this for our implementation we normalized different initial constellations to 1 and plotted the norm after every time step during the evolution. The result is depicted in the following.
\begin{figure}[H]
	\centering
	\includegraphics[scale = 0.25]{figures/norm}
	\caption{Proving the conservation of the norm by our implementation of the Split-Step Fourier method for different initial constellations.}
\end{figure}
As one can see, for our implementation the norm ist conserved within a limit of $\mathcal{O}(10^{-13})$.


\section{Forward and Futures}

\subsection*{Forward Interest rates}
The forward interest rate between time $t-1$ and $t$ satisfies:\\
$ (1+r_t)^t = (1+r_{t-1})^{t-1}(1+f_t) $ \\
$ f_t = \frac{B_{t-1}}{B_t} -1 =\frac{(1+r_t)^t}{(1+r_{t-1})^{t-1}} -1 $

\subsection*{Forward price}
$F_T=e^{(r-y)T} S_0$

\subsection*{Swaps}
The swap rate is a weighted average of forward rates: \\
$r_s = \frac{\sum_{t=1}^{T}B_t f_t}{\sum_{u=1}^{T}B_u} = \sum_{t=1}^{T} w_t  f_t$ ,
with the weights $w_t = \frac{B_t}{\sum_{u=1}^{T} B_u}$ \\

Alternative method: $r_S=\frac{1-B_T}{\sum_{u=1}^{T}B_u}$ in the 3 year case: $r_S=\frac{1-B_3}{B_1+B_2+B_3}=\frac{1-1/(1+r_3)^2}{1/(1+r_1)+1/(1+r_2)^2+1/(1+r_3)^3}$

\subsection*{R11Q1:Forward Interest Rates and Arbitrage }Question 1:
Problem is to find arbitrage from 2 spot rates and 1 forward rate, the general approach to solving this problem is to:
1. Invest $x$ at spot rate, $r_1$
2. Invest $y$ at spot rate, $r_2$
3. Invest $z$ at 1-yr forward rate in year 1, $f_1$
(a) Pays \$100 today and nothing in the future:
$-x-y = 100$ |
$(1 + r_1)x -z = 0$ |
$(1 + r_2)^2y + (1 + f_1)z = 0$ then solve system to find amounts.

\subsection*{R11Q3: Currency Forward}
Suppose that USD/JPY is trading at 105, and the 1-year forward on USD/JPY
is trading at 106. The risk-free rate in the US is 1\%, and in Japan it is 3\%.
Construct an arbitrage strategy that gives you \$100 today and nothing in the
future. {\bf solution}: Year 1 income in JPY = Year 1 liabilities in JPY
$105\times 1.03 \times (x - 100) = 106 \times 1.01 \times x$
Solving yields $x= 9922 USD$


\textbf{Modify Program 20 to make use of matricves instead of FFT. Make sure to do this elegantly, using matrix-matrix multiplications rather than explicit loops. You will find that the code gets much shorter, and faster, too. How much faster is it? Does increasing $N$ from 24 to 48 tip the balance}
\newline

In the following table we have the times spent by the different codes to compute the first derivative (the plotting section was commented out). It can be seen that as $N$ gets large using matrices is more expensive.

\begin{table}[h]
\centering
 \begin{tabular}{||c c c||}
 \hline
 $N$ & FFT & Matrices \\ [0.5ex]
 \hline\hline
 24 & 0.1420280 & 0.0113390 \\
 \hline
 48 & 1.0236130 & 3.7638280 \\ [1ex]
 \hline
\end{tabular}
\caption{Time spent in computing the first derivative of $f$ using FFTs and matrices.}
\end{table}

\subsection*{Matlab code for this problem}
\begin{verbatim}
format long
% Grid and initial data:
Nvector = [24,48];
time = zeros(length(Nvector),1);
for k = 1:length(Nvector)
    N=Nvector(k);
    [D,x] = cheb(N);  y = x;
    D2 = D^2;
    D2 = D2(2:end-1,2:end-1);
    L = kron(eye(N-1),D2)+kron(D2,eye(N-1));
    dt = 6/N^2;
    [xx,yy] = meshgrid(x(2:N),y(2:N));
    x = xx(:); y = yy(:);

    plotgap = round((1/3)/dt); dt = (1/3)/plotgap;
    u = exp(-40*((x-.4).^2 + y.^2));
    uold = u;

    % Time-stepping by leap frog formula:
    [ay,ax] = meshgrid([.56 .06],[.1 .55]); clf
    tic
    for n = 0:3*plotgap
        t = n*dt;
%         if rem(n+.5,plotgap)<1     % plots at multiples of t=1/3
%           uu = reshape(u,N-1,N-1);
%           i = n/plotgap+1;
%           subplot('position',[ax(i) ay(i) .36 .36])
%           [xxx,yyy] = meshgrid(-1:1/16:1,-1:1/16:1);
%           uuu = interp2(xx,yy,uu,xxx,yyy,'cubic');
%           mesh(xxx,yyy,uuu), axis([-1 1 -1 1 -0.15 1])
%           colormap(1e-6*[1 1 1]); title(['t = ' num2str(t)]), drawnow
%         end
    % -------------------- %
    % -- Using matrices -- %
    % -------------------- %
        unew = 2*u - uold + dt^2*L*u;
        uold = u; u = unew;
    % --------------- %
    % -- Using fft -- % Done in Program 20
    % --------------- %
    %     uxx = zeros(N+1,N+1); uyy = zeros(N+1,N+1);
    %     ii = 2:N;
    %     for i = 2:N                % 2nd derivs wrt x in each row
    %       v = vv(i,:); V = [v fliplr(v(ii))];
    %       U = real(fft(V));
    %       W1 = real(ifft(1i*[0:N-1 0 1-N:-1].*U)); % diff wrt theta
    %       W2 = real(ifft(-[0:N 1-N:-1].^2.*U));    % diff^2 wrt theta
    %       uxx(i,ii) = W2(ii)./(1-x(ii).^2) - x(ii).* ...
    %                      W1(ii)./(1-x(ii).^2).^(3/2);
    %     end
    %     for j = 2:N                % 2nd derivs wrt y in each column
    %       v = vv(:,j); V = [v; flipud(v(ii))];
    %       U = real(fft(V));
    %       W1 = real(ifft(1i*[0:N-1 0 1-N:-1]'.*U));% diff wrt theta
    %       W2 = real(ifft(-[0:N 1-N:-1]'.^2.*U));   % diff^2 wrt theta
    %       uyy(ii,j) = W2(ii)./(1-y(ii).^2) - y(ii).* ...
    %                      W1(ii)./(1-y(ii).^2).^(3/2);
    %     end
    end
    time(k) = toc;
end
\end{verbatim}


\chapter{Functional Derivatives}\label{app:functional-derivatives}

A functional is just a function of a function -- i.e.\ some rule $F$ that maps a function $f$ into a number $F[f]$.  Definite integrals are a common example.
In order to optimize a functional $F$ with respect to its argument $f$, one needs to take a \textit{functional derivative}.\footnote{\url{http://en.wikipedia.org/wiki/Functional_derivative}}
To motivate the definition of a functional derivative, first consider the definition of an ordinary derivative
\begin{align}
  \fd{f(x)}{x}
\equiv
  \lim_{\e\rightarrow0}
  \fr{f(x+\e)-f(x)}{\e}
\end{align}
and note the following identity, which you can verify using
$
  f(x+\e)
=
  f(x)
+
  \dfd{f(x)}{x}
  \e
+
  \mc{O}(\e^2)
$.
\begin{align}\label{eq:scalar-derivative-trick}
  \lim_{\e\rightarrow0}
  \fr{f(x+\e)-f(x)}{\e}
=&\
\left.
  \fr{df(x+\e)}{d\e}
\right|_{\e=0}
\end{align}
For multivariate functions, we have the concept of a \textit{directional derivative}
\begin{align}\label{eq:directional-derivative}
  \bo{y}\cdot
  \pd{f(\bo{x})}{\bo{x}}
=
  \lim_{\e\rightarrow0}
  \fr{f(\bo{x}+\e\bo{y}) - f(\bo{x})}{\e}
\end{align}
which measures the change in $f(\bo{x})$ in the direction $\bo{y}$.
Using equation \ref{eq:scalar-derivative-trick}, the directional derivative can be evaluated as an ordinary scalar derivative with respect to $\e$.
\begin{align}\label{eq:vector-derivative-trick}
  \bo{y}\cdot
  \pd{f(\bo{x})}{\bo{x}}
=
  \left.
  \fd{f(\bo{x} + \e\bo{y})}{\e}
  \right|_{\e=0}
\end{align}
The functional derivative $\dfr{\d F}{\d f}$ is defined to satisfy an equation analogous to \ref{eq:directional-derivative}, playing the role of the gradient.
\begin{align}
  \int_{-\infty}^{\infty}
  dx'\,
  g(x')
  \fr{\d F[f]}{\d f(x')}
\equiv
  \lim_{\e\rightarrow0}
  \fr{F[f+\e g] - F[f]}{\e}
\end{align}
This left-hand side could be called a \textit{functional directional derivative}, giving the change in $F$ upon displacing its argument along the function $g$.
Here, the integral takes the role of the dot product in \ref{eq:directional-derivative}.
Using the same trick as in equation \ref{eq:vector-derivative-trick}, the functional derivative can be expressed as an ordinary scalar derivative.
\begin{align}
\label{eq:functional-derivative-trick}
  \int_{-\infty}^{\infty}
  dx'\,
  g(x')
  \fr{\d F[f]}{\d f(x')}
=
  \left.
  \fd{F[f+\e g]}{\e}
  \right|_{\e=0}
\end{align}
The standard procedure for evaluating the functional derivative is to first evaluate the right-hand side of equation~\ref{eq:functional-derivative-trick} for an arbitrary $g$ and then infer what $\dfr{\d F[f]}{\d f(x)}$ must be by comparing to the left-hand side.
Equivalently, $g(x')$ can be replaced with a Dirac delta $\d(x-x')$ in order to arrive at $\dfr{\d F[f]}{\d f(x)}$ directly.

Using eq. \ref{eq:functional-derivative-trick} and the lemma in \cref{app:fundamental-lemma-of-calculus-of-variations}, we find that the stationarity condition for a functional
\begin{align}
  \fr{\d F[f]}{\d f}
\overset{!}{=}
  0
\end{align}
is equivalent to the following condition.
\begin{align}
  \left.
  \fd{F[f+\e g]}{\e}
  \right|_{\e=0}
\overset{!}{=}
  0
&&
  \text{for all $g(x)$}
\end{align}


\vfill~\columnbreak

\section{Useful math}

% ===
\emph{Probabilities}

$\mathbb{E}_x[X] = \begin{cases}
   \int x \cdot p(x) \diff x  & \text{if continuous}\\
   \sum_x x \cdot p(x) & \text{discrete}
  \end{cases}$

$\operatorname{Var}[X] = \mathbb{E}[(X-\mu_X)^2] = \mathbb{E}[X^2] - \mathbb{E}[X]^2$

\textbf{Bayes Rule:} (using \textbf{chain} rule)

\highlight*{$P(A\vert B) = \frac{P(B\vert A) \cdot P(A)}{P(B)}$},
\enskip
$P(Z\vert X,\theta) = \frac{ P(X,Z\vert\theta) }{ P(X\vert\theta) }$

% ===
\emph{p-Norm:}
$\norm{x}_p = (\sum_{i=1}^n \abs{x_i}^p)^{\frac{1}{p}}$, $1 \leq p < \infty$

% ===
\emph{Some gradients}

$\nabla\!_x \norm{x}_2^2 = 2 x$

$f(x) = x^\top A x$; $\nabla\!_x f(x) = (A + A^\top) x$

E.g. $\nabla\!_w \log(1+\exp(-y w^\top x)) = ...$\\
\phantom{E.g.} $=\frac{1}{1+\exp(-y w^\top x)} \cdot \exp(-y w^\top x) \cdot (-y x)$\\
\phantom{E.g.} $=\frac{1}{1 + \exp(y w^\top x)} \cdot(-yx)$

% ===
\emph{Convex / Jensen's inequality}

$f(x) \textbf{ \normalcolor convex }
\myiff f''(x)\!>\!0
\myiff x_i\!\in\!\mathbb{R}, \lambda\!\in\![0,1]:
f(\lambda x_1 + (1\!-\!\lambda) x_2) \leq \lambda f(x_1) + (1\!-\!\lambda) f(x_2)$

\textbf{Jensen's inequality:}
$g(\mathbb{E}[X]) \leq \mathbb{E}[g(X)]$

% ===
\emph{Gaussian / Normal distribution}

$\highlight{\mathcal{N}(\mu,\sigma^2)} \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp\big( \!-\frac{(x-\mu)^2}{2\sigma^2} \big)$

\textbf{Multivariate Gaussian:}\\
$f(x) = \frac{1}{2\pi\sqrt{\abs{\Sigma}}} \exp\left( \frac{1}{2} (x-\mu)^\top \Sigma^{-1} (x-\mu) \right)$\\
with $\textstyle \Sigma =
\begin{bmatrix}
	\sigma_{11}^2	& \sigma_{12}\\
	\sigma_{21}		& \sigma_{22}^2
\end{bmatrix}$
and $\mu =
\begin{bmatrix}
	\mu_1 \\ \mu_2
\end{bmatrix}$

% ===
\emph{Multivariate Gaussian}

$\Sigma =$ covariance matrix, $\mu$ = mean\\
$f(x) = \frac{1}{2\pi \sqrt{|\Sigma|}} e^{- \frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu)}$\\
Empirical: $\Sigma = \frac{1}{n}\sum_{i=1}^n x_i x_i^T$ (needs centered data points)

% ===
\emph{Positive semi-definite matrices}

$M \in \mathbb{R}^{n\times n}$ is psd $\Leftrightarrow$\\
$\forall x \in \mathbb{R}^n: x^TMx \geq 0 \Leftrightarrow$\\
all eigenvalues of $M$ are positive: $\lambda_i\geq 0$


\subsection{Normal equation}

\subsubsection{Least squares}

The square error is \(\sum_i (\hat{y_i}-y_i)^2\).

The differential of this with respect to \(\hat{\theta_j }\) is:

\(2\sum_i \dfrac{\delta \hat{y_i}}{\delta \hat{\theta_j}}(\hat{y_i}-y_i)\)

The stationary point is where this is zero:

\(\sum_i \dfrac{\delta \hat{y_i}}{\delta \hat{\theta_j}}(\hat{y_i}-y_i)=0\)

\subsubsection{Linear least squares}

Here, \(\hat{y_i}= \sum_j x_{ij}\hat{\theta_j}\)

Therefore: \(\dfrac{\delta \hat{y_i}}{\delta \hat{\theta_j}}=x_{ij}\)

And so the stationary point is where

\(\sum_i x_{ij}( \sum_j x_{ij}\hat{\theta_j }-y_i)=0\)

\(\sum_i x_{ij}( \sum_j x_{ij}\hat{\theta_j)}= \sum_i x_{ij}y_i\)

\subsubsection{Normal equation}

We can write this in matrix form.

\(X^TX\hat{\theta }=X^Ty\)

We can solve this as:

\(\hat{\theta }=(X^TX)^{-1}X^Ty\)

\subsubsection{Perfectly correlated variables}

If variables are perfectly correlated then we cannot solve the normal equation.

Intuitively, this is because for perfectly correlated variables there is no single best parameter, as changes to one parameter can be counteracted by changes to another.


\textbf{Suppose the discretization of a boundary value problem, such as the Poisson equation, leads to the matrix equation}
\begin{align*}
uA + Bu = f,
\end{align*}
\textbf{that needs to be solved for $u$. Here $u$ is $n \times m$, $A$ is $m \times m$, $B$ is $n \times n$, and $f$ is $n \times m$.}

\textbf{Assuming both $A$ and $B$ are non-defective matrices, with $A = V_A \Lambda_AV_A^{-1}$ and $B = V_B \Lambda_BV_B^{-1}$, show that the solution $u$ is given by $u = V_B UV_A^{-1}$, where}
\begin{align*}
U_{i,j} = \frac{F_{i,j}}{\alpha_j + \beta_i},
\end{align*}
\textbf{$F = V_B^{-1}fV_A$, and $\alpha_k$ and $\beta_k$ are the diagonal entries of $\Lambda_A$ and $\Lambda_B$ respectively.}

We start with the eigenvalue decomposition of the matrices $A$ and $B$,
\begin{align*}
uV_A \Lambda_AV_A^{-1} + V_B \Lambda_BV_B^{-1} u = f,
\end{align*}
and premultiplying by $V_B^{-1}$ multiplying from the right by $V_A$,
\begin{align*}
V_B^{-1}uV_A\Lambda_A + \Lambda_B V_B^{-1} u V_A = V_B^{-1} f V_A.
\end{align*}
Then let $U = V_B^{-1}uV_A$ and $F = V_B^{-1} f V_A$ so that. Therefore, we have
\begin{align*}
U\Lambda_A + \Lambda_B U = F,
\end{align*}
which in matrix form is
\begin{align*}
&\begin{bmatrix}
\tilde{u}_{11}& \cdots & \tilde{u}_{1m} \\
\tilde{u}_{21}& \cdots & \tilde{u}_{2m} \\
\vdots& ~ & \vdots \\
\tilde{u}_{n1}& \cdots & \tilde{u}_{nm}
\end{bmatrix} \begin{bmatrix}
\alpha_1 & 0 & \cdots & 0 \\
0 & \alpha_2 & ~ & \vdots \\
\vdots & ~& \ddots & ~ \\
0 & \cdots & 0 & \alpha_m
\end{bmatrix} + \begin{bmatrix}
\beta_1 & 0 & \cdots & 0 \\
0 & \beta_2 & ~ & \vdots \\
\vdots & ~& \ddots & ~ \\
0 & \cdots & 0 & \beta_m
\end{bmatrix}\begin{bmatrix}
\tilde{u}_{11}& \cdots & \tilde{u}_{1m} \\
\tilde{u}_{21} & \cdots & \tilde{u}_{2m} \\
\vdots& ~ & \vdots \\
\tilde{u}_{n1} & \cdots & \tilde{u}_{nm}
\end{bmatrix} = \begin{bmatrix}
\tilde{f}_{11} & \cdots & \tilde{f}_{1m} \\
\tilde{f}_{21} & \cdots & \tilde{f}_{2m} \\
\vdots & ~ & \vdots \\
\tilde{f}_{n1} & \cdots & \tilde{f}_{nm}
\end{bmatrix}.
\end{align*}
We can infer from the matrix equation above that
\begin{align*}
U_{i,j} = \frac{F_{i,j}}{\alpha_j + \beta_i}.
\end{align*}
Then, to get back our solution, we use $u = V_B U V_A^{-1}$.

\textbf{Using part (a), solve the Poisson equation $\Delta u = \sin(x)\cos(100y)$ on $[0,1] \times [0,1]$ (with zero Dirichlet boundary conditions) using 100 points in the $x$ direction and 300 in the $y$ direction. Estimate the accuracy of your solution.}

Coding into \textsl{Matlab} the previous method, we obtain a solution for $u$ shown in the figure below,

\begin{figure}[H]
\center{\includegraphics[scale=.75]{problem2_u.png}}
\caption{Solution0 $u$ of the poisson equation.}
\end{figure}

The error, found by comparing $uD2A + D2Bu$ and $f$, where $D2A$ and $D2B$ are the centered finite difference matrices for $x$ and $y$ respectively, is $2.10806\cdot 10^{-12}$.

\subsection*{Matlab code for this part}
\begin{verbatim}
%% Problem 2
Nx=100; Ny=300;
hx=1/(Nx-1); hy=1/(Ny-1);

D2x=gallery('tridiag',Nx,1,-2,1)/(hx^2);
D2y=gallery('tridiag',Ny,1,-2,1)/(hy^2);

[Vx,LambdaX]=eig(full(D2x));
[Vy,LambdaY]=eig(full(D2y));
[x,y]=meshgrid(linspace(0,1,100),linspace(0,1,300));
f=sin(x).*cos(100*y);
F=Vy'*f*Vx;
U=zeros(size(f));
lambdax=diag(LambdaX);
lambday=diag(LambdaY);
% whos U F lambdax lambday
for j=1:Nx
    U(:,j)=F(:,j)./(lambday+lambdax(j));
end

u=Vy*U*Vx';
% Calculate error
err=norm(u*D2x+D2y*u-f,inf)
figure
surf(x,y,u)
shading interp
xlabel('$x$','Interpreter','latex')
ylabel('$y$','Interpreter','latex')
set(get(gca,'ylabel'),'rotation',0)
txt=[path,'problem2_u'];
saveas(gcf,txt,'png')

\end{verbatim}


\subsubsection{The gramian matrix $\trans{A}A$}

We brought the \cref{fig:fund} not only to illustrate the
Fundamental Theorem of Linear Algebra, but also to justify the
introduction of the matrix $\trans{A}A$ (also called the gramian
matrix of $A$). From the figure, it can be seen that the matrix $A$
takes the row space \C{\trans{A}} into the column space \C{A}; and we
know that both subspaces have the same dimension $r =
\func{rank}(A)$. As Strang explains in \cite{strang88}, the
dimensions of the domain \R{n} and codomain \R{m} do not tell the real
story about the linear transformation behind $A$; it is rather the
dimensions of \C{\trans{A}} and \C{A}. \\

If we just pay attention to
those subspaces, then the matrix $A$ behaves like a bijection (this is
proved in \cite{strang88}); that is, if we took the submatrix of
dimensions $r \times r$ that results from eliminating dependent rows
and columns in $A$, such matrix would be invertible and the inverse would
take the column space \C{A} into the row space \C{\trans{A}}.
Thus, the real information of matrix $A$ lies in the one-to-one
transformation of \C{\trans{A}} into \C{A}; the null spaces in both
sides (\N{A} and \N{\trans{A}}) do not contribute much to the
the transformation $A$, as they just get compressed to the zero vector. \\

Alright, putting aside the null spaces and focusing only in the
bijection that $A$ performs between the row and column spaces, one may
be tempted to think from \cref{fig:fund} that the transpose of $A$
(denoted as \trans{A}), is actually the inverse of $A$ in the context
of those subspaces. But as Strang promptly clarifies in
\cite{strang88}, that honor belongs only to the actual inverse of
A. We refer to the inverse here, not in the regular sense, but
restricted to the subspaces \C{\trans{A}} and \C{A} (that is, we are
talking about the inverse of the submatrix of $k \times k$ that we
mentioned above). The effect of \trans{A} is
correct at the level of the whole subspace \C{A}; it takes it back to
\C{\trans{A}}. But the vectors that were originally mapped by $A$, are
not necessarily recovered after applying \trans{A} to that image
$A\vec{x}$. \\

One particular way of reinforcing the fact that \trans{A} is not the
inverse, is by an indirect measure. If we take the dot product between
the starting point \vec{x} in \R{n}, and the result of applying
\trans{A} to its image $A\vec{x}$, that would give us an indication of
how close or distant they are (in the end, the dot product and the
orthogonal projection are intimately related). If the starting and
final vectors happen to be the same, the cited dot product shall be
$\norm{x}_2^2$ (where $\norm{.}_2$ represents the known euclidean
distance). Let us confirm ourselves that is not the case: \\

\begin{align*}
& \vec{x} \cdot (\trans{A} A\vec{x}) &= \\
& \trans{\vec{x}} (\trans{A} A\vec{x}) &= \\
& (\trans{\vec{x}} \trans{A}) (A\vec{x}) &= \\
& \trans{(A\vec{x})} (A\vec{x}) &= \\
& (A\vec{x}) \cdot (A\vec{x}) &= \\
  & \norm{A\vec{x}}_2^2 &\le
& \norm{A}_2^2 \norm{\vec{x}}_2^2
\end{align*}
\hfill

From the above development, we can see indeed that $\vec{x}
\cdot (\trans{A} A\vec{x}) \ne \norm{x}_2^2$; actually, in the last step
we used a general property of matrix norms, which in particular applies
to the extension of the vector norm $\norm{.}_2$ to matrices. We will not
define it formally, but it suffices to keep the intuition that norm of
$A$, denoted as $\norm{A}_2$, is a measure of the distortion that the linear
transformation $A$ does on the space (is actually the maximum
distortion on the unit sphere). In the last step we can see that
such measure of distortion, is precisely one of the factors that
prevents that simply taking the transpose \trans{A} as a way back,
sends us to the starting point in the row space. \\

Above reasoning is a further argument for $\trans{A} \ne \inv{A}$;
such special property only applies to orthogonal matrices (like the
$U$ and $V$, which appear in the SVD factorization). For orthogonal
matrices, $\trans{A} A = I$, where $I$ is the identity matrix; and
though that does not occur in general for an arbitrary matrix $A$, the
function composition represented by $\trans{A}A$ is quite interesting;
it may not be the identity function, but at least it has one of its
properties: \\

\[
\trans{(\trans{A}A)} = \trans{A}\trans{(\trans{A})} = \trans{A}A
\]
\hfill

The matrix $\trans{A}A$ (called gramian) is equal to its transpose, which is the
definition of a symmetric matrix. This particular symmetric matrix
is quite important for us, as it represents the bridge between the SVD
factorization and the Spectral Theorem; in short, the Spectral Theorem
guarantees that any symmetric matrix is diagonalizable, and applying such
factorization to our special matrix $\trans{A}A$, give us the two bases
that we need to build the SVD factorization (which happen contain, the
bases of the four subspaces of $A$). These details will be developed in
the next two sections. For the moment, we will just finish the current one
by establishing a few more facts about $\trans{A}A$, which
show its close connection with $A$. \\

Is not hard to show that $\trans{A}A$ and $A$ share the same null
space; and that actually implies that the rank of both matrices is
the same. Let us state that in a theorem and prove it:

\begin{theorem}[Rank of the Gramian Matrix]
\label{thm:gramr}
Let $A$ be a real matrix of rank $r$ $\implies$ its gramian matrix
$\trans{A}A$ has the same rank $r$.
\end{theorem}
\hfill

\begin{proof}
Let us begin proving that \N{A} = \N{\trans{A}A}.  Let \vec{x} $\in
\N{A}$, then: \\

\[
A\vec{x} = \vec{0} \ds{\iff} \trans{A}(A\vec{x}) = \trans{A}\vec{0}
\ds{\iff} \trans{A}A\vec{x} = \vec{0}
\]
\hfill

Above just proves that $\N{A} \subset \N{\trans{A}A}$, but the other
contention can also be deduced. Let $\vec{x} \in \N{\trans{A}A}$, then: \\

\begin{align*}
& \trans{A}A\vec{x} = \vec{0} &\iff \\
& \trans{\vec{x}}\trans{A}A\vec{x} = \trans{\vec{x}}\vec{0} &\iff \\
& \trans{(A\vec{x})} A\vec{x} = 0 & \iff \\
& (A\vec{x}) \cdot (A\vec{x}) = 0 & \iff \\
& \norm{A\vec{x}}_2^2 = 0 & \iff \\
& A\vec{x} = \vec{0} & \iff \\
& \vec{x} \in \N{A}
\end{align*}
\hfill

The two developments above show that $\N{\trans{A}A} = \N{A}$, that in
particular means that $\dim{\N{\trans{A}A}} = \dim{\N{A}}$. If we
apply the \cref{thm:ortdim} theorem to each matrix, we get the same
dimension for the row space (the row spaces of both matrices live in
\R{n}, which has dimension $n$ of course): \\

\[
\dim{\C{\trans{A}}} = n - \dim{\N{A}} = n - \dim{\N{\trans{A}A}} = \dim{\C{\trans{(\trans{A}A)}}}
\]
\hfill

Knowing that the dimension of the row spaces is the same, we just need
to recall that $\dim{\C{\trans{A}}} = \func{rank}(A)$, and then we can
conclude that $\func{rank}(A) = \func{rank}(\trans{A}A)$.
\end{proof}
\hfill

Another useful result about the matrix $\trans{A}A$, that we will need
when proving the SVD theorem, talks about the qualities of its
eigenvalues. \\

\begin{theorem}
\label{thm:grameig}
Let $A$ be a real matrix of rank $r$, then its gramian matrix
$\trans{A}A$ has $r$ positive eigenvalues.
\end{theorem}
\hfill

Besides reusing \cref{thm:gramr}, the key step in proving
this result, has to do with the previously
used fact that $\trans{\vec{x}}\trans{A}A\vec{x} =
\norm{A}_2^2 \ge 0$; which implies that $\trans{A}A$ is not only symmetric
but also semipositive-definite (by definition). And it turns out, that
semipositive-definite matrices have the desired property of having $r$
positive eigenvalues. We will not prove this theorem here, but
\cite{strang88} can be consulted for further details.


\chapter{Preface}
This manuscript is devoted to a mathematical introduction to deep
neural networks (DNN) through the lens of finite element and multigrid
methods.  Deep neural networks is the core tool in the most important class of
machine learning, namely deep learning, that have been successfully
applied to many artificial intelligence (AI) tasks such as image
classification and natural language processing.  Despite of its great
success in practical applications, deep learning has thus far very
limited mathematical theories.
%
On the other hand, finite element and multigrid methods are two major
numerical methods for solving partial differential equations arising
from science and engineering.  Both finite element and multigrid
method also have rich mathematical theories.
%
In this manuscript, we will introduce and study deep neural networks
through the lens of finite element and multigrid methods in many
different ways.  After we give brief introduction to finite element
method, general iterative methods and then multigrid method, we will
then introduce shallow neural network as a natural generalization of
finite element method.  More specifically, we will illustrate that the
most widely used deep neural network, ReLU-DNN, is identically the
same as the set of all linear finite element functions.  We will
further derive convolutional neural networks (CNN) as some simple
variations of multigrid method.

With the aforementioned mathematical relationships between deep neural
networks and finite element and multigrid method, this manuscript will
provide an easy introduction to machine learning, especially deep
learning to readers with background in numerical analysis, especially
numerical partial differential equations.  On the other hand, this
set of notes will be a good references for readers who are mainly working in
the applications of deep learning but wish to gain some mathematical
understanding the relevant machine learning models.

While there is a vast and rapidly growing literature in machine
learning and especially deep learning, there are relatively very few
books in deep learning, see, for examples,
\cite{goodfellow2016deep,deng2014deep,nielsen2015neural}.
This manuscript will be the first manuscript to present deep neural networks
from more a mathematical viewpoint.

The reading of this book requires some basic knowledge of
multivariable calculus, linear algebra, and numerical analysis.  Some
chapters, especially those chapters involving approximation theory of
DNN require some basic knowledge of real analysis, functional analysis
and approximation theory.  Some special efforts have been made to make
the presentations self-contained.

The manuscript currently, as the first draft, consists of 16
chapters.   The first three chapters cover some basics of machine
learning.

Chapter 1 gives some mathematical discussions of the concept of
linearly separable sets and two linear models, namely logistic
regression and support vector machine (SVM).

Chapter 2  gives a brief introduction to machine learning for its
application to image classification, and also simple
descriptions of some popular data sets, including MNIST, CIFAR and
ImageNet.

Chapter 3 discusses some optimization algorithms, including the widely
used gradient descent method, stochastic gradient descent method and
some convergence theory.

Chapter 4

Chapter 5 introduces the basic artificial neural network, deep neural
network (DNN) from the view of the finite element methods and gives a
discussion about the relation between the linear finite element and
the general ReLU DNN.

Chapters 6-8 discusses some iterative methods and
preconditioning techniques for a general system of equation which is
widely used in FEM and deep learning.  Especially, iterative methods
based on expanded systems are given in Chapter 7 to illustrate the
effectiveness of over-parameterization in the solution of quadratic
optimization.

Chapter 9 discusses a special class of multigrid algorithm for solving
2nd order elliptic boundary value problem on the unit square
discretized by piecewise linear finite element function.  One special
feature of such a presentation is the the multigrid method and its
relevant components are given in terms of discrete convolutions that
are used in machine learning.

Chapter 10 gives some simple descriptions of
convolutional operations and some examples of convolution filters and
convolutional neural network (CNN) models.

Chapter 11 presents a special class of convolutional neural networks, namely
MgNet, by making some minor modification of the multigrid method
presented in Chapter 8.  Such a special derivation of MgNet directly
from multigrid method is hoped to give some mathematical insight to
the convolutional neural network for which mathematical understanding
is still very limited.

Chapter 12 discusses basic initialization and normalization
techniques that are used in deep learning.

Chapter 12 presents some error estimates and adaptivity of finite
element methods. These results will be used by and compared with the corresponding
error estimates for neural networks in the later chapters.

Chapter 13-15 discusses the approximation properties of shallow neural
networks, including the qualitative approximation and some asymptotic
approximation properties.  The materials in these chapters are
relatively more theoretical than the other chapters.

Chapter 16 discuss some applications of deep neural network  functions
in the numerical solution of partial differential equations.

This manuscript is based on notes accumulated from the lectures given in the following two
summer schools:
(1) Summer school on numerical methods of partial differential
equations, Guangzhou, China, 2017; (2)
MATH 497: Deep Learning Alogrithms and Analysis, PSU, United States, 2020
and also the following courses offered at Penn State
(1) MATH 497: An Introduction to Deep Learning in Summer 2019;
(2) MATH 597: Deep Learning in  Spring 2019;
(3) MATH 556: Finite Element Methods in Fall 2018; and
(4) MATH 597: Hierarchical Algorithms and Deep Learning in Fall 2017

Many of my former students, postdocs and collaborators have helped in
the preparations of the aforementioned lectures and in particular the
set of notes, including Jianhong Chen, Juncai He, Qingguo Hong, Li
Jiang, Limin Ma, Jonathan Siegel and Lian Zhang.  I would also like to
acknowledge that support by the Verne M. William Professorship Fund
from Penn State University and the National Science Foundation (Grant
No. DMS-1819157) for the research related to this manuscript.

\bigskip

\noindent Jinchao Xu

\noindent December 2020 State College


\chapter{Constrained Optimization}\label{app:constrained-optimization}

The standard method of optimizing a function subject to a constraint is called Lagrangian optimization.
Taking a function of two variables $f(x,y)$ as an example, suppose we want to optimize it subject to a constraint of the form $g(x,y)=c$.
In this approach, we define the ``Lagrangian function'' $\mc{L}$ as
\begin{align}
  \mc{L}(x,y,\la)
\equiv
  f(x,y)
-
  \la(g(x,y)-c)
\end{align}
where the parameter $\la$ is called the Lagrange multiplier.
The constrained optimization problem can be solved solved by optimizing $\mc{L}$ with respect to $x$, $y$, and $\la$.
To see why, consider the stationarity conditions for $\mc{L}$.
\begin{align}
  \pd{\mc{L}}{x}
=
  \pd{f}{x}
-
  \la\pd{g}{x}
\overset{!}=0
&&
  \pd{\mc{L}}{y}
=
  \pd{f}{y}
-
  \la\pd{g}{y}
\overset{!}=0
&&
  \pd{\mc{L}}{\la}
=
  c
-
  g(x,y)
\overset{!}=0
\end{align}
The last equation is simply the requirement that the constraint $g(x,y)=c$ be satisfied -- i.e.\ that the point $(x,y)$ lies along the contour of $g(x,y)$ specified by $g(x,y)=c$.
The first two equations correspond to the requirement that the gradients of the function $f(x,y)$ and the constraint surface $g(x,y)$ be parallel
\begin{align}
  \nabla f
=
  \la\nabla g
\end{align}
which is always true at the point $(x,y)$ of closest approach along the line $g(x,y)=c$ to a minimum or maximum of the function $f(x,y)$.
This is best understood visually.
\begin{center}
  \includegraphics[width=0.5\linewidth]{lagrangian-optimization}
\end{center}
If the gradients were not parallel, we could move along $g(x,y)=c$ to a higher contour of $f(x,y)$ by following the component of $\nabla f$ parallel to $g(x,y)=c$.


\subsection{Weak law of large numbers}

The sample mean is:

\(\bar X_n=\dfrac{1}{n}\sum_{i=1}^nX_i\)

The variance of this is:

\(Var[\bar X_n]=Var[\dfrac{1}{n}\sum_{i=1}^nX_i]\)

\(Var[\bar X_n]=\dfrac{1}{n^2}nVar[X]\)

\(Var[\bar X_n]=\dfrac{\sigma^2}{n} \)

We know from Chebyshev’s inequality:

\(P(|X-\mu | \ge k\sigma )\le \dfrac{1}{k^2}\)

Use \(\bar X_n\) as \(X\):

\(P(|\bar X_n-\mu | \ge \dfrac{k\sigma }{\sqrt n})\le \dfrac{1}{k^2}\)

Update \(k\) so \(k:=\dfrac{k\sqrt n}{\sigma}\)

\(P(|\bar X_n-\mu | \ge k)\le \dfrac{\sigma^2}{nk^2}\)

As \(n\) increases, the chance that the sample mean lies outside a given distance from the population mean approaches \(0\).


\textit{CLRS appendix A page 1145}

\subsection{Linearitet}
For any real number $ c $ and any finite sequences $ a_1,a_2,\ldots,a_n $ and $ b_1,b_2,\ldots,b_n $,
$$ \sum_{k=1}^{n}\left( ca_k+b_k\right) = c \sum_{k=1}^{n}a_k + \sum_{k=1}^{n}b_k. $$

\subsection{Arithmetic series}
The summation
$$ \sum_{k=1}^{n}k = 1+2+\cdots+n $$
is an \textbf{\textit{arithmetic series}} and has the value
\begin{align*}
\sum_{k=1}^{n}k &= \dfrac{1}{2}n(n+1)\\
&= \Theta\left( n^2\right) .
\end{align*}

\subsection{Sums of squares and cubes}'
We have the following summations of squares and qubes:
\begin{align*}
\sum_{k=0}^{n}k^2&=\dfrac{n(n+1)(2n+1)}{6}\\
\sum_{k=0}^{n}k^3&=\dfrac{n^2(n+1)^2}{4}.
\end{align*}

\subsection{Geometric series}
For real $ x \neq 1 $, the summation
$$ \sum_{k=0}^{n}x^k = 1+x+x^2+\cdots+x^n $$
is a \textit{\textbf{geometric}} or \textbf{\textit{exponential series}} and has the value
$$ \sum_{k=0}^{n}x^k = \dfrac{x^{n+1}-1}{x-1} $$

Algorithms and Data Structures - Reference/summations.tex


\subsection{Matrix rank}
\subsubsection{Rank function}

The rank of a matrix is the dimension of the span of its component columns.

\(rank (M)=span(m_1,m_2,...,m_n)\)

\subsubsection{Column and row span}

The span of the rows is the same as the span of the columns.


\subsection{General activation functions}
Assume that $\sigma$ is a locally Riemann integrable nonzero function
and $\sigma\in L^1(\mathbb{R})$ and thus the Fourier transform of
 $\sigma$ is well-defined and continuous.
 Since $\sigma$ is non-zero and
 \begin{equation}\label{key}
 \hat \sigma(\omega) = \int_{\mathbb{R}} \sigma(t)e^{-2\pi i\omega t}dt,
 \end{equation}
 this implies that $\hat{\sigma}(a)\neq 0$ for
 some $a\neq 0$. Via a change of variables $t = \omega\cdot x + b$ and $dt = db$,
  this means that for all $x$ and $\omega$, we have
 \begin{equation}
 \begin{aligned}
  0\neq \hat{\sigma}(a)&= \int_{\mathbb{R}}\sigma(\omega\cdot x+b)e^{-2\pi ia(\omega\cdot x+b)}db \\
 & = e^{-2\pi ia\omega \cdot x} \int_{\mathbb{R}}\sigma(\omega\cdot x+b)e^{-2\pi iab}db ,
 \end{aligned}
 \end{equation}
 and so
 \begin{equation}
  e^{2\pi ia\omega \cdot x} = \frac{1}{\hat{\sigma}(a)}\int_{\mathbb{R}}\sigma(\omega\cdot x+b)e^{-2\pi iab}db.
 \end{equation}
 Likewise, since the growth condition also implies that $\sigma^{(k)}\in L^1$, we can differentiate the above expression  under the integral with respect to $x$.

 This allows us to write the Fourier mode $e^{2\pi ia\omega \cdot x}$ as an integral of neuron output functions. We substitute this
 into the Fourier representation of $u$
 (note that the assumption we make implies that $\hat{u}\in L^1$ so this
 is rigorously justified for a.e. $x$) to get
 \begin{equation}\label{integral_representation}
 \begin{split}
  u(x) &= \int_{\mathbb{R}^d} e^{2\pi i\omega\cdot x}\hat{u}(\omega)d\omega =
  \int_{\mathbb{R}^d}\int_\mathbb{R}\frac{1}{ \hat{\sigma}(a)}
  \sigma\left(a^{-1}{\omega}\cdot
    x+b\right)\hat{u}(\omega)e^{-2\pi iab}dbd\omega
\\
&=  \int_{\mathbb{R}^d}\int_\mathbb{R} k(x,\theta) dbd\omega
\end{split}
 \end{equation}
where $\theta=(\omega, b)$
and
$$
k(x,\theta)= \frac{1}{ \hat{\sigma}(a)}
  \sigma\left(a^{-1}{\omega}\cdot
    x+b\right)\hat{u}(\omega)e^{-2\pi iab}.
$$


\subsection{Euler's totient function}

This functions counts numbers up to \(n\) which are relatively prime

eg for 10 we have \(1\), \(3\), \(7\), \(9\).

So \(\phi (10)=4\)


\chapter{Perceptron}

\section{Linear Classifier}
A linear classifier implements a decision boundary represented by a straight line in the multidimensional feature space.
The discriminant function:
\begin{equation*}
\begin{split}
f(\mathbf{x}) &=\sum_{i=0}^{I} w_i x_i \\
&= \mathbf{w^{T}x}
\end{split}
\end{equation*}
The decision boundary is given by:
$$f(\mathbf{x})=0$$

\section{Simple Perceptron}
Net synaptic input:
$$\mathbf{u} = \sum_{i=0}^{I} x_i w_i $$
\begin{center}where $w_0 = \theta$ and $x_0=-1$ \end{center}
The output of the neuron unit is:
$$y = \Phi(\mathbf{u})$$
For the rest of this section, we look at learning algorithm for discrete perceptron, using hard limiter as the output activation function.

\subsection{Basic Perceptron Learning Algorithm (Minsky)}
In each iteration, the weight is updated by:
$$w_{new} =
\begin{cases}
    w_{old} + x & y=0\ and\ d=1 \\
    w_{old} - x & y=1\ and\ d=0 \\
    w_{old} & if y==d
\end{cases}
$$

\subsection{Modified Perceptron Learning Algorithm}
The weights change proportional to the difference between the desired output and perceptron output:
$$Error, \varepsilon = d - \Phi(u)$$
$$\mathbf{w}^{new}=\mathbf{w}^{old} + \alpha \varepsilon \mathbf{x}$$
\begin{center}
where learning factor ($0< \alpha \le 1$) and
$\varepsilon \in \{+1, 0, 1\}$
\end{center}
\section{Continuous Perceptron}
The adjustment of weight vector is resolved by using \emph{gradient descent} algorithm:
$$\mathbf{\triangle w} = - \alpha \frac{\partial E(\mathbf{w})}{\partial \mathbf{w}}$$
\begin{equation}
\begin{split}
\mathbf{w^{new}} &= \mathbf{w^{old} + \triangle w } \\
&= \mathbf{w^{old}} - \alpha \frac{\partial E(\mathbf{w})}{\partial w}
\end{split}
\label{weight_update}
\end{equation}
The error function is:
$$E(\mathbf{w})=\frac{1}{2}(d-o)^{2}$$
\begin{center}
where
$o = f(u)$,
$u=\mathbf{w^{T}x}$
\end{center}
The gradient of error function:
\begin{equation}
\begin{split}
\frac{\partial E(\mathbf{w})}{\partial \mathbf{w}} &= \frac{1}{2} \frac{\partial (d-o)^{2}}{\partial \mathbf{w}} \\
&= - (d-o) \frac{\partial f(u)}{\partial \mathbf{w}} \\
&= - (d-o) \frac{\partial f(u)}{\partial u} \frac{\partial u}{\partial \mathbf{w}}
\end{split}
\label{error_derive}
\end{equation}
The synaptic input is:
\begin{equation}
\begin{split}
u &= \sum_{i=0}^{I} w_i x_i \\
\frac{\partial u}{\partial w_i} &= x_i \\
\frac{\partial u}{\partial \mathbf{w}} &= \mathbf{x}
\end{split}
\label{input_derive}
\end{equation}
\noindent Substituting \ref{input_derive} to \ref{error_derive}:
\begin{equation}
\frac{\partial E(\mathbf{w})}{\partial \mathbf{w}} = - (d-o) \frac{\partial f(u)}{\partial u} \mathbf{x}
\label{error_derive_input_derive}
\end{equation}
\noindent Substituting \ref{error_derive_input_derive} to \ref{weight_update}
\begin{equation}
\begin{split}
\mathbf{w}^{new} &= \mathbf{w}^{old} + \alpha (d-o) \frac{\partial f(u)}{\partial u} \mathbf{x} \\
\mathbf{w}^{new} &= \mathbf{w}^{old} + \alpha \delta \mathbf{x}
\end{split}
\label{weight_update_2}
\end{equation}
\begin{center}where $\delta = (d-o) \frac{\partial f(u)}{\partial u}$\end{center}

\subsubsection{Unipolar Sigmoid Function}
$$f(u) = o = \frac{a}{1+exp(-bu)}$$
$$\frac{\partial f(u)}{\partial u} = by \Big(1- \frac{y}{a}\Big)$$

\subsubsection{Bipolar Sigmoid Function}
$$f(u) = o = \frac{a(1-exp(-bu))}{1+exp(-bu)}$$
$$\frac{\partial f(u)}{\partial u} = \frac{ab}{2} \Big(1-\frac{o^{2}}{a^{2}}\Big)$$

\section{Limitation of Perceptron}
\begin{itemize}
\item A perceptron can perform pattern classification only on linearly separable patterns
\item The algorithm may stuck in local minima, sensitive to the starting point
\end{itemize}


\section{Parametric \& Vector Functions}
Up to this point, almost all the graphs we have worked with have been of the form $y=f(x)$, defining the $y$ coordinate in terms of the $x$ coordinate.
These sorts of functions are limited in the types of graphs they can draw.
If we instead let both the $x$ and $y$ coordinates be defined in terms of another variable $t$, like $(x(t),y(t))$, then we can draw much more interesting graphs.
For example, a unit circle, which can't be defined with a single function $y=f(x)$, would be $(\cos{t}, \sin{t})$. \\

We are always able to translate a function of the form $y=f(x)$ into a parametric function as $(t, f(t))$.
Sometimes, but not always, we are also able to translate parametric functions into $y$ as a function of $x$.

\begin{example}
	Given the following parametric function, find $y$ as a function of $x$.
	\begin{equation*}
		(\sqrt{t}, t-2).
	\end{equation*}
\end{example}
\begin{answer}
	Squaring both sides of the $x$ equation,
	\begin{equation*}
		x^2 = t.
	\end{equation*}

	Substituting our expressing for $t$ in terms of $x$ into the $y$ equation,
	\begin{equation*}
		y = x^2 - 2.
	\end{equation*}
\end{answer}

\subsection{Vector Functions}
Vector and parametric functions are essentially the same thing.
In fact, in multivariable calculus, we drop the idea of parametric functions almost completely and exclusively talk about vector-valued functions.
Both can graph the exact same functions.
Visually, you might imagine an arrow rooted at the origin tracing out the graph of a vector function.
You're more likely to see vector functions written in the following form
\begin{equation*}
	\vec{r}(t) = \langle x(t), y(t) \rangle.
\end{equation*}

All the normal vector operations, like addition and subtraction, scalar multiplication, and dot products work exactly the same.
If we think of $\vec{r}(t)$ as a position function,
\begin{align*}
	\textbf{Velocity: }& \vec{v}(t) = \vec{r^\prime}(t) = \langle x^\prime(t), y^\prime(t) \rangle \\
	\textbf{Speed: }& \abs{\vec{v}(t)} = \sqrt{\left(x^\prime(t)\right)^2 + \left(y^\prime(t)\right)^2} \\
	\textbf{Acceleration: }& \vec{a}(t) = \vec{v^\prime}(t) = \langle x^{\prime\prime}(t), y^{\prime\prime}(t) \rangle \\
	\textbf{Direction: }& \frac{\vec{v}(t)}{\abs{\vec{v}(t)}}.
\end{align*}

\subsection{Slope \& Concavity}
Just like with functions like $y=f(x)$, we can find the slope and concavity of parametric functions using first and second derivatives respectively.
We just apply the chain rule.
\begin{align*}
	\dd{y}{x} &= \frac{\dd{y}{t}}{\dd{x}{t}} \\
	\dd{^2y}{x^2} &= \dd{y^\prime}{x} = \frac{\d{y^\prime}/\d{t}}{\d{x}/\d{t}}.
\end{align*}

\begin{example}
	Consider the following parametric function:
	\begin{equation*}
		(t^2-5, 2\sin{t}), 0\leq t\leq\pi.
	\end{equation*}
	Find the first and second derivatives of $y$ with respect to $x$.
\end{example}
\begin{answer}
	Differentiating both $x$ and $y$ with respect to $t$,
	\begin{align*}
		x^\prime(t) &= 2t \\
		y^\prime(t) &= 2\cos{t} \\
		\dd{y}{x} &= \frac{2\cos{t}}{2t} = \frac{\cos{t}}{t}.
	\end{align*}
	Finding the derivative of $y^\prime$ with respect to $t$,
	\begin{align*}
		\dd{}{t}y^\prime &= \dd{}{t}\frac{\cos{t}}{t} \\
		&= \frac{-t\sin{t}-\cos{t}}{t^2} \\
		\dd{^2y}{x^2} &= \frac{\d{y^\prime}/\d{t}}{\d{x}/\d{t}} \\
		&= \frac{\frac{-t\sin{t}-\cos{t}}{t^2}}{2t} \\
		&= -\frac{t\sin{t}+\cos{t}}{2t^3}.
	\end{align*}
\end{answer}

\subsection{Arc Length}
Remember that we had the following formula for $\d{s}$ when deriving arc length.
\begin{equation*}
	\d{s} = \sqrt{\left(\d{x}\right)^2 + \left(\d{y}\right)^2}.
\end{equation*}
Since we now have $x$ and $y$ as functions of $t$, we can rewrite this formula to get a formula for arc length of a parametric function.
\begin{align*}
	\d{s} &= \sqrt{\left(\dd{x}{t}\right)^2 + \left(\dd{y}{t}\right)^2}\d{t} \\
	s &= \int_{a}^{b}{\sqrt{\left(\dd{x}{t}\right)^2 + \left(\dd{y}{t}\right)^2}\d{t}}.
\end{align*}

When talking about vector-valued functions or working in a more physics-based context, you might hear the term ``distance traveled'' instead of arc length and see the following formula.
They are equivalent ideas.
\begin{equation*}
	s = \int_{a}^{b}{\abs{\vec{v}(t)}\d{t}}.
\end{equation*}

\begin{example}
	A circle of radius $r$ is defined parametrically as
	\begin{equation*}
		(r\cos{t}, r\sin{t}), 0 \leq t \leq 2\pi.
	\end{equation*}
	Use this definition to find its circumference.
\end{example}
\begin{answer}
	\begin{align*}
		\dd{x}{t} &= -r\sin{t} \\
		\left(\dd{x}{t}\right)^2 &= r^2\sin^2{t} \\
		\dd{y}{t} &= r\cos{t} \\
		\left(\dd{y}{t}\right)^2 &= r^2\cos^2{t} \\
		C &= \int_{0}^{2\pi}{\sqrt{r^2\sin^2{t}+r^2\cos^2{t}}\d{t}} \\
		&= \int_{0}^{2\pi}{r\sqrt{\sin^2{t}+\cos^2{t}}\d{t}} \\
		&= \int{0}^{2\pi}{r\d{t}} \\
		&= 2\pi r.
	\end{align*}
\end{answer}


\subsection{Gaussian elimination}

\subsubsection{Simultaneous equations}

Matricies can be used to solve simultaneous equations. Condsider the following set of equations.

\begin{itemize}
\item \(2x+y-z=8\)
\item \(-3x-y+2z=-11\)
\item \(-2x+y+2z=-3\)
\end{itemize}

We can write this in matrix form.

\(Ax=y\)

\(A=\begin{bmatrix}2 & 1&-1\\-3 & -1&2\\-2&1&2\end{bmatrix}\)

\(x=\begin{bmatrix}x \\y \\z\end{bmatrix}\)

\(y=\begin{bmatrix}8 \\-11 \\-3\end{bmatrix}\)

\subsubsection{Augmented matrix}

Consider a form for summarising these equations. This is the augmented matrix.

\((A|y)=\begin{bmatrix}2 & 1&-1&|&-8\\-3 & -1&2&|&-11\\-2&1&2&|&-3\end{bmatrix}\)

We can take this and recovery our original \(A\) and \(y\).

However we can also do things to this augmented matrix which preserve solutions to the set of equations. These are:

Undertaking combinations of these can make it easier to solve the equation. In particular, if we can arrive at the form:

\((A|y)=\begin{bmatrix}1 & 0&0&|&a\\0 & 1&0&|&b\\0&0&1&|&c\end{bmatrix}\)

The solutions for \(x,y,z\) are \(a,b,c\).

\subsubsection{Echeleon / triangular form}

We first aim for:

\((A|y)=\begin{bmatrix}a_{11} & a_{12}&a_{13}&|&a\\0 & a_{22}&a_{23}&|&b\\0&0&a_{33}&|&c\end{bmatrix}\)

If this cannot be reached there is no single solution. There may be infinite or no solutions.

\subsubsection{Solving}

Once we have the triangular form, we can easily solve.

\((A|y)=\begin{bmatrix}a_{11} & a_{12}&a_{13}&|&a\\0 & a_{22}&a_{23}&|&b\\0&0&a_{33}&|&c\end{bmatrix}\)

\((A|y)=\begin{bmatrix}1 & 0&0&|&a\\0 & 1&0&|&b\\0&0&1&|&c\end{bmatrix}\)

This process is back substitution (or forward substitution if the matrix is triangular the other way).

\subsubsection{Matrix inversion}

We can think of the inverse of a matrix as one which which takes a series of reverible operations and does these to a matrix then arriving at the identity matrix.

That is, only the three elementary row operations, and combinations of them, can transform a matrix in a way in which it can be reversed. As such All reversible matricies are combinations of the identity matrix and a series of elementary row operations. The inverse matrix is then those series of row operations, in reverse.

We can find identify an inversion by undertaking gaussian elimination. Each step done on the matrix is done to the identify matrix, reversing the process. The end result is the inverted matrix.

Instead of:

\((A|y)=\begin{bmatrix}2 & 1&-1&|&-8\\-3 & -1&2&|&-11\\-2&1&2&|&-3\end{bmatrix}\)

Take:

\((A|I)=\begin{bmatrix}2 & 1&-1&|&1&0&0\\-3 & -1&2&|&0&1&0\\-2&1&2&|&0&0&1\end{bmatrix}\)

When we solve this we get:

\((I|A^{-1})=\begin{bmatrix}1 & 0&0&|&\dfrac{3}{4}&\dfrac{1}{2}&\dfrac{1}{4}\\0& 1&0&|&\dfrac{1}{2}&1&\dfrac{1}{2}\\0&0&1&|&\dfrac{1}{4}&\dfrac{1}{2}&\dfrac{3}{4}\end{bmatrix}\)


\section{Projection Operator: Continuous Fourier Expansion}
	\vspace{0.2cm}

	We define the projection operator denoted as $\mathcal{P}_N$ as the truncated Fourier series, i.e.,
	\begin{equation}
	\label{proyection_operator}
		\mathcal{P}_N u(x) \equiv  \displaystyle \sum_{ |n| \leq \frac {N}{2}} \hat{u}_{n} e^{inx}.
	\end{equation}
	We will denote to $\hat{B}_N$ as the finite subset of $B = span\{e^{inx}: |n| \leq \infty \}$ on which it is projected the function, represented as follows
	\begin{align*}
		\hat{B}_{N} = span \left\{e^{inx}: |n| \leq \frac {N}{2} \right\},\hspace{0.2cm} dim(\hat{B}_{N}) = N + 1.
	\end{align*}
	Then by the orthogonality relation (\ref{ortho_phi}), it can be seen that for $u(x) \in L^2 [0, 2\pi]$
	\begin{align*}
		\langle \mathcal{P}_N u, v \rangle = \langle u, v \rangle, \hspace{3mm} \forall v \in S_N.
	\end{align*}
	This shows that $\mathcal{P}_N u$ is the orthogonal projection of $u$ upon the space of the trigonometric polynomials of degree $N$.\\

	Equivalently, $\mathcal{P}_N u$ is the closest element to $u$ in $\hat{B}_N$ with respect to the inner product
	\begin{align*}
		\langle u, v \rangle = \displaystyle \int_{0}^{2 \pi} u(x) \overline{v(x)} dx,
	\end{align*}
	and this also defines the norm
	\begin{align}
	\label{L2_dot}
		\| u \|^2 = \displaystyle \int_{0}^{2 \pi} |u(x)|^2 dx.
	\end{align}

	A full characterization of the functions for which the Fourier series is convergent is the framework of Lebesgue integration for convergence in mean. This convergence can be defined in $L^2 (0, 2 \pi)$ (square-integrable functions), also is a complex Hilbert space with inner product defined by (\ref{L2_dot}). Then for $u \in L^2 (0, 2 \pi)$ the Fourier series $F(u)$ given by (\ref{fourier_series}) is said to be convergent in mean (or $L^2$-convergent) to $u$ if
	\begin{align}
		\label{L2_mean}
		\displaystyle \int_{0}^{2 \pi} |u(x) - \mathcal{P}_N u(x) |^2 dx \rightarrow 0, \hspace{2mm} \text{as} \hspace{2mm} N \rightarrow \infty,
	\end{align}

	Then the Functions in $L^2 (0, 2 \pi)$ can be characterized in terms of their Fourier coefficients, according to the Riesz theorem, in the following sense. If $u \in L^2 (0, 2 \pi)$, then its Fourier series converges to $u$ in the sense of (\ref{L2_mean}), and by Parseval's identity show us that
	\begin{align}
	\label{parseval}
		\| u \|^2 = 2 \pi \displaystyle \sum^{\infty}_{-\infty} |\hat{u}_n|^2.
	\end{align}
	Conversely, if for any complex sequence $\{\hat{u}_n \}$, $n = 0, \pm 1, \dots $, and $\sum^{\infty}_{n=-\infty} |\hat{u}_n|^2 < \infty$, there exists a unique function $u \in L^2 (0, 2 \pi)$ such that its Fourier coefficients are precisely the $\hat{u}_n$$'$s for any $n$. Thus, for any function $u \in L^2 (0, 2 \pi)$ can be written as
	\begin{align}
		u = \displaystyle \sum^{\infty}_{n=-\infty} \hat{u}_n \phi_n.
	\end{align}

	The Riesz theorem states that the finite Fourier transform is an isomorphism between $L^2 (0, 2\pi)$ and the space $l^2$ of complex sequences $\{\hat{u}_n \}$, $n = 0, \pm1, \pm2, \dots$, such that $\sum^{\infty}_{n=-\infty} |\hat{u}_n|^2 < \infty$. The above can be summed up in the following theorem.

	\begin{teor}
		 If the sum of squares of the Fourier coefficients is bounded
		\begin{align*}
			\displaystyle \sum_{ |n| \leq \infty} |\hat{u}_n|^2 < \infty
		\end{align*}
		then the truncated series converges in the $L^2$ norm
		\begin{align*}
			\|u -  \mathcal{P}_N u \|_{L^2 [0, 2\pi]} \rightarrow 0 \hspace{0.5cm} \text{as} \hspace{0.5cm} N \rightarrow \infty.
		\end{align*}
		If, moreover, the sum of the absolute values of the Fourier coefficients is bounded
		\begin{align*}
			\displaystyle \sum_{ |n| \leq \infty} |\hat{u}_n| < \infty
		\end{align*}
		then the truncated series converges uniformly
		\begin{align*}
			\|u -  \mathcal{P}_N u \|_{L^{\infty} [0, 2\pi]} \rightarrow 0 \hspace{0.5cm} \text{as} \hspace{0.5cm} N \rightarrow \infty.
		\end{align*}
	\end{teor}

	Note that if the truncated sum converges implies that the error is dominated by the tail of the series, i.e.,
	\begin{align*}
		\|u -  \mathcal{P}_N u \|^2_{L^2 [0, 2\pi]} = 2 \pi	\displaystyle \sum_{ |n| > \frac{N}{2} } |\hat{u}_n|^2,
	\end{align*}
	and
	\begin{align*}
		\|u -  \mathcal{P}_N u \|_{L^{\infty} [0, 2\pi]} \leq
		\displaystyle \sum_{|n| > \frac{N}{2}} |\hat{u}_n|.
	\end{align*}
	Thus, the error committed by replacing $u(x)$ with its $N$th-order Fourier series depends solely on how fast the expansion coefficients of $u(x)$ decay. \\

	To appreciate this, suppose that $u(x) \in L^2_p [0, 2 \pi]$ and that its derivative $u'(x) \in L^2_p [0, 2\pi]$, where the subscript $p$ indicate that the function is periodic. then for $n \neq 0$ we have to
	\begin{align*}
		2\pi \hat{u}_N &= \displaystyle \int_{0}^{2\pi} u(x) e^{-inx} dx \\
		&= - \frac{1}{in} (u(2\pi) - u(0)) - \frac{1}{in} \displaystyle \int_{0}^{2\pi} u'(x) e^{inx} dx,
	\end{align*}
	therefore
	\begin{align*}
		|\hat{u}_N| \propto \frac{1}{n}.
	\end{align*}

	In general, if for $u(x)$ and its derivatives $(m - 1)$, and its periodic extensions are all continuous, and also if its derivative $m$th is measurable at $[0, 2 \pi]$, also known in the literature as the regularity of the function, in this particular case in $L^2_p$, we have to $\forall n \neq 0$, repeating the previous procedure successively, the behavior of Fourier coefficients $\hat{u}_n$ of $u(x)$ is similar, i.e.,
	\begin{align*}
		|\hat{u}_n| \propto \left(\frac{1}{n}\right)^m.
	\end{align*}

	 This is known as spectral convergence, which means that the smoother the function, the series converges faster.\\

	 This result is important since it will allow us to investigate the convergence rate of the methods, which we will define in detail later. Therefore, we will focus on periodic functions expanded in Fourier series since its rapid decay of the coefficients implies that the Fourier series truncated after just a few more terms represents an exceedingly good approximation of the function. However, in practice, this decay is not exhibited until there are enough coefficients to represent all the essential structures of the function but in general, functions can be described both through their values in physical space and through their coefficients in transform space. The following examples illustrate the previous results. \\

	\begin{example}
	    Consider the function $u(x) \in C^{\infty} [0, 2 \pi]$ given by
    	\begin{align}
    		\label{Example1}
    	    u(x) = \frac{1}{5 - 4 \cos(x)}
    	\end{align}
    	with its expansion coefficients
    	\begin{align*}
    	     \hat{u}_{n} = \frac{2^{-|n|}}{3}.
    	\end{align*}

    	In Figure \ref{fig1} we can clearly observe the convergence of the Fourier series and that in addition, the convergence of the approximation is almost uniform. This is due to the periodicity of the function and its derivatives.

    	\begin{figure}[H]
        \includegraphics[width=\textwidth]{preliminaries/figures/example21.png}
        \caption{(a) Continuous Fourier series approximation of the equation (\ref{Example1}). (b) The Pointwise error of approximation.}
        \label{fig1}
        \end{figure}
	\end{example}

	\begin{example}
	    The expansion coefficients of the function
    	\begin{align}
    		\label{Example2}
    	    u(x) = \frac{\pi}{2} \sin(\frac{x}{2})
    	\end{align}
    	are given by
    	\begin{align*}
    	     \hat{u}_{n} = \frac{1}{(1 - 4n^2)}.
    	\end{align*}

    	Note that is infinitely differentiable in $[0, 2 \pi]$, but $u'(0) \ne u' (2 \pi)$. In Figure \ref{fig2} we can see that the convergence is much slower than in the Example \ref{Example1}, as expected
    	\begin{figure}[H]
        \includegraphics[width=\textwidth]{preliminaries/figures/example22.png}
        \caption{(a) Continuous Fourier series approximation of the equation (\ref{Example2}). (b) The Pointwise error of approximation for increasing resolution.}
        \label{fig2}
    	\end{figure}
	\end{example}

	\subsection{Differentiation of the Continuous Expansion}

	To find solutions of partial differential equations using the spectral methods, in addition to approximating a function $u(x)$ by the finite Fourier series $\mathcal{P}_N u$, we also need to obtain its derivatives. Due to the linearity of the derivative and that these functions are exponential, we can easily obtain the derivatives of $\mathcal{P}_N u$ by simply differentiating the basis functions term by term. Therefore, if we have the following series truncated \\
 	\begin{align*}
   		\mathcal{P}_N u(x) =  \displaystyle \sum_{ |n| \leq \frac {N}{2}} \hat{u}_{n} e^{inx},
   	\end{align*}
    from this, we can get
   	\begin{align*}
   		\frac{d^q}{dx^q} \mathcal{P}_N u(x) = \displaystyle \sum_{ |n| \leq \frac {N}{2}} \hat{u}_{n} \frac{d^q}{dx^q} e^{inx} = \displaystyle \sum_{ |n| \leq \frac {N}{2}} (in)^q \hat{u}_{n}e^{inx}.
   	\end{align*}
    Therefore the projection and differentiation operators commute, i.e.,
    	\begin{align*}
    	\mathcal{P}_N \frac{d^q}{dx^q} u = \frac{d^q}{dx^q}\mathcal{P}_N u.
    	\end{align*}
    This property implies that for any differentiation operator $\mathcal{L}$ with constant coefficients,
    	\begin{align*}
    	\mathcal{P}_N \mathcal{L} (I - \mathcal{P}_N)u
    	\end{align*}
     vanishes, which known as the truncation error. Thus, the Fourier approximation to the equation $u_t = \mathcal{L}u$ is exactly the projection of the analytic solution.

    \subsection{Approximation theory for Continuous Expansion.}

    The behavior of the functions and their derivatives that we have shown is relevant when the solutions of the differential equations are approximated using spectral methods since it allows us to investigate how fast and precise they can be. In this subsection, we will present these properties based in \cite{gottlieb2007} as detailed as possible some useful results for our main objective regarding the analysis of the projection operator already defined above. \\

    When using the Fourier approximation to discretize the spatial part of the equation
    \begin{align*}
        u_t = \mathcal{L}u,
    \end{align*}
    it is important that our approximation, both to $u$ and to $\mathcal{L}u$, be accurate,i.e., we must consider not only the difference between $u$ and $\mathcal{P}_N u$ if not also the distance between $\mathcal{L} u$ and $\mathcal{L} \mathcal{P}_N u$, measured in an appropriate norm. This is because the actual rate of convergence is determined by the truncation error
    \begin{align*}
    	\mathcal{P}_N \mathcal{L} (I - \mathcal{P}_N)u.
    \end{align*}
   	Thus, the error is determined not only by the behavior of the Fourier approximations of the function but also of its derivatives, as we have seen previously. Therefore, the Sobolev $q$-norm denoted by $H^q_p [0, 2\pi]$, It is appropriate to estimate the truncation error since it measures the smoothness of the derivatives and the function. This norm is defined as follows
    \begin{align}
    \label{sobolev_norm}
    	 \|u\|^2_{H^q_p [0, 2\pi]} = \displaystyle \sum^{q}_{m=0} \int^{2\pi}_{0} \left| u^m (x) \right|^2 dx.
    \end{align}
    The subscript $p$ indicates the fact that all functions are periodic. By substituting the Fourier expansion for each derivative in (\ref{sobolev_norm}), the Sobolev norm can be written as
    	\begin{align*}
    	    \|u\|^2_{H^q_p [0, 2\pi]} = 2\pi \displaystyle \sum^{q}_{m=0} \sum_{|n| \leq \infty} |n|^{2m} |\hat{u}_n|^2 = 2\pi \sum_{|n| \leq \infty} \left(\sum^{q}_{m=0} |n|^{2m} \right) |\hat{u}_n|^2,
    	\end{align*}
    where the interchange of the summation is allowed provided $u(x)$ has sufficient smoothness. \\

   Before starting the analysis, without loss of generality, we first consider the continuous Fourier series given by
        \begin{align*}
            \displaystyle \mathcal{P}_{2N} u(x) = \sum_{|n| \leq N} \hat{u}_n e^{in x}.
        \end{align*}
    The first important result is the estimate in $L^2$ for the distance between $u$ and its trigonometric approximation $\mathcal{P}_{2N} u$, which shows everything we've seen previously.
    \\
    \begin{teor}
    \label{estimating_error_PN_L2}
    For any $u(x) \in H_p^r [0, 2\pi]$, there exists a positive constant $C$, independent of $N$, such that
        \begin{align*}
    	    \|u - \mathcal{P}_{2N} u \|_{L^2 [0, 2\pi]} \leq C N^{-q} \|u^{(q)}\|_{L^2 [0, 2\pi]},
    	\end{align*}
    provided $0 \leq q \leq r$.
    \end{teor}
    \begin{proof}
    By Parseval’s identity given by (\ref{parseval}) we get
    	\begin{align*}
    	    \|u - \mathcal{P}_{2N} u \|^2_{L^2 [0, 2\pi]} = 2\pi \displaystyle \sum_{|n| > N} |\hat{u}_n|^2.
    	\end{align*}
    We rewrite this summation as follows
    	\begin{align*}
    	    \displaystyle \sum_{|n| > N} |\hat{u}_n|^2 &= \sum_{|n| > N} \frac{n^{2q}}{n^{2q}} |\hat{u}_n|^2 \\
    	    &\leq N^{-2q} \sum_{|n| > N} n^{2q} |\hat{u}_n|^2 \\
    	    &\leq N^{-2q} \sum_{|n| \geq 0} n^{2q} |\hat{u}_n|^2 \\
    	    &= \frac{1}{2\pi} N^{-2q} \|u^{(q)}\|^2_{L^2 [0, 2\pi]}.
    	\end{align*}
    Putting all the above together and taking out the square root, we get our result.
	\end{proof}

    \noindent Note that the smoother the function, the larger the value of $q$ and therefore, the better the approximation, as seen before. Now let's notice the following. Suppose that $u(x)$ is analytical, so we have to
    	\begin{align*}
    		 u^{(q)} = \displaystyle
    		 \sum_{ |n| \leq \infty} (in)^q \hat{u}_{n}e^{inx}.
    	\end{align*}
    Since $u^{(q)} \in W^q_p$, and by (\ref{parseval})
        \begin{align*}
            \|u^{(q)}\|_{L^2 [0, 2\pi]} = \sum_{ |n| \leq \infty} |n|^{2q} |\hat{u}_{n}|^2 \leq C q! \sum_{ |n| \leq \infty} |\hat{u}_{n}|^2 \leq C q! \| u \|_{L^2 [0, 2\pi]},
        \end{align*}
    and so by the previous theorem
        \begin{align*}
            \|u - \mathcal{P}_{2N} u \|_{L^2 [0, 2\pi]} \leq  N^{-q} \|u^{(q)}\|_{L^2 [0, 2\pi]} \leq C \frac{q!}{N^{q}} \| u \|_{L^2 [0, 2\pi]}.
        \end{align*}
    Using Stirling’s formula, $q! \sim q^q e^{-q}$, and assuming that $q \propto N$, we obtain
        \begin{align*}
            \|u - \mathcal{P}_{2N} u \|_{L^2 [0, 2\pi]} \leq \sim C \left(\frac{q}{N}\right)^q e^{-q} \| u \|_{L^2 [0, 2\pi]} \sim K e^{-c N} \| u \|_{L^2 [0, 2\pi]}.
        \end{align*}
    Thus, for an analytic function, its spectral convergence is exponential convergence. \\

    We must not forget that the theory we have previously presented is with the assumption that the functions and their derivatives are all periodic. But it is possible to do a similar analysis considering some other class of functions, such as functions that vanish at the borders. However, these kinds of functions belong to spaces very similar to those we have studied, and it is possible to use the same results.


% !TEX root = Main.tex
\section{K-means Algorithm}
\textbf{Target:} $\min_{\mathbf{U}, \mathbf{Z}} J(\mathbf{U}, \mathbf{Z}) = \|\mathbf{X} - \mathbf{U} \mathbf{Z}\|_F^2$\\
$= \sum_{n=1}^N \sum_{k=1}^K \mathbf{z}_{k,n} \|\mathbf{x}_n - \mathbf{u}_k\|_2^2$\\
1. \textbf{Initiate:} choose $K$ centroids $\mathbf{U} = [\mathbf{u}_1, \ldots, \mathbf{u}_K]$\\
2. \textbf{Assign:} data points to clusters. $k^\star(\mathbf{x}_n) = \argmin_k \{ \|\mathbf{x}_n - \mathbf{u}_k\|_2 \}$ returns cluster $k^\star$, whose centroid $\mathbf{u}_{k^\star}$ is closest to data point $\mathbf{x}_n$. Set $\mathbf{z}_{k^\star,n} = 1$, and for $ l \neq k^\star~ \mathbf{z}_{l,n}=0$.\\
3. \textbf{Update} centroids: $\mathbf{u}_k = \frac{\sum_{n=1}^N z_{k,n} \mathbf{x}_n}{\sum_{n=1}^N z_{k,n}}$.\\
4. Repeat from step 2, stops if $\|\mathbf{Z} - \mathbf{Z}^\text{new}\|_0 = \|\mathbf{Z} - \mathbf{Z}^\text{new}\|^2_F = 0$.\\
Computational cost: $O(k\cdot n \cdot d)$


\subsection{The Naive Bayes posterior}

\subsubsection{Bayes theorem}

Consider Bayes' theorem

\(P(y|x_1,x_2,...,x_n)=\dfrac{P(x_1,x_2,...,x_n|y)P(y)}{P(x_1,x_2,...,x_n)}\)

Here, \(y\) is the label, and \(x_1,x_2,...,x_n\) is the evidence. We want to know the probability of each label given evidence.

The denominator, \(P(x_1,x_2,...,x_n)\), is the same for all, so we only need to identify:

\(P(y|x_1,x_2,...,x_n)\propto P(x_1,x_2,...,x_n|y)P(y)\)

\subsubsection{The assumption of Naive Bayes}

We assume each \(x\) is independent. Therefore:

\(P(x_1,x_2,...,x_n|y)=P(x_1|y)P(x_2|y)...P(x_n|y)\)

\(P(y|x_1,x_2,...,x_n)\propto P(x_1|y)P(x_2|y)...P(x_n|y)P(y)\)


\chapter{Linear Discriminant Analysis}
\newpage

\section{Binary LDA}
    Linear Discriminant Analysis aims at projecting data samples onto a hyperplane, such that samples of the same class get as close to each other as possible, while samples of different classes get as far from each other as possible.

    \subsection{Within- and Between- class Distances}
        Let $\mu_1$ and $\mu_2$ be the mean value of two classes.
        \[ \mu_1 = \frac{1}{N_1}\sum_{i\in C_1}x^{(i)} \qquad \mu_2 = \frac{1}{N_2}\sum_{i\in C_2}x^{(i)} \]

        \subsubsection{Between-class Distance}
        Let $w$ be the linear mapping,
        \[ d_{between} = |w^Tu_1 - w^Tu_2| \]
        We want to maximize $d_{between}$, which is equivalent to maximizing
        \[ \|d_{between}\|^2 = w^T(\mu_1-\mu_2)(\mu_1-\mu_2)^Tw\]

        \subsubsection{Within-class Distance}
        We also want to minimize
        \[ d_{within}^{(i)} = \sum_{j\in C}|w^Tx^{(j)} -w^T\mu_i| \]
        for all $j$, which is equivalent to minimizing
        \[ \|d_{within}^{(i)}\|^2 = \sum_{j\in C}w^T(x^{(j)}-\mu_i)(x^{(j)}-\mu_i)^Tw = w^T\Sigma_iw \]
        where $\Sigma_i$ is the covariance matrix of the $i$-th class.

        Therefore define
        \[ d_{within}^2 = \sum_i (d_{within}^{(i)})^2 \]

    \subsection{Opimization Formulation}
        Notice that we have two objective functions, the key is to combine the two objective into one single objective funtion.

        This can be done by simply setting the objective function to be
        \[ \max J = \frac{d_{between}}{d_{within}} = \frac{w^T(\mu_1-\mu_2)(\mu_1-\mu_2)^Tw}{w^T(\Sigma_1 + \Sigma_2)w} \]

        We define the \textbf{within-class scatter matrix} and \textbf{between-class scatter matrix} by
        \[ S_w = \Sigma_1 + \Sigma_2 \]
        \[ S_b = (\mu_1 - \mu_2)(\mu_1-\mu_2)^T \]

        Therefore
        \[ J = \frac{w^TS_bw}{w^TS_ww} \]
        this is also known as the \textbf{generalized Rayleight quotient}.

    \subsection{Solution}
        \[ J = \frac{w^TS_bw}{w^TS_ww} \]
        Notice that both the numerator and denominator are quadratic, so the norm of $w$ does not matter.

        \subsubsection{By Lagrangian}
        Since the norm of $w$ does not matter, the problem is equivalent to
        \begin{align*}
            \min &\quad -w^TS_bw\\
            \text{s.t.} &\quad w^TS_ww=1
        \end{align*}
        By Lagrangian multipliers
        \[ S_bw = \lambda S_ww \]

        Notice that $(\mu_1-\mu_2)^Tw$ is a scalar, and therefore $S_bw$ is parallel with $\mu_1-\mu_2$. WLOG let
        \[ S_bw = \lambda(\mu_1-\mu_2) \]

        And therefore
        \[ w = S_w^{-1}(\mu_0-\mu_1) \]

        \begin{remark}
            For numerical stability we usually compute the inverse by SVD.
        \end{remark}

        \subsubsection{By Gradient}
        We can directly take gradient w.r.t. $w$.
        \[ \nabla J = \frac{2(w^TS_ww)S_bw-2(w^TS_bw)S_ww}{(w^TS_ww)^2} \]
        which yields
        \[ S_bw=JS_ww \Longrightarrow S_w^{-1}S_bw = Jw \]
        and it becomes a eigenvalue decomposition problem.

        To solve this, we perform eigenvalue decomposition on $S_w^{-1}S_b$, and choose the eigenvector with the largest eigenvalue.

        Or alternatively, we can take more than one eigenvectors.

\section{Multi-class LDA}
    We introduce the \textbf{total scatter matrix} $S_T$
    \[ S_T = \sum_{i=1}^N(x^{(i)}-\mu)(x^{(i)}-\mu)^T \]
    where $\mu$ is the mean vector of all samples.

    \begin{align*}
        S_T &= \sum_{j=1}^{K}\sum_{i\in C_j}(x^{(i)}-\mu_j+\mu_j-\mu)(x^{(i)}-\mu_j+\mu_j-\mu)^T\\
        &= \sum_j\sum_{i\in C_j}(x^{(i)}-\mu_j)(x^{(i)}-\mu_j)^T\\
        &\quad + \sum_j\sum_i(\mu_j-\mu)(\mu_j-\mu)^T + 2 \sum_j\sum_i(x^{(i)}-\mu_j)(x^{(i)}-\mu_j)^T\\
        &= S_w + S_b
    \end{align*}
    Notice that the last term is zero.

    The solution is the same as the binary case.

    However, notice that
    \[ S_b = \sum_{j=1}^k N_j(\mu_j-\mu)(\mu_j-\mu)^T \]
    has at most $k-1$ positive eigenvalues, and therefore we can at most reduce the dimension to $k-1$, but not $k$.


\section{The Diffie-Hellman key exchange}
Since public key cryptosystems are relatively slowly compared to private key ones, it is often more realistic to use them in conjunction with classical schemes, giving them the limited role of key exchange and agreement.

A major flaw of symmetric cryptosystems, in fact, is that key establishment must be done through secure means, and \textbf{information cannot directly be sent over the channel}. The two parts must have prior knowledge and trust of each other, and cannot publicly share information.

This can be solved encrypting a symmetric key using a public key algorithm, so that after the decryption it can be used by both parts to encrypt and decrypt using the symmetric cipher.

The first published proposal for a \textbf{key exchange based on public cryptography} is attributed to W. Diffie and M. E. Hellman, and was based on the discrete logarithm problem.

The process involves two individuals agreeing on an arbitrary starting number which does not have to be kept secret, while each one secretly chooses another value to keep to himself. Then, the secret is mixed with the public key, and the result is exchanged.

Finally, the received output is once again combined with the other secret, giving the original message. If a third party listened to the exchange, it would only know the common number and the mixed one, with the inability to retrieve the original value in an efficient way.

Diffie-Hellman is not an encryption algorithm, but rather a \textit{protocol} defining the necessary series of step in order to achieve the exchange of encrypted information between two parties through asymmetrical cryptography.

The basic idea behind the Diffie-Hellman Key Exchange (DHKE) is that exponentiation in $\mathbb{Z}^*_p$ with $p$ prime is a one-way function and the exponentiation is commutative, i.e.:
$$k = (a^x)^y \equiv (a^y)^x \mod p$$

The value $k$ is the joint secret which can be used as session key between the two parties.

The set-up protocol consists of the following steps:
\begin{enumerate}
	\item Choose a large prime $p$;
	\item Choose an integer $\alpha \in \{2, 3, \dots, p - 2\}$, which ideally should be a generator;
	\item Publish $p$ and $\alpha$ (domain parameters).
\end{enumerate}

If Alice and Bob both know the public parameters $p$ and $\alpha$, they can generate a joint secret key $k$ with the DHKE protocol:

\begin{figure}[h]
	\includegraphics[scale=0.35]{DiffieHellmanPaar.png}
	\centering
\end{figure}

Enciphering and deciphering between Alice and Bob can now be performed by both parts:
$$k_{AB} = k^{k_{pr, A}}_{k_{pub, B}} \equiv B^a \mod p$$
$$k_{AB} = k^{k_{pr, B}}_{k_{pub, A}} \equiv A^b \mod p$$

The correctness of this protocol is simple to prove:
\begin{itemize}
	\item Alice computes $B^a \equiv (\alpha^b)^a \equiv \alpha^{ab} \mod p$;
	\item Bob computes $A^b \equiv (\alpha^a)^b \equiv \alpha^{ab} \mod p$.
\end{itemize}
Now both of them share the session key $k_{AB} \equiv \alpha^{ab} \mod p$ which can be used to establish a secure connection for a symmetric algorithm.

This protocol can also be extended with more than two parties: any number of users can take part in the exchange by performing more iterations of the algorithm and exchanging intermediate data.

\subsection{Security of Diffie-Hellman}
Basic Diffie-Hellman protocol is not secure against active attacks: messages can be either modified or falsely generated by a malicious third part, harming its security through a so called man-in-the-middle attack.

However, if generators and finite groups are chosen carefully and large enough, the protocol is considered secure.

Passive attacks. i.e.\ ones in which an attacker can only listen but not alter information, take place trying to compute the session key $k_{AB}$ shared between Alice and Bob.

Just by observing the protocol, one could obtain:
\begin{itemize}
	\item $\alpha$ and $p$, since those are public parameters;
	\item Values $A = k_{pub, A}$ and $B = k_{pub, B}$ by eavesdropping on the channel during an execution of a key exchange.
\end{itemize}
The question is whether $k = \alpha^{ab}$ can be computed as well, assuming that $\alpha$, $p$, $A \equiv \alpha^a \mod p$ and $B \equiv \alpha^b \mod p$ are known.

The previously stated problem is called the Diffie-Hellman problem, and can be generalized to arbitrary finite cyclic groups.

Given a finite cyclic group $G$ of order $n$, a primitive element $\alpha \in G$ and two elements $A = \alpha^a$ and $B = \alpha^b$ in $G$, the Diffie-Hellman problem is to find the group element $\alpha^{ab}$.

One general approach to solve it consists in considering the DHP in the multiplicative group $\mathbb{Z}^*_p$, and requires an effective way to compute discrete logarithms within it. Then, the key can be found with those steps:
\begin{enumerate}
	\item Computing Alice's private key $a = k_{pr, A}$ by solving $a \equiv \log_\alpha A \mod p$;
	\item Computing the session key $k_{A, B} \equiv B^a \mod p$.
\end{enumerate}

It is unknown whether solving the DLP is the only way to crack the Diffie-Hellman protocol without computing the discrete logarithm, but at the moment this is the only available method.

The order of the finite group cardinality should have a large prime factor $p$ to prevent usage of attacking algorithms to obtain the exponent. For this reason, so called safe primes $q$ are used to calculate $p = 2q + 1$, such that the order of the group is only divisible by 2 and $q$.

Hence, in order to ensure security in practice, there must be the assumption that this problem cannot be solved in an efficient way, choosing $p$ sufficiently large to make the computation infeasible.

Furthermore, private keys should always stem from a true random generator in order to prevent an attacker to guess them.

$p$ is consequently generated using a probabilistic prime-finding algorithm, and should have a length of at least 1024 bits in order to provide strong security.

\subsubsection{Key generation}
The session key $k_{AB}$ is supposedly a large randomly chosen integer (or a collection of such). Since the algorithm has application within large finite fields, random numbers need to be adjusted with a modulo operation, associating an integer in $[0, p - 1]$.

When the field corresponds to $\mathbb{Z}^*_p$, not all elements have an inverse.

Nevertheless, it should have the same length as $p$; to improve computational speed, it can also be used as symmetric key taking the first 128 most significant bits or hashed.

$A$, $B$ and the session key can be calculated using the repeated squares algorithm. Public keys are typically precomputed, and the main computation is therefore the exponentiation for the session key.

The integer $\alpha$ needs to have a special property: it should be a primitive element.

\subsubsection{Applications}
DHKE is implemented in many open and commercial cryptographic protocols such as \textbf{SSH}, \textbf{TLS} and \textbf{IPSec}.

SSH offers Diffie-Hellman as an option, since it implements all of the standard cryptographic algorithms.

TLS, on the other hand, primarily relies on Diffie-Hellman. The \textit{TLS handshake} between client and server begins with a negotiation to determine the crypto algorithms used for the session. The client sends a list of supported ciphersuites, them being different versions of Diffie-Hellman within a message, specifying a key exchange algorithm and other primitives.

The server arbitrarily selects an algorithm from the client’s list and signals it, along with selecting the Diffie-Hellman parameters. It chooses a group $(p, g)$, computes $g^b$, and sends a message containing a signature using the long-term signing key from its certificate.

The client then verifies the signature and responds with a message containing $g^a$.To ensure agreement on the negotiation messages, each party computes the TLS master secret from $g^{ab}$ and calculates a MAC of its view of the handshake transcript.

These MACs are exchanged and verified by the recipients. Thereafter, client and server start exchanging application data, protected by an authenticated encryption scheme with keys also derived from $g^{ab}$.

However, much internet traffic uses one of a handful of commonly-known groups that involves primes having less than 1024 bits. There are good reasons for this: using a standard set of groups reduces implementation complexity, ensures interoperability between different pieces of software, and enables the research community to focus its cryptanalytic effort on a few important groups.

These facts, however, can be exploited to attack TLS, find the session key and therefore compromising a large amount of servers along with their security. An attacker can perform preprocessing attacks knowing which groups will be used, doing precomputation of possible values to then find discrete logarithm in a much shorter amount of time.

In fact, in 2015 the \textbf{Logjam} attack was exploited, taking advantage of the Number Field Sieve algorithm and Diffie-Hellman vulnerabilities to both perform a man-in-the-middle and compute the discrete logarithm of a 512-bit key. 1024-bit keys are still considered secure.

Seminar - Algebraic Methods and Algorithms in Cryptology  /chapters/diffie-hellman.tex


%
% Chapter 2.9
%

\section*{2.9 Linear Approximation and Differentials}

\textbf{Linear approximation} (tangent line approximation) is a way of finding the slope of a curve using the tangent line at point \((a, f(a))\) as an approximation to the curve \(y=f(x)\) when \(x\) is near \(a\).\\\\
The linear function whose graph is the tangent line of \(f(x)\) at \((a, f(a))\) is called the \textbf{linearization} of \(f\) at \(a\).

$$ L(x) = f(a) + f'(a)(x-a) $$

\subsection*{Differentials}

If \(y=f(x)\) is a differentiable function, then differential \(dx\) is an independent variable. The differential \(dy\) is defined in terms of \(dx\) by the equation
$$ dy=f'(x)dx $$


\subsection{Complements and disjoint sets}

\subsubsection{Disjoint sets}

Sets are disjoint is there is no overlap in their elements. Two sets are \(s_i\) and \(s_j\) are mutually exclusive if:

\(s_i\land s_j=\emptyset\)

A collection of events \(s\) are all mutually exclusive if all pairs are mutually exclusive. That is:

\(\forall s_i \in s\forall s_j\in s[s_i\land s_j\ne \emptyset \rightarrow s_i=s_j]\)

\subsubsection{Complement function}

\(x^C\) is the completement. It is defined such that:

\(\forall x [x\land x^C=\varnothing ]\)

For a set \(b\), the complement with respect to \(a\) is all elements in \(a\) which are not in \(b\).

\(\forall x \in a \forall y \in b [x \in (a \setminus b) \land y\in (a \setminus b)]\)

\(b\land (a \setminus b)= \varnothing \)

That is, \(b\) and \(a \setminus b\) are disjoint.

\subsubsection{Existence of the complement}

For two sets \(a\) and \(b\) we can write \((a \setminus b)\). This is the set of elements of \(a\) which are not in \(b\).

Consider the axiom of specification:

\(\forall x \forall a \exists s[(P(x)\land x\in a )\leftrightarrow (x\in s)]\)

We can also write

\(\forall x \forall a \forall b\exists s[(x\not\in b\land x\in a )\leftrightarrow (x\in s)]\)

Which provides the complement, \(s\).


Consider the differential equation
\begin{align*}
y' = \sin(t)(y^2 - \cos^2(t) -1),~~y(0)=1,~~t>0.
\end{align*}
The exact solution to this nonlinear ODE is $y(t) = \cos(t)$.

\begin{questions}

\question{Use \texttt{rk4} to solve this equation for $t \in [0,50]$. Plot the error vs. values of $\Delta t$ (use \texttt{loglog}). Does the error decay as $\mathcal{O}(\Delta t^4)$? }

\begin{solution}

\begin{figure}[H]
\center{\includegraphics[scale=.5]{P1F1.eps}}
\caption{Error Vs. $\Delta t$}
\end{figure}

The figure above shows that the error does indeed seem to decay as $\mathcal{O}(\Delta t^4)$.

\end{solution}

\question{You will now compare \texttt{rk4}, forward-Euler, and modified Euler methods. Solve the problem with these three methods for several $\Delta t$. on a \texttt{loglog} plot, show the error vs. the number of function evaluations used by eahc method. Using the number of function evaluations as a measure of cost, is it worth using several stages in RK methods? }

\begin{solution}

\begin{figure}[H]
\center{\includegraphics[scale=.5]{P1F2.eps}}
\caption{Error Vs. cost}
\end{figure}

The figure above indicates that it is indeed worth it to use several stages in RK methods despite their cost. Even evaluated at 4 times the cost, RK4 still performs markedly better than euler and modified euler, equivalent to RK1 and RK2 respectively.

\end{solution}

\end{questions}

\subsection*{MATLAB}

\subsection*{Contents}

\begin{itemize}
\setlength{\itemsep}{-1ex}
   \item part b
\end{itemize}
\begin{verbatim}
close all; clear variables;
% Part a

N = 2.^(4:12);
err = 0*N;
f = @(t,y) sin(t)*(y^2 - (cos(t))^2 - 1);

for k = 1:length(N)
    [t,w] = rk4(f,[0 50],1,N(k));
    err(k) = norm(w - cos(t),inf);
end
dt = 50./N;

figure
loglog(dt,err,'*',dt,1e5*N.^(-4),'--')
grid on
\end{verbatim}

\subsection*{part b}

\begin{verbatim}
for k = 1:length(N)
    [t,w] = rk4(f,[0 50],1,N(k));
    err(k) = norm(w - cos(t),inf);
    [t,w] = euler(f,[0 50],1,N(k));
    err1(k) = norm(w - cos(t),inf);
    [t,w] = modified_euler(f,[0 50],1,N(k));
    err2(k) = norm(w - cos(t),inf);
end

figure
loglog(N,err1,'b',2*N,err2,'r',4*N,err,'g')
legend('euler','modified_euler','rk4')
grid on
\end{verbatim}

\subsection*{rk4.m}

    \begin{verbatim}
function [t,w] = rk4(dydt,tspan,y0,N)

h = diff(tspan)/N;
t = tspan(1) + h*(0:N)';
w = zeros(length(y0),N+1);
w(:,1) = y0(:).';

for i = 1:N
    k1 = h*dydt(t(i)    , w(:,i)     );
    k2 = h*dydt(t(i)+h/2, w(:,i)+k1/2);
    k3 = h*dydt(t(i)+h/2, w(:,i)+k2/2);
    k4 = h*dydt(t(i)+h  , w(:,i)+k3  );
    w(:,i+1) = w(:,i) + (k1 + 2*k2 + 2*k3 + k4)/6;
end
w = w.';
end
\end{verbatim}

\subsection*{euler.m}

    \begin{verbatim}
function [t,w] = euler(dydt,tspan,y0,N)

h = diff(tspan)/N;
t = tspan(1) + h*(0:N)';
w = zeros(length(y0),N+1);
w(:,1) = y0(:).';

for i = 1:N
    k1 = h*dydt(t(i)    , w(:,i)     );
    w(:,i+1) = w(:,i) + k1;
end
w = w.';
end
\end{verbatim}

\subparagraph*{modified euler.m}

    \begin{verbatim}
function [t,w] = modified_euler(dydt,tspan,y0,N)

h = diff(tspan)/N;
t = tspan(1) + h*(0:N)';
w = zeros(length(y0),N+1);
w(:,1) = y0(:).';

for i = 1:N
    k1 = h*dydt(t(i)    , w(:,i)     );
    k2 = h*dydt(t(i)+h, w(:,i)+k1);
    w(:,i+1) = w(:,i) + (k1 + k2)/2;
end
w = w.';
end
\end{verbatim}


\subsection{Momentum gradient descent}

\subsubsection{Batch gradient descent}

:= used to denote an update of variable. Used in programming, eg x=x+1.

\(\theta _j := \alpha \dfrac{\delta }{\delta \theta _j}J(\theta _0,\theta _1)\)

\(\alpha \) sets rate of descent.

\(\theta 0 := \theta 0 - \alpha/m \sum(h0(x) - y)\)

\(\theta j := \theta j - \alpha/m \sum(h0(x) - y)xj\)

Can check if j theta increasing, means bad methodology, lower alpha

Get run for x iterations,evaluate j(theta)

Can use matrices to do each step

Can check convergence by checking cost over last 1000 or so, rather than all

Smaller learning rate can get to better solution, as can circle drain for small samples

Slowly decreasing learning rate can get better solutions

\(\alpha = const1/(i + cost2)\)

Do gradient descent on all samples

The standard gradient descent algorithm above is also known as batch gradient descent. There are other implementations.

\subsubsection{Mini-batch gradient descent}

Use \(b\) samples on each iteration, \(b\) is parameter, between stochastic and batch

\(b=2-100\) for example

\subsubsection{Stochastic gradient descent}

Do gradient descent on one (?!) sample only

Not guaranteed for each step to go towards minimum, but each step much faster

\subsubsection{Stochastic gradient descent with momentum}

The gradient we use is not just determined by the single sample, it is a moving average of past samples.

\subsubsection{Epochs}

This refers to the number of times the whole dataset has been run.


\subsection{Exponential Rule}
Let's find the derivative of the most natural exponential function: $f(x) = e^x$.
Using the limit definition of the derivative,
\begin{align*}
	f^\prime(x) &= \lim_{h \to 0}{\frac{e^{x+h}-e^x}{h}} \\
	&= \lim_{h \to 0}{\frac{e^x\left(e^h - 1\right)}{h}} \\
	&= e^x \lim_{h \to 0}{\frac{e^h - 1}{h}}
\end{align*}
Remembering the following definition of $e$,
\begin{equation*}
	e = \lim_{n \to \infty}{\left(1+\frac{1}{n}\right)^n}.
\end{equation*}
Substituting $h = 1/n$,
\begin{equation*}
	e = \lim_{h \to 0}{\left(1+h\right)^{1/h}}.
\end{equation*}
Putting substituting this definition for $e$ into our work,
\begin{align*}
	f^\prime(x) &= e^x \lim_{h \to 0}{\frac{\left(\left(1+h\right)^{1/h}\right)^h-1}{h}} \\
	&= e^x \lim_{h \to 0}{\frac{\left(1+h\right)-1}{h}} \\
	&= e^x \lim_{h \to 0}{\frac{h}{h}} \\
	&= e^x.
\end{align*}
Amazingly, this function is equal to it's own derivative. In fact, aside from the trivial example of $0$, $e^x$ is the only function with this property. \\

We can apply the chain rule to find the derivative of $b^x$ for real, positive values of $b$.
\begin{equation*}
	f(x) = b^x	= e^{x\ln{b}}.
\end{equation*}
Let $u(x) = x\ln{b}$.
\begin{equation*}
	f(x) = b^x = e^{u(x)}.
\end{equation*}
Applying the chain rule,
\begin{align*}
	f^\prime(x) &= \dd{}{u}e^u \hspace{3pt} \dd{}{x}x\ln{b}. \\
	&= e^u \ln{b} \\
	&= e^{x\ln{b}} \ln{b} \\
	&= b^x \ln{b}.
\end{align*}

\begin{example}
	Find the derivative of $f(x) = 2^{x^2}$.
\end{example}
\begin{answer}
	Let $u(x) = x^2$.
	\begin{equation*}
		f(x) = e^u(x).
	\end{equation*}
	Using the exponential and chain rules,
	\begin{align*}
		f^\prime(x) &= \dd{f}{u} \hspace{3pt} \dd{u}{x} \\
		&= \dd{}{u}e^u \hspace{3pt} \dd{}{x}x^2 \\
		&= 2^u \ln{(2)} 2x \\
		&= 2^{x^2} \ln{(2)} 2x \\
		&=  2x\ln{(2)}2^{x^2}.
	\end{align*}
\end{answer}


\chapter{Transformation of variables}
For this chapter we will show tranformation of variables from a distribution to another distribution. From \cite{casella2002statistical} we have a theorem for tranformation of variables The theorem in one dimension is given as follows.
\begin{theorem}
Let $X$ have pdf $f_X (x)$, let $Y=g(X)$, where $g$ is a monotone function. Let $\mathcal{X} = \{x: f_X (x) > 0\}$ and $\mathcal{Y} = \{ y: y = g(x)$ for some $x \in \mathcal{X} \}$. Suppose that $f_x(x)$ is continuous on $\mathcal{X}$ and that $g^{-1}(y)$ has a continuous derivative on $\mathcal{y}$. Then the pdf of $Y$ is given by
\begin{equation}
f_Y (y) =
\begin{cases}
f_X(g^{-1}(y)) \left| \frac{d}{dy} g^{-1}(y) \right| & y \in \mathcal{Y} \\
0 & \text{otherwise.}
\end{cases}
\end{equation}
\end{theorem}
For multivariate the pdf of $Y$ is given by
\begin{equation}
f_Y (y) =
\begin{cases}
f_X(g^{-1}(y)) \left| J(g^{-1}(y)) \right| & y \in \mathcal{Y} \\
0 & \text{otherwise,}
\end{cases}
\end{equation}
where $J$ is the Jacobian matrix. The determinant of Jacobian matrix can be rewritten to $\left| J(g^{-1}(y)) \right| = \left| J(g(y)) \right|^{-1}$