\subsection{Work}

The mathematical concept of work is an application of vectors in $\R^n$.  The physical concept
of work differs from the notion of work employed in
ordinary conversation. For example, suppose you were to slide an object of mass 150 kilograms
off a table which is one meter high and carry the object to another table 50 meters away, keeping the object always exactly one meter above ground. The physical concept
of work would indicate that the force exerted by your arms did no work
during this project. The reason for this definition is that even
though your arms exerted considerable force on the object, the direction of motion was at right angles to the force they
exerted. The only part of a force which does work in the sense of physics is
the component of the force in the direction of motion.

Work is defined to be the magnitude of the component of
this force times the distance over which it acts, when the
component of force points in the direction of motion. In the case where the force points in exactly the opposite direction of motion
work is given by $(-1) $
times the magnitude of this component times the distance.
Thus the work done by a force on an object as
the object moves from one point to another is a measure of the extent to
which the force contributes to the motion. This is illustrated in the
following picture in the case where the given force contributes to the
motion.

\begin{center}
\begin{tikzpicture}
\draw[blue, ->](0,0)--(4,1);
\draw[purple, ->](0,0)--(2,1.5);
\draw[red, ->](0,0)--(-0.5, 2);
\node[below] at (0,0){$P$};
\node[right] at (4,1){$Q$};
\node[left] at (-0.5,1.5){$\vect{F}_{\perp}$};
\node[left] at (1.75,1.5){$\vect{F}$};
\node[below] at (3,0.75){$\vect{F}_{||}$};
\node[right] at (0.75, 0.5){$\theta$};
\end{tikzpicture}
\end{center}

Recall that for any vector $\vect{u}$ in $\R^n$, we can write $\vect{u}$ as a sum
of two vectors, as in
\begin{equation*}
\vect{u} = \vect{u}_{||} + \vect{u}_{\perp}
\end{equation*}
For any force $\vect{F}$,
we can write this force as the sum of a vector in the direction of the motion and a vector
perpendicular to the motion. In other words,
\begin{equation*}
\vect{F} = \vect{F}_{||} + \vect{F}_{\bot}
\end{equation*}

In the above picture the force, $\vect{F}$ is applied to an object which moves
on the straight line from $P$ to $Q$. There are two vectors shown, $\vect{F}_{||}$ and $\vect{F}_{\bot }$ and the
picture is intended to indicate that when you add these two vectors you get
$\vect{F}$. In other words, $\vect{F} = \vect{F}_{||} + \vect{F}_{\bot}$. Notice that
 $\vect{F}_{||}$ acts in the direction of motion and
$\vect{F}_{\bot }$ acts perpendicular to the direction of motion. Only
$\vect{F}_{||}$ contributes to the work done by $\vect{F}$ on the object
as it moves from $P$ to $Q$. $\vect{F}_{||}$ is
called the component of the force in
\index{component of a force} the direction of motion. From trigonometry, you
see the magnitude of $\vect{F}_{||}$ should equal $\norm{\vect{F}
} \abs{\cos \theta}$. Thus, since $\vect{F}_{||}$ points
in the direction of the vector from $P$ to $Q$,
the total work done should equal
\begin{equation*}
\norm{\vect{F}} \norm{
\longvect{PQ}} \cos \theta =\norm{
\vect{F}} \norm{\vect{q}-\vect{p}} \cos \theta
\end{equation*}

Now, suppose the included angle had been obtuse. Then the work done by the force
$\vect{F}$ on the object would have been negative because $\vect{F}_{||}$
would point in $-1$ times the direction of the motion.  In this case, $\cos \theta $ would also be negative and
so it is still
the case that the work done would be given by the above formula. Thus from
the geometric description of the dot product given above, the work equals
\begin{equation*}
\norm{\vect{F}} \norm{\vect{q}-\vect{p}} \cos
\theta =\vect{F}\dotprod (\vect{q}-\vect{p})
\end{equation*}
This explains the following definition.

\begin{definition}{Work done on an object by a force}{work-done-by-force}
Let $\vect{F}$ be a force acting on an object which moves from the point
$P$ to the point $Q$, which have position vectors given by $\vect{p}$ and $\vect{q}$ respectively.
 Then the \textbf{work} done
\index{work} on the object by the given force equals $\vect{F}\dotprod (
\vect{q}-\vect{p})$.
\end{definition}

Consider the following example.

\begin{example}{Finding work}{finding-work}
Let $\vect{F}=
\begin{mymatrix}{rrr}
2 & 7 & -3
\end{mymatrix}^T$ Newtons. Find the work
done by this force in moving from the point $(1,2,3) $ to the
point $(-9,-3,4) $ along the straight line segment joining these
points where distances are measured in meters.
\end{example}

\begin{solution}
First, compute the vector $\vect{q} - \vect{p}$, given by
\begin{equation*}
\begin{mymatrix}{rrr}
-9 & -3 & 4
\end{mymatrix}^T
-
\begin{mymatrix}{rrr}
1 & 2 & 3
\end{mymatrix}^T
=
\begin{mymatrix}{rrr}
-10 & -5 & 1
\end{mymatrix}^T
\end{equation*}

According to Definition~\ref{def:work-done-by-force} the work done is
\begin{align*}
\begin{mymatrix}{rrr}
2 & 7 & 3
\end{mymatrix}^T
 \dotprod
\begin{mymatrix}{rrr}
-10 & -5 & 1
\end{mymatrix}^T
& =-20+(-35) +(
-3) \\
& =-58\text{ Newton meters}
\end{align*}
\end{solution}

Note that if the force had been given in pounds and the distance had been
given in feet, the units on the work would have been foot pounds. In
general, work has units equal to units of a force times units of a length.
Recall that $1$ Newton meter is equal to $1$ Joule.  Also notice that the work done by the force can be negative as in the
above example.


\section{Numerical implementation of the Bogoliubov-de Gennes equations}
In section \ref{sec:bogo}, we have shown the BdG equations, which are used to examine the stability of a state in BECs. To solve the Bogoliubov equations numerically requires searching for the solution to a generalised non-Hermitian eigenvalue problem. Thus, the system must be formally specified in matrix form. The derivative operators of $H_0$ are specified using second-order central differences, represented by
\begin{equation}
    \partial^2_i = \frac{U_{i+1,j} + U_{i-1,j} - 2U_{i,j}}{h^2}
\end{equation}
where $i$ is the respective dimension for the derivative, and $h$ is the step size. In matrix form, this becomes
\begin{equation}
    \mathbf{B} =
    \begin{bmatrix}
            -2      &   1    &    0   &  \cdots   &  \cdots   &  \cdots   & 0 \\
            1       &   -2   &    1   &           &           &     &  \vdots \\
            0       &    1   &   -2   & \ddots    &           &     &  \vdots \\
            \vdots  &        & \ddots & \ddots    & {\ddots}  &     &  \vdots \\
            \vdots  &        &        & \ddots    &    -2     &  1  &       0 \\
            \vdots  &        &        &           &     1     & -2  &       1 \\
            0       & \cdots & \cdots & \cdots    &     0     &  1  &      -2 \\
        \end{bmatrix}.
\end{equation}
where the number of rows and columns equal the number of elements along the dimension to be differentiated. To construct a multidimensional version, we take a Kronecker sum of the matrix $\mathbf{B}$ along each respective dimension of the system. This can be represented as
\begin{equation}
    \mathbf{B}_{i,j} = \mathbf{B}_i \oplus \mathbf{B}_j = \mathbf{B}_i \otimes \mathbf{I}_j + \mathbf{I}_i \otimes \mathbf{B}_j
\end{equation}
were $i,j$ are the indices of the respective dimensions, and $\mathbf{I}$ is the identity equal in size to dimensions $i,j$. With this operation, we can obtain a block diagonal matrix that represents the Laplacian operator as

\begin{equation}
    \nabla^2=
    \begin{bmatrix}
        \mathbf{B}    & \mathbf{I} &      0      &  \cdots      &      0       \\
        \mathbf{I}    & \mathbf{B} &  \ddots     &              &  \vdots      \\
        0             & \ddots     &  \ddots     &  \ddots      &      0       \\
        \vdots        &            &  \ddots     &  \ddots      &  \mathbf{I}  \\
        0             & \cdots     &      0      &  \mathbf{I}  &  \mathbf{B}
    \end{bmatrix}.
\end{equation}

Similarly, the angular momentum operator $L_z = i\hbar(x\partial_y + y\partial_x)$ can be defined in terms of first derivative matrices. The non-derivative operators require a reshaping into lexicographical indexing ($N$-D to 1-D), and will sit along the diagonal. As these systems have many more 0's than elements, it makes sense to store them in a sparse matrix format. Given the proposed system is not Hermitian, a generalised eigenvalue solver is required. For the sake of simplicity, these systems can be solved in MATLAB, using \textit{eigs}, which makes use of the Arnoldi (non-Hermitian) or Lanczos (Hermitian) algorithm for finding the specified number of eigenvectors and values.


\input{../qxfkindle}
\begin{document}

\chapter{Linear Algebra}

\textbf{Matrix Definitions:}
\Creset
\begin{align*}
\C~&A_{m\times n}=\begin{bmatrix}
      a_{11} &~a_{12} &~\cdots &~a_{1n}\\
      a_{21} &~a_{22} &~\cdots &~a_{2n}\\
      \vdots &~\vdots &~       &~\vdots\\
      a_{m1} &~a_{m2} &~\cdots &~a_{mn}
      \end{bmatrix}=[a_{ij}]_{m\times n }\\
   &\text{Abbreviate as}~A.~\text{For}~m=n,\,\text{write as}~A_n~\tag{Matrix}\\
   &\text{For}~B=[b_{ij}]_{m\times n},\,a_{ij}=b_{ij},\\
   &\text{write as}~A=B\tag{Matrix Equivalence}\\
   &\text{For}~\forall a_{ij}=0,~\text{write as}~O\tag{Zero Matrix}\\
   &\text{For}~m=n,\,\forall(i\neq j)\,a_{ij}=0,\,\forall(i=j)\,a_{ij}=1,\\
   &\text{write as}~E\tag{Identity Matrix}\\
   &\text{Recollect here.}\tag{Block Matrix}\\
\C~&A+B=[a_{ij} + b_{ij}]_{m\times n}\tag{Matrix Addition}\\
   &C=AB=[a_{ij}]_{m\times s}[b_{ij}]_{s\times n}=[c_{ij}]_{m\times n},\\
   &c_{ij}=\sum_{k=1}^{s}a_{ik}b_{kj}\tag{Matrix Multiplication}\\
   &kA=k[a_{ij}]_{m\times n}=[ka_{ij}]_{m\times n}\tag{Matrix Scalar Multiplication}\\
   &A=[a_{ij}]_{m\times n},\,A^T=[a_{ji}]_{n\times m}\tag{Transposition}\\
\C~&\text{For}~A_n,~\text{there is}~B~{that let}~AB=BA=E,\\
   &\text{write}~B~\text{as}~A^{-1}\tag{Inverse of Matrix}\\
   &\text{For}~A~\text{that has}~A^{-1},\\
   &\text{call it invertible matrix}\tag{Invertible Matrix}\\
\C~&\text{Elementary Row Operations,}\\
   &\text{matrix row write as}~R_i,\\
   &R_i\leftrightarrow R_j\tag{Row Switching}\\
   &kR_i\rightarrow R_i\tag{Row Multiplication}\\
   &kR_j + R_i\rightarrow R_i\tag{Row Addition}\\
\C~&\text{Elementary Matrix}\\
   &\text{Apply row switching to}~E\tag{\(E_{ij}\)}\\
   &\text{Apply row multiplication to}~E\tag{\(E_i(k)\)}\\
   &\text{Apply row addition to}~E\tag{\(E_{ij}(k)\)}\\
\C~&\text{For}~AA^T=A^TA=E\Leftrightarrow A^T=A^{-1}\tag{Orthogonal Matrix}\\
\C~&|A_n|=\begin{vmatrix}
      a_{11} &~a_{12} &~\cdots &~a_{1n}\\
      a_{21} &~a_{22} &~\cdots &~a_{2n}\\
      \vdots &~\vdots &~       &~\vdots\\
      a_{n1} &~a_{n2} &~\cdots &~a_{nn}
      \end{vmatrix}\\
   &\qquad=\sum_{j_1 j_2 \cdots j_n} (-1)^{\tau{(j_1 j_2 \cdots j_n)}}
      a_{1j_1}a_{2j_2}\cdots a_{nj_n}\tag{Determinant}\\
\C~&\text{Pick}~p~\text{rows and}~q~\text{columns from}~A,\\
   &\text{use the cross elements of these rows and columns}\\
   &\text{to construct a matrix.}\tag{Submatrix}\\
   &\text{For}~p=q=k,\\
   &\text{write as}~\ps{A}_k\tag{Principal Submatrix}\\
\C~&\text{For}~\exists{r},\,\forall|\ps{A}_r|\neq 0,\,\forall|\ps{A}_{r+1}|=0,\\
   &\text{write as}~r(A)\tag{Rank of Matrix}\\
\C~&M_{ij}=\begin{vmatrix}
      a_{11}&~\cdots&~a_{1,j-1}&~a_{1,j+1}&~\cdots&~a_{1n}\\
      \vdots&~      &~\vdots   &~\vdots   &~      &~\vdots\\
      a_{i-1,1}&~\cdots&~a_{i-1,j-1}&~a_{i-1,j+1}&~\cdots&~a_{i-1,n}\\
      a_{i+1,1}&~\cdots&~a_{i+1,j-1}&~a_{i+1,j+1}&~\cdots&~a_{i+1,n}\\
      \vdots&~      &~\vdots   &~\vdots   &~      &~\vdots\\
      a_{n1}&~\cdots&~a_{n,j-1}&~a_{n,j+1}&~\cdots&~a_{nn}
      \end{vmatrix}\tag{Minor}\\
\C~&A_{ij}=(-1)^{i+j}M_{ij}\tag{Cofactor}\\
\C~&A^* = \begin{bmatrix}
      A_{11} &~ A_{12} &~ \cdots &~ A_{1n}\\
      A_{21} &~ A_{22} &~ \cdots &~ A_{2n}\\
      \vdots &~ \vdots &~        &~ \vdots\\
      A_{n1} &~ A_{n2} &~ \cdots &~ A_{nn}\\
      \end{bmatrix}\tag{Cofactor Matrix}\\
\C~&\text{For}~AA^*=A^*A=E\Leftrightarrow A^*=A^{-1}\tag{Unitary Matrix}
\end{align*}

\textbf{Vector:}
\Creset
\begin{align*}
\C~&\text{For 1 column matrix}~A_{n\times 1},\\
   &\text{call it column vector, or simply vector},\\
   &\text{write it as}~\V{\alpha}\tag{n-dimensional Vector}\\
   &\text{For all zero element vector,}\\
   &\text{write as}~\V{0}\tag{Zero Vector}\\
\C~&\text{Vector Operation}\\
   &\text{For the rest of this \textbf{Vector} section,}\\
   &\V{\alpha} = \begin{bmatrix}
      a_1\\a_2\\\vdots\\a_n
   \end{bmatrix};\;\V{\beta} = \begin{bmatrix}
      b_1\\b_2\\\vdots\\b_n
   \end{bmatrix};\;k_1,k_2,\cdots,k_n \in \V{Z}\\
   &\V{\alpha} + \V{\beta} = \begin{bmatrix}
      a_1 + b_1\\a_2 + b_2\\\vdots\\a_n + b_n
   \end{bmatrix}\tag{Addition}\\
   &k\V{\alpha} = \begin{bmatrix}
      k a_1\\k a_2\\\vdots\\k a_n
   \end{bmatrix}\tag{Scalar Multiplication}\\
   &\V{\alpha}^T\V{\beta} = \begin{bmatrix}
      a_1~a_2~\cdots~a_n
   \end{bmatrix}\begin{bmatrix}
      b_1\\b_2\\\vdots\\b_n
   \end{bmatrix}\\
   &= a_1b_1 + a_2b_2 + \cdots + a_nb_n\tag{Dot/Inner Product}\\
\C~&\V{\beta} = k_1\V{\alpha}_1 + k_2\V{\alpha}_2 + \cdots + k_n\V{\alpha}_n\tag{Linear Combination \& Linear Representation}\\
\C~&\V{\alpha}
\end{align*}

\textbf{Elementary Matrix properties:}
\begin{align*}
\Creset
\C~&E_{ij}^{-1}=E_{ij}\\
\C~&|E_{ij}|=-1\Rightarrow |E_{ij}A|=-|A|\N\\
\Creset
\C~&E_i^{-1}(k)=E_{i}(1/k)\\
\C~&E_i(k)~\text{and}~E_i^{-1}(k)~\text{are diagonal matrices}\\
\C~&|E_i(k)|=k\Rightarrow |E_i(k)A|=k|A|\N\\
\Creset
\C~&E_{ij}(k)~\text{is kind of shear mapping}\\
\C~&E_{ij}^{-1}(k)=E_{ij}(-k)\\
\C~&E_{ij}(k)~\text{and}~E_{ij}^{-1}(k)~\text{are triangular matrices}\\
\C~&|E_{ij}(k)|=1\Rightarrow |E_{ij}(k)A|=|A|\N
\end{align*}

\textbf{Determinant properties:}
\begin{align*}
\Creset
&\C~\text{row/column exchange update sign}\\
&\C~\text{common divisor can be extracted}\\
&\C~\text{row/column can be splitted}\\
&\C~\text{multiply\&plus equivalent}\\
&\C~|A^T|=|A|\N
\end{align*}

\textbf{Special forms' properties:}
\Creset
\begin{align*}
\C~&\text{Triangular form,}\,|A| = \prod_{i=1}^n{a_{ii}}\\
   &\text{Secondary coefficient:}\:(-1)^{\frac{n(n-1)}{2}}\\
\C~&\text{Laplace form,}\,\begin{vmatrix}A_m&~O\\O&~B_n\end{vmatrix} = |A|\cdot|B|\\
   &\text{Secondary coefficient:}\:(-1)^{mn}\\
\C~&\text{Vandermonde form,}\\
   &|A|=\begin{vmatrix}
   1     &~ 1     &~ \cdots &~ 1\\
   x_1   &~ x_2   &~ \cdots &~ x_n\\
   x_1^2 &~ x_2^2 &~ \cdots &~ x_n^2\\
   \vdots&~ \vdots&~        &~ \vdots\\
   x_1^{n-1} &~ x_2^{n-1} &~ \cdots &~ x_n^{n-1}
   \end{vmatrix} = \prod_{1\leq j\leq i\leq n}{(x_i - x_j)}\\
   &\text{Another style to represent}~A:\\
   &A_{i,j} = x_j^{i-1},\,(1\leq i,j \leq n)\N
\end{align*}

\textbf{Useful formulas:}
\begin{align*}
\Creset
\C~&AA^*=A^*A=|A|E\\
\C~&|A| = \sum_{k=1}^{n} a_{ik} A_{ik},\,\text{true for}~i=1,2,\cdots,n\\
   &\text{Above column form is true}\\
\C~&\sum_{k=1}^{n} a_{ik} A_{jk} = 0,\,(i\neq j),~\text{column form is true}\\
\C~&|kA_n| = k^n|A|\\
\C~&|A_nB_n| = |A|\cdot|B|\\
\C~&|A^*_n| = |A|^{n-1}\\
\C~&|A^{-1}_n|=|A|^{-1}\\
\C~&|A_n|=\prod_{i=1}^{n}\lambda_i,\,(\lambda_i~\text{is eigenvalue of}~A_n)\\
\C~&A\sim B \Rightarrow |A| = |B|\N
\end{align*}

\end{document}


% !TEX root = main.tex

Expectation is to random variables what probability is to events.
\bit
\it Probability quantifies the `size' of a random event.
\it Expectation quantifies the `size' of a random variable.
\eit

\subsubsection*{Elementary probability}
Let $\Omega$ be a finite sample space and let $p:\Omega\to[0,1]$ be a probability mass function on $\Omega$. In elementary probability the \emph{expectation} of a random variable $X:\Omega\to\R$ is defined to be
\[
\expe(X) = \sum_{\omega\in\Omega} X(\omega)\prob(\omega)
\]
This is a sum over the \emph{domain} of $X$. Expectation can also be defined over the \emph{range} of $X$.

%\subsubsection*{Frequentist model}
%Let $\Omega$ be the sample space of a random experiment which can be repeated many times under the same conditions, let $X$ be a simple random variable on $\Omega$, and let $\{x_1,x_2,\ldots,x_n\}$ be its range.
%
%Suppose we repeat the experiment $N$ times. Let $N(\omega)$ be the number of times outcome $\omega$ is observed. record the sequence of values $\xi_1,\xi_2,\ldots,\xi_N$ taken by $X$. Let $N_i$ be the number of times the event $\{X=x_i\}$ is observed. Under the \emph{frequentist model}, the sample mean of the observations satisfies
%
%\[
%\bar{X}
%	= \frac{1}{N}\sum_{j=1}^{N} \xi_j
%	= \frac{1}{N}\sum_{i=1}^{n} x_i N_i
%	= \sum_{i=1}^{n} x_i \left(\frac{N_i}{N}\right)
%	\longrightarrow \sum_{i=1}^{n} x_i P(X=x_i) \quad\text{as $N\to\infty$}.
%\]
%
%Suppose we repeat the experiment $N$ times and record the sequence of values $\xi_1,\xi_2,\ldots,\xi_N$ taken by $X$. Let $N_i$ be the number of times the event $\{X=x_i\}$ is observed. Under the \emph{frequentist model}, the sample mean of the observations satisfies
%\[
%\bar{X}
%	= \frac{1}{N}\sum_{j=1}^{N} \xi_j
%	= \frac{1}{N}\sum_{i=1}^{n} x_i N_i
%	= \sum_{i=1}^{n} x_i \left(\frac{N_i}{N}\right)
%	\longrightarrow \sum_{i=1}^{n} x_i P(X=x_i) \quad\text{as $N\to\infty$}.
%\]
%
%This motivates the following definition of expectation for simple random variables.

%-------------------------------------------------
\section{Simple variables}\label{sec:expe-simple}

\begin{definition}[Expectation of simple random variables]
%Let $X$ be a simple random variable and $\{x_1,x_2,\ldots,x_n\}$ be its range. The expectation of $X$ is
The expectation of a simple random variable $X:\Omega\to\R$ is
\[
%\expe(X) = \sum_{i=1}^n x_i \prob(X=x_i).
\expe(X) = \sum_{i=1}^n x_i f(x_i).
\]
where $f$ is the PMF of $X$ and $\{x_1,x_2,\ldots,x_n\}$ is the range of $X$.
\end{definition}

In particular, the expectation of an indicator variable $I_A:\Omega\to\R$ is
\[
\expe(I_A) = \prob(A)
\]
which shows the connection between an event and its indicator function.

%-----------------------------
%\subsection{Properties}
%Before deriving the properties of expectation for simple random variables we need the following.

% definition: positivity & domination
\begin{definition}
Let $X$ and $Y$ be random variables.
\ben
\it We say that $X$ is \emph{non-negative} if $X(\omega)\geq 0$ for all $\omega\in\Omega$. This is denoted by $X\geq 0$.
%\it We say that $X$ is \emph{non-negative} (denoted by $X\geq 0$) if $X(\omega)\geq 0$ for all $\omega\in\Omega$.
\it We say that $X$ \emph{dominates} $Y$ if $X(\omega)\geq Y(\omega)$ for all $\omega\in\Omega$. This is denoted by $X\geq Y$.
\een
\end{definition}

% theorem: properties
\begin{theorem}[Properties of expectation for simple random variables]
Let $X$ and $Y$ be simple random variables.
\ben
\it \textbf{Linearity}.  For every $a,b\in\R$,\ \ $\expe(aX+bY) = a\expe(X) + b\expe(Y)$.
\it \textbf{Positivity}. If $X\geq 0$ then $\expe(X)\geq 0$.
\it \textbf{Monotonicity}. If $X\geq Y$ then $\expe(X)\geq\expe(Y)$.
\een
\end{theorem}

\begin{proof}
\ben
\it % << (ii)
\textbf{Linearity}. Let $\{x_1,x_2,\ldots,x_m\}$ be the range of $X$ and $\{y_1,y_2,\ldots,y_n\}$ be the range of $Y$. We define two partitions $\{A_1,A_2,\ldots,A_m\}$ and $\{B_1,B_2,\ldots,B_n\}$ of the underlying sample space $\Omega$ by
\[
A_i = \{\omega:X(\omega)=x_i\} \quad\text{and}\quad B_j = \{\omega:Y(\omega)=y_j\}.
\]
These can be combined to produce another partition of $\Omega$:
\[
\{A_i\cap B_j: i=1,2,\ldots,m;\ j=1,2,\ldots,n\}.
\]
The composite variable $aX + bY$ takes the value $ax_i+by_j$ on the set $A_i\cap B_j$. Because there are only finitely many such sets, it follows that $aX+bY$ is also a simple random variable, so its expectation is given by
\begin{align*}
\expe(aX+bY)
	& = \sum_{i=1}^m\sum_{j=1}^n (ax_i + by_j) \prob(A_i\cap B_j) \\
	& = a\sum_{i=1}^m x_i \sum_{j=1}^n \prob(A_i\cap B_j) + b\sum_{j=1}^n y_j \sum_{i=1}^m \prob(A_i\cap B_j).
\end{align*}
By the additivity of probability measures:
\bit
\it $\{A_i\cap B_j\}_{j=1}^n$ is a partition of $A_i$, so $\sum_{j=1}^n \prob(A_i\cap B_j) = \prob(A_i)$.
\it $\{A_i\cap B_j\}_{i=1}^m$ is a partition of $B_j$, so $\sum_{i=1}^m \prob(A_i\cap B_j) = \prob(B_j)$.
\eit
Hence,
\[
\expe(aX+bY)
	= a\sum_{i=1}^n x_i\prob(A_i) + b\sum_{j=1}^m y_j\prob(B_j)
	= a\expe(X) + b\expe(Y).
\]

\it % << (ii)
\textbf{Positivity}. If $X(\omega)\geq 0$ for all $\omega$ we must have that each $x_i\geq 0$. Thus $\expe(X) = \sum_{i=1}^n x_i \prob(X=x_i)$ is a sum of non-negative terms, so $\expe(X)\geq 0$.

\it % << (iii)
\textbf{Monotonicity}. If $X\geq Y$ then $X-Y\geq 0$, so
\bit
\it by positivity, $\expe(X-Y)\geq 0$;
\it by linearity, $\expe(X)-\expe(Y)\geq 0$, so $\expe(X)\geq\expe(Y)$.
\eit
\een
\end{proof}

% expectation of transformed variables
For a transformed variable, we need not compute its PMF to compute its expectation. The following result is sometimes called the \emph{law of the unconscious statistician}.

\begin{theorem}[Expectation of transformed simple random variables]\label{thm:expe-simple-transformed}
If $X$ be a simple random variable and $g:\R\to\R$ is a transformation, the expected value of the transformed variable $g(X)$ is
\[
\expe\big[g(X)\big] = \sum_{i=1}^{n} g(x_i) f(x_i).
\]
\end{theorem}
\begin{proof}
Let $Y=g(X)$. Because $X$ is simple, $Y$ must also be simple. Let $\{x_1,\ldots,x_m\}$ and $\{y_1,\ldots,y_n\}$ be the range of $X$ and $Y$ respectively and let $C_j$ be the set containing those $x_i$ that are mapped to $y_j$:
\[
C_j  = \{x_i:g(x_i)=y_j\} \qquad\text{for $i=1,2,\ldots,m$.}
\]
Clearly, $\prob(Y=y_j) = \displaystyle\sum_{x_i\in C_j}\prob(X=x_i)$, so
\begin{align*}
\expe(Y)
	& = \sum_{j=1}^n y_j\prob(Y=y_j) = \sum_{j=1}^n y_j\left(\sum_{x_i\in C_j} \prob(X=x_i)\right) \\
	& = \sum_{j=1}^n \sum_{x_i\in C_j} y_j\prob(X=x_i) \\
	& = \sum_{j=1}^n \sum_{x_i\in C_j} g(x_i)\prob(X=x_i) \text{\quad because $y_j=g(x_i)$ whenever $x_i\in C_j$,}\\
	& = \sum_{i=1}^m g(x_i)\prob(X=x_i),
\end{align*}
where the last equality follows because $\{C_1,C_2,\ldots,C_m\}$ is a partition of $\{x_1,\ldots,x_n\}$.
\end{proof}


\subsection{Ordinary vs. Partial}
\noindent
Another useful classification is by the type. There are two broad types of differential equation: ordinary and partial.
\begin{definition}
	An ordinary differential equation (ODE) is a differential equation where only 1 independent variable involved in the derivatives.
\end{definition}
\begin{definition}
	A partial differential equation (PDE) is a differential equation where 2 or more independent variables are involved in the derivative.
\end{definition}
\noindent
Below is a table of types and equation numbers.
\begin{table}[H]
	\centering
	\begin{tabular}{c|c}
		Type & Equation Number \\
		\hline
		ODE &  1, 2, 5, 6, 7, 8, 10 \\
		PDE & 3, 4, 9 \\
	\end{tabular}
\end{table}
\noindent
We'll work mostly with ODEs.


\chapter{Expectation Maximization}

\section{EM-algorithm}
Lets first explore the data and let’s motivate the reason of why you would use a mixture model by using an example. Suppose we have the following density plot \ref{5quakes_depth_dist} and \ref{5faith_wait_dist}:

\begin{figure}[h]
\centering
\includegraphics[scale=0.75, keepaspectratio]{ex5/5quakes_depth_dist.png}
\caption{Density plot for  variable \texttt{quakes\$depth}}
\label{5quakes_depth_dist}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[scale=0.75, keepaspectratio]{ex5/5faith_wait_dist.png}
\caption{Density plot for  variable \texttt{faith\$waiting}}
\label{5faith_wait_dist}
\end{figure}

We can immediately see that the resulting distribution appears to be bi-modal (i.e. there are two bumps in both the plots) suggesting that these in case of both the data, they might be coming from two different sources.

Putting the the \texttt{quakes} data into context suggests that the earthquakes in Fiji may be coming from two different sub-populations, depending upon the depth of the region. Similarly, putting the the \texttt{faithful} data into context suggests that the eruption times may be coming from two different sub-populations. There could be several reasons for this. For instance, maybe at different times of the year the geyser eruptions are more frequent. We can probably take an intuitive guess as to how we could split this data.

For instance, we can observe in Figure \ref{5quakes_depth_dist}, there likely is a sub-population with a mean depth of $\sim 80$ with some variance around this mean (red vertical line in figure \ref{5quakes_depth_dist}.) Another population with a mean depth of $\sim 565$ with again some variance around this mean (blue vertical line in figure \ref{5quakes_depth_dist}). Similarly, as we can observe in Figure \ref{5faith_wait_dist}, there likely is a sub-population with a mean eruption of $\sim 53$ with some variance around this mean (red vertical line in figure \ref{5faith_wait_dist}.) Another population with a mean eruption of $\sim 80$ with again some variance around this mean (blue vertical line in figure \ref{5faith_wait_dist}).

\section{Gaussian Mixture Model}
What we have done is a naive attempt at trying to group the data into sub-populations or clusters.
But surely there must be some objective and “automatic” way of defining these clusters? This is where mixture models come in by providing a “model-based approach” to clustering through the use of statistical distributions.

A mixture model consist of a mixture of distributions. The first thing we need to do when performing mixture model clustering is to determine what type of statistical distribution we want to use for the components.

In our exercise we will use one of the most common statistical distributions used for mixture model clustering which is the Gaussian/Normal Distribution. When Gaussian distributions are used for mixture model clustering, they are referred to as \emph{Gaussian Mixture Models (GMM).} As it turns out, our earlier intuition on where the means and variance of the sub-population in the plot above is a perfect example of how we could apply a GMM. Specifically, we could try to represent each sub-population as its own distribution (aka. mixture component). The entire set of data could then be represented as a mixture of 2 Gaussian distributions (aka. 2-component GMM)

 The procedures is based on the iterative \emph{expectation maximization (EM) algorithm.} The following two points are important to note here. First, the EM algorithm is an iterative procedure, and the time required for it to reach convergence, if it converges at all depends strongly on the problem to which it is applied.  The second key point is that because it is an iterative procedure, the EM algorithm requires starting values for the parameters, and algorithm performance can depend strongly on these initial values.

 In our exercise, for both the data sets, we are going take initial values,
 $$\theta^ 0 =(\mu_{1}^{(0)},\mu_{2}^{(0)},\sigma_{1}^{(0)}, \sigma_{2}^{(0)},p^{(0)}) =(m-sd/2, m + sd/2, sd, sd, 0.5)$$
where $m$ is the mean of all observation and $sd$ the standard deviation.

Observe the following plots \ref{5quakes_depth_mix} and \ref{5faith_wait_mix}. We have built a 2-component GMM. So how do we interpret figures \ref{5quakes_depth_mix} and \ref{5faith_wait_mix}? It’s actually quite simply, the red and blue lines simply indicate $2$ different fitted Gaussian distributions.
\begin{figure}[h]
\centering
\includegraphics[scale=0.75, keepaspectratio]{ex5/5quakes_depth_mix.png}
\caption{Density plot for  GMM with barplot for \texttt{quakes\$depth}}
\label{5quakes_depth_mix}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[scale=0.75, keepaspectratio]{ex5/5faith_wait_mix.png}
\caption{Density plot for  GMM with barplot for \texttt{faith\$waiting}}
\label{5faith_wait_mix}
\end{figure}

\section{Stopping criteria}
In the EM-algorithm we have used the (log-)likelihood function, which is a non-decreasing sequence. Also, is strictly increasing unless $\theta^{(k)} = \theta^{(k-1)}.$ We can observe this fact for the two data sets.

\begin{figure}[h]
\centering
\includegraphics[scale=0.75, keepaspectratio]{ex5/5quakes_depth_iter.png}
\caption{Values of the log-likelihood function per iteration for \texttt{quakes\$depth}}
\label{5quakes_depth_iter}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[scale=0.75, keepaspectratio]{ex5/5faith_wait_iter.png}
\caption{Values of the log-likelihood function per iteration for  \texttt{faith\$waiting}}
\label{5faith_wait_iter}
\end{figure}

We can this be used to formulate a stopping criterion for the EM-Algorithm. We can decide a threshold on $\theta^{(k)} - \theta^{(k-1)}$, if it is below the threshold then, we can stop the algorithm. For eg. we have take the threshold as $0.00001$. In the case of \texttt{quakes\$depth} algorithm stopped after $12$ iterations, where as in the case of \texttt{faith\$waiting} it stopped after $9$ iterations.  This just assures that the improvement in the maximization step is small and the algorithm can be stopped.


\section*{Part B, Problem 3: Kernel SVM (15 pts) (Adams)}
After Homework 2, you must be very familiar with SVM.
In this question, we are considering its Kernel version:
\begin{align*}
\min_{w\in \mathbb{R}^d, b, \xi_i\in\mathbb{R},i=1,2,...,n} &\quad \frac{1}{2}||w||_2^2 + C\sum_{i=1}^n\xi_i\\
\text{s.t.} &\quad y^{(i)}(w^\mathsf{T}\phi(x^{(i)}) + b) \geq 1-\xi_i,~~  \forall i = 1,\ldots, n\\
 &\quad \xi_i \geq 0, ~~ \forall i = 1,\ldots, n
\end{align*}
where $x^{(i)}\in\mathbb{R}^p, i=1,2,...,n$ is the original training data coming along with the label $y^{(i)}\in\{-1,1\}$, $C>0$ is a constant and $\phi: \mathbb{R}^p \rightarrow \mathbb{R}^d$ is a mapping function that maps the original data to a new space. Generally speaking, $d>p$ (In fact, $d$ can be $+\infty$). Please answer the following questions. Note that the questions with complexity could be answered with big-O notation.

\begin{enumerate}
\item \textbf{[6 points]} First write down the Lagrangian of the above problem and then derive the dual problem step by step.
\\\textbf{Answer:}\\

$$J(\vec{w},b,\vec{\xi},\vec{\alpha},\vec{\gamma}) = \frac{1}{2}\vec{w}^T\vec{w}+C\sum_{i=1}^{n}\xi_i + \sum_{i=1}^{n} \alpha_i(1-\xi_i-y^{(i)}(\vec{w}^T\phi(\vec{x}^{(i)})+b)) - \sum_{i=1}^{n}\gamma_i\xi_i~~~~(\alpha_i\geq0,~\gamma_i\geq0)$$

Because,
\begin{equation}
\nonumber
\begin{array}{rcl}
\left.\frac{\partial J}{\partial \vec{w}}\right|_{\vec{w}^*} & = & \vec{w}^*-\sum_{i=1}^{n}\alpha_iy^{(i)}\phi(\vec{x}^{(i)}) = 0 \\
									& \Rightarrow & \vec{w}^* = \sum_{i=1}^{n}\alpha_iy^{(i)}\phi(\vec{x}^{(i)}) \\

\left.\frac{\partial J}{\partial b}\right|_{b^*} & = & -\sum_{i=1}^{n}\alpha_iy^{(i)} = 0 \\
& \Rightarrow & \sum_{i=1}^{n}\alpha_iy^{(i)} = 0 \\

\left.\frac{\partial J}{\partial \xi_i}\right|_{\xi_i^*} & = & C-\alpha_i-\gamma_i = 0 \\
& \Rightarrow & C = \alpha_i+\gamma_i \\
\end{array}
\end{equation}
Therefore,
$$J(\vec{w}^*,b^*,\vec{\xi}^*,\vec{\alpha},\vec{\gamma}) = -\frac{1}{2}\sum_{i=1}^{n}\sum_{j=1}^{n}\alpha_i\alpha_j y^{(i)} y^{(j)} \phi^T(\vec{x}^{(i)})\phi(\vec{x}^{(j)})+\sum_{i=1}^{n}\alpha_i$$
Therefore, the dual form is:
$$\max_{\vec{\alpha},\vec{\gamma}} -\frac{1}{2}\sum_{i=1}^{n}\sum_{j=1}^{n}\alpha_i\alpha_j y^{(i)} y^{(j)} \phi^T(\vec{x}^{(i)})\phi(\vec{x}^{(j)})+\sum_{i=1}^{n}\alpha_i$$
s.t.
$$\sum_{i=1}^{n}\alpha_i y^{(i)} = 0$$
$$0\leq\alpha_i\leq C$$

\item \textbf{[9 points, (3 for each subproblem)]} Suppose we have obtained the solution of the dual problem, which is denoted as $\alpha_i^*$ for $i=1,2,...,n$. Please find out the corresponding primal solution $w^*$ and $b^*$ in the course slides. Given a test data point $z\in \mathbb{R}^p$,
\\\textbf{Answer:}\\
$$\vec{w}^* = \sum_{i=1}^{n}\alpha_i^*y^{(i)}\phi(\vec{x}^{(i)})$$
$$b^* = y^{(k)}-\sum_{i=1}^{n}\alpha_i^*y^{(i)}\phi^T(\vec{x}^{(i)})\phi(\vec{x}^{(k)})~~~~(for~any~k~where~C>\alpha_k>0)$$

\begin{itemize}
\item What is the time complexity of making the classification decision if the mapping is given by $$\phi(x) = (\underbrace{x_p^2, x_{p-1}^2, ..., x_1^2}_{p}, \underbrace{ \sqrt{2}x_{p}x_{p-1},..., \sqrt{2}x_{p}x_{1}}_{p-1}, \underbrace{\sqrt{2}x_{p-1}x_{p-2},...,\sqrt{2}x_{p-1}x_{1}}_{p-2},...,\sqrt{2}x_{2}x_{1}, \sqrt{2c}x_{p},...,\sqrt{2c}x_{1}, c)^\mathsf{T}$$
with a constant $c>0$ and we need to compute it from scratch?
\\\textbf{Answer:}\\
\begin{itemize}
	\item For $\phi(\vec{x}^{(i)})$: the number of multiply operations is $n\frac{(p+2)(p+1)}{2}$
	\item For $\vec{w}^*$: the number of multiply operations is $n\frac{(p+2)(p+1)}{2}$
	\item For $b^*$: the number of multiply operations is $2n\frac{(p+2)(p+1)}{2}$
	\item For $\phi(\vec{z})$: the number of multiply operations is $\frac{(p+2)(p+1)}{2}$
	\item For $sgn(\vec{w}^*\cdot\phi(\vec{z})+b)$:	the number of multiply operations is $\frac{(p+2)(p+1)}{2}$
	\item Therefore, the time complexity is $O(np^2)$
\end{itemize}

\item  Let $K(u,v) = \phi(u)^\mathsf{T}\phi(v)$ where $\phi(\cdot)$ has the same definition as above. Please give a compact form of $K(u,v)$. What is the time complexity of making the classification decision if we directly compute $K(\cdot, \cdot)$?
\\\textbf{Answer:}\\
$$K(\vec{u},\vec{v}) = \phi(\vec{u})^T\phi(\vec{v}) = (\vec{u}^T\vec{v}+C)^2$$
\begin{itemize}
	\item For $\vec{w}^*\cdot\phi(\vec{z})=\sum_{i=1}^{n}\alpha_iy_iK(\vec{z},\vec{x}_i)$: the number of multiply operations is $n(p+2)$
	\item For $b^*=y_k-\sum_{i=1}^{n}\alpha_iy_iK(\vec{x}_k,\vec{z})$: the number of multiply operations is $n(p+2)$
	\item For $sgn(\vec{w}^*\cdot\phi(\vec{z})+b)$:	the number of multiply operations is $0$
	\item Therefore, the time complexity is $O(np)$
\end{itemize}

\item Now please go back to the dual formulation you derived previously. To avoid repetitive computation, one can precompute all the inner products $K(x^{(i)},x^{(j)})=\phi(x^{(i)})^\mathsf{T}\phi(x^{(j)})$ before solving the dual problem. What is the space complexity of this approach and what might be a problem if $n$ is huge?
\\\textbf{Answer:}\\
The space complexity of $K(x^{(i)},x^{(j)})$ is $O(n^2)$, and this may lead memory overflow if $n$ is huge.

\end{itemize}
\end{enumerate}

\newpage


\color{black}
\subsection*{Inference with Markov models}

Recall the state transition diagram and transition probability matrix of a Markov chain ${\lambda_1}$ which models the behavior of an NPC who is ``patrolling the map''
\begin{center}
\includegraphics[width=0.4\textwidth]{dungeonMC1.pdf}
\qquad
\raisebox{2.5cm}{$\mat{P}_{\lambda_1} = \begin{bmatrix} 0.3 & 0.2 & 0.5 \\ 0.5 & 0.3 & 0.2 \\ 0.2 & 0.5 & 0.3 \end{bmatrix}$}
\end{center}

The probability of the event $E_1 = $ ``the NPC goes from room $C$ to room $B$ and then to room $A$'' amounts to
\begin{align*}
\prob{E_1} & = \cprob{X_1 = B}{X_0 = C} \cdot \cprob{X_2 = A}{X_1 = B}
\intertext{and the corresponding log-likelihood amounts to}
\mathcal{L} \bigl( E_1 \bigr) & =  \ln \cprob{X_1 = B}{X_0 = C} + \ln \cprob{X_2 = A}{X_1 = B}
\end{align*}

Compute both these values for the model $\lambda_1$. Round your results to \emph{two} decimal places.
\color{blue}
%%%%%
%%%%% enter your results after the '=' signs
%%%%%
\begin{equation*}
\prob{E_1} = 0.04
\end{equation*}
\begin{equation*}
\mathcal{L} \bigl( E_1 \bigr) = -3.22
\end{equation*}
%%%%%
%%%%%
%%%%%
\color{black}

Also for model $\lambda_1$, compute the probability and log-likelihood of the event $E_2 = $ ``the NPC goes from room $C$ to room $A$ and then to room $B$''
\color{blue}
%%%%%
%%%%% enter your results after the '=' signs
%%%%%
\begin{equation*}
\prob{E_2} = 0.25
\end{equation*}
\begin{equation*}
\mathcal{L} \bigl( E_2 \bigr) = -1.39
\end{equation*}
%%%%%
%%%%%
%%%%%
\color{black}
\newpage


\subsection{Projections}\label{ssec:projections}

It is sometimes important to find the component of a vector in a
particular direction. Consider the following picture:
\begin{center}
  \begin{tikzpicture}[scale=0.8,rotate=13]
    \draw[draw=green!50!black] (3,0.5) -- (2.5,0.5) -- (2.5,0);
    \draw[->, thick, red](0,0) -- node[left]{$\vect{v}$} (3,3.5);
    \draw[->, thick, blue](0,0) -- node[below, near end]{$\vect{u}$} (8,0);
    \draw[dashed] (3,3.5) -- (3,0);
    \draw[fill](0,0) circle [radius=2.25pt] node[below=1ex]{$0$};
    \draw[fill](3,0) circle [radius=2.25pt] node[below=1ex]{$P$};
    \draw[fill](3,3.5) circle [radius=2.25pt] node[right=1ex]{$Q$};
  \end{tikzpicture}
\end{center}
Here, $\vect{u}$ is a non-zero vector specifying a {\em direction},
and $\vect{v}$ is any vector. We have given the label $Q$ to the tip
of $\vect{v}$. The point $P$ lies at the place along $\vect{u}$ that
is closest to $Q$, or equivalently, such that $(0,P,Q)$ forms a right
triangle. The distance from $0$ to $P$ (measured positively in the
direction of $\vect{u}$) is called the \textbf{component}%
\index{component!of a vector}%
\index{vector!component} of $\vect{v}$ in the direction of $\vect{u}$,
and is denoted $\comp_{\vect{u}}(\vect{v})$. The vector
$\longvect{0P}$ is called the \textbf{projection}%
\index{projection!vector to vector}%
\index{vector!projection of} of $\vect{v}$ onto $\vect{u}$, and is
denoted $\proj_{\vect{u}}(\vect{v})$. We wish to find formulas for
these quantities.

Let $\theta$ be the included angle between $\vect{v}$ and
$\vect{u}$. From trigonometry, considering the right triangle
$(0,P,Q)$, we know that
\begin{equation*}
  \cos \theta = \frac{|0P|}{|0Q|} = \frac{|0P|}{\norm{\vect{v}}},
\end{equation*}
and therefore
\begin{equation}\label{eqn:component-1}
  |0P| = \norm{\vect{v}} \cos \theta.
\end{equation}
On the other hand, from Proposition~\ref{prop:dot-product-angle},  we
have
\begin{equation*}
  \vect{u}\dotprod \vect{v}=\norm{\vect{u}} \norm{\vect{v}} \cos \theta,
\end{equation*}
and therefore
\begin{equation}\label{eqn:component-2}
  \norm{\vect{v}} \cos \theta = \frac{\vect{u}\dotprod \vect{v}}{\norm{\vect{u}}}.
\end{equation}
Putting equations {\eqref{eqn:component-1}} and
{\eqref{eqn:component-2}} together, we obtain the desired formula for
the component of $\vect{v}$ in the direction of $\vect{u}$:
\begin{equation}\label{eqn:component}
  \comp_{\vect{u}}(\vect{v})
  = |0P|
  = \frac{\vect{u}\dotprod \vect{v}}{\norm{\vect{u}}}.
\end{equation}
Note that it is possible for this quantity to be negative; this
happens when the angle between $\vect{v}$ and $\vect{u}$ is obtuse.
In this case, $\vect{v}$ will have a negative component along
$\vect{u}$.

The vector $\proj_{\vect{u}}(\vect{v})=\longvect{0P}$ can now be
computed by re-scaling $\vect{u}$ to the correct length. Specifically,
we first normalize $\vect{u}$ by dividing it by its own length, and
then multiply by $|0P|$. In formulas:
\begin{equation}
  \proj_{\vect{u}}(\vect{v})
  = \longvect{0P}
  = \frac{|0P|}{\norm{\vect{u}}}\vect{u}
  = \frac{\vect{u}\dotprod \vect{v}}{\norm{\vect{u}}^2}\,\vect{u}
  = \frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod \vect{u}}\,\vect{u}.
\end{equation}
The following definition summarizes what we have just found.

\begin{definition}{Vector projection}{projection}
  Let $\vect{u}$ be a non-zero vector and $\vect{v}$ any vector. Then
  the \textbf{component of $\vect{v}$ in the direction of $\vect{u}$}%
  \index{vector!component} is defined to be the scalar
  \begin{equation*}
    \comp_{\vect{u}}(\vect{v})
    = \frac{\vect{u}\dotprod \vect{v}}{\norm{\vect{u}}}.
  \end{equation*}
  The \textbf{projection of $\vect{v}$ onto $\vect{u}$}%
  \index{vector!projection of} is defined to be the vector
  \begin{equation*}
    \proj_{\vect{u}}(\vect{v}) =
      \frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod \vect{u}}\, \vect{u}
    =
    \frac{\vect{u}\dotprod \vect{v}}{\norm{\vect{u}}^2}\,\vect{u}.
  \end{equation*}
  These two operations are also called the \textbf{scalar projection}%
  \index{scalar projection} and \textbf{vector projection}%
  \index{vector projection}, respectively.
\end{definition}

\begin{example}{Find the projection of one vector onto another}{vector-projection}
  Find $\proj_{\vect{u}}(\vect{v})$ if
  \begin{equation*}
    \vect{u}=
    \begin{mymatrix}{r}
      2 \\
      3 \\
      -4
    \end{mymatrix}
    \quad\mbox{and}\quad
    \vect{v}=
    \begin{mymatrix}{r}
      1 \\
      -2 \\
      1
    \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  We can use the formula provided in Definition~\ref{def:projection}
  to find $\proj_{\vect{u}}(\vect{v})$.  First, compute
  $\vect{u} \dotprod \vect{v}$.  This is given by
  \begin{equation*}
    \begin{mymatrix}{r}
      2 \\
      3 \\
      -4
    \end{mymatrix}
    \dotprod
    \begin{mymatrix}{r}
      1 \\
      -2 \\
      1
    \end{mymatrix}
    = (2)(1) + (3)(-2) + (-4)(1)
    = -8.
   \end{equation*}
   Similarly, $\vect{u} \dotprod \vect{u}$ is given by
   \begin{equation*}
     \begin{mymatrix}{r}
       2 \\
       3 \\
       -4
     \end{mymatrix}
     \dotprod
     \begin{mymatrix}{r}
       2 \\
       3 \\
       -4
     \end{mymatrix}
     = 2^2 + 3^2 + (-4)^2
     = 29.
  \end{equation*}
  Therefore, the projection is equal to
  \begin{equation*}
    \proj_{\vect{u}}(\vect{v})
    =-\frac{8}{29}
        \begin{mymatrix}{r}
          2 \\
          3 \\
          -4
        \end{mymatrix}
    =
        \begin{mymatrix}{r}
          -16/29 \\
          -24/29 \\
          32/29
        \end{mymatrix}.
  \end{equation*}
\end{solution}

An important application of projections is that every vector
$\vect{v}$ can be uniquely written as a sum of two orthogonal vectors,
one of which is a scalar multiple of some given non-zero vector
$\vect{u}$, and the other of which is orthogonal to $\vect{u}$.

\begin{theorem}{Decomposition into components}{decomposition-into-components}
  Let $\vect{u}$ be a non-zero vector, and let $\vect{v}$ be any
  vector. Then there exist unique vectors $\vect{v}_{\coll}$ and
  $\vect{v}_{\bot}$ such that
  \begin{equation}\label{projection}
    \vect{v}=\vect{v}_{\coll}+\vect{v}_{\bot}
  \end{equation}
  where $\vect{v}_{\coll}$ is a scalar multiple of $\vect{u}$,
  and $\vect{v}_{\bot}$ is orthogonal to $\vect{u}$.
\end{theorem}

\begin{proof}
  \begin{center}
    \begin{tikzpicture}[scale=0.7,rotate=-8]
      \draw[draw=green!50!black] (3,0.5) -- (2.5,0.5) -- (2.5,0);
      \draw[->, thick, blue](0,0) -- node[left]{$\vect{v}$} (3,3.5);
      \draw[->, thick, blue](0,0) -- node[below]{$\vect{u}$} (10,0);
      \draw[->, thick, red] (0,0) -- node[below]{$\vect{v}_{\coll}$} (3,0);
      \draw[->, thick, red] (3,0) -- node[right]{$\vect{v}_{\bot}$} (3,3.5);
      \draw[fill](0,0) circle [radius=2.25pt];
      \draw[fill](3,0) circle [radius=2.25pt];
      \draw[fill](3,3.5) circle [radius=2.25pt];
    \end{tikzpicture}
  \end{center}
  To show that such a decomposition exists, let
  \begin{equation*}
    \vect{v}_{\coll} = \proj_{\vect{u}}(\vect{v}) = \frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod \vect{u}}\, \vect{u},
  \end{equation*}
  and define $\vect{v}_{\bot} = \vect{v}-\vect{v}_{\coll}$. By
  definition, {\eqref{projection}} is satisfied, and
  $\vect{v}_{\coll}$ is a scalar multiple of $\vect{u}$. We must show
  that $\vect{v}_{\bot}$ is orthogonal to $\vect{u}$. For this,
  we verify that their dot product equals zero:
  \begin{eqnarray*}
    \vect{u}\dotprod\vect{v}_{\bot}
    &=& \vect{u}\dotprod(\vect{v}-\vect{v}_{\coll}) \\
    &=& \vect{u}\dotprod\vect{v}-\vect{u}\dotprod\vect{v}_{\coll} \\
    &=& \vect{u}\dotprod \vect{v}-\frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod\vect{u}}\,\vect{u}\dotprod \vect{u} \\
    &=& \vect{u}\dotprod\vect{v}-\vect{u}\dotprod \vect{v} \\
    &=& 0.
  \end{eqnarray*}
  To show uniqueness, suppose that {\eqref{projection}} holds and
  $\vect{v}_{\coll}= k \vect{u}$.  Taking the dot product of both
  sides of {\eqref{projection}} with $\vect{u}$ and using
  $\vect{u}\dotprod\vect{v}_{\bot}=0$, this yields
  \begin{equation*}
    \begin{array}{ll}
      \vect{u}\dotprod \vect{v} & = \vect{u}\dotprod( \vect{v}_{\coll}+\vect{v}_{\bot}) \\
                                & =  \vect{u} \dotprod k\vect{u} + \vect{u}\dotprod\vect{v}_{\bot} \\
                                & = k \norm{\vect{u}}^2,
    \end{array}
  \end{equation*}
  which implies
  $k =\vect{u}\dotprod \vect{v} / \norm{\vect{u}}^2$.  Thus there
  can be no more than one such vector $\vect{v}_{\coll}$. Since
  $\vect{v}_{\bot}$ must equal $\vect{v}-\vect{v}_{\coll}$, it follows
  that there can be no more than one choice for both
  $\vect{v}_{\coll}$ and $\vect{v}_{\bot}$, proving their uniqueness.
\end{proof}

\begin{example}{Decomposition into components}{decomposition-into-components}
  Decompose the vector $\vect{v}$ into $\vect{v}=\vect{a}+\vect{b}$
  where $\vect{a}$ is parallel to $\vect{u}$ and $\vect{b}$ is
  orthogonal to $\vect{u}$.
  \begin{equation*}
    \vect{v} = \begin{mymatrix}{r} -5\\3\\-5 \end{mymatrix},\quad
    \vect{u} = \begin{mymatrix}{r} 1\\2\\-2 \end{mymatrix}.
  \end{equation*}
\end{example}

\begin{solution}
  We can let $\vect{a}=\vect{v}_{\coll}$ and $\vect{b}=\vect{v}_{\bot}$
  as in the proof of Theorem~\ref{thm:decomposition-into-components}.
  Then
  \begin{equation*}
    \vect{a} ~=~ \vect{v}_{\coll}
    ~=~ \proj_{\vect{u}}(\vect{v})
    ~=~ \frac{\vect{u}\dotprod \vect{v}}{\vect{u}\dotprod \vect{u}}\, \vect{u}
    ~=~ \frac{11}{9}\, \begin{mymatrix}{r} 1\\2\\-2 \end{mymatrix}
    ~=~ \begin{mymatrix}{r} 11/9\\22/9\\-22/9 \end{mymatrix}
  \end{equation*}
  and
  \begin{equation*}
    \vect{b} ~=~ \vect{v}_{\bot}
    ~=~ \vect{v}-\vect{v}_{\coll}
    ~=~ \begin{mymatrix}{r} -5\\3\\-5 \end{mymatrix}
    - \begin{mymatrix}{r} 11/9\\22/9\\-22/9 \end{mymatrix}
    ~=~ \begin{mymatrix}{r} -56/9\\5/9 \\-23/9 \end{mymatrix}.
  \end{equation*}
\end{solution}


\documentclass[fleqn,leqno]{article}

% Defining page margins
\usepackage[top=1in, bottom=1.5in, left=1in, right=1in]{geometry}

\usepackage{mathtools}
\usepackage{amsmath}
\usepackage{empheq}
\usepackage{amssymb}

% argmin function
\DeclareMathOperator*{\argmin}{arg\,min}

\begin{document}

\setlength{\parindent}{0pt}

\section*{Auxiliary notation: subset of $k$-minimal values}

Given the set $X \subset \mathbb{N}$, and some $k \in \mathbb{N}$, for the sake of convenience let's introduce an auxiliary notation for the subset of $k$-minimal values: $min_k(X) \subseteq X$, such that $\forall a \in min_k(X), \forall b \in X \setminus min_k(X)$ it follows, that $a < b$, and $|min_k(X)|=k$ (in case if $|X| < k$, then $min_k(X) = X$).

\section*{K smallest edit distances}

Let $lev_{a,b}(i, j)$ be the set of $k$ smallest edit distances between the first $i$ characters of the string $a$ and the first $j$ characters of the string $b$. \\

Let's introduce the following sets:
\begin{equation}
\begin{split}
	      & I_{a,b}(i, j) := \{ d + 1 \mid d \in lev_{a,b}(i - 1, j) \}  && \text{The set of $k$ edit distances obtained through insertions} \\
	      & D_{a,b}(i, j) := \{ d + 1 \mid d \in lev_{a,b}(i, j - 1) \} && \text{The set of $k$ edit distances obtained through deletions} \\
	      & S_{a,b}(i, j) := \{ d + 1_{(a_i \neq b_j)} \mid d \in lev_{a,b}(i - 1, j - 1) \} && \text{The set of $k$ edit distances obtained through substitutions} \\
	      & E_{a,b}(i, j) := I_{a,b}(i, j) \cup D_{a,b}(i, j) \cup S_{a,b}(i, j) && \text{The set of all $3k$ edit distances}  \\
\end{split}
\end{equation}

Where $1_{(a_i \neq b_j)}$ is the indicator function equal to $0$ when $a_{i}=b_{j}$ and equal to $1$ otherwise. \\

Then the set of $k$ smallest edit distances between the first $i$ characters of the string $a$ and the first $j$ characters of the string $b$ is defined as follows:
\begin{equation}
\begin{cases}
	lev_{a,b}(i, j) := \{ max(i, j) \}  		& \text{when $i = 0$ or $j = 0$} \\
	lev_{a,b}(i, j) := min_k(E_{a,b}(i, j)) 	& \text{otherwise} \\
\end{cases}
\end{equation}

\section*{Proof of correctness}

\subsection*{Lemma 1}
Whenever the arbitrary natural number $a$ doesn't belong to the set $min_k(X)$ and there exists some item of $min_k(X)$ which is greater than $a$, it means that $a \not \in X$. \\

More formally, let's prove the following statement for any $a \in \mathbb{N}$ and any $X \subset \mathbb{N}$:
\begin{equation}
	\Big( (a \not \in min_k(X)) \wedge (\exists b \in min_k(X), a < b) \Big) \Rightarrow \Big( a \not \in X \Big)
\end{equation}

\subsubsection*{Proof}

According to the contrapositive  proof scheme, let's show that:
\begin{equation}
	\neg \Big( a \not \in X \Big) \Rightarrow \neg \Big( (a \not \in min_k(X)) \wedge (\exists b \in min_k(X), a < b) \Big)
\end{equation}

Which is equivalent to:
\begin{equation}
	\Big( a \in X \Big) \Rightarrow \Big( (a \in min_k(X)) \lor (\forall b \in min_k(X), a \geq b) \Big)
\end{equation}

The latter statement is equivalent to:
\begin{equation}
	\Big( a \in X \Big) \Rightarrow \Big( (a \in min_k(X)) \lor (a \in X \setminus min_k(X)) \Big)
\end{equation}

Which is a tautology. $\blacksquare$

\subsection*{Proof by the smallest counterexample}

\textit{Induction Basis:} \\ \\
In case if $i = 0$ or $j = 0$ there is possible only one edit distance, hence the set $lev_{a,b}(i, j)$ contains only one item, namely $max(i, j)$. As far as there is only one possible edit distance - it means, that $lev_{a,b}(i, j) = max_k(\{ max(i, j) \}) = \{ max(i, j) \}$, which complies to the definition of the subset of $k$-minimal values. \\

\textit{Induction Hypothesis:}
\begin{itemize}
	\item The set $lev_{a,b}(i - 1, j)$ contains the $k$ minimal edit distances between the first $i - 1$ characters of the string $a$ and the first $j$ characters of the string $b$. \\
		Hence, for every edit distance $y$ between the first $i - 1$ characters of the string $a$ and the first $j$ characters of the string $b$, such that $y \not \in lev_{a,b}(i - 1, j)$ it follows, that $\forall x \in lev_{a,b}(i - 1, j), y > x$.
	\item The set $lev_{a,b}(i, j - 1)$ contains the $k$ minimal edit distances between the first $i$ characters of the string $a$ and the first $j - 1$ characters of the string $b$. \\
		Hence, for every edit distance $y$ between the first $i$ characters of the string $a$ and the first $j - 1$ characters of the string $b$, such that $y \not \in lev_{a,b}(i, j - 1)$ it follows, that $\forall x \in lev_{a,b}(i, j - 1), y > x$.
	\item The set $lev_{a,b}(i - 1, j - 1)$ contains the $k$ minimal edit distances between the first $i - 1$ characters of the string $a$ and the first $j - 1$ characters of the string $b$. \\
		Hence, for every edit distance $y$ between the first $i - 1$ characters of the string $a$ and the first $j - 1$ characters of the string $b$, such that $y \not \in lev_{a,b}(i - 1, j - 1)$ it follows, that $\forall x \in lev_{a,b}(i - 1, j - 1), y > x$.
\end{itemize}

\textit{Inductive Step:} \\ \\
We want to show, that the \textit{Induction Hypothesis} implies, that the set $lev_{a,b}(i, j)$ contains the $k$ smallest edit distances between the first $i$ characters of the string $a$ and the first $j$ characters of the string. \\ \\
\textbf{For the sake of contradictions} let's assume, that the set $lev_{a,b}(i, j)$ doesn't contain the $k$ smallest edit distances. Thus, there there exists some edit distance $y \not \in lev_{a,b}(i, j)$ between the first $i$ characters of the string $a$ and the first $j$ characters of the string $b$, such that $\exists x \in lev_{a,b}(i, j)$ for which $y < x$:
\begin{equation}
\begin{split}
	\Big( y \not \in lev_{a,b}(i, j) \Big) \land \Big( \exists x \in lev_{a,b}(i, j), y < x \Big)
\end{split}
\end{equation}

Let's rewrite the expression in a following way:
\begin{equation}
\begin{split}
					& \Big( y \not \in lev_{a,b}(i, j) \Big) \land \Big( \exists x \in lev_{a,b}(i, j), y < x \Big) \\
	\Leftrightarrow \ 	& \Big( y \not \in min_k(E_{a,b}(i, j)) \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{By definition of $lev_{a,b}(i, j)$} \\
	\Leftrightarrow \ 	& \Big( y \not \in min_k(E_{a,b}(i, j)) \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{Idempotence} \\
	\Rightarrow \ 		& \Big( y \not \in E_{a,b}(i, j) \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{According to the Lemma 1} \\
	\Leftrightarrow \ 	& \Big( y \not \in I_{a,b}(i, j) \cup D_{a,b}(i, j) \cup S_{a,b}(i, j) \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{By definition of $E_{a,b}(i, j)$} \\
	\Leftrightarrow \ 	& \Big( y \not \in I_{a,b}(i, j) \Big) \land \Big( y \not \in D_{a,b}(i, j) \Big) \land \Big( y \not \in S_{a,b}(i, j) \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{De-Morgan's law} \\
	\Leftrightarrow \ 	& \Big( (y - 1) \not \in lev_{a,b}(i - 1, j) \Big) \land \Big( (y - 1) \not \in lev_{a,b}(i, j - 1) \Big) \land && \text{By definition of $I_{a,b}(i, j),$}\\
		\land			& \Big( (y - 1_{(a_i \neq b_j)}) \not \in lev_{a,b}(i - 1, j - 1) \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{$D_{a,b}(i, j)$ and $S_{a,b}(i, j)$} \\
	\Rightarrow \ 		& \Big( \forall m \in lev_{a,b}(i - 1, j), y - 1 > m \Big) \land \Big( \forall n \in lev_{a,b}(i, j - 1), y - 1 > n \Big) \land && \text{By Induction Hypothesis}\\
		\land			& \Big( \forall u \in lev_{a,b}(i - 1, j - 1), y - 1_{(a_i \neq b_j)} > u \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{} \\
	\Leftrightarrow \ 	& \Big( \forall m \in I_{a,b}(i, j), y > m \Big) \land \Big( \forall n \in D_{a,b}(i, j), y > n \Big) \land && \text{By definition of $I_{a,b}(i, j),$}\\
		\land			& \Big( \forall u \in S_{a,b}(i, j), y > u \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{$D_{a,b}(i, j)$ and $S_{a,b}(i, j)$} \\
	\Leftrightarrow \ 	& \Big( \forall m \in I_{a,b}(i, j) \cup D_{a,b}(i, j) \cup S_{a,b}(i, j), y > m \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{Reordering} \\
	\Leftrightarrow \ 	& \Big( \forall m \in E_{a,b}(i, j), y > m \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{By definition of $E_{a,b}(i, j)$} \\
	\Rightarrow \ 		& \Big( \forall m \in min_k(E_{a,b}(i, j)), y > m \Big) \land \Big( \exists x \in min_k(E_{a,b}(i, j)), y < x \Big) && \text{Because $min_k(X) \subseteq X$} \\
	\Leftrightarrow \ 	& \Big( \forall m \in lev_{a,b}(i, j), y > m \Big) \land \Big( \exists x \in lev_{a,b}(i, j), y < x \Big) && \text{By definition of $lev_{a,b}(i, j)$} \\
	\Rightarrow \ 		& Contradiction. \\
\end{split}
\end{equation}

Hence, our assumption was wrong. Consequently, the set $lev_{a,b}(i, j)$ contains the $k$-minimal edit distances. $\blacksquare$

\end{document}


\subsection{Subtraction of natural numbers}

We have inverse functions for addition. This is subtraction.

For function \(\oplus\), its inverse is \(\oplus' \), as defined below:

$a\oplus b=c$

$b=c\oplus 'a$

$f(a,b)=c\rightarrow f^{-1}(c,b)=a$

\subsubsection{Subtraction}

$a+b=c\rightarrow b=c-a$

There is no natural number \(b\) that satisfies:

$3+b=2$

While addition and multiplication are defined across all natural numbers, subtraction is not.
\subsubsection{Properties of subtraction}

Subtraction is not commutative:

$x-y\ne y-x$

Subtraction is not associative:

$x-(y-z)\ne (x-y)-z$


\section{Markov Chains}
\label{sec:markov-chains}

\begin{definition}[Markovian Property]
\label{def:markovian-property}
	The Markovian property states that the conditional distribution of any future state $X_{n+1}$, given past states $X_{0},X_{1},...,X_{n-1}$, and given the present state $X_{n}$, is independent of past states and depends only on the present state.
\end{definition}

\begin{definition}[Discrete-Time Markov Chain]
\label{def:discrete-time-markov-chain}

	A Discrete-Time Markov Chain (DTMC) is a stochastic process $\{X_{n \in \natural}\}$, where $X_{n}$ denotes the state at time $n$ and such that

	\begin{equation}
	\label{eqn:discrete-time-markov-chain}
		\begin{split}
			\probability{X_{n+1} = j | X_{n} = i,X_{n-1} = i_{n}-1,...,X_{0} = i_{0}}
			& = \probability{X_{n+1} = j | X_{n} = i} \\
			& = P_{i,j} \\
			& \forall n \geq 0, \forall i,j, \forall i_{0},...,i_{n-1}
		\end{split}
	\end{equation}

	where $P_{i,j}$ is independent of time and past history.
\end{definition}

\begin{definition}[Continuous-Time Markov Chain]
\label{def:continuous-time-markov-chain}

	A Continuous-Time Markov Chain (CTMC) is a stochastic process $\{X(t),t \in \Re^{+}\}$, where $X_{n}$ denotes the state at time $n$ and such that

	\begin{equation}
	\label{eqn:continuous-time-markov-chain}
		\begin{split}
			\probability{X(s+t) = j | X(s) = i,X(u) = x(u)}
			& = \probability{X(s+t) = j | X(s) = i} \\
			& = \probability{X(t) = j | X(0) = i} \\
			& = P_{i,j}(t) \\
			& \forall s,t \geq 0, \forall i,j,x(u)
		\end{split}
	\end{equation}

	where $P_{i,j}$ is independent of time and past history.
\end{definition}

Alternatively

\begin{definition}[Continuous-Time Markov Chain 2]
	\label{def:continuous-time-markov-chain-2}

	A Continuous-Time Markov Chain (CTMC) is a stochastic process with the property that every time it enters state $i$, the following hold:

	\begin{enumerate}
		\item The amount of time the process spends in state $i$ is a r.v. $X \sim Exp(\lambda)$.

		\item The probability $p_{i,j}$ to transit from state $i$ to state $j$ is independent of the time spent in state $i$.
	\end{enumerate}
\end{definition}

\begin{definition}[Transition Probability Matrix]
\label{def:transition-probability-matrix}

	The transition probability matrix associated with any Markov chain is a matrix, whose $(i,j)$-th entry $P_{i,j}$ is the probability of moving in the next transition from state $i$ to state $j$.
\end{definition}

By definition, we have that

\begin{equation*}
\sum_{j} P_{i,j} = 1 \quad \forall i
\end{equation*}

A queue network that exposes the Markovian property can be modeled by a Markov chain.

Non Markovian workloads can be approximated by mixtures of Exponential distributions, hence lending themselves to Markov chain analysis.

\begin{theorem}[Balance equations for CTMC]
\label{thm:balance-equations-continuous-time-markov-chains}

	Given an irreducible CTMC, we have that

	\begin{equation}
		\pi_{j} \sum_{i} q_{j,i} = \sum_{i} \pi_{i} q_{i,j}
	\end{equation}

	and

	\begin{equation}
		\sum_{i} \pi_{i} = 1
	\end{equation}

	where $\pi_{i}$ is the limiting probability for the CTMC to be in state $i$.
\end{theorem}

\begin{definition}[Time Reversibility]
\label{def:time-reversibility}

	A CTMC is time-reversible if, for all states $i$ and $j$, the rate of transitions from state $i$ to state $j$ equals the rate of transitions from state $j$ to state $i$. That is

	\begin{equation}
		\pi_{i} q_{i,j} = \pi_{j} q_{j,i} \quad \forall i,j
	\end{equation}
\end{definition}

All birth-death processes are time-reversible.
For a time-reversible process, the time-reversibility equations are always solvable.

\begin{theorem}
\label{thm:statistical-description-time-reversible}

	If a CTMC is time-reversible, then its reverse chain is statistically identical to the forwards chain, that is

	\begin{equation}
	\label{eqn:statistical-description-time-reversible}
	P_{i,j} = P_{i,j}^{*}
	\end{equation}
\end{theorem}

\Cref{thm:statistical-description-time-reversible} means that the reverse chain can be described by the same CTMC as the forwards chain

\subsection{PASTA}
\label{sec:pasta}

In many stochastic models, particularly in queueing theory, Poisson arrivals both observe a stochastic process and interact with it. In some cases it has been shown that the fraction of arrivals that see the process in some state is equal to the fraction of time the process is in that state.

This property il called PASTA, standing for \textit{Poisson Arrivals See Time Averages}.

The PASTA property is the adaptation of the Arrival Theorem for Poisson processes \cite{asmussen2003queueing,el2012sample} \footnote{also referred to as the Random Observer Property (ROP) \cite{el2012sample}.}.
The property states that the probability of the state as seen by an outside random observer is the same as the probability of the state seen by an arriving customer \cite{wolff1982poisson}.
The property also holds for the case of a doubly stochastic Poisson process where the rate parameter is allowed to vary depending on the state \cite{van1988conditional}.

\begin{theorem}[PASTA Property]
\label{thm:pasta}

	For any system with Poisson arrival process, we have that

	\begin{equation}
		a_{n} = d_{n} = p_{n}
	\end{equation}

	where
	$a_{n}$ is the probability that an arrival sees $n$ jobs in the system,
	$d_{n}$ is the probability that a departure leaves $n$ jobs in the system, and
	$p_{n}$ is the probability that there are $n$ jobs in the system.

\end{theorem}

PASTA is useful in system simulation. When simulating a Poisson arrival process, it suffices to average over what arrivals see at the moment they enter the system to know averages of the whole system. It is not the case when arrivals do not follow a Poisson process.


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 10.1 One-Dimensional Real Fourier Transform

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

This subroutine computes discrete Fourier transforms for real data using the
fast Fourier transform. The relations between the real values $x$ and the
complex Fourier coefficients $\xi $ are as follows:

For Fourier analysis:
\begin{equation}\label{fa}
\xi _k=\frac 1N\sum_{j=0}^{N-1}x_jW^{-jk},\quad k=0,1...,N-1,
\end{equation}
For Fourier synthesis:
\begin{equation}
\label{fs}x_j=\sum_{k=0}^{N-1}\xi _kW^{jk},\quad j=0,1,...,N-1,
\end{equation}
where $N=2^{\text{M}}$ and $W=e^{2\pi i/N}=\cos (2\pi /N)+i\ \sin (2\pi /N)$, and $%
i^2=-1$. Note that the sign on the exponent of $W$ is negative for analysis
and positive for synthesis.

\subsection{Usage}

\subsubsection{Program Prototype, Single Precision}

\begin{description}
\item[\bf INTEGER]  \ {\bf M, MS}
\item[\bf REAL]  \ {\bf A}($\geq 2^{\text{M}}$)\bf
 {, S}($\geq 2^{\text{M}-2}-1$)
\item[\bf CHARACTER]  \ {\bf MODE}
\end{description}
On the initial call set MS = 0 to indicate the array S() does not yet
contain a sine table. Assign values to A(), MODE, and M.%
$$
\fbox{{\bf CALL SRFT1(A, MODE, M, MS, S)}}
$$
On return A() will contain the computed results. S() will contain the sine
table used in computing the Fourier transform. MS may have been changed.

\subsubsection{Argument Definitions}

\begin{description}
\item[A()]  [inout] If the argument MODE selects Analysis, A() must contain $%
x$ on input and will contain $\xi $ on output. If MODE selects Synthesis,
A() must contain $\xi $ on input and will contain $x$ on output. When A()
contains $x$'s, $A(k+1)$ contains $x_k$, $k=0$, 1, ..., $N-1$. When
A() contains $\xi $'s, the real coefficients $\xi _0$ and $\xi
_{N/2}$ are in A(1) and A(2), respectively and the complex coefficients, $%
\xi _k$, for $k=1$, ..., $(N/2)-1$, are in the paired locations $(A(2k+1)$, $%
A(2k+2))$. The remaining complex coefficients satisfy $\xi _{N-k} =
\overline{\xi}_k$, for $k$ = 1, ..., $\frac N2-1$.

\item[MODE]  [in] MODE\ =\ $^{\prime }\text{A}^{\prime }$
or $^{\prime }\text{a}^{\prime }$ selects Analysis.
MODE\ =\ $^{\prime }\text{S}^{\prime }$ or $^{\prime }\text{s}^{\prime }$
selects Synthesis.

\item[M]  [in] The number of data points is $N=2^{\text{M}}$. If $\text{M}=0$,
the subroutine returns taking no action. Require $0\leq \text{M}\leq 31.$

\item[MS]  [inout] Gives the state of the sine table in S().  Let
$\text{MS}_{in}\text{ and MS}_{out}$ denote the values of MS on entry
and return respectively. If the sine table has not previously been
computed, set $\text{MS}_{in} = 0$ or $-$1 before the call. Otherwise
the value of $\text{MS}_{out}$ from the previous call using the same
S() array can be used as $\text{MS}_{in}$ for the current call.

Certain error conditions described in Section E cause the subroutine
to set $\text{MS}_{out} = -2$ and return.  Otherwise, with M $>$ 0, the
subroutine sets $\text{MS}_{out} = \max (\text{M}, \text{MS}_{in}).$

If $\text{MS}_{out} > \max (2, \text{MS}_{in}),$ the subroutine sets
NT = $2^{\text{MS}_{out}-2}$ and fills S() with NT $-$ 1 sine values.

If $\text{MS}_{in}=-1$, the subroutine returns after the above
actions, not transforming the data in A().  This is intended to allow
the use of the sine table for data alteration before a subsequent Fourier
transform, as discussed in Section~G of Chapter~16.0, and as
illustrated in the example of Section~C.

\item[S()]  [inout] When the sine table has been computed, $\text{S}(j)=
\sin \pi j/(2\times NT)$, $j=1,2,...,NT-1$, see MS above.
\end{description}

\subsubsection{Modifications for Double Precision}

Change SRFT1 to DRFT1, and the REAL type statement to DOUBLE PRECISION.

\subsection{Examples and Remarks}

Program DRSRFT1 computes an estimate of the spectral composition of
\begin{equation}
\label{e1}f(t)=[\sin 2\pi (t+.01)+4\cos 2\pi (\sqrt{2}\,t+.3)]
\end{equation}
with output in ODSRFT1. It is assumed $t$ has units of seconds, $f$ has
discrete frequencies with mutual separation of at least 0.4~cycles/second,
and that there is no interest in frequencies $>2$ cycles/second. To resolve
different frequencies we select $\Delta \omega =0.4/10=.04$ cycles/second,
which according to Eq.~(20) of Chapter~16.0 implies a sampling period of $%
T=1/\Delta \omega =25$ seconds. According to Eq.~(24) of Chapter~16.0, $\Delta
t\leq \frac 12\omega _{N/2}=\frac 14$~second, if any information about the
highest frequency is to be obtained. Thus, $N=T/\Delta t\geq 100$. Since
modest accuracy is sufficient, we give results for $N=128$ in the program
below. Lanczos sigma factors are used as described in Section G of
Chapter~16.0. The complex transform has peaks for $k=26$ and for $k=36$.
These values of $k$ correspond to frequencies of 1 and~1.4~cycles/second.
Results for this problem, but without the use of Sigma factors are
given in Chapter~16.3.

\subsection{Functional Description}

From Eq.~(\ref{fa}), the fact that $x_j$ is real, and $W^N=e^{2\pi i}=1$, it
follows that $\xi _{N-k}=\overline{\xi }_k$. Thus $\xi _{N/2}$ is real, and
clearly $\xi _0$ is also real. Thus only half the $\xi $'s need be
stored if $\xi _0$ and $\xi _{N/2}$ are counted as one. The $\xi $'s
are stored in the same space used for the $x$'s by taking advantage
of this symmetry.

The primitive FFT routine SFFT (or DFFT) is used to compute $\zeta $ from $x$%
, or $x$ from $\zeta $ where $\zeta $ is defined by
\begin{equation}\label{e4}
\zeta _k=\frac 2N\sum_{j=0}^{\frac
N2-1}(x_{2j}+ix_{2j+1})W^{-2jk},k=0,1,...,\frac N2-1.
\end{equation}
Using Eq.\ (\ref{e4}), one finds that $\zeta _k+\overline{\zeta }_{(N/2)-k}$
depends only on $x_{2j}$ and $\zeta _k-\overline{\zeta }_{(N/2)-k}$ only on $%
x_{2j+1}$. From this and Eq.\ (\ref{fa}) there follows immediately%
\begin{equation}\label{e5}
\begin{split}
\xi_k  &=\frac 14 \left[(\zeta _k+\overline{\zeta }_{(N/2)-k})\right.\\
 -i&\left.W^{-k}(\zeta_k-\overline{\zeta}_{(N/2)-k})\right] ,
 \ \ 0<k<N/4\\
 \overline{\xi}_{(N/2)-k} &= \frac 14 \left[(\zeta _k+\overline{\zeta }_{(N/2)
 -k})\right.\\
 +i&\left.W^{-k}(\zeta _k-\overline{\zeta }_{(N/2)-k})\right] ,
 \ \ 0<k<N/4\\
  \xi_0 &=\frac 12 [\Re\zeta _0+\Im\zeta _0]\\
  \xi_{N/2} &=\frac 12 [\Re\zeta _0-\Im\zeta _0]\\
  \xi _{N/4} &=\frac 12 \overline {\zeta}_{N/4}
\end{split}
\end{equation}
%\begin{eqnarray}
%    \nonumber \xi_k  & = &  \frac 14
%        \left[(\zeta _k+\overline{\zeta }_{(N/2)-k}) \hspace{1in}\right. \\
% \nonumber &  &  \left. \hspace{-.35in} \mbox{}-iW^{-k}
%                        (\zeta _k-\overline{\zeta }_{(N/2)-k})\right] ,\  0<k<N/4\\
% \nonumber \overline{\xi}_{(N/2)-k} & = &
%                     \frac 14 \left[(\zeta _k+\overline{\zeta }_{(N/2)-k})\right. \\
%   & & \left. \hspace{-.35in} \mbox{}+iW^{-k}(\zeta _k-\overline{\zeta }_{(N/2)-k})\right] ,
%     \  0<k<N/4\ \ \ \ \\
%  \nonumber \xi_0 & = & \frac 12 [\Re\zeta _0+\Im\zeta _0] \\
%   \nonumber \xi_{N/2} & = & \frac 12 [\Re\zeta _0-\Im\zeta _0] \\
%  \nonumber      \xi _{N/4} & = & \frac 12 \overline {\zeta}_{N/4} \\
%  \nonumber
%\end{eqnarray}
Thus to compute $\xi $ given $x$, $\zeta /4$ is computed
using Eq.~(\ref{e4}) (divided by~4) and the $\xi $'s are computed
from Eq.~(\ref{e5}). To compute $x$ given $\xi $, the above process is reversed.
More details can be found in \cite{Krogh:1970:RF1}.

\bibliography{math77}
\bibliographystyle{math77}

\subsection{Error Procedures and Restrictions}

M must satisfy 0 $\leq \text{M} \leq 31$ and MODE must have one of its
allowed values.  If these conditions are violated an error message will be
issued using the error processing procedures of Chapter~19.2 with a
severity level of~2 to stop execution.  A return will be made with $%
\text{MS}=-2$ instead of stopping if the statement ``CALL ERMSET($-1$)''
is executed before calling this subroutine.

If the sine table does not appear to have valid data, an error message is
printed, and the sine table and then the transform are computed.

\subsection{Supporting Information}

The source language is ANSI Fortran~77.

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DRFT1 & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
 DFFT, DRFT1, ERFIN, ERMSG, IERM1, IERV1\rule[-5pt]{0pt}{8pt}}\\
SRFT1 & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
 ERFIN, ERMSG, IERM1, IERV1, SFFT, SRFT1}\\
\end{tabular}

Subroutine designed and written by: Fred T. Krogh, JPL Section~373, August
1969; Revised January~1988.

\begcodenp

\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSRFT1}\vspace{10pt}
\lstinputlisting{\codeloc{srft1}}

\vspace{30pt}\centerline{\bf \large ODSRFT1}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{srft1}}
\end{document}


\subsection{Support Vector Machine} \label{svm}

Support Vector Machines (SVM) are highly used due to high empirical performance and somewhat-simple computational complexity, being binary classifiers that when allied with strategies, such as \textit{one-vs-all}, can be expanded to become multi-class classifiers.

In our case, SVM was our first approach to the face detection problem, because the dataset, even after the process of data augmentation, is not very large and we are dealing with a binary classification problem, so we decided to start our initial approach with a simpler model like SVM.

Initially we divide our data in two subsets: training subset with \(80\%\) of the images and testing subset with \(20\%\) of the images.

In SVM there are two main hyper-parameters that can be tweaked to improve the performance of the model, C and Gamma. To get the best performance out of the model we made every combination possible for C and gamma, starting with one of the values present in table \ref{table:svm-initial-configuration}.

\begin{table}[htbp]
\centering
\caption{SVM - Initial Configuration}
\begin{tabular}{ |c|c| }
 \hline
 \textbf{Train Data} & 80\% \\
 \hline
 \textbf{Test Data} & 20\% \\
 \hline
 \textbf{Gamma} & [100, 33, 10, 3, 1, 0.3, 0.1, 0.03 ] \\
 \hline
 \textbf{C} & [0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30] \\
 \hline
\end{tabular}
\label{table:svm-initial-configuration}
\end{table}

To validate which combination of hyper-parameters enabled the best performance out of our model we used K-Fold Cross Validation where our training set was divided in \(K = 5\) smaller sets, where 4 were used for training and 1 was used to validate the model. With this approach we didn't need to reduce the training dataset by creating a validation dataset and the results did not depend on a particular random choice for the pair of (train, validation) datasets.

After validating every combination, as we can see in figure \ref{fig:svm-cross-validation}, the values of C and gamma where our model performed the best are: C = 30 and Gamma = 1.

\begin{figure}[htbp]
\centerline{\includegraphics[width=1\linewidth]{images/svm_cross_validation.png}}
\caption{SVM - Cross-Validation using multiple values for C and Gamma}
\label{fig:svm-cross-validation}
\end{figure}

Based on the Cross Validation data, we created an SVM model with a Radial Basis Function (RBF) kernel, and with the hyper-parameters C = 30 and Gamma = 1, that returns 1 when the image is a face, and 0 otherwise, as we can see in table \ref{table:svm-final-configuration}.

\begin{table}[htbp]
\centering
\caption{SVM - Final Configuration}
\begin{tabular}{ |c|c| }
 \hline
 \textbf{Train Data} & 80\% \\
 \hline
 \textbf{Test Data} & 20\% \\
 \hline
 \textbf{Kernel} &  Radial Basis Function (RBF) \\
 \hline
 \textbf{Gamma} & 1 \\
 \hline
 \textbf{C} & 30 \\
 \hline
\end{tabular}
\label{table:svm-final-configuration}
\end{table}

Finally, to ensure that the model didn't end up in a local minimum, and that our model was actually learning we run the all process 50 times and calculated the mean values to both train and test phases.


\section{Evaluation}
\label{sec:evaluation}
If not differently specified, all experiments discussed in this section are run with the following parameters: $N = 500$, $n_{max} = 200$, $n_D = 100$ and $c = 0$.

\subsection{Storage Capacity}
\label{subsec:capacity}
\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/base}
    \caption{Storage success rate of a Rosenblatt perceptron as a function of $\alpha = P / N$. The experiments use $N = 500$, $n_{max} = 200$ and $n_D = 100$.}
	\label{fig:base}
\end{figure}
\cref{fig:base} shows the results of the base experiment.
The x-axis represent different values of $\alpha = P / N$, while the y-axis the success rate $Q_{l.s.}$.
As expected, the function looks like a step function from $1$ to $0$.
For $\alpha \approx 1.7$, the success rate $Q_{l.s.}$ drops from $1$ to $0$ very quickly.

The value of $\alpha$ for which the function drops is called storage capacity of the perceptron.
For $N \to \infty$ (very large number of examples) and $n_{max} \to \infty$ (no limit on the maximum number of training iterations), the theoretical storage capacity of the Rosenblatt perceptron is $\alpha = 2$.

\subsection{Number of Iterations}
\label{subsec:epochs}
\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/multiple_epochs}
    \caption{Storage success rate of a Rosenblatt perceptron as a function of $\alpha = P / N$ for different values of $n_{max}$.}
	\label{fig:multiple_epochs}
\end{figure}

The difference between the theoretical value and the experimental one are mainly due to the limited number of training iterations.
\cref{fig:multiple_epochs} gives an experimental proof of this statement:
for a very small number of iterations (eg. $n_{max} = 10$), the step is close to $\alpha = 1$, while for higher values of iterations the step moves closer and closer to the theoretical value $\alpha = 2$ found with \cref{eq:prob-lin-sep-alpha}.
The theoretical result still remains far from the practical one because of the limited size of input's dimension $N$.

\subsection{Number of Dimensions}
\label{subsec:dimensions}
\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/multiple_n}
    \caption{Storage success rate of a Rosenblatt perceptron as a function of $\alpha = P / N$ for different values of $N$.}
	\label{fig:multiple_n}
\end{figure}
The theoretical results are valid for $N \to \infty$.
However, real datasets have a limited number of features.
\cref{fig:multiple_n} shows the behaviour of the perceptron for different values of $N$.
For high values of $N$, the shape of the success rate $Q_{l.s.}$ as a function of $\alpha$ is similar to a step function.
For small values of $N$, the function looks like a smoothed step function:
the smaller $N$ is, the higher is the smoothing.

\subsection{Weight Update Criterion}
\label{subsec:c}
\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/bonus_2_c}
    \caption{Storage success rate of a Rosenblatt perceptron as a function of $\alpha = P / N$ for different values of $c$.}
	\label{fig:multiple_c}
\end{figure}
\cref{fig:multiple_c} shows the effect of changing the values of $c$ in the training procedure of the perceptron.
For higher values of $c$, the curve is shifted to the left.
Since an example $\xi^\mu$ is considered correctly classified only when its local potential is greater than $c$ ($E = \mathsf{\bm{w}} \cdot \xi^\mu S^\mu > c$), the potential only depends on $\mathsf{\bm{w}}$ for a fixed $\xi^\mu$ and the update of $\mathsf{\bm{w}}$ is fixed for a given wrong classified example, the perceptron will need a higher number of updates to increase the norm of $\mathsf{\bm{w}}$ and make the local potentials higher than a threshold $c > 0$.
Formally, we can show that the value of $c > 0$ is irrelevant (provided the perceptron is trained long enough):
\begin{equation*}
	\mathsf{\bm{w}}_1 : \{E_1^\mu \geq c\}_{\mu = 1}^{P} \Leftrightarrow \mathsf{\bm{w}}_2 = \lambda \mathsf{\bm{w}}_1 : \{E_2^\mu \geq c\}_{\mu = 1}^{P}, \lambda > 0
\end{equation*}

\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/bonus_2_epoch}
    \caption{Storage success rate of a Rosenblatt perceptron as a function of $\alpha = P / N$ for different numbers of iterations with fixed value of $c=2$.}
	\label{fig:fixed_c_multiple_epoch}
\end{figure}
To give an empirical proof of this, we fix the value of $c$ and train the perceptron for different number of iterations $n_{max}$.
We expect to see the curve shifted to the left for small $n_{max}$, and approximate a step function centered in $\alpha = 2$ for big $n_{max}$.
\cref{fig:fixed_c_multiple_epoch} shows that the results of the experiment confirm our hypothesis.

\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/bonus_2_c_epoch}
    \caption{Storage success rate of a Rosenblatt perceptron as a function of $\alpha = P / N$ for different numbers of iterations and $c$.}
	\label{fig:multiple_c_multiple_epoch}
\end{figure}
\cref{fig:multiple_c_multiple_epoch} compares the curves for $c = 0$ and $c = 2.0$ for different values of $n_{max}$:
by increasing $n_{max}$, the curve is ``pushed'' towards the right, contrasting the effect of the increased value of $c$.
We can somehow see $c$ as a simple version of the learning rate that is used in more complex neural networks: it ``regulates'' the speed of the training.

\subsection{Inhomogeneous Hyperplanes}
\label{subsec:homogeneous}
\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/homogeneous}
    \caption{Storage success rate of a Rosenblatt perceptron and its inhomogeneous version for $N = 500$ as a function of $\alpha = P / N$.}
	\label{fig:homogeneous}
\end{figure}
We run an experiment to verify the behaviour of $Q_{l.s.}$ by allowing inhomogeneous hyperplanes:
we train both a normal and modified perceptron for $N = 500$.
\cref{fig:homogeneous} shows the results of the experiment.
As expected, the success rate $Q_{l.s.}$ of the inhomogeneous perceptron is slightly higher than homogeneous one.
However, the difference is not significant, since data points follow a normal distribution $\xi^\mu_j \sim \mathcal{N}(0,\,1)$ and are therefore distributed around the origin.

The problem of finding an inhomogeneously solution in $R^{N}$ can be solved by finding a homogeneously solution in $R^{N + 1}$.
In a second experiment, we compare the success rate $Q_{l.s.}$ of an inhomogeneous perceptron for $N = 500$ with an homogeneous perceptron for $N = 501$.
\cref{fig:homogeneous_n_n1} shows the result of this experiment.
As expected, the success rates for the $2$ perceptrons are very close to each other.

\begin{figure}[t]
	\centering
	\includegraphics[width=\columnwidth]{figures/homogeneous_n_n1}
    \caption{Storage success rate of a homogeneous Rosenblatt perceptron for $N = 501$ and its inhomogeneous version for $N = 500$ as a function of $\alpha = P / N$.}
	\label{fig:homogeneous_n_n1}
\end{figure}


% !Mode:: "TeX:UTF-8"
% !TEX program  = xelatex
\section{Applications}\label{S:applications}
\subsection{Solution to Article Title}
From the definition of quadratic equation in Section~\ref{S:quadratic}, we can finally answer the question, \emph{what Angle Should We Throw a Football for Maximum Range?}

Firstly, we establish a rectangular coordinate system, as shown in Figure~\ref{F:parabolic-throwing}. We denote $\theta$ as shot angle, $V$ as initial velocity, $g$ as gravity, $S$ as distance, $t$ as time.

\begin{figure}
    \centering
    \includegraphics[width=.45\textwidth]{figures/quadratic_equation-3.png}
    \caption{Displacement and coordinates of parabolic throwing}\label{F:parabolic-throwing}
\end{figure}

Then, we have both horizontal and vertical velocity,
\begin{equation}\label{E:solution-1}
    \begin{cases}
        V_x &= V\cos(\theta) \\
        V_y &= V\sin(\theta) - gt
    \end{cases}
\end{equation}

Therefore, we have both horizontal and vertical distance at time $t$,
\begin{equation}\label{E:solution-2}
    \begin{cases}
        S_x &= V\cos(\theta)t \\
        S_y &= V\sin(\theta)t - \frac{1}{2} gt^2
    \end{cases}
\end{equation}

From Equation~\eqref{E:solution-1} and Equation~\ref{E:solution-1}, we can find all of the coordinate $(S_x, S_y)$ with angle $\theta$. From Equation~\eqref{E:solution-2}, we have $t=\tfrac{2V\sin(\theta)}{g}$, then we substitute the expression of $t$ back to the $S_y$, we have the quadratic equation
\begin{equation}\label{E:solution-3}
    S_y = \tan(\theta)S_x - \frac{g}{2V^2\cos^2(\theta)}S_x^2.
\end{equation}

From Equation~\eqref{E:quadratic-formula}, we can solve this quadratic equation, the solutions are $0$ and $\tfrac{2V^2}{g}\sin(\theta)$. Obviously, $0$ is the initial position, then the football will land at $\tfrac{2V^2}{g}\sin(\theta)$, that is, $\tfrac{V^2}{g}\sin(2\theta)$.

According to knowledge of trigonometric functions, we have the maximum range of the football is $\tfrac{V^2}{g}$, which is taken at $\theta=45$.

\subsection{Simulation of Projectile Motion}
You can use quadratic equation to simulate projectile motion, which can solve free fall problems. If you are familiar with \texttt{Python}, you can use \texttt{Python} to draw the pictures of trajectories, and you will find it easy to use Python to solve quadratic equations, especially in symbolic calculations. And the code is in Appendix~\ref{A:python}.