\chapter{Skills from National 5}

\section{Distance Formula}
The Distance Formula allows us to find the distance between two points given their Cartesian coordinates. To do this we treat the line between the two points as though it were the hypotenuse of a right-angled triangle, and use Pythagoras' theorem to find the length of the hypotenuse.

DIAGRAM GOES HERE.

To find each side we simply take the difference between the two $x$-coordinates $y$-coordinates, and substitute into Pythagoras' theorem.

Pythagoras' theorem:
\begin{equation*}
	a^2 = b^2+c^2
\end{equation*}

Rearranged, with our method to find the distances:
\begin{equation*}
	d = \sqrt{(x_2-x_1)^2+(y_2-y_1)^2}
\end{equation*}

\section{Factorisation}
Factorisation is one of the most important skills that must be known at Higher. Though it's possible to pass National 5 maths with an A without knowing how to factorise, this isn't the case with the Higher course. There are three kinds of factorisation which should be able to be done by hand: looking for the greatest common factor, recognising a difference of two squares, and factorising a quadratic.

Note that when reading resources written in American English, factorising and to factorise is often shortened to simply factoring and to factor (and American spelling changes -ise to -ize in general, so it'll be called "factorizing``).

\subsection{Greatest Common Factor}
The greatest common factor should have been learnt about in primary school when simplifying fractions. For example, the greatest common factor of $12$ and $18$ is $6$.

When asked to factorise an expression such as $12 + 18$, it should be recognised that the greatest common factor of all terms is $6$. So the expression can be factorised to look like $6(2 + 3)$. This is incredibly useful when dealing with many large terms, such as $126 - 612 + 531$, which can be factorised to become $9(14 - 68 + 59)$.

The strategy for algebraic expressions is the same as with normal numbers: look for the greatest value that everything can be divided by. For example, $3x^5 + 9x^2$. Both coefficients can be divided by 3, the variables can both be divided by $x^2$. Factoring this expression gives $3x^2(x^3+3)$. If unsure, the bracket(s) can always be multiplied out again to double-check.

\subsubsection{Partial Factorisation ("Grouping``)}
It's possible to only partially factorise and take out a common factor. For example, $4x^2+2y+7x+11$ can be partially factorised to $2(2x^2+y)+7x+11$. This works, as shown below.

\begin{align*}
	&4x^2+2y+7x+11\\
	&=\frac{4x^2}{2}+\frac{2y}{2}+\frac{7x}{2}+\frac{11}{2}\\
	&=2x^2+y+3.5x+5.5\\
	&=2(2x^2+y+3.5x+5.5)\\
	&=2(2x^2+y+3.5x)+11\\
	&=2(2x^2+y)+7x+11\\
\end{align*}

This is also called factorisation by grouping, as you're essentially grouping a large expression into smaller ones. The above expression can be re-written into two groups like $(4x^2+2y)+(7x+11)$ where it's a bit more obvious that a factor can be taken out. Note that for a final answer any unnecessary brackets must be removed, otherwise marks might not be awarded in an exam.

\subsection{Difference of Two Squares}
It's very common that expressions end up as the difference of two squares. That name is very literal, $16 - 9$ is a difference of two squares, as it can be re-written as $4^2 - 3^2$.

Any expression in the form $a^2 - b^2$ can be re-written as $(a+b)(a-b)$.

For example, $9x^4-x^2$ can be factorised to give $(3x^2+x)(3x^2-x)$. Note that sometimes you can do this twice, for example, $16x^4-81$ can be factorised to $(4x^2+9)(4x^2-9)$, where the second bracket can be further factorised to give $(4x^2+9)(2x+3)(2x-3)$.

\subsection{Factorising A Quadratic}
Some quadratics can be factorised, which is essential when trying to solve them. There are usually two different methods that are taught.

\subsubsection{Method 1 — St. Andrew's Cross}
My friend always called this the St. Andrew's Cross method, even though the table that is built isn't layed out diagonally like the real St. Andrew's Cross is. I liked the name though, which is why I'll call it like so.

A quadratic in the form $ax^2+bx+c$ where $a=1$ can be factorised by looking for factors of $c$, and then checking which of those factors adds to give $b$. Once a suitable pair of factors is found, they are placed in brackets in the form $(x+a)(x+b)$ where $a$ and $b$ are the two factors (which could potentially be negative).

For example, consider $x^2+6x+9$. The factors of $9$ are $1 \cdot 9$, and $3 \cdot 3$, where either both have to be positive or negative. To get $6$, positive $3$ and $3$ have to be added, $1$ and $9$ would give $10$ when added which isn't what is needed. So $x^2+6x+9$ can be factorised to give $(x+3)(x+3)$, which can be simplified to $(x+3)^2$.

The cross part comes when laying this method out. Suppose $x^2-3x-4$. First, the factors of $4$ should be written out in column of a table, and the sum of the pair should be written in the other column, as shown. In practice, after enough repetitiion, this can be done mentally, many people wouldn't write this table out.

\medskip

\begin{tabular}{r | l}
	Factors of $-4$ & Sums\\
	\hline
	$4 \cdot -1$ & $3$\\
	$-4 \cdot 1$ & $-3$\\
	$2 \cdot -2$ & $0$\\
	$-2 \cdot 2$ & $0$\\
\end{tabular}

From the table, it can be seen that the factors $-4 \cdot 1$ give $-3$, so when $x^2-3x-4$ is factorised, $(x-4)(x+1)$ is received. Note that in practice, the table can be stopped after $-4 \cdot 1$ is written out, since continuing is pointless and the table gives no extra marks.

When $a \neq 1$, it gets a little bit more difficult and is where the second method really shines.

First, check if the whole quadratic can be simplified by taking out a common factor. For example, $6x^2+72x+120$ might look daunting, but can be simplified to $6(x^2+12x+20)$.

If $a$ is a prime number, one of the factors of $c$ has to be multiplied by $a$. This will make the table a lot longer.

Suppose $5x^2+11x-12$. Here, the first factor is multiplied by $a$ and then, after the horizontal line, the second factor is multiplied by $a$; so for the first row, instead of $1 \cdot -12$, $1$ is multiplied by $a$ (in this case $5$) to give $5 \cdot -12$.

\medskip

\begin{tabular}{r | l}
	Factors & Sums\\
	\hline
	$5 \cdot -12$ & $-7$\\
	$-5 \cdot 12$ & $7$\\
	$10 \cdot -6$ & $-4$\\
	$-10 \cdot 6$ & $4$\\
	$15 \cdot -4$ & $11$\\
	$-15 \cdot 4$ & $-11$\\
	\hline
	$1 \cdot -60$ & $-59$\\
	$-1 \cdot 60$ & $59$\\
	$2 \cdot -30$ & $-28$\\
	$-2 \cdot 30$ & $28$\\
	$3 \cdot -20$ & $-17$\\
	$-3 \cdot 20$ & $17$\\
\end{tabular}

Again, the table can be stopped after $15 \cdot -4$ is found, as that gives the correct sum of $11$, but for demonstration purposes the complete table is shown.

Now that the factors have been found, they have to be placed in such a way so that when multiplied back out it'll give the correct value for $b$. This means that the factor that has been multiplied by $a$ has to be placed in the bracket that doesn't include $a$. So the factorisation of $5x^2+11x-12$ is $(5x-4)(x+3)$ (note that the unmultiplied factor is placed in the bracket!).

When $a$ is not a prime number, all of its factors have to be considered. If $a$ is 4, for example, then the brackets could look like $(4x+a)(x+b)$ or $(2x+a)(2x+b)$. Even more possibilities have to be considered when making a table. A very difficult question might choose a value for $a$ which has many factors, such as $12$.

Suppose $4x^2+8x+3$ had to be factorised. Firstly, it cannot be simplified by taking out a common factor, so all the factors of $c$ (in this case $3$) have to be listed and individually multiplied by $4$ or $2$. In the table, the first factor is multiplied by 4, then the second, then both terms are multiplied by $2$ and $2$. Since it's the same number, this doesn't need to be done twice. Negative numbers can be excluded, as both $b$ and $c$ are positive.

\medskip

\begin{tabular}{r | l}
	Factors & Sums\\
	\hline
	$4 \cdot 3$ & $7$\\
	\hline
	$1 \cdot 12$ & $13$\\
	\hline
	$2 \cdot 6$ & $8$\\
\end{tabular}

So finally, $4x^2+8x-3$ can be factorised to give $(2x+3)(2x+1)$.

If $a<0$ (if $a$ is negative), take out a common factor of $-1$. Suppose $-4x^2-8x-3$ had to be factorised.

\begin{align*}
	&-4x^2-8x-3\\
	&=-(4x^2+8x+3)\\
	&=-(2x+3)(2x+1)\text{ This has been factorised above already}\\
	&=(-2x-3)(2x+1)
\end{align*}

It is most likly not necessary to remove the negative outside the bracket, but some might prefer the look of this.

\subsubsection{Method 2 — Splitting Up $b$}
At first, this might seem similar to the St. Andrew's Cross, but the only thing that's similar is that both require a table. This method is good because it doesn't require too many extra steps when $a \neq 1$.

Firstly, two numbers (later referred to as $p$ and $q$) have to be found such that their product is $ac$ and their sum is $b$. Then, re-write the quadratic $ax^2+bx+c$ in the form $ax^2+px+qx+c$. Partially factorise $ax^2+px$ and $qx+c$, and take out a common factor. As in method 1, if $a<0$, take out a common factor of $-1$ first.

For example, suppose $9x^2-3x-2$ had to be factorised. First, find $p$ and $q$ such that their product is $9 \cdot -2$ ($=-18$) and their sum is $-3$.

\medskip

\begin{tabular}{r | l}
	Factors of -18 & Sums\\
	\hline
	$1 \cdot -18$ & $-17$\\
	$-1 \cdot 18$ & $17$\\
	$2 \cdot -9$ & $-7$\\
	$-2 \cdot 9$ & $7$\\
	$3 \cdot -6$ & $-3$\\
	$-3 \cdot 6$ & $3$\\
\end{tabular}

The factors $3 \cdot -6$ give $-3$, so $p=3$ and $q=-6$.

\begin{align*}
	&9x^2-3x-2\\
	&=9x^2+3x-6x-2\\
	&=3x(3x+1)-2(3x+1)\\
	&=(3x+1)(3x-2)
\end{align*}

\section{Completing the Square}
Completing the square is turning a quadratic from the form $ax^2 + bx + c$ to the form $p(x + q)^2+r$. There are multiple ways to do this, all of which will use the example quadratic $2x^2-6x+77$.

The question will always give the completed square form, that is $p(x - q)^2+r$ (though the variables might be different ones, like $a$, $b$, $c$, or so). This can be expanded, as shown.
\begin{align*}
	&p(x+q)^2+r\\
	&=p(x^2+2qx+q^2)+r\\
	&=px^2+2pqx+pq^2+r
\end{align*}
It can be seen that $a$ became $p$, $b$ became $2pq$, and $c$ became $pq^2+r$. From there, a series of equations can be made and solved (practically this is done from left to right as shown).
\begin{align*}
	p &= a & 2pq &= b         & pq^2+r &= c\\
	p &= 2 & 2 \cdot 2q &=-24 & 2 \cdot (-6)^2 + r &= 77\\
	  &    & 4q &= -24        & 2 \cdot 36 + r &= 77\\
	  &    & q &= -6          & 72 + r &= 77\\
	  &    & &                & r &= 77 - 72\\
	  &    & &                & r &= 5
\end{align*}
Now that $p$, $q$, and $r$ have been found, they can be substituted back into completed square form.
\begin{equation*}
	2x^2-24x+77 = 2(x-6)^2+5
\end{equation*}

\section{Rationalising the Denominator}
When a fraction uses an irrational number as its denominator it can be difficult to understand what it's actually quantifying (try to imagine 5 $\sqrt{2}$ pieces of pizza!). Instead, the denominator can be rationalised. This has to be done in the final answer to an exam question, but isn't necessary (and sometimes even unhelpful) to be done mid-question.

When the denominator contains a root, the whole fraction should be multiplied by $1$ in the form of this root.

\begin{align*}
	&\frac{5}{\sqrt{2}}\\
	&=\frac{5}{\sqrt{2}} \cdot \frac{\sqrt{2}}{\sqrt{2}}\\
	&=\frac{5\sqrt{2}}{\sqrt{2}\sqrt{2}}\\
	&=\frac{5\sqrt{2}}{2}
\end{align*}

And that is all there is to it, simply multiply the fraction by whatever root the denominator has. Here's another example.

\begin{align*}
	&\frac{300\sqrt{2}}{5\sqrt{40}}\\[5pt]
	&=\frac{300\sqrt{2}}{5\sqrt{4 \cdot 10}}\\[5pt]
	&=\frac{300\sqrt{2}}{5 \cdot 2\sqrt{10}}\\[5pt]
	&=\frac{300\sqrt{2}}{10\sqrt{10}}\\[5pt]
	&=\frac{30\sqrt{2}}{\sqrt{10}}\\[5pt]
	&=\frac{30\sqrt{2}}{\sqrt{10}} \cdot \frac{\sqrt{10}}{\sqrt{10}}\\[5pt]
	&=\frac{30\sqrt{2}\sqrt{10}}{\sqrt{10}\sqrt{10}}\\[5pt]
	&=\frac{30\sqrt{2}\sqrt{10}}{10}\\[5pt]
	&=3\sqrt{2}\sqrt{10}\\
	&=3\sqrt{20}\\
	&=3\sqrt{4 \cdot 5}\\
	&=3 \cdot 2 \sqrt{5}\\
	&=6\sqrt{5}
\end{align*}


\subsection{Tidal Shear Tensor}
\label{section:forero-romero}

This method \cite{2009MNRAS.396.1815F} works on density fields grids
obtained either from numerical simulations or reconstructions from
redshift surveys.

The method improves on the work by \citep{2007MNRAS.375..489H}. It
also uses the Hessian of the gravitational potential
\begin{equation}
T_{\alpha\beta} = \frac{\partial^2\phi}{\partial x_\alpha\partial x_\beta},
\end{equation}
%
where the physical gravitational potential has been normalized by
$4\pi G\bar{\rho}$ so that $\phi$ satisfies the Poisson
equation
\begin{equation}
\nabla^2\phi=\delta,
\end{equation}
%
with $\delta$ the dimensionless matter overdensity, $G$ the
gravitational constant and $\bar{\rho}$ the average density of the
Universe.

This tidal tensor can be represented by a real symmetric $3\times 3$
matrix with eigenvalues $\lambda_1>\lambda_2>\lambda_3$ and
eigenvectors ${\bf e}_1$, ${\bf e}_2$ and ${\bf e}_3$. The eigenvalues
are indicators of orbital stability along the directions defined by
the eigenvectors.

This method introduces a threshold $\lambda_{\rm th}$ to gauge the
strenght of the eigenvalues of the Tidal Shear Tensor. The number of
eigenvalues larger than the threshold is used to classify the cosmic
web into four kinds of environments: voids (3 eigenvalues smaller than
$\lambda_{\th}$), sheets (2), filaments (1) and knots (0).

In practice the density is interpolated over a grid using the particle
data and a Cloud-In-Cell scheme. The Poisson equation is solved in
Fourier space to obtain the potential over a grid. At each grid cell
the shear tensor is computed to obtain and store the corresponding
eigenvalues and eigenvectors. The grid cell has a size of $\sim
1$Mpc/h and the threshold is fixed to be $\lambda_{\th}=0.2$ as
suggested by previous studies that aim at capturing the visual
impression of the cosmic web \cite{2009MNRAS.396.1815F}.

%%%%%%%%%%References used
%@ARTICLE{2007MNRAS.375..489H,
%   author = {{Hahn}, O. and {Porciani}, C. and {Carollo}, C.~M. and {Dekel}, A.
%	},
%    title = "{Properties of dark matter haloes in clusters, filaments, sheets and voids}",
%  journal = {\mnras},
%   eprint = {astro-ph/0610280},
% keywords = {methods: N-body simulations , galaxies: haloes , cosmology: theory , dark matter , large-scale structure of Universe},
%     year = 2007,
%    month = feb,
%   volume = 375,
%    pages = {489-499},
%      doi = {10.1111/j.1365-2966.2006.11318.x},
%   adsurl = {http://adsabs.harvard.edu/abs/2007MNRAS.375..489H},
%  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
%}

%@ARTICLE{2009MNRAS.396.1815F,
%   author = {{Forero-Romero}, J.~E. and {Hoffman}, Y. and {Gottl{\"o}ber}, S. and
%	{Klypin}, A. and {Yepes}, G.},
%    title = "{A dynamical classification of the cosmic web}",
%  journal = {\mnras},
%archivePrefix = "arXiv",
%   eprint = {0809.4135},
% keywords = {methods: numerical , cosmology: large-scale structure of Universe},
%     year = 2009,
%    month = jul,
%   volume = 396,
%    pages = {1815-1824},
%      doi = {10.1111/j.1365-2966.2009.14885.x},
%   adsurl = {http://adsabs.harvard.edu/abs/2009MNRAS.396.1815F},
%  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
%}


\section{Modelling with Matrices}

    \subsection{Intro to Matrices}
    A matrix is a rectangular array of numbers that is used to organize data.
    If you are used to computer programming, you will recognize them as 2D arrays.
    If you are good with set notation, they are in essence, a set of sets.
    Matrices follow the format below:
    \begin{equation*}
        X =
        \begin{bmatrix}
            a & b & c \\
            d & e & f
        \end{bmatrix}
    \end{equation*}
    Matrices have various properties, and unique terms created to describe them.
    For instance, looking at the matrix above, we can safely say that $a$ in an entry, as defined below.
    \begin{definition}
        Entry: An entry refers to a number in a matrix.
    \end{definition}
    How did we figure out the number of entries in the matrix? Simple, we can use the matrix's \emph{dimensions} to figure that out.
    \begin{definition}Dimensions:
        These refer to the size of the matrix, described by the number of rows and columns.
        The formula below is used to solve for dimensions, where D is the result, $R$ is the rows, and $C$ is the columns.
        \begin{equation*}
                D = R \cdot C
        \end{equation*}
    \end{definition}
    Matrix's entries are numbered in a way so that it is easy to refer to a specific entry.
    Capital letters are used for the names of matrices, and lowercase characters are used for the names of entries.
    Shown below are the entries named, with numbers instead of letters.
    The first character represents the y value, whereas the second represents the x value.
    In the equation below, $a_{13}$ represents the entry in row 1, column 3.
    \begin{equation*}
        X =
        \begin{bmatrix}
            a_{11} & a_{12} & a_{13} \\
            a_{21} & a_{22} & a_{23}
        \end{bmatrix}
    \end{equation*}

    \subsection{Variations and Classifications of Matrices}
    Matrices can be transformed, and compared to each other, much like sets.
    The definitions of the variations and classifications of the matrices are listed below.
    \begin{definition}Transpose Matrix:
        A transpose matrix is a matrix obtained by interchanging the rows and columns. This is written as $X^{+}$. Shown below is Matrix W, and the transposed equivalent.
        \begin{equation*}
            W =
            \begin{bmatrix}
                a & b & c \\
                d & e & f
            \end{bmatrix}
            -> X^{+} =
            \begin{bmatrix}
                a & d \\
                b & e \\
                c & f
            \end{bmatrix}
        \end{equation*}
    \end{definition}

    \begin{definition}Column Matrix:
        A Column Matrix is a matrix that has only one x value. Instead of rows and columns, it is merely a column.
        For instance, Matrix X in this case would be a Column Matrix.
        \begin{equation*}%Column Matrix
            X =
            \begin{bmatrix}
                a&\\
                b&\\
                c
            \end{bmatrix}
        \end{equation*}
    \end{definition}

    \begin{definition}Row Matrix:
        A Row Matrix is a matrix that has only one y value. It is the transpose of the Column Matrix For instance, Matrix Y in this case would be a Row Matrix.
        \begin{equation*}%Row Matrix
            Y =
            \begin{bmatrix}
                a & b & c
            \end{bmatrix}
        \end{equation*}
    \end{definition}

    \begin{definition}Square Matrix:
        A Square Matrix is a matrix in which the rows and columns are equal.
        For instance, Matrix Z below is a Square Matrix.
        \begin{equation*}%Column Matrix
            Z =
            \begin{bmatrix}
                a & b & c\\
                d & e & f\\
                g & h & i
            \end{bmatrix}
        \end{equation*}
    \end{definition}

    \subsection{Basic Operations with Matrices}
        Matrices may only be added or subtracted when they have the same dimensions.
        Adding matrices is done by adding entry by entry to the other matrix.
        $A_{12}$ will be added with $B_{12}$ and so forth.
        Shown below is a valid example of adding with matrices in which the entries are labeled to show proper addition:
        \begin{equation*}
            \begin{bmatrix}
                a_{11} & a_{12} & a_{13} \\
                a_{21} & a_{22} & a_{23}
            \end{bmatrix}
            +
            \begin{bmatrix}
                b_{11} & b_{12} & b_{13} \\
                b_{21} & b_{22} & b_{23}
            \end{bmatrix}
            =
            \begin{bmatrix}
                (a+b)_{11} & (a+b)_{12} & (a+b)_{13} \\
                (a+b)_{21} & (a+b)_{22} & (a+b)_{23}
            \end{bmatrix}
        \end{equation*}
        They may also be multiplied by a coefficient, by multiplying the coefficient with every entry in the matrix.
        Shown below is a proper solution in which a matrix is multiplied by a coefficient, with entries labeled:
         \begin{equation*}
            x
            \begin{bmatrix}
                y_{11} & y_{12} & y_{13} \\
                y_{21} & y_{22} & y_{23}
            \end{bmatrix}
            =
            \begin{bmatrix}
                xy_{11} & xy_{12} & xy_{13} \\
                xy_{21} & xy_{22} & xy_{23}
            \end{bmatrix}
        \end{equation*}

    \subsection{Multiplying Matrices Together}
        Matrices may also be multiplied with other matrices, yet to do so we must follow certain rules.
        Let's say we have Matrix A and Matrix B as shown below.
        To multiply, we need to make sure the \emph{inner dimensions} are the same.
        We know that Matrix A's dimensions are $3 \cdot 2$, whereas for Matrix B, they are $2 \cdot 3$.
        Since we know this, we can find the \emph{inner dimensions}.
        The \emph{inner dimensions}, equals the columns of Matrix A and the rows of Matrix B.
        In this case, that would work out to be $3$ and $3$.
        Since they are the same, we are allowed to multiply these matrices.
        \begin{equation*}%Demonstrating Matrices.
            A =
            \begin{bmatrix}
                a & b & c\\
                d & e & f\\
            \end{bmatrix}
            B =
            \begin{bmatrix}
                u & v\\
                w & x\\
                y & z
            \end{bmatrix}
        \end{equation*}
        Next, the rows of Matrix A times the columns of Matrix B, also known as the \emph{outer dimensions}, will give us the dimensions of the resultant matrix.
        Since Matrix A has two rows, and Matrix B has two columns, our resultant Matrix will possess the dimensions $2\cdot2$.
        Shown below is the updated equation.
        \begin{equation*}
            \begin{bmatrix}
                a & b & c\\
                d & e & f\\
            \end{bmatrix}
        \cdot
            \begin{bmatrix}
                u & v\\
                w & x\\
                y & z
            \end{bmatrix}
            =
            \begin{bmatrix}
                ?&?\\
                ?&?
            \end{bmatrix}
        \end{equation*}
        Multiplying them is a bit strange at first, because to do it, we multiply row by row.
        Starting at the first row in Matrix A, we can see we have $a_{11},b_{12},c_{13}$ as our values, with their locations being labeled for convenience.
        We must multiply these values each column in Matrix B, starting with the values $u_{11},w_{21},y_{31}$.
        To find entry (1,1) we can use the formula shown below, where $E$ represents the entry, $r$ represents a row in Matrix A, and c represents a column in Matrix B.
        \begin{equation*}
            E_{11} = r_{11} \cdot c_{11}  +  r_{12} \cdot c_{21}  +   r_{13} \cdot c_{31}
        \end{equation*}
        If you want to do it just by eye, you can multiply $A_{11}$ by $B_{11}$, $A_{12}$ by $B_{21}$, and $A_{13}$ by $B_{31}$, and add them all up.
        It will seem totally awful at first, but after you do a couple it'll all work out fine.

    \subsection{Identity Matrices}
        An Identity Matrix is a $n\cdot n$ size matrix in which every number along the main diagonal is a 1, and the rest are zeros.
        Shown below is Matrix I, a $3\cdot 3$ Identity Matrix.
        \begin{equation*}
        I =
            \begin{bmatrix}
                1 & 0 & 0\\
                0 & 1 & 0\\
                0 & 0 & 1
            \end{bmatrix}
        \end{equation*}
        This matrix is very special, as any square matrix multiplied by its inverse will equal an Identity Matrix.
        We can further formalize this assumption, by referring to a given matrix as Matrix A.
        \begin{equation*}
            A\cdot A^{-1} = I
        \end{equation*}
        Its too bad we can't divide matrices, how can we ever solve for I now? Good thing we can use something called an Inverse Matrix.

    \subsection{Finding Inverse Matrices}
        Finding the inverse of large matrices is one of the most mathematically intensive processes out there.
        Good thing we'll only be finding the inverse of a $2\cdot2$ matrix instead. To find the inverse of a $2\cdot 2$ matrix, we must use first define the matrices.
        \begin{equation*}
        A =
            \begin{bmatrix}
                a & b\\
                c & d
            \end{bmatrix}
        \end{equation*}
        Now that we have the starting matrix, it is time to find the inverse. To find it, we must multiply the determinant, by the matrix, and swap several spots of numbers. Again, sounds odd, but we just need to follow the formula below.
        \begin{equation*}
        A^{-1} = \frac{1}{ad-bc}
            \begin{bmatrix}
                d & -b\\
                -c & a
            \end{bmatrix}
        \end{equation*}
        After this step, simply multiply the determinant into the matrix, and you have found $A^{-1}$!


\subsection{Document embedding}
    \label{sec:document_embedding}

    \paragraph{TF-IDF.} Before texts can be fed into machine learning models, they need to become numbers, usually vectors. These vectors are called text embeddings, and, in the case of long documents, document embeddings. There are many document embedding techniques available, but, for the scope of this work, we will use TF-IDF \cite{robertson2004understanding}.

    Briefly, for each document, and for each word, TF-IDF counts the frequency of this word relative to this document (term frequency - TF). This count is ``normalized'' by the rarity of the word in the entire set of documents (inverse document frequency - IDF). More formally, for a term $t$ inside document $d$ with frequency $f_{t, d}$,
    \[\text{tf-idf}(t, d) = \text{tf}(t, d) \times \text{idf}(t),\]
    \[\text{tf}(t, d) = \frac{f_{t, d}}{\sum_{t' \in d} f_{t', d}}, \ \ \text{idf}(t) = \ln \frac{1+n}{1+\text{df}(t)}+1,\]
    where $n$ is the number of documents and $\text{df}(t)$ is the number of documents that contain $t$. The TF-IDF weights, in the end, construct a vector that represents the document in a very high dimensional space.

    \paragraph{Dimensionality reduction.} Because vectors' dimensionality is too high, we perform a dimensionality reduction to have the vectors treatable by our models. The vectors will be reduced from dimensionality of thousands to dimensionality 50 using truncated singular value decomposition (SVD). It consists of decomposing our matrix $X$ (with rows being the TF-IDF vectors) in
    \[X = U \Sigma V^\intercal,\]
    where $\Sigma$ is a diagonal matrix of the singular values of $X$. If we take only the $k$ greatest singular values, we will have
    \[X_{n \times p} = U_{n \times k} \Sigma_{k \times k} \left[V^\intercal\right]_{k \times p},\]
    and our new data points will be $X_{n \times p} V_{p \times k}$, with reduced dimensionality $k$. This is very similar to principal component analysis, and equivalent when data is centered (mean zero in each component).


\subsection{Newton's Method}\label{sec:Newton}
A well known numeric method is \ifont{Newton's Method} (also sometimes
referred to as \ifont{Newton-Raphson's Method}), named after Isaac Newton
and Joseph Raphson.  This method is used to find roots, or
$x$-intercepts, of a function.  While we may be able to find the roots
of a polynomial which we can easily factor, we saw in the previous
chapter on {\bf Limits}, that for example the function $e^x + x = 0$
has a solution ($i.e.$ root, or $x$-intercept) at $x \approx
-0.56714$.  By the \ifont{Intermediate Value Theorem} we know that the
function $e^x + x = 0$ does have a solution. We cannot here simply
solve for such a root algebraically, but we can use a numerical method
such as $Newton's$.  Such a process is typically classified as an
$iterative$ method, a name given to a technique which involves
repeating similar steps until the desired accuracy is obtained.   Many
computer $algorithms$ are coded with a for-loop, repeating an
iterative step to converge to a solution.

The idea is to start with an initial value $x_0$ (approximating the
root), and use linear approximation to create values $x_1$, $x_2$, $\cdots$ getting closer and closer to a root.

The first value $x_1$ corresponds to the intercept of the tangent line
of $f(x_0)$ with the $x$-axis, which is:
\[ x_1 = x_0 -\frac{f(x_0)}{f'(x_0)} \]

\figure[!ht]
$$\includegraphics[width=1.75in]{images/newton_figure_1}$$
\caption{First iteration of Newton's Method. \label{fig:Newton1}}
\endfigure

We can see in Figure~\ref{fig:Newton1}, that if we compare the point $(x_0,0)$ to
$(x_1,0)$, we would likely come to the conclusion that $(x_1,0)$ is
closer to the actual root of $f(x)$ than our original guess,
$(x_0,0)$.  As will be discussed, the choice of $x_0$ must be done
correctly, and it may occur that $x_1$ does not yield a better
estimate of the root.

Newton's method is simply to repeat this process again and again
in an effort to obtain a more accurate solution.  Thus at the next step we obtain:

\[ x_2 = x_1 -\frac{f(x_1)}{f'(x_1)} \]

\figure[!ht]
$$\includegraphics[width=1.75in]{images/newton_figure_2}$$
\caption{Second iteration of Newton's Method. \label{fig:Newton2}}
\endfigure

We can now clearly see how $(x_2,0)$ is a better estimate of the root
of $f(x)$, rather than any of the previous points.  Moving forward, we
will get:
\[ x_3 = x_2 -\frac{f(x_2)}{f'(x_2)} \]
Rest assured, $(x_3,0)$ will be an even better estimate of the root!  We express the general iterative step as:
\[ x_{n+1} = x_n -\frac{f(x_n)}{f'(x_n)} \]

The idea is to iterate these steps to obtain the desired accuracy. Here is an example.

\begin{example}{Newton method to approximate roots}{newtonroot}
Use Newton's method to approximate the roots of $f(x)=x^3-x+1$.
\end{example}

\begin{solution}
You can try to find solve the equation algebraically to see that this
is a difficult task, and thus it make sense to try a numerical method
such as Newton's.

To find an initial value $x_0$, note that $f(-1)=-5$ and $f(0)=1$,
and by the Intermediate Value Theorem this $f$ has a root between these two values, and we decide to start with $x_0=-1$ (you can try other values to see what happens).

Note that $f'(x)=3x^2-1$, and thus we get
\[ x_{n+1} = x_n -\frac{f(x_n)}{f'(x_n)} = x+n - \frac{x^3_n-x_n+1}{3x^2_n-1} \]
Thus we can produce the following values (try it):
\[ \begin{array}{l}
x_0 = -1 \\
x_1= -1.5000\\
x_2 = -1.347826.. \\
x_3= -1.325200.. \\
x_4 = -1.324718.. \\
x_5 = -1.324717..  \\
x_6 = -1.324717..  \\
\cdots
\end{array} \]
and we can now approximate the root as $-1.324717$.
\end{solution}

As with any numerical method, we need to be aware of the weaknesses of
any technique we are using.

\figure[!ht]
$$\includegraphics[width=2.5in]{images/newton_figure_4}$$
\caption{Function with three distinct solutions. \label{fig:Newton4}}
\endfigure

%% need an easy example here

If we know our root is somewhere near $a$, we would make our guess
$x_0=a$.  Generally speaking, a good practice is to make our guess as
close to the actual root as possible. In some cases we may have no idea where the root is, so it would be prudent to
perform the algorithm several times on several different initial
guesses and analyze the results.

For example we can see in Figure~\ref{fig:Newton4} that $f(x)$ in fact has three roots, and depending on our initial
guess, we may get the algorithm to converge to different roots.  If we
did not know where the roots were, we would try the technique several
times.  In one instance, if our initial guess was $x_a$, we'd
likely converge to $(a, 0)$.  Then if we were to choose another
guess, $x_b$, then we'd likely converge to $(b, 0)$.
Eventually, using various initial guesses we'd get one of three roots:
$a$, $b$, or $c$.  Under these circumstances we can
clearly see the effectiveness of this numeric method.

\figure[!ht]
$$\includegraphics[width=3.5in]{images/newton_figure_3}$$
\caption{Newton's Method applied to $\sin x$. \label{fig:Newton3}}
\endfigure

As another example if we attempt to use $Newton's$ $Method$ on $f(x) =
\sin x$ using $x_0=\pi/2$, then $f'(x_0)=0$ so $x_1$ is undefined and we cannot proceed.
Even in general $x_{n+1}$ is typically nowhere near $x_n$,
and in general not converging to the root nearest to our initial guess
of $x_0$.  In effect, the algorithm keeps "bouncing around". An example of which is depicted in Figure~\ref{fig:Newton3}. Based on
our initial guess for such a function, the algorithm may or may not
converge to a root, or it may or may not converge to the root {\bf
closest} to the initial guess.  This gives rise to the more common
issue: Selection of the initial guess, $x_0$.

Here is a summary.

{\bf Key Points in using Newton's method to approximate a root of $f(x)$}

\begin{enumerate}
\item Choosing $x_0$ as close as possible to the root we wish to find.
\item A guess for $x_0$ which makes the algorithm ``bounce around'' is considered $unstable$.
\item Even the smallest changes to $x_0$ can have drastic effects: We may converge to another root,
we may converge very slowly (requiring many more iterations), or we may encounter an unstable point.
\item We may encounter a $stationary$ $point$ if we choose $x_0$ such that $f'(x)=0$ ($i.e.$ at a critical point!) in which case the algorithm fails.
\end{enumerate}

This is all to say that your initial guess for $x_0$ can be extremely
important.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:Newton}}

\begin{enumialphparenastyle}

%%%%%%%%%%
\begin{ex}
Use Newton's Method to find all roots of $f(x)=3x^2-9x-11$. (Hint: use Intermediate Value Theorem to choose an appropriate $x_0$)
\begin{sol}
	Notice that $f(-2)=19$, $f(0)=-11$, and $f(5)=19$ and $f$ is a continuous function. By the
	Intermediate Value Theorem there exists a root in $[-2,0]$ and $[0,5]$. Choose $x_0=0$, then $x_4\approx -0.93242$.
	Choose $x_0=5$, then $x_4\approx 3.93242$.
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Consider $f(x)=x^3-x^2+x-1$.
\begin{enumerate}
	\item	Using initial approximation $x_0=2$, find $x_4$.
	\item	What is the exact value of the root of $f$? How does this compare to our approximation $x_4$ in part (a)?
	\item	What would happen if we chose $x_0=0$ as our initial approximation?
\end{enumerate}
\begin{sol}
\begin{enumerate}
	\item	$x_4\approx 1.00022\ldots$
	\item	$x=1$ is the root of $f$. Our approximation in part (a) was correct to 3 decimal places.
	\item	$x_1=1$. The root is found in one iteration of Newton's Method.
\end{enumerate}
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Consider $f(x)=\sin x$. What happens when we choose $x_0=\pi/2$? Explain.
\begin{sol}
	$\cos (\pi/2)=0$, so $x_1$ is undefined.
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\documentclass[11pt]{article}

\usepackage{amsmath}
\usepackage[margin=1in]{geometry}

\begin{document}

\section{Cold Neutrals}

The number of reactions per unit time per unit volume is given by

\begin{equation}
	R_{12} (\mathbf{v_i},\ \mathbf{v_n}) = \frac{n_i n_n}{1 + \delta_{12}} v_r \sigma(v_r) f_i(\mathbf{v_i}) f_n(\mathbf{v_n}) \mathrm{d}^3 \mathbf{v_i} \mathrm{d}^3 \mathbf{v_n}
\end{equation}

Where $\delta_{ij} = 1$ only if $i = j$, and it 0 otherwise, is included to avoid double counting when considering intra-species reactions.

We can plug in the Maxwellian for ions in the rotating plasma with a bulk fluid velocity of $\mathbf{v_f}$

\begin{equation}
	f_i(\mathbf{v_i}) = \left( \frac{m_i}{2 \pi T_i} \right)^{\frac{3}{2}} \exp \left( -\frac{m_i (\mathbf{v_i} - \mathbf{v_f})^2}{2 T_i} \right)
\end{equation}

And the delta function for the cold neutrals

\begin{equation}
	f_n(\mathbf{v_n}) = \delta( \mathbf{v_n} )
\end{equation}

And we get

\begin{equation}
	R_{12} (\mathbf{v_i},\ \mathbf{v_n}) = \Lambda v_r \sigma(v_r) \exp \left[ - \frac{m_i (\mathbf{v_i} - \mathbf{v_f})^2}{2 T_i} \right] \delta( \mathbf{v_n} ) \mathrm{d}^3 \mathbf{v_i} \mathrm{d}^3 \mathbf{v_n}
\end{equation}

Where

\begin{equation}
	\Lambda = \frac{n_i n_n}{1 + \delta_{12}} \left( \frac{m_i}{2 \pi T_i} \right)^{\frac{3}{2}}
\end{equation}

We then make the assumption that $v_r \approx v_i$ and make a transformation into spherical coordinates where the Jacobian is $\mathrm{d}^3 \mathbf{v_i} = v_i^2 \sin \theta_i \mathrm{d} v_i \mathrm{d} \theta_i \mathrm{d} \phi_i$. We also align the z-axis of the transform with the fluid velocity:

\begin{align*}
	v_{ix} &= v_i \sin \theta_i \cos \phi_i, \quad &v_{fx} &= 0 \\
	v_{iy} &= v_i \sin \theta_i \sin \phi_i, \quad &v_{fy} &= 0 \\
	v_{iz} &= v_i \cos \theta_i, \quad &v_{fz} &= v_f
\end{align*}

So that the integrand now becomes

\begin{equation}
	R_{12} (v_i) = \Lambda v_i^3 \sigma(v_i) \sin \theta_i \exp \left[ -\frac{m_i (v_i^2 + v_f^2 - 2 v_i v_f \cos \theta_i)}{2 T_i} \right] \delta( \mathbf{v_n} ) \mathrm{d} v_i \mathrm{d} \theta_i \mathrm{d} \phi_i  \mathrm{d}^3 \mathbf{v_n}
\end{equation}

Only the delta function is a function of $\mathbf{v_n}$, so that integrates out to 1. We can then solve the integral up to $v_i$.

\begin{align}
	R_{12} &= \Lambda \exp \left( -\frac{m_i v_f^2}{2 T_i} \right)  \int_0^\infty \int_0^{\pi} \int_0^{2 \pi} v_i^3 \sigma(v_i) \sin \theta_i \exp \left[ -\frac{m_i (v_i^2 - 2 v_i v_f \cos \theta_i)}{2 T_i} \right] \mathrm{d} \phi_i \mathrm{d} \theta_i \mathrm{d} v_i \\
	&= 2 \pi \Lambda \exp \left( -\frac{m_i v_f^2}{2 T_i} \right)  \int_0^\infty \int_0^{\pi} v_i^3 \sigma(v_i) \sin \theta_i \exp \left[ - \frac{m_i (v_i^2 - 2 v_i v_f \cos \theta_i)}{2 T_i} \right] \mathrm{d} \theta_i \mathrm{d} v_i \\
	&= \frac{4 \pi \Lambda T_i}{m_i v_f} \exp \left( -\frac{m_i v_f^2}{2 T_i} \right)  \int_0^\infty v_i^2 \sigma(v_i) \sinh \left( \frac{m_i v_i v_f}{T_i} \right) \exp \left( -\frac{m_i v_i^2}{2 T_i} \right) \mathrm{d} v_i \\
	&= \frac{2 n_i n_n}{v_f (1 + \delta_{12})} \left( \frac{m_i}{2 \pi T_i} \right)^{\frac{1}{2}} \exp \left( -\frac{m_i v_f^2}{2 T_i} \right)  \int_0^\infty v_i^2 \sigma(v_i) \sinh \left( \frac{m_i v_i v_f}{T_i} \right) \exp \left( -\frac{m_i v_i^2}{2 T_i} \right) \mathrm{d} v_i
\end{align}

We can then define everything in terms of the mach number normalized to the sound speed, $M = \frac{v_f}{v_s}$, where $v_s = \sqrt{\frac{Z T_e}{m_i}}$ and $Z$ is the ion charge state and $T_e$ the electron temperature. The thermal speed is $v_{th} = \sqrt{\frac{2 T_i}{m_i}}$, so we can say that the thermal Mach number is $M_{th} = M \sqrt{\frac{Z T_e}{2 T_i}}$. We can also simplify by expanding the $\sinh$ term.

\begin{align}
	R_{12} &= \frac{n_i n_n}{\sqrt{\pi} (1 + \delta_{12})} \frac{1}{v_f v_{th}} \exp \left( -\frac{v_f^2}{v_{th}^2} \right)  \int_0^\infty v_i^2 \sigma(v_i) \left[ \exp \left( \frac{2 v_i v_f}{v_{th}^2} \right) - \exp \left( -\frac{2 v_i v_f}{v_{th}^2} \right) \right] \exp \left( -\frac{v_i^2}{v_{th}^2} \right) \mathrm{d} v_i\\
	&= \frac{n_i n_n}{M_{th} v_{th}^2 \sqrt{\pi} (1 + \delta_{12})} \int_0^\infty v_i^2 \sigma(v_i) \left\lbrace \exp \left[ -\left( M_{th} - \frac{v_i}{v_{th}} \right)^2 \right] - \exp \left[ -\left( M_{th} + \frac{v_i}{v_{th}} \right)^2 \right] \right\rbrace \mathrm{d} v_i
\end{align}

Then making the substitution that $u = \frac{v_i}{v_{th}}$ we're left with

\begin{equation}
	R_{12} = \frac{n_i n_n v_{th}}{M_{th} \sqrt{\pi} (1 + \delta_{12})} \int_0^\infty u^2 \sigma(v_{th} u) \left\lbrace \exp \left[ -\left( M_{th} -  u \right)^2 \right] - \exp \left[ -\left( M_{th} + u \right)^2 \right] \right\rbrace \mathrm{d} u
\end{equation}

The same can be applied to electron impact cross sections, where $v_{th} = \sqrt{\frac{2 T_e}{m_e}}$ and $M_{th} = M \sqrt{\frac{Z}{2}}$

\begin{equation}
	R_{12} = \frac{n_e n_n v_{th}}{M_{th} \sqrt{\pi} (1 + \delta_{12})} \int_0^\infty u^2 \sigma(v_{th} u) \left\lbrace \exp \left[ -\left( M_{th} -  u \right)^2 \right] - \exp \left[ -\left( M_{th} + u \right)^2 \right] \right\rbrace \mathrm{d} u
\end{equation}

\section{Hot Neutrals}

Now looking at the regime where a charge exchange has occurred, and the neutrals are `hot' and moving at the fluid velocity, $v_f$, with a thermal spread equal to that of the ions, $T_i$. In the rotating frame of reference, there is no fluid velocity offset in the Maxwellian. Therefore, we can look at what the rate reaction is for ion and electon impact. The distribution function for the hot neutrals is Maxwellian.

\begin{equation}
	f_n(\mathbf{v_n}) = \left( \frac{m_n}{2 \pi T_n} \right)^{\frac{3}{2}} \exp \left( -\frac{m_n v_n^2}{2 T_n} \right)
\end{equation}

\subsection{Ion Impact}

The distribution functions for the ions in the rotating reference frame is

\begin{equation}
	f_i(\mathbf{v_i}) = \left( \frac{m_i}{2 \pi T_i} \right)^{\frac{3}{2}} \exp \left( -\frac{m_i v_i^2}{2 T_i} \right)
\end{equation}

Assuming that the thermal spread is equal for the neutral and ion, we have $T_i = T_n$. The rate coefficient for two Maxwellians in thermal equilibrium is

\begin{equation}
	R_{12} = 4 \pi \left( \frac{\mu}{2 \pi T_i} \right)^{3/2} \int_0^\infty \sigma(v_r) v_r^3 \exp{ \left( -\frac{\mu v_r^2}{2 T_i} \right) }
\end{equation}

Where $\mu = \frac{m_i m_n}{m_i + m_n}$ is the reduced mass, and $\mathbf{v_r} = |\mathbf{v_i} - \mathbf{v_n}|$ is the relative velocity. We can convert this expression to be in terms of center-of-mass energy, $\varepsilon = \frac{1}{2} \mu v_r^2$.

\begin{equation}
	R_{12} = \frac{4}{(2 \pi \mu)^{\frac{1}{2}} T_i^{\frac{3}{2}}} \int_0^\infty \varepsilon \sigma(\varepsilon) \exp{ \left( -\frac{\varepsilon}{T_i} \right) } \mathrm{d} \varepsilon
\end{equation}

\subsection{Electron Impact}

In a collision between an electron hot neutral and a hot neutral, both are moving at the fluid velocity with a thermal spread of $T_e$ and $T_i$, respectively. However, the electrons have a significantly higher thermal velocity, so the distribution function for the neutrals is again a delta function. The rate coefficient is then

\begin{equation}
	R_{12} (\mathbf{v_e},\ \mathbf{v_n}) = \Lambda v_r \sigma(v_r) \exp \left[ - \frac{m_e v_e^2}{2 T_e} \right] \delta( \mathbf{v_n} ) \mathrm{d}^3 \mathbf{v_e} \mathrm{d}^3 \mathbf{v_n}
\end{equation}

Where $\Lambda$ is a prefactor

\begin{equation}
	\Lambda = n_e n_n \left( \frac{m_e}{2 \pi T_e} \right)^{\frac{3}{2}}
\end{equation}

Because $m_e \ll m_i$, and $v_e \gg v_n$, we can assume that $v_r \approx v_e$. We can also change to the center-of-mass energy, $\varepsilon = \frac{1}{2} \mu v_r^2 \approx \frac{1}{2} m_e v_e^2$

\begin{equation}
	R_{12} = \frac{n_e n_n}{\sqrt{\pi m_e}} \left( \frac{2}{T_e} \right)^{\frac{3}{2}}  \int_0^\infty \varepsilon \sigma(\varepsilon) \exp{ \left( -\frac{\varepsilon}{T_e} \right) } \mathrm{d} \varepsilon
\end{equation}

\end{document}


\section{Radix Sort}
\label{sec:200}

Radix Sort is a non-comparative sorting algorithm that works on integer or string keys. The ordering is done by grouping keys into buckets, according to their individual digits.\\

The approach used here is the \emph{Least Significant Digit} one, which is iterative in nature, running through all the elements once for every digit

As a small example, consider the following list of integers:

\begin{lstlisting}
	arr = [170, 45, 75, 90, 24, 2, 66]
\end{lstlisting}

In the first iteration, the least significant digit is considered. After sending each key into the correct bucket, the result is the following:

\begin{lstlisting}
	0: [170, 90],
	2: [2],
	4: [24],
	5: [45, 75],
	6: [66]
\end{lstlisting}

And after merging all buckets:

\begin{lstlisting}
	arr = [170, 90, 2, 24, 45, 75, 66]
\end{lstlisting}

This completes the first iteration. The process is repeated until there are no more digits do process.


% !Mode:: "TeX:UTF-8"
% !TEX program  = xelatex
\section{LPPL Model Implement}
The formula in 1.4 of LPPL describes a nonlinear function of logarithm asset price about time, involving logarithm, periodic and super-exponential tendency with 7 unknown parameters. We chose the Levenberg-Marquardt algorithm (LM) to find out the global optimal solution about seven unknown parameters in this non-linear least squares problem.

Firstly, we aim to minimize the distance between real price data with the estimated price from the model. We measured the $L_2$-norm distance as aim function for better curve fitting after comparing the consequence with $L_1$-norm.

Secondly, we put several limits and bounds on parameters and fitted the data with MATLAB. The limits on parameters refer to the conclusive analysis of successful experiments by Professor Didier with his group.

Thirdly, we tested different time intervals(Start Date to End Date) on three different stocks --- SSE Composite Index, NASDAQ, HK Seng Index. The number of iterations and the limit conditions on parameters, to some extent, did affect the consequences of critical time to get closer to the real historical crush points. The LPPL model is only valid in the super-exponential tendency.

In our report, we used the  MATLAB to fit seven parameters especially the critical time, which can verify the precision of our model and algorithm.

Finally, the figures of three stocks embody the precise prediction on the critical time of bubbles ending.

Detailed results are in the Appendix.


\chapter{Shor's algorithm}
OK, now for Shor's Algorithm:
how to factor $M = pq$ in $O\left( (\log M)^2 \right)$ time.

\section{The classical (inverse) Fourier transform}
The ``crux move'' in Shor's algorithm is the so-called
quantum Fourier transform.
The Fourier transform is used to extract \emph{periodicity} in data,
and it turns out the quantum analogue is a lot faster than the classical one.

Let me throw the definition at you first.
Let $N$ be a positive integer, and let $\omega_N = \exp\left( \frac{2\pi i}{N} \right)$.
\begin{definition}
	Given a tuple of complex numbers
	\[ \left( x_0, x_1, \dots, x_{N-1} \right) \]
	its \vocab{discrete inverse Fourier transform} is
	the sequence $(y_0, y_1, \dots, y_{N-1})$ defined by
	\[ y_k = \frac1N \sum_{j=0}^{N-1} \omega_N^{jk} x_j. \]
	Equivalently, one is applying the matrix
	\[
		\frac 1N
		\begin{bmatrix}
			1 & 1 & 1 & \dots & 1 \\
			1 & \omega_N & \omega_N^2 & \dots & \omega_N^{N-1} \\
			1 & \omega_N^2 & \omega_N^4 & \dots & \omega_N^{2(N-1)} \\
			\vdots & \vdots & \vdots & \ddots & \vdots \\
			1 & \omega_N^{N-1} & \omega_N^{2(N-1)} & \dots & \omega_N^{(N-1)^2}
		\end{bmatrix}
		\begin{bmatrix} x_0 \\ x_1 \\ \vdots \\ x_{N-1} \end{bmatrix}
		=
		\begin{bmatrix} y_0 \\ y_1 \\ \vdots \\ y_{N-1} \end{bmatrix}.
	\]
\end{definition}
The reason this operation is important is because it lets
us detect if the $x_i$ are periodic:
\begin{example}
	[Example of discrete inverse Fourier transform]
	Let $N = 6$, $\omega = \omega_6 = \exp(\frac{2\pi i}{6})$
	and suppose $(x_0,x_1,x_2,x_3,x_4,x_5)=(0,1,0,1,0,1)$
	(hence $x_i$ is periodic modulo $2$).
	Thus,
	\begin{align*}
		y_0 &= \tfrac16\left(\omega^0 + \omega^0+ \omega^0\right) = 1/2 \\
		y_1 &= \tfrac16\left(\omega^1 + \omega^3 + \omega^5\right) = 0 \\
		y_2 &= \tfrac16\left( \omega^2 + \omega^{6} + \omega^{10} \right) = 0 \\
		y_3 &= \tfrac16\left( \omega^3 + \omega^9 + \omega^{15} \right) = -1/2 \\
		y_4 &= \tfrac16\left( \omega^4 + \omega^{12} + \omega^{20} \right) = 0 \\
		y_5 &= \tfrac16\left( \omega^5 + \omega^{15} + \omega^{25} \right) = 0.
	\end{align*}
	Thus, in the inverse transformation the ``amplitudes''
	are all concentrated at multiples of $3$;
	thus this reveals the periodicity of the original
	sequence by $\frac N3 = 2$.
\end{example}
More generally, given a sequence of $1$'s appearing with period $r$,
the amplitudes will peak at inputs which are divisible by $\frac{N}{\gcd(N,r)}$.
\begin{remark}
	The fact that this operation is called the ``inverse''
	Fourier transform is mostly a historical accident
	(as my understanding goes).
	Confusingly, the corresponding quantum operation is the
	(not-inverted) Fourier transform.
\end{remark}
If we apply the definition as written, computing the transform takes $O(N^2)$ time.
It turns out that by an algorithm called the \vocab{fast Fourier transform}
(whose details we won't discuss), one can reduce this to $O(N \log N)$ time.
However, for Shor's algorithm this is also insufficient;
we need something like $O\left( (\log N)^2 \right)$ instead.
This is where the quantum Fourier transform comes in.

\section{The quantum Fourier transform}
Note that to compute a Fourier transform, we need to multiply an $N \times N$ matrix
with an $N$-vector, so this takes $O(N^2)$ multiplications.
However, we are about to show that with a quantum computer,
one can do this using $O( (\log N)^2 )$ quantum gates when $N = 2^n$,
on a system with $n$ qubits.

First, some more notation:
\begin{abuse}
	In what follows, $\ket{x}$ will refer to
	$\ket{x_n} \otimes \ket{x_{n-1}} \otimes \dots \otimes \ket{x_1}$
	where $x = x_n x_{n-1} \dots x_1$ in binary.
	For example, if $n = 3$
	then $\ket{6}$ really means $\ket1 \otimes \ket1 \otimes \ket 0$.
\end{abuse}
Observe that the $n$-qubit space now has an
orthonormal basis $\ket0$, $\ket1$, \dots, $\ket{N-1}$

\begin{definition}
	Consider an $n$-qubit state
	\[ \ket\psi = \sum_{k=0}^{N-1} x_k \ket{k}. \]
	The \vocab{quantum Fourier transform} is defined by
	\[
		\UQFT(\ket\psi) = \frac{1}{\sqrt N}\sum_{j=0}^{N-1}
		\left( \sum_{k=0}^{N-1} \omega_N^{jk} \right) \ket{j}.
	\]
	In other words, using the basis $\ket0$, \dots, $\ket{N-1}$,
	$\UQFT$ is given by the matrix
	\[
		\UQFT = \frac{1}{\sqrt N}
		\begin{bmatrix}
			1 & 1 & 1 & \dots & 1 \\
			1 & \omega_N & \omega_N^2 & \dots & \omega_N^{N-1} \\
			1 & \omega_N^2 & \omega_N^4 & \dots & \omega_N^{2(N-1)} \\
			\vdots & \vdots & \vdots & \ddots & \vdots \\
			1 & \omega_N^{N-1} & \omega_N^{2(N-1)} & \dots & \omega_N^{(N-1)^2}
		\end{bmatrix}
	\]
\end{definition}
This is the exactly the same definition as before,
except we have a $\sqrt N$ factor added so that $\UQFT$ is unitary.
But the trick is that in the quantum setup, the matrix can be rewritten:
\begin{proposition}
	[Tensor representation]
	Let $\ket x = \ket{x_n x_{n-1} \dots x_1}$.
	Then
	\begin{align*}
		\UQFT( \ket{x_n x_{n-1} \dots x_1} )
		= \frac{1}{\sqrt N} &
		\left( \ket0 +\exp(2\pi i \cdot 0.x_1) \ket 1 \right) \\
		&\otimes \left( \ket0 +\exp(2\pi i \cdot 0.x_2x_1) \ket 1 \right) \\
		&\otimes \dots  \\
		&\otimes \left( \ket0 +\exp(2\pi i \cdot 0.x_n\dots x_1) \ket 1 \right)
	\end{align*}
\end{proposition}
\begin{proof}
	Direct (and quite annoying) computation.
	In short, expand everything.
\end{proof}

So by using mixed states, we can deal with the quantum Fourier transform
using this ``multiplication by tensor product'' trick that isn't possible classically.

Now, without further ado, here's the circuit.
Define the rotation matrices
\[ R_k = \begin{bmatrix} 1 & 0 \\ 0 & \exp(2\pi i/2^k) \end{bmatrix}. \]
Then, for $n=3$ the circuit is given by by using controlled $R_k$'s as follows:
\[
	\Qcircuit @C=1em @R=.7em {
		\lstick{\ket{x_3}} & \gate{H} & \gate{R_2} & \qw & \gate{R_3}  & \qw & \qw & \rstick{\ket{y_1}} \\
		\lstick{\ket{x_2}} & \qw & \ctrl{-1} & \gate{H} & \qw & \gate{R_2} & \qw & \rstick{\ket{y_2}} \\
		\lstick{\ket{x_1}} & \qw & \qw & \qw & \ctrl{-2} & \ctrl{-1} & \gate{H} & \rstick{\ket{y_3}} \\
	}
\]
\begin{exercise}
	Show that in this circuit, the image of of $\ket{x_3x_2x_1}$
	(for binary $x_i$) is
	\[
		\Big(\ket0+\exp(2\pi i \cdot 0.x_1) \ket1\Big)
		\otimes \Big(\ket0+\exp(2\pi i \cdot 0.x_2x_1) \ket1\Big)
		\otimes \Big(\ket0+\exp(2\pi i \cdot 0.x_3x_2x_1) \ket1\Big)
	\]
	as claimed.
\end{exercise}

For general $n$, we can write this as inductively as
\[
 \Qcircuit @C=1em @R=.7em {
  \lstick{\ket{x_n}}     & \multigate{5}{\text{QFT}_{n-1}} & \gate{R_n} & \qw                    & \qw & \cdots & & \qw                    & \qw & \cdots & & \qw              & \qw      & \rstick{\ket{y_1}} \qw \\
  \lstick{\ket{x_{n-1}}}     & \ghost{\text{QFT}_{n-1}}        & \qw                    & \gate{R_{n-1}} & \qw & \cdots & & \qw                    & \qw & \cdots & & \qw              & \qw      & \rstick{\ket{y_2}} \qw \\
  \lstick{\vdots\ \ }    & \pureghost{\text{QFT}_{n-1}}    &                        &                        &     &        & &                        &     &        & &                  &          & \rstick{\ \ \vdots} \\
  \lstick{\ket{x_i}}     & \ghost{\text{QFT}_{n-1}}        & \qw                    & \qw                    & \qw & \cdots & & \gate{R_i} & \qw & \cdots & & \qw              & \qw      & \rstick{\ket{y_{n-i+1}}} \qw \\
  \lstick{\vdots\ \ }    & \pureghost{\text{QFT}_{n-1}}    &                        &                        &     &        & &                        &     &        & &                  &          & \rstick{\ \ \vdots} \\
  \lstick{\ket{x_2}} & \ghost{\text{QFT}_{n-1}}        & \qw                    & \qw                    & \qw & \cdots & & \qw                    & \qw & \cdots & & \gate{R_2} & \qw      & \rstick{\ket{y_{n-1}}} \qw \\
  \lstick{\ket{x_1}}     & \qw                             & \ctrl{-6}               & \ctrl{-5}               & \qw & \cdots & & \ctrl{-3}               & \qw & \cdots & & \ctrl{-1}         & \gate{H} & \rstick{\ket{y_n}} \qw
 }
\]
\begin{ques}
	Convince yourself that when $n=3$ the two circuits displayed are equivalent.
\end{ques}

Thus, the quantum Fourier transform is achievable with $O(n^2)$ gates,
which is enormously better than the $O(N \log N)$ operations achieved by
the classical fast Fourier transform (where $N=2^n$).

\section{Shor's algorithm}
The quantum Fourier transform is the key piece of Shor's algorithm.
Now that we have it, we can solve the factoring problem.

Let $p,q > 3$ be odd primes, and assume $p \neq q$.
The main idea is to turn factoring an integer $M = pq$ into a problem
about finding the order of $x \pmod M$; the latter is a ``periodicity''
problem that the quantum Fourier transform will let us solve.
Specifically, say that an $x \pmod M$ is \emph{good} if
\begin{enumerate}[(i)]
	\ii $\gcd(x,M) = 1$,
	\ii The order $r$ of $x \pmod M$ is even, and
	\ii Factoring $0 \equiv (x^{r/2}-1)(x^{r/2}+1) \pmod M$,
	neither of the two factors is $0 \pmod M$.
	Thus one of them is divisible by $p$, and the other
	is divisible by $q$.
\end{enumerate}
\begin{exercise}
	[For contest number theory practice]
	Show that for $M = pq$ at least half of the residues
	in $\Zm M$ are good.
\end{exercise}

So if we can find the order of an arbitrary $x \in \Zm M$,
then we just keep picking $x$ until we pick a good one
(this happens more than half the time);
once we do, we compute $\gcd(x^{r/2}-1,M)$ using the Euclidean
algorithm to extract one of the prime factors of $M$, and we're home free.

Now how do we do this?  The idea is not so difficult:
first we generate a sequence which is periodic modulo $r$.
\begin{example}[Factoring $77$: generating the periodic state]
	Let's say we're trying to factor $M = 77$,
	and we randomly select $x = 2$, and want to find its order $r$.
	Let $n = 13$ and $N = 2^{13}$, and start by initializing the state
	\[ \ket\psi = \frac{1}{\sqrt N} \sum_{k=0}^{N-1} \ket k. \]
	Now, build a circuit $U_x$ (depending on $x=2$!)
	which takes $\ket k \ket 0$ to $\ket k \ket{2^k \mod M}$.
	Applying this to $\ket\psi \otimes \ket0$ gives
	\[ U(\ket\psi\ket0) =
		\frac{1}{\sqrt N} \sum_{k=0}^{N-1} \ket k \otimes \ket{2^k \mod M}. \]
	Now suppose we measure the second qubit, and get a state of $\ket{128}$.
	That tells us that the collapsed state now, up to scaling, is
	\[ (\ket{7} + \ket{7+r} + \ket{7+2r} + \dots) \otimes \ket{128}. \]
\end{example}
The bottleneck is actually the circuit $U_x$;
one can compute $x^k \pmod M$ by using repeated squaring,
but it's still the clumsy part of the whole operation.

In general, the operation is:
\begin{itemize}
	\ii Pick a sufficiently large $N = 2^n$ (say, $N \ge M^2$).
	\ii Generate $\ket\psi = \sum_{k=0}^{2^n-1} \ket{k}$.
	\ii Build a circuit $U_x$ which computes $\ket{x^k \mod M}$.
	\ii Apply it to get a state
	$\frac{1}{\sqrt N} \sum_{k=0}^{2^n-1} \ket k \otimes \ket{x^k \mod M}$.
	\ii Measure the second qubit to cause the first qubit to
	collapse to something which is periodic modulo $r$.
	Let $\ket\phi$ denote the left qubit.
\end{itemize}

Suppose we apply the quantum Fourier transform to the left qubit $\ket\phi$ now:
since the left bit is periodic modulo $r$, we expect the transform
will tell us what $r$ is.
Unfortunately, this doesn't quite work out, since $N$ is a power of two,
but we don't expect $r$ to be.

Nevertheless, consider a state
\[ \ket\phi = \ket{k_0} + \ket{k_0+r} + \dots \]
so for example previously we had $k_0=7$ if we measured $128$ on $x=2$.
Applying the quantum Fourier transform, we see that the
coefficient of $\ket j$ in the transformed image is equal to
\[
	\omega_N^{k_0j} \cdot
	\left( \omega_N^{0} + \omega_N^{jr} + \omega_N^{2jr}
	+ \omega_N^{3jr} + \dots \right)
\]
As this is a sum of roots of unity, we realize we have
destructive interference unless $\omega_N^{jr} = 1$ (since $N$ is large).
In other words, we approximately have
\[
	\UQFT(\ket\phi)
	\approx
	\sum_{\substack{0 \le j < N \\ jr/N \in \ZZ}} \ket j
\]
up to scaling as usual.
The bottom line is that
\begin{moral}
	If we measure $\UQFT\ket\phi$ we obtain a $\ket j$ such that
	$\frac{jr}{N}$ is close to an $s \in \ZZ$.
\end{moral}
And thus given sufficient luck we can use continued fractions
to extract the value of $r$.

\begin{example}
	[Finishing the factoring of $M = 77$]
	As before, we made an observation to the second qubit,
	and thus the first qubit collapses to the state
	$\ket\phi = \ket7 + \ket{7+r} + \dots$.
	Now we make a measurement and obtain $j = 4642$, which means that
	for some integer $s$ we have
	\[ \frac{4642r}{2^{13}} \approx s. \]
	Now, we analyze the continued fraction of $\frac{4642}{2^{13}}$;
	we find the first few convergents are
	\[
		0, \;
		1, \;
		\half, \;
		\frac{4}{7}, \;
		\frac{13}{23}, \;
		\frac{17}{30}, \;
		\frac{1152}{2033}, \;
		\dots
	\]
	So $\frac{17}{30}$ is a very good approximation,
	hence we deduce $s = 17$ and $r = 30$ as candidates.
	And indeed, one can check that $r = 30$ is the desired order.
\end{example}

This won't work all the time (for example, we could get unlucky and
measure $j=0$, i.e.\ $s=0$, which would tell us no information at all).

But one can show that we succeed any time that \[ \gcd(s,r) = 1. \]
This happens at least $\frac{1}{\log r}$ of the time,
and since $r < M$ this means that given sufficiently many trials,
we will eventually extract the correct order $r$.
This is Shor's algorithm.


%&LaTeX

\section{More Feedforward Filters}

This lab discusses the different ways in which to express a discrete
feedforward filter. When completed you should feel comfortable
representing a filter in any of its equivalent forms and know the
different advantages of expressing a filter in these forms.  When you
get down to it, feedforward filters can be expressed in the following
equivalent ways:
\begin{enumerate}
\item Defining equation, $y[n]$ as a (linear) function of $x[n],
  x[n-1], \ldots$, or equivalently as this equation's coefficients.
  % \item Impulse response, $h[n]$.
\item Transfer function, $H(z)$.
\item Frequency response (i.e., the expression for $\mathcal{H}(\hat{\omega})$).
\item Zero locations, $z_i$, often plotted in the complex plane.
\item Amplitude $|\mathcal{H}(\hat{\omega})| = |H(e^{j\hat{\omega}})|$ (and phase
  $\theta(\hat{\omega})$) of the frequency response, often plotted as
  functions of $\hat{\omega}$.
\item Block diagram.
\end{enumerate}

In this lab, you will be given a number of filters, each expressed in
one of the above ways, for each, you are asked to generate the other
representations using pencil and paper. For our purposes here, it will
only be necessary to sketch the amplitude of the filter's frequency
response (for item 5 above).  For items 4 and 5, please use J-DSP to
check your answers. If there is disagreement between your answers and
what you get in J-DSP, please spend some time out of class discussing
the discrepancy with your classmates or with me to determine its
source.

\paragraph{Step 1: Three-point averager} Yes, we've done some of this already,
	so it's a good, familiar place to start. The defining equation is
	$y[n] = \frac{1}{3}x[n] + \frac{1}{3}x[n-1] + \frac{1}{3}x[n-2]$;
	determine the other representations.

\paragraph{Step 2: First-difference filter} A \emph{first-difference} filter is an
	approximation to a discrete derivative operation. Its transfer
	function is $H(z) = 1 - z^{-1}$. Determine the other representations.

\paragraph{Step 3: Second-difference filter} Just as we can compute a discrete
	first derivative with a first-difference filter, we can compute a
	discrete second derivative with a \emph{second difference filter}. Its
	block diagram (in a little trickier format) is:

	\centerline{\includegraphics[width=2.5in]{lab5/second-difference-bdiag}}

	After you've developed the alternative representations, show that
	this filter can also be implemented as a cascade of two
	first-difference filters.

\paragraph{Step 4: Complex conjugate zeros} Feedforward filters with real
	coefficients in their defining equations have either real or complex
	conjugate zeros. Re-represent the filter with zeros $z_{1,2} = 0.5 \pm
	j 0.5$ and $z_3 = 0.75$.

\paragraph{Step 5: Symmetric filters} Feedforward filters with
        symmetric coefficients have interesting properties, such as
        linear phase terms. Even symmetric filters have a midpoint
        around which they mirror their coefficients. Re-represent the
        filter with a coefficients given by $\{1,2,2,1\}$ (i.e.,
        $y[n]=x[n]+2x[n-1]+2x[n-2]+x[n-3]$). In this filter, the
        coefficients \{1,2\} are mirrored.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\chapter{Appendix: Incrementally Calculating Partition Functions on Graphs}
\label{chap:appendix-partition}

\paragraph{Problem Definition.}
Consider a directed graph $G=(V,E)$ endowed with an edge weight
function $w : E \rightarrow \mathbb{R}$.
Between any two vertices $x,y \in V$,
there exist a (potentially infinite) set of paths $P_{xy}$,
\marginnote{Note that we do not restrict consideration
to only simple paths on $G$.}
with the length of any path given by
$\mbox{len} : P_{xy} \rightarrow \mathbb{R}$ as in
(\ref{eqn:lazysp:len-definition}).
Between each such pair of vertices $x,y$,
one can compute the \emph{partition function} denoted $Z_{xy}$,
defined as a sum over all possible paths as follows:
\begin{equation}
   Z_{xy} = \sum_{p_{xy} \in P_{xy}} \exp(- \beta \, \mbox{len}(p_{xy})),
   \label{eqn:def}
\end{equation}
with $\beta$ a fixed real-valued parameter
(commonly called an inverse temperature in statistical mechanics).
We wish to compute and maintain this partition function value
between all pairs of vertices $x$ and $y$ on $G$.

\paragraph{An Incremental Approach.}
We propose an incremental method for calculating $Z$ between all pairs
on a graph.
We will proceed by initializing the values $Z_{xy}$ for each pair
on a graph with no edges,
and then show how we can update these values as edges are added
or removed.

First,
consider the graph with no edges $G_0 = (V, \emptyset)$.
On this graph, there exists no paths between each pair of vertices
$x,y$ with $x \neq y$.
On the other hand,
for $x=y$ there exists exactly one path
containing no edges (and therefore with length $0$).
Thus, we can initialize our values $Z_{xy}$ with:
\begin{equation}
   Z_{xy} = \left\{ \begin{array}{cl}
      1 & \mbox{if } x = y, \\
      0 & \mbox{otherwise}.
   \end{array}\right.
\end{equation}

\paragraph{Adding Edges.}
We can next establish the inductive step.
Suppose we compare two directed graphs $G$ and $G'$,
where $V' = V$ and $E' = E \cup \{ e_{ab} \}$,
and with new edge $e_{ab} : a \rightarrow b$ having weight $w_{ab}$.
For arbitrary vertices $x$ and $y$, we can write $Z'_{xy}$ as follows:
\begin{equation}
   Z'_{xy} = S^{(0)} + S^{(1)} + S^{(2)} + \dots
   \label{eqn:sums}
\end{equation}
where $S^{(k)}$ is the sum from (\ref{eqn:def})
over only the subset of paths that traverse $e_{ab}$ exactly $k$ times.
We note immediately that $S^{(0)} = Z_{xy}$.

Next, we consider the sum $S^{(1)}$,
that is, the sum over all paths which use the new edge $e_{ab}$
exactly once.
We note that each path which contributes to this sum consists of
three segments:
(a) a path segment from $x$ to $a$,
followed by (b) the new edge $e_{ab}$,
followed by (c) a path segment from $b$ to $y$.
Since the sum in question consists exactly of the sum over all unique
paths which follow this template,
we see that we can write $S^{(1)}$ as follows:
\begin{equation}
   S^{(1)} = \sum_{p_{xa}} \sum_{p_{by}}
      \exp\left(
         - \beta \left[ \mbox{len}(p_{xa}) + w_{ab} + \mbox{len}(p_{by}) \right]
         \right)
\end{equation}
Note that both $p_{xa} \in P_{xa}$ and $p_{by} \in P_{by}$
are over all paths on the original graph $G$.
We can then rewrite this simply as:
\begin{equation}
   S^{(1)} = Z_{xa} \, \exp(-\beta \, w_{ab}) \, Z_{by}.
\end{equation}
We can likewise write $S^{(2)}$ as:
\begin{equation}
   S^{(2)} = Z_{xa} \, \exp(-\beta \, w_{ab}) \, Z_{ba} \, \exp(-\beta \,  w_{ab}) \, Z_{by}.
\end{equation}
That is,
this sum consists of paths which arrive from $x$ to $a$ through $G$,
traverse the edge $e_{ab}$ once,
then traverse the original $G$ in some way from $b$ back to $a$,
traverse the edge $e_{ab}$ a second time,
and then proceed from $b$ to $y$.
This decomposition allows us to rewrite
the entire sum from (\ref{eqn:sums}) as:
\begin{equation}
   Z'_{xy} = Z_{xy} + Z_{xa} \, \exp(-\beta \, w_{ab}) \, Z_{by}
      \sum_{k=0}^{\infty} \left[ Z_{ba} \, \exp(-\beta \, w_{ab}) \right]^k.
\end{equation}
As this is a geometric series, it will converge as long as
$Z_{ba} < \exp(\beta \, w_{ab})$.
In this case,
we have:
\begin{equation}
   Z'_{xy} = Z_{xy} + \frac{Z_{xa} \, Z_{by}}{\exp(\beta \, w_{ab}) - Z_{ba}}.
   \label{eqn:partition:addition}
\end{equation}
\marginnote{Note that the addition of an undirected edge between
$a$ and $b$ must entail two successive applications of
(\ref{eqn:partition:addition}).}
This allows us to accommodate arbitrary edge additions while maintaining
the correct values $Z_{xy}$ over all pairs of vertices on $G$.

\paragraph{Removing Edges.}
We can establish a similar result in the case that an existing edge
is removed from the graph.
Consider the case that you have a graph $G'$,
with partition function values $Z'$ between all pairs of vertices.
You then remove some existing edge $e_{ab}$ with weight $w_{ab}$,
yielding new graph $G$.
What can we say about the new values $Z$?

We proceed by applying (\ref{eqn:partition:addition})
to the edge to be removed $e_{ab}$:
\begin{equation}
   Z'_{ba} - Z_{ba} - \frac{Z_{ba}^2}{\exp(\beta w_{ab}) - Z_{ba}} = 0
\end{equation}
\begin{equation}
   \left[\exp(\beta w_{ab}) - Z_{ba} \right] Z'_{ba} - \exp(\beta w_{ab}) Z_{ba} = 0
\end{equation}
\begin{equation}
   \left[ \exp(\beta w_{ab}) - Z_{ba} \right]
   \left[ \exp(\beta w_{ab}) + Z'_{ba} \right]
   = \exp(2 \beta w_{ab})
\end{equation}
\begin{equation}
   \exp(\beta w_{ab}) - Z_{ba}
   = \frac{\exp(2 \beta w_{ab})}{\exp(\beta w_{ab}) + Z'_{ba}}
   \label{eqn:zba-convert}
\end{equation}
We can then use (\ref{eqn:zba-convert}) to express
$Z_{xa}$ and $Z_{by}$ in terms of $Z'_{xa}$ and $Z'_{by}$,
respectively:
\begin{equation}
   Z_{xa} + \frac{Z_{xa} \, Z_{ba}}{\exp(\beta w_{ab}) - Z_{ba}} = Z'_{xa}
   \longrightarrow
   Z_{xa} = \frac{\exp(\beta w_{ab})}{\exp(\beta w_{ab}) + Z'_{ba}} Z'_{xa}
\end{equation}
\begin{equation}
   Z_{by} + \frac{Z_{ba} \, Z_{by}}{\exp(\beta w_{ab}) - Z_{ba}} = Z'_{by}
   \longrightarrow
   Z_{by} = \frac{\exp(\beta w_{ab})}{\exp(\beta w_{ab}) + Z'_{ba}} Z'_{by}
\end{equation}
Lastly, we can combine (\ref{eqn:partition:addition})
and (\ref{eqn:zba-convert}) to express the desired value $Z_{xy}$
only in terms of values on $G'$:
\begin{equation}
   Z_{xy} = Z'_{xy} -
   \frac{
   \left[ \frac{\exp(\beta w_{ab})}{\exp(\beta w_{ab}) + Z'_{ba}} Z'_{xa} \right]
   \,
   \left[ \frac{\exp(\beta w_{ab})}{\exp(\beta w_{ab}) + Z'_{ba}} Z'_{by} \right]
   }{
   \left[ \frac{\exp(2 \beta w_{ab})}{\exp(\beta w_{ab}) + Z'_{ba}} \right]
   }
\end{equation}
\begin{equation}
   Z_{xy} = Z'_{xy} -
   \frac{Z'_{xa} Z'_{by}}{\exp(\beta w_{ab}) + Z'_{ba}}
\end{equation}
This allows us to accommodate edge deletions.


\chapter{Optimizing the Neural Network Model}
{\color{red}LZ: this chapter can be removed}
\section{General Training Procedure}
We have discussed the Gradient-based back propagation procedure for vanilla neural networks. In this chapter we will discuss in detail various optimization techniques. Let's start with summarizing the basic procedure of training a neural network model.

The key elements of training a neural network model can be summarized as follows:
\begin{enumerate}
\item Model construction: choose hyper-parameters, activation function, loss function.
\item Obtain a training data set and a test data set.
\item Adjust the parameters with an optimization methods such as gradient descent to reduce the value of the loss function over the training set. The gradient of the loss function is back propagated through each layer.
\item Apply the model to the test set to check for overfitting or underfitting.
\item Repeat the training and testing step until the model is well-trained.
\end{enumerate}

\section{Other Optimization Methods}

Stochastic Gradient Descent (SGD) is currently the favourite optimization method in deep learning. Its popularity is gained thanks to its ease of implementation and the nonconvexity of the optimization problem.

Despite its popularity and its low cost per step, SGD has well-known
deficiencies that can make it inefficient, or at least tedious to use in
practice. Two main issues are that 1) SGD requires a step size (learning
rate) that has drastic effect on the algorithm's efficiency, is often
difficult to choose well, and virtually never optimal for each individual
descent step; 2) SGD is inherently sequential: it is very difficult to
parallelize them using GPUs or distribute them using computer clusters.

Batch methods, such as Limited memory BFGS (L-BFGS) or Conjugate Gradient
(CG), with the presence of a line search procedure, are usually much more
stable to train and easier to check for convergence. These methods also
enjoy parallelism by computing the gradient on GPUs and/or distributing that
computation across machines. These methods, conventionally considered to be
slow, can be fast thanks to the availability of large amounts of RAMs,
multicore CPUs, GPUs and computer clusters with fast network hardware.

A weakness of batch L-BFGS and CG, which require the computation of the
gradient on the entire dataset to make an update, is that they do not scale
gracefully with the number of examples.

\subsection{Learning with Momentum}
The momentum technique is a compromise that can smooth out the erratic behaviour caused by a fixed learning rate in SGD without slowing down the learning process too much. The idea is to update the weights with the moving average of the individual weight changes corresponding to single training patterns.
If we label everything by the time $t$, then implementing a moving average can be computed as:

\begin{equation}
\label{emom}
\aligned
\Delta \bm W_k(t+1) :=& \bm{W_k}(t+1) - \bm{W_k}(t) = r\frac{\partial L_{total}}{\partial \bm{W_k}}(\bm{W_1}(t),...,\bm{W_d}(t),\bm{\theta_1}(t),..,\bm{\theta_d}(t)) -  \eta\Delta\bm W_k(t)\\
\Delta \bm \theta_k(t+1) :=& \bm{\theta_k}(t+1)-\bm{\theta_k}(t) = r\frac{\partial L_{total}}{\partial \bm{\theta_k}}(\bm{W_1}(t),...,\bm{W_d}(t),\bm{\theta_1}(t),..,\bm{\theta_d}(t)) - \eta\Delta \bm\theta_k(t)
\endaligned
\end{equation}
Here we introduce a new hyper-parameter $\eta$ and a momentum term $\eta\Delta
\bm W_k(t)$ or $\eta\Delta\bm \theta_k(t)$. The momentum simply adds a fraction of the previous update to the current one.  When the gradient keeps pointing in the same direction, this will increase the size of the steps taken towards the minimum. If $\eta=0$, then it is the same as the gradient descent algorithm used before. When using the momentum method, it is usually necessary to reduce the learning rate $r$ accordingly so that the learning step is not too large.

\subsection{Gradient Descent with Line Search}

In neural network model we are seeking parameter values such that the loss function
$$L_{total}(\bm{W_1},...,\bm{W_d},\bm{\theta_1},..,\bm{\theta_d})$$
is minimal. One idea is to consider this as a one-dimensional optimization problem that optimizes one parameter while holding the other parameters fixed. We can do this for each parameter, and repeat the process until all weights are simultaneously at optimal values.

Another idea about training neural network is to find the optimal learning rate with line search. Instead of a fixed learning rate, let's consider it as another variable $r$. Let $\bm{W_1}(t)$,...,$\bm{W_d}(t)$, $\bm{\theta_1}(t)$,..., $\bm{\theta_1}(t)$ be the value of weight matrices and bias vectors at time $t$. By the update rule \eqref{ebpup}, the updated value for these parameters are
\begin{equation}
\aligned
\bm{W_k}(t+1) =& \bm{W_k}(t) - r_k\frac{\partial L_{total}}{\partial \bm{W_k}}(\bm{W_1}(t),...,\bm{W_d}(t),\bm{\theta_1}(t),..,\bm{\theta_d}(t))\\
\bm{\theta_k}(t+1) =& \bm{\theta_k}(t) - r_k'\frac{\partial L_{total}}{\partial \bm{\theta_k}}(\bm{W_1}(t),...,\bm{W_d}(t),\bm{\theta_1}(t),..,\bm{\theta_d}(t))
\endaligned
\end{equation}

The loss function can be treated as a function of one variable
$$
L_{total} = L_{total}(r_k) = L_{total}(\bm{W_k}(t) - r_k\frac{\partial L_{total}}{\partial \bm{W_k}}).
$$
Its derivative can be computed as
\begin{equation}
L_{total}'(r_k) =\frac{\partial L_{total}}{\partial \bm{W_k}}(\bm W_k(t+1))\circ \frac{\partial L_{total}}{\partial \bm{W_k}}(\bm W_k(t)).
\end{equation}

Treat the gradient $\frac{\partial L_{total}}{\partial \bm{W_k}}$ as a vector. If $L_{total}'(r_k) =0$, the direction of the new update $\frac{\partial L_{total}}{\partial \bm{W_k}}(\bm W_k(t+1))$ should be orthogonal to the previous update $\frac{\partial L_{total}}{\partial \bm{W_k}}(\bm W_k(t))$. Together with one-dimensional optimal search such as the Golden section method and the Brent's method, one can find a small interval for $r$ such that the value corresponding to the minimum of the loss function lies in the interval. Note that during this process multiply forward passes are needed, as the target function needs to be evaluated at several different points. Also for each weight matrix and each bias vector, an optimal learning rate needs to be computed separately. If we want to find a global optimal learning rate that applies to all parameters, we will need to consider to following equation:

\begin{equation}
\aligned
&L'(r)\\ =& \sum_{k=1}^d \big(\frac{\partial L}{\partial \bm{W_k}}\frac{\partial \bm{W_k}}{\partial r} + \frac{\partial L}{\partial \bm{\theta_k}}\frac{\partial \bm{\theta_k}}{\partial r}\big)\\
=& -\sum_{k=1}^d\big(\frac{\partial L}{\partial \bm{W_k}}(\bm{W_1}(t+1),...,\bm{W_d}(t+1),\bm{\theta_1}(t+1),..,\bm{\theta_d}(t+1))\frac{\partial L_{total}}{\partial \bm{W_k}}(\bm{W_1}(t),...,\bm{W_d}(t),\bm{\theta_1}(t),..,\bm{\theta_d}(t))\\
 &+ \frac{\partial L}{\partial \bm{\theta_k}}(\bm{W_1}(t+1),...,\bm{W_d}(t+1),\bm{\theta_1}(t+1),..,\bm{\theta_d}(t+1))\frac{\partial L_{total}}{\partial \bm{\theta_k}}(\bm{W_1}(t),...,\bm{W_d}(t),\bm{\theta_1}(t),..,\bm{\theta_d}(t))\big)
\endaligned
\end{equation}

In this case the derivative is a polynomial in $r$ of degree $d$. Solving such equation is going to be more costly. Therefore in general gradient descent with line search is not faster than the basic gradient descent method.

%----------------------------------------------------------------------------
\subsection{Limited Memory BFGS Method}

Besides first-order optimization methods, one can use second-order methods which seek the place where the gradient vanishes. We will start with the classical Newton's method:
\begin{equation}
\bm{z} \leftarrow \bm{z} - \big[HL(\bm{z})\big]^{-1}\nabla L(\bm{z}).
\end{equation}
Here, $\bm{z}$ is a vector representing all parameters of a neural network model, $HL(\bm{z})$ is the Hessian matrix, which is a square matrix of second-order partial derivatives of the function, and $\nabla L(\bm{z})$ is the gradient vector. More precisely, $\bm{z}$, $HL(\bm{z})$, and $\nabla L(\bm{z})$ are defined as follows:
\begin{equation}
\aligned
\bm{w_j} :=& ((\textit{1st column of }\bm{W_j})^T, (\textit{2nd column of }\bm{W_j})^T,..., (\textit{last column of }\bm{W_j})^T), j=1,...,d\\
\bm{z} :=& (\bm{w_1}^T,..., \bm{w_d}^T, \bm{\theta_1}^T,...,\bm{\theta_d}^T)^T = (z_1,...,z_N)\\
h_{ij} =& \frac{\partial^2 L}{\partial z_i \partial z_j}, i,j=1,...,N\\
HL(\bm{z}) =& \big(h_{ij}\big)_{i,j=1}^N\\
\nabla L(\bm{z}) =& (\frac{\partial L}{\partial z_1},...,\frac{\partial L}{\partial z_N})
\endaligned
\end{equation}

Intuitively, the Hessian matrix describes the local curvature of the loss function, which allows us to perform a more efficient update. In particular, multiplying by the inverse Hessian matrix leads the optimization to take more aggressive steps in directions of shallow curvature and shorter steps in directions of steep curvature. However, the update above is impractical if the depth of the neural network model is large, since it requires the inversion of the entire Hessian matrix corresponding to parameters from each layer. In this section we will introduce a \textit{quasi-Newton} method called the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm that seek to approximate the inverse of the Hessian matrix.

The derivation starts with the quadratic model
\begin{equation}
m_k(\bm{p}) = f(\bm{x_k}) + \nabla f(\bm{x_k})\bm{p} + \frac{1}{2}\bm{p}^TB_k\bm{p},
\end{equation}
where $B_k$ is a symmetric positive definite matrix that will be updated in every iteration.

Given this convex quadratic model, we can write down the minimizer $\bm{p_k}$ explicitly as
\begin{equation}
\bm{p_k}=-B_k^{-1}\nabla f(\bm{x_k}).
\end{equation}

The key difference compared with the classical Newton's method is that we are using an approximate Hessian $B_k$ that can be obtained by updating the previous $B_{k-1}$ with only a few arithmetic operations. We impose the following condition on each update: $m_{k+1}(\bm{p})$ must match the gradient of the objective function in $\bm{x_k}$ and $\bm{x_{k+1}}$. Thus we have
\begin{equation}
\aligned
B_k(\bm{p_k}) =& -\nabla f(\bm{x_k})\\
B_{k+1}(\bm{p_{k+1}}) =& -\nabla f(\bm{x_{k+1}})\\
B_{k+1}(\bm{x_{k+1}} - \bm{x_k}) =& \nabla f(\bm{x_{k+1}}) - \nabla f(\bm{x_k})\\
%\nabla m_{k+1}(-\alpha_k\bm{p_k})=&\nabla f(\bm{x_{k+1}}) - \alpha_k B_{k+1}\bm{p_k} = \nabla f(\bm{x_k})
\endaligned
\end{equation}

%We derive that
% \begin{equation}
% \label{ebk}
% \alpha_k B_{k+1} \bm{p_k} = \nabla f(\bm{x_{k+1}}) - \nabla f(\bm{x_k})
% \end{equation}

%We clean up the notations by introducing
\begin{equation}
\aligned
\bm{s_k} :=& \bm{x_{k+1}} - \bm{x_{k}}\\
\bm{y_k} :=& \nabla f(\bm{x_{k+1}}) - \nabla f(\bm{x_{}})\\
\endaligned
\end{equation}

Now the condition on $B_{k+1}$ can be expressed as
\begin{equation}\label{esecant}
\bm{B_{k+1}s_k} = \bm{y_k}.
\end{equation}
There are infinitely many symmetrical matrices that satisfy equation \eqref{esecant}. The BFGS method seeks the one that requires fewest changes upon $\bm B_k$, i.e., the solution to the following problem:
\begin{equation}
\aligned
\min_{\bm{B_{k+1}}}&\|B_{k+1} - B_k\|\\
s.t.&\bm{B_{k+1}s_k} = \bm{y_k}\\
&\bm{B_{k+1}^T} = \bm {B_{k+1}}
\endaligned
\end{equation}
Here we use the \textit{weighted Frobenous norm} {(\color{red}definition needed)}. The optimal solution is given by
\begin{equation}
\label{ebfgs}
\bm{B_{k+1}} = \big(I - \rho_k\bm{y_ks_k^T}\big)\bm{B_k}\big(I - \rho_k\bm{y_ks_k^T}\big)+\rho_k\bm{s_ks_k^T},
\end{equation}
where $\rho_k = (\bm{y_ks_k^T})^{-1}.${\color{red}(proof?)}

{\bf Limited-memory BFGS method.} By equation \eqref{ebfgs}, each $\bm{B_k}$ can be obtained recursively from $(\bm{s_k, y_k})$, $(\bm{s_{k-1}, y_{k-1}})$,...,$(\bm{s_1, y_1})$. This would require the algorithm to maintain a history of all these vectors, which could be a huge burden. The Limited-memory BFGS (L-BFGS) method simply truncates this procedure, using only last $m$ pairs of vectors to construct the matrix $\bm B_k$. This results in a loss of accuracy in approximation, but it will greatly reduce the momery requirement of this algorithm.
% In order to ensure that the solution is a local minimum, we will impose the \textit{strong Wolfe condition} on curvature:
% \begin{equation}
% Wolfe condition.
% \end{equation}

% By pre-multiplying equation \eqref{ebk} by $\bm{s_k^T}$ we have
% \begin{equation}
% 0 < \bm{s_k^TB_{k+1}s_k} = \bm{s_k^T y_k}.
% \end{equation}

{\bf Cost and convergence rate.} The cost of the BFGS method per iteration is dominated by matrix-vector multiplication in equation \eqref{ebfgs} ($O(n^2)$). Other computations include function evaluations and gradient evaluations. The convergence rate is super-linear. Newton's method converges quadratically, but the cost per iteration is higher since it requires the solution of a linear system. In addition, Newton's method requires the calculation of second derivatives whereas the BFGS method does not.

\section{Back-Propagation of Convolutional Neural Networks}

In general, the convolution operation is a mathematical operation on two functions (f and g); it produces a third function, that is typically viewed as a modified version of one of the original function.

\begin{itemize}
\item
Convolution
$(f*g)(x) := \int_{-\infty}^{\infty}f(t)g(x-t)dt$
\item
Discrete convolution
$(f*g)[n] := \sum_{-\infty}^\infty f[m]g[n-m]$
\item
2-Dimensional convolution
$(F*G)[i,j] := \sum_{s = -\infty}^\infty\sum_{t = -\infty}^\infty F[s, t]G[i-s, j-t]$
\end{itemize}

In neural network applications, the input is a multidimensional array (referred to as tensors), and so are the parameters adapted by the algorithm. In CNN terminology, the first function $f(x)$ is referred to as the input, the second function $g(x)$ is referred to as the kernel (or filter), and the output function $(f*g)(x)$ is referred to as the feature map. If we use 2D images $M\in\mathbb{R}^{n\times n}$ as input and use a 2D filter $F\in\mathbb{R}^{d\times d}$ we have
$$(M*F)[i,j] := \sum_{s = 0}^{d-1}\sum_{t=0}^{d-1} F[s, t]G[i+s, j+t], i,j,=0,...,n-d-1$$. Here the output matrix $(M*F)$ is an $(n-d)\times(n-d)$ matrix, and the kernel is flipped comparing to the previous definition. Convolution in CNN is illustrated as in Figure \ref{fconv}
\begin{figure}[ht]
\label{fconv}
\centering
\includegraphics[scale=0.4]{Convolution1.png}
\caption{Convolution in CNN}
\end{figure}
{\bf Convolutional Neural Network (CNN)} is the most widely-used neural network structure for tasks on image pattern recognition. A CNN mainly comprises three types of layers: fully-connected (FC) layers, convolutional layers, and pooling layers
\begin{figure}[ht]
\centering
\includegraphics[scale=0.7]{CNN.png}
\caption{Convolutional Neural Network}
\end{figure}

{\bf Structure of CNN}
A convolution layer in CNN is a composition of three functions: $f^{conv}$, $f^{sig}$, and $f^{pool}$

{\bf Convolutional Layer} consists of a set of filters/feature maps $F_{ij} = \big(f_{ijkl}\big)_{k,l}$ (matrices of size $d\times d$ such as $d=3,5$). During the forward pass, each filter will conduct a 2D convolution with the input matrices $X_i=\big(x_{ikl}\big)_{kl}$ across its width and height. In this way local-features can be extracted and fed into subsequent layers to get higher-order features. We use 3-dimensional tensor $\mathcal{X}=\big(x_{ikl}\big)_{i,k,l}$ to represent the set of input matrices, and 4-dimensional tensor $\mathcal{F}=\big(f_{ijkl}\big)_{i,j,k,l}$ to denote the set of filters in a convolutional layer. We have that
%At a convolution layer, the previous layer’s feature maps are convolved with learnable kernels and put through the activation function to form the output feature map. Each output map may combine convolutions with multiple input maps. In general, we have that
\begin{equation}
\aligned
f^{conv}(\mathcal{X}, \mathcal{F}) =& \big(y_{jst}\big)_{j,s,t}=:\mathcal{Y}\\
y_{jst} =& \sum_i\sum_{k,l} x_{i,s+k, t+l}f_{ijkl}\\
f^{sig}(\mathcal{Y}) =& \big(\sigma(y_{jst})\big)_{j,s,t}=:\mathcal{Z}
\endaligned
\end{equation}

{\bf Pooling layer}. Pooling layer subsamples statistics to obtain summary statistics that are invariant to shifts and distortions in some degree. Commonly used pooling functions are
\begin{enumerate}
\item
Mean pooling: $f^{pool}(\mathcal{Z})=\big(\frac{\sum_{k=1}^m z_k}{m}\big)$
\item
Max pooling: $f^{pool}(\mathcal{Z})=\big(\max_{1\le k\le m}\{z_k\}\big)$
\item
$l^p$ pooling: $f^{pool}(\mathcal{Z})=\big(\|(z_1,...,z_m)\|_p\big)$
\end{enumerate}

\begin{figure}[ht]\label{fcnnbp}
\includegraphics[scale=0.5]{CNNBP}
\caption{Convolutional Layer}
\end{figure}

Figure \ref{ffea} illustrates features extracted the first three convolutional Layers  on a facial recognition task.

\begin{figure}[ht]
\label{ffea}
\centering
\includegraphics[scale=1]{Features.png}
\caption{Features}
\end{figure}

{\bf Backpropagation in CNN}.
To back-propagate through one convolutional layer, one need to compute the derivative of three functions:
\begin{equation}
\aligned
\bm\delta^{conv}_{ikl} =& \frac{\partial L}{\partial x_{ikl}} =
\sum_{j,k,l}\frac{\partial L}{\partial y_{jkl}}\frac{\partial y_{jkl}}{\partial x_{ikl}} = \sum_{k,l=0}^{r-1}\bm\delta^{sig}_{ikl}\\
\bm\delta^{sig}_{jkl} =& \frac{\partial L}{\partial y_{jkl}} = \sigma'(y_{jkl})\\
\bm\delta^{pool}_{jkl} =& \frac{\partial L}{\partial z_{jkl}} = \frac{\partial L}{\partial x^{next}_{jkl}}\frac{\partial x^{next}_{jkl}}{\partial z_{jkl}} = \bm\delta^{next, conv}_{jkl}\\
\frac{\partial L}{\partial f_{ijkl}} =& \sum_{j,k,l}\frac{\partial L}{\partial y_{jkl}}\frac{\partial y_{jkl}}{\partial f_{ijkl}}=\sum_{s,t=0}^{d-1}\bm\delta^{sig}_{jst}x_{i, s+k, t+l}
\endaligned
\end{equation}
The partial derivative of the pooling function is usually called the \textit{upsample} function. For example. for the max pooling function, the corresponding upscale function is $\mathbbm{1}_{z_{jkl} = \max\{z_k\}}$ (distributing the gradient to the entry with maximum value). Also note that the partial derivative of the filer parameters are computed through a discrete convolution.

\section{Tensor Computation and Decomposition}

{\bf Tensor}
\begin{itemize}
\item
Definition of tensor: an element $\mathcal{F}\in\mathbb{R}^{I_1\times\cdots\times I_N}$.
\item
Element-wise:
$$
\mathcal{F}[i_1,...,i_N]\in\mathbb{R}, i_j=1,...,I_j, j=1,...,N.
$$
\item
Fiber: mode-1 tensor (vector) obtained by fixing every index but one.
\item
Slice: mode-2 tensor (matrix) obtained by fixing every index but two.
\item
Norm:
$$
\|\mathcal{F}\| := \sqrt{\sum_{i_1=1}^{I_1}\cdots\sum_{i_N=1}^{I_N} \mathcal{F}[i_1,...,i_N]^2}
$$
\end{itemize}
\begin{figure}[ht]
\centering
\includegraphics[scale=0.5]{fiber.png}
\end{figure}
\begin{figure}[ht]
\centering
\includegraphics[scale=0.5]{slice.png}
\end{figure}

{\bf Metricization}
The mode-$n$ metricization of a tensor $\mathcal{X}\in\mathbb{R}^{I_1\times\cdots\times I_N}$ is denoted by $X_{(n)}$ and arranges the mode-$n$ fibers to be the columns of the resulting matrix.
\begin{figure}[ht]\label{fmat}
\centering
\includegraphics[scale=0.6]{metricization.png}
\caption{Tensor Matricization}
\end{figure}
{\color{red} Definition of Matricization}. An example is given in Figure \ref{fmat}.

{\bf CP Decomposition}
The CANDECOMP/PARAFAC (CP) decomposition attempts to decompose a tensor as a sum of rank-1 tensors:

\begin{equation}
\mathcal{X}[i_1,...,i_N] = \sum_{k=1}^r a^{(1)}_{i_1k}a^{(2)}_{i_2k}\cdots a^{(N)}_{i_Nk}
\end{equation}
The number of summation is call the CP-rank of tensor $\mathcal{X}$.
\begin{figure}[ht]
\centering
\includegraphics[scale=0.6]{CP_decomposition.png}
\end{figure}

{\bf Challenges about CP Decomposition}
\begin{itemize}
\item
The problem of finding the CP-rank of a tensor is NP-hard.
\item
For a general 3-mode tensor $\mathcal{X}\in\mathbb{R}^{I\times J\times K}$, its maximal attainable rank is bounded by
$$
rank(\mathcal{X})\le \min\{IJ, JK, KI\}
$$
\item
Unlike matrices, the low-rank approximation problem for tensors is not well-defined: \textit{any rank-3 tensor can be infinitely approximated by a rank-2 tensor}.
\end{itemize}

{\bf Tucker Decomposition}

The \textit{Tucker-decomposition} is a form of higher-order PCA. It decomposes a tensor into a core tensor multiplied by a matrix along each mode.
\begin{figure}[ht]
\centering
\includegraphics[scale=0.6]{Tucker.png}
\end{figure}
\begin{equation}
\mathcal{X}[i,j,k] = \sum_{p=1}^P\sum_{q=1}^Q\sum_{r=1}^R
g_{pqr}a_{ip}b_{jq}c_{kr}, i=1,...,I, j=1,...,J, k=1,...,K.
\end{equation}

{\bf Tensor-Matrix Multiplication}
Tucker decomposition is best illustrated through \textit{tensor-matrix multiplication}. A product of a tensor $\mathcal{X}$ and a matrix $\bm M=\big(m_{ij}\big)_{i,j}$ on the $k$-th mode is a tensor defined as:
\begin{equation}
    \big(\mathcal{X}\times_kM\big)[i_1,...,i_n] =
    \sum_j \mathcal{X}[i_1,...,i_{k-1},j,i_{k+1},...,i_n]m_{ji_k}.
\end{equation}
$M\times_k\mathcal{X}$ can be defined similarly as
\begin{equation}
    \big(M\times_k\mathcal{X}\big)[i_1,...,i_n] =
    \sum_j m_{i_kj}\mathcal{X}[i_1,...,i_{k-1},j,i_{k+1},...,i_n]m_{i_kj}.
\end{equation}
The following properties are easily provable:
\begin{lemma}
[Tensor-Matrix Multiplication]
\begin{equation}
\aligned
&(\mathcal{X}\times_k M)\times_l N = (\mathcal{X}\times_l N)\times_k M =: \mathcal{X}\times_k M\times_l N\\
&(\mathcal{X}\times_k M)\times_k N = \mathcal{X}\times_k(M\cdot N)
\endaligned
\end{equation}
\end{lemma}
{\color{red}proof?}

With the notation above, the Tucker decomposition can be written as
\begin{equation}
\label{etucker}
\mathcal{X} = \bm P\times_1 \bm Q\times_2 \bm R\times_3 \mathcal{G}
\end{equation}
{Tucker Rank}
Tucker rank (P, Q, R) is defined through the matricization:
\begin{align*}
P = rank(\mathcal{X}_{(1)})\\
Q = rank(\mathcal{X}_{(2)})\\
R = rank(\mathcal{X}_{(3)})\\
\end{align*}

As a result, matrix SVD and low-rank approximation can be extended to the computation of Tucker ranks (HOSVD).

{\bf Tensor-Train Decomposition}
The need for storing the $I_1\times\cdots\times I_n$ core tensor makes the Tucker decomposition increasingly unattractive as $N$ gets larger. The following definition of Tensor Train decomposition solves the problem, while preserving the possibility of SVD-based compression and low-rank approximation:
\begin{definition}
A tensor $\mathcal{X}\in\mathbb{R}^{I_1\times\cdots\times I_n}$ is said to have \textit{tensor train rank (TT)} $(r_1, r_2,...,r_{N-1}, r_N)$ with $r_1 = r_N = 1$ if there exists matrices $G_\mu(i_\mu)$ for $\mu=2,...,N$ and $i_\mu=1,2,...,I_\mu$ such that
\begin{equation}
\mathcal{X}[i_1,...,i_N] = G_2(i_2) G_3(i_3)\cdots G_N(i_N), G_\mu(i_\mu)\in\mathbb{R}^{r_{\mu-1}\times r_\mu}
\end{equation}
\end{definition}