\chapter{Vectors and Matrices \label{chapter:linear-algebra.tex}}
Certain applications, for example data mining and machine learning, require an efficient support of both
vectors and matrices.
Although vectors and matrices could easily be implemented in \setlx, this would not be very
efficient because of the overhead of the interpreter loop.  Therefore, \setlx\ supports both vectors and
matrices natively.  This support is based on the \textsl{Java} library
\href{http://math.nist.gov/javanumerics/jama/}{\textsl{Jama}}.  This library is the result of a joint effort of
\href{http://www.mathworks.com/}{Mathworks} and the \href{https://www.nist.gov}{National Institute
  of Standards and Technology}.  \textsl{Jama} has been integrated into
\setlx\ by Patrick Robinson.  This library provides the basic means for computations involving
matrices and vectors.  In the following exposition we assume that the reader has some familiarity with
\href{http://en.wikipedia.org/wiki/Linear_algebra}{linear algebra}.

\section{Vectors}
\setlx\ supports real valued vectors of arbitrary dimensions.  Conceptually, a vector can be viewed as
a list of floating point numbers.  A vector is constructed from a list of numbers via the
function \texttt{la\_vector} as follows:
\\[0.2cm]
\hspace*{1.3cm}
\texttt{v := la\_vector([1/2, 1/4, 1/5]);}
\\[0.2cm]
When executed, this statement yields the result
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<<0.5 0.25 0.2>>}.
\\[0.2cm]
This result shows that the fractions in the argument list have been converted to floating point
numbers.  Conceptually, the vector \texttt{v} is a column vector.  Therefore, mathematically
\texttt{v} would be written as
\\[0.2cm]
\hspace*{1.3cm}
$
\left(\begin{array}{l}
  0.5  \\
  0.25 \\
  0.2
\end{array}\right)
$.
\\[0.2cm]
Instead of using the function \texttt{la\_vector,} we could also have used the command
\\[0.2cm]
\hspace*{1.3cm}
\texttt{v := <<1/2 1/4 1/5>>;}
\\[0.2cm]
to define the vector \texttt{v}.  This shows that the operators ``\texttt{<<}'' and ``\texttt{>>}''
can be used to define a vector.  \textbf{Note} that the components of a vector are \textbf{not} separated by the
character ``\texttt{,}'' but \textbf{rather} are separated by white space.  Furthermore, at the
moment the syntax that uses the operators ``\texttt{<<}'' and ``\texttt{>>}'' only works with
numbers, it does not work with arbitrary expressions!

\setlx\ support the basic arithmetic operations that are defined for vectors.  If \texttt{a} and
\texttt{b} are two vectors, then
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a + b}
\\[0.2cm]
computes the sum of \texttt{a} and \texttt{b}, while
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a - b}
\\[0.2cm]
computes their difference.  Concretely, if we have
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a = <<a$_1$ $\cdots$ a$_n$>>} \quad and \quad
\texttt{b = <<b$_1$ $\cdots$ b$_n$>>},
\\[0.2cm]
then ``\texttt{a+b}'' and ``\texttt{a-b}'' are defined componentwise:
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a + b := <<(a$_1$+b$_1$) $\cdots$ (a$_n$+b$_n$)>>} \quad and \quad
\texttt{a - b := <<(a$_1$-b$_1$) $\cdots$ (a$_n$-b$_n$)>>}.
\\[0.2cm]
For example, if we define
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a := <<1 2 3>>; \ b := <<4 5 6>>;}
\\[0.2cm]
then the expressions ``\texttt{a + b}'' and ``\texttt{a - b}'' yield the results
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<<5.0 7.0 9.0>>} \quad and \quad \texttt{<<-3.0 -3.0 -3.0>>},
\\[0.2cm]
respectively.  Additionally, the shortcut assignment operators ``\texttt{+=}'' and ``\texttt{-=}''
are available for vectors.  They work as expected.  Note that for the expressions ``\texttt{a + b}''
and ``\texttt{a - b}'' to be defined, \texttt{a} and \texttt{b} have to have
\textbf{the same number of elements}.

Vectors support
\href{http://en.wikipedia.org/wiki/Scalar_multiplication}{\emph{scalar multiplication}}.  If
$\alpha$ is a real number and
\\[0.2cm]
\hspace*{1.3cm}
\texttt{v = <<v$_1$ $\cdots$ a$_n$>>}
\\[0.2cm]
is an $n$-dimensional number, then the scalar products \texttt{$\alpha$ * v} and \texttt{v * $\alpha$}
are both defined as the vector
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<<($\alpha$ * v$_1$) $\cdots$ ($\alpha$ * v$_n$)>>}.
\\[0.2cm]
For example, if \texttt{a} is defined as above, then the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{1/2 * a}
\\[0.2cm]
yields the vector
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<<0.5 1.0 1.5>>}.
\\[0.2cm]
It does not matter whether we multiply the scalar from the left or from the right, so the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a * (1/2)}
\\[0.2cm]
yields the same result.  We could also have written the previous expression as
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a / 2}.
\\[0.2cm]
In general, if \texttt{x} is a vector and $\alpha$ is a real number, then the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{x / $\alpha$} \quad is equivalent to \quad
\texttt{x * (1/$\alpha$)},
\\[0.2cm]
However, an expression of the form
\\[0.2cm]
\hspace*{1.3cm}
\texttt{$\alpha$ / x}
\\[0.2cm]
is not defined if \texttt{x} is a vector.

 If \texttt{v} is a vector and \texttt{n} is a
number, then the assignment statement
\\[0.2cm]
\hspace*{1.3cm}
\texttt{v *= n;}
\\[0.2cm]
is equivalent to the statement
\\[0.2cm]
\hspace*{1.3cm}
\texttt{v = v * n;}
\\[0.2cm]
For two $n$-dimensional vectors
\\[0.2cm]
\hspace*{1.3cm}
$\vec{x} = \left(
  \begin{array}[c]{c}
  x_1 \\
  \vdots \\
  x_n
  \end{array}
\right)
$
\quad and \quad
$\vec{y} = \left(
  \begin{array}[c]{c}
  y_1 \\
  \vdots \\
  y_n
  \end{array}
\right),
$
\\[0.2cm]
the \href{http://en.wikipedia.org/wiki/Dot_product}{\emph{dot product}}, which is also known as the
\emph{scalar product}, is defined as
\\[0.2cm]
\hspace*{1.3cm}
$\vec{x} \cdot \vec{y} := \sum\limits_{i=1}^n x_i \cdot y_i$.
\\[0.2cm]
Students often confuse scalar multiplication and the scalar product.
Therefore, we have decided to use the same operator for both products:
If \texttt{a} and \texttt{b} are two vectors of the same dimension, the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a * b}
\\[0.2cm]
yields their scalar product.  For example,
using the definition of \texttt{a} and \texttt{b} given above, the expression
``\texttt{a * b}''
yields the result $32$ since
\\[0.2cm]
\hspace*{1.3cm}
$1 \cdot 4 + 2 \cdot 5 + 3 \cdot 6 = 32$.
\\[0.2cm]
Finally, \setlx\ supports the
\href{http://en.wikipedia.org/wiki/Cross_product}{\emph{cross product}}.  For two three-dimensional
vectors
\\[0.2cm]
\hspace*{1.3cm}
$\vec{x} = \left(
  \begin{array}[c]{c}
  x_1 \\
  x_2 \\
  x_3
  \end{array}
\right)
$
\quad and \quad
$\vec{y} = \left(
  \begin{array}[c]{c}
  y_1 \\
  y_2 \\
  y_3
  \end{array}
\right),
$
\\[0.2cm]
the cross product $\vec{x} \times \vec{y}$ is defined as
\\[0.2cm]
\hspace*{1.3cm}
$\vec{x} = \left(
  \begin{array}[c]{c}
  x_1 \\
  x_2 \\
  x_3
  \end{array}
\right) \times \left(
  \begin{array}[c]{c}
  y_1 \\
  y_2 \\
  y_3
  \end{array}
\right) = \left(
  \begin{array}[c]{c}
  x_2 \cdot y_3 - x_3 \cdot y_2 \\
  x_3 \cdot y_1 - x_1 \cdot y_3 \\
  x_1 \cdot y_2 - x_2 \cdot y_1
  \end{array}
\right)
$.
\\[0.2cm]
If \texttt{a} and
\texttt{b} are both 3-dimensional vectors, then the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a >< b}
\\[0.2cm]
computes the cross product of \texttt{a} and \texttt{b}.  For example, if \texttt{a} and \texttt{b}
are the vectors defined at the beginning of this section, then the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{a >< b}
\\[0.2cm]
yields the result
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<<-3.0  6.0  -3.0>>}
\\[0.2cm]
since we have
\\[0.2cm]
\hspace*{1.3cm}
$\left(
 \begin{array}[c]{l}
   1 \\ 2 \\ 3
 \end{array}
 \right)
   \times
 \left(
 \begin{array}[c]{l}
    4 \\ 5 \\ 6
 \end{array}
 \right)
    =
 \left(
  \begin{array}[c]{l}
    2 \cdot 6 - 3 \cdot 5 \\ 3 \cdot 4 - 6 \cdot 1 \\ 1 \cdot 5 - 2 \cdot 4
  \end{array}
 \right)
    =
 \left(
  \begin{array}[c]{r}
    -3 \\ 6 \\ -3
  \end{array}
 \right)
$.
\\[0.2cm]
Note that the cross product of two vectors is only defined iff both \texttt{a} and \texttt{b} are
three-dimensional.  There are generalisations of the cross product for $n$-dimensional vectors.
However, in that case the cross product is no longer a binary operator but rather takes $n-1$ arguments.
These generalisations are not supported in \setlx.

Vectors provide the same access operations as list.  Therefore, to extract the $i$-th component of a
vector \texttt{v} we can use the expression ``\texttt{v[$i$]}''.  Furthermore, the operator
``\texttt{\#}'' returns the dimension of a given vector.  Therefore, given the vector \texttt{a}
defined above, the expression \texttt{\#a} returns the value 3.

\section{Matrices}
\setlx\ supports real valued matrices.  The function \texttt{la\_matrix} can be used to construct a
matrix from a list of list of numbers.  For example, the assignment
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A := la\_matrix([[1,2],[3,4]]);}
\\[0.2cm]
constructs a matrix that can be written as
\\[0.2cm]
\hspace*{1.3cm}
$
\left(
\begin{array}[c]{ll}
  1 & 2 \\
  3 & 4
\end{array}
\right)
$.
\\[0.2cm]
We can see that the inner lists used as an argument to the function \texttt{la\_matrix} correspond
to the rows of the resulting matrix.   In \setlx, the matrix is printed as
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<< <<1.0 2.0>> <<3.0 4.0>> >>}.
\\[0.2cm]
Note that a matrix is written as a list of vectors where instead of the square brackets
``\texttt{[}'' and ``\texttt{]}''  the tokens ``\texttt{<<}'' and
``\texttt{>>}'' are used as opening and closing delimiters.  However, whereas the elements of a list
are separated by commas, the row vectors making up a matrix are separated by white space.
Instead of using the function \texttt{la\_matrix} we could have defined the matrix \texttt{A} using
the following command:
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A := << <<1.0 2.0>> <<3.0 4.0>> >>;}
\\[0.2cm]
The function \texttt{la\_matrix} can also be called with a single vector as its argument.  If $v$ is an
$n$-dimensional vector, then
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_matrix($v$)}
\\[0.2cm]
interprets this vector as a column vector and turns it into an $n \times 1$ matrix.  For example,
the statement
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A := la\_matrix(<<1 2 3>>);}
\\[0.2cm]
yields the result
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<< <<1.0>> <<2.0>> <<3.0>> >>}.
\\[0.2cm]
This result shows that \texttt{A} is a matrix that consists of three rows that are themselves
vectors of length 1.

Similar to vectors, matrices can be added and subtracted using the operators ``\texttt{+}'' and
``\texttt{-}''.   If we define
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A := << <<1 2>> <<3 4>> >>;} \quad and \quad \texttt{B := << <<5 6>> <<7 8>> >>;}
\\[0.2cm]
then the expressions ``\texttt{A + B}'' and ``\texttt{A - B}'' work componentwise and yield the
results
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<< <<6.0 8.0>> <<10.0 12.0>> >>} \quad and \quad
\texttt{<< <<-4.0 -4.0>> <<-4.0 -4.0>> >>}.
\\[0.2cm]
In addition to ``\texttt{+}'' and ``\texttt{-}'', the assignment operators ``\texttt{+=}'' and
``\texttt{-=}'' are supported for matrices and work as expected.

Furthermore, matrices support scalar multiplication in the same way as vectors.  For example, if the
matrix \texttt{A} is defined as above, then the expressions
``\texttt{2 * A}'' and ``\texttt{A * 2}'' both yield the result
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<< <<2.0 4.0>> <<6.0 8.0>> >>}.
\vspace*{0.2cm}

Next,
\setlx\ supports \href{http://en.wikipedia.org/wiki/Matrix_multiplication}{matrix multiplication}.
If
\\[0.2cm]
\hspace*{1.3cm}
 $\texttt{A} = (\mathtt{a}_{i,j})_{1 \leq i \leq m \atop 1 \leq j \leq n}$ \quad is an $m \times n$ matrix
\\
and
\\
\hspace*{1.3cm}
$\texttt{B} = (\mathtt{b}_{j,l})_{1 \leq j \leq n \atop 1 \leq l \leq k}$ \quad is an $n \times k$ matrix,
\\[0.2cm]
then the matrix product \texttt{A * B} is defined as the matrix
\\[0.2cm]
\hspace*{1.3cm}
$\Bigl(\sum\limits_{j=1}^n \texttt{a}_{i,j}\cdot \texttt{b}_{j,l}\Bigr)_{1 \leq i \leq n \atop 1 \leq l \leq k}$.
\\[0.2cm]
For example, given the definitions of
\texttt{A} and \texttt{B} shown above,  the expression ``\texttt{A * B}'' yields the result
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<< <<19.0 22.0>> <<43.0 50.0>> >>}.
\\[0.2cm]
Furthermore, if \texttt{A} is an $m \times n$ matrix and \texttt{v} is an $n$ dimensional vector,
then the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A * v}
\\[0.2cm]
is computed as a matrix multiplication, where $v$ is interpreted as an $n \times 1$ matrix.  In
this case, the resulting $m \times 1$ matrix is then automatically converted back into an $m$ dimensional vector.
For example, if we define the matrix \texttt{A} as
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A := << <<1 2>> <<3 4>> >>;}
\\[0.2cm]
and the vector \texttt{x} is defined as
\\[0.2cm]
\hspace*{1.3cm}
\texttt{x := <<5 6>>;}
\\[0.2cm]
then evaluating the product \texttt{A * x} yields the result
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<<17.0 39.0>>}
\\[0.2cm]
since we have
\\[0.2cm]
\hspace*{1.3cm}
$1 \cdot 5 + 2 \cdot 6 = 17$ \quad and \quad $3 \cdot 5 + 4 \cdot 6 = 39$.
\\[0.2cm]
In addition to matrix multiplication, \setlx\ also support exponentiation of a square matrix by an
integer number.  For example,
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A ** 2;}
\\[0.2cm]
returns the square  of \texttt{A}, while
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A ** -1}
\\[0.2cm]
return the \href{http://en.wikipedia.org/wiki/Invertible_matrix}{\emph{inverse}} of a matrix,
provided the matrix is not \emph{singular}.  If the matrix \texttt{A} is singular, evaluation of the
expression \texttt{A ** $n$} raises an exception if the exponent $n$ is negative.

Matrices can be \href{http://en.wikipedia.org/wiki/Transpose}{\emph{transposed}} via the postfix
operator ``\texttt{!}''.  For example, using the definition of the matrix \texttt{A} shown above,
the expression ``\texttt{A!}'' yields
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<< <<1.0 3.0>> <<2.0 4.0>> >>}.
\\[0.2cm]
If \texttt{A} is an $m \times n$ matrix, the expression ``\texttt{\#A}'' yields the dimension $m$.
In order to compute the dimension $n$, we can use the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{\#A[1]}.
\\[0.2cm]
The reason is that \texttt{A[1]} returns the first row of the matrix \texttt{A} as a list and the
length of this list is the dimension $n$.  In order to access the element in row $i$ and column $j$
of matrix \texttt{A} we can use the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A[$i$][$j$]}.

\section{Numerical Methods for Matrices and Vectors}
In this section we will discuss the numerical methods that are provided.  These methods are inherited
from \href{http://math.nist.gov/javanumerics/jama/}{\textsl{Jama}}, which is a \textsl{Java} matrix
package.  The names of all methods inherited from \textsl{Jama} start with ``\texttt{la\_}''.  This
is short for \underline{l}inear \underline{a}lgebra.

\subsection{Computing the Determinant}
If \texttt{A} is a square matrix, the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_det(A)}
\\[0.2cm]
computes the \href{http://en.wikipedia.org/wiki/Determinant}{\emph{determinant}} of \texttt{a}.
For example, if we define
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A := << <<1 2>> <<3 4>> >>;}
\\[0.2cm]
then the expression ``\texttt{la\_det(A)}'' yields the result $-2$.  The determinant can be used to
check whether a matrix is invertible as a matrix is invertible if and only if the determinant
is different from $0$.  However, note that due to rounding errors the result of the expression
``\texttt{la\_det(A)}''  might be a small non-zero number even if the matrix \texttt{A} really is
singular.

\subsection{Solving a System of Linear Equations}
A system of linear equations of the form
\\[0.2cm]
\hspace*{1.3cm}
$\texttt{A} \cdot \texttt{x} = \texttt{b}$
\\[0.2cm]
where \texttt{A} is a square matrix $n \times n$  matrix and  \texttt{b} is an $n$-dimensional
vector can be solved using the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_solve(A, b)}.
\\[0.2cm]
For example, to solve the system of equations
\\[0.2cm]
\hspace*{1.3cm}
$
\begin{array}[c]{lcl}
  1 \cdot x + 2 \cdot y & = & 5 \\[0.1cm]
  3 \cdot x + 4 \cdot y & = & 6
\end{array}
$
\\[0.2cm]
we define
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A := << <<1 2>> <<3 4>> >>;} \quad and \quad \texttt{b := <<5 6>>;}
\\[0.2cm]
Then, the solution is computed via the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_solve(A, b)}.
\\[0.2cm]
Note that this expression throws an exception if the given system of equations is not solvable.

Note that it is also posible to solve a system of equations that is
\href{http://en.wikipedia.org/wiki/Overdetermined_system}{\emph{overdetermined}}.  For example,
assume that the system of equations is given as
\\[0.2cm]
\hspace*{1.3cm}
$
\begin{array}[c]{lcl}
  1 \cdot x + 2 \cdot y & = & 1 \\[0.1cm]
  2 \cdot x + 3 \cdot y & = & 2 \\[0.1cm]
  3 \cdot x + 4 \cdot y & = & 3
\end{array}
$
\\[0.2cm]
In order to compute the solution $\vec{x}$ that minimizes the error
 $\|a \cdot \vec{x} - \vec{b}\|_2$ we can
define \texttt{A} and \texttt{b} via
\\[0.2cm]
\hspace*{1.3cm}
\texttt{A := << <<1 2>> <<2 3>> <<3 4>> >>;} \quad and \quad \texttt{b := <<1 2 3>>;}
\\[0.2cm]
and then call  the function \texttt{la\_solve} as
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_solve(A, b);}
\\[0.2cm]
In this case, \texttt{la\_solve} computes the result
\\[0.2cm]
\hspace*{1.3cm}
\texttt{<<0.9999999999999982 1.4430967688835178E-15>>}
\\[0.2cm]
which is pretty close to the exact result $\left(\begin{array}[c]{l} 1 \\ 0 \end{array}\right)$.
\vspace*{0.3cm}

\noindent
When solving a system of linear equations, care has to be taken that the solution is well defined.
To this end, \setlx\ provides the function \texttt{la\_cond} that computes the
\href{http://en.wikipedia.org/wiki/Condition_number}{condition number} of a given matrix.
Let us describe the problem via an example borrowed from
\\[0.2cm]
\hspace*{1.3cm}
\href{http://nm.mathforcollege.com/mws/gen/04sle/mws_gen_sle_spe_adequacy.pdf}{\texttt{http://nm.mathforcollege.com/mws/gen/04sle/mws\_gen\_sle\_spe\_adequacy.pdf}}.
\\[0.2cm]
Assume the matrix $A$ and the vector $\vec{b}$ are defined as
\\[0.2cm]
\hspace*{1.3cm}
$A := \left(
  \begin{array}[c]{ll}
    1.000 &  2.000 \\
    2.000 &  3.999
  \end{array}
  \right)
$  \quad and \quad
$\vec{b} := \left(
  \begin{array}[c]{l}
    4.000 \\ 7.999
  \end{array}
  \right).
$
\\[0.2cm]
It can easily be verified that the solution $\vec{x}$ to the system of equations
\\[0.2cm]
\hspace*{1.3cm}
$\vec{b} = A \cdot \vec{x}$
\\[0.2cm]
is given as
\\[0.2cm]
\hspace*{1.3cm}
$\vec{x}  =
 \left(
   \begin{array}[c]{l}
     2 \\ 1
   \end{array}
  \right)
$.
\\[0.2cm]
However, let us assume that the vector $\vec{b}$ is distorted into the vector $\vec{c}$ that is
given as
\\[0.2cm]
\hspace*{1.3cm}
$\vec{c} := \left(
  \begin{array}[c]{l}
    4.001 \\ 7.998
  \end{array}
  \right).
$
\\[0.2cm]
If we solve the system
\\[0.2cm]
\hspace*{1.3cm}
$\vec{c} = A \cdot \vec{x}$,
\\[0.2cm]
the solution for $\vec{x}$ becomes
\\[0.2cm]
\hspace*{1.3cm}
$\vec{x}  =
 \left(
   \begin{array}[c]{r}
     -3.999 \\ 4.000
   \end{array}
  \right)
$.
\\[0.2cm]
We see that a tiny change in the right hand side of the system of equations has caused a big change
in the solution $\vec{x}$.  The reason is that the matrix \texttt{A} is \emph{ill-conditioned}.
We can verify this using the function \texttt{la\_cond}:  The expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_cond(<< <<1.000 2.000>> <<2.000 3.999>> >>);}
\\[0.2cm]
yields the result
\\[0.2cm]
\hspace*{1.3cm}
\texttt{24992.000959995028}.
\\[0.2cm]
This shows that small errors in the right hand side of a system of linear equations involving the
matrix \texttt{A} are multiplied by a factor of nearly $25\,000$.  Our recommendation is that if you
ever have to solve a system of equations, you should first check whether the system is
well-conditioned by computing the condition number of the matrix associated with the system.  This
condition number tells us by how much an error in the right hand side of the system of equations is
magnified when we compute the solution.  In particular, if the condition number is so big that it
will magnify known uncertainties in  the right hand side of the equations to an extend that the
result is meaningless, then it is not possible to solve the given system of equations in a meaningful
way.

\subsection{The Singular Value Decomposition and the Pseudo-Inverse}
The function \texttt{la\_svd} can be used to compute the
\href{http://en.wikipedia.org/wiki/Singular_value_decomposition}{\emph{singular value decomposition}}
of a given matrix \texttt{A}.  For a given $m \times n$ matrix \texttt{A}, the expression
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_svd(A)}
\\[0.2cm]
returns a list of the form
\\[0.2cm]
\hspace*{1.3cm}
\texttt{[U, S, V]}
\\[0.2cm]
where \texttt{U} is an
\href{http://en.wikipedia.org/wiki/Orthogonal_matrix}{orthogonal} $m \times m$ matrix, \texttt{S}
is an $m \times n$
\href{http://en.wikipedia.org/wiki/Diagonal_matrix}{diagonal matrix}, and \texttt{V} is an $n \times n$
orthogonal matrix such that
\\[0.2cm]
\hspace*{1.3cm}
$A = U \cdot S \cdot V^{\mathtt{T}}$,
\\[0.2cm]
where $V^{\mathtt{T}}$ denotes the transpose of the matrix $V$.
  In practice, the singular value decomposition is often used to compute the
\href{http://en.wikipedia.org/wiki/Moore-Penrose_pseudoinverse}{\emph{pseudo-inverse}}
 of a singular matrix.   However, this can be done directly as for a matrix \texttt{A} the call
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_pseudoInverse(A);}
\\[0.2cm]
computes the pseudo-inverse of \texttt{A}.

\subsection{Eigenvalues and Eigenvectors}
In order to compute the
\href{http://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors}{eigenvalues and eigenvectors} of a
square matrix, \setlx\ provides the functions
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_eigenValues} \quad and \quad \texttt{la\_eigenVectors}.
\\[0.2cm]
Both of these function take a single argument \texttt{A}, where \texttt{A} must be a square matrix.
The function \texttt{la\_eigenValues} returns a list of the eigenvalues of \texttt{A}.  The
function call \texttt{la\_eigenVectors(a)} returns a list of the eigenvectors of the matrix \texttt{A}.
For example, in order to compute the eigenvalues and eigenvectors of the matrix
\\[0.2cm]
\hspace*{1.3cm}
$\mathtt{A} :=
\left(\begin{array}[c]{ll}
  1 & 2 \\
  2 & 3
\end{array}\right)
$
\\[0.2cm]
we can execute the following commands:

\begin{Verbatim}[ frame         = lines,
                  framesep      = 0.3cm,
                  firstnumber   = 1,
                  labelposition = bottomline,
                  numbers       = left,
                  numbersep     = -0.2cm,
                  xleftmargin   = 0.8cm,
                  xrightmargin  = 0.8cm,
                ]
    A        := << <<1 2>> <<2 3>> >>;
    [l1, l2] := la_eigenValues(a);
    [x1, x2] := la_eigenVectors(a);
\end{Verbatim}

Then, the eigenvalues will be computed as
\\[0.2cm]
\hspace*{1.3cm}
\texttt{l1 = -0.23606797749978958} \quad and \quad \texttt{l2 = 4.23606797749979}
\\[0.2cm]
In this case, the exact values of these eigenvalues are
\\[0.2cm]
\hspace*{1.3cm}
$\mathtt{l1} = 2 - \sqrt{5}$ \quad and \quad $\mathtt{l2} = 2 + \sqrt{5}$.
\\[0.2cm]
The eigenvectors are computed as
\\[0.2cm]
\hspace*{1.3cm}
\texttt{x1 = <<-0.8506508083520399 0.5257311121191336>>} \quad and \\
\hspace*{1.3cm}
\texttt{x2 = <<0.5257311121191336 0.8506508083520399>>}.
\\[0.2cm]
The eigenvectors computed are \emph{normalized}, that is the length of these vectors is $1$.
Furthermore, as the matrix \texttt{A} happens to be symmetric, i.e.~we have \texttt{A = A!},
the eigenvectors are orthogonal to each other.

Observe that not every square $n \times n$ matrix is
\href{http://en.wikipedia.org/wiki/Diagonalizable_matrix}{\emph{diagonalizable}}
and hence has $n$ different eigenvectors.  In case the matrix is not diagonalizable in the real
numbers, the functions \texttt{la\_eigenValues} and \texttt{la\_eigenVectors} both throw an
exception.  Note, however, that every
\href{http://en.wikipedia.org/wiki/Symmetric_matrix}{\emph{symmetric}} matrix is diagonalizable.
Therefore, if the matrix \texttt{A} is symmetric, neither of the two calls
\\[0.2cm]
\hspace*{1.3cm}
\texttt{la\_eigenValues(A)} \quad or \quad \texttt{la\_eigenVectors(A)}
\\[0.2cm]
will throw an exception.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "tutorial.tex"
%%% End:


\lab{K-Means Clustering}{K-Means Clustering}
\objective{Clustering is the one of the main tools in unsupervised learning---machine learning problems where the data comes without labels.
In this lab we implement the k-means algorithm, a simple and popular clustering method, and apply it to geographic clustering and color quantization.
}

\section*{Clustering}

Previously, we analyzed the iris dataset from \li{sklearn} using PCA; we have reproduced the first two principal components of the iris data in Figure \ref{fig:iris_data}.
Upon inspection, a human can easily see that there are two very distinct groups of irises.
Can we create an algorithm to identify these groups without human supervision?
This task is called \emph{clustering}, an instance of \emph{unsupervised learning}.

\begin{figure}
\centering
\includegraphics[width=.7\textwidth]{iris_pca.pdf}
\caption{The first two principal components of the iris dataset.}
\label{fig:iris_data}
\end{figure}

The objective of clustering is to find a partition of the data such that points in the same subset will be ``close'' according to some metric.
The metric used will likely depend on the data, but some obvious choices include Euclidean distance and angular distance.
Throughout this lab we will use the metric $d(x,y) = \|x-y\|_2$, the Euclidean distance between $x$ and $y$.

More formally, suppose we have a collection of $\mathbb{R}^K$-valued observations $X = \{x_1,x_2,\ldots,x_n\}$.
Let $N \in \mathbb{N}$ and let $\mathcal{S}$ be the set of all $N$-partitions of $X$, where an $N$-partition is a partition with exactly $N$ nonempty elements.
We can represent a typical partition in $\mathcal{S}$ as $S = \{S_1,S_2,\ldots,S_N\}$, where
\[
X = \bigcup_{i=1}^N S_i
\]
and
\[
|S_i| > 0, \qquad i=1,2,\ldots,N.
\]
We seek the $N$-partition $S^*$ that minimizes the within-cluster sum of squares, i.e.
\[
S^* = \underset{S\in\mathcal{S}}{\arg\min} \sum_{i=1}^N\sum_{x_j\in S_i}\|x_j-\mu_i\|_2^2,
\]
where $\mu_i$ is the mean of the elements in $S_i$, i.e.
\[
\mu_i = \frac{1}{|S_i|}\sum_{x_j\in S_i}x_j.
\]

\subsection*{The K-Means Algorithm}

Finding the global minimizing partition $S^*$ is generally intractable since the set of partitions can be very large indeed,
but the \emph{k-means} algorithm is a heuristic approach that can often provide reasonably accurate results.

We begin by specifying an initial cluster mean $\mu_i^{(1)}$ for each $i = 1, \cdots, N$ (this can be done by random initialization, or according to some heuristic).
For each iteration, we adopt the following procedure.
Given a current set of cluster means $\mu^{(t)}$, we find a partition $S^{(t)}$ of the observations such that
\begin{equation*}
S_{i}^{(t)} = \{x_j \; : \; \|x_j - \mu_{i}^{(t)}\|_2^2 \leq \|x_j - \mu_{l}^{(t)}\|_2^2,\,\,\,  l = 1, \cdots, N\}.
\end{equation*}
We then update our cluster means by computing for each $i = 1, \cdots, N$.
We continue to iterate in this manner until the partition ceases to change.

Figure \ref{fig:iris_clusterings} shows two different clusterings of the iris data produced by the \emph{k-means} algorithm.
Note that the quality of the clustering can depend heavily on the initial cluster means.
We can use the within-cluster sum of squares as a measure of the quality of a clustering (a lower sum of squares is better).
Where possible, it is advisable to run the clustering algorithm several times, each with a different initialization of the means,
and keep the best clustering.
Note also that it is possible to have very slow convergence.
Thus, when implementing the algorithm, it is a good idea to terminate after some specified maximum number of iterations.
%
\begin{figure}[h]
    \centering
    \begin{tabular}{cc}
    \includegraphics[width=.49\textwidth]{iris_means_1.pdf} &
    \includegraphics[width=.49\textwidth]{iris_means_2.pdf}
    \end{tabular}
    \caption{Two different K-Means clusterings for the iris dataset.
            Notice that the clustering on the left predicts the flower species to a high degree of accuracy,
            while the clustering on the right is less effective.}
    \label{fig:iris_clusterings}
\end{figure}
%
The algorithm can be summarized as follows.
\begin{enumerate}
    \item Choose $k$ initial cluster centers.
    \item For $i=0,\ \ldots,$\ \li{max_iter},
    \begin{enumerate}
        \item Assign each data point to the cluster center that is closest, forming $k$ clusters.
        \item Recompute the cluster centers as the means of the new clusters.
        \item If the old cluster centers and the new cluster centers are sufficiently close, terminate early.
    \end{enumerate}
\end{enumerate}

\begin{problem}
Write a \li{KMeans} class for doing basic $k$-means clustering.
Implement the following methods, following \li{sklearn} class conventions.
\begin{enumerate}
\item \li{__init__()}: Accept a number of clusters $k$, a maximum number of iterations, and a convergence tolerance.
Store these as attributes.

\item \li{fit()}: Accept an $m \times n$ matrix $X$ of $m$ data points with $n$ features.
Choose $k$ random rows of $X$ as the initial cluster centers.
Run the $k$-means iteration until consecutive centers are within the convergence tolerance, or until iterating the maximum number of times.
Save the cluster centers as attributes.

If a cluster is empty, reassign the cluster center as a random row of $X$.

\item \li{predict()}: Accept an $l \times n$ matrix $X$ of data.
Return an array of $l$ integers where the $i$th entry indicates which cluster center the $i$th row of $X$ is closest to.
\end{enumerate}
%
Test your class on the iris data set after reducing the data to two principal components.
Plot the data, coloring by cluster.
\end{problem}

\section*{Detecting Active Earthquake Regions} % ==============================

Suppose we are interested in learning about which regions are prone to experience frequent earthquake activity.
We could make a map of all earthquakes over a given period of time and examine it ourselves, but this, as an unsupervised learning problem, can be solved using our $k$-means clustering tool.

% % Old approach (required students to parse 6 files for some reason).
% These files contain a lot of information which isn't of interest to us at the present time; all we would like to extract from them is the location of each earthquake, which appears in characters $21$ through $33$ of each line.
% Characters $21$ through $26$ contain the latitude of each epicenter, character $26$ denoting North or South, and characters $27$ through $33$ contain the longitude of each epicenter, character $33$ denoting East or West.
% We need to divide each value by $1,000$ to represent these as degrees and decimals.
\begin{figure}[H]
    \centering
    \includegraphics[width=.7\textwidth]{earthquakes.png}
    \caption{Earthquake epicenters over a 6 month period.}
    \label{fig:earthquakes}
\end{figure}

The file \texttt{earthquake\_coordinates.npy} contains earthquake data throughout the world from January 2010 through June 2010.
Each row represents a different earthquake; the columns are scaled longitude and latitude measurements.
We want to cluster this data into active earthquake regions.
For this task, we might think that we can regard any epicenter as a point in $\mathbb{R}^{2}$ with coordinates being their latitude and longitude.
This, however, would be incorrect, because the earth is not flat.
Instead, latitude and longitude should be viewed in \emph{spherical coordinates} in $\mathbb{R}^{3}$, which could then be clustered.

A simple way to accomplish this transformation is to first transform the latitude and longitude values to spherical coordinates, and then to Euclidean coordinates.
Recall that a spherical coordinate in $\mathbb{R}^3$ is a triple $(r,\theta,\varphi)$, where $r$ is the distance from the origin, $\theta$ is the radial angle in the $xy$-plane from the $x$-axis, and $\varphi$ is the angle from the $z$-axis.
In our earthquake data, once the longitude is converted to radians it is an appropriate $\theta$ value; the latitude needs to be offset by $90^\circ$ degrees, then converted to radians to obtain $\varphi$.
For simplicity, we can take $r=1$, since the earth is roughly a sphere.
We can then transform to Euclidean coordinates using the following relationships.
\[
\theta = \frac{\pi}{180}\left(\text{longitude}\right)
\qquad
\varphi = \frac{\pi}{180}\left(90 - \text{latitude}\right)
\]
\begin{align*}
r & = \sqrt{x^{2} + y^{2} + z^{2}} & x & = r \sin \varphi \cos \theta \\
\varphi & = \arccos \frac{z}{r} & y & = r \sin \varphi \sin \theta \\
\theta & = \arctan \frac{y}{x} & z & = r \cos \varphi
\end{align*}

There is one last issue to solve before clustering.
Each earthquake data point has norm 1 in Euclidean coordinates, since it lies on the surface of a sphere of radius 1.
Therefore, the cluster centers should also have norm 1.
Otherwise, the means can't be interpreted as locations on the surface of the earth, and the \emph{k-means} algorithm will struggle to find good clusters.
A solution to this problem is to normalize the mean vectors at each iteration, so that they are always unit vectors.

\begin{problem}
Add a keyword argument \li{normalize=False} to your \li{KMeans} constructor.
Modify \li{fit()} so that if \li{normalize} is \li{True}, the cluster centers are normalized at each iteration.

Cluster the earthquake data in three dimensions by converting the data from raw data to spherical coordinates to euclidean coordinates on the sphere.
\begin{enumerate}
\item Convert longitude and latitude to radians, then to spherical coordinates.
\\(Hint: \li{np.deg2rad()} may be helpful.)
\item Convert the spherical coordinates to euclidean coordinates in $\mathbb{R}^3$.
\item Use your \li{KMeans} class with normalization to cluster the euclidean coordinates.
\item Translate the cluster center coordinates back to spherical coordinates, then to degrees.
Transform the cluster means back to latitude and longitude coordinates.
\\(Hint: use \li{numpy.arctan2()} for $\arctan$, so that that correct quadrant is chosen).
\item Plot the data, coloring by cluster.
Also mark the cluster centers.
\end{enumerate}
With 15 clusters, your plot should resemble the Figure \ref{fig:earthquakeclusters}.
\end{problem}

\begin{figure}[H]
    \centering
    \includegraphics[width=.7\textwidth]{earthquake_clusters.png}
    \caption{Earthquake epicenter clusters with $k = 15$.}
    \label{fig:earthquakeclusters}
\end{figure}

\begin{comment}
Though plotting our results in two dimensions gives us a good picture, we can see that this is not entirely accurate.  There are points that appear to be closer to a different cluster center than the one to which they belong.  This comes from viewing the results in only two dimensions.  When viewing in three dimensions, we can see more clearly the accuracy of our results.
\begin{problem}
Add a keyword argument \li{3d=False} to your \li{kmeans} function, and add code to show the three-dimensional plot instead of the two-dimensional scatter plot should this argument be set to \li{True}.  Maintain the same color-coding scheme as before.  Use \li{mpl_toolkits.mplot3d.Axes3D} to make your plot.
\end{problem}
\end{comment}

\section*{Color Quantization} % ===============================================

The $k$-means algorithm uses the euclidean metric, so it is natural to cluster geographic data.
However, clustering can be done in any abstract vector space.
The following application is one example.

Images are usually represented on computers as $3$-dimensional arrays.
Each $2$-dimensional layer represents the red, green, and blue color values, so each pixel on the image is really a vector in $\mathbb{R}^3$.
Clustering the pixels in $RGB$ space leads a one kind of image segmentation that facilitate memory reduction.

Reading: \url{https://en.wikipedia.org/wiki/Color_quantization}

\begin{problem}
Write a function that accepts an image array (of shape $(m,n,3)$), an integer number of clusters $k$, and an integer number of samples $S$.
Reshape the image so that each row represents a single pixel.
Choose $S$ pixels to train a $k$-means model on with $k$ clusters.
Make a copy of the original picture where each pixel has the same color as its cluster center.
Return the new image.
For this problem, you may use \li{sklearn.cluster.KMeans} instead of your \li{KMeans} class from Problem 1.

Test your function on some of the provided NASA images.
\end{problem}

\newpage

\section*{Additional Material} % ==============================================

\subsection*{Spectral Clustering}
We now turn to another method for solving a clustering problem, namely that of Spectral Clustering.  As you can see in Figure ???, it can cluster data not just by its location on a graph, but can even separate shapes that overlap others into distinct clusters.  It does so by utilizing the spectral properties of a Laplacian matrix.  Different types of Laplacian matrices can be used.  In order to construct a Laplacian matrix, we first need to create a graph of vertices and edges from our data points.  This graph can be represented as a symmetric matrix $W$ where $w_{ij}$ represents the edge from $x_i$ to $x_j$.  In the simplest approach, we can set $w_{ij} = 1$ if there exists an edge and $w_{ij} = 0$ otherwise.  However, we are interested in the similarity of points, so we will weight the edges by using a \emph{similarity measure}.  Points that are similar to one another are assigned a high similarity measure value, and dissimilar points a low value.  One possible measure is the \emph{Gaussian similarity function}, which defines the similarity between distinct points $x_i$ and $x_j$ as
\begin{equation*}
s(x_i,x_j) = e^{- \frac{\| x_i - x_j \| ^2}{2 \sigma ^2}}
\end{equation*}
for some set value $\sigma$.

Note that some similarity functions can yield extremely small values for dissimilar points.  We have several options for dealing with this possibility.  One is simply to set all values which are less than some $\epsilon$ to be zero, entirely erasing the edge between these two points.  Another option is to keep only the $T$ largest-valued edges for each vertex.  Whichever method we choose to use, we will end up with a weighted \emph{similarity matrix} $W$.  Using this we can find the diagonal \emph{degree matrix} $D$, which gives the number of edges found at each vertex.  If we have the original fully-connected graph, then $D_{ii} = n-1$ for each $i$.  If we keep the $T$ highest-valued edges, $D_{ii} = T$ for each $i$.

As mentioned before, we may use different types of Laplacian matrices.  Three such possibilities are:
\begin{enumerate}
    \item The \emph{unnormalized Laplacian}, $L = D - W$
    \item The \emph{symmetric normalized Laplacian}, $L_{sym} = I - D^{-1/2}WD^{-1/2}$
    \item The \emph{random walk normalized Laplacian}, $L_{rw} = I - D^{-1}W$.
 \end{enumerate}

Given a similarity measure, which type of Laplacian to use, and the desired number of clusters $k$, we can now proceed with the Spectral Clustering algorithm as follows:

\begin{itemize}
    \item Compute $W$, $D$, and the appropriate Laplacian matrix.
    \item Compute the first $k$ eigenvectors $u_1, \cdots , u_k$ of the Laplacian matrix.
    \item Set $U = [u_1, \cdots , u_k]$, and if using $L_{sym}$ or $L_{rw}$ normalize $U$ so that each row is a unit vector in the Euclidean norm.
    \item Perform $k$-means clustering on the $n$ rows of $U$.
    \item The $n$ labels returned from your \li{kmeans} function correspond to the label assignments for $x_1, \cdots, x_n$.
\end{itemize}

As before, we need to run through our $k$-means function multiple times to find the best measure when we use random initialization.  Also, if you normalize the rows of $U$, then you will need to set the argument \li{normalize = True}.

\begin{problem}
Implement the Spectral Clustering Algorithm by calling your \li{kmeans} function, using the following function declaration:
\begin{lstlisting}
def specClus(measure,Laplacian,args,arg1=None,kiters=10):
    """
    Cluster a dataset using the k-means algorithm.

    Parameters
    ----------
    measure : function
        The function used to calculate the similarity measure.
    Laplacian : int in {1,2,3}
        Which Laplacian matrix to use. 1 corresponds to the unnormalized,
        2 to the symmetric normalized, 3 to the random walk normalized.
    args : tuple
        The arguments as they were passed into your k-means function,
        consisting of (data, n_clusters, init, max_iter, normalize). Note
        that you will not pass 'data' into your k-means function.
    arg1 : None, float, or int
        If Laplacian==1, it should remain as None
        If Laplacian==2, the cut-off value, epsilon.
        If Laplacian==3, the number of edges to retain, T.
    kiters : int
        How many times to call your kmeans function to get the best
        measure.

    Returns
    -------
    labels : ndarray of shape (n,)
        The i-th entry is an integer in [0,n_clusters-1] indicating
        which cluster the i-th row of data belongs to.
    """
    pass
\end{lstlisting}
\end{problem}

We now need a way to test our code.  The website http://cs.joensuu.fi/sipu/datasets/ contains many free data sets that will be of use to us.  Scroll down to the ``Shape sets" heading, and download some of the datasets found there to use for trial datasets.
\begin{problem}
Create a function that will return the accuracy of your spectral clustering implementation, as follows:
\begin{lstlisting}
def test_specClus(location,measure,Laplacian,args,arg1=None,kiters=10):
    """
    Cluster a dataset using the k-means algorithm.

    Parameters
    ----------
    location : string
        The location of the dataset to be tested.
    measure : function
        The function used to calculate the similarity measure.
    Laplacian : int in {1,2,3}
        Which Laplacian matrix to use. 1 corresponds to the unnormalized,
        2 to the symmetric normalized, 3 to the random walk normalized.
    args : tuple
        The arguments as they were passed into your k-means function,
        consisting of (data, n_clusters, init, max_iter, normalize). Note
        that you will not pass 'data' into your k-means function.
    arg1 : None, float, or int
        If Laplacian==1, it should remain as None
        If Laplacian==2, the cut-off value, epsilon.
        If Laplacian==3, the number of edges to retain, T.
    kiters : int
        How many times to call your kmeans function to get the best
        measure.

    Returns
    -------
    accuracy : float
        The percent of labels correctly predicted by your spectral
        clustering function with the given arguments (the number
        correctly predicted divided by the total number of points.
    """
    pass
\end{lstlisting}
\end{problem}


\documentclass[../main.tex]{subfiles}

\begin{document}
        \section{Curves \& Arc Length}
        \begin{definition}{Smoothness}{}
                A vector valued function \(\vec{r}(t)\) is \emph{smooth} on an interval \((a,b)\) if \(\vec{r}'(t)\) exists \emph{and} \(\vec{r}'(t)\neq\vec{0}\) on the interval.
        \end{definition}
        In the above definition, the case where \(\vec{r}'(t)=\vec{0}\) corresponds to coming to a full stop, at which point motion may resume in any direction while maintaining the existence of the derivative, so a stronger condition is necessary to eliminate these cases.
        \begin{definition}{Closure}{}
                A vector valued function \(\vec{r}(t)\) is \emph{closed} on an interval \((a,b)\) if \(\vec{r}(a)=\vec{r}(b)\).
        \end{definition}
        \begin{definition}{Simplicity}{}
                A vector valued function \(\vec{r}(t)\) is \emph{simple} on an interval \((a,b)\) if it has no self-intersections on the interval, except possibly where \(\vec{r}(a)=\vec{r}(b)\).
        \end{definition}
        Given a curve described by some function \(\vec{r}(t)\) on some interval \((a,b)\), we may wish to find its arc length. We assume that \(\vec{r}\) is continuous, smooth, and 1-to-1. This may be done by approximating the curve as a sequence of line segments, and taking the limit as their length goes to \(0\). We consider some set of points \(\{t_0,\ldots,t_n\}\), where \(a=t_0 < t_1 < \ldots < t_n=b\), which yield the corresponding line segments \(\{\vec{r}(t_1)-\vec{r}(t_0),\ldots,\vec{r}(t_n)-\vec{r}(t_{n-1})\}\) whose lengths may be summed up to obtain an approximation.
        \begin{definition}{Rectifiability}{}
                A curve is rectifiable if there exists some \(k>0\) such that the length of an approximation of the curve in terms of line segments is less than \(k\) for any number of line segments. In other words, if the length of the approximation approaches a limit.
        \end{definition}
        If a curve is rectifiable with some \(k\), then the smallest such \(k\) is the length of the curve. Formulaically,
        \[
        L = \int_a^b \left| \vec{r}'(t) \right|\, dt = \int_a^b v(t)\,dt
        \]
        which may be proven by noting that the length of the line segment between \(\vec{r}(t_i)\) and \(\vec{r}(t_{i-1})\) is \(|\vec{r}(t_i)-\vec{r}(t_{i-1})|\), which approaches \(|\vec{v}'(t_i)|\) as \(t_i\) and \(t_{i-1}\) become very close.
        \begin{example}{}{}
                Find the length of the helix described by \(\vec{r}(t)=a\cos{t}\i+a\sin{t}\j+bt\) on the interval \(0\leq t \leq T\).
                \tcblower
                \(\vec{r}'(t)=-a\sin{t}\i+a\cos{t}\j+b\k\), so \(v(t)=\sqrt{a^2\sin^2{t}+a^2\cos^2{t}+b^2}=\sqrt{a^2+b^2}\) and
                \[
                L = \int_0^T \sqrt{a^2+b^2}\,dt = \left[\sqrt{a^2+b^2}t\right]_0^T = T\sqrt{a^2+b^2}
                \]
        \end{example}
        \begin{example}{}{}
                Find the length of the curve described by \(\vec{r}(t)=2t\i+t^2\j+\frac{1}{3}t^3\k\) on the interval \(1\leq t \leq 2\).
                \tcblower
                \(\vec{r}'(t)=2\i+2t\j+t^2\k\), so \(v(t)=\sqrt{4+4t^2+t^4}=\sqrt{(2+t^2)^2}=2+t^2\), and the length of the curve is
                \[
                L = \int_1^2 2+t^2\,dt = \left[2t+\frac{t^3}{3}\right]_1^2 = \frac{13}{3}
                \]
        \end{example}
        Let's say that \(\vec{r}:[a,b]\to\mathbb{R}^d\) is a smooth parametric curve, and \(s(t)\) is the length of the curve from \(\vec{r}(a)\) to \(\vec{r}(t)\). We may then use \(s\) to parametrize the curve instead of \(t\), which yields a parameterization with a constant speed of \(1\).

        Note that
        \[
        s(t) = \int_a^t v(u)\,du
        \]
        so
        \[
        \frac{ds}{dt}=v(t)
        \]
        by FTC.
        \begin{theorem}{}{}
                Prove that an arc length parametrized curve has constant speed \(1\).
                \tcblower
                Note that
                \[
                \left|\frac{d\vec{r}}{dt}\right|=\left|\frac{d\vec{r}}{dt}\frac{dt}{ds}\right|
                \]
                but by the definition of arc length parameterization we have \(\frac{dt}{ds}=\frac{1}{v(t)}\), which is always defined because \(\vec{r}\) is smooth. In that case,
                \[
                \frac{d\vec{r}}{dt}=|v(t)|\frac{1}{|v(t)|}=1
                \]
        \end{theorem}
        \begin{example}{}{}
                Find the arc length parameterization of the helix described by \(\vec{r}(t)=a\cos{t}\i+a\sin{t}\j+bt\) on the interval \(0\leq t \leq T\).
                \tcblower
                \(\vec{r}(t)=a\cos{t}\i+a\sin{t}\j+b\k\), so \(v(t)=\sqrt{a^2+b^2}\) and
                \[
                s(t) = \int_0^t \sqrt{a^2+b^2}\,du = t\sqrt{a^2+b^2}
                \]
                which we may use this to reparameterize the curve in the form
                \[
                        \vec{r}=a\cos\left(\frac{s}{\sqrt{a^2+b^2}}\right)\i + a\sin\left(\frac{s}{\sqrt{a^2+b^2}}\right)\j + b\frac{s}{\sqrt{a^2+b^2}}\k
                \]
        \end{example}
        \begin{example}{}{}
                Find the arc length parameterization of the curve described by \(\vec{r}(t)=2t\i+t^2\j+\frac{t^3}{3}\k\) on the interval \(0 \leq t \leq T\).
                \tcblower
                \(\vec{r}(t)=2t\i+t^2\j+\frac{t^3}{3}\k\), so \(v(t)=t^2+2\) and
                \[
                s(t) = \int_0^tu^2+2\,du = \frac{t^3}{3}+2t
                \]
                which may be solved for \(t\) to find a parameterization. The algebra involved is tedious, and thus elided.
        \end{example}
\end{document}


\section{Integral Curves}
\noindent
An integral curve, $\vec{r}$, of a vector field, $\vec{F}$, is a VVF such that $\vec{r^\prime}(t) = \left(\vec{F}\circ\vec{r}\right)(t)$.\\

\noindent
For example, let's find an integral curve to $\vec{F}(x,y) = \langle x, 2y\rangle$.
\begin{equation*}
	\frac{\mathrm{d}}{\mathrm{d}t}\vec{r}(t) = \vec{F}\circ\vec{r}
\end{equation*}
\begin{equation*}
	\frac{\mathrm{d}}{\mathrm{d}t}\langle x(t), y(t) \rangle = \langle x(t), 2 y(t) \rangle \to
	\begin{cases}
		x^\prime(t) = x(t) \\
		y^\prime(t) = 2 y(t)
	\end{cases}
	\implies \vec{r}(t) = \langle c_1 e^t, c_2 e^{2t} \rangle
\end{equation*}

\noindent
These integral curves have applications in physics and show up naturally in nature alongside vector fields. In physics, the integral curves for an electric field are called “field lines” and integral curves of the velocity field of a fluid like water or air are called “streamlines.” You may recognize these curves as tracing out a slope field.

[INSERT IMAGE]


%!TEX root=ClassNotes.tex
\section{Fundamental Theorem of Calculus}

The Fundamental Theorem of Calculus is a remarkable theorem that connects the notions of derivatives and integrals.

If we denote by $\mathcal{D}$ the {\it operator} that takes a function and produces it's derivative i.e. $\mathcal{D}(f) = f'$ and by $\mathcal{I}$ the {\it operator} that takes a function and produces (one of) it's antiderivative i.e. $\mathcal{I}(f) = \int_a^x f(t) \: \:dt$, for some real number $a$, then the Fundamental Theorem of Calculus says that
\begin{enumerate}
	\item $\mathcal{D}(\mathcal{I}(f)) = f$, and
	\item  $\mathcal{I}(\mathcal{D}(f)) = f + $ a constant.
\end{enumerate} so that Integral and Derivative are (almost) inverse operators.\footnote{Compare this to the definition of inverse functions: $f(g(x)) = x$ and $g(f(x)) = x$.} This is why $\int_a^x f(t) \: \:dt$ is called (an) {antiderivative}.\\

\begin{theorem}[Fundamental Theorem of Calculus]$ $
	\begin{description}
		\item[Form 1] Let $f$ be a continuous function. For a real number $a$, let $ F_a(x) = \int \limits_a^x f(t) \:dt$.
		      Then $F_a(x)$ is differentiable with
		      \begin{align*}
			      F_a'(x) = f(x).
		      \end{align*}
		\item[Form 2]
		      Let $f$ be a differentiable function with $f'$ continuous. Then for any real numbers $a$ and $x$,
		      \begin{align*}
			      \int \limits_a^x f'(t) \:dt = f(x) - f(a).
		      \end{align*}
	\end{description}
\end{theorem}

The idea of the proof of Form 1 is very straightforward. By definition of derivative,
\begin{align}
	\begin{split}
		F_a'(x)
		& = \lim \limits_{h \rightarrow 0} \dfrac{F_a(x+h) - F_a(x)}{h}                                                                                \\
		& = \lim \limits_{h \rightarrow 0} \dfrac{\int \limits_a^{x+h} f(t) \:dt - \int \limits_a^x f(t) \:dt}{h}                                                           \\
		& = \lim \limits_{h \rightarrow 0} \dfrac{\int \limits_x^{x+h} f(t) \:dt}{h}
		\qquad
		\qquad
		\mbox{ by Exercise } \ref{q:difference_antiderivatives}
	\end{split}
	\label{eq:Fundamental_Theorem_Reduction}
\end{align}
For small $h$ this last integral can be approximated by the area of a rectangle with height $f(x)$ and base $h$.
\begin{align*}
	\int \limits_x^{x+h} f(t) \:dt
	 & \approx f(x) \cdot h \\
	\implies \dfrac{\int \limits_x^{x+h} f(t) \:dt}{h}
	 & \approx f(x)
\end{align*}
We'll make this last statement precise using $\epsilon, \delta$ arguments. We'll start with a small Proposition.
\begin{figure}[t]
	\centering
	\includegraphics[width=0.6\textwidth]{FundamentalTheorem.png}
	\caption*{The area $\int_x^{x+h} f(t)\:dt$ can be approximated by a rectangle with height $f(x)$ and base $h$.}
\end{figure}

\begin{prop}
	\label{theorem:Fundamental_theorem_lemma}
	If for all $ y$ in the interval $(x,x+h)$,
	\begin{align*}
	m < f(y) < M
	\end{align*}
	for some real numbers $m$, $M$, then
	\begin{align*}
		mh < \int \limits_{a}^{b} f(t) \: \:dt < Mh
	\end{align*}
\end{prop}
\begin{exercise}$ $
	\begin{enumerate}
		\item Prove the above Proposition using a {\it geometric} argument.
		\item \textbf{Optional: } Prove the above Proposition rigorously using the definition of integral involving Riemann sums.
	\end{enumerate}
\end{exercise}

Now we get back to the proof of the Form 1 of the Fundamental Theorem of Calculus.
\begin{proof}[Proof of Form 1 of the Fundamental Theorem of Calculus]
	Let $f(x)$ be a continuous function, let $a$ be a real number. We want to show that
	\begin{align*}
		F_a'(x) = f(x)
	\end{align*}
	By Equations \eqref{eq:Fundamental_Theorem_Reduction} this is equivalent to proving that
	\begin{align*}
		\lim \limits_{h \rightarrow 0}\dfrac{\int \limits_x^{x+h} f(t) \:dt}{h}
		 & = f(x)
	\end{align*}
	We'll only prove this for the right hand limit, the left hand limit is proved similarly. We want to show that
	\begin{align}
		\label{eq:Fundamental_theorem_limit}
		\lim \limits_{h \rightarrow 0^+}\dfrac{\int \limits_x^{x+h} f(t) \:dt}{h}
		 & = f(x)
	\end{align}
	\begin{exercise}
		\label{q:epsilon_delta_limit}
		Convert Equation \eqref{eq:Fundamental_theorem_limit} into an $\epsilon, \delta$ statement using the definition of limit.
	\end{exercise}

	\begin{exercise}
		Recall what it means for $f(x)$ to be continuous from the right, using the  $\epsilon, \delta$ definition of continuity.
	\end{exercise}

	\begin{exercise}
		Using this $\delta$ and Proposition \ref{theorem:Fundamental_theorem_lemma} prove the statement in Exercise \ref{q:epsilon_delta_limit}, thereby completing the proof of the Fundamental Theorem of Calculus.
	\end{exercise}
\end{proof}

\begin{exercise}[{\bf Optional}]
	Prove that $\lim \limits_{h \rightarrow 0^-}\dfrac{\int \limits_x^{x+h} f(t) \:dt}{h}
	= f(x)$. (Be careful of the signs.)
\end{exercise}

The Form 2 of the fundamental theorem follows directly from Form 1 using a small Proposition about derivatives.
\begin{prop}
	\label{theorem:Fundamental_theorem_lemma2}
	If $g(x)$ is a differentiable function such that $g'(x) = 0$ for all real numbers $x$, then $g(x)$ is a constant function i.e. $g(x) = g(y)$ for all real numbers $x$, $y$.
\end{prop}
\begin{exercise}
	Prove the above Proposition by Contradiction\hint{Use the Mean Value Theorem}.
\end{exercise}

\begin{proof}[Proof of Form 2 of the Fundamental Theorem of Calculus]
	Let $f$ be a diffentiable function with $f'$ continuous, and let $a$ be a real number. We want to show that
	\begin{align*}
		\int \limits_a^x f'(t) \:dt = f(x) - f(a)
	\end{align*}
	Consider the function
	\begin{align*}
		g(x) = f(x) - \int \limits_a^x f'(t) \:dt.
	\end{align*}
	\begin{exercise}
		Using Form 1 of the Fundamental Theorem and Proposition \ref{theorem:Fundamental_theorem_lemma2} prove that $g(x)$ is a constant function.
	\end{exercise}
	\begin{exercise}
		As $g$ is a constant function, for all real numbers $x$, $g(x) = g(a)$. Use this to prove Form 2 of the Fundamental Theorem of Calculus.
	\end{exercise}
\end{proof}

\begin{remark}
	For real numbers $b$, $c$,
		\begin{align*}
			\int \limits_b^c f(t) \:dt
			&= \int \limits_b^a f(t) \:dt + \int \limits_a^c f(t) \:dt\\
			&= -\int \limits_a^b f(t) \:dt + \int \limits_a^c f(t) \:dt\\
			&= F_a(c) - F_a(b)
		\end{align*}
	where $F_a(x)$ is {\it an} antiderivative of $f(x)$ i.e. it is a function whose derivative is $f(x)$. This is the form is which the Fundamental Theorem of Calculus is most often used. Which $a$ we use to find the antiderivative does not matter, so the integral can be written in the following forms.
	\begin{align*}
		\int \limits_b^c f(t) \: \:dt
		&= F_a(c) - F_a(b) \\
		&= \left. F_a(x) \right|_{b}^c \\
		&= \left. \int f(x) dx \right|_{b}^c \\
		&= \left. \int f(x) dx \right|_{x=b}^{x=c}
	\end{align*}
	It is acceptable to use any of these forms as they all mean the same thing.
\end{remark}

\newpage
\subsection{Understanding the Fundamental Theorem of Calculus}
The Fundamental Theorem has a LOT of applications. We'll spend the next several sections just understanding this theorem.
\begin{exercise}
	Let $a$ be a real number. Find derivatives of the following functions using the Fundamental Theorem and simple algebraic manipulations.
	\begin{enumerate}
		\item $\int \limits_{x}^{a} f(t) \: dt$.
		\item $\int \limits_a^{x^2} f(t) \: dt$.
		\item $\int \limits_a^{g(x)} f(t) \: dt$, where	$g(x)$ is a differentiable function.
		\item $\int \limits_{x}^{x^2} f(t) \: dt$.
		\item $\int \limits_{h(x)}^{g(x)} f(t) \: dt$, where	$g(x)$, $h(x)$ are differentiable functions.
		% \item $\int \limits_a^{x} t \cdot f(t) \: dt$.
		\item $\int \limits_a^{x} x \cdot f(t) \: dt$. (Be careful!)
	\end{enumerate}
\end{exercise}

\begin{exercise}
	Prove that if
	\begin{align*}
		f'(t) > 0
	\end{align*}
	for all $t \in [a,b]$, then $f$ is increasing on the interval $[a,b]$ i.e. for all $x$, $y$ in the interval $[a,b]$ if $x < y$ then $f(x) < f(y)$.\hint{Compute the integral $\int \limits_x^y f'(t) \: dt$.}
\end{exercise}

\begin{theorem}[u-substitution]
	Let $f$, $g$ be differentiable functions. Then,
	\begin{align*}
		\int_a^x f(g(t)) g'(t)\: dt = \int_{g(a)}^{g(x)} f(u) \: du
	\end{align*}
\end{theorem}
\begin{exercise}[Proof of u-substitution] Consider the function
	\begin{align*}
		h(x)
		=
		\int_a^x f(g(t)) g'(t)\: dt - \int_{g(a)}^{g(x)} f(u) \: du
	\end{align*}
\begin{enumerate}
	\item Show that $h'(x) = 0$.
	\item Show that $h(a) = 0$.
	\item Argue that these two statements imply that $h(x) = 0$ for all $x$, thereby proving u-substitution.
\end{enumerate}
\end{exercise}

\begin{theorem}[Integration by parts]
	Let $f$, $g$ be differentiable functions. Then,
	\begin{align}
		\label{eq:integration_by_parts}
		\int_a^x f(t) g'(t)\: dt = f(x)g(x) - \int_{a}^{x} f'(t) g(t) \: dt + c
	\end{align}
	for some constant $c$.
\end{theorem}
\begin{exercise}[Proof of Integration by parts] Consider the function
	\begin{align*}
		h(x)
		=
		\int_a^x f(t) g'(t)\: dt - \left(f(x)g(x) - \int_{a}^{x} f'(t) g(t) \: dt \right)
	\end{align*}
Show that $h'(x) = 0$. Argue that this proves Integration by parts.
\end{exercise}

\begin{exercise}
	Find the contant $c$ in Equation \eqref{eq:integration_by_parts}.
\end{exercise}

Because of the constant $c$ that shows up in Equation \eqref{eq:integration_by_parts} integration by parts is often expressed using indefinite integrals as,
\begin{align*}
	\int f(x) g'(x)\: dx = f(x)g(x) - \int f'(x) g(x) \: dx + c
\end{align*}

\begin{remark}
	These two theorems
	\begin{itemize}
		\item u-substitution
		\item integration by parts
	\end{itemize}
	provide the main techniques for computing integrals. In the later sections we'll do several problems to practice these.
\end{remark}


\section{Cap product and ``\v{C}ech'' cohomology}
Let $R$ be a commutative ring with coefficients. The cap product is a map $\cap: H^p(X)\otimes H_n(X)\to H_{q}(X)$ where $p+q=n$. This comes from a chain level map $S^p(X)\otimes S_n(X)\xrightarrow{1\otimes\alpha} S^p(X)\otimes S_p(X)\otimes S_q(X)\xrightarrow{\langle-,-\rangle\otimes 1}R\otimes S_q(X)\cong S_q(X)$. Using our explicit formula for $\alpha$, we can write:
\begin{equation*}
\cap:\beta\otimes\sigma\mapsto\beta\otimes(\sigma\circ\alpha_p)\otimes(\sigma\circ\omega_q)\mapsto\left(\beta(\sigma\circ\alpha_p)\right)\cdot (\sigma\circ\omega_q)
\end{equation*}
There's many things to say:
\begin{enumerate}
\item $ H_\ast(X)$ is a module for $ H^\ast(X)$.
\item The only reasonable thing to ask for in terms of naturality is the following. Suppose $f:X\to Y$, and let $b\in H^p(Y)$ and $x\in H_n(X)$. We then have $f_\ast(f^\ast(b)\cap x)=b\cap f_\ast(x)$, where $f^\ast(b)\cap x\in H_q(X)$ and $b\cap f_\ast(x)\in H_n(X)$. This is called a projection formula. To see this, let $[\beta]=b$. Then:
\begin{align*}
f_\ast(f^\ast(\beta)\cap\sigma)& =f_\ast(\left(f^\ast(\beta)(\sigma\circ\alpha_p)\right)\cdot(\sigma\circ\omega_q))\\
& =f_\ast(\beta(f\circ\sigma\circ\alpha_p)\cdot(\sigma\circ\omega))\\
& =\beta(f\circ\sigma\circ\alpha_p)\cdot f_\ast(\sigma\circ\omega_q)\\
& = \beta(f\circ\sigma\circ\alpha_p)\cdot(f\circ\sigma\circ\omega_q)\\
& = \beta\cap f_\ast(\sigma)
\end{align*}
So we're done.
\item There's a relation between the cap and Kronecker product. Any space has an augmentation $\varepsilon:X\to\ast$, so I get $\varepsilon_\ast: H_\ast(X)\to R$. Maybe we should compute $\varepsilon_\ast(\beta\cap \sigma)$. I will get zero unless $p=n$ and $q=0$. What does our formula say? This just says that $\varepsilon_\ast(b\cap x)=\varepsilon_\ast(\beta(\sigma)\cdot c^0_{\sigma(n)})=\beta(\sigma)\varepsilon_\ast(c^0_{\sigma(n)})=\beta(\sigma)=\langle \beta,\sigma\rangle$ because $\varepsilon_\ast$ counts the number of points, i.e., it's $1$. Hence $\varepsilon_\ast(b\cap x)=\langle b,x\rangle$.
\item What is $\langle a\cup b,x\rangle$? This is $\varepsilon_\ast((a\cup b)\cap x)=\varepsilon_\ast(a\cap(b\cap x))$ by an assertion in the previous lecture (namely that $(\alpha\cup\beta)\cap x=\alpha\cap(\beta\cap x)$ and $1\cap x=x$), which becomes $\langle a,b\cap x\rangle$. In other words, $\langle a\cup b,x\rangle=\langle a,b\cap x\rangle$. So the cup product is adjoint to the cap product.
\end{enumerate}
\subsection{Relative $\cap$}
There's a lot of structure, but we want more. We want to now try to understand the relative cap product. Suppose $A\subseteq X$ is a subspace. We have:
\begin{equation*}
\xymatrix{
	0\ar[d] & & 0\ar[d]\\
	S^p(X)\otimes S_n(A)\ar[d]^{1\otimes i_\ast}\ar[r]^{i^\ast\otimes 1} & S^p(A)\otimes S_n(A)\ar[r]^{\cap} & S_q(A)\ar[d]\\
	S^p(X)\otimes S_n(X)\ar[rr]^\cap\ar[d] & & S_q(X)\ar[d]\\
	S^p(X)\otimes S_n(X,A)\ar[d]\ar@{-->}[rr] & & S_q(X,A)\ar[d]\\
	0 & & 0
}
\end{equation*}
The left sequence is exact because $0\to S_n(A)\to S_n(X)\to S_n(X,A)\to 0$ splits and tensoring with $S^p(X)$ still leaves it exact. We have to check that this diagram commutes.

Let $\beta\otimes \sigma\in S^p(X)\otimes S_n(A)$. We then get:
\begin{equation*}
\beta\otimes\sigma\xrightarrow{i^\ast\otimes 1}i^\ast\beta\otimes\sigma\to i^\ast(\beta)\cap\sigma\xrightarrow{i_\ast}i_\ast(i^\ast(\beta)\cap\sigma)
\end{equation*}
And:
\begin{equation*}
\beta\otimes\sigma\xrightarrow{1\otimes i_\ast}\beta\otimes i_\ast\sigma\to \beta\cap i_\ast(\sigma)
\end{equation*}
So they're equal by the projection formula. Hence you get $\cap: H^p(X)\otimes H_n(X,A)\to H_q(X,A)$ that makes $ H_\ast(X,A)$ a $ H^\ast(X)$-module.
\subsection{A different perspective on excision}
Recall what excision is. We know that $ H_\ast(X-U,A-U)\cong H_\ast(X,A)$. There's another perspective on this. Suppose $K\subseteq U\subseteq X$ such that $\overline{K}\subseteq\mathrm{Int}(U)$. To simplify things, suppose $K$ is closed and $U$ is open. Let $A=X-K\supseteq X-U=V$. Then excision says that $ H_\ast(X-V,A-V)= H_\ast(X-(X-U),(X-K)-(X-U))\cong H_\ast(X,A)= H_\ast(X,X-K)$. There's a simpler expression: $ H_\ast(X-(X-U),(X-K)-(X-U))= H_\ast(U,U-K)$, so $ H_\ast(U,U-K)\cong H_\ast(X,X-K)$, i.e., it depends only on an open neighborhood of $K$. A question that we now have is: how does this depend on $ H_\ast(K)$? $ H^\ast(K)$? This is really what Poincar\'{e} duality wants to understand.
\begin{example}
We'll eventually be talking, for example, about $X=S^3$ and $K=\text{knot}$.
\end{example}
We want to understand $ H_\ast(X,X-K)$ better. We have a cap product $ H^p(X)\otimes H_n(X,X-K)\to H_q(X,X-K)$. We just decided that $ H_n(X,X-K)\cong H_n(U,U-K)$, and so I have the cap product $ H^p(U)\otimes H_n(U,U-K)\to H_q(U-U-K)$. Hence I get the cap product map like $ H^p(U)\otimes H_n(X,X-K)\to H_q(X,X-K)$. But this seems to depend upon a choice of $U$. What if I make $U$ smaller?
\begin{lemma}
Let $U\supseteq V\supseteq K$. Then:
\begin{equation*}
\xymatrix{
	 H^p(U)\ar[dd]^{i^\ast\otimes 1}\otimes H_n(X,X-K)\ar[dr]^\cap & \\
	 & H_q(X,X-K)\\
	 H^p(V)\otimes H_n(X,X-K)\ar[ur]^\cap
}
\end{equation*}
\end{lemma}
\begin{proof}
Hint: use projection formula again.
\end{proof}
Let $\mathcal{U}_K$ be the set of open neighborhoods of $K$ in $X$. This is a poset (actually a directed set because you can take intersections), under reverse-inclusion as the ordering. This lemma says that $ H^p:\mathcal{U}_K\to\mathbf{Ab}$.
\begin{definition}
$\cHH^p(K):=\varinjlim_{U\in\mathcal{U}_K} H^p(U)$.
\end{definition}
This is bad notation because it depends on the way $K$ is sitting in $X$.

You therefore get $\cHH^p(K)\otimes H_n(X,X-K)\xrightarrow{\cap} H_q(X,X-K)$. This is the best you can do. It's the natural structure that this relative homology has, i.e., $ H_\ast(X,X-K)$ is a module over $\cHH^\ast(K)$.

Sometimes, $\cHH^\ast(K)$ will just be $ H^\ast(K)$. Suppose $K\subseteq X$ satisfies the condition (called the ``regularity'' condition) that for every open $U\supseteq K$, there exists an open $V$ such that $U\supseteq V\supseteq K$ such that $K\to V$ is a homotopy equivalence (or actually just a homology isomorphism). (for example, a smooth knot in $S^3$) Then:
\begin{lemma}
Suppose $\cI$ is a directed set (nonempty). Let $F:\cI\to\mathbf{Ab}$, and suppose I have a natural transformation $\theta:F\to c_A$ (for example, a map from $F$ to its direct limit). This expresses $A$ as $\varinjlim_\cI F$ provided that for all $i$, there is $j\geq i$ such that $F(i)\to A$ factors through $F(j)\to A$, which should be an isomorphism, i.e.:
\begin{equation*}
\xymatrix{
	F(i)\ar[dr]\ar[rr] & & A\\
	& F(j)\ar[ur]^\cong
}
\end{equation*}
\end{lemma}
\begin{proof}
Given $a\in A$, it has to come from somewhere to be a direct limit. This is obviously true. Also, for any $i$ and $a_i\in F(i)$ such that $a_i\mapsto 0$, then there exists $j\geq i$ such that $a_i\mapsto 0\in F(j)$. This is also obvious.
\end{proof}
\begin{remark}
This is a really strong condition by the way. It is a really stupid way.
\end{remark}
This works in the case that $K$ is regular in $X$. Thus, under this condition, $\cHH^p(K)\cong H^p(K)$. One other comment is that more generally, if $X$ is an Euclidean neighborhood retract (a retract of a neighborhood in some $\RR^n$), and $K$ is locally compact, then $\cHH^p(K)$ depends only on $K$, and it is isomorphic to \v{C}ech cohomology (which is a different type of cohomology theory).


\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

% Main maths packages
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathrsfs}

\begin{document}

\section*{Matrices}

\subsection*{Basic matrix}

\[
\begin{matrix}
	a_{11} & \cdots & a_{1k}\\
	\vdots & \ddots & \vdots\\
	a_{k1} & \cdots & a_{kk}
\end{matrix}
\]

\subsection*{Matrix with braces}

\[
\begin{pmatrix}
	a_{11} & \cdots & a_{1k}\\
	\vdots & \ddots & \vdots\\
	a_{k1} & \cdots & a_{kk}
\end{pmatrix}
\]

A lot of environments exist with different brackets.

\[
\begin{vmatrix}
	a_{11} & \cdots & a_{1k}\\
	\vdots & \ddots & \vdots\\
	a_{k1} & \cdots & a_{kk}
\end{vmatrix}
\]

\[
\begin{Vmatrix}
	a_{11} & \cdots & a_{1k}\\
	\vdots & \ddots & \vdots\\
	a_{k1} & \cdots & a_{kk}
\end{Vmatrix}
\]

\[
\begin{bmatrix}
	a_{11} & \cdots & a_{1k}\\
	\vdots & \ddots & \vdots\\
	a_{k1} & \cdots & a_{kk}
\end{bmatrix}
\]

\[
\begin{Bmatrix}
	a_{11} & \cdots & a_{1k}\\
	\vdots & \ddots & \vdots\\
	a_{k1} & \cdots & a_{kk}
\end{Bmatrix}
\]

\end{document}


\documentclass{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{datetime}
\usepackage{lipsum}
\graphicspath{{images/}}

\setlength\parindent{0pt}	% no indent for entire document

\title{Backpropagation for FC Neural Network}

\newcommand\blfootnote[1]
{%
  \begingroup
  \renewcommand\thefootnote{}\footnote{#1}%
  \addtocounter{footnote}{-1}%
  \endgroup
}

\begin{document}

\begin{titlepage}
\author{Hao Huang}
\date{\today}
\maketitle
\end{titlepage}

\tableofcontents
\newpage

\section{Network Structure}
\begin{center}
\includegraphics[scale=0.3]{neuralnetwork}
\end{center}

\blfootnote{Credit and Copyright:}
\blfootnote{http://blog.csdn.net/sysstc/article/details/75305276}
\blfootnote{http://blog.csdn.net/sysstc/article/details/75269563}

\section{Notations}
\subsection{Output of activation}
\begin{center}
\includegraphics[scale=0.3]{activation_output}
\end{center}
Output of a neuron: $a^l_i$ (Superscript $l$ is the layer, and subscript is the index in layer)

Output of a layer: $a^l$ (a vector)

\subsection{Weights}
\begin{center}
\includegraphics[scale=0.4]{weights}
\end{center}
$w^l_{ij}$: from neuron $j$ (layer $l-1$) to neuron $i$ (layer $l$)

\[
W^l
=
\begin{bmatrix}
    w^l_{11} & w^l_{12} & \dots & w^l_{1j} & \dots  & w^l_{1N_{l-1}} \\
    w^l_{21} & w^l_{22} & \dots & w^l_{2j} & \dots  & w^l_{2N_{l-1}} \\
    \vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\
    w^l_{N_l1} & w^l_{N_l2} & \dots & w^l_{N_lj} & \dots & w^l_{N_lN_{l-1}}
\end{bmatrix}_{N_l\times N_{l-1}}
\]

\subsection{Biases}
\begin{center}
\includegraphics[scale=0.4]{biases}
\end{center}
$b^l_i$: bias for neuron $i$ at layer $l$
\[
b^l
=
\begin{bmatrix}
    b^l_{1} & b^l_{2} & \dots & b^l_{i} & \dots  & b^l_{N_l}
\end{bmatrix}^T
\]

\subsection{Input of activation}
\begin{center}
\includegraphics[scale=0.4]{activation_input}
\end{center}
$z^l_i$: input of the activation function of neuron $i$ at layer $l$

$z^l$: input of the activation function of all neurons at layer $l$

\[
z^l_i = \sum_j^{N_{l-1}} w^l_{ij}a^{l-1}_j + b^l_i
\]

\[
z^l =
\begin{bmatrix}
    z^l_{1} & z^l_{2} & \dots & z^l_{i} & \dots  & z^l_{N_l}
\end{bmatrix}^T
\]

\subsection{Relations between layers}
\begin{center}
\includegraphics[scale=0.4]{layer_1}
\end{center}
\[
z^l_1 = w^l_{11}a^{l-1}_1 + w^l_{12}a^{l-1}_2 + \dots + w^l_{1j}a^{l-1}_j + \dots + w^l_{1N_{l-1}}a^{l-1}_{N_{l-1}} + b^l_1
\]
\[
z^l_2 = w^l_{21}a^{l-1}_1 + w^l_{22}a^{l-1}_2 + \dots + w^l_{2j}a^{l-1}_j + \dots + w^l_{2N_{l-1}}a^{l-1}_{N_{l-1}} + b^l_2
\]
\[
\vdots
\]
\[
z^l_i = w^l_{i1}a^{l-1}_1 + w^l_{i2}a^{l-1}_2 + \dots + w^l_{ij}a^{l-1}_j + \dots + w^l_{iN_{l-1}}a^{l-1}_{N_{l-1}} + b^l_i
\]
\[
\vdots
\]
\[
z^l_{N_l} = w^l_{N_l1}a^{l-1}_1 + w^l_{N_l2}a^{l-1}_2 + \dots + w^l_{N_lj}a^{l-1}_j + \dots + w^l_{N_lN_{l-1}}a^{l-1}_{N_{l-1}} + b^l_{N_l}
\]
Rewrite them as matrix multiplication form:
\[
\begin{bmatrix}
z^l_{1} \\ z^l_{2} \\ \vdots \\ z^l_{i} \\ \vdots  \\ z^l_{N_l}
\end{bmatrix}
=
\begin{bmatrix}
    w^l_{11} & w^l_{12} & \dots & w^l_{1j} & \dots  & w^l_{1N_{l-1}} \\
    w^l_{21} & w^l_{22} & \dots & w^l_{2j} & \dots  & w^l_{2N_{l-1}} \\
    \vdots & \vdots & \ddots & \vdots & \ddots & \vdots \\
    w^l_{i1} & w^l_{i2} & \dots & w^l_{ij} & \dots  & w^l_{iN_{l-1}} \\
    \vdots & \vdots & \ddots & \vdots & \ddots & \vdots \\
    w^l_{N_l1} & w^l_{N_l2} & \dots & w^l_{N_lj} & \dots & w^l_{N_lN_{l-1}}
\end{bmatrix}
\begin{bmatrix}
a^{l-1}_{1} \\ a^{l-1}_{2} \\ \vdots \\ a^{l-1}_{j} \\ \vdots  \\ a^{l-1}_{N_{l-1}}
\end{bmatrix}
+
\begin{bmatrix}
b^l_{1} \\ b^l_{2} \\ \vdots \\ b^l_{i} \\ \vdots  \\ b^l_{N_l}
\end{bmatrix}
\]
That is:
\[
z^l = W^l a^{l-1} + b^l
\]

\begin{center}
\includegraphics[scale=0.4]{layer_2}
\end{center}
For neuron $i$ in layer $l$:
\[
a^l_i = \sigma(z^l_i)
\]
For the layer $l$:
\[
\begin{bmatrix}
a^l_{1} \\ a^l_{2} \\ \vdots \\ a^l_{i} \\ \vdots  \\ a^l_{N_l}
\end{bmatrix}
=
\begin{bmatrix}
\sigma(z^l_1) \\ \sigma(z^l_2) \\ \vdots \\ \sigma(z^l_i) \\ \vdots  \\ \sigma(z^l_{N_l})
\end{bmatrix}
\]
Rewrite it as:
\[
a^l = \sigma(z^l)
\]

\begin{center}
\includegraphics[scale=0.4]{layer_3}
\end{center}
\[
\begin{rcases*}
a^l = \sigma(z^l)\\
z^l = W^l a^{l-1} + b^l
\end{rcases*} \Rightarrow a^l=\sigma(W^la^{l-1}+b^l)
\]

\subsection{Summary of notation}
\begin{center}
\includegraphics[scale=0.4]{notation_1}
\end{center}
\[
a^1=\sigma(W^1x+b^1)
\]
\[
a^2=\sigma(W^2a^1+b^2)
\]
\[
\vdots
\]
\[
y=a^L=\sigma(W^{L}a^{L-1}+b^L)
\]
Consider a neural network as a function:
\[
y=f(x)=\sigma(W^L...\sigma(W^2\sigma(W^1x+b^1)+b^2)...+b^L)
\]

\section{Three gradient descent approaches}
$\theta$ is the parameter set of a neural network (e.g. $\{W^1, b^1, ..., W^L, b^L\}$), and $C$ is the cost function. The superscript $i$ indicates the $i$-th update. $N$ is the number of training samples.
\subsection{Gradient descent}
\[
\theta^i = \theta^{i-1} - \eta\nabla C(\theta^{i-1})
\]
\[
\nabla C(\theta^{i-1}) = \frac{1}{N}\sum_{r = 1}^N\nabla C _r(\theta^{i-1})
\]
\subsection{Stochastic gradient descent}
Randomly pick a training sample $x_r$:
\[
\theta^i = \theta^{i-1} - \eta \nabla C_r(\theta^{i-1})
\]
\subsection{Mini-batch gradient descent}
Shuffle the training data and then pick $M$ samples as a batch:
\[
\theta^i = \theta^{i-1} - \eta \frac{1}{M}\sum_{x_r\in batch}\nabla C_r(\theta^{i-1})
\]

\section{Backpropagation}
The change in $w^l_{ij}$ leads to the change in $C_r$, so we need to compute $\nabla C_r$ with respect to $w^l_{ij}$.

\subsection{Partial derivative of $C_r$ w.r.t $w^l_{ij}$}
\begin{center}
\includegraphics[scale=0.4]{grad_ctow}
\end{center}
\[
\Delta w^l_{ij}\rightarrow \Delta z^l_i\rightarrow \Delta a^l_i \rightarrow \dots \rightarrow \Delta C_r
\]
\[
\frac{\partial C_r}{\partial w^l_{ij}} = \frac{\partial C_r}{\partial z^l_i}\frac{\partial z^l_i}{\partial w^l_{ij}}
\]

\subsection{Partial derivative of $z^l_i$ w.r.t $w^l_{ij}$}
We divide this partial derivative into two parts: input layer and hidden layer(s).

\subsubsection{Input layer}
Suppose the input layer has $N_0$ input nodes. The superscript $r$ of $x$ means the $r$-th sample.
\begin{center}
\includegraphics[scale=0.5]{grad_ztow_input}
\end{center}
\[
z^1_i = \sum_j^{N_0}w^1_{ij}x^r_j + b^1_i
\]
\[
\frac{\partial z^1_i}{\partial w^1_{ij}} = x^r_j
\]

\subsubsection{Hidden layer}
\begin{center}
\includegraphics[scale=0.5]{grad_ztow_hidden}
\end{center}
\[
z^l_i = \sum_j^{N_{l-1}}w^l_{ij}a^{l-1}_j + b^l_i
\]
\[
\frac{\partial z^l_i}{\partial w^l_{ij}} = a^{l-1}_j
\]

\subsection{Partial derivative of $C_r$ w.r.t $z^l_{i}$}
\[
\frac{\partial C_r}{\partial w^l_{ij}} = \frac{\partial C_r}{\partial z^l_i}\frac{\partial z^l_i}{\partial w^l_{ij}}
\]
We denote $\frac{\partial C_r}{\partial z^l_i}$ as $\delta^l_i$ and define $\delta^l = [\delta^l_1 \: \delta^l_2 \: \dots \delta^l_i \: \dots \: \delta^l_{N_l}]^T$.

\begin{center}
\includegraphics[scale=0.4]{delta}
\end{center}

We need to compute $\delta^l$. However, computing $\delta^l$ can be divided into two steps:
\begin{enumerate}
\item
computing $\delta^L$ (output layer)
\item
figuring out the relationship between $\delta^{l+1}$ and $\delta^l$
\end{enumerate}

\subsubsection{Computing $\delta^L$ (output layer)}
\begin{center}
\includegraphics[scale=0.5]{grad_last}
\end{center}
\[
\Delta z^L_n\rightarrow \Delta a^L_n = \Delta y^r_n \rightarrow \Delta C_r
\]
\[
\delta^L_n = \frac{\partial C_r}{\partial z^L_n} = \frac{\partial C_r}{\partial y^r_n}\frac{\partial y^r_n}{\partial z^L_n} = \frac{\partial C_r}{\partial y^r_n}\sigma^\prime(z^L_n)
\]
In addition, $\frac{\partial C_r}{\partial y^r_n}$ depends on the definition of the cost function.
\[
\delta^L
=
\begin{bmatrix}
\delta^L_1 \\ \delta^L_2 \\ \vdots \\ \delta^L_n \\ \vdots  \\ \delta^L_{N_L})
\end{bmatrix}
=
\begin{bmatrix}
(\partial C_r \mathbin{/} \partial y^r_1)\times\sigma^\prime(z^L_1) \\
(\partial C_r \mathbin{/} \partial y^r_2)\times\sigma^\prime(z^L_2) \\
\vdots \\
(\partial C_r \mathbin{/} \partial y^r_n)\times\sigma^\prime(z^L_n) \\
\vdots \\
(\partial C_r \mathbin{/} \partial y^r_{N_L})\times\sigma^\prime(z^L_{N_L})
\end{bmatrix}
=
\begin{bmatrix}
(\partial C_r \mathbin{/} \partial y^r_1) \\
(\partial C_r \mathbin{/} \partial y^r_2) \\
\vdots \\
(\partial C_r \mathbin{/} \partial y^r_n) \\
\vdots \\
(\partial C_r \mathbin{/} \partial y^r_{N_L})
\end{bmatrix}
\odot
\begin{bmatrix}
\sigma^\prime(z^L_1) \\ \sigma^\prime(z^L_2) \\ \vdots \\
\sigma^\prime(z^L_n) \\ \vdots \\ \sigma^\prime(z^L_{N_L})
\end{bmatrix}
\]
$\odot$ is element-wise multiplication. We rewrite this formula is a vector form:
\[
\delta^L = \nabla C_r(y^r) \odot \sigma^\prime(z^L)
\]

\subsubsection{Relationship between $\delta^{l+1}$ and $\delta^l$}
\begin{center}
\includegraphics[scale=0.6]{grad_delta}
\end{center}
Because the output of $i$-th node in layer $l$ (e.g., $a^l_i$) will be passed in to every node in layer $l+1$, so we have:
\[
\Delta z^l_i \rightarrow \Delta a^l_i \rightarrow
\begin{dcases*}
\begin{rcases*}
\Delta z^{l+1}_1 \\ \Delta z^{l+1}_2 \\ \vdots \\ \Delta z^{l+1}_k \\ \vdots \\ \Delta z^{l+1}_{N_{l+1}}
\end{rcases*}
\end{dcases*} \dashrightarrow \Delta C_r
\]
\[
\delta^l_i = \frac{\partial C_r}{\partial z^l_i} = \sum^{N_{l+1}}_k\frac{\partial C_r}{\partial z^{l+1}_k}\frac{\partial z^{l+1}_k}{\partial a^l_i}\frac{\partial a^l_i}{\partial z^l_i} = \frac{\partial a^l_i}{\partial z^l_i}\sum^{N_{l+1}}_k\frac{\partial C_r}{\partial z^{l+1}_k}\frac{\partial z^{l+1}_k}{\partial a^l_i} = \frac{\partial a^l_i}{\partial z^l_i}\sum^{N_{l+1}}_k\delta^{l+1}_k\frac{\partial z^{l+1}_k}{\partial a^l_i}
\]
\[
\because a^l_i = \sigma(z^l_i) \; \land \; z^{l+1}_k = \sum^{N_l}_i w^{l+1}_{ki}a^l_i + b^{l+1}_k
\]
\[
\therefore \delta^l_i = \sigma^\prime(z^l_i)\sum^{N_{l+1}}_k\delta^{l+1}_kw^{l+1}_{ki} = \sigma^\prime(z^l_i)\sum^{N_{l+1}}_kw^{l+1}_{ki}\delta^{l+1}_k
\]
Let us rewrite it in vector (and matrix) form:
\[
\sigma^\prime(z^l)
=
\begin{bmatrix}
\sigma^\prime(z^l_1) \\ \sigma^\prime(z^l_2) \\ \vdots \\
\sigma^\prime(z^l_i) \\ \vdots \\ \sigma^\prime(z^l_{N_l})
\end{bmatrix}
\; \land \;
\delta^{l+1}
=
\begin{bmatrix}
\delta^{l+1}_1 \\ \delta^{l+1}_2 \\ \vdots \\
\delta^{l+1}_k \\ \vdots \\ \delta^{l+1}_{N_{l+1}}
\end{bmatrix}
\]
\[
\delta^l
=
\begin{bmatrix}
\delta^l_1 \\ \delta^l_2 \\ \vdots \\
\delta^l_i \\ \vdots \\ \delta^l_{N_l}
\end{bmatrix}
=
\begin{bmatrix}
\sigma^\prime(z^l_1) \\ \sigma^\prime(z^l_2) \\ \vdots \\
\sigma^\prime(z^l_i) \\ \vdots \\ \sigma^\prime(z^l_{N_l})
\end{bmatrix}
\odot
\begin{pmatrix}
\begin{bmatrix}
w^{l+1}_{11} & w^{l+1}_{21} & \dots & w^{l+1}_{k1} & \dots & w^{l+1}_{N_{l+1}1}\\
w^{l+1}_{12} & w^{l+1}_{22} & \dots & w^{l+1}_{k2} & \dots & w^{l+1}_{N_{l+1}2}\\
\vdots & \vdots & \ddots & \vdots & \ddots & \vdots\\
w^{l+1}_{1i} & w^{l+1}_{2i} & \dots & w^{l+1}_{ki} & \dots & w^{l+1}_{N_{l+1}i}\\
\vdots & \vdots & \ddots & \vdots & \ddots & \vdots\\
w^{l+1}_{1N_l} & w^{l+1}_{2N_l} & \dots & w^{l+1}_{kN_l} & \dots & w^{l+1}_{N_{l+1}N_l}\\
\end{bmatrix}
\begin{bmatrix}
\delta^{l+1}_1 \\ \delta^{l+1}_2 \\ \vdots \\
\delta^{l+1}_k \\ \vdots \\ \delta^{l+1}_{N_{l+1}}
\end{bmatrix}
\end{pmatrix}
\]
That is:
\[
\delta^l = \sigma^\prime(z^l)\odot[(W^{l+1})^T\delta^{l+1}]
\]
Thus we get the relationship between $\delta^{l+1}$ and $\delta^l$.

\subsubsection{Another viewpoint}
From this formula: $\delta^l_i = \sigma^\prime(z^l_i)\sum^{N_{l+1}}_kw^{l+1}_{ki}\delta^{l+1}_k$, we can view it as a new type of neuron, where $\sigma^\prime(z^l_i)$ is an "amplifier".
\begin{center}
\includegraphics[scale=0.6]{new_neuron}
\end{center}
The relationship between the layer $l$ and layer $l+1$ (e.g., $\delta^l = \sigma^\prime(z^l)\odot[(W^{l+1})^T\delta^{l+1}]$) can be viewed as:
\begin{center}
\includegraphics[scale=0.6]{new_layer}
\end{center}
Now we compare the forward-pass network and the backward-pass network:
\begin{enumerate}
\item
Forward-pass network
\begin{center}
\includegraphics[scale=0.4]{forward}
\end{center}
\[
a^{l+1} = \sigma(W^{l+1}a^l+b^{l+1})
\]
\item
Backward-pass network
\begin{center}
\includegraphics[scale=0.4]{backward}
\end{center}
\[
\delta^l = \sigma^\prime(z^l)\odot[(W^{l+1})^T\delta^{l+1}]
\]
\end{enumerate}

\section{Summary}
Our goal is:
\[
\frac{\partial C_r}{\partial w^l_{ij}} = \frac{\partial C_r}{\partial z^l_i}\frac{\partial z^l_i}{\partial w^l_{ij}}
\]
\[
\frac{\partial z^l_i}{\partial w^l_{ij}}
=
\begin{cases}
x^r_j & \textrm{input layer}\\
a^{l-1}_j & \textrm{hidden layer}
\end{cases}
\]
To compute $\delta^l_i = \frac{\partial C_r}{\partial z^l_i}$, we divide it into two step:
\begin{enumerate}
\item
For output layer $L$:
\[
\delta^L = \nabla C_r(y^r)\odot \sigma^\prime(z^L)
\]
\item
For hidden layers $\delta^{l+1}$ and $\delta^l$:
\[
\delta^l = \sigma^\prime(z^l)\odot [(W^{l+1})^T\delta^{l+1}]
\]
\end{enumerate}
To visualize forward-pass and backward-pass, we have:
\begin{center}
\includegraphics[scale=0.35]{summary}
\end{center}
The black dot in the figure is equivalent to $\odot$ and means element-wise multiplication.

\end{document}


\chapter{Examples in Matlab}

We now want to find approximate numerical solutions using Fourier spectral methods. In this section we focus primarily on the heat equation with periodic boundary conditions for $x\in[0,2\pi)$. Many of the techniques used here will also work for more complicated partial differential equations for which separation of variables cannot be used directly.
%%%%%%%
%Section
%%%%%%%
\section{1D Heat Equation}
The 1D heat equation
\begin{eqnarray}\label{eq:heat}
\frac{\partial u}{\partial t} = \alpha \frac{\partial^2 u}{\partial x^2}
\end{eqnarray}
is a well known second order PDE for which exact series solutions can be found using separation of variables.  It arises in several contexts such as in predicting the temperature in a thin uniform cross section rod. The equation and its derivation can be found in introductory books on partial differential equations and calculus, for example \cite{BoyDip10}, \cite{CouJoh98} and \cite{HugEtAl08},   The constant $\alpha$ is the thermal diffusivity and $u(x,t)$ is temperature. We have already described how to solve the heat equation using separation of variables. Let us first discretize $x$ such that $x_j$ where $j=0,1,2,...,n$. $x_j$ are uniformly spteaced in $[0,2\pi)$. Let's now take the FFT of both sides of the 1D heat equation to obtain
\begin{eqnarray}
\widehat{\frac{\partial u}{\partial t}} = \alpha \widehat{\frac{\partial^2 u}{\partial x^2}}.
\end{eqnarray}
We then rewrite the spatial derivative using eq.\ \eqref{eq:Spectral_Method_Deriv} \footnote{The $k$ subscript denotes the coefficient of the k$^{th}$ Fourier mode.}
\begin{eqnarray}
\frac{\partial \hat{u}_k}{\partial t} = \alpha (ik)^2 \hat{u}_k,
\end{eqnarray}
so that the partial differential equation now becomes a collection of independent ODEs. While we can solve these ODEs in time exactly, we will use techniques that will also allow us to obtain approximate solutions to PDEs we cannot solve exactly. We will discuss two methods for solving these ODEs, forward Euler and backward Euler.

%subsection
\subsection{Forward Euler}
Using the forward Euler method in time, we obtain
\begin{eqnarray}
\frac{\hat{u}_k^{n+1}-\hat{u}_k^n}{h} = \alpha (ik)^2 \hat{u}_k^n \\
\hat{u}_k^{n+1} =\hat{u}_{k}^n+\alpha h(ik)^2\hat{u}_k^n
\end{eqnarray}
All that is left is to take the IFFT of the computed solution after all timesteps are taken to transfer it back to real space. This is a linear PDE, so only one IFFT is needed at the end. We will later see that this is different for a nonlinear PDE. A Matlab implementation of this is in listing \ref{lst:HeatFE1dMatlab}.

\lstinputlisting[style=matlab_style,label=lst:HeatFE1dMatlab,caption={A Matlab program to solve the heat equation using forward Euler timestepping.}]{./ExamplesInMatlab/Programs/Heat_Eq_1D_Spectral_FE.m}

 %subsection
\subsection{Backward Euler}
To derive this method, we start by applying the FFT and then perform timestepping using backward Euler. We then rewrite the implicit form into a form that gives the next iterate,
\begin{eqnarray}
\frac{\partial \hat{u}_k}{\partial t} &=& \alpha (ik)^2 \hat{u}_k \\
\frac{\hat{u}_k^{n+1}-\hat{u}_k^n}{h} &=& \alpha (ik)^2 \hat{u}_k^{n+1} \\
\hat{u}_k^{n+1}(1-\alpha h(ik)^2) &=&\hat{u}_k^n\\
\hat{u}_k^{n+1} &=&\frac{\hat{u}_k^n}{(1-\alpha h(ik)^2)}.
\end{eqnarray}
Below is a graph of the numerical solution to the heat equation\footnote{Methods to obtain the exact solution can be found in, among other places,  Boyce and DiPrima~\cite{BoyDip10}. } where $n=64$ obtained using the Matlab program in listing \ref{lst:HeatBE1dMatlab}.
\begin{figure}
 \begin{center}
\includegraphics[scale=.35]{./ExamplesInMatlab/Heat_Equation_1D_BE_Color}
\caption{A numerical solution to the heat equation, eq.\ \eqref{eq:heat} computed using the backward Euler method.} \label{fig:HeatBE1dMatlab}
\end{center}
\end{figure}

\lstinputlisting[style=matlab_style,label=lst:HeatBE1dMatlab,caption={A Matlab program to solve the heat equation using backward Euler timestepping.}]{./ExamplesInMatlab/Programs/Heat_Eq_1D_Spectral_BE.m}

%subsection
\subsection{Exercises}
\begin{enumerate}
\item[1)] Write a program to solve the heat equation using the Crank-Nicolson method.
\item[2)] Solve the advection equation $u_t=u_x$ for $x\in[0,2\pi)$ with the initial data
\begin{enumerate}
\item[a)] $u(t=0,x)=\cos(x)$
\item[b)] $u(t=0,x)=\left\{{0\quad x<\pi}\atop{1\quad x\geq\pi}\right.$
\end{enumerate}
up to a time $T=1$. You can do this either by using separation of variables or by assuming that the solution is of the form $u(x,t)=f(x+t)$ and deducing what $f$ is in order to satisfy the initial conditions. In both cases please use the forward Euler, backward Euler and Crank-Nicolson timestepping schemes. After calculating the exact solution in each of these cases, examine how the maximum error at the final time depends on the timestep for each of these three methods.
\end{enumerate}

%%%%%%%
%Section
%%%%%%%
\section{Nonlinear Equations}
%subsection
\subsection{The 1D Allen-Cahn Equation}
So far we have dealt only with linear equations. Now it's time for a nonlinear PDE. The \emph{Allen-Cahn equation} models the separation of phases in a material. It was introduced by Sam Allen and J. W. Cahn~\cite{AllCah79}  and is
\begin{eqnarray}
\frac{\partial u}{\partial t} = \epsilon\frac{\partial^2 u}{\partial x^2}+u-u^3, \label{eq:AllCah}
\end{eqnarray}
where $\epsilon$ is a small but positive constant.  The way to numerically solve this is similar to the method used for the heat equation, but there are some notable differences. The biggest difference is that FFT($u^3$)$\neq$FFT($u$)$^3$, so the $u^3$ must be computed before taking the FFT. The FFT is a linear operation but cubing is non-linear operation, so the order matters
\begin{eqnarray}
\frac{\partial \hat{u}_k}{\partial t} = \epsilon\frac{\partial^2 \hat{u}_k}{\partial x^2}+\hat{u}_k-\widehat{u^3}_k. \label{Allen_Cahn_1D}
\end{eqnarray}
Next rewrite the first term on the right hand side, just like we did in the heat equation
\begin{eqnarray}
\frac{\partial \hat{u}_k}{\partial t} = \epsilon(ik)^2\hat{u}_k+\hat{u}_k-\widehat{u^3}_k.
\end{eqnarray}
In order to solve this numerically we are going to use a combination of implicit (backward Euler) and explicit (forward Euler) methods. We are going to skip forward Euler because it is unstable.

%subsubsection
\subsubsection{Implicit-Explicit Method}
You might have already noticed that backward Euler is not going to work for the Allen-Cahn in its present state because of the nonlinear term. If you go to implement backward Euler you can see that you can't factor out all of the $\hat{u}_k^{n+1}$. Luckily there is a simple intuitive way around this that isn't detrimental to the accuracy of the solution. Write all the terms implicitly (backwards Euler) except for the nonlinear term which is expressed explicitly.  Applying this to Allen-Cahn we find that \footnote{Notice that when programming you are going to have to update the nonlinear term ($u^3$) each time you want to calculate the next timestep $n+1$. The reason this is worth mentioning is because for each timestep you are going to have to go from real space to Fourier space to real space, then repeat. For, the heat equation you can perform any number of timesteps in Fourier space and only convert back when you record your data.}
\begin{eqnarray}
\frac{\hat{u}_k^{n+1}-\hat{u}_k^n}{h} &=& \epsilon(ik)^2\hat{u}_k^{n+1}+\hat{u}_k^n-\widehat{(u^n)^3}_k \\
\hat{u}_k^{n+1}\left(-\epsilon(ik)^2+\frac{1}{h}\right)&=&\frac{1}{h}\hat{u}_k^n+\hat{u}_k^n-\widehat{(u^n)^3}_k \\
\hat{u}_k^{n+1}&=&\frac{\hat{u}_k^n(\frac{1}{h}+1)-\widehat{(u^n)^3}_k}{\left(-\epsilon(ik)^2+\frac{1}{h}\right)}.
\end{eqnarray}
Now we have a form that we can work with. We can set the initial conditions to be $u(x,0)=\frac{1}{4} \sin(x)$ and plot the computed space-time evolution calculated by the Matlab code in listing \ref{lst:AllenCahnIE1dMatlab}. The computed result is in Fig.\ \ref{fig:AllenCahnIE1dMatlab}.

\lstinputlisting[style=matlab_style,label=lst:AllenCahnIE1dMatlab,caption={A Matlab program to solve the 1D Allen-Cahn equation using implicit explicit timestepping.}]{./ExamplesInMatlab/Programs/Allen_Cahn_1D_Spectral_IE.m}

\begin{figure}
\begin{center}
\includegraphics[scale=.2]{./ExamplesInMatlab/Allen_Cahn_BE_1D}
\caption{A numerical solution to the 1D Allen-Cahn equation, eq.\ \eqref{eq:AllCah}, with $\epsilon=0.001$ and $u(x,t=0)=0.25\sin(x)$ computed using an implicit explicit method.} \label{fig:AllenCahnIE1dMatlab}
\end{center}
\end{figure}

%subsection
\subsection{The 2D Allen-Cahn Equation}
Now we will look at the 2D form of the Allen-Cahn Equation, where $u(x,y,t)$ satisfies
\begin{eqnarray}
\frac{\partial u}{\partial t} = \epsilon\left(\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial y^2}\right)+u-u^3. \label{eq:AllCah2D}
\end{eqnarray}
The convert it into Fourier space by taking the FFT of both sides
\begin{eqnarray}
\frac{\partial \hat{u}_k}{\partial t} &=& \epsilon\left(\frac{\partial^2 \hat{u}_k}{\partial x^2}+\frac{\partial^2 \hat{u}_k}{\partial y^2}\right)+\hat{u}_k-\widehat{u^3}_k\\
\frac{\partial \hat{u}_k}{\partial t} &=& \epsilon\left((ik_x)^2\hat{u}_k+(ik_y)^2\hat{u}_k\right)+\hat{u}_k-\widehat{(u^3)}_k \label{eq:Allen_Cahn_2D}
\end{eqnarray}
where $k_x$ and $k_y$ is to remind us that we take the FFT in respected directions. We will also define
\begin{eqnarray}
f(u) \equiv u-u^3 \label{eq:nonlinear}
\end{eqnarray}
The way to deal with the first two terms on the right hand side is to take the FFT in the x-direction and then take it in the y-direction. The order in which the FFT is done, $x$ first or $y$ first is not important. Some software libraries offer a two dimensional FFT. It usually depends on the equation being solved whether it is more efficient to use a multidimensional FFT or many one dimensional FFTs. Typically, it is easier to write a program which uses a multidimensional FFT, but in some situations this is not very efficient since one can immediately reuse data that has just been Fourier transformed.
%subsubsection
\subsubsection{Implicit-Explicit Method}
In this method, the nonlinear term in eq.\ \eqref{eq:nonlinear} is calculated explicitly, while the rest of the terms will be written implicitly such that
\begin{align}
&{} \frac{\hat{u}_k^{n+1}-\hat{u}_k^n}{h} = \epsilon\left((ik_x)^2\hat{u}_k^{n+1}+(ik_y)^2\hat{u}_k^{n+1}\right)+\widehat{f(u^n)}_k
\\&{} \hat{u}_k^{n+1}\left(-\epsilon(ik_x)^2-\epsilon(ik_y)^2+\frac{1}{h}\right)=\frac{\hat{u}_k^n}{h}+\widehat{f(u^n)}_k
\\&{} \hat{u}_k^{n+1} = \frac{\frac{\hat{u}_k^n}{h}+\widehat{f(u^n)}_k}{ \left(-\epsilon(ik_x)^2-\epsilon(ik_y)^2+\frac{1}{h}\right)}
\end{align}
we can then substitute in for $f(u)$
\begin{align}
\hat{u}_k^{n+1} = \frac{\hat{u}_k^n\left(\frac{1}{h}+1\right)-\widehat{(u^n)^3}_k}{ \left(-\epsilon(ik_x)^2-\epsilon(ik_y)^2+\frac{1}{h}\right)}. \label{eq:Allen_Cahn_IE_2D}
\end{align}
\begin{figure}
\begin{center}
\includegraphics[scale=.2]{./ExamplesInMatlab/Allen_Cahn_2D_IE}
\caption{A numerical solution to the 2D Allen-Cahn equation, eq.\ \eqref{eq:AllCah2D} at time $t=500$ with $\epsilon=0.1$ and $u(x,y,t=0)=\sin(2\pi x)+0.001\cos(16\pi x)$ computed using an implicit explicit method.} \label{fig:AllenCahnIE2dMatlab}
\end{center}
\end{figure}
The Matlab code used to generate Fig.\ \ref{fig:AllenCahnIE2dMatlab} is in listing \ref{lst:AllenCahnIE2dMatlab}.

\lstinputlisting[style=matlab_style,label=lst:AllenCahnIE2dMatlab,caption={A Matlab program to solve the 2D Allen-Cahn equation using implicit explicit timestepping.}]{./ExamplesInMatlab/Programs/Allen_Cahn_2D_Spectral_IE.m}

%subsection
\subsection{Exercises}

Many of these exercises are taken from Uecker~\cite{Uec09}. Another introductory source of information on these equations is Trefethen and Embree~\cite{TreEmb01}.

\begin{enumerate}
\item[1)] Burgers equation is given by:
$$\frac{\partial u}{\partial t}=\nu \frac{\partial^2 u}{\partial x^2} - u\frac{\partial u}{\partial x}$$
where $\nu\in\mathbb{R}^+$ and $u$ has periodic boundary conditions. Solve this equation using an implicit-explicit method. If you take $\nu$ to be small, ensure that a sufficient number of grid points are used to get the correct numerical solution. A simple way to check this is to keep increasing the number of grid points and checking that there is no change in the solution. Another way to check this is to calculate the Fourier coefficients and check that the highest ones decay to machine precision.
\item[2)] The Kuramoto-Sivashinsky equation is given by:
$$\frac{\partial u}{\partial t}=-\frac{\partial^2 u}{\partial x^2}-\frac{\partial^4 u}{\partial x^4}-u\frac{\partial u}{\partial x}$$
where $u$ has periodic boundary conditions.
\begin{itemize}
\item[a)] What does this equation model and what type of behavior do you expect its solutions to have?
\item[b)] Find numerical solutions to this equation using an implicit-explicit method.
\end{itemize}
\item[3)] The 1D Gray-Scott equations are given by:
$$\frac{\partial u}{\partial t}=d_1\frac{\partial^2 u}{\partial x^2}-uv^2+f(1-u),$$
$$\frac{\partial v}{\partial t}=d_2\frac{\partial^2 v}{\partial x^2} + uv^2 - (f+k)v$$
where $d_1$, $d_2$, $f$ and $k$ are constants.
\begin{itemize}
\item[a)] What does this equation model and what type of behavior do you expect its solutions to have?
\item[b)] Find numerical solutions to this equation using an implicit-explicit method. Try several different values of $d_1$, $d_2$, $f$ and $k$ and compare the resulting patterns to what you can find in the literature.
\end{itemize}
\item[4)] The 2D Swift-Hohenberg equation is given by:
$$\frac{\partial u}{\partial t}=-\Delta^2u+2\Delta u+ (\alpha-1)u-u^3,$$
\begin{itemize}
\item[a)] What does this equation model and what type of behavior do you expect its solutions to have?
\item[b)] Find numerical solutions to this equation using an implicit-explicit method for several values of $\alpha$.
\end{itemize}
\item[5)] The 2D Gray-Scott equations are given by:
$$\frac{\partial u}{\partial t}=d_1\Delta u -uv^2+f(1-u)$$
$$\frac{\partial v}{\partial t}=d_2\Delta v  + uv^2 - (f+k)v$$
where $d_1$, $d_2$, $f$ and $k$ are constants.
\begin{itemize}
\item[a)] What does this equation model and what type of behavior do you expect its solutions to have?
\item[b)] Find numerical solutions to this equation using an implicit-explicit method.
\end{itemize}
\item[6)] The 2D Complex Ginzburg-Landau equation is given by:
$$\frac{\partial A}{\partial t}=A+(1+i\alpha)\Delta A- (1+i\beta)|A|^2A.$$
An introductory tutorial to this equation can be found at \url{http://codeinthehole.com/static/tutorial/index.html}
\begin{itemize}
\item[a)] What does this equation model and what type of behavior do you expect its solutions to have?
\item[b)] Find numerical solutions to this equation using an implicit-explicit method for several values of $\alpha$ and $\beta$.
\end{itemize}
\end{enumerate}