\chapter{Linear equations}

\section{Introduction}

The numerical solution of systems of linear equations can be a rather
taxing problem.    Solving a system like
%
\begin{align*}
  \left\{
    \begin{aligned}
      2 x + 3 y & = 7, \\
      x - y & = 1,
    \end{aligned} \right.
\end{align*}
%
is easy.  Subtract twice the second equation from the first and obtain
$y=1$ and $x=2$.  This procedure is fine  for a small system of linear
equations.  However,  there are many  numerical  problems that require
the solution of linear systems with many equations.  For example, when
integrating numerically a Partial Differential  Equation it is typical
to have to solve a $10000 \times 10000$ system of linear equations.

There are other, less obvious, snags in solving a linear system of
equations.     Consider the linear system
%
\begin{align*}
  \left\{
    \begin{aligned}
      x + y & = 1, \\
      2 x + 2 y & = 3.
    \end{aligned} \right.
\end{align*}
%
The second equation is incompatible with the first one and, therefore,
the system has no solution.    However, if the coefficients stored in
the computer are ever so slightly altered, for example to
%
\begin{align*}
  \left\{
    \begin{aligned}
      0.999999 \, x + y & = 0.999999, \\
      2 x + 2 y &= 2.9999999 ,
    \end{aligned} \right.
\end{align*}
%
then the system will have a solution.    This is an example of an
ill-conditioned system: a very small change in the matrix of the
coefficients induces an enormous change in the solutions (from
non-existence to existence in this case).

From these two examples, we can see the problems that have to
be solved in   devising  algorithms  for  solving systems  of   linear
equations.  The algorithms  should be fast, so   that huge systems  of
equations can be solved in a reasonable length of time and they should
be accurate, i.e.\ they  should not introduce approximation errors that
grow out of  bounds.  Moreover, if a linear  system cannot be  solved
accurately (second example), we must be  able to find functions of the
matrix coefficients that  can  be used as   an health warning  against
putting too much trust in the solutions.

If we want to know how close the  numerical solution is to the correct
solution we  must    first define  the   concept  of  ``length''   and
``distance'' for vectors  and matrices.  Once this is  done we will be
able to find appropriate functions of  the matrix coefficients that we
can use to  check the accuracy of  the numerical results.   Only after
this ground work has been  carried out we will be  able to discuss how
to solve a linear system of equations.

\section{Some definitions}

\subsection{Introduction}

Unless otherwise stated, we plan to solve
%
\begin{equation*}
  A \bx = \bb ,
\end{equation*}
%
where $A$ is a $n \times n$ real  matrix, $\bx = (x_1, \ldots, x_n)^T$
is the vector of unknowns and $\bb = (b_1, \ldots,  b_n)^T$ is a given
real vector.  Moreover,  we indicate with $I$  the identity matrix and
with $0$ the zero matrix.

The basic theory of systems of linear equations states that the
equations $A \bx = \bb $ have a unique solution if and only if
$\det(A) \ne 0$, that solution being $\bx = A^{-1} \bb$.  For the case
$\det(A) = 0$, the equations either have no solutions (inconsistent
set) or an infinite number of solutions (undetermined set).

\smallskip

\noindent {\bf Remark} - Numerically, finding $\bx$ using $\bx =
A^{-1} \bb$ is invariably bad: there are faster and more accurate
methods to solve a linear system of equations.

\subsection{Vector and matrix norms}

\subsubsection{Vector norms}

Let   $\bRn$ be  the   set of all    $n$-dimensional vectors with real
components.   A \textit{norm}  on $\bRn$ is  a function  $\norm{\cdot}$
which assigns a real value to each vector in $\bRn$ (the ``length'' of
the vector) and satisfies
%
\begin{enumerate}
\item $\norm{\bx} \ge 0 \, \, \forall \bx \in \bRn$, with $\norm{\bx} = 0$
if and only if $\bx = 0$.
\item $\norm{\alpha \bx} = | \alpha| \norm{\bx} \, \, \forall \bx \in \bRn,
\, \forall \alpha \in \bR$.
\item $\norm{\bx + \by} \le \norm{\bx} + \norm{\by}, \, \, \forall \bx, \by
\in \bRn$.
\end{enumerate}

Some common norms are:
%
\begin{center}
  \begin{tabular}{ll@{\hspace{5mm}}l}
    $\displaystyle \norm{\bx}_1 \equiv \sum_{j=1}^n |x_j|$ ,
    & & $\displaystyle \norm{\bx}_2 \equiv \left [ \sum_{j=1}^n (x_j)^2 \right ]^{1/2}$, \\
    $\displaystyle \norm{\bx}_p \equiv \left [ \sum_{j=1}^n (x_j)^p \right ]^{1/p}
    \, p>0$,
    & & $\displaystyle \norm{\bx}_\infty = \max_j |x_j|$ .
  \end{tabular}
\end{center}

\noindent {\bf Remark} - $\norm{\cdot}_2$ is called the
\textit{Euclidean norm}: it is the generalisation to $n$-dimensional
spaces of the distance between two points in the coordinate plane.

\subsubsection{Matrix norms}

Let $M_n$ denote the set of all real $n \times n$ matrices.   A
\textit{norm} on $M_n$ is a function $\norm{ \cdot }$ which assigns a
real value to each matrix in $M_n$ and that satisfies
%
\begin{enumerate}
\item $\norm{ A } \ge 0 \, \, \, \forall A \in M_n$.  Moreover $\norm{ A } =
  0$ if and only if $A = 0$.
\item $\norm{ \alpha A } = | \alpha | \norm{ A } \, \, \, \forall A \in M_n
  \, \, \, \forall \alpha \in \bR$.
\item $\norm{A + B} \le \norm{A} + \norm{ B } \, \, \, \forall A,B \in
  M_n$.
\item $ \norm{ A B } \le \norm{ A } \norm{ B } $
\end{enumerate}

\smallskip

Vector norms can be used to define a family of matrix norms, called
\textit{induced norms}: these are the only ones used in this unit. The norm
of a vector $\norm{ \bx }$ measures its length; therefore, $\norm{ A \bx }$
is the length of the vector $\bx$ transformed by the matrix $A$.  We
can define a matrix norm as the maximum relative ``stretching''
induced by the matrix $A$.  More formally, if $\norm{ \cdot }$ is a norm
defined in $\bRn$ then we a define a function (norm) $\norm{ \cdot }$ on
the matrix space $M_n$ as
%
\begin{equation*}
\norm{ A } \equiv \max_{\norm{ \bx } = 1} \norm{ A \bx } .
\end{equation*}

\smallskip

These norms are said to be \textit{induced} by the corresponding
vector norm and we use the same symbol for the matrix and the vector
norm.   For example,
%
\begin{align*}
  \norm{ A }_1 & \equiv \max_{\norm{ \bx }_1 = 1} \norm{ A \bx }_1 , \\
  \norm{ A }_p & \equiv \max_{\norm{ \bx }_p = 1} \norm{ A \bx }_p , \\
  \norm{ A }_\infty & \equiv \max_{\norm{ \bx }_\infty = 1} \norm{ A \bx }_\infty .
\end{align*}
%
\smallskip

\noindent {\bf Remark 1} - The vector and matrix norms are compatible:
%
\begin{equation*}
  \norm{ A \bx }_p \le \norm{ A }_p \norm{ \bx }_p .
\end{equation*}
%
However, we cannot mix norms.  For example, it is \textbf{not} true in
general that $\norm{ A \bx }_1 \le \norm{ A }_2 \norm{ \bx }_2$.

\smallskip

\noindent {\bf Remark 2} - The value of the matrix 1-norm is the
maximum of the 1-norms of the column vectors of the matrix:
%
\begin{equation*}
 \norm{ A }_1 = \max_{1 \le k \le n} \sum_{j=1}^n | a_{j k} | .
\end{equation*}
%

\smallskip

\noindent {\bf Remark 3} - The value of the matrix infinity-norm is
the maximum of the 1-norms of the row vectors of the matrix:
%
\begin{equation*}
  \norm{ A }_\infty = \max_{1 \le k \le n} \sum_{j=1}^n | a_{k j} | .
\end{equation*}

\subsection{The condition number}

We  now  have the   tools  to assess  the reliability  of  a numerical
solution of  a system of linear equations.    Given a system  $A \bx =
\bb$ call $\bx_c$ the computed solution and $\bx_t$ the true solution.
Define an additional vector $\br$ (\textit{residual}) as,
%
\begin{equation*}
 \br = A \bx_c - \bb .
\end{equation*}
%
Any numerical    method that attempts  to solve   the linear system of
equations tries to  minimise the  vector $\br$: if  $\br =  0$ the
problem is solved exactly.  We can rewrite $\br$ as
%
\begin{equation*}
  \br = A \bx_c - A \bx_t \implies \bx_c - \bx_t = A^{-1} \br .
\end{equation*}
%
We can  see from this expression  that if $A^{-1}$  is ``ill behaved''
then  even   though $\br$  is very  small   the difference between the
computed and the true solution  can be very  large.  The
\textit{condition number of the matrix}, $K(A)$, is the
quantity
%
\begin{equation*}
  K(A) = \norm{ A } \, \norm{ A^{-1} },
\end{equation*}
%
and we call ${\cal E}$ the \textit{weighted residual},
%
\begin{equation*}
  {\cal E} = \frac{\norm{ \br }}{\norm{ \bb }} =
  \frac{\norm{ A \bx_c - \bb }}{\norm{ \bb }} .
\end{equation*}
%
One can show that
%
\begin{equation}
 \frac{1}{K(A)} {\cal E} \le
      \frac{ \norm{ \bx_c - \bx_t }}{\norm{ \bx_t }} \le
      K(A) {\cal E} .
 \label{cond_and_res}
\end{equation}
%
The condition number is always greater or equal to one, $K(A) \ge 1$.
If $K(A) \simeq 1$ then the relative error is of the same order of the
weighted residual: if the numerical method we have used has converged
to a solution with small weighted residual we can be confident of the
accuracy of the solution.  However, if the condition number is big,
$K(A) \gg 1$, then, even though the weighted residual may be very
small, the relative error in the solution may be extremely large.

\subsection{Ill-conditioned systems}

The condition number gives us a warning that the solution may not be
accurate.     We must discuss a little bit more at length what this
implies.     Consider the linear system
%
\begin{align}
  \left\{
    \begin{aligned}
      x + 2 y & = 3 , \\
      0.499 x + 1.001 y & = 1.5 .
    \end{aligned} \right.
 \label{ill_cond_1}
\end{align}
%
The solution is $x = y = 1$.       Consider now a system with the same
first equation and a slightly different second equation:
%
\begin{align}
  \left\{
    \begin{aligned}
      x + 2 y & = 3 , \\
      0.5 x + 1.001 y & = 1.5 .
    \end{aligned} \right.
 \label{ill_cond_2}
\end{align}
%
The solution of this second system is $x = 3$ and $y=0$.    In other
words, an extremely small change in one of the coefficients has
produced an enormous change in the solution.

We can analyse this result in terms of the condition number.   The
matrix of the coefficients and its inverse are
%
\begin{equation*}
  A =
  \begin{pmatrix}
    1 & 2 \\
    \dfrac{499}{1000} & \dfrac{1001}{1000} \\
  \end{pmatrix}
  \qquad
  A^{-1} =
  \begin{pmatrix}
   \dfrac{1001}{3} & \dfrac{-2000}{3} \\
   \dfrac{-499}{3} & \dfrac{1000}{3} \\
  \end{pmatrix}
\end{equation*}
%
with infinity norms (maximum of the 1-norm of the rows)
%
\begin{equation*}
  \norm{ A }_\infty = 3 , \quad \norm{ A^{-1} }_\infty = \frac{3001}{3}
  \implies K(A) = 3001 .
\end{equation*}
%
The condition number is big, as expected. To summarise this example,
the following statements are roughly equivalent:
%
\begin{itemize}
%
\item The linear system is such that a small change in specification of
the problem can lead to a large change in relative error; \par
%
\item The condition number is large, $K(A) \gg 1$; \par
%
\item The system is ill conditioned; \par
%
\item As for all ill-conditioned problems, the relative error may be
disastrous even if the residual is very small.
%
\end{itemize}

Being or not being ill-conditioned is a property of the linear system
under study and cannot be eliminated by some cunning numerical
algorithm. The best we can hope for is to have some warning that
things may go horribly wrong: this is the purpose of the condition
number.

While there are techniques based on the Singular Value Decomposition
that try to extract as much information as possible from an
ill-conditioned system, they are well beyond the scope of this unit
and from now on we will assume that the linear system we intend to
solve is perfectly well behaved, i.e.\ has a condition number of the
order of unity.

\section{Direct methods}

\subsection{Introduction}

It is not  possible to  dwell  on all  the  techniques that have  been
developed to  solve  linear  systems of  equations.    The most recent
methods    can be  very  involved    and their  implementation may  be
considered skilled craftsmanship.  Instead,  we introduce the simplest
examples  of  the two  main  categories  of  methods to  solve  linear
systems.  While it is  true that no present  day linear system  solver
uses these methods as we describe them, it is also true that most
of the present day techniques are deeply rooted in these methods.

There are two big families of algorithms for solving linear systems:
the first, the \textit{direct methods}, consist of a finite list of
transformations of the original matrix of the coefficients that reduce
the linear systems to one that is easily solved.  The second family,
the \textit{indirect or iterative methods}, consists of algorithms
that specify a series of steps that lead closer and closer to the
solution without, however, ever exactly reaching it.  This may not
seem a very desirable feature until we remember that we cannot in any
case represent an exact solution: most iterative methods provide us
with a highly accurate solution in relatively few iterations.

\subsection{Gaussian elimination}

The idea behind Gaussian elimination (and all direct methods) is that
some linear systems of equations are very easy to solve.   Suppose for
example, that we wish to solve the problem $A \bx = \bb$ where $A$ is
an upper triangular matrix, i.e.\ $a_{i j} = 0$ if $j < i$.    This
equation can be solved by simple back-substitution.   Consider, for
example, the system
%
\begin{equation*}
  A \bx =
  \begin{pmatrix}
    3 & 2 & 1 \\ 0 & 5 & 4 \\ 0 & 0 & 6
  \end{pmatrix}
  \begin{pmatrix}
    x_1 \\ x_2 \\ x_3
  \end{pmatrix}
 =
 \begin{pmatrix}
   1 \\ 2 \\ 3
 \end{pmatrix}
 = \bb .
\end{equation*}
%
The last equation is trivial and we can easily obtain $x_3$.  Once
$x_3$ is known it can be substituted in the second equation
(back-substitution) and so on:
%
\begin{align*}
 6 x_3               & = 3 && \implies & x_3                 & = \frac{1}{2} , \\
 5 x_2 + 4 x_3       & = 2 && \implies & 5 x_2 + 2           & = 2 \\
                     &     && \implies & x_2                 & = 0 , \\
 3 x_1 + 2 x_2 + x_3 & = 1 && \implies & 3 x_1 + \frac{1}{2} & = 1 \\
                     &     && \implies & x_1                 & = \frac{1}{6} .
\end{align*}

The Gauss elimination algorithm consists of a series of steps that
transform a generic $n \times n$ matrix $A$ with elements $a_{i j}$
into an upper triangular matrix so that the ensuing linear system can
be solved by back substitution.  The algorithm is as follows:

\begin{enumerate}

\item Replace the $j$-th equation with
%
  \begin{equation*}
    -\frac{a_{j 1}}{a_{1 1}} \times ( 1^{\rm st} \text{equation} ) +
    ( j\text{-th equation} ) ,
  \end{equation*}
%
where $j$ is an index that runs from 2 to $n$.  In implementing this
algorithm by hand it is convenient to write $A$ and $\bb$ as a single
$n \times (n+1)$ matrix called the \textit{augmented matrix}.
Consider for example
%
\begin{equation*}
  A =
  \begin{pmatrix}
    1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 0
  \end{pmatrix}
 , \quad
  \bb =
  \begin{pmatrix}
    10 \\ 11 \\ 12
  \end{pmatrix},
\end{equation*}
%
and write them as
%
\begin{equation*}
\left (
  \begin{array}{c c c|c}
   1 & 2 & 3 & 10 \\ 4 & 5 & 6 & 11 \\ 7 & 8 & 0 & 12 \\
  \end{array}
 \right ) .
\end{equation*}
%
By applying the first step of the algorithm to this matrix we obtain
%
\begin{equation*}
  \left (
  \begin{array}{c c c|c}
   1 & 2 & 3 & 10 \\
   0 & 5 - 4 \times 2 & 6 - 4 \times 3 & 11 - 4 \times 10 \\
   0 & 8 - 7 \times 2 & 0 - 7  \times 3 & 12 - 7 \times 10
  \end{array}
 \right ) =
 \left (
  \begin{array}{c c c|c}
   1 & 2 & 3 & 10 \\
   0 & -3 & -6 & -29 \\
   0 & -6 & -21 & -58
  \end{array}
 \right )
\end{equation*}
%
{\bf Remark} - If $a_{1 1}=0$ swap the first row with one whose first
element is non zero.

\item Repeat the previous step, but starting with the next row down
and with $j$ greater than the row number.   If the current row is row
$k$ then we must replace row $j$, $j > k$, with
%
%
\begin{equation*}
   -\frac{a_{j k}}{a_{k k}} \times ( k^{\rm th} \text{ equation} ) +
   ( j^{\rm th}\text{ equation} ) ,
 \end{equation*}
%
where $j$ is an index, $k < j \le n$.    The coefficient $a_{k k}$ is
called the \textit{pivot}. In the case of our example system we have:
%
\begin{equation*}
  \left (
  \begin{array}{c c c|c}
   1 & 2 & 3 & 10 \\
   0 & -3 & -6 & -29 \\
   0 & -6 - 2 (-3) & -21 - 2 (-6) & -58 - 2 (-29)
  \end{array}
 \right ) =
 \left (
  \begin{array}{c c c|c}
   1 & 2 & 3 & 10 \\
   0 & -3 & -6 & -29 \\
   0 & 0 & -9 & 0
  \end{array}
 \right ) .
\end{equation*}

\item When all the rows have been reduced, the matrix has been
transformed in an upper triangular matrix and the linear system can be
solved by back-substitution.

\end{enumerate}

\noindent {\bf Remark} - When designing an algorithm it is important
to keep track of the number of operations required to solve the
problem.   For example, an algorithm which involves a number of
operations that increases exponentially with the problem size becomes
very quickly useless (such problems are called non-polynomial and are
some of toughest nuts to crack in numerical analysis.  The most famous
example is the travelling salesman problem).    The Gauss elimination
algorithm is an $n^3$ algorithm, i.e.\ it requires a number of floating
point operations that grows with the cube of the problem size.

\subsection{Pivoting strategies}

The Gauss elimination method just outlined suffers from poor accuracy
if the matrix coefficients are of very different size.   Consider, for
example, the system
%
\begin{equation*}
  \begin{cases}
    10^{-5} x_1 + x_2 &= 1 , \\ x_1 + x_2 &= 2 .
  \end{cases}
\end{equation*}
%
The 1-norm of the matrix of coefficients, $A$, is $\norm{ A }_1 =
2$.  The inverse of $A$ is
%
\begin{equation*}
  A^{-1} = \frac{1}{10^{-5}-1}
  \begin{pmatrix}
    1 & -1 \\ -1 & 10^{-5}
  \end{pmatrix} ,
\end{equation*}
%
and has 1-norm $\norm{A^{-1}}_1 = 2/(1-10^{-5}) \simeq 2$ so that the
matrix condition number is a most benign $K(A) \simeq 4$.   The problem
is not ill-conditioned and we should not expect any problem in finding
an accurate solution.

We can solve the problem exactly using Gauss elimination.   By
subtracting $10^5$ times the first equation from the second  we obtain
the following augmented matrix and solution:
%
\begin{equation*}
  \left (
   \begin{array}{c c|c}
    10^{-5} & 1 & 1 \\
    0 & -99999 & -99998
   \end{array}
  \right )
  \implies
  \begin{aligned}
    x_2 &=  \dfrac{-99998}{-99999} &&= 0.9999899\ldots && \simeq 1 , \\*[2mm]
    x_1 &=  \dfrac{10^5}{99999} &&= 1.00001000\ldots && \simeq 1 .
  \end{aligned}
\end{equation*}
%
If, however, we now solve the same problem using floating point
notation and only four significant figures, i.e.\ $m=4$, we obtain a
rather different result.    The augmented matrix is now
%
\begin{align}
   \left (
    \begin{array}{c c|c}
      0.1000 \times 10^{-4} & 0.1000 \times 10 & 0.1000 \times 10 \\
      0.1000 \times 10 & 0.1000 \times 10 & 0.2000 \times 10
    \end{array}
  \right ) & \implies \\
   \left (
    \begin{array}{c c|c}
      0.1000 \times 10^{-4} & 0.1000 \times 10 & 0.1000 \times 10 \\
      0.0000 \times 10^{0} & -0.1000 \times 10^6 & -0.1000 \times 10^6
    \end{array}
  \right ) & \implies
  \begin{cases}
    x_2 &= 1, \\ x_1 &= 0 .
  \end{cases}
\end{align}
%
This is an example of \textit{poor scaling}: whenever the coefficients
of a linear system are of greatly varying sizes, we may expect that
rounding errors may build up due to loss of significant figures.
While ill-conditioning is an incurable ``illness'' because it is
intrinsic to the matrix of coefficients, poor scaling is related
to the algorithm and can be eliminated by suitably modifying the
procedure to solve the linear system.     In the case of this example,
everything works fine if we first exchange the two rows so that the
first row is the one with the biggest coefficient.    In this case,
the second row is transformed during Gaussian elimination by
subtracting from it the first row multiplied by $10^{-5}$ and no
rounding errors occur:
%
\begin{align}
  \left (
    \begin{array}{c c|c}
      0.1000 \times 10 & 0.1000 \times 10 & 0.2000 \times 10 \\
      0.1000 \times 10^{-4} & 0.1000 \times 10 & 0.1000 \times 10
    \end{array}
  \right ) & \implies \\
  \left (
    \begin{array}{c c|c}
      0.1000 \times 10 & 0.1000 \times 10 & 0.2000 \times 10 \\
      0.0000 \times 10^{0} & 0.1000 \times 10 & 0.1000 \times 10
    \end{array}
  \right ) & \implies
  \begin{cases}
    x_2 &= 1, \\ x_1 &= 1 .
  \end{cases}
\end{align}
%
The solutions are very close to the exact solution as we would have
expected from the analysis of the condition number.

The procedure of  exchanging rows  (or columns) in  order  to have the
biggest pivot  (i.e.\ the first  coefficient of the  row that is  to be
subtracted from all the  subsequent rows) is called \textit{pivoting}.
From the above example we can conclude that, in the $k$-th step of the
Gauss elimination algorithm, we want to replace row $j$ with
%
\begin{equation*}
  \text{Row}_j \rightarrow \text{Row}_j - \varepsilon \, \text{Row}_k,
\end{equation*}
%
where $\varepsilon$ is a small number, ideally as small as possible.
This can be easily arranged: at each step of the algorithm we
rearrange the set of equations so that we are eliminating using the
largest pivot in modulus.

Notice that the factors that multiply the rows are always smaller than
unity.    This procedure is called \textit{partial pivoting}.

An algorithm that is similar to the one above, but that exchanges both
rows and columns in order that the biggest element in the matrix is
the pivot is called \textit{total pivoting}.  However, total pivoting
involves moving around huge chunks of computer memory and also doing
an exhaustive search for the biggest matrix element.  This practical
problems are such that Gaussian elimination with partial pivoting
is preferred.

\subsection{Decomposition methods}

\subsubsection{Introduction}

Decomposition methods attempt to rewrite the matrix of coefficients as
the product of two matrices.  The advantage of this approach is that
the solution of a linear system of equations is split into smaller and
considerably easier tasks.  There are many decomposition techniques,
but we are going to discuss only the simplest scheme, the $LU$
decomposition, in order to understand the principles behind this
approach to finding the solution of a linear system of equations.

Suppose that we are able to write the matrix $A$ of coefficients of
the linear system $A \bx = \bb$ as
%
\begin{equation*}
  A = L U ,
\end{equation*}
%
where  $L$ is lower  triangular   and $U$  is  upper triangular.   The
solution  of the linear   system of equations becomes straightforward.
We can write
%
\begin{equation*}
  A \bx = \bb \implies ( L \, U ) \, \bx = \bb \implies
  L \left ( U \bx \right ) = \bb .
\end{equation*}
%
If we call $\by = U \bx $, then we have transformed the original
linear system in two systems,
%
\begin{align}
  L \by &= \bb , \label{L_eq} \\
  U \bx &= \by . \label{U_eq}
\end{align}
%
Each of these systems is triangular.  The system~(\ref{L_eq}) is lower
triangular and can be solved by forward substitution while the
system~(\ref{U_eq}) is upper triangular and can be solved by back
substitution.   Therefore, even though we must solve two linear
systems instead of only one, each of the two is very easy to solve.
This is an advantage with respect to Gauss elimination especially if
we have to solve $A \bx = \bb$ for many different values of the vector
$\bb$: we need to factorise the matrix $A$ only once and then use the
factorisation to find the solution vector $\bx$ for all the different
values of the known vectors $\bb$.

\medskip

There are two questions that we must now answer: ``How can we
factorise a matrix?'' and ``Is the factorisation possible?''  We will
answer the first question first and then discuss at more length the
answer to the second.

\subsubsection{Factorisation of a matrix}

A first thing to note is that the $L U$ factorisation as described
above is not unique.   $A$ is a $n \times n$ matrix and has therefore
$n^2$ coefficients.   $L$ and $U$ are both triangular and thus have
$n(n+1)/2$ entries each for a total of $n^2 + n$ entries.
In other words, $L$ and $U$ have together $n$ coefficients more than the
original matrix $A$.  Therefore we can choose $n$ coefficients of $L$
and $U$ to our own liking.

To derive a formula for the $LU$ factorisation we start by writing
explicitly in terms of the components the decomposition $A = L U$.
Call $a_{i j}$, $\ell_{i j}$ and $u_{i j}$ the elements respectively
of $A$, $L$ and $U$.    The following relation must hold:
%
\begin{equation}
  a_{i j} = \sum_{s=1}^n \ell_{i s} u_{s j} =
  \sum_{s=1}^{\min(i,j)} \ell_{i s} u_{s j} ,
  \label{LU_fact}
\end{equation}
%
where we have used the fact that $\ell_{i s} = 0$ for $s>i$ and $u_{s
j}=0$ for $s > j$.

Let us start with $i=j=1$.   Equation~(\ref{LU_fact}) reduces to
%
\begin{equation*}
 a_{1 1} = \ell_{1 1} u_{1 1} .
\end{equation*}
%
We can now make use of the freedom of choosing $n$ coefficients of $L$
and $U$.     The most common choices are:
%
\begin{itemize}
%
\item \textit{Doolittle's factorisation} - Set $\ell_{i i} = 1$,
i.e.\ the matrix $L$ is \textit{unit} lower triangular.
%
\item \textit{Crout's factorisation} - Set $u_{i i} = 1$, i.e.\ the
matrix $U$ is \textit{unit} upper triangular.
%
\end{itemize}

\begin{table}
  \begin{center}
    \fbox{
      \parbox[t]{80mm}{
        \setlength{\parindent}{0pt} \footnotesize
        \textbf{for} $k = 1,2,\ldots,n$ \textbf{do} \par
        \hspace{5mm}
        Specify a nonzero value for either $\ell_{k k}$ (Doolittle) \par
        \hspace{5mm}or $u_{k k}$ (Crout) and compute the other from
        \begin{equation*}
          \ell_{k k} u_{k k} =
          a_{k k} - \sum_{s=1}^{k-1} \ell_{k s} u_{s k} .
        \end{equation*}

        \hspace{5mm} Build the $k$-th row of $U$:\par
        \hspace{5mm} \textbf{for} $j=k+1,k+2, \ldots, n$ \textbf{do} \par
        \begin{equation*}
          u_{k j} = \left( a_{k j} -
            \sum_{s=1}^{k-1} \ell_{k s} u_{s j} \right) / \ell_{k k}
        \end{equation*}
        \hspace{5mm} \textbf{end}

        \smallskip

        \hspace{5mm} Build the $k$-th column of $L$:\par
        \hspace{5mm} \textbf{for} $i=k+1,k+2, \ldots, n$ \textbf{do} \par
        \begin{equation*}
          \ell_{i k} = \left( a_{i k} -
            \sum_{s=1}^{k-1} \ell_{i s} u_{s k} \right) / u_{k k}
        \end{equation*}
        \hspace{5mm} \textbf{end}

        \smallskip

        \textbf{end}
      }
    }
  \end{center}
  \caption{\label{doocrout} \it Algorithm for the Doolittle and Crout
    factorisation methods.}
\end{table}

Following the Doolittle's factorisation we set $\ell_{1 1} = 1$ and
obtain
%
\begin{equation*}
  u_{1 1} = a_{1 1} .
\end{equation*}
%
We can now compute all the elements of the first row of $U$ and of the
first column of $L$, by setting either $i=1$ or $j=1$:
%
\begin{align}
  u_{1 j} & = a_{1 j} , & i & = 1, \, j > 1 \\
  \ell_{i 1} &= a_{i 1} / u_{1 1} , & j & = 1, \, i > 1 .
\end{align}
%
Consider now $a_{2 2}$.   Equation~(\ref{LU_fact}) becomes:
%
\begin{equation*}
  a_{2 2} = \ell_{2 1} u_{1 2} + \ell_{2 2} u_{2 2} .
\end{equation*}
%
If, according to the Doolittle's scheme, we set $\ell_{2 2} = 1$ we
have:
%
\begin{equation*}
  u_{2 2} = a_{2 2} - \ell_{2 1} u_{1 2} ,
\end{equation*}
%
where $\ell_{2 1}$ and $u_{1 2}$ are known from the previous steps.
We can now compute the second row of $U$ and the second column of $L$
by setting either $i=2$ or $j=2$:
%
\begin{align}
  u_{2 j} & = a_{2 j} - \ell_{2 1} u_{1 j}, & i & = 2, \, j > 2 \\
  \ell_{i 2} & = \left ( a_{i 2} - \ell_{i 1} u_{1 2} \right )/ u_{2
    2} & j & = 2, \, i > 2.
\end{align}
%
This procedure can be repeated for all the rows and columns of $U$ and
$L$.   The compact version of the algorithm for $LU$ decomposition using
either Doolittle's or Crout's method is listed in
Table~\ref{doocrout}.

\smallskip

\noindent \textbf{Remark 1} - Notice that for the algorithm to work
$\ell_{k k}$ and $u_{k k}$ must be different from zero.  However, this
should not be interpreted as a condition for the matrix to have an
$LU$ decomposition.   The following matrix,
%
\begin{equation*}
  \begin{pmatrix}
    2 & 1 & -2 \\ 4 & 2 & -1 \\ 6 & 3 & 11
  \end{pmatrix}
\end{equation*}
%
has $u_{2 2} =0 $, but it can be written as the product of
%
\begin{equation*}
  L =
  \begin{pmatrix}
    1 & 0 & 0 \\ 2 & 1 & 0 \\ 3 & 0 & 1
  \end{pmatrix}
  \quad \text{and} \quad
  U =
  \begin{pmatrix}
    2 & 1 & -2 \\ 0 & 0 & 3 \\ 0 & 0 & 17
  \end{pmatrix} .
\end{equation*}

\noindent \textbf{Remark 2} - The $LU$ decomposition with $L$ unit
triangular and  the  Gauss elimination algorithm are  closely related.
The matrix $U$ is the matrix of the coefficients reduced to triangular
form using Gaussian  elimination.  The matrix  $L$ can be obtained  by
writing  each multiplier in   the location corresponding  to  the zero
entry in the matrix it was responsible for creating.   Therefore one
can use Gaussian elimination to find the $LU$ decomposition of a
matrix.   The advantage of $LU$ decomposition over Gaussian elimination
is that if we have to solve many linear systems $A \bx = \bb$ for many
different values of $\bb$, we need to do only one $LU$ decomposition.

\noindent \textbf{Remark 3} - Pivoting is essential for the accuracy
of this algorithm.

\subsubsection{Conditions for factorisation}

We must now specify under what conditions a matrix can be decomposed
as the product of two triangular matrices.  What follows is a list of
sufficient conditions for the decomposition to be possible.  We start
with a theorem that involves the minors of $A$.

\smallskip

\begin{theorem}
\label{LUexists}
If all $n-1$ leading principal minors\footnote{A principal minor of
order $k$ is the minor
%
\begin{equation*}
  \begin{pmatrix}
    a_{1 1} & \cdots & a_{1 k} \\
    \vdots & \ddots & \vdots \\
    a_{k 1} & \cdots & a_{k k}
  \end{pmatrix}
\end{equation*}
%
} of the   $n \times n$  matrix  $A$ are nonsingular,  then  $A$ has an
$LU$ decomposition.   If the $LU$ decomposition exists and $A$ is non
singular, then the $LU$ decomposition is unique and $\det(A)=u_{11}
u_{22} \ldots u_{nn}$ (assuming that the matrix has been decomposed
according the the Doolittle method).
\end{theorem}

\noindent Proof: Golub \& Van Loan, \textit{Matrix computations}, page
97 (Third edition, 1996)

\smallskip

\noindent
\textbf{Remark 1} - This theorem is a consequence of the relation
between Gaussian elimination and $LU$ decomposition.   It is possible to
decompose the matrix in a lower and a upper triangular form if all the
pivots in the Gaussian elimination are different from zero.   The
hypotheses in the above theorem ensure that this is the case.

\smallskip

\noindent
\textbf{Remark 2} - It is not sufficient for a matrix to be
non-singular in order for it to have an $LU$ decomposition.  However, if
the matrix is non singular then there exists a suitable permutation of
its rows such that the permuted matrix has a (unique)
$LU$ decomposition.

A concept that is extremely useful in determining the behaviour and
convergence of many algorithms to solve linear system is that of
\textit{diagonal dominance}.   A matrix is strictly diagonally dominant if the
modulus of each diagonal element is greater than the sum of the moduli
of all the other elements in its row:
%
\begin{equation*}
  | a_{i i} | > \sum_{\genfrac{}{}{0pt}{}{j=1}{j \ne i}}^n
  | a_{i j} |, \qquad (1 \le i \le n) .
\end{equation*}
%
This may seem a rather peculiar condition to require on a matrix.   As
a matter of fact, it is not so for many applications of practical
interest.   For example, most algorithms that solve numerically
partial differential equations using finite difference methods involve
the solutions of large linear systems whose matrix of coefficients is
diagonally dominant.

The importance of diagonal dominance from the point of view of
direct methods to solve linear systems of equations stems from the
following theorems:

\begin{theorem}
\label{diagdom1}
Every strictly diagonally dominant matrix is nonsingular and has an
$LU$ decomposition.
\end{theorem}

\noindent Proof: Kincaid pages 190

\begin{theorem}
\label{diagdom2}
Gaussian elimination of a diagonally dominant matrix does not require
pivoting provided each row is first scaled with its maximum.
\end{theorem}

\noindent Proof: Kincaid page 190

\subsection{Cholesky factorisation}

In the case of a symmetric, positive definite matrix (i.e.\ such that
$\bx^T A \bx > 0 \, \, \forall \bx \ne 0$) then it is possible to
factorise the matrix as $A = L L^T$, in which $L$ is lower triangular with
a positive diagonal (\textit{Cholesky factorisation}).

To find the Cholesky decomposition of the matrix $A$ we can use an
algorithm similar to that developed for the Doolittle and Crout
methods and derived from it by assuming $U=L^T$.  Its description is
in Table~\ref{cholesky}.

\begin{table}
  \begin{center}
    \fbox{
      \parbox[t]{80mm}{
        \setlength{\parindent}{0pt}
        \small
        \textbf{for} $k = 1,2,\ldots,n$ \textbf{do} \par
        \hspace{5mm} Compute the diagonal element \par
        \begin{equation*}
          \ell_{k k} =
          \left ( a_{k k} - \sum_{s=1}^{k-1} \ell_{k s}^2 \right )^{1/2} .
        \end{equation*}

        \hspace{5mm} Build the $k$-th column of $L$:\par
        \hspace{5mm} \textbf{for} $i=k+1,k+2, \ldots, n$ \textbf{do} \par
        \begin{equation*}
          \ell_{i k} = \left ( a_{i k} -
            \sum_{s=1}^{k-1} \ell_{i s} \ell_{k s} \right ) / \ell_{k k}
        \end{equation*}
        \hspace{5mm} \textbf{end}

        \smallskip

        \textbf{end}
      }
    }
  \end{center}
  \caption{\label{cholesky} \it Algorithm for the Cholesky
    factorisation method}
\end{table}

\subsection{Tridiagonal systems}

\noindent Consider a $4\times 4$ system
%
\begin{equation*}
  \begin{pmatrix}
    b_1 & c_1 & 0 & 0 \\
    a_1 & b_2 & c_2 & 0 \\
    0 & a_2 & b_3 & c_3 \\
    0 & 0 & a_3 & b_4
  \end{pmatrix}
  \begin{pmatrix}
    x_1 \\ x_2 \\ x_3 \\ x_4
  \end{pmatrix}
  =
  \begin{pmatrix}
    f_1 \\ f_2\\ f_3 \\ f_4
  \end{pmatrix}.
\end{equation*}
%
It is called a \textit{tridiagonal system}. Tridiagonal systems give
rise to particularly simple results when using Gaussian elimination.
Forward elimination at each step yields a system
%
\begin{equation*}
  \begin{pmatrix}
    1 & c_1/d_1 & 0 & 0 \\
    0 & 1 & c_2/d_2 & 0 \\
    0 & 0 & 1 & c_3/d_3 \\
    0 & 0 & 0 & 1
  \end{pmatrix}
  \begin{pmatrix}
    x_1 \\ x_2 \\ x_3 \\ x_4
  \end{pmatrix}
  =
  \begin{pmatrix}
    y_1 \\ y_2\\ y_3 \\ y_4
  \end{pmatrix},
\end{equation*}
%
where
%
\begin{align*}
    d_1 &= b_1, & y_1 &= f_1/d_1, \\
    d_2 &= b_2-a_1c_1/d_1, & y_2 &= (f_2-y_1 a_1)/d_2, \\
    d_3 &= b_3-a_2c_2/d_2, & y_3 &= (f_3-y_2 a_2)/d_3, \\
    d_4 &= b_4-a_3c_3/d_3, & y_4 &= (f_4-y_3 a_3)/d_4.
\end{align*}
%
Finally, the backward substitution procedure gives the answer
%
\begin{equation*}
  \left\{
    \begin{aligned}
      x_4 &= y_4,\\
      x_3 &= y_3-x_4 c_3/d_3,\\
      x_2 &= y_2-x_3 c_2/d_2,\\
      x_1 &= y_1-x_2 c_1/d_1.
    \end{aligned} \right .
\end{equation*}
%
It is clear what will happen in general case of $n\times n$
tridiagonal system.  The forward elimination procedure is

\begin{enumerate}
  %
\item At the first step $d_1=b_1$ and $y_1=f_1/d_1$;
  %
\item At the $k$-th step $d_k=b_k-a_{k-1}c_{k-1}/d_{k-1}$ and
  $y_k=(f_k-y_{k-1} a_{k-1})/d_k$.
%
\item Once all the $y_k$ have been computed, the $x_k$ can be
  determined using backward substitution: $x_n = y_n$, \ldots,
  $x_{k-1}=y_{k-1}-x_k c_{k-1}/d_{k-1}$, \ldots
  %
\end{enumerate}

\section{Iterative methods}

\subsection{Introduction}

The direct methods for solving linear systems of order $n$ require
about $n^3/3$ operations.   In addition in practical computations with
these methods, the errors which are necessarily introduced through
rounding may become quite large for large $n$.   Now we consider
iterative methods in which an approximate solution is sought by using
fewer operations per iteration.   In general, these may be described
as methods which proceed from some initial ``guess'', $\bx^{(0)}$, and
define a sequence of successive approximations $\bx^{(1)}, \bx^{(2)},
\ldots$ which, in principle, converge to the exact solution.    If the
convergence is sufficiently rapid, the procedure may be terminated at
an early stage in the sequence and will yield a good approximation.
One of the intrinsic advantages of such methods is the fact that the
errors, due to roundoff or even blunders, may be damped out as the
procedure continues.   In fact, special iterative methods are
frequently used to improve ``solutions'' obtained by direct methods.

A large class of iterative methods may be defined as follows.   Let
the system to be solved be
%
\begin{equation}
 A \bx = \bb , \label{Axb}
\end{equation}
%
where $\det(A) \ne 0$.   Then the coefficient matrix can be
split, in an infinite number of ways, into the form
%
\begin{equation*}
  A = N - P ,
\end{equation*}
%
where $N$ and $P$ are matrices of the same order as $A$.    The
system~(\ref{Axb}) is then written as
%
\begin{equation}
  N \bx = P \bx + \bb. \label{NxPxb}
\end{equation}
%
Starting from some \textit{arbitrary} vector $\bx^{(0)}$, we define a
sequence of vectors $\{ \bx^{(n)}\}$, by the recursion
%
\begin{equation}
  N \bx^{(n)} = P \bx^{(n-1)} + \bb, \quad n=1,2,3,\ldots
  \label{rec1}
\end{equation}
%
The different iterative methods are characterised by their choice of
$N$ and $P$.      By looking at~(\ref{rec1}) we can already find
some restrictions on the choice of $N$:
%
\begin{enumerate}
  %
\item The matrix $N$ should not be singular, i.e.\ $\det(N) \ne 0$.
  %
\item The matrix $N$ should be chosen such that a system of the form
  $N \by = \bz$ is ``easily'' solved.
  %
\end{enumerate}
%

\subsection{Iteration schemes}

To simplify the notation we will assume in what follows that all the
diagonal elements of the matrix $A$ are one.   This can be done by
dividing each row by its diagonal element.   If by chance, one row has
a zero diagonal element we would just need to reorder the rows.

\subsubsection{Jacobi's method}

In Jacobi's method the matrix is split into diagonal and off-diagonal
part:
%
\begin{equation*}
  A = I - ( A_L + A_U ) , \implies
  \begin{cases}
    N &= I , \\ P &= A_L+A_U
  \end{cases}
\end{equation*}
%
where we have written the off-diagonal part as the sum of a lower and
an upper triangular matrix, $A_L$ and $A_U$ respectively.   The iteration
scheme is
%
\begin{equation*}
  \bx^{(n+1)} = \bb + ( A_L + A_U ) \bx^{(n)}
\end{equation*}

%
and the convergence matrix is $M \equiv N^{-1} P = P$.

\subsubsection{Gauss-Seidel's method}

In Jacobi's method the old guess is used to estimate all the
elements of the new guess.   In Gauss-Seidel's method each new iterate
is used as soon as it becomes available.    The iteration scheme is as
follows:
%
\begin{align}
  A &= I - (A_L+A_U) , \nonumber \\
  \bx^{(n+1)} &= \bb + A_L \bx^{(n+1)} + A_U \bx^{(n)} , \implies
  \begin{cases}
    N &= I - A_L , \\ P &= A_U ,
  \end{cases}
  \label{seidel}
\end{align}
%
where $A_L$ and $A_U$ are respectively the  lower and upper triangular
parts of $A$ with diagonal elements  set to zero.   Notice that if we
start computing  the   elements   of $\bx^{(n+1)}$ from     the first,
i.e.\ from  $x_1^{(n+1)}$, then all the  terms  on the  right hand side
of~(\ref{seidel}) are known by the time they are used.

\subsubsection{S.O.R. method}

We can interpret the Gauss-Seidel algorithm as a method that applies
at each guess $\bx^{(n)}$ a correction term $\bc$.  We can rewrite the
Gauss-Seidel iteration scheme as:
%
\begin{align*}
  && \bx^{(n+1)} & = \bb + A_L \bx^{(n+1)} + A_U \bx^{(n)} \\
  \implies && \bx^{(n+1)} & = \bx^{(n)} + \left [ \bb + A_L \bx^{(n+1)}
    + (A_U - I) \bx^{(n)} \right ] \\
  &&             & = \bx^{(n)} + \bc,
\end{align*}
%
where the correction term is $\bc = \bb + A_L \bx^{(n+1)} + (A_U - I)
\bx^{(n)}$.  The idea of the S.O.R. (Successive Over-Relaxation)
method is that the convergence may be pushed to go a bit faster by
using a slightly larger correction.  In other words, we introduce a
parameter $\omega > 1$ (\textit{SOR parameter}) that multiplies the
correction term and write the iteration scheme as
%
\begin{align*}
  \bx^{(n+1)} & =  \bx^{(n)} + \omega \bc \\
              & =  \bx^{(n)} + \omega
        \left [ \bb + A_L \bx^{(n+1)} + (A_U - I) \bx^{(n)} \right ].
\end{align*}

\noindent \textbf{Remark 1} - The introduction of the relaxation
parameter $\omega$ can speed up convergence, but may also promote
divergence.    Moreover, it is not clear a priori what is the optimal
value of $\omega$ to achieve convergence in the smallest number of
iterations.    A typical choice is to set $\omega$ quite close to one.

\subsection{Analysis of convergence}

We want to know when an iterative scheme will converge to the correct solution.
We can gain some insight from the scalar case.
Here we have
%
\begin{equation}
  a x = b
\end{equation}
%
with $a, b$ being real numbers, with the solution $x_\text{exact} = b / a$.
Re-writing this as an iterative scheme we have
%
\begin{equation}
  x^{(k+1)} = \frac{b + p x^{(k)}}{n}
\end{equation}
%
where $n - p = a$.

To check convergence we want to see what happens when $x$ is near the correct solution.
We set $x^{(0)} = b / a + \epsilon$. We see that
%
\begin{align}
  && x^{(1)} &= \frac{b + b (n - a) / a + \epsilon (n - a)}{n} \\
  \implies && x^{(1)} - x_\text{exact} &= \epsilon \left( 1 - \frac{a}{n} \right) \\
  \implies && x^{(N)} - x_\text{exact} &= \epsilon \left( 1 - \frac{a}{n} \right)^N.
\end{align}
%
So we expect this iteration scheme to converge to the exact solution if $|1 - a/n| < 1$.
More usefully for our intuition we can write this as $| p / n | < 1$.

In the general matrix case a similar analysis goes through, except instead of $| p / n|$ we look at $M = N^{-1} P$.
This is precisely the convergence matrix mentioned above.
The condition $| p / n | < 1$ becomes the condition on the eigenvalues of $M$ instead.
One can show that an iterative method converges if and only if the matrix $M = N^{-1} P$ exists and has all eigenvalues $\lambda_i$ with modulus strictly less than 1:
%
\begin{equation*}
  \varrho(M) \equiv \max_i | \lambda_i | < 1 \Longleftrightarrow
  \text{Method converges}
\end{equation*}
%
where the symbol $\varrho(M)$ is called the \textit{spectral radius} of $M$.
It is sometimes hard to verify whether this condition is satisfied.
A weaker, but more easily verifiable statement is that the method converges if at least one norm of the matrix $M$ is strictly less than one:
%
\begin{equation*}
  \norm{ M } < 1 \implies \text{Method converges.}
\end{equation*}
%
However, the reverse is not true: a method may converge but some of the norms of $M$ may be larger than or equal to one.

As it can be difficult to compute the full spectral radius, approximate conditions for special matrices have been developed.
A concept that is extremely useful in determining the behaviour and convergence of many algorithms to solve linear system is that of \textit{diagonal dominance}.
A matrix is strictly diagonally dominant if the modulus of each diagonal element is greater than the sum of the moduli of all the other elements in its row:
%
\begin{equation*}
  | a_{i i} | > \sum_{\genfrac{}{}{0pt}{}{j=1}{j \ne i}}^n
  | a_{i j} |, \qquad (1 \le i \le n) .
\end{equation*}
%
This may seem a rather peculiar condition to require on a matrix.
As a matter of fact, it is not so for many applications of practical interest.
For example, most algorithms that solve numerically partial differential equations using finite difference methods involve the solutions of large linear systems whose matrix of coefficients is diagonally dominant.

The importance of diagonally dominance from the point of view of direct methods to solve linear systems of equations stems from the following theorems:

\begin{theorem}
If the matrix of the coefficients $A$ is strictly diagonally dominant then the Jacobi method converges.
\end{theorem}

\medskip

\begin{theorem}
If the matrix of the coefficients $A$ is strictly diagonally dominant then the Gauss-Seidel method converges.
\end{theorem}

\noindent
\textbf{Remark} - Note that diagonal dominance (i.e.\ the sum can be equal to the diagonal element) in itself does not ensure convergence.

\smallskip

\begin{theorem}
If the matrix of the coefficients $A$ is symmetric and positive definite then Gauss-Seidel's method converges.
\end{theorem}

\noindent
\textbf{Remark 1} - This theorem does not hold for Jacobi's method.

\noindent
\textbf{Remark 2} - Convergence theorem for the SOR methods are much
harder to prove.

\section{Determinants and inverses}

The entire gist of this section on linear systems has been that we may
not wish to know the inverse or the determinant of the matrix of the
coefficients in order to solve a linear system of equations.
However, sometimes we need to know these two quantities.  We can use
the methods that we have listed to compute them accurately.

The $LU$ decomposition offers the determinant as an easy to obtain side
product.  If $A = L U$ then
%
\begin{equation*}
  \det(A) = \det(L) \times \det(U).
\end{equation*}
%
In the case of Doolittle's factorisation we have $\det(L) = 1$ and
therefore
%
\begin{equation*}
\det(A) = \det(U) = \prod_{i=1}^n u_{i i} .
\end{equation*}
%
If partial pivoting has been used for the Gaussian elimination then
%
\begin{equation*}
  \det(A) = \pm \prod_{i=1}^n u_{i i} ,
\end{equation*}
%
where the sign is positive or negative depending if the number of row
permutations is even or odd.

\noindent
\textbf{Remark} - To find the determinant using Gaussian elimination
requires a number of operations of the order of $n^3$.   The expansion
in minors requires a number of operations of the order $n!$.

\medskip

Finally, to find the inverse we can solve $n$ systems of $n$ linear
equations.  Let $\be_i$ denote the vector whose components are all
zero except the $i$-th component that is, instead, equal to 1.   Let
$\bc_i$ be the $i$-th column of $A^{-1}$.   Then
%
\begin{equation*}
  A A^{-1} = I \implies A \bc_i = \be_i , \quad i=1,2,\ldots,n.
\end{equation*}
%
Therefore for each value of $i$ we have to solve a system of $n$
linear equations for the $i$-th column of the inverse of $A$.   This
method is much faster than Cramer's rule, that requires the evaluation
of $n+1$ determinants.

\section*{Further reading}

Topics covered here are also covered in
\begin{itemize}
\item Chapters 3 and 8 of Linz \& Wang, \textit{Exploring Numerical
    Methods} (QA297 LIN),
\item Chapter 4 (and extended through chapter 5) of Kincaid \& Cheney,
  \textit{Numerical Analysis} (QA297 KIN),
\item Chapters 2 and 3 of S{\"u}li \& Mayers, \textit{An Introduction
    to Numerical Analysis} (not in library),
\item throughout the extremely detailed Golub \& Van Loan,
  \textit{Matrix Computations} (QA263 GOL).
\end{itemize}


\subsection{Ellipsoids}
\noindent
Ellipsoids look like ellipses that have been rotated and extruded about their axis. They are radially symmetric about this axis. They have the general form
\begin{equation*}
	d = \frac{x^2}{a^2} + \frac{y^2}{b^2} + \frac{z^2}{c^2}
\end{equation*}
Note that the only difference in the equation between an ellipsoid and hyperboloid is the signs are all positive.

[INSERT IMAGE]


%!TEX root =  ../main.tex

\subsection{Definition}

\objective{Interpret piece-wise functions and their limits}

In human observations, there is almost nothing which follows one equation for the
entirety of its domain.  The only constant in the universe is change!  For example,
the population of the world, the stock market, or even one stock might generally follow one
equation for a significant stretch of time, but not forever.  So it is that most functions are
defined in pieces, and are therefore called piece-wise functions.

For example, the absolute value function which you have been using for some time now is
actually a piecewise function.\index{Absolute Value!piece-wise}

$$
|x|=
\begin{cases}
x, x\ge 0\\
-x,x<0
\end{cases}
$$

In light of this reality, it would behoove us to think more carefully about limits, recognizing that
we might be evaluating at the seems between cases.
We should formally define the nomenclature of a ``sided'' limit:

\begin{derivation}{Right-sided limit}\index{limit!right-sided}
``The limit as $x$ approaches $c$ from the right is $L$'' is true if and only if for every $\epsilon > 0$, there exists a $\delta > 0$ such that
$|f(x)-L|<\epsilon$ whenever $0<x-c<\delta$.

$$\lim_{x \to c^+}f(x) = L$$
\end{derivation}

\begin{derivation}{Left-sided limit}\index{limit!left-sided}
Similarly, ``The limit as $x$ approaches $c$ from the left is $L$'' is true if and only if for every $\epsilon > 0$, there exists a $\delta > 0$ such that
$|f(x)-L|<\epsilon$ whenever $0<c-x<\delta$.

$$\lim_{x \to c^-}f(x) = L$$
\end{derivation}

\personfeature[-4in]{\chapdir/pics/George_Boole_color}{George
    Boole}{1815-1864, French}{was an English mathematician
    who said, ``No general method for the solution of questions
    in the theory of probabilities can be established which does
    not explicitly recognise, not only the special numerical bases
    of the science, but also those universal laws of thought which
    are the basis of all reasoning, and which, whatever they
    may be as to their essence, are at least mathematical as to their form.''
    \href{https://en.wikipedia.org/wiki/George_Boole}{(Wikipedia)}}

\subsubsection{Boolean Variables}\index{booleans}
Perhaps surprisingly, your TI-8* can graph piece-wise functions.  We will start with a simple
piecewise-function :

$$
f(x)=
\begin{cases}
x, x<1\\
x^2,x\ge 1
\end{cases}
$$

\index{TI-8*!piece-wise}

If we wanted to graph the sections separately, we could make $Y_1=(X^2)/(X\ge{}1)$ and
$Y_2=(X)/(X<1)$.  (The equality and inequality signs are under 2ND-MATH --- TEST.)  This will
allow you to make the different sections different colors or different shading, but that might be
what you want.  To graph everything in $Y_1$, use $(X^2)/(X\ge{}1)+(X)/(X<1)$.

\subsection{Other Discontinuities}
\reminder{\lefthand}{The different TI-8* behave differently around holes.  Newer calculators will attempt to make a hole apparent, while older models do not show it.

Only the new models draw points visibly, but even then they are very small.  We recommend
against even trying to represent them in the TI-8*s.}

All together, there are five kinds of discontinuities.  We are only responsible to rigorously prove
instanes of the first two and the last:

\subsubsection{Removable}
Here the limit exist.  The graph has a hole in it, which may or may not be defined as a point somewhere unexpected.

\inlinefig{\chapdir/pics/Removable-discontinuity.png}{The limits is removable.  The function jumps to a value.}

\subsubsection{Jump}
The limits does not exist.  The graph is not continuous, but ``leaps'' from one output to another without passing in between, at one or more points.

\inlinefig{\chapdir/pics/Discontinuity_jump.png}{The limit does not exist and there is an unavoidable jump.}

\subsubsection{Infinite}
The limit may or may not exist.  The function itself goes up and/or down without limit.  The most common example is a vertical asymptote.

\inlinefig{\chapdir/pics/infinite}{At 1, the limit exists, because both sides are going to (positive) infinity.  At -1, the limit does not exist, because the two sides disagree.}

\subsubsection{Oscillating}
The limit does not exist.  The graph varies between outputs in way that never resolves.

\inlinefig{\chapdir/pics/The_function_sin1_over_x.png}{No matter how much we zoom in to the function $\sin{\frac{1}{x}}$ around (0,0), it will only increase in complexity.}

\subsubsection{Domain}
The limits does not exist because on one side, the function ceases to exist.

\inlinefig{\chapdir/pics/CorolarLocalizareRadacina.jpg}{The limit does not exist at $a$ or $b$ because one side of the function there does not exist.}


\section{Vector spaces}
As we've seen in \autoref{chapter:linear algebra intuitive} vectors are found at the heart of linear algebra. We first defined them in a geometric way as objects with magnitude and direction, and later as lists of real numbers, analyzing the connections between these two mostly parallel definitions. We also spoke about vector spaces of the type $\Rs{n}$ as the structures vectors exist in. However, we haven't defined vectors nor vector spaces formally - which is exactly what we do in this section, by defining the concept of \emph{vector spaces}.

\begin{note}{$\bm{\Rs{n}}$ as a guide to general vector spaces}{}
	While reading the definition below, it is worthwhile to reflect on each of the given axioms as it relates to the familiar vector space $\Rs{n}$.
\end{note}

\begin{definition}{Vector space}{vector space}
	A vector space over a field $\mathbb{F}$ is a set $V$ which, together with two operations described below, fulfils a list of axioms. The two operations are
	\begin{listitemize}
	\item[Vector addition] an operation which takes two elements of $V$ and returns a single element of $V$, i.e. $+:V\times V\to V$.
	\item[Scalar multiplication] an operation which takes a single element of $\mathbb{F}$ and a single element of $V$ and returns a single element of $V$, i.e. $\cdot:\mathbb{F},V \to V$.
	\end{listitemize}

	The axioms to be fulfilled are:
	\begin{descitemize}
		\item[Commutativity of vector addition] for any $u,v\in V$,
			\[
				u+v=v+u.
			\]

		\item[Associativity of vector addition] for any $u,v,w\in V$,
			\[
				u+(v+w) = (u+v)+w.
			\]

		\item[Additive identity] there exist an element $0\in V$ for which, for any $v\in V$,
			\[
				v+0 = v.
			\]

		\item[Scalar multiplicative identity] for any $v\in V$
			\[
				1\cdot v = v,
			\]
			where $1$ is the multiplicative identity in $\mathbb{F}$.

		\item[Additive inverse] for any $v\in V$ there exist an element $u\in V$ for which
			\[
				v+u = 0.
			\]

		\item[Associativity of scalar multiplication] for any $\alpha,\beta\in\mathbb{F}$ and $v\in V$
			\[
				\alpha\cdot(\beta\cdot v) = (\alpha\beta)\cdot v,
			\]
			where $\alpha\beta$ is the multiplication defined for $\mathbb{F}$.

		\item[Distributivity of vector addition] for any $\alpha\in\mathbb{F}$ and $u,v\in V$,
			\[
				\alpha\cdot(u+v) = (\alpha\cdot u) + (\alpha\cdot v).
			\]

		\item[Distributivity of scalar addition] for any $\alpha,\beta\in\mathbb{F}$ and $v\in V$,
			\[
				(\alpha+\beta)\cdot v = (\alpha\cdot v) + (\beta\cdot v).
			\]
	\end{descitemize}

	The elements of $V$ are then called \emph{vectors}, and the elements of $\mathbb{F}$ are called \emph{scalars}.
\end{definition}

Since we discussed $\Rs{n}$ thoroughly in \autoref{chapter:linear algebra intuitive}, let's prove that it is indeed a vector space under the above definition. First, the claim:

\begin{theorem}{$\bm{\Rs{n}}$ is a vector space}{Rn vector space}
	The set of elements of the form
	\[
		\vec{v} = \colvec{v_{1};v_{2};\vdots;v_{n}}
	\]
	where $v_{i}\in\mathbb{R}$, forms a vector space over $\mathbb{R}$ together with the following two operations:
	\begin{descitemize}
		\item[Vector addition]
			\[
				\vec{u}+\vec{v} = \colvec{u_{1};u_{2};\vdots;u_{n}} + \colvec{v_{1};v_{2};\vdots;v_{n}} = \colvec{u_{1}+v_{1};u_{2}+v_{2};\vdots;u_{n}+v_{n}}.
			\]
		\item[Scalar multiplication]
			\[
				\alpha\cdot\vec{v} = \colvec{\alpha v_{1};\alpha v_{2};\vdots;\alpha v_{n}}.
			\]
	\end{descitemize}
\end{theorem}

The proof itself is pretty easy, based on the fact that $\mathbb{R}$ is a field:

\begin{proof}{$\bm{\Rs{n}}$ is a vector space}{Rn vector space}
	Since the results of both operations defined for $\Rs{n}$ only depend on the respective components of a vector $v\in\Rs{n}$, all the axioms of a vector space apply, since they derive directly from the fact that $\mathbb{R}$ is a field. As an example, we will elaborate on two of the axioms:
	\begin{descitemize}
	\item[Additive inverse] Given a vector $\vec{v}\in\Rs{n}$, each of its components $v_{i}$ has an inverse under $\mathbb{R}$, namely $-v_{i}$. Therefore,
		\[
			\colvec{v_{1};v_{2};\vdots;v_{n}} + \colvec{-v_{1};-v_{2};\vdots;-v_{n}} = \colvec{v_{1}-v_{1};v_{2}-v_{2};\vdots;v_{n}-v_{n}} = \colvec{0;0;\vdots;0} = \vec{0},
		\]
		which is the additive identity in $\Rs{n}$.

	\item[Distributivity of vector addition] for each component of two vectors $\vec{u},\vec{v}\in\Rs{n}$, given the rules for vector addition and scalar multiplication, together with the distributivity of numbers in $\mathbb{R}$:
		\begin{align*}
			\alpha (\vec{u}+\vec{v}) &= \alpha\left(\colvec{u_{1};u_{2};\vdots;u_{n}} + \colvec{v_{1};v_{2};\vdots;v_{n}}\right) = \alpha\colvec{u_{1}+v_{1};u_{2}+v_{2};\vdots;u_{n}+v_{n}}\\
			&= \colvec{\alpha u_{1}+\alpha v_{1};\alpha u_{2} + \alpha v_{2};\vdots;\alpha u_{n} + \alpha v_{n}} = \colvec{\alpha u_{1};\alpha u_{2};\vdots;\alpha u_{n}} + \colvec{\alpha v_{1};\alpha v_{2};\vdots;\alpha v_{n}} = \alpha\vec{u} + \alpha\vec{v}.
		\end{align*}
	\end{descitemize}
\end{proof}
(it is adviceable for the reader to go over the rest of the axioms and prove them for $\Rs{n}$)


\documentclass{article}

\usepackage{mathrsfs,amssymb,amsmath}

\begin{document}

\section{If $S:l_p \to l_p$ is defined by $S(\alpha_1, \alpha_2, ...) =(0, \alpha_1, \alpha_2, ...)$ describe the lattice of invariant closed subspaces of S}

$\mathscr{M}_n=\{x\in l^p:x(k)=0 \text{ for } 1 \le k \le n\}$, then $\mathscr{M}_n \in \text{Lat}S$, if $x \in \mathscr{M}_n$, so is $S(x)$, since they both start with at least $n$ 0s.

$\mathscr{M}_{n+1} \in \mathscr{M}_n$ since if a series begins w n+1 0s it will also begin with n zeros.

Claim: These subspaces, together with the zero element and all of $l^p$, represent all of Lat T (which is thus a totally ordered set).

Start with the element $x_1=(1, 0, 0, ...)$. It will be shown that the smallest closed invariant subspace $X$ that contains $x_1$ is fact the whole space.

If $x_1 \in X$, so is any element of the form $(\alpha_1, 0, 0, ...)$, since $X$ is a subspace so should be closed under scalar multiplication. Also, if $x_1 \in X$, so is $S(x_1) = (0, 1, 0, 0, ...) = x_2$, as well as $(0, \alpha, 0, 0)$. Since a subspace is closed under vector addition, all elements of the form $(\alpha_1, \alpha_2, \alpha_3, ..., 0, 0, ...)$ are thus in $X$. Call the set of all such points $Y \subset X$.

Now we must show that $Y$ is dense in $l_p$, hence that $X=l_p$. Given an arbitrary element of $l_p$ and $\epsilon$, since $l_p -> 0$, we can produce an element $y \in Y$ such that $|| l_p - y || < \epsilon$

By the same argument, any element whose first n entries are 0 will be identical to $\mathscr{M}_n$, thus proving the claim.

\end{document}


%---------------------------Minimum Angle-----------------------------
\section{Minimum Angle\label{s:tet-min-angle}}

The (nonoriented) dihedral angle of two faces of the
tetrahedron that are adjacent along edge $i$ ($0\le{}i\le5$), is,
measured in degrees,
\[
  \alpha_i = \frac{180\dgr}{\pi}
    \arccos{\left(\vec{n_{i1}} \cdot \vec{n_{i2}}\right)},
\]
where $\vec{n_{i1}}$ and $\vec{n_{i2}}$ are unit vectors normal to the
two tetrahedron faces that are adjacent to edge $i$. Subsequently,
the minimum (nonoriented) dihedral angle of the tetrahedron, measured
in degrees, is
\[
  q =
    \min_{i\in\{0,1,2,3,4,5\}}{\alpha_i}.
\]

\tetmetrictable{minimum dihedral angle}%
{$A^1$}%                                              Dimension
{$[40\dgr,\frac{180\dgr}{\pi}\arccos\tfrac{1}{3}]$}%  Acceptable range
{$[0\dgr,\frac{180\dgr}{\pi}\arccos\tfrac{1}{3}]$}%   Normal range
{$[0\dgr,360\dgr]$}%                                  Full range
{$\frac{180\dgr}{\pi}\arccos\tfrac{1}{3}\approx70.528779\dgr$}% Regular tetrahedron value
{--}%                                                 Reference(s)
{v\_tet\_minimum\_angle}%                             Verdict function name


\section{Basics}

% ===
\emph{Fundamental Assumption}

Data is iid for unknown $P$: $(x_i, y_i) \sim P(X,Y)$

\textbf{Empirical risk}: $\hat{R}_D(w) = \frac{1}{\abs{D}}\sum_{(x,y)\in D} (y-w^\top x)^2$ \\
\textbf{True risk}: $R(w) = \int p(x,y)\, r_i^2 \,\partial x\partial y = \mathbb{E}_{x,y}[r_i^2]$

%\subsection*{True risk and estimated error}
%True risk: $R(w) = \int P(x,y) (y-w^\top x)^2 \partial x \partial y = \mathbb{E}_{x,y}[(y-w^\top x)^2]$ \\
%%Est. error: $\hat{R}_D(w) = \frac{1}{\abs{D}}\sum_{(x,y)\in D} (y-w^\top x)^2$

% ===
\emph{Standardization:} \enskip
{\color{gray} (for $x_k\!\in\! X$, $k=1,\ldots,d$)}

Centered data with unit variance:
\highlight*{$\!\widetilde{x}_{i,k} = \frac{x_{i,k}-\hat{\mu}_k}{\hat{\sigma}_k}\!\!$}

\vspace{-2pt}$\hat{\mu}_k = \frac{1}{n}\sum_{i=1}^n x_{i,k}$, \enskip $\hat{\sigma}_k^2 = \frac{1}{n}\sum_{i=1}^n {(x_{i,k}-\hat{\mu}_k)}^2$

% ===
\emph{Parametric vs. Nonparametric:}

\textbf{Parametric}: have finite set of parameters.
e.g. linear regression, linear perceptron\\
\textbf{Nonparametric}: grow in complexity with the size of the data, more expressive.
e.g. k-NN

% ===
\emph{Gradient Descent:}

\begin{highlightbox}
    \begin{enumerate}
        \item Pick arbitrary $w_0 \in \mathbb{R}^d$
        \item $w_{t+1} = w_t - \eta_t \nabla\!_w \hat{R}(w_t)$
    \end{enumerate}

    \vspace*{-25pt}\hfill
    \begin{highlightbox*}<.4>
    	\footnotesize
    	$\nabla\!_w = \begin{bmatrix}
    		\pderiv{}{w_1} & \ldots & \pderiv{}{w_d}
    	\end{bmatrix}$
    \end{highlightbox*}
\end{highlightbox}

% ===
\emph{Stochastic Gradient Descent (SGD):}

\begin{highlightbox}
\begin{enumerate}
    \item Pick arbitrary $w_0 \in \mathbb{R}^d$
	\item $w_{t+1} = w_t - \eta_t \nabla_w \ell(w_t;x',y')$, with u.a.r.\\ (random) data point $(x',y') \in D$
\end{enumerate}
\end{highlightbox}

works if $\sum\limits_t \eta_t=\infty$ and $\sum\limits_t \eta_t^2 < \infty$,
e.g. $\eta_t \!=\! \frac{1}{t}$


\documentclass[]{article}
\usepackage{amsmath}

\title{MTH 343 Numerical Analysis Lecture 4: Bisection Method (f(x) = 0)}
\author{Sheikh Abdul Raheem Ali}
\date{February 3, 2019}

\begin{document}

	\maketitle

	\section*{Strategy/Algorithm}

	Assume $ f(x) $ is continuous.

	\begin{enumerate}
		\item Begin with two values $ x = a \& x = b $ that bracket the root by finding $ f(a) \cdot f(b) < 0  $ (i.e they are of opposite signs).

		\item The method successfully divides the interval in half \& replaces one endpoint by the midpoint so that the root is again bracketed.

	\end{enumerate}

	\[ x^2 - 2 = 0, (\text{ solution } x = \sqrt{2}) \text{ on the interval} \ [1,2] \ \begin{cases}
	f(a = 1) = -1 & < 0 \\
	f(b = 2) = 2 & > 0
	\end{cases}\]

	\begin{tabular}{c c c c c}
		\# of iterations \textbf{$ n $} & \textbf{$ a_n $} & \textbf{$ b_n $} & midpoint \textbf{$ P_n $} & f($ P_n $) \\

		0&1&2&1.5&0.25 \\
		1&1&1.5&1.25& -0.4375\\
		2&1.25&1.5&1.375& -0.1094\\
		3&1.375&1.5&1.4375& +0.0664\\
		4&1.4375&1.375&1.40625& -0.0225\\
		5&1.4375&1.40625&1.421875& -0.0217\\
		6&1.40625&1.421895&1.4140625& -0.0004\\
		7&1.4140625&1.421875&1.41796875& -0.0106

	\end{tabular}
\section*{Remarks}

\begin{enumerate}

	\item The main advantage of the bisection method is that it is guaranteed to work if $ f(x) $ is continuous on $ [a,b] $ and $ a,b $ bracket the root.

	 \item Its accuracy after n iterations is known in advance which is $ \le |\frac{b-a}{2^n}| $

	\[ |P_n - P| \le \frac{b-a}{2^n} \]

	Where $ P $ = exact value, and $ P_n $ = midpoint (approx)

	In the previous example, find how many iterations are needed to achieve an accuracy for $ 10^{-4}  (x^2 - 2 = 0  \text{ on }  [1,2])$.

	\begin{align*}
	 |E_n| \le \frac{2-1}{2^n} < 0.0001 \\
	 2^n > 10,000 \\
	 n \cdot \ln2 > \ln(10,000) \\
	 n > \frac{\ln(10,000)}{\ln2} \approx 13.28 \\
	 n = 14 \ terms
	\end{align*}

	\item A minor disadvantage is that it is slow to converge, however with speedy computers available the slowness is of less concern.

	\item When multiple roots are concerned, the method may not be applicable since it might not change signs.
\end{enumerate}

\[ 	0 = x^2 - 6x + 9 = (x-3)^2, \  x = 3  \]

\end{document}


\section{Hamiltonian Monte Carlo}
As we mention in section \ref{sec:Metropolis_Hastings}, a significant inefficiency in the Metropolis algorithm, is that it often behaves like a random walk and thus the simulations take a longer road while moving through the target distribution, which gives rise to slow simulation. This is especially a problem when concerning high-dimensional problems, such as Bayesian Neural Networks (\cite{neal2012bayesian}).
Another way to generate proposals with a higher efficiency, is by updating the parameters dynamically by simulating Hamiltonian dynamics and then use the Metropolis algorithm to accept or reject these proposals. Such an algorithm suppress the local random walk behavior and thus allowing it to move much faster and more rapidly through the distribution. This method, which we will present in this chapter, is called Hamilton Monte Carlo (HMC), and is commonly used in computational physics and statistics. The algorithm was originally proposed by \cite{Duane1987216} for calculations used in lattice quantum chromodynamics, but was later introduced to the field of computation statistics when it was used for Bayesian Neural Networks n \cite{neal2012bayesian}\\
\\
The HMC algorithm reduces correlation between successive sampled states, compared to the Metropolis Hastings algorithm in section \ref{sein tc:Metropolis_Hastings}, by proposing moves to distant states which maintain a high probability of acceptance due to the approximate energy conserving properties of the simulated Hamiltonian dynamic when using a symplectic integrator\footnote{In mathematics, a symplectic integrator is a numerical integration scheme for Hamiltonian systems.}. The reduced correlation means fewer Markov chain samples are needed to approximate integrals with respect to the target probability distribution for a given Monte Carlo error. For the HMC algorithm we need to construct a differential equation system, that is known to keep the target distribution invariant. If we can achieve this, we can simulate transitions following the solution to this differential equation system and these transitions can be shown to leave the target distribution invariant. Such a differential equations system is often the Hamiltonian dynamics named after William Rowan Hamilton and will be introduced in the next section.

\subsection{Hamiltonian dynamics}
Before we move to the actual HMC algorithm, we need to become familiar with the concept of Hamiltonian dynamics. Hamiltonian dynamics are used to describe how an object move around in a system. The Hamiltonian dynamics are defined by the objects position $\boldsymbol{\theta}\in \mathbb{R}^d$ and it's momentum $\boldsymbol{\rho}\in \mathbb{R}^d$, which in the physics are equivalent to an object's mass times it's velocity at some point in time. The object's position is associated with a potential energy $U(\boldsymbol{\theta})$ and likewise the momentum is associated with a kinetic energy $K(\boldsymbol{\rho})$. The sum of the potential and kinetic energy is regarded as the total energy of the system and often called the Hamiltonian,
\begin{equation*}
H(\boldsymbol{\theta},\boldsymbol{\rho})=U(\boldsymbol{\theta})+K(\boldsymbol{\rho})
\end{equation*}
one important feature of the Hamiltonian is that it is constant over time. Taking the partial derivative with respect to time of the position and momentum, shows us how they evolve over time,
\begin{equation}\label{eq:hamilton_equations}
\begin{split}
\frac{d \theta_{i}}{d t}=\frac{\partial H}{\partial \rho_{i}}=\frac{\partial K(,\boldsymbol{\rho})}{\partial \rho_{i}} \\
\frac{d \rho_{i}}{d t}=-\frac{\partial H}{\partial \theta_{i}}=-\frac{\partial U(,\boldsymbol{\theta})}{\partial \theta_{i}}
\end{split}
\end{equation}
where $i=1,2, \ldots,d$. These are named Hamiltonian equations and represents differential equations system. The Hamiltonian equations are very useful, since if we are able to evaluate the partial derivatives from equation \ref{eq:hamilton_equations}, we are able to predict the position and momentum variable of the object at any point in the future $t^\prime>t$.
\subsection{Properties of the Hamiltonian Dynamics}
The Hamiltonian dynamics have several properties, which are very crucial when used in relation to MCMC simulations.\\
\\
\textit{Property of time reversibility:}\\
This property is very important for showing that the HMC transitions, generated by the dynamics, leave the target distribution invariant, since the reversibility of the chain transitions requires reversibility of the dynamics. \\
\\
\textit{Property of conservation:}\\
Secondly we have the property of conservation, that is the dynamics keeps the Hamiltionian invariant. This can easily be verified from equation \ref{eq:hamilton_equations}
\begin{equation}\label{eq:Hamilton_conservation}
    \frac{d \H}{d t}=\sum_{i=1}^{d}\left[\frac{d \theta_{i}}{d t} \frac{\partial \H}{\partial \theta_{i}}+\frac{d \rho_{i}}{d t} \frac{\partial \H}{\partial \rho_{i}}\right]=\sum_{i=1}^{d}\left[\frac{\partial \H}{\partial \rho_{i}} \frac{\partial \H}{\partial \theta_{i}}-\frac{\partial \H}{\partial \theta_{i}} \frac{\partial \H}{\partial \rho_{i}}\right]=0
\end{equation}
In the HMC method, we will use Metropolis updates, with proposals found by the Hamiltonian dynamic. We have that the acceptance probability is one if $\H$ is kept invariant (see e.g \cite{neal2012mcmc}). This is however not possible in practise, and we are only able to keep $\H$ approximately invariant, but we will come back to this issue later. \\
\\
\textit{Property of volume preservation:}
The last crucial property of the Hamiltonian dynamics is the volume preservation property, which gives us that the dynamics preserves volume in the $(q,p)$ space.

\begin{equation*}
    \sum_{i=1}^{d}\left[\frac{\partial}{\partial \theta_{i}} \frac{d \theta_{i}}{d t}+\frac{\partial}{\partial \rho_{i}} \frac{d \rho_{i}}{d t}\right]=\sum_{i=1}^{d}\left[\frac{\partial}{\partial \theta_{i}} \frac{\partial \H}{\partial \rho_{i}}-\frac{\partial}{\partial \rho_{i}} \frac{\partial \H}{\partial \theta_{i}}\right]=\sum_{i=1}^{d}\left[\frac{\partial^{2} \H}{\partial \theta_{i} \partial \rho_{i}}-\frac{\partial^{2} \H}{\partial \rho_{i} \partial \theta_{i}}\right]=0
\end{equation*}

\subsection{Discretizing Hamiltonian Equations}
The Hamiltonian equations describe how an objective evolve in continuous time, but for simulating Hamiltonian dynamics on a computer system, we have to approximate the differential equations numerically in some way, this is done by discretizing time. We do this by splitting the time interval $dt$ into small intervals $\varepsilon$. We will now present the two most common ways of handling this. The first is the Euler's Method and lastly the Leapfrog Method.
\subsubsection*{The Euler’s Method}
This is one of the most common methods in computational science for solving a differential equation system. For Hamiltonian equations the Euler methods update the position and momentum variable as follows,

\begin{equation*}
\begin{split}
\rho_{i}(t+\varepsilon)=\rho_{i}(t)+\varepsilon \frac{d \rho_{i}}{d t}(t)=\rho_{i}(t)-\varepsilon \frac{\partial U}{\partial \theta_{i}(t)} \\
\theta_{i}(t+\varepsilon)=\theta_{i}(t)+\varepsilon \frac{d \theta_{i}}{d t}=\theta_{i}(t)+\varepsilon \frac{\partial K}{\partial \rho_{i}(t)}
\end{split}
\end{equation*}
According to \cite{neal2012mcmc} a slightly better result can be obtained by,
\begin{equation*}
\begin{split}
\rho_{i}(t+\varepsilon) &=\rho_{i}(t)-\varepsilon \frac{\partial U}{\partial \theta_{i}}(\theta(t)) \\
\theta_{i}(t+\varepsilon) &=\theta_{i}(t)+\varepsilon \frac{\rho_{i}(t+\varepsilon)}{m_{i}}
\end{split}
\end{equation*}
\subsubsection*{The Leapfrog method}
Another discretizing scheme often used for simulation Hamiltonian equations, is the Leapfrog method. Whereas the Euler’s method takes full steps for updating position and momentum, the leapfrog method takes a half steps to update momentum value,
\begin{equation*}
\begin{split}
\rho_{i}(t+\varepsilon / 2)=\rho_{i}(t)-(\varepsilon / 2) \frac{\partial U}{\partial \theta_{i}(t)} \\
\theta_{i}(t+\varepsilon)=\theta_{i}(t)+\varepsilon \frac{\partial K}{\partial \rho_{i}(t+\varepsilon / 2)} \\
\rho_{i}(t+\varepsilon)=\rho_{i}(t)-(\varepsilon / 2) \frac{\partial U}{\partial \theta_{i}(t+\varepsilon)}
\end{split}
\end{equation*}
According to \cite{neal2012mcmc} the Leapfrog method, preserves volume exactly, \textcolor{red}{which is good??} and it is also reversible.

\subsection{Hamiltonian and Probability: Canonical Distributions}
We now have a slightly better understanding understanding of what is a Hamiltonian and how we can simulate it's dynamics by either the Euler method or the Leapfrog method. We now need to connect this with the MCMC theory from the previous sections. In order to perform this connection, we need to relate the target distribution $\hat{p}(\boldsymbol{\theta})$ and the Hamiltonian, such that we can use the Hamiltonian equations to the target distribution. A way of doing this, proposed by \cite{neal2012bayesian}, is use a concept from statistical mechanics known as the canonical (Boltzman) distribution. The probability distribution of $\boldsymbol{\theta}$ under the canonical distribution can be written as
\begin{equation*}
    \hat{p}(\boldsymbol{\theta})\propto \exp(\frac{-U(\boldsymbol{\theta})}{T})
\end{equation*}
where $U(\boldsymbol{\theta})$ again is the potential energy. $T$ is often called the temperature of the system and usually chosen to be equal to one (\cite{neal2012bayesian}).
One should note that any probability distribution that is nowhere zero can be put into this form by letting $U(\boldsymbol{\theta})=-\log \hat{p}(\boldsymbol{\theta})-\log Z$, for any convenient choice of $Z$.  Since the Hamiltonian is an energy function for the joint state of both position and momentum, we need a joint distribution, which is given by
\begin{equation*}
p(\boldsymbol{\theta},\boldsymbol{\rho})\propto \exp(-\H(\boldsymbol{\theta},\boldsymbol{\rho}))   = \exp(-U(\boldsymbol{\theta}))\exp(-K(\boldsymbol{\rho}))=\hat{p}(\boldsymbol{\theta})p(\boldsymbol{\rho})
\end{equation*}
where the last equality hold since we assume independence between $\boldsymbol{\theta}$ and $\boldsymbol{\rho}$. We now have a joint distribution, in terms of the Hamiltonian function, which we know how to simulate. But we are in fact only interested in the position variable $\boldsymbol{\theta}$ and not the momentum variable $\boldsymbol{\rho}$, which we in some way can interpret as a "helper" variable that enable us to simulate the joint distribution. In order to obtain marginal samples from the target distribution only, on can simply throw away the samples the momentum distribution, because they are independent anyway. Since we can interpret the momentum variable as a  helper variable, we are allowed freely to decide the marginal distribution $p(\boldsymbol{\rho})$. The literature often choose it to be Gaussian, $\boldsymbol{\rho}\sim \mathcal{N}\left(0, \Sigma \right)$, where $\Sigma$ is some symmetric, positive-definite mass matrix and often chosen to be diagonal, such that $\boldsymbol{\rho}$ is d-dimensional multivariate normal where the variables are independent.
The Hamiltonian equations from equation \ref{eq:hamilton_equations} can now be written as,
\begin{equation*}
\begin{split}
\frac{d \theta_{i}}{d t}&=\left[\Sigma^{-1}\rho\right]_i \\
\frac{d \rho_{i}}{d t}&=-\frac{\partial U}{\partial \theta_i}
\end{split}
\end{equation*}

\subsection{Hamiltonian Monte Carlo}
We start the HMC algorithm from an initial state $\boldsymbol{\theta}_0$ $\boldsymbol{\rho}_0$, and then we simulate the Hamiltonian dynamics for $t+\varepsilon$ using the Leapfrog method. We choose the Leapfrog method since \cite{betancourt2017conceptual} shows it to be more effective. The states generated for the position and momentum variable at the end of the Leapfrog simulation is used as proposals for a new state $(\boldsymbol{\theta}^\prime,\boldsymbol{\rho}^\prime)$. The proposed stats are accepted according to the Metropolis acceptance criteria,
\begin{equation*}
\begin{split}
    \alpha\left((\boldsymbol{\theta},\boldsymbol{\rho}) \mapsto (\boldsymbol{\theta}^\prime , \boldsymbol{\rho}^\prime )\right) &= \min\left\{1, \frac{p(\boldsymbol{\theta}^\prime,\boldsymbol{\rho}^\prime)}{p(\boldsymbol{\theta},\boldsymbol{\rho})} \right\}\\
    &= \min\{1,\exp\left(\log p(\boldsymbol{\theta}^\prime,\boldsymbol{\rho}^\prime)- \log p(\boldsymbol{\theta}, \boldsymbol{\rho})  \right)\\
    &= \min \left\{1,\exp\left(-\H(\boldsymbol{\theta}^\prime,\boldsymbol{\rho}^\prime) +\H(\boldsymbol{\theta},\boldsymbol{\rho})\right) \right\}
\end{split}
\end{equation*}
If we could simulate the Hamiltonian dynamics exactly, the Metropolis acceptance criteria would always be equal to one, due to the Hamiltonian conservation criteria in equation \ref{eq:Hamilton_conservation} which would give $\min \{1, \exp (0)\}=1$. But since we cannot simulate the Hamilton dynamics exactly and we need to approximate them with the Leapfrog scheme introduced earlier. We can however see that if we choose a proper way of discretize the dynamics, the term  $\H(\boldsymbol{\theta},\boldsymbol{\rho})-\H(\boldsymbol{\theta}^\prime,\boldsymbol{\rho}^\prime)$ in the exponent should be small, thus a high acceptance rate. This is a very clever way of making proposals, since we can make very large and uncorrelated moves in the state space, while keeping a high acceptance of probability. The algorithm is written in pseudo code in algorithm \ref{alg:HMC}. \\
\\
The Algorithm relies heavily on choosing values for $\varepsilon$ and $L$. If we pick a too large value for $\varepsilon$, then the simulations will be inaccurate and give us a low acceptance rate. If we on the other hand choose $\varepsilon$ which is too small, we will waste computation by taking small steps. A too small $L$ will give us successive samples that lies close to each other, which gives us the same undesirable random walk behavior as with the Metropolis algorithm from section \ref{sec:Metropolis_Hastings}. A too large choice for $L$, will produce trajectories that loops back and retrace their steps again and even worse if we choose $L$ such that the samples jumps from one side of the space to the other in each iteration, then accoding to.

\begin{algorithm}[h!]\label{alg:HMC}

\SetAlgoLined
$\text{Given } \boldsymbol{\theta}^{0}, \varepsilon, L, \mathcal{L}, M$:\\
\For{m=1 to M}{
Sample $\boldsymbol{\rho}^{0} \sim \mathcal{N}(0, I)$ \\
Set $\boldsymbol{\theta}^{m} \leftarrow \boldsymbol{\theta}^{m-1}, \boldsymbol{\theta}^\prime \leftarrow \boldsymbol{\theta}^{m-1}, \boldsymbol{\rho}^\prime \leftarrow \boldsymbol{\rho}^{0}$\\
\For{i = 1 to L}{
Set $\boldsymbol{\theta}^\prime, \boldsymbol{\rho}^\prime \leftarrow \operatorname{Leapfrog}(\boldsymbol{\theta}^\prime, \boldsymbol{\rho}^\prime, \varepsilon)$\\
With probability $\alpha=\min \left\{1, \frac{\exp \left\{\mathcal{L}(\boldsymbol{\theta}^\prime)-\frac{1}{2} \boldsymbol{\rho}^\prime \cdot \boldsymbol{\rho}^\prime\right\}}{\exp \left\{\mathcal{L}\left(\boldsymbol{\theta}^{m-1}\right)-\frac{1}{2} \boldsymbol{\rho}^{0} \cdot \boldsymbol{\rho}^{0}\right\}}\right\}, \text { set } \boldsymbol{\theta}^{m} \leftarrow \boldsymbol{\theta}^\prime, \boldsymbol{\rho}^{m} \leftarrow-\boldsymbol{\rho}^\prime$

}
}

\SetKwFunction{Lepfrog}{Lepfrog}
\SetKwProg{Fn}{Function}{:}{\KwRet{$\boldsymbol{\theta}^\prime,\boldsymbol{\rho}^\prime$}}
\Fn{\Lepfrog{$\boldsymbol{\theta}$, $\boldsymbol{\rho}$, $\varepsilon$}}{
 Set $\boldsymbol{\rho}^\prime \leftarrow r+(\varepsilon / 2) \nabla_{\theta} \mathcal{L}(\theta)$\\
Set $\boldsymbol{\theta}^\prime \leftarrow \theta+\varepsilon \boldsymbol{\rho}^\prime$\\
Set  $\boldsymbol{\rho}^\prime \leftarrow \boldsymbol{\rho}^\prime+(\varepsilon / 2) \nabla_{\theta} \mathcal{L}(\boldsymbol{\theta}^\prime)$
  }
\caption{Hamiltonian Monte Carlo}
\end{algorithm}

\subsection{No-U-Turn Hamiltonian Monte Carlo}
No-U-Turn Sampler (NUTS) introduced by \cite{hoffman2011nouturn}
extends the HMC by eliminating the need to specify the trajectory length $L$.


\section{Burke's Theorem}
\label{sec:Burke-Theorem}

\begin{theorem}[Burke]
\label{thm:Burke}
	For any $M/M/m$ with arrival rate $\lambda$, we have that:

	\begin{enumerate}
		\item The departure process is $Poisson(\lambda)$
		\footnote{that is, inter-departure times are Exponentially distributed with rate $\lambda$.};

		\item The number of jobs in the system is independent of the sequence of previous departures
		\footnote{that is, independent of times and patterns}.
	\end{enumerate}
	\begin{proof}
		For a formal demonstration, see \cite{harchol2013performance}.
	\end{proof}
\end{theorem}

The Burke's Theorem allows us to instantly analyze a large class of queueing networks.
Let us start with its application on (i) tandem systems and (ii) acyclic networks with probabilistic routing.

\subsection{Burke Applications: Tandem Systems}
\label{sec:Burke-Application-Tandem-Systems}

By the Burke's Theorem, for any tandem system made by m $M/M/1$ nodes arrival rate $\lambda$, we have that:

\begin{itemize}
	\item the arrival stream to each server is Poisson distributed with rate $\lambda$;

	\item the number of jobs in each server is independent of the number of jobs in every other server;

	\item the order of the sequence of servers are uninfluential for system performance;

	\item the state probability is

	\begin{equation}
	\label{eqn:Burke-Application-Tandem-Systems-Probability-State}
	\pi_{n_{1},...,n_{m}} = \prod_{i=1}^{m} \varrho_{i}^{n_{i}} (1 - \varrho_{i})
	\end{equation}

	where $\varrho_{i}$ is the utilization of the $i$-th server.

	\item the probability to have $n_{i}$ jobs in the $i$-th server is

	\begin{equation}
	\label{eqn:Burke-Application-Tandem-Systems-Probability-Jobs-Server}
	\probability{N_{i}=n_{i}} = \varrho_{i}^{n_{i}} (1 - \varrho_{i})
	\end{equation}

	where $\varrho_{i}=\frac{\lambda}{\mu_{i}}$ is the utilization of the $i$-th server.

	\item the mean number of jobs in the $i$-th server is

	\begin{equation}
	\label{eqn:Burke-Application-Tandem-Systems-Expected-Jobs-Server}
	\expected{N_{i}} = \frac{\varrho_{i}}{1-\varrho_{i}}
	\end{equation}
\end{itemize}

%\begin{figure}[tp]
%	\label{fig:tandem-systems}
%	\centering
%	\includegraphics{fig/Tandem-System}
%	\caption{A Tandem system}
%\end{figure}

\subsection{Burke Application: Acyclic Networks with Probabilistic Routing}
\label{sec:Burke-Application-Acyclic-Networks-Probabilistic-Routing}

By the Burke's Theorem, for any acyclic system with probabilistic routing made by $m$ $M/M/1$ nodes with arrival rate $\lambda$, everything stated in \Cref{sec:Burke-Application-Tandem-Systems} holds, but the arrival stream to each server is (splitted/merged) Poisson distributed with rate proportional to $\lambda$.

%\begin{figure}[tp]
%\label{fig:Acyclic-Networks-Probabilistic-Routing}
%	\centering
%	\includegraphics{fig/Acyclic-Network-Probabilistic-Routing}
%	\caption{An Acyclic Networks with Probabilistic Routing}
%\end{figure}


\chapter{Week 2: The vector space}

\section{Linear combinations}
\begin{definition}
  An expression
  \begin{equation*}
    \alpha_1\vv_1 + \cdots + \alpha_n\vv_n
  \end{equation*}
  is a linear combination of the vectors $\vv_1, \ldots, \vv_n$.
  The scalars $\alpha_1, \ldots, \alpha_n$ are the coefficients of the linear combination.
\end{definition}

% ================================================================

\section{Span}
\begin{definition}[Span]
  The set of all linear combinations of some vectors $\vv_1, \ldots, \vv_n$ is called the span of these vectors. Written $\Span\{\vv_1,\ldots,\vv_n\}$.
\end{definition}

\begin{definition}
  Let $\sV$ be a set of vectors. If $\vv_1, \ldots, \vv_n$ are vectors such that $\sV = \Span\{ \vv_1, \ldots, \vv_n \}$, then
  \begin{itemize}
  \item We say $\{ \vv_1, \ldots, \vv_n \}$ is a generating set for $\sV$.
  \item We refer to the vectors $\vv_1, \ldots, \vv_n$ as generators for $\sV$.
  \end{itemize}
\end{definition}

If we use the vectors $[1, 0, 0]$, $[0, 1, 0]$, and $[0, 0, 1]$:
\begin{equation*}
  [x, y, z] = x[1, 0, 0] + y[0, 1, 0] + z[0, 0, 1]
\end{equation*}
These are called \emph{standard generators} for $\R^3$.

% ================================================================

\section{Geometry of sets of vectors}
Span of a single nonzero vector $\vv$:
\begin{equation*}
  \Span\{\vv\} = \{ \alpha\vv : \alpha\in\R \}
\end{equation*}
This is the line through the origin and $\vv$, One-dimensional. The span of the empty set: just the origin, Zero-dimensional.

One way to specify a plane can be with:
\begin{equation*}
  \{ (x, y, z) : ax + by + cz = 0 \}
\end{equation*}

Using dot-product, we could rewrite as
\begin{equation*}
  \{ [x, y, z] : [a, b, c] \cdot [x, y, z] = 0 \}
\end{equation*}

We can specify a line in three dimensions:
\begin{equation*}
  \{ [x, y, z] : \va_1 \cdot [x, y, z] = 0, \va_2 \cdot [x, y, z] = 0 \}
\end{equation*}

Two ways to represent a geometric object (line, plane, etc.) containing the origin:
\begin{itemize}
\item Span of some vectors.
\item Solution set of some system of linear equations with zero right hand sides.
\end{itemize}

What is common with these two representations? R: Subset $\fF^D$ satisfies three properties:
\begin{enumerate}
\item Subset contains the zero vector $\vzero$.
\item If subset contains $\vv$ then it contains $\alpha\vv$ for every scalar $\alpha$.
\item If subset contains $\vu$ and $\vv$ then it contains $\vu + \vv$.
\end{enumerate}

\begin{definition}[Vector space]
  Any subset $\sV$ of $\fF^{D}$ satisfying the three properties is called a vector space.
\end{definition}

\begin{definition}
  If $\sU$ is also a vector space and $\sU$ is a subset of $\sV$ then $\sU$ is called a subspace of $\sV$.
\end{definition}

\subsection{Convex hull}
\begin{definition}
  For vectors $\vv_1, \ldots, \vv_n$ over $\R$, a linear combination
  \begin{equation*}
    \alpha_1\vv_1, \ldots, \alpha_n\vv_n
  \end{equation*}
  is a convex combination if the coefficients are all nonnegative and they sum to $1$.
\end{definition}

% ================================================================

\section{Vector spaces}
To represent an object that doesn't contain the origin we sum a vector $\vc$ and redefine the definition as:
\begin{equation*}
  \{ \vc + \vv : \vv \in \sV \}
\end{equation*}
It can also be abbreviated as $\{ \vc + \sV \}$.

\begin{definition}
  A linear combination
  \begin{equation*}
    \alpha_1\vu_1 + \alpha_2\vu_2 + \cdots + \alpha_n\vu_n
  \end{equation*}
  where
  \begin{equation*}
    \alpha_1 + \alpha_2 + \cdots + \alpha_n = 1
  \end{equation*}
  is an affine combination.
\end{definition}

\begin{definition}
  The set of all affine combinations of vectors $\vu_1, \vu_2, \ldots, \vu_n$ is called the \emph{affine hull} of those vectors.
\end{definition}

Affine hull of $\vu_1, \vu_2, \ldots, \vu_n = \{ \vu_1 + \Span\{ \vu_2 - \vu_1, \ldots, \vu_n - \vu_1 \} \}$. This shows that the affine hull is an affine space.

In general, a geometric object can be expressed as the solution set of a system of linear equations.
\begin{equation*}
  \{ \vx : \va_1\cdot\vx = \beta_1, \cdots, \va_m\cdot\vx = \beta_m \}
\end{equation*}

Conversely, is the solution set an affine space?
Consider solution set of a contradictory system of equations, e.g. $1x = 1, 2x = 1$:
\begin{itemize}
\item Solution set is empty
\item but a vector space $\sV$ always contains the zero vector,
\item so an affine space $\vu_1 + \sV$ always contains at least one vector.
\end{itemize}
Turns out this is the only exception:
\begin{theorem}
  The solution set of a linear system is either empty or an affine space.
\end{theorem}

\begin{definition}
  A linear equation $\va\cdot\vx = 0$ with zero right-hand side is a homogeneous linear equation. A system of homogeneous linear equations is called a homogeneous linear system.
\end{definition}

\begin{lemma}
  Let $\vu_1$ be a solution to a linear system. Then, for any other vector $\vu_2$, $\vu_2$ is also a solution if and only if $\vu_2-\vu_1$ is a solution to the corresponding homogeneous linear system.
\end{lemma}