\par
\chapter{Algorithm Design}
\label{chapter:algorithmDesign}
\par
Let us begin with a very quick description of the sparse
factorizations we will use.
We assume that the matrix $A$ has symmetric structure, or more
generally, if $a_{i,j} \ne 0$, then we treat $a_{j,i}$ as nonzero.
\par
Let us ignore pivoting for a moment.
The matrix will be factored as $A = (L+I)D(I+U)$.
The rows and columns of $A$ are partitioned into {\it fronts}.
We use the upper case Roman letters $I$ and $J$ to refer to
index sets for a front.
We use $\bnd{J}$ to represent the {\it boundary} of front $J$,
namely those rows and columns $k \notin J$
such that $l_{k,j} \ne 0$ and or $u_{j,k} \ne 0$.
\par
There are two steps to compute the entries in a front.
The first is to form the temporary matrix
\begin{equation}
\label{alg:eqn:1}
\left \lbrack \begin{array}{cc}
T_{J,J} & T_{J, \bnd{J}} \\
T_{\bnd{J},J} & T_{\bnd{J},\bnd{J}}
\end{array} \right \rbrack
=
\left \lbrack \begin{array}{cc}
A_{J,J} & A_{J, \bnd{J}} \\
A_{\bnd{J},J} & 0
\end{array} \right \rbrack
-
\left \lbrack \begin{array}{c}
L_{J,*} \\
L_{\bnd{J},*}
\end{array} \right \rbrack
D_{*,*}
\left \lbrack \begin{array}{cc}
U_{*,J} & U_{*,\bnd{J}}
\end{array} \right \rbrack.
\end{equation}
The $*$ subscript means all rows and columns that precede $J$.
We think of the temporary matrix on the left
as {\it fully assembled}, i.e.,
the original entries are present plus all updates from rows and
columns that precede the front.
\par
The second step is to factor the front as
\begin{equation}
\label{alg:eqn:2}
\left \lbrack \begin{array}{cc}
T_{J,J} & T_{J, \bnd{J}} \\
T_{\bnd{J},J} & T_{\bnd{J},\bnd{J}}
\end{array} \right \rbrack
=
\left \lbrack \begin{array}{cc}
L_{J,J} + I & 0 \\
L_{\bnd{J},J} & I
\end{array} \right \rbrack
\left \lbrack \begin{array}{cc}
D_{J,J} & 0 \\
0 & H_{\bnd{J},\bnd{J}}^J
\end{array} \right \rbrack
\left \lbrack \begin{array}{cc}
I + U_{J,J} & U_{J,\bnd{J}} \\
0 & I
\end{array} \right \rbrack.
\end{equation}
\par
In equation~(\ref{alg:eqn:1}) there will likely be many columns $i$ such
that $L_{J,i}$ and $U_{i,J}$ are zero, in which case they need not
take part in the first equation.
Since all preceding rows and columns are found in fronts
themselves, we can rewrite equation~(\ref{alg:eqn:1}) as
\begin{equation}
\label{alg:eqn:3}
\left \lbrack \begin{array}{cc}
T_{J,J} & T_{J, \bnd{J}} \\
T_{\bnd{J},J} & T_{\bnd{J},\bnd{J}}
\end{array} \right \rbrack
=
\left \lbrack \begin{array}{cc}
A_{J,J} & A_{J, \bnd{J}} \\
A_{\bnd{J},J} & 0
\end{array} \right \rbrack
-
\sum_{\bnd{I} \cap J \ne \emptyset}
\left \lbrack \begin{array}{c}
L_{J,I} \\
L_{\bnd{J},I}
\end{array} \right \rbrack
D_{I,I}
\left \lbrack \begin{array}{cc}
U_{I,J} & U_{I,\bnd{J}}
\end{array} \right \rbrack.
\end{equation}
\par
The fronts can be grouped into a {\it front tree} using a {\it
parent} relation: $par(I) = J$ means that $J$ is the parent of $I$
in the front tree.
There is one special property that relates the boundaries of fronts
and the front tree: if $\bnd{I} \cap J \ne \emptyset$, then $I$ is
a descendent of $J$, or conversely, $J$ is an ancestor of $I$.
Front $I$ need not update all of its ancestors, nor does front $J$
need to be updated by all of its descendents, but when an update
does occur, it is between two fronts that have a
ancestor-descendent relationship.
The front tree is defined by the parent relation:
the {\it parent} of $I$ is the front that contains the first row
and column in $\bnd{I}$, in other words, the closest supported
ancestor to $I$.
(Of course, if $\bnd{I} = \emptyset$, then its parent does not
exist, and $I$ is a root of the tree, actually a forest if $A$ is
reducible.)
There is one important property that relates the boundary sets of a
front and its parent:
if $par(I) = J$, then $\bnd{I} \subseteq J \cup \bnd{J}$.
\par
Using the front tree, let us describe the fronts that are
descendents of a front.
We use the ${\widehat {\ }}$ operator to denote a subtree.
${\widehat J}$, the subtree rooted at $J$,
is the set of fronts that contains $J$ and all of its descendents.
We can write the subtree relation in a recursive form.
$$
{\widehat J} = J \cup \bigcup_{par(I) = J} {\widehat I}
$$
Using these two relations, we can rewrite
equation~(\ref{alg:eqn:3}) as follows.
\begin{eqnarray}
\label{alg:eqn:4}
\left \lbrack \begin{array}{cc}
T_{J,J} & T_{J, \bnd{J}} \\
T_{\bnd{J},J} & T_{\bnd{J},\bnd{J}}
\end{array} \right \rbrack
& = &
\left \lbrack \begin{array}{cc}
A_{J,J} & A_{J, \bnd{J}} \\
A_{\bnd{J},J} & 0
\end{array} \right \rbrack
-
\sum_{par(I) = J}
\left \lbrack \begin{array}{c}
L_{J,{\widehat I}} \nonumber \\
L_{\bnd{J},{\widehat I}}
\end{array} \right \rbrack
D_{{\widehat I},{\widehat I}}
\left \lbrack \begin{array}{cc}
U_{{\widehat I},J} & U_{{\widehat I},\bnd{J}}
\end{array} \right \rbrack \\
& = &
\left \lbrack \begin{array}{cc}
A_{J,J} & A_{J, \bnd{J}} \\
A_{\bnd{J},J} & 0
\end{array} \right \rbrack
-
\sum_{par(I) = J}
L_{\bnd{I}, {\widehat I}}
D_{{\widehat I}, {\widehat I}}
U_{{\widehat I}, \bnd{I}}
\end{eqnarray}
One more trick is necessary.
Recall equation~(\ref{alg:eqn:2}).
Once we have assembled the temporary matrix, we factor into the
$L$, $D$ and $U$ portions plus an additional matrix
$H_{\bnd{J},\bnd{J}}^J$ which we call the update matrix.
Using the update matrices of the children, we can rewrite
equation~(\ref{alg:eqn:4}) as follows.
\begin{equation}
\label{alg:eqn:mf}
\left \lbrack \begin{array}{cc}
T_{J,J} & T_{J, \bnd{J}} \\
T_{\bnd{J},J} & T_{\bnd{J},\bnd{J}}
\end{array} \right \rbrack
=
\left \lbrack \begin{array}{cc}
A_{J,J} & A_{J, \bnd{J}} \\
A_{\bnd{J},J} & 0
\end{array} \right \rbrack
-
\sum_{par(I) = J}
H_{\bnd{I},\bnd{I}}^I
\end{equation}
This is the defining equation for the multifrontal method
\cite{duf83-multifrontal}.
The matrix on the left is the {\it frontal matrix} for front $J$,
and is usually dense and almost always treated as if it were dense.
The first matrix on the right consists of original entries and is
usually sparse.
Each of the $H_{\bnd{I},\bnd{I}}^I$ update matrices are treated as
if they were dense.
\par
There are many advantages to the multifrontal method.
The computations are almost all dense matrix operations.
In a serial environment, the temporary storage for the update
matrices can be handled as a stack, a last-in first-out list
structure.
This makes the transition to an out-of-core implementation very easy.
But, the one disadvantage is that the frontal matrices can take up
an immense amount of space for the largest fronts, typically near
the top of the tree.
This is the main reason that we do not use this algorithm
in the {\bf SPOOLES} library.
\par
Return to equation~(\ref{alg:eqn:3}) for a moment.
A left-looking block general sparse algorithm computes three of the
four submatrices on the left.
\begin{eqnarray*}
T_{J,J} & = & A_{J,J}
- \sum_{\bnd{I} \cap J \ne \emptyset}
L_{\bnd{I} \cap J, I} D_{I, I} U_{I, \bnd{I} \cap J}
\\
T_{\bnd{J},J} & = & A_{\bnd{J},J}
- \sum_{\bnd{I} \cap J \ne \emptyset}
L_{\bnd{I} \cap \bnd{J}, I} D_{I, I} U_{I, \bnd{I} \cap J}
\\
T_{J,\bnd{J}} & = & A_{J,\bnd{J}}
- \sum_{\bnd{I} \cap J \ne \emptyset}
L_{\bnd{I} \cap J, I} D_{I, I} U_{I, \bnd{I} \cap \bnd{J}}
\end{eqnarray*}
and then factors the three as follows.
\begin{eqnarray*}
T_{J,J}       & = & (L_{J,J} + I)D_{J,J}(I + U_{J,J}) \\
T_{\bnd{J},J} & = & L_{\bnd{J},J}D_{J,J}(I + U_{J,J}) \\
T_{J,\bnd{J}} & = & (L_{J,J} + I)D_{J,J}U_{J,\bnd{J}}
\end{eqnarray*}
If the fronts are large there are good computational kernels to be had.
The disadvantage lies in the irregular access patterns for the
computed factor submatrices; this makes an out-of-core
implementation problematic.
On the other hand, working storage is modest, for there are no
update matrices waiting to be assembled.
\par \bigskip \par
\noindent {\large\bf Pivoting and delayed rows and columns}
\par \bigskip \par
Let us now turn to pivoting during the factorization.
It is not the actual swapping of rows and columns that gives
problems --- one must merely keep track
of indices and permute rows and columns.
What does complicate matters is when rows and columns cannot be
eliminated from a front due to stability reasons.
(The remaining lower right matrix in $T_{J,J}$ may be zero, or
eliminating rows and columns may result in large entries in
$L_{\bnd{J},J}$ or $U_{J,\bnd{J}}$.)
The delayed rows and columns must be {\it passed} up to the parent
front, enlarging the parent front where an attempt will be made
to eliminate them with the rows and columns in the parent's front.
\par
Let us assume symmetric pivoting for the moment, where the row and
column permutation matrices are identical.
Let the rows and columns in front $I$ be partitioned into two sets,
$I_e$ contains the eliminated rows and columns
and
$I_d$ contains the delayed rows and columns.
Let us look first at the multifrontal method for it is easier to
understand.
\begin{equation}
\label{alg:eqn:mf-delayed}
\left \lbrack \begin{array}{cc}
T_{I,I} & T_{I, \bnd{I}} \\
T_{\bnd{I},I} & T_{\bnd{I},\bnd{J}}
\end{array} \right \rbrack
=
\left \lbrack \begin{array}{cc}
L_{I_e,I_e} + I & 0 \\
L_{I_d \cup \bnd{J},I_e}  & I
\end{array} \right \rbrack
\left \lbrack \begin{array}{cc}
D_{I_e,I_e} & 0 \\
0           & H_{I_d \cup \bnd{I},I_d \cup \bnd{I}}^I
\end{array} \right \rbrack
\left \lbrack \begin{array}{cc}
I + U_{I_e,I_e} & U_{I_e,I_d \cup \bnd{J}} \\
0 & I \\
\end{array} \right \rbrack.
% \left \lbrack \begin{array}{cc}
% T_{I,I} & T_{I, \bnd{I}} \\
% T_{\bnd{I},I} & T_{\bnd{I},\bnd{J}}
% \end{array} \right \rbrack
% =
% \left \lbrack \begin{array}{ccc}
% L_{I_e,I_e} + I & 0 & 0 \\
% L_{I_d,I_e}     & I & 0 \\
% L_{\bnd{J},I_e}   & 0 & I
% \end{array} \right \rbrack
% \left \lbrack \begin{array}{ccc}
% D_{I_e,I_e} & 0                 & 0 \\
% 0           & H_{I_d,I_d}^I     & H_{I_d,\bnd{I}}^I \\
% 0           & H_{\bnd{I},I_d}^I & H_{\bnd{I},\bnd{I}}^I
% \end{array} \right \rbrack
% \left \lbrack \begin{array}{ccc}
% I + U_{I_e,I_e} & U_{I_e,I_d}, & U_{I_e,\bnd{J}} \\
% 0 & I & 0 \\
% 0 & o & I \\
% \end{array} \right \rbrack.
\end{equation}
\par
The update matrix for $I$ contains the delayed rows and columns
as well as the usual update matrix $H^I_{\bnd{I},\bnd{I}}$
that would normally occur.
$$
H^I_{I_d \cup \bnd{I}, I_d \cup \bnd{I}}
=
\left \lbrack \begin{array}{cc}
H^I_{I_d,I_d}     & H^I_{I_d, \bnd{I}} \\
H^I_{\bnd{I},I_d} & H^I_{\bnd{I},\bnd{I}}
\end{array} \right \rbrack
$$
Recall, the $I_e$ rows and columns are fully assembled, so they can
be merged into the parent front. Let us define the new front after
merging all delayed rows and columns from the children as
$$
{\widetilde J} = J \cup \bigcup_{par(I) = J } I_d.
$$
We can now write the multifrontal equation with pivoting as follows.
\begin{equation}
\label{alg:eqn:mf-pivoting}
\left \lbrack \begin{array}{cc}
T_{{\widetilde J},{\widetilde J}} & T_{{\widetilde J}, \bnd{J}} \\
T_{\bnd{J},{\widetilde J}} & T_{\bnd{J},\bnd{J}}
\end{array} \right \rbrack
=
\left \lbrack \begin{array}{cc}
A_{J,J} & A_{J, \bnd{J}} \\
A_{\bnd{J},J} & 0
\end{array} \right \rbrack
-
\sum_{par(I) = J}
H_{I_d \cup \bnd{I}, I_d \cup \bnd{I}}^I
\end{equation}
Delaying rows and columns from one front to its parent really
doesn't change the multifrontal algorithm very much.
This is the reason that a factorization with pivoting has usually
been implemented by a multifrontal algorithm.
\par
Now let us return to a left-looking general sparse factorization
and try to incorporate pivoting.
The equations that accumulate updates from the descendent fronts
must be modifed to include the delayed rows and columns from the
children.
\begin{eqnarray*}
T_{{\widetilde J},{\widetilde J}} & = & A_{J,J}
- \sum_{\bnd{I} \cap J \ne \emptyset}
L_{\bnd{I} \cap J, I_e}
D_{I_e, I_e}
U_{I_e, \bnd{I} \cap J}
+ \sum_{par(I) = J}
\left \lbrack \begin{array}{cc}
H_{I_d, I_d} & H_{I_d, \bnd{I} \cap J} \\
H_{\bnd{I} \cap J, I_d} & 0 \\
\end{array} \right \rbrack
\\
T_{\bnd{J},{\widetilde J}} & = & A_{\bnd{J},J}
- \sum_{\bnd{I} \cap J \ne \emptyset}
L_{\bnd{I} \cap \bnd{J}, I_e} D_{I_e, I_e} U_{I_e, \bnd{I} \cap J}
+ \sum_{par(I) = J} H_{\bnd{I} \cap \bnd{J}, I_d}
\\
T_{{\widetilde J},\bnd{J}} & = & A_{J,\bnd{J}}
- \sum_{\bnd{I} \cap J \ne \emptyset}
L_{\bnd{I} \cap J, I_e} D_{I_e, I_e} U_{I_e, \bnd{I} \cap \bnd{J}}
+ \sum_{par(I) = J} H_{I_d, \bnd{I} \cap \bnd{J}}
\end{eqnarray*}
It appears that since we do not know the structure of ${\widetilde J}$
until the delayed rows and columns of the children of $J$ are known,
we cannot start the computation until the children of $J$ are finished.
This is not quite true.
Write the above equations as follows, marking the ${\widetilde T}$
matrices on the left as distinct from the $T$ matrices on the right.
\begin{eqnarray*}
{\widetilde T}_{{\widetilde J},{\widetilde J}} & = & T_{J,J}
+ \sum_{par(I) = J}
\left \lbrack \begin{array}{cc}
H_{I_d, I_d} & H_{I_d, \bnd{I} \cap J} \\
H_{\bnd{I} \cap J, I_d} & 0 \\
\end{array} \right \rbrack
\\
{\widetilde T}_{\bnd{J},{\widetilde J}} & = & T_{\bnd{J},J}
+ \sum_{par(I) = J} H_{\bnd{I} \cap \bnd{J}, I_d}
\\
{\widetilde T}_{{\widetilde J},\bnd{J}} & = & T_{J,\bnd{J}}
+ \sum_{par(I) = J} H_{I_d, \bnd{I} \cap \bnd{J}}
\end{eqnarray*}
The $T$ matrices have the following form.
\begin{eqnarray*}
T_{J,J} & = & A_{J,J}
- \sum_{\bnd{I} \cap J \ne \emptyset}
L_{\bnd{I} \cap J, I_e}
D_{I_e, I_e}
U_{I_e, \bnd{I} \cap J}
\\
T_{\bnd{J},J} & = & A_{\bnd{J},J}
- \sum_{\bnd{I} \cap J \ne \emptyset}
L_{\bnd{I} \cap \bnd{J}, I_e} D_{I_e, I_e} U_{I_e, \bnd{I} \cap J}
\\
T_{J,\bnd{J}} & = & A_{J,\bnd{J}}
- \sum_{\bnd{I} \cap J \ne \emptyset}
L_{\bnd{I} \cap J, I_e} D_{I_e, I_e} U_{I_e, \bnd{I} \cap \bnd{J}}
\end{eqnarray*}
The computation of $T_{J,J}$, $T_{\bnd{J},J}$ and $T_{J,\bnd{J}}$
does not depend on any delayed rows and columns from the children,
and so it can begin to execute before the children are complete.
It is key to note that $J$ and $\bnd{J}$ are known before the
factorization starts and do not change due to any delayed rows or
columns.
It is true that we do not know $I_e$ for each descendent of $J$
{\it until} front $I$ is complete, but in some sense that does not
matter.
All that is necessary is to keep track of which descendent fronts $I$
have $I_e \ne \emptyset$ and $\bnd{I} \cap J \ne \emptyset$.
The three equations with ${\widetilde T}$ on the left
are simply an assembly of matrices ---
one matrix comes from the updates from the descendents,
one matrix from delayed rows and columns.
\par \bigskip \par
\noindent {\bf A serial factorization}
\par \bigskip \par
There are five simple steps to a serial factorization:
initialize the front, load original entries, accumulate updates
from descendents, assemble any delayed rows and columns from the
children, then factor the front and update any delayed rows and
columns.
Here is the algorithm step by step.
\par
\noindent Loop over the fronts in a post-order traversal
\begin{enumerate}
\item
Initialize
$\displaystyle
\left \lbrack \begin{array}{cc}
T_{J,J} & T_{J, \bnd{J}} \\
T_{\bnd{J},J} & 0
\end{array} \right \rbrack
= 0.
$
\item
Load with original entries
\par
$\displaystyle
\mbox{\qquad}
\left \lbrack \begin{array}{cc}
T_{J,J} & T_{J, \bnd{J}} \\
T_{\bnd{J},J} & 0
\end{array} \right \rbrack
\mbox{\tt +=}
\left \lbrack \begin{array}{cc}
A_{J,J} & A_{J, \bnd{J}} \\
A_{\bnd{J},J} & 0
\end{array} \right \rbrack
$
\item
Accumulate updates from descendents.
\par
For $I_e \ne \emptyset$ and $\bnd{I} \cap J \ne \emptyset$.
\par
$\mbox{\qquad}\displaystyle
\left \lbrack \begin{array}{cc}
T_{J,J} & T_{J, \bnd{J}} \\
T_{\bnd{J},J} & 0
\end{array} \right \rbrack
\mbox{\tt -=}
\left \lbrack \begin{array}{c}
L_{\bnd{I}\cap J,I_e} \\
L_{\bnd{I}\cap \bnd{J},I_e}
\end{array} \right \rbrack
D_{I_e,I_e}
\left \lbrack \begin{array}{cc}
U_{I_e, \bnd{I} \cap J} & U_{I_e, \bnd{I}\cap \bnd{J}}
\end{array} \right \rbrack
$
\item
Assemble postponed rows and columns.
\par $\widetilde J = J$.
\par for $par(I) = J$ and $I_d \ne \emptyset$
\par
$\mbox{\qquad} {\widetilde J} := {\widetilde J} \cup I_d$
\par
$\mbox{\qquad} \displaystyle
\left \lbrack \begin{array}{cc}
T_{{\widetilde J},{\widetilde J}} & T_{{\widetilde J}, \bnd{{\widetilde J}}} \\
T_{\bnd{{\widetilde J}},{\widetilde J}} & 0
\end{array} \right \rbrack
\mbox{\tt +=}
\left \lbrack \begin{array}{cc}
H_{I_d \cup (\bnd{I} \cap J), I_d \cup (\bnd{I} \cap J)}
& H_{I_d \cup (\bnd{I} \cap J), \bnd{I} \cap \bnd{J}} \\
H_{\bnd{I} \cap \bnd{J}), I_d \cup (\bnd{I} \cap J)} & 0
\end{array} \right \rbrack
$
\item
Factor the front.
\par
$T_{J_e,J_e}
    =  (L_{J_e,J_e} + I)D_{J_e,J_e} (I + U_{J_e,J_e})$
\par
$T_{J_d \cup \bnd{J},J_e}
   = L_{J_d \cup \bnd{J},J_e} D_{J_e,J_e} (I + U_{J_e,J_e}) $
\par
$T_{J_e,J_d \cup \bnd{J}}
   = (L_{J_e,J_e} + I)D_{J_e,J_e} U_{J_e,J_d \cup \bnd{J}})$
\par
$H_{J_d,J_d}^J
   = T_{J_d,J_d} - L_{J_d, J_e} D_{J_e,J_e} U_{J_e,J_d}$
\par
$H_{J_d,\bnd{J}}^J
   = T_{J_d,\bnd{J}} - L_{J_d, J_e} D_{J_e,J_e} U_{J_e,\bnd{J}}$
\par
$H_{\bnd{J},J_d}^J
   = T_{\bnd{J},J_d} - L_{\bnd{J}, J_e} D_{J_e,J_e} U_{J_e,J_d}$
\end{enumerate}
The last step needs a bit of elaboration.
The factorization of a front is a complex process.
The matrix $D_{J_e,J_e}$ is diagonal or block diagonal.
Our codes try to detect large block pivots as a way of taking
advantage of the speed of BLAS3 kernels.
\par
Consider symmetric pivoting for now. (Nonsymmetric pivoting is much
the same but the notation gets more complicated.)
\begin{center}
\begin{minipage}{3.5 in}
\begin{tabbing}
XXX\=XXX\=XXX\=XXX\=XXX\kill
$J_e = J_d = \emptyset$ \\
while a pivot block $(\bfj, \bfj)$ can be found \\
\> $J_e := J_e \cup \bfj$,
   ${\widetilde J} := {\widetilde J} \setminus \bfj$ \\
\> $D_{\bfj,\bfj} = T_{\bfj,\bfj}$\\
\> $L_{{\widetilde J}\cup\bnd{J},\bfj}
      = A_{{\widetilde J}\cup\bnd{J},\bfj} D_{\bfj,\bfj}^{-1}$\\
\> $U_{{\widetilde J}\cup\bnd{J},\bfj}
      = D_{\bfj,\bfj}^{-1} A_{\bfj,{\widetilde J}\cup\bnd{J}} $ \\
\> $T_{{\widetilde J}, {\widetilde J}}
   := T_{{\widetilde J}, {\widetilde J}}
    - L_{{\widetilde J}, \bfj} D_{\bfj,\bfj} U_{\bfj,{\widetilde J}}$ \\
\> $T_{\bnd{J}, {\widetilde J}}
   := T_{\bnd{J}, {\widetilde J}}
    - L_{\bnd{J}, \bfj} D_{\bfj,\bfj} U_{\bfj,{\widetilde J}}$ \\
\> $T_{{\widetilde J}, \bnd{J}}
   := T_{{\widetilde J}, \bnd{J}}
    - L_{{\widetilde J}, \bfj} D_{\bfj,\bfj} U_{\bfj,\bnd{J}}$\\
end while \\
$J_d = {\widetilde J}$
\end{tabbing}
\end{minipage}
\end{center}
Note the matrix-matrix multiply with the explicit inverse
of $D_{\bfj,\bfj}$.
We actually compute the inverse
$D_{\bfj,\bfj}^{-1}$ as part of the test for acceptability
of the pivot block.
(See Section~\ref{section:PivotFinder} for more details.)
\par \bigskip \par
\noindent {\bf Parallel factorization }
\par \bigskip \par
We have one major assumption as we move towards a parallel
algorithm. {\it
Once a front is complete (all updates from descendent fronts
have been made and any postponed rows and columns from the children
have been assembled), the its factorization takes place inside
one thread or processor.}
This does not lead to a {\it scalable} algorithm
\cite{sch93-scalability}, but it is important for efficiency to
have the pivot selection process inside one thread of computation.
\par
In light of this constraint, let us look at the five steps of the
factorization.
Steps 1, 2, 4 and 5 must be done by one thread, the owner of the
front $J$.
It is Step 3 that must be parallelized across the processors.
Now for the major question: if front $I$ updates front $J$,
who performs the computation? There are three possibilities.
\begin{itemize}
\item
The owner of front $I$. (The fan-in method
\cite{ash90-fan-in}.)
\item
The owner of front $J$. (The fan-out method
\cite{geo87-fan-out}.)
\item
Some other processor. (The fan-both method
\cite{ash93-fan-both}.)
\end{itemize}
We have chosen the fan-in paradigm for this library.
It is a simple method, fairly efficient and can take advantage of
subtree-subcube mappings \cite{geo87-fan-out} better than either
the fan-out or fan-both methods.
Any algorithm will distribute the factor entries among the threads,
but there can be only one owner that computes
the factor pivots of a given front.
Furthermore, the fan-in algorithm features no exchange of factor
entries among the threads, rather it is partial updates, or
{\it aggregate} updates that are exchanged.
\par
Step 3 needs some modification for thread $q$.
\begin{itemize}
\item[$3^\prime.$]
For $I$ owned by thread $q$,
$I_e \ne \emptyset$
and $\bnd{I} \cap J \ne \emptyset$
\par
$\mbox{\qquad}\displaystyle
\left \lbrack \begin{array}{cc}
T^q_{J,J} & T^q_{J, \bnd{J}} \\
T^q_{\bnd{J},J} & 0
\end{array} \right \rbrack
\mbox{\tt -=}
\left \lbrack \begin{array}{c}
L_{\bnd{I}\cap J,I_e} \\
L_{\bnd{I}\cap \bnd{J},I_e}
\end{array} \right \rbrack
D_{I_e,I_e}
\left \lbrack \begin{array}{cc}
U_{I_e, \bnd{I} \cap J} & U_{I_e, \bnd{I}\cap \bnd{J}}
\end{array} \right \rbrack
$
\begin{tabbing}
XXX\=XXX\=XXX\=XXX\=\kill
If $J$ is owned by thread $q$ then \\
\> for each supporting thread $r$ \\
\>\> $\displaystyle
\left \lbrack \begin{array}{cc}
T^q_{J,J} & T^q_{J, \bnd{J}} \\
T^q_{\bnd{J},J} & 0
\end{array} \right \rbrack
\mbox{\tt +=}
\left \lbrack \begin{array}{cc}
T^r_{J,J} & T^r_{J, \bnd{J}} \\
T^r_{\bnd{J},J} & 0
\end{array} \right \rbrack
$ \\
\> end for \\
% else \\
% \> send
% $\displaystyle
% \left \lbrack \begin{array}{cc}
% T^q_{J,J} & T^q_{J, \bnd{J}} \\
% T^q_{\bnd{J},J} & 0
% \end{array} \right \rbrack
% $ to the owner of $J$ \\
end if
\end{tabbing}
\end{itemize}
Of course we are omitting the rendevous (in a threaded context)
or send/receive (in a distributed context) logic in the algorithm
description. It should be clear that the aggregate updates
(on the left)
and the delayed rows and columns
(on the right)
$$
T_J^q =
\left \lbrack \begin{array}{cc}
T^q_{J,J} & T^q_{J, \bnd{J}} \\
T^q_{\bnd{J},J} & 0
\end{array} \right \rbrack
\qquad \qquad
H^I =
\left \lbrack \begin{array}{cc}
H_{I_d \cup (\bnd{I} \cap J), I_d \cup (\bnd{I} \cap J)}
& H_{I_d \cup (\bnd{I} \cap J), \bnd{I} \cap \bnd{J}} \\
H_{\bnd{I} \cap \bnd{J}), I_d \cup (\bnd{I} \cap J)} & 0
\end{array} \right \rbrack
$$
must be made available to the threads
or communicated to the processors that need them.
\par
The presence of pivoting, and thus the possibility that a front may
not have any eliminated rows and columns, makes an implementation
somewhat tricky.
This is a key point and needs to be explained further.
It is simple to determine {\it prior to the factorization}
which threads will support which fronts, but it is possible that a
none of the fronts owned by a thread will have any eliminated rows
and columns that support some given front $J$.
The aggregate matrix $T_J^q$ would never been created but the owner
of front $J$ will be expecting it.
In short, the communication patterns of the factorization without
pivoting must be obeyed for the process to terminate satisfactorily.
If an aggregate matrix is indeed empty, meaning that due to delayed
rows and columns the updates that would have been performed will
not be forthcoming, the entries are not actually transfered,
but the notification is made.
\par \bigskip \par
\noindent {\bf Parallel solve }
\par \bigskip \par
Let us now consider the forward and backsolves.
We make one assumption: the data partition of the factors $L$, $D$
and $U$ will be maintained during the solve.
In other words, the entries in $L$, $D$ and $U$
are found in basic submatrices,
$L_{J_d \cup \bnd{J}, J_e}$,
$D_{J_e, J_e}$ and
$L_{J_e, J_d \cup \bnd{J}}$,
and these submatrices are not split
up across threads or processors.
There is still the concept of a thread or processor owning a front.
In the threaded code we do allow for different maps from fronts
to owners.\footnote{
For example, the forward and backsolves could be done with fewer
threads than the factorization.
}
\par
We do not actually compute $A = (L + I)D(I + U)$, instead we
compute $(L + I)(D + {\widehat U})$ where ${\widehat U} = DU$.
Recall, the eliminated rows and columns for a front, $J_e$,
are split into pivot blocks that we denote by $\bfj$.
By convention, let the boundary of $\bfj$ be
$$
\bnd{\bfj} = \{k\ |\ l_{k,j} \ne 0 \mbox{\ for some\ }j \in \bfj\}.
$$
Note, $\bnd{\bfj} \subset J_e \cup J_d \cup \bnd{J}$.
The serial solve $(I+L)(D+U)x = b$ is found in
Figure~\ref{fig:serial-solve}.
\begin{figure}
\caption{Serial forward and back solve}
\label{fig:serial-solve}
\begin{center}
\begin{minipage}{3.5 in}
\begin{tabbing}
XXX\=XXX\=XXX\=XXX\=XXX\=\kill
for $J$ in a post-order traversal \\
\> for $\bfj \in J_e$ in ascending order \\
\>\> $y_{\bfj} = b_{\bfj}$ \\
\>\> $b_{\bnd{\bfj}} :=
      b_{\bnd{\bfj}} - L_{\bnd{\bfj},\bfj} y_{\bfj}$ \\
\> end for\\
end for\\
for $J$ in a pre-order traversal \\
\> for $\bfj \in J_e$ in descending order \\
\>\> $y_{\bfj} := y_{\bfj}
        - {\widehat U}_{\bfj,\bnd{\bfj}} x_{\bnd{\bfj}}$ \\
\>\> $x_{\bfj} = D_{\bfj,\bfj}^{-1} y_{\bfj}$ \\
\> end for\\
end for
\end{tabbing}
\end{minipage}
\end{center}
\end{figure}
\par
As we distribute the forward solve across threads or processors,
we see that the right hand side vector $b$ accumulates updates of
the form $L_{\bnd{\bfj},\bfj} y_{\bfj}$ from descendents of $J$.
Since these updates are made in a distributed fashion, each thread
or processor needs a copy of $b$, or at least of copy of the
entries in $b$ that it will interact with.
The same holds in a slightly different sense for the backward solve.
Instead of a vector $b$ that needs to be gathered and accumulated,
it is a solution vector $x$ that needs to be scattered to threads
or processors that need it.
\par
A well designed map of fronts to threads or processors will no
doubt take advantage of locality within the front tree, e.g.,
a subtree-subcube map.
In these cases, the entries that {\it active} on a thread or
processor will be a fraction of the total entries.
It thus pays to take have local vectors $b^q$ and $x^q$ for thread $q$.
These vectors should contain only those entries that are active
on thread $q$.
This is particularly important when we are solving several right
hand sides at once.
\par
Figure~\ref{fig:parallel-solve} describes the
the parallel solve $(I+L)(D+U)x = b$ as done by thread $q$.
\begin{figure}
\caption{Parallel forward and back solve}
\label{fig:parallel-solve}
\begin{center}
\begin{minipage}{3.5 in}
\begin{tabbing}
XXX\=XXX\=XXX\=XXX\=XXX\=\kill
for $J$ in a post-order traversal \\
\> if  $J$ owned by $q$ then \\
\>\> gather $b_{J_e} = \sum_r b_{J_e}^r$ \\
\>\> for $\bfj \in J_e$ in ascending order \\
\>\>\> $y_{\bfj} = b_{\bfj}$ \\
\>\>\> $b^q_{\bnd{\bfj}} :=
      b^q_{\bnd{\bfj}} - L_{\bnd{\bfj},\bfj} y_{\bfj}$ \\
\>\> end for\\
\> else if $b_{J_e}^q \ne 0$ then \\
\>\> communicate $b_{J_e}^q$ to $b_{J_e}$ somehow \\
\> end if\\
end for\\
for $J$ in a pre-order traversal \\
\> if  $J$ owned by $q$ then \\
\>\> for $\bfj \in J_e$ in descending order \\
\>\>\> $y_{\bfj} := y_{\bfj}
        - {\widehat U}_{\bfj,\bnd{\bfj}} x^q_{\bnd{\bfj}}$ \\
\>\>\> $x_{\bfj} = D_{\bfj,\bfj}^{-1} y_{\bfj}$ \\
\>\> end for\\
\>\> store $x_{J_e}$ in $x^q_{J_e}$ \\
\>\> communicate $x_{J_e}$ to those who need it \\
\> else if $x_{J_e}$ needed by thread $q$ then \\
\>\> obtain $x_{J_e}$ and store in $x^q_{J_e}$ \\
\> end if \\
end for
\end{tabbing}
\end{minipage}
\end{center}
\end{figure}
\par
The interplay between local and global information can take many
forms.
In our present threaded solves there are global right hand side and
solution vectors that are protected by locks.
In the MPI version explicit messages will communicate information
among the processors.


% !Mode:: "TeX:UTF-8"

\chapter{The Theory of FDTD}
\section{Yee Cell}
Maxwell's equations are a set of equations which can be written in differential form or integral form. They are the foundation of macroscopic electromagnetic phenomenas. There are two kinds of numerical solver to Maxwell's equation. One kind of solvers were developed from integral form of Maxwell's equations, called integral equation solvers, including MoM, BEM etc. Another kind of solvers, like FDTD and FEM, were developed from differential form of Maxwell's equations, called differential equation solvers.

FDTD is based on Maxwell's equations in differential form, which are shown as follows. Then Maxwell's equations are modified by discretized in central-difference way.
\begin{equation}\label{ch2 eq:maxwellH}
\nabla\times\mathbf{\mathit{H}}=\frac{\partial \mathbb{\mathit{D}}}{\partial t}+\mathbf{\mathit{J}},
\end{equation}
\begin{equation}\label{ch2 eq:maxwellE}
\nabla\times\mathbf{\mathit{E}}=-\frac{\partial \mathbb{\mathit{B}}}{\partial t}-\mathbf{\mathit{J}}_m.
\end{equation}

%各向同性线性介质中的本构关系为
%\begin{equation}\label{bengouguanxi}
%\begin{cases}
%\mathbf{\mathit{D}}&=\varepsilon\mathbf{\mathit{E}}\\
%\mathbf{\mathit{B}}&=\mu\mathbf{\mathit{H}}\\
%\mathbf{\mathit{J}}&=\sigma\mathbf{\mathit{E}}\\
%\mathbf{\mathit{J}}_m&=\sigma_m \mathbf{\mathit{H}}
%\end{cases}.
%\end{equation}
%真空中$$\sigma=0$$，$$\sigma_m=0$$，以及
%\begin{eqnarray*}
%\varepsilon=\varepsilon_0=8.85\times 10^{-12}F/m\\
%\mu=\mu_0=4\pi\times 10^{-7}H/m.
%\end{eqnarray*}

In  Cartesian coordinate system，the equation\eqref{ch2 eq:maxwellH} and \eqref{ch2 eq:maxwellH} are written in following forms:
\begin{equation}\label{ch2 eq:3dmaxwellE}
\begin{cases}
\frac{\partial H_z}{\partial y}-\frac{\partial H_y}{\partial z}=\varepsilon\frac{\partial E_x}{\partial t}+\sigma E_x\\
\frac{\partial H_x}{\partial z}-\frac{\partial H_z}{\partial x}=\varepsilon\frac{\partial E_y}{\partial t}+\sigma E_y\\
\frac{\partial H_y}{\partial x}-\frac{\partial H_x}{\partial y}=\varepsilon\frac{\partial E_z}{\partial t}+\sigma E_z
\end{cases}
\end{equation}

\begin{equation}\label{ch2 eq:3dmaxwellH}
\begin{cases}
\frac{\partial E_z}{\partial y}-\frac{\partial H_y}{\partial z}=-\mu\frac{\partial H_x}{\partial t}-\sigma_m H_x\\
\frac{\partial E_x}{\partial z}-\frac{\partial H_z}{\partial x}=-\mu\frac{\partial H_y}{\partial t}-\sigma_m H_y\\
\frac{\partial E_y}{\partial x}-\frac{\partial H_x}{\partial y}=-\mu\frac{\partial H_z}{\partial t}-\sigma_m H_z
\end{cases}.
\end{equation}

The six equations in \eqref{ch2 eq:3dmaxwellE} and \eqref{ch2 eq:3dmaxwellH} are a set of partial differential equations in space-time formulations form which represent each filed vector component. They are hard to deal in that form, so we need to discretize them in space and time at first.Let $u(x,y,z,t)$ represent any field vector component of $\mathbf{\mathit{E}}$ or $\mathbf{\mathit{H}}$ in Cartesian coordinate system. On the aspect of space, we assume that the discreteness on space is uniform, which means the space steps, also called the lengths of grids, are equal to each other in $x$, $y$, and $z$ directions and written as $\Delta x$, $\Delta y$, and $\Delta z$ respectively. We also use $i$, $j$, and $k$ to represent the grid index in directions of $x$, $y$, and $z$ respectively. On the aspect of time, wo assume the discreteness is uniform, too. Besides, we adopt the symbol $\Delta t$ to represent the length of time step, which also called the distance between two iterations, and $n$ to represent the index of time steps. After all, we can represent any field vector component in the following notion to indicate the location where the field vector components are sampled in space and time:
\begin{equation}
u(x,y,z,t)=u(i\Delta x,j\Delta y,k\Delta z,n\Delta t)=u^n(i,j,k).
\end{equation}

There are three forms of finite difference, forward, backward, and central difference, which are considered commonly. Here we pick the central difference as it has second-order numerical accuracy. Let us take the $x$ direction as an example to illustrate a field vector component's spatial first partial derivative:
\begin{equation}\label{ch2 eq:space discrete}
\frac{\partial u^n(i,j,k)}{\partial x} \approx \frac{
	u^n(i+\frac{1}{2},j,k)-u^n(i-\frac{1}{2},j,k)
	}{\Delta x}.
\end{equation}
And the its first partial derivative on time is:
\begin{equation}\label{ch2 eq:time discrete}
\frac{\partial u^n(i,j,k)}{\partial t} \approx \frac{
	u^{n+\frac{1}{2}}(i,j,k)-u^{n-\frac{1}{2}}(i,j,k)
}{\Delta x}.
\end{equation}

Now we have discretized the six equations in \eqref{ch2 eq:3dmaxwellH} and \eqref{ch2 eq:3dmaxwellE}. The next step we should do is considering how we position those discrete points of every field vector component. The answer is Yee cell.

In space, we position those discrete points like what illustrated in \ref{ch2 fig: yee cell}. This is the spatial structure of Yee cell.

\begin{figure}[hp]
\centering
		\begin{tikzpicture}
		\def \len {6}
		\def \hlen {3}
		\def \coe {-0.75}

		%back rectangle wigh dashline
		\draw(0,0) rectangle +(\len,\len);
		\draw [dashed] ($(0,0)+0.5*(0,\len)$) -- +(\len,0);
		\draw [dashed] ($(0,0)+0.5*(\len,0)$) -- +(0,\len);

		%front rectangle wigh dashline
		\draw($(0,0)+\coe*(\hlen,\hlen)$) rectangle +(\len,\len);
		\draw [dashed] ($(0,0)+(0,\hlen)+\coe*(\hlen,\hlen)$) -- +(\len,0);
		\draw [dashed] ($(0,0)+(\hlen,0)+\coe*(\hlen,\hlen)$) -- +(0,\len);

		%connect two rectangle
		\draw (0,0) -- ($(0,0)+\coe*(\hlen,\hlen)$);
		\draw ($(0,0)+(0,\len)$) -- ($(0,0)+(0,\len)+\coe*(\hlen,\hlen)$);
		\draw ($(0,0)+(\len,0)$) -- ($(0,0)+(\len,0)+\coe*(\hlen,\hlen)$);
		\draw ($(0,0)+(\len,\len)$) -- ($(0,0)+(\len,\len)+\coe*(\hlen,\hlen)$);

		%all dashline
		\draw [dashed] ($(0,0)+(0,\hlen)$) -- ($(0,0)+(0,\hlen)+\coe*(\hlen,\hlen)$);
		\draw [dashed] ($(0,0)+(\hlen,0)$) -- ($(0,0)+(\hlen,0)+\coe*(\hlen,\hlen)$);
		\draw [dashed] ($(0,0)+(\len,\hlen)$) -- ($(0,0)+(\len,\hlen)+\coe*(\hlen,\hlen)$);
		\draw [dashed] ($(0,0)+(\hlen,\len)$) -- ($(0,0)+(\hlen,\len)+\coe*(\hlen,\hlen)$);

		\draw [dashed] ($(0,0)+0.5*\coe*(\hlen,\hlen)$) -- ++(\len,0) -- ++(0,\len) -- ++(-\len,0) -- cycle;

		%axis
		\draw [->] (0,0) -- +(\len+2,0) node[right]{$y$};
		\draw [->] (0,0) -- +(0,\len+2) node[above]{$z$};
		\draw [->] (0,0) -- +($(0,0)-(\hlen,\hlen)$) node[below,left]{$x$};

		%nodes
		%Hx
		\node[shape=circle,fill=cyan,inner sep=0pt] at (\hlen,\hlen) {$H_x$};
		\node[shape=circle,fill=cyan,inner sep=0pt] at ($(\hlen,\hlen)+\coe*(\hlen,\hlen)$) {$H_x$};
		%Hz
		\node[shape=circle,fill=cyan,inner sep=0pt] at ($(0,0)+(\hlen,0)+0.5*\coe*(\hlen,\hlen)$) {$H_z$};
		\node[shape=circle,fill=cyan,inner sep=0pt] at ($(0,0)+(\hlen,0)+0.5*\coe*(\hlen,\hlen)+(0,\len)$) {$H_z$};
		%Hy
		\node[shape=circle,fill=cyan,inner sep=0pt] at ($(0,0)+(0,\hlen)+0.5*\coe*(\hlen,\hlen)$) {$H_y$};
		\node[shape=circle,fill=cyan,inner sep=0pt] at ($(0,0)+(0,\hlen)+0.5*\coe*(\hlen,\hlen)+(\len,0)$) {$H_y$};
		%Ez
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(0,\hlen)$) {$E_z$};
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(0,\hlen)+\coe*(\hlen,\hlen)$) {$E_z$};
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(0,\hlen)+(\len,0)$) {$E_z$};
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(0,\hlen)+\coe*(\hlen,\hlen)+(\len,0)$) {$E_z$};
		%Ex
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(0,\hlen)+0.5*\coe*(\hlen,\hlen)+(0,\hlen)$) {$E_x$};
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(0,\hlen)+0.5*\coe*(\hlen,\hlen)+(\len,0)+(0,\hlen)$) {$E_x$};
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(0,\hlen)+0.5*\coe*(\hlen,\hlen)+(0,-\hlen)$) {$E_x$};
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(0,\hlen)+0.5*\coe*(\hlen,\hlen)+(\len,0)+(0,-\hlen)$) {$E_x$};
		%Ey
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(\hlen,\hlen)+(0,\hlen)$) {$E_y$};
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(\hlen,\hlen)+\coe*(\hlen,\hlen)+(0,\hlen)$) {$E_y$};
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(\hlen,\hlen)+(0,-\hlen)$) {$E_y$};
		\node[shape=circle,fill=pink,inner sep=0pt] at ($(\hlen,\hlen)+\coe*(\hlen,\hlen)+(0,-\hlen)$) {$E_y$};
		\end{tikzpicture}
\caption{The spatial discrete structure of Yee cell}\label{ch2 fig: yee cell}
\end{figure}

We can see from the figure \ref{ch2 fig: yee cell} that in Yee cell each three components of $\mathbf{\mathit{E}}$ and $\mathbf{\mathit{H}}$ are discretized on the surface. Electric field components $\mathit{E}_x$, $\mathit{E}_y$, and $\mathit{E}_z$ are on the center of edges, and magnetic field components $\mathit{H}_x$, $\mathit{H}_y$, and $\mathit{H}_z$ are on the center of surfaces. In this way, for each electric discrete point there are four magnetic discrete points encircling it. Conversely, for each magnetic discrete point there will be four electric discrete points encircling it. This way of distributing discrete points fit into Faraday's law and Ampere's law in nature. In time aspect, Yee cell adopted the time-stepping manner. For any discrete point in space, the updated value of the electric filed in time is dependent on the stored value of magnetic field of the last time. Iterating the process in a marching-in-time way, we can make an analog to the continuous electromagnetic waves's propagation.

In summary, Yee cell can describe the interaction between electrics and magnetics naturally. Given a specific problem with specific coefficients, initial state, and boundary condition, we can use FDTD to obtain the distribution of electromagnetic waves in any given transient.

To indicate the location of discrete points in space and time, we need to give any location of Yee cell a serial number. Here, we follow this rule: in any direction, the serial number of Yee cell's edges is integer, and the serial number of the center of Yee cell's edge is half integer; all time location of electric discrete points are integer, and all time location of magnetic discrete points are half integer. For each field vector component, the serial number in space is illustrated in the following table.

\begin{table}
\caption{The serial number of components of $\mathbf{\mathit{E}}$ and $\mathbf{\mathit{H}}$ in Yee cell}
	\centering
	\begin{tabular}{ccccc}
		\toprule
		\multirow{2}{5em}{Field components} & \multicolumn{3}{c}{The location in space} & \multirow{2}{6em}{The location in time}\\
		\cline{2-4}
		& $x$ & $y$ & $z$ &\\

		\midrule
		$E_x$ & $i+\frac{1}{2}$ & $j$ & $k$ & \multirow{3}{1em}{$n$}\\
		\cline{1-4}
		$E_y$ & $i$ & $j+\frac{1}{2}$ & $k$ & \\
		\cline{1-4}
		$E_z$ & $i$ & $j$ & $k+\frac{1}{2}$ & \\
		\cline{1-5}

		$H_x$ & $i$ & $j+\frac{1}{2}$ & $k+\frac{1}{2}$ & \multirow{3}{3em}{$n+\frac{1}{2}$}\\
		\cline{1-4}
		$H_y$ & $i+\frac{1}{2}$ & $j$ & $k+\frac{1}{2}$ & \\
		\cline{1-4}
		$H_z$ & $i+\frac{1}{2}$ & $j+\frac{1}{2}$ & $k$ & \\
		\bottomrule
	\end{tabular}
\end{table}

\section{The Updating Formulations}

Let us take the \eqref{ch2 eq:3dmaxwellE} as an example to explain the how to update a field vector component and the updating formulations. According to the \eqref{ch2 eq:space discrete} and \eqref{ch2 eq:time discrete}, we change \eqref{ch2 eq:3dmaxwellH} into the discrete form, which shown below:

\begin{equation}\label{ch2 eq:ex discrete}
\begin{split}
E^{n+1}_{x}\left( i+\frac{1}{2},j,k \right)=&CA(m) \cdot E^{n}_{x}\left( i+\frac{1}{2},j,k \right)+CB(m)\\
{}&\cdot\left[
\frac{H^{n+\frac{1}{2}}_{z}\left(i+\frac{1}{2},j+\frac{1}{2},k\right)-H^{n+\frac{1}{2}}_{z}\left(i+\frac{1}{2},j-\frac{1}{2},k\right)}{\Delta y}\right.\\
{}&-
\left.\frac{H^{n+\frac{1}{2}}_{y}\left(i+\frac{1}{2},j,k+\frac{1}{2}\right)-H^{n+\frac{1}{2}}_{z}\left(i+\frac{1}{2},j,k-\frac{1}{2}\right)}{\Delta z}
\right].
\end{split}
\end{equation}

And there has
\begin{equation}
CA(m)=\frac{
	1-\frac{\sigma(m)\Delta t}{2\varepsilon(m)}
	}{
	1-\frac{\sigma(m)\Delta t}{2\varepsilon(m)}
	},
\end{equation}
and
\begin{equation}
CB(m)=\frac{
	\frac{\Delta t}{\varepsilon(m)}
}{
1+\frac{\sigma(m)\Delta t}{2\varepsilon(m)}
}.
\end{equation}\label{ch2 eq:ex discrete simple}
We use $m$ to represent the spatial location the discrete point have which is at the right of equal sign.

To explain the algorithm in a clear way, we assume all mediums are lossless medium, ie $\sigma=\sigma(m)=0$. Hence, there has $CA(m)=1$, and $CB(m)=\frac{\Delta t}{\varepsilon(m)}$. So, now, \eqref{ch2 eq:ex discrete} is

\begin{equation}
\begin{split}
E^{n+1}_{x}\left( i+\frac{1}{2},j,k \right)=&E^{n}_{x}\left( i+\frac{1}{2},j,k \right)+\frac{\Delta t}{\varepsilon(m)}\\
{}&\cdot\left[
\frac{H^{n+\frac{1}{2}}_{z}\left(i+\frac{1}{2},j+\frac{1}{2},k\right)-H^{n+\frac{1}{2}}_{z}\left(i+\frac{1}{2},j-\frac{1}{2},k\right)}{\Delta y}\right.\\
{}&-
\left.\frac{H^{n+\frac{1}{2}}_{y}\left(i+\frac{1}{2},j,k+\frac{1}{2}\right)-H^{n+\frac{1}{2}}_{z}\left(i+\frac{1}{2},j,k-\frac{1}{2}\right)}{\Delta z}
\right].
\end{split}
\end{equation}

In the same way we modified the field vector components of $\mathbf{\mathit{H}}$. Let us take the $H_x$ as an example, now the first equation in \eqref{ch2 eq:3dmaxwellH} is:

\begin{equation}\label{ch2 eq:hx discrete simple}
\begin{split}
H^{n+\frac{1}{2}}_{x}\left( i,j+\frac{1}{2},k+\frac{1}{2} \right)=&H^{n-\frac{1}{2}}_{x}\left( i,j+\frac{1}{2},k+\frac{1}{2} \right)+\frac{\Delta t}{\mu}\\
{}&\cdot\left[
\frac{
	E^n_y\left(i,j+\frac{1}{2},k+1\right)-E^n_y\left(i,j+\frac{1}{2},k\right)
	}{\Delta z}\right.\\
{}&+\left.
\frac{
	E^n_z\left(i,j,k+\frac{1}{2}\right)-E^n_z\left(i,j+1,k+\frac{1}{2}\right)
}{\Delta z}
\right]
\end{split}
\end{equation}
All other field vector components can be processed in the exactly same way we stated above. Therefore, we obtain each field vector component's discrete form updating formulation.

\section{Courant Condition}
According to the last section, we know that instead of a set of continuous partial differential equations, FDTD solving a set of discretized Maxwell's equations. So, like other numerical approximate methods, FDTD has a necessary condition for convergence while solving partial differential equations numerically. As a consequence, the time step must be less than a certain time.

At first, we consider the time step, $\Delta t$. As any field can be deposited into several harmonic electromagnetic fields, we examine the limitation of $\Delta t$ in harmonic electromagnetic field.

Here is a harmonic electromagnetic field:
\begin{equation}\label{stability: shi xie chang}
u(x,y,z,t)=u_0exp(j\omega t).
\end{equation}
The first-order partial differential equation of \eqref{shi xie chang} respect to time is：
\begin{equation}\label{stability: u partial t}
\frac{\partial u}{\partial t}=j\omega u.
\end{equation}

By using central difference approximations to the left side of equation \eqref{stability: u partial t}, we obtain:
\begin{equation}\label{stability: chafen}
\frac{u^{n+\frac{1}{2}}-u^{n-\frac{1}{2}}}{\Delta t}=j\omega u^n,
\end{equation}
In which $u_n=u(x,y,z,n\Delta t)$.

Let the growth factor $q$ be:
\begin{equation}\label{stability: q}
	q=\frac{u^{n+\frac{1}{2}}}{u^n}=\frac{u^n}{u^{n-\frac{1}{2}}}.
\end{equation}

Then, combining the equation \eqref{stability: q} and \eqref{stability: chafen}, we obtain this equation:
\begin{equation}
q^2-j\omega \Delta tq-1=0
\end{equation}\label{ch2 eq:temp eq}

Solving the equation \eqref{ch2 eq:temp eq}, we will have:
\begin{equation}
q=\frac{j\omega \Delta t}{2}\pm\sqrt{1-\left(\frac{\omega \Delta  t}{2}\right)^2}.
\end{equation}

If we want the value of fields converge along the marching of time, the condition $|q|\leqslant 1$ must be satisfied. Hence, there is
\begin{equation}\label{ch2 eq:stability: dt}
\frac{\omega \Delta t}{2}\leqslant 1.
\end{equation}

Equation \eqref{ch2 eq:stability: dt} is the necessary condition of $\Delta t$ required by a field which need to be stable.

For Maxwell's equations, which have six field components, we know that all rectangular components of electromagnetic field fit in the homogeneous wave equation which is following:
\begin{equation}\label{stability: qicibodong}
\frac{\partial^2 f}{\partial x^2}+\frac{\partial^2 f}{\partial y^2}+\frac{\partial^2 f}{\partial z^2}+\frac{\omega^2}{c^2}f=0.
\end{equation}

As any wave can be expanded to plain waves, so we consider the solution of plain waves of homogeneous wave equation:
\begin{equation}\label{stability: pingmianbo}
u(x,y,z,t)=u_0 exp[-j(k_x x+k_y y+k_z z-\omega t)].
\end{equation}

Substituting the equation \eqref{stability: pingmianbo} into \eqref{stability: qicibodong}, and discretize the result by using central difference, then we obtain the following equations:
\begin{equation}\label{stability: qicibodong lisan}
\frac{\sin^2 \left( \frac{k_x\Delta x}{2} \right)}{\left(\frac{\Delta x}{2}\right)^2}+
\frac{\sin^2 \left( \frac{k_y\Delta y}{2} \right)}{\left(\frac{\Delta y}{2}\right)^2}+
\frac{\sin^2 \left( \frac{k_z\Delta z}{2} \right)}{\left(\frac{\Delta z}{2}\right)^2}-
\frac{\omega^2}{c^2}=0.
\end{equation}
The $c$ is the speed of light in media among it.

Simplify the equation \eqref{stability: qicibodong lisan} and substitute it into \eqref{ch2 eq:stability: dt}. Then we obtain
\begin{equation}
\left(\frac{c\Delta t}{2}\right)^2
\left[
\frac{\sin^2 \left( \frac{k_x\Delta x}{2} \right)}{\left(\frac{\Delta x}{2}\right)^2}+
\frac{\sin^2 \left( \frac{k_y\Delta y}{2} \right)}{\left(\frac{\Delta y}{2}\right)^2}+
\frac{\sin^2 \left( \frac{k_z\Delta z}{2} \right)}{\left(\frac{\Delta z}{2}\right)^2}
\right]=
\left(\frac{\omega\Delta t}{2}\right)^2\leqslant 1.
\end{equation}
This equation is true under the following condition:
\begin{equation}\label{stability: courant}
c\Delta t\leqslant
\frac{1}{
	\sqrt{\frac{1}{(\Delta x)^2}\frac{1}{(\Delta y)^2}+\frac{1}{(\Delta z)^2}}
	}.
\end{equation}
The equation \eqref{stability: courant} give out the relationship between time step $\Delta t$ and space step $\Delta x$, which called courant condition.

\section{The limitation of $\Delta x$}
In the last section, the relationship between the time step and the space step was been determined. Taking the 1D situation as an instance, from the equation \eqref{stability: qicibodong lisan} we have:
\begin{equation}
\frac{\sin^2 \left( \frac{k_x\Delta x}{2} \right)}{\left(\frac{\Delta x}{2}\right)^2}-
\frac{\omega^2}{c^2}=0.
\end{equation}\label{ch2 eq:dispersion}

We can observe the fact from the equation \eqref{ch2 eq:dispersion} that the dispersion can be avoided only if $\Delta x\rightarrow 0$. So, we need to evaluate to what extent the $\Delta x$ can be treated as closed to 0 in numerical approximation. According to triangle functions, when $\phi\leqslant \pi/12$, $sin\phi\approx\phi$. So, there has
\begin{equation}\label{stability: space}
	\Delta x\leqslant \frac{2\pi}{12k}=\frac{\lambda}{12}.
\end{equation}

Equation \eqref{stability: space} is the requirement for space step $\Delta x$. For circumstances of 2D or 3D, they are the same as 1D situation, just make space steps of each dimension meet the condition \eqref{stability: space}. To signals whose band are wide and belong to non-homogeneous waves, make sure the space step meet the requirement \eqref{stability: space} of the shortest wave length.

\section{Boundary conditions of FDTD}
In theory, by following the rules of FDTD we can simulate almost every electromagnetic wave in the infinite space and the length of time is infinite, too. However, as the power of computation is limited, and problems are always have significant big size, we can, and should simulate the the waves in the target area we concerned even in fact those waves are propagating in the infinite space. One possible way to do that is to design boundary conditions properly. Given a problem, we can compute discrete field points which are in the target area to observe how the wave changes in the whole area. There are two kinds of boundary conditions, absorbing boundary conditions (ABC) and truncating boundary conditions (TBC). In reality, people always adopt ABC, which allow them simulate the situation a wave propagating in the infinite space. Among ABCs, Mur and perfectly matched layer (PML) are two main boundary conditions.

\section{The ways of parallel computing of FDTD}

To satisfy the Courant condition, the grids of Yee cell must be fine enough, which means the number of discrete field points we need to compute will be significantly massive, therefore, solving a real problem in a big size seems not practical. Parallel computing, as to this problem, is a quite efficient solution.

There are three level of parallelism. The top level is task level. In this level, a huge task need to be completed is divided into several independent smaller tasks. After those smaller tasks computed by some computing cores, we merge the solutions of those smaller tasks into a integral solution, which answers the original task. According to the parallelism nature of FDTD, that all computing cores need exchange some boundary data with its adjacent computing cores, FDTD is suitable to be computed in parallel computing. So far, we have massage passing interface (MPI) method and parallel virtual machine (PVM) method of this parallelism level.

The lower level of parallelism is instruction level parallelism. This level of parallelism always considered by CPU manufacturers, hardly by users.

The lowest level of parallelism if data level. In this level, several data can be evaluated by one instruction simultaneously if a task have been reorganized appropriately. In FDTD, as all discrete field points of the same field vector component have a shared updating formulation, we can compute several discrete field points by every single instruction when those points belong to the same field vector component. For parallelism in this level, there are some implementations of FDTD by using vector processor instruction sets.

\section{Conclusion}
In this chapter, de discussed several aspects of FDTD. First, we introduce the theories of FDTD algorithm. Then we describe the updating formulations for every field vector components, which are the key of FDTD. After that, the necessary condition, Courant condition was stated. Satisfying this condition make sure all field vector components keep stable and being convergent, which make FDTD useful in reality. Then we discussed the boundary conditions of FDTD, which influence the accuracy of FDTD. Finally, we introduced several ways of parallel computing to make FDTD faster. All things we discussed above in this chapter are the foundations of the works in this paper.


\section{Introduction}

One of the important scientific computing problems is computing the real roots
of polynomials. It has wide applications in computer graphics. For the
polynomials with low order, like quadratic or cubic, we can use formulas to get
roots directly. However, according to the Abel–Ruffini theorem, there is no
solution in radicals to general polynomial equations of five degrees or higher
with arbitrary coefficients\cite{Abel-Ruffini}.

There are a lot of algorithms to find the real roots of polynomials. Although
most root-finding algorithms, Newton’s method for example, may produce some real
roots, the convergence of algorithms are not guaranteed. Furthermore, these
methods cannot generally certify having found all real roots. Which means if
such methods do not find any root, one cannot know if there are real roots or
not. Moreover, there are some cases where one does not need the exact roots of
polynomial equations. Take ray tracing as an example. When computing the
distance from intersection point to ray’s end point, the object with smallest
distance is important to us, instead of exact distance to point of intersection.
In these cases, computing exact roots is not very essential.

Real-root isolation is an appropriate method for the above problems. It will
generate a sequence of disjoint intervals. Each interval contains only one real
root of the polynomial. And combining these intervals together, all real roots
could be found. Besides that, isolating the roots instead of computing them out
might speed up the cases that do not need exact roots, as mentioned above.

This project aims to implement a real-root isolation program based on Budan’s
theorem and continued fraction. These two methods are based on Descartes’ rule
of signs, which describes how to get information on the number of positive real
roots of a polynomial and first introduced by René Descartes\cite{rule_of_sign}. Budan’s theorem,
developed from Descartes’ rule of signs, provides the methods to bound the
number of real roots of a polynomial in an interval\cite{Budan}. Vincent introduced the
continued fraction method in his work in 1834\cite{Vincent}. Both methods work only on
square-free polynomials. Therefore, this project takes Yun’s algorithm to
perform square free decomposition\cite{Yuns}.

The organization of this report is as follows. Section \ref{methods} will introduce the
theory basis of this project. Section \ref{implementation} describes how the program implemented
and what have been tried to optimize. Running time comparison and analysis of
how error propagation through computation will be discussed in Section
\ref{analysis}. Finally, a conclusion is drawn in Section \ref{conclusion}.


% ------------------------------------------------------ %
%  BIOE60011 Probability and Statistics For Bioengineers %
%  Formula Booklet for Practice and Exams %
% ----------------------------------------------------- %
% Author:
%           Binghuan Li  <binghuan.li19@imperial.ac.uk>,
%           Peter Xie    <peter.xie19@imperial.ac.uk>
% ------------------------------------------------------ %
%  Reviewer:
%           Zhuang Liu, Bishr Al-Badri,
%           Eliott Stoclet, Rea Tresa
% ------------------------------------------------------ %
%  Update Date: 29th December 2021  %
% ------------------------------------------------------ %
%  Graph credentials by Dr Joseph van Batenburg-Sherwood %
% ------------------------------------------------------ %
% This work is licensed under a Creative Commons Attribution 4.0 International License %
% ------------------------------------------------------ %

% keep the font size by default
\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

% set font
\usepackage[sfdefault]{arimo}

% set page margin
\usepackage[top=2cm, bottom=1cm, left=2cm, right=2cm]{geometry}

% set math env
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{mathrsfs}
\usepackage{mathtools}
\usepackage{amssymb}
\usepackage{changepage}
% set graphics env
\usepackage{float}
\usepackage{graphics}

% table scaling
\usepackage{adjustbox}

% write pdf info
\usepackage{hyperref}
%\hypersetup{hidelinks}
\hypersetup{pdfauthor={Li, Binghuan},
            pdftitle={Probs-and-Stats, Formula Sheet},
            pdfsubject={last update date: \today},
            }

% env to insert external pdf
\usepackage{pdfpages}

\usepackage{booktabs}

\usepackage{framed}

% set section title size
\usepackage{titlesec}
\titleformat*{\section}{\large\bfseries}
\titleformat*{\subsection}{\normalsize\bfseries}

% caption setting env
\usepackage{caption}
\captionsetup[table]{labelformat=empty}

% set header/footer
\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\lhead{\textit{Probability and Statistics for Bioengieering - Formula Sheet}}
\rhead{\thepage}

%% start the doc!
\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Mean, Median and Standard Deviation}

\begin{table}[H]
    \centering
    \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c | c}
        Mean & Median & Standard Deviation & MAD\\ [0.7ex] \hline
        $\displaystyle \bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_{i} = \frac{x_{1}+x_{2}+...+x_{n}}{n}$ &
        $\Tilde{x} = \begin{cases}
        x(\frac{n+1}{2}) \\
        0.5(x(\frac{n}{2})+x(\frac{n}{2}+1))
        \end{cases}$ &
        $\displaystyle s = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n} (x_{i}-\bar{x})^{2}}$
        & $\displaystyle MAD = k|\widetilde{x_{i} - \hat{x}}|$ \\
          &  &  &  $k = 1.4826$\\
    \end{tabular}
    \end{adjustbox}
\end{table}

\begin{itemize}
    \item[-] $s$, $s^{2}$ and $\bar{x}$ denote \underline{sample} S.D., variance and mean.
    \item[-] $\sigma$, $\sigma^{2}$ and $\mu$ denote \underline{population} S.D., variance and mean (expected value).\\

    \item[-] \textbf{Trimmed Mean at $p\%$} - Remove p/2 percentile from both ends of the data and calculate the mean
    \item[-] \textbf{Standard Error on the mean} -
    $\displaystyle S_{\bar{X}} = \frac{S_{X}}{\sqrt{n}}$, where $n$ is the sample size, $S_{X}$ is the standard deviation
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Probability Theory}
\subsection*{Set Theory}
\begin{table}[H]
    \centering
    \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c |c}
        Union & Intersection & Complement & Relative Complement\\ [0.7ex] \hline \vspace{1ex}
       A \underline{OR} B & A \underline{AND} B & \underline{NOT} A & A \underline{AND} \underline{NOT} B \\
       \includegraphics[width=0.2\textwidth]{./images/AorB.pdf}&\includegraphics[width=0.2\textwidth]{./images/AandB.pdf}&\includegraphics[width=0.2\textwidth]{./images/notA.pdf}&\includegraphics[width=0.2\textwidth]{./images/Aand_notB.pdf}\\
       $P(A\cup B) = P(A)+P(B)-P(A\cap B)$ &&$P(A') = 1 - P(A)$ &
    \end{tabular}
    \end{adjustbox}
\end{table}

%%
\subsection*{Counting}
\begin{itemize}
\item Product Rule - $\displaystyle n = n_{1}\times n_{2}\times n_{3}\times...\times n_{k}$
    \item Permutations: Ordered, dependent selections (without replacement) -
    $\displaystyle P_{k,n} = \frac{n!}{(n-k)!}$
    \item Combinations: Unordered selections (without replacement) -
    $\displaystyle C_{k,n} = \frac{P_{k,n}}{k!} = {n \choose k} = \frac{n!}{k!(n-k)!}$
\end{itemize}

%%
\subsection*{Conditional Probability}
\begin{itemize}
    \item Conditional probability: knowing that A has happened, the probability of B given A - $\displaystyle P(B|A) = \frac{P(B \cap A)}{P(A)}$
    \item Bayes Theorem - $\displaystyle P(A_{j}|B) = \frac{P(B|A_{j})P(A_{j})}{\sum_{i=1}^{k}P(B|A_{i})P(A_{i})} $\\\\
    (Applies if events A are mutually exclusive and exhaustive (defines the whole set)).
\end{itemize}

%%
\subsection*{Independence}
\begin{itemize}
    \item Independent: event A is unaffected by event B
    \[P(A|B) = P(A), \quad\quad P(B|A) = P(B), \quad\quad P(A\cap B)=P(A)P(B)\]
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Probability Distributions and Discrete Random Variables}
\subsection*{Binomial and Poisson Distribution}
\begin{minipage}{.8\textwidth}
\begin{table}[H]
    \centering
    \begin{tabular}{l c c}
        & Binomial Distribution & Poisson Distribution\\ [0.7ex] \hline \vspace{1ex}
        Probability Density Function & $ \displaystyle b(x;n,p) ={ n \choose x} p^{x}(1-p)^{n-x}$  & $\displaystyle po(x,\lambda) = \frac{e^{-\lambda}\lambda^{x}}{x!}$  \\ [0.5ex]
        Expected Value, $E(x)$ & $E(x) = np$ & $E(x) = \lambda$\\ [0.5ex]
        Variance, $V(x)$ & $\displaystyle V(x) = np(1-p) = npq$ & $\displaystyle V(x) = \lambda(1-\frac{\lambda}{n})$  \\ [0.5ex]
    \end{tabular}
\end{table}
\end{minipage}
\begin{minipage}{.2\textwidth}
\begin{framed}
Note: \\\\
$V(x)$ can also be expressed as $V(x)=E(x^{2})-(E(x))^{2}$.
\end{framed}

\end{minipage}

%%
\subsection*{Discrete Random Variables}
% \begin{adjustwidth}{-100pt}{-100pt}
\begin{table}[H]
    \centering
    \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c}
        Bernoulli RV & Expected Value & Variance\\ [1.0ex] \hline
        $p(x;\alpha) = \begin{cases}
        1-\alpha, & if \ x=0 \\
        \alpha, & if \ x=1\\
        0, & otherwise
        \end{cases}$
        &
        $\displaystyle E(x)=\mu_{X}=\sum_{x\in D}xp(x)$
        &
        $\displaystyle V(X) = \sigma_{X}^{2}=\sum_{x\in D}(x-\mu_{X})^{2}p(x)=E[(X-\mu_{X})^{2}]$
    \end{tabular}
    \end{adjustbox}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Continuous Random Variables and Joint Probability}
\subsection*{Random Variables}
\begin{table}[H]
    \centering
    \begin{tabular}{|l|c|}
    \hline
     \addlinespace[.3ex] Probability density function & $\displaystyle P(a \leq X \leq b) = \int_{a}^{b}f(x)dx$ \\ [.5ex] \hline
    \addlinespace[.3ex] Cumulative distribution function & $\displaystyle F(x) = P(X\leq x) = \int_{-\infty}^{x} f(y)dy \  \text{and} \ \int_{-\infty}^{+\infty} f(x)dx = 1$ \\ [.5ex] \hline
    \addlinespace[.3ex] Expected value & $\displaystyle \mu_{X} = E(X) = \int_{-\infty}^{+\infty} xf(x) dx$ \\ [.5ex] \hline
    \addlinespace[.3ex] Variance & $\displaystyle \sigma_{X}^{2} = V(X) = \int_{-\infty}^{+\infty} (x-\mu)^{2}f(x) dx$ \\ [.3ex] \hline
    \end{tabular}
\end{table}

\subsection*{Joint Probability Distributions}
\begin{itemize}
    \item Discrete Joint random variables
    \begin{table}[H]
    \centering
    % \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c}
        Joint probability mass function (PMF) & Expected Value & Marginal PMFs\\ [0.7ex] \hline
        \addlinespace[.4ex] $p(x,y)=P(X=x$ and $Y=y)$
        &
        $\displaystyle E[h(X,Y)]=\sum_{X}\sum_{Y}h(x,y)p(x,y)$
        &
        $P_{X}(x)=\sum_{y}p(x,y)$\\

        $\sum_{x}\sum_{y}p(x,y)=1$
        &
        &
        $P_{Y}(y)=\sum_{x}p(x,y)$\\
    \end{tabular}
\end{table}

    \item Continuous Joint random variables
    \begin{table}[H]
    \centering
   \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c| c| c}
        Joint probability mass function (PDF) & Expected Value & Marginal PDFs\\ [0.7ex] \hline
        \addlinespace[.4ex] $\displaystyle P[(X,Y)\in A]=\int\int_{A} f(x,y) dx dy$
        &
        $\displaystyle E[h(X,Y)]=\int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty}h(x,y)f(x,y)dx dy$
        &
        $\displaystyle f_{X}(x)=\int_{-\infty}^{+\infty} f(x,y)dy$ \\ [2ex]

        &
        $\displaystyle P[a\leq X\leq b, c \leq Y \leq d]=\int_{c}^{d}\int_{a}^{b}f(x,y)dx dy$
        &
        $\displaystyle f_{Y}(y)=\int_{-\infty}^{+\infty} f(x,y)dx$
    \end{tabular}
    \end{adjustbox}
\end{table}
\end{itemize}

\vspace{-1cm}
\subsection*{Independence}
$X$ and $Y$ are independent if for every pair $x, y$:
\begin{itemize}
    \item Discrete: \quad $p(x,y) = p_{x}(x)p_{y}(y)$
    \item Continuous: \quad $f(x,y) = f_{x}(x)f_{y}(y)$
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Normal Distribution}

\begin{minipage}{0.7\textwidth}
\begin{itemize}
    \item Probability Density Function (pdf):\[f(x) = \frac{1}{\sqrt{2\pi \sigma^{2}}}e^{-\frac{(x-\mu)^{2}}{2\sigma^{2}}}\]
    \item Cumulative Distribution Function (cdf): \[F(X) = \frac{1}{2}\bigg[ 1+ \text{erf}\bigg( \frac{x-\mu}{\sqrt{2\sigma^{2}}} \bigg) \bigg]\]
    where \[\text{erf}(x) = \frac{2}{\sqrt{\pi}} \int_{0}^{x} e^{-t^{2}}dt\]
\end{itemize}
\end{minipage}\hfill
\begin{minipage}{0.3\textwidth}
\begin{table}[H]
    \centering
    \begin{tabular}{l c}\hline
        Discrete & Continuous \\ \hline
        $X=A$ & $A-0.5 < Y < A+0.5$\\
        $X\leq A$ & $Y<A+0.5$\\
        $X<A$ & $Y<A-0.5$ \\
        $X\geq A$ & $Y>A-0.5$ \\
        $X>A$ & $Y>A+0.5$ \\
    \end{tabular}\caption{Continuity Correction - Normal approximation of discrete distributions (e.g. Binomial)}
\end{table}
\end{minipage}

%%
\subsection*{Standard Normal Distribution}
Standard normal distribution is a special case of normal distribution with $\mu = 0$ and $\sigma =1$.
\begin{table}[H]
    \centering
    \begin{tabular}{c|c}
        Probability Density Function (PDF) &
        Cumulative Distribution Function (CDF)\\ \hline
        $\displaystyle f(Z) = \frac{1}{\sqrt{2\pi }}e^{-\frac{z^{2}}{2}}$ &
        $\displaystyle \Phi(z) = F(Z) = \frac{1}{2}\bigg[ 1+ \text{erf}\bigg( \frac{z}{\sqrt{2}} \bigg) \bigg]$ and \ $\displaystyle Z = \frac{X-\mu}{\sigma}$
    \end{tabular}
\end{table}

%%
\subsection*{Central Limit Theorem}
For sufficiently large n:
\begin{itemize}
    \item Mean sample mean is the population mean - $\displaystyle E[\bar{X}] = \mu_{\bar{X}} = \mu$
    \item Sample mean variance decreases as sample size increase - $\displaystyle  V[\bar{X}] = \sigma_{\bar{X}}^{2} = \frac{\sigma^{2}}{n}$
    \item Standard error on the mean (SEM) is the SD of sample means - $\displaystyle \sigma_{\bar{X}} = \frac{\sigma}{\sqrt{n}}$
\end{itemize}

%%
\subsection*{Margin of Error, Confidence Intervals}
Unknown $\sigma$, large $n$.
\begin{table}[H]
    \centering
    \begin{tabular}{c|c}
        Confidence interval (CI) &
        Margin of Error (ME)\\ \hline
        $\displaystyle CI = \bar{X} [\bar{X_{l}}, \bar{X_{u}}] \ =\  \bigg[\bar{X}-Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}},\ \bar{X}+Z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}\bigg]$ &
        $\displaystyle ME = \bar{X}\pm ME_{\bar{X}} = \lvert Z_{\frac{\alpha}{2}} \rvert \frac{\sigma}{\sqrt{n}}$
    \end{tabular}
\end{table}

%%
\subsection*{$t$-distribution}
Unknown $\sigma$, small n. Test statistic - $\displaystyle T=\frac{\bar{X}-\mu}{S/\sqrt{n}}$.
\begin{table}[H]
    \centering
    \begin{tabular}{c|c}
        Confidence interval (CI) &
        Margin of Error (ME)\\ \hline
        $\displaystyle CI = \bar{X} [\bar{X_{l}}, \bar{X_{u}}] \ =\  \bigg[\bar{X}-t_{\lvert{\frac{\alpha}{2},n-1} \rvert}\frac{s}{\sqrt{n}},\ \bar{X}+t_{\lvert{\frac{\alpha}{2},n-1}\rvert}\frac{s}{\sqrt{n}}\bigg]$ &
        $\displaystyle ME = \bar{X}\pm ME_{\bar{X}} = \lvert Z_{\frac{\alpha}{2},n-1} \rvert \frac{s}{\sqrt{n}}$
    \end{tabular}
\end{table}
%%
\subsection*{$\chi^{2}$-distribution and $F$-distribution}
\begin{itemize}
    \item $\chi^{2}$-distribution is obtained by squaring normal distribution.
    \item For $n$ observations with unknown $\mu$ and $\sigma$, test statistic - $\displaystyle \frac{(n-1)S^{2}}{\sigma^{2}} $

    \item $F$-statistic is the ratio of two variances (see ANOVA).\[F(\nu_{1}, \nu_{2}) = \chi_{1}^{2}(\nu_{1})/ \chi_{2}^{2}(\nu_{2})\]
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Hypothesis Testing}
%%
\subsection*{NHST Types of Errors}
\begin{minipage}{0.5\textwidth}
\begin{table}[H]
    \begin{tabular}{|c|c|c|}\hline
        $\mathbf{H_{0}}$ & \textbf{True} & \textbf{False} \\\hline
        \textbf{Do not reject} & True negative, $1-\alpha$ & False negative, $\beta$\\ \hline
        \textbf{Reject} & False positive, $\alpha$ & True positive, $1-\beta$\\ \hline
    \end{tabular}
\end{table}
\end{minipage} \hfill
\begin{minipage}{0.45\textwidth} \vspace{.2cm}
\begin{itemize}
    \item $\alpha$ denotes the probability of type I error.
    \item $\beta$ denotes the probability of type II error.
\end{itemize}
\end{minipage}

\subsection*{$z$-test \& $t$-test}
\begin{itemize}
\item Assumptions:
    \begin{itemize}
        \item{Data are sampled from a normal distribution. Often lognormal distribution is more appropriate (Use SW or AD test to confirm)}
        \end{itemize}
    \item Null Hypothesis $H_{0}$: \begin{itemize}
        \item one sample: no differences between the means and a population mean.
        \item two sample: no differences between the two sample means.
        \item p-value is an estimate of the probability of getting a test-statistic more extreme. Reject $H_{0}$ if $p \leq \alpha$.
    \end{itemize}
\end{itemize}
\begin{table}[H]
    \centering
    \begin{tabular}{l | c | c}\hline
    & Expression & Notes \\ \hline
    $z$-test statistic &   $ \displaystyle Z = \frac{\bar{X}-\mu}{\sigma / \sqrt{n}}$ & $\displaystyle \frac{\text{effect}}{\text{error}}$, same below \\ [1.2em] \hline

    $t$-test one-sample statistic &   $ \displaystyle T = \frac{\bar{X}_{B}-\mu_{A}}{s_{B}/\sqrt{n}}$ & $v = n-1$\\ [1.2em] \hline

    $t$-test two-sample statistic &   $ \displaystyle T = \frac{\bar{X}_{B}-\bar{X}_{A}}{\sqrt{\frac{S_{A}^{2}}{n_{A}}+\frac{S_{B}^{2}}{n_{B}}}}$ & assume similar variances, $v = n_{A}+n_{B}-2$\\ [1.5em] \hline

    Welch-Satterthwaite Equation  & $\displaystyle \nu = \frac{(s_{\bar{X}_{1}}^{2}+s_{\bar{X}_{2}}^{2})^{2}}{\frac{s_{\bar{X}_{1}}^{4}}{n_{1}-1}+ \frac{s_{\bar{X}_{2}}^{4}}{n_{2}-1}}$ & $s_{\bar{X}_{1}}^{2} = \frac{s_{1}^{2}}{n_{1}}, \quad s_{\bar{X}_{2}}^{2} = \frac{s_{2}^{2}}{n_{2}}$
    \end{tabular}
\end{table}
\begin{figure}[H]
    \centering
    \includegraphics[width=.75\textwidth]{./images/zTest_and_tTest.pdf}
\end{figure}

%%
\vspace{-2cm}
\subsection*{ANOVA}
Assumptions:
\begin{itemize}
\item{Data are sampled from a normal distribution. Often lognormal distribution is more appropriate (Use SW or AD test to confirm)}
\item{All groups have similar variance (homoscedastic) (Use Levene’s or Bartlett’s test to confirm)}
\item{Hypothesis $H_{0}$: no differences between the means of any of the groups}
\end{itemize}
%%
\subsection*{One-way ANOVA}
\begin{itemize}
    \item Sample means and grand mean:
        \begin{table}[H]
        \centering
        \begin{tabular}{c|c}
        Sample Means & Grand Mean \\ \hline
        $\displaystyle \bar{X}_{i} = \frac{1}{J} \sum_{j=1}^{J}X_{i, j}$
        &
        $\displaystyle \bar{X} =
        \frac{1}{N}\sum^{I}_{i=1}\sum^{J}_{j=1}X_{i,j} = \frac{1}{I}\sum_{i=1}^{I}\bar{X}_{i}$\\
         \end{tabular}
        \end{table}
        \begin{itemize}
            \item $I$ denotes the number of groups
            \item $J_{i}$ denotes the number of observations for a given group.
            \item $N$ is the total number of observations, $N = \sum_{i=1}^{I}J_{i}$.
            \item $X_{i,j}$ is a random variable that denotes the $j^{th}$ observation from the $i^{th}$ group.
        \end{itemize}

    \item Sums of Squares: $SS_{T} = SS_{R}+SS_{M}$
        \begin{table}[H]
        \centering
        \begin{tabular}{c|c|c|c}
        & Sum of Squares & DoF & Notes \\ \hline
        Total & $\displaystyle SS_{T} = \sum_{i=1}^{I}\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X})^{2}=(N-1)S_{grand}^{2}$ & $\displaystyle \nu_{T}=N-1$ & $\displaystyle S^{2}_{grand} = \frac{1}{N-1}\sum^{I}_{i=1}\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X})^{2} $\\ \hline

        Model & $\displaystyle SS_{M} = \sum^{I}_{i=1}J_{i}(\bar{X}_{i}-\bar{X})^{2}$ & $\displaystyle \nu_{M}=I-1$\\ \hline

        Residuals & $\displaystyle SS_{R} = \sum_{i=1}^{I}\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X}_{i})^{2}=\sum_{i=1}^{I}S_{i}^{2}(J_{i}-1)$ & $\displaystyle \nu_{R} = N-I$ & $\displaystyle S_{i}^{2} = \frac{\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X}_{i})^{2}}{J_{i}-1}$\\
         \end{tabular}
        \end{table}

   \item Mean square values and test-statistic:
   \[MS_{M} = \frac{SS_{M}}{\nu_{M}} = \frac{1}{I-1}\sum^{I}_{i=1}J_{i}(\bar{X}_{i}-\bar{X})^{2} \quad \quad \quad  MS_{R} = \frac{SS_{R}}{\nu_{R}} = \frac{1}{N-I} \sum_{i=1}^{I}\sum_{j=1}^{J_{i}}(X_{i,j}-\bar{X}_{i})^{2}\]
   \[F = \frac{MS_{M}}{MS_{R}} \quad \quad \quad p =1-f(F, \nu_{M}, \nu_{R})\]
\end{itemize}

\subsection*{Two-way ANOVA}
        \begin{table}[H]
        \begin{adjustbox}{width=\columnwidth,center}
        \begin{tabular}{c|c|c|c|c|c}
        & Sum of Squares & DoF & Mean Square & F-Statistic & Notes \\ \hline
        Total & $\displaystyle SS_{T} = \sum_{g=1}^{G}\sum_{i=1}^{I}\sum_{j=1}^{J}(X_{g,i,j}-\bar{X})^{2}$ & $\displaystyle \nu_{T}=N-1$ & & & $N=GIJ$\\ \hline

        Model & $\displaystyle SS_{M} = \sum_{g=1}^{G}\sum^{I}_{i=1}J(\bar{X}_{gi}-\bar{X})^{2}$ & $\displaystyle \nu_{M}=GI-1$ & & &\\ \hline

        Residuals & $\displaystyle SS_{R} = \sum_{g=1}^{G}\sum_{i=1}^{I}\sum_{j=1}^{J}(X_{g,i,j}-\bar{X}_{gi})^{2}$ & $\displaystyle \nu_{R} = N-GI$ & $\displaystyle MS_{R} = \frac{SS_{R}}{\nu_{R}}$ & & \\ \hline

        $SS_{A}$  & $\displaystyle SS_{A} = IJ \sum_{g=1}^{G}(\bar{X}_{g}-\bar{X})^{2}$ & $\displaystyle \nu_{A} = G-1$ & $\displaystyle MS_{A} = \frac{SS_{A}}{\nu_{A}}$ & $\displaystyle F_{A} = \frac{MS_{A}}{MS_{R}}$ & $SS_{M}$ grouped by G\\ \hline

        $SS_{B}$  & $\displaystyle SS_{B} = GJ \sum_{i=1}^{I}(\bar{X}_{i}-\bar{X})^{2}$ & $\displaystyle \nu_{B} = I-1$ & $\displaystyle MS_{B} = \frac{SS_{B}}{\nu_{B}}$ & $\displaystyle F_{B} = \frac{MS_{B}}{MS_{R}}$ & $SS_{M}$ grouped by I\\ \hline

        $SS_{A \times B}$  & $\displaystyle SS_{A \times B} = SS_{M}-SS_{A}-SS_{B}$ & $\displaystyle \nu_{A \times B} = \nu_{M}-\nu_{A}-\nu_{B}$ & $\displaystyle MS_{A\times B} = \frac{SS_{A\times B}}{\nu_{A\times B}}$ & $\displaystyle F_{A\times B} = \frac{MS_{A\times B}}{MS_{R}}$ & left over $SS$\\
         \end{tabular}
         \end{adjustbox}
        \end{table}
\begin{figure}[H]
    \centering
    \includegraphics[width=.5\textwidth]{./images/twoWayANOVA.pdf}
\end{figure}
\subsection*{Post-Hoc Tests}
\begin{itemize}
            \item Bonferroni - pairwise t-tests. Statistically significant if $p < p_{crit}$
            \item Tukey Kramer Test (less conservative than Bonferroni).
        \end{itemize}
\subsection*{Non-parametric Tests}
\begin{itemize} \itemsep0.4cm
    \item \textbf{Sign test}: one sample, asymmetric data.
    \begin{itemize}
        \item Compare each sample to a pre-defined median value $\tilde{\mu}_{0}$: +ve/-ve difference?
        \item Make hypothesis:
        \begin{itemize}
            \item $H_{0}$: the population median $\tilde{\mu}$ from which the sample is taken is equal to $\tilde{\mu}_{0}$.
            \item $H_{1}$: $\tilde{\mu} \neq \tilde{\mu}_{0}$.
        \end{itemize}
        \item If $H_{0}$ is true, we should obtain at least $n/2$ of data points that are negative.
        \item find $P$ by taking binomial test by taking $p=0.5$.
    \end{itemize}

    \item \textbf{Wilcoxon Sign-rank test}: one sample, non-asymmetric data.
    \begin{itemize}
        \item Compare each sample to a pre-defined median value $\tilde{\mu}_{0}$. Consider both the \textbf{sign} and \textbf{magnitude} of differences.
        \item Sort and find the rank of the differences. Multiply the ranks by their sign.
        \begin{table}[H]
            \centering
            \begin{tabular}{l|c|c|c}
            rank  &  ...  &  ...&  ...\\ \hline
            $\lvert$difference$\rvert$ & ...  &  ...&  ...\\  \hline
            sign & ...  &  ...&  ...\\ \hline
            signed rank & ... &  ... &  ...\\
            \end{tabular}
        \end{table}
        \item calculate the absolute sum of the signed ranks, for +ve and -ve, respectively.
        \begin{itemize}
            \item $W^{+} = \sum_{i}^{n^{+}}R_{i}^{+}$ and $W^{-} = \sum_{i}^{n^{-}}-R_{i}^{-}$
            \item the smaller one becomes the test statistic.
        \end{itemize}
        \item Critical value $W_{crit}$ can be located from the table.
    \end{itemize}

    \item \textbf{Mann-Whitney U-test}: two samples.
    \begin{itemize}
        \item Make hypothesis:
        \begin{itemize}
            \item $H_{0}$: the mean rank of the two levels are equal. (If samples are independent are from similar underlying distributions : mean rank -> median.)
            \item $H_{1}$: the mean rank of the two levels are different.
        \end{itemize}
        \item Group both samples, rank and correct averages \textbf{in one table}.
         \begin{table}[H]
            \centering
            \begin{tabular}{l|c|c|c}
            rank  &  ...  &  ...&  ...\\ \hline
            data value & ...  &  ...&  ...\\
            \end{tabular}
        \end{table}
        \item Calculate the sum ranks for each group $W$.
        \[W_{X}=\sum_{i}^{n_{X}}R_{i}^{X} \quad \quad W_{Y}=\sum_{i}^{n_{Y}}R_{i}^{Y}\]
        \item Calculate U-statistic for each group
        \[U_{i} = W_{i}-\frac{n_{i}(n_{i}+1)}{2}\]
        \item For $n<8$: $U_{crit}$ can be found from the table. Reject $H_{0}$ if $min(U_{1}, U_{2})<u_{crit}$
        \item For $n\geq 8$: \[\mu_{U}=\frac{n_{1}n_{2}}{2} \quad \quad \sigma_{U}^{2}= \frac{n_{1}n_{2}(n_{1}+n_{2}+1)}{12}\]
        z-statistic can be calculated by \[Z_{i}=\frac{U_{i}-\mu_{U}(+continuity)}{\sigma_{U}}\]
    \end{itemize}

    \item \textbf{Kruskall-Wallis test}: more than two samples.
    \begin{itemize}
        \item $H_{0}$: the $I$ independent random samples all come from identical populations.
        \item Calculate the sum ranks for each group:
        \[H=-3(N+1)+\frac{12}{N(N+1)}\sum_{i=1}^{k}\frac{W_{i}^{2}}{J_{i}}\]
        \item For all $J_{i} > 5$, and if $H_{0}$ is true, $H$ is approximated to $\chi^{2}$ distribution with $\nu = I-1$.
    \end{itemize}
\begin{figure}[H]
    \centering
    \includegraphics[width=\textwidth]{./images/nonparasummary.pdf}
\end{figure}

    % \item K-S test statistic: \[K = \sqrt{n}D_{n}\] p-value is $\displaystyle p = 1-KS(\sqrt{n}D_{n})$.

    % \item Shapiro-Wilk test statistic: \[W = \frac{(\sum a_{i}z_{i})^{2}}{(\sum x_{i}-\bar{x})^{2}}\]
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Correlation and Regression}
\subsection*{Correlation}
\begin{table}[H]
    \centering
    \begin{adjustbox}{width=\columnwidth,center}
    \begin{tabular}{c|c}
    Covariance (discrete and continuous)& Correlation Coefficient \\ \hline
    $\displaystyle cov(X, Y)= E[(x-\mu_{X})(y-\mu_{Y})]
    = E(XY)-E(X)E(Y)=\begin{cases}
    \displaystyle \sum_{x}\sum_{y}(x-\mu_{X})(y-\mu_{Y})p(x,y)\\
    \displaystyle \int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty}  (x-\mu_{X})(y-\mu_{Y})f(x,y) dxdy %& continuous
    \end{cases} $
     &  $\displaystyle \rho_{X,Y} = \frac{cov(X,Y)}{\sigma_{X}\sigma_{Y}}$\\
    \end{tabular}
    \end{adjustbox}
\end{table}
  \begin{itemize}
     \item \textbf{Pearson's correlation coefficient}: $r$ is an point estimator of $R = \hat{\rho}$.  Assume linear relationship between $X$ and $Y$.
     \[\sigma_{X}^{2} = S_{XX} = \frac{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}{n-1}, \quad \quad \sigma_{Y}^{2} = S_{YY} = \frac{\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}{n-1}, \quad \quad cov(x,y) = S_{XY} = \frac{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})^{2}}{n-1}\]
    % \[\Downarrow\]
    \[\Rightarrow \quad r_{X, Y} = \frac{S_{XY}}{\sqrt{S_{XX}}\sqrt{S_{YY}}} = \frac{\sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}\sqrt{\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}} \quad \in [-1, 1]\]

    \item \textbf{Test statistic and DoF}
    \[T = \frac{R\sqrt{n-2}}{\sqrt{1-R^{2}}}, \quad \quad \nu = n-2\]
     \item \textbf{Spearman's rank correlation coefficient}: no linear relationship between $X$ and $Y$ -  $ \displaystyle \rho = 1- \frac{6\sum(x_{i}-y_{i})^{2}}{n(n^{2}-1)}$, where $x_{i}, y_{i}$ are ranks. Otherwise we evaluate Pearson's r on the ranks (if $x_{i}, y_{i}$ are tied).
  \end{itemize}
  %%

\subsection*{Regression}
\begin{itemize}
    \item \textbf{Regression model} - $Y = b_{0} + b_{1}X + \epsilon$, where $\epsilon$ is a normally distributed random variable with $E(\epsilon) = 0$ and $V(\epsilon) = \sigma_{\epsilon}^{2}$.
    \item \textbf{Estimate of the best fit line}

    \begin{minipage}{.45\textwidth}
    \begin{table}[H] \centering
    \begin{tabular}{c|c|c}
    \multicolumn{3}{c}{$\mathbf{Y=\hat{b}_{0}+\hat{b}_{1}X}$}\\ \hline
    slope & $\displaystyle \hat{b}_{1} = \frac{S_{xy}}{S_{xx}}$ & $\displaystyle s_{\hat{b}_{1}} = \frac{s_{\epsilon}}{\sqrt{S_{xx}}}$\\ \hline
    intercept & $\displaystyle \hat{b}_{0} = \bar{y}-\hat{b}_{1}\bar{x} $ & $\displaystyle s_{\hat{b}_{0}} = \sqrt{\frac{1}{n}+\frac{\bar{x}^{2}}{S_{xx}}}$
    \end{tabular}\end{table}
    \end{minipage}
    \begin{minipage}{.05\textwidth}\[\xrightarrow{\text{with}}\]
    \end{minipage}
    \begin{minipage}{.45\textwidth}
    \begin{tabular}{c|c|c} \centering
    $S_{xx}$ & $S_{yy}$ & $S_{xy}$ \\ \hline
    $\displaystyle \sum_{i=1}^{n}(x_{i}-\bar{x})^{2}$ & $\displaystyle \sum_{i=1}^{n}(y_{i}-\bar{y})^{2}$ & $\displaystyle \sum_{i=1}^{n}(x_{i}-\bar{x})(y_{i}-\bar{y})^{2}$
    \end{tabular}\end{minipage}
    \begin{itemize}
        \item Best estimator for variance of residuals -  $\displaystyle \hat{\sigma}_{\epsilon}^{2} = s_{\epsilon}^{2} = \frac{SS_{R}}{\nu_{R}} = \frac{\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}{n-2}$
    \end{itemize}

    \item \textbf{Coefficient of determination}: \[r^{2} = 1-\frac{SS_{R}}{SS_{T}}, \quad \quad \text{where} \quad SS_{T}=S_{yy}, \quad SS_{R}=s_{\epsilon}^{2}\nu_{R}, \quad \quad \]

    \item \textbf{Confidence interval}:
     \begin{table}[H] \centering
    \begin{tabular}{c|c|c}

   slope & intercept & variance of residual \\ \hline
   $CI=\hat{b}_{1} + t_{1-\frac{\alpha}{2}, \nu}s_{\hat{b}_{1}}$ & $CI=\hat{b}_{0} + t_{1-\frac{\alpha}{2}, \nu}s_{\hat{b}_{0}}$ &
   $s_{\epsilon}^{2}= SS_{R}/\nu_{R}$ and $\nu_{R}=n-2$
    \end{tabular}\end{table}

    \item \textbf{Confidence bands}: uncertainty on the mean
    \[CI = \hat{y}^{*}\pm t_{\frac{\alpha}{2}, n-1}s_{\hat{y}^{*}} \quad \quad s_{\hat{y}^{*}} = s_{\epsilon}\sqrt{\frac{1}{n}+\frac{(x^{*}-\bar{x})^{2}}{\sum_{i=1}^{n}(x_{i}-\bar{x})^{2}}}\]

    \item \textbf{Prediction bands}: uncertainty on a single measurement
    \[s_{pred}^{2} = s_{\hat{y}^{*}}^{2} + s_{\epsilon}^{2} \quad \quad PI=\hat{y}^{*}\pm t_{\frac{\alpha}{2}, n-2} s^{*}_{pred}\]
\end{itemize}

\vspace*{\fill}
\framebox{\footnotesize  \href{https://www.overleaf.com/read/yrxtbmqnvzgh}{Scripted} by B Li \& P Xie. \ Last update: \today}
% \begin{flushright}
%     \begin{figure}[H]
%     \includegraphics[width=.1\textwidth]{by-nceu.eps}
%     This work is licensed under a Creative Commons Attribution 4.0 International License
% \end{figure}
% \end{flushright}

% insert data tables from the external pdf source
\includepdf[pages=-]{./images/Z_t_F_Tables.pdf}

% end the doc
\end{document}


\section{M/M/\texorpdfstring{$\infty$}{Infinity} Queues}
\label{sec:M-M-inf-Queues}

A $M/M/\infty$ is a queue where
(i) the arrival process is Poissonian with rate $\lambda$,
(ii) the service process is Exponential with rate $\mu$,
(iii) there are infinite servers,
(iv) the buffer has infinite capacity,
(v) the scheduling policy is FCFS.

%\begin{figure}[tp]
%\label{fig:M-M-inf-Queue}
%	\centering
%	\includegraphics{fig/M-M-inf-Queue}
%	\caption{An M/M/k queue and its corresponding CTMC.}
%\end{figure}

The key question in these types of systems is determining the \textit{Queue Probability} $P_{Q}$, that is the probability that an arriving job is enqueued.

\begin{theorem}[Utilization]
	\label{thm:M-M-inf-Utilization}
	For any $M/M/\infty$, the utilization is

	\begin{equation}
	\label{eqn:M-M-inf-Utilization}
	\varrho = \frac{\lambda}{\mu}
	\end{equation}
\end{theorem}

\begin{theorem}[State Probability]
\label{thm:M-M-inf-Probability-State}
	For any $M/M/\infty$, the state probability is

	\begin{equation}
	\label{eqn:M-M-inf-Probability-State}
	\pi_{i} = \Big(\frac{\lambda}{\mu})^{i} \frac{1}{i!} \pi_{0}
	\end{equation}

	with

	\begin{equation}
	\label{eqn:M-M-inf-Probability-State-Zero}
	\pi_{0} = e^{-\frac{\lambda}{\mu}}
	\end{equation}

	That is $N^{M/M/\infty} \sim Poisson(\frac{\lambda}{\mu})$.
\end{theorem}

Since, $N^{M/M/\infty} \sim Poisson(\frac{\lambda}{\mu})$, then

\begin{equation}
\label{eqn:M-M-inf-Mean-System-Jobs}
\expected{N} = \frac{\lambda}{\mu}
\end{equation}

The remaining metrics ($\expected{N_{Q}},\expected{T},\expected{T_{Q}}$) could be determined by applying the Little's Law, the basic definitions
$\expected{N}=\expected{N_{Q}}+\expected{N_{S}}$,
$\expected{N_{S}}=\varrho$,
$\expected{T}=\expected{T_{Q}}+\expected{T_{S}}$,
$\expected{T_{S}}=\frac{1}{\mu}$,
considering that $\expected{N_{Q}}=0$ and $\expected{T_{Q}}=0$, because every arrival il always served immediately.

In particular we obtain

\begin{equation}
\label{eqn:M-M-inf-Mean-Response-Time}
\expected{T} = \frac{1}{\mu}
\end{equation}


\section {Introduction to DFA Framework}
\setlength{\parindent}{0pt}
(Prepared by Namrata Priyadarshani and Shivam Bansal)

\vspace{0.3cm}

In this module, we build up common framework to express DFA. Some common characteristics of DFA are described below:

\subsection{Transfer Functions and Meet operator}

Let s be a statement and the value before statement is in[s] and after statement is out[s]. Then, for a forward DFA, \textbf{Transfer function} is defined as the function that takes in[s] as argument and converts it into out[s]. \textbf{Meet operator} expresses in[s] as the function of out[p] for all predecessor p of s. These functions operates in reverse direction in the case of Backward DFA.
\newline
Most of the time, we consider running the algorithm at the Basic Block level granularity and define Transfer function and meet operator for the values in[B] and out[B] i.e. the input and output values of the basic block B.

\subsection{Partial ordered values}

There is a partial order within the elements of Domain represented by $\geq$ operator.
Directed Acyclic Graph is one way to represent the ordering. The graph contains an edge from $a \rightarrow b$ iff $a \geq b$.
%insert image
\includegraphics[scale=0.3]{images/91_1.png}
There is one value greater than all values in Domain known as Top value (T) and one value less than all values in Domain known as Bottom value ($ \bot $).

\subsection{Constant Propagation DFA}
Different DFA has different direction, boundary condition, meet operator and transfer function. The structure and flow of algorithm remains same for all DFA. In fixed-point-algorithm first the boundary condition is specified, all other values are set to top value and then iterate until no change occur in any value.

For constant propagation the DFA is specified by:
\begin{itemize}
    \item \textbf{Domain}: set of constant definitions
    \item \textbf{Direction}: Forward
    \item \textbf{Transfer function}: $f_{B} = \lambda x. GEN[B] \cup (x-KILL[B]) $
    \item \textbf{Meet operator}: $\cap$ (set intersection)
    \item \textbf{Boundary condition}: $out[entry] = \phi$
\end{itemize}


\section*{Ex.34.4-6}
\subsection*{Formula satisfiability}

We are given an algorithm, call it $A$, which can decide formula satisfiability in polynomial time. To find a satisfying assignment with the use of $A$ do the following:

Let $x_1,\ldots,x_k$ be the $k$ input variables to the formula problem. First call $A$, to decide if the formula instance has a satisfying assignment. If not it will return 0, and no more work needs to be done. If it returns 1, there exists an assignment which we can find.

Note that if we set $x_i=1$ for some $i\in\{1,\ldots,k\}$, and then run $A$ it will tell us if there exists a solution when $x_i=1$. If there doesn't, then set $x_1 =0$. Do this for all $i\in\{1,\ldots,k\}$, and you will get a satisfying assignment. For each variable we call $A$ once, which is $O(n^m)$, and we do this $k$ times plus some constant work. This yields an algorithm that finds the satisfying assignment in $O(kn^m)=O(n^m)$ time. Which is polynomial.


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
    \DeclareGraphicsExtensions{.png, .jpeg}
\usepackage{caption}
\usepackage[top=1in, bottom=1in, left=1in, right=1in]{geometry}

\title{STAT 775: Machine Learning \\ HW 02}
\author{Terence Henriod}
\date{\today}

\begin{document}

\clearpage            % All
\maketitle            % this,
\thispagestyle{empty} % removes the page number from the title page

\begin{abstract}
A short implementation and demonstration of na\"{i}ve Bayes classifiers on hand-written digit images.
\end{abstract}

\newpage
\section{Problem Description}
The task: using the \texttt{zipcode} data from the Elements of Statistical Learning book website and Bayes rule, do the following:
\begin{enumerate}
  \item Construct Gaussian classifiers for all of the digits (0-9) using the training data set. Do so by computing the means and covariances of each class in order to fit the Gaussian distribution for the class.
  \item Use the constructed classifiers to classify the test set.
  \item Display the results in a ``confusion matrix."
\end{enumerate}
\textit{Hint}: If you find that a matrix (specifically any of the covariance matrices) is numerically or actually singular, and therefore cannot be inverted, regularize it by adding an identity matrix multiplied by a small factor to ``breathe" on it. This can help make the matrix invertible, but one needs to exercise caution as it can adversely affect the results of computations.

\section{Results}
The na\"{i}ve Bayes classifiers performed rather well, achieving nearly 94\% accuracy. While this does leave something to be desired, improving on this performance benchmark might require improving the numerical accuracy of the program (using something better than double-precision variables), finding a better regularization factor for regularizing non-invertible matrices ($0.5$ seemed to work well enough, no more effort was put forth to improve on this), application of additional techniques (like PCA), or even using this classifier as an element of a composite classifier.

\begin{figure}[h]
  \centering
  \includegraphics[width=.6\linewidth]{R_confusion_matrix_output}
  \caption{R output of a confusion matrix displaying the classification results using the na\"{i}ve Bayes classifier.}
  \label{fig:confusion_matrix}
\end{figure}

\section{Code}
After struggling for too long with R code and singular matrices, it was deemed better to switch to a familiar and high performance language - namely C++. Thus, the classification code was written in C++ and the confusion matrix was ported to R for data presentation.

\subsection{C++ Code}
\begin{verbatim}
#include <string>
#include <vector>
#include <Eigen/Dense>
#include <cmath>

#include <iostream>
#include <fstream>
#include <exception>

#define PI 3.141592653589793238462643383279502884L
#define REGULARIZATION_FACTOR 0.5

const std::string TRAINING_FILE_NAME = "zip.train"; // 2007 items
const std::string TEST_FILE_NAME = "zip.test"; // 7291 items

typedef struct {
  unsigned label;
  Eigen::VectorXd features;
} ClassificationObject;

typedef struct {
  unsigned label;
  double prior;
  Eigen::VectorXd mean;
  Eigen::MatrixXd covariance;
  double covarianceDeterminant;
  Eigen::MatrixXd covarianceInverse;
} ClassInfo;

std::vector<ClassificationObject>
ReadData(
  const std::string& fileName,
  const unsigned featureDim,
  const unsigned numItems) {

  std::ifstream fin;
  fin.clear(); fin.open(fileName.c_str());

  if (!fin.good()) {
    throw std::exception();
  }

  std::vector<ClassificationObject> data; data.reserve(numItems);

  for (unsigned i = 0; i < numItems; ++i) {
    ClassificationObject classificationObject;
    double dummy;
    fin >> dummy;
    classificationObject.label = (unsigned) dummy;

    classificationObject.features.resize(featureDim);
    for (unsigned j = 0; j < featureDim; ++j) {
      fin >> classificationObject.features(j);
    }
    data.push_back(classificationObject);
  }

  return data;
}

ClassInfo
ComputeClassInfo(
  const std::vector<ClassificationObject>& data,
  const unsigned classLabel,
  const unsigned featureDim) {

  ClassInfo classInfo;
  classInfo.label = classLabel;
  classInfo.mean.setZero(featureDim);
  classInfo.covariance.setZero(featureDim, featureDim);

  // compute mean and prior probability at once
  std::vector<ClassificationObject> subset;
  for (const ClassificationObject& classificationObject : data) {
    if (classificationObject.label == classLabel) {
      classInfo.mean += classificationObject.features;
      subset.push_back(classificationObject);
    }
  }
  classInfo.mean /= subset.size();
  classInfo.prior = (double) subset.size() / (double) data.size();

  // covariance matrix
  // Note: I use a nifty trick that is simpler in code (not sure if
  //       simpler computationally). Let $P_i = x_i - mu$, where $i$
  //       corresponds to a particular observation vector. Then
  //       $A$ is the concatenation of all the $P_i$ as column
  //       vectors ($A = [P_1 ... P_m]$). Then
  //       $\frac{1}{m} * A * A^t$ results in the same computations
  //       that create the covariance matrix as more traditional
  //       formulae.
  Eigen::MatrixXd A(featureDim, subset.size());
  for (unsigned j = 0; j < subset.size(); ++j) {
    A.col(j) = subset[j].features - classInfo.mean;
  }
  classInfo.covariance = A * A.transpose();
  classInfo.covariance /= (double) subset.size();
  if (classInfo.covariance.determinant() - 0.1 <= 0.0) {
    Eigen::MatrixXd regularizationMatrix;
    regularizationMatrix.setIdentity(featureDim, featureDim);
    regularizationMatrix *= REGULARIZATION_FACTOR;
    classInfo.covariance += regularizationMatrix;
  }
  classInfo.covarianceDeterminant = classInfo.covariance.determinant();
  classInfo.covarianceInverse = classInfo.covariance.inverse();

  return classInfo;
}

double
GaussianPdf(
  const Eigen::VectorXd& testFeatureVector,
  const ClassInfo& classSummary) {

  const Eigen::VectorXd& x = testFeatureVector;
  const Eigen::VectorXd& mu = classSummary.mean;
  const double& sigmaDet = classSummary.covarianceDeterminant;
  const Eigen::MatrixXd& sigmaInv = classSummary.covarianceInverse;

  double scalingFactor = 1.0 / sqrt(pow(2.0 * PI, mu.size()) * sigmaDet);
  double exponent = -0.5 * (((x - mu).transpose() * sigmaInv).dot((x - mu)));

  return scalingFactor * exp(exponent);
}

unsigned
ClassifyObject(const ClassificationObject& object, const std::vector<ClassInfo>& classSummaries) {
  unsigned mostLikelyClass = 0;
  double highestProbability = classSummaries.front().label;

  for (const ClassInfo& classSummary : classSummaries) {
    double probability = GaussianPdf(object.features, classSummary) * classSummary.prior;
    if (probability > highestProbability) {
      mostLikelyClass = classSummary.label;
      highestProbability = probability;
    }
  }

  return mostLikelyClass;
}

Eigen::MatrixXi
PerformClassifications(const std::vector<unsigned>& labelSet,
  const std::vector<ClassInfo>& classSummaries,
  const std::vector<ClassificationObject>& testObjects) {

  Eigen::MatrixXi confusionMatrix(labelSet.size(), labelSet.size());
  confusionMatrix.setZero(labelSet.size(), labelSet.size());

  unsigned i = 1;
  unsigned onePercent = testObjects.size() / 100;
  for (const ClassificationObject& object : testObjects) {
    if (i % onePercent == 0) {
      std::cout << i << "% processed." << std::endl;
    }
    i++;

    unsigned classifiedAs = ClassifyObject(object, classSummaries);
    confusionMatrix(object.label, classifiedAs)++;
  }

  return confusionMatrix;
}

int
main(const int argc, const char** argv) {
  std::cout << "Reading Data..." << std::endl;
  std::vector<ClassificationObject> trainingData = ReadData(
    TRAINING_FILE_NAME,
    256,
    7291
  );
  std::vector<ClassificationObject> testData = ReadData(
    TEST_FILE_NAME,
    256,
    2007
  );

  std::cout << "Training Models..." << std::endl;
  std::vector<ClassInfo> classSummaries; classSummaries.reserve(10);
  for (unsigned label = 0; label <= 9; ++label) {
    classSummaries.push_back(ComputeClassInfo(trainingData, label, 256));
  }

  std::cout << "Classifying Objects..." << std::endl;
  Eigen::MatrixXi confusionMatrix = PerformClassifications(
    {0,1,2,3,4,5,6,7,8,9},
    classSummaries,
    testData
  );

  std::cout << "Results:" << std::endl;
  std::cout << confusionMatrix << std::endl;

  return 0;
}
\end{verbatim}

\subsection{R Code}
\begin{verbatim}
############
# Plotting #
############
require(caret)

results <-data.matrix(read.table("results"))
dimnames(results) <- list(c(0,1,2,3,4,5,6,7,8,9), c(0,1,2,3,4,5,6,7,8,9))

actuals.f <- list()
predictions.f <- list()
for (i in 1:nrow(results)) {
  for (j in 1:ncol(results)) {
    actuals.f <- c(actuals.f, rep(i - 1, results[i, j]))
    predictions.f <- c(predictions.f, rep(j - 1, results[i, j]))
  }
}
actuals.f <- factor(unlist(actuals.f))
predictions.f <- factor(unlist(predictions.f))

confusionMatrix(data = predictions.f, reference = actuals.f)
\end{verbatim}

\subsection{The student would like to thank...}
The student would like to thank the authors of the Eigen C++ matrix library. This library has proved quick, easy and effective numerous times, this time being no exception. The Eigen library and more information can both be found at\hfill\\
\texttt{http://eigen.tuxfamily.org}.

\end{document}