\subsection{$QR$ factorization}

In this section, a reliable factorization of matrices is studied. Called the $QR$ factorization of a matrix, it \textit{always} exists. While much can be said about the $QR$ factorization, this section will be limited to real matrices. Therefore we assume the dot product used below is the usual dot product. We begin with a definition.

\begin{definition}{$QR$ factorization}{qr-factorization}
Let $A$ be a real $m\times n$-matrix. Then a $QR$ factorization of $A$ consists
of two matrices, $Q$ orthogonal and $R$ upper
triangular, such that $A=QR$.\index{QR factorization}
\end{definition}

The following theorem claims that such a factorization exists.

\begin{theorem}{Existence of $QR$ factorization}{existence-qr}
Let $A$ be any real $m\times n$-matrix with linearly independent columns. Then there exists an orthogonal
matrix $Q$ and an upper triangular matrix $R$ having non-negative entries
on the main diagonal such that
\begin{equation*}
A=QR
\end{equation*}
\end{theorem}

The procedure for obtaining the $QR$ factorization for any matrix $A$ is as follows.

\begin{procedure}{$QR$ factorization}{qr-factorization}
Let $A$ be an $m \times n$-matrix given by $A = \begin{mymatrix}{cccc}
A_1 & A_2 & \cdots & A_n
\end{mymatrix}$ where the $A_i$ are the linearly independent columns of $A$.
\begin{enumerate}
\item
Apply the Gram-Schmidt Process~\ref{algo:gram-schmidt-algorithm} to the columns of $A$, writing $B_i$ for the resulting columns.

\item
Normalize the $B_i$, to find $C_i = \frac{1}{\norm{B_i}} B_i$.

\item
Construct the orthogonal matrix $Q$ as $Q=\begin{mymatrix}{cccc}
C_1 & C_2 & \cdots & C_n
\end{mymatrix}$.

\item
Construct the upper triangular matrix $R$ as
\[ R =
\begin{mymatrix}{ccccc}
\norm{B_1} & A_2 \dotprod C_1 & A_3 \dotprod C_1 & \cdots & A_n \dotprod C_1 \\
0 & \norm{B_2} & A_3 \dotprod C_2 & \cdots & A_n \dotprod C_2 \\
0 & 0 & \norm{B_3} & \cdots & A_n \dotprod C_3 \\
\vdots & \vdots & \vdots & & \vdots \\
0 & 0 & 0 & \cdots & \norm{B_n}
\end{mymatrix}
\]

\item
Finally, write $A=QR$ where $Q$ is the orthogonal matrix and $R$ is the upper triangular matrix obtained above.
\end{enumerate}
\end{procedure}

Notice that $Q$ is an orthogonal matrix as the $C_i$ form an orthonormal set. Since $\norm{B_i} > 0$ for all $i$ (since the length of a vector is always positive), it follows that $R$ is an upper triangular matrix with positive entries on the main diagonal.

Consider the following example.

\begin{example}{Finding a $QR$ factorization}{qr-factorization}
Let \[
A = \begin{mymatrix}{rr}
1 & 2 \\
0 & 1 \\
1 & 0
\end{mymatrix}
\]
Find an orthogonal matrix $Q$ and upper triangular matrix $R$ such that $A=QR$.
\end{example}

\begin{solution}
First, observe that $A_1$, $A_2$, the columns of $A$, are linearly independent. Therefore we can use the Gram-Schmidt Process to create a corresponding orthogonal set $\set{B_1, B_2 }$ as follows:
\begin{eqnarray*}
B_1 &=& A_1 = \begin{mymatrix}{r}
1 \\
0 \\
1
\end{mymatrix} \\
B_2 &=& A_2 - \frac{A_2 \dotprod B_1}{\norm{B_1} ^2} B_1 \\
&=& \begin{mymatrix}{r}
2 \\
1 \\
0
\end{mymatrix}
- \frac{2}{2} \begin{mymatrix}{r}
1 \\
0 \\
1
\end{mymatrix} \\
&=&
\begin{mymatrix}{r}
1 \\
1 \\
-1
\end{mymatrix}
\end{eqnarray*}

Normalize each vector to create the set $\set{C_1, C_2 }$ as follows:
\begin{eqnarray*}
C_1 &=& \frac{1}{\norm{B_1}} B_1 = \frac{1}{\sqrt{2}}  \begin{mymatrix}{r}
1 \\
0 \\
1
\end{mymatrix} \\
C_2 &=& \frac{1}{\norm{B_2}} B_2 = \frac{1}{\sqrt{3}} \begin{mymatrix}{r}
1 \\
1 \\
-1
\end{mymatrix}
\end{eqnarray*}

Now construct the orthogonal matrix $Q$ as
\begin{eqnarray*}
Q &=& \begin{mymatrix}{cccc}
C_1 & C_2 & \cdots & C_n
\end{mymatrix} \\
&=& \begin{mymatrix}{rr}
\vspace{0.05in}\frac{1}{\sqrt{2}} & \vspace{0.05in}\frac{1}{\sqrt{3}} \\
0  & \vspace{0.05in}\frac{1}{\sqrt{3}} \\
\vspace{0.05in}\frac{1}{\sqrt{2}} & -\vspace{0.05in}\frac{1}{\sqrt{3}}
\end{mymatrix}
\end{eqnarray*}

Finally, construct the upper triangular matrix $R$ as
\begin{eqnarray*}
R &=&
\begin{mymatrix}{cc}
\norm{B_1} & A_2 \dotprod C_1  \\
0 & \norm{B_2}
\end{mymatrix} \\
&=&
\begin{mymatrix}{cc}
\sqrt{2} & \sqrt{2} \\
0 & \sqrt{3} \\
\end{mymatrix}
\end{eqnarray*}

It is left to the reader to verify that $A=QR$.
\end{solution}


\subsection{Boundedness theorem}

If \(f(x)\) is closed and continuous in \([a,b]\) then \(f(x)\) is bounded by \(m\) and \(M\). That is:

\(\exists m \in \mathbb{R} \exists M\in \mathbb{R}\forall x\in [a,b](m<f(x)<M\)


\section{Neural Networks}
\begin{frame}{\insertsec}
	\input{drawings/neural_network.tikz.tex}
\end{frame}

\subsection{Parameters}
\begin{frame}{\insertsubsec}
    \begin{columns}[t]
        \column{.6\textwidth}
        \begin{itemize}
            \item $l$: Current layer
            \item $w_{i, j}^{[l]}$: Weight from unit $j$ to $i$ at layer $l$
            \item $b_{i}^{[l]}$: Bias for unit $i$ at layer $l$
            \item $n^{[l]}$: Number of units in layer $l$
            \item $W^{[l]} \in \mathbb{R}^{n^{[l]} \times n^{[l - 1]}}$: Weight matrix
            \item $\bm{b}^{[l]} \in \mathbb{R}^{n^{[l]}}$: Bias
            \item $
            \begin{aligned}[t]
                \bm{a}^{[l]} &= g(W^{[l]}\cdot \bm{a}^{[l - 1]} + \bm{b}^{[l]}) \\
                \bm{a}^{[l]} &\in \mathbb{R}^{n^{[l]}}
            \end{aligned}
            $ Activations for layer $l$
            \item $g(x)$: Activation function
        \end{itemize}
        \column{.5\textwidth}
        \centering
        \input{drawings/neural_activations.tikz.tex}
    \end{columns}
\end{frame}
\begin{frame}
    \begin{columns}
        \column{.6\textwidth}
        \begin{block}{Cost function}
            $$
            J(\bm{W}, \bm{b}, \hat{\bm{y}}, \bm{y}) =
            \frac{1}{m} \sum_{i=1}^{m} \mathcal{L}(\hat{\bm{y}}^{(i)}, \bm{y}^{(i)})
            $$
        \end{block}
        \begin{block}{Loss function}
            $$
            \mathcal{L}(\hat{\bm{y}}, \bm{y}) = ||\hat{\bm{y}} - \bm{y}||^2
            $$
        \end{block}
        \begin{block}{Parameters update}
            \begin{align*}
            W^{[l]} &:= W^{[l]} - \alpha \cdot \frac{\partial J}{\partial W^{[l]}} \\
            \bm{b}^{[l]} &:= \bm{b}^{[l]} - \alpha \cdot \frac{\partial J}{\partial \bm{b}^{[l]}}
            \end{align*}
        \end{block}
        \column{.4\textwidth}
        To achieve better results at predicting $\hat{\bm{y}}$
        we have to minimize the cost function $J(\bm{W}, \bm{b}, \hat{\bm{y}}, \bm{y})$.
    \end{columns}
\end{frame}

\subsection{Regularization}
\begin{frame}{\insertsubsec}
    To prevent overfitting add the weights to the cost function so we also try to minimize
    big weights.

    \begin{align*}
        J(\bm{W}, \bm{b}, \hat{\bm{y}}, \bm{y}) &=
        \frac{1}{m} \sum_{i = 1}^m \mathcal{L}(\hat{\bm{y}}, \bm{y}) +
        \frac{\lambda}{2m} \sum_{l=1}^L ||W^{[l]}||^2_F \\
        \underbrace{||W^{[l]}||^2_F}_{\text{Frobenius Norm}} &= \sum_i \sum_j (w_{ij}^{[l]}) \\
        dW^{[l]} &= \text{from backprop } + \frac{\lambda}{m} \cdot W^{[l]} \\
        \lambda &\rightarrow \text{Regularization parameter}
    \end{align*}

    It cancels some effects from some units creating a simple model.
\end{frame}
\begin{frame}[fragile]{Dropout Regularization}
    Go through each layer of the network and set some probability of elimination for each unit.
    This is done for \textbf{each sample}.

    \begin{figure}
        \begin{verbatim}
drop = np.random.randn(*a_prev.shape) < keep_prob
a_next = np.multiply(a_prev, drop)
a_next /= keep_prob
        \end{verbatim}
        \caption{Example using \texttt{numpy}}
    \end{figure}

    \begin{alertblock}{Predictions during test time}
        No dropout is used during test time since it wouldn't make sense.
    \end{alertblock}
\end{frame}

\begin{frame}{Adam optimization algorithm}
    Training a NN is a minimization problem. In this case we always use gradient descent
    to find the minimum. Sometimes the gradient can go too fast.

    \begin{onlyenv}<1>
        \begin{align*}
            V_{dW} &= \beta_1 V_{dW} + (1 - \beta_1) \cdot dW \\
            S_{dW} &= \beta_2 S_{dW} + (1 - \beta_2) \cdot dW^2 \\
            W &:= W - \alpha \cdot \frac{V_{dW}}{\sqrt{S_{dW}} + \varepsilon}
        \end{align*}
    \end{onlyenv}
    \begin{onlyenv}<2>
        \input{drawings/momentum.tikz.tex}
    \end{onlyenv}
\end{frame}

\subsection{Hyper-parameters}
\begin{frame}{\insertsubsec}
    \begin{itemize}
        \item $m$: Mini-batch size
        \begin{itemize}
            \item $m = 1$: Stochastic gradient descent
            \item $m = $ set size: Batch gradient descent
        \end{itemize}
        \item $\alpha$: Learning rate
        \begin{itemize}
            \item Learning rate decay
            $\alpha = \frac{1}{1 + \text{decay-rate}\times \text{epoch}} \cdot \alpha_0$
            \item Exponential decay
            $\alpha = 0.95^{\alpha_0}$

        \end{itemize}
        \item $L$: Number of layers
        \item Number of iterations
        \item Number of hidden units
    \end{itemize}
\end{frame}


\subsection{Row picture}

Involves viewing matrix as linear equations graphed on a line or plane. Take the example $A\tb{x}=\tb{b}$ below:

\[\begin{bmatrix}
        1&2&3\\
        3&4&5\\
        4&5&6
    \end{bmatrix}
    \begin{bmatrix}
        x\\y\\z
    \end{bmatrix}
    =
    \begin{bmatrix}
        8\\9\\10
    \end{bmatrix}
\]

This can be viewed as the following system:

\[
    \spalignsys{
    1x + 2y + 3z = 8 ;
    3x + 4y + 5z = 9 ;
     4 + 5y + 6z = 10
    }
\]

\subsection{Column picture}

Involves viewing this setup as a linear combination of column vectors. Take $A\tb{x}=\tb{b}$ again:

\[
    x\begin{bmatrix}
        1\\3\\4
    \end{bmatrix}
    +y\begin{bmatrix}
        2\\4\\5
    \end{bmatrix}
    +z\begin{bmatrix}
        3\\5\\6
    \end{bmatrix}
    =
    \begin{bmatrix}
        8\\9\\10
    \end{bmatrix}.
\]

\subsection{Visualization in Space and Solutions}

\subsubsection{2D space}

In $\R^2$, the equations form a line. Independent column vectors means infinite linear combinations of these to get
a set of $\tb{b}$ in $\R^2$. If one column vector is dependent on another, they are parallel and various
combinations of $\tb{b}$ are on a line.

\subsubsection{3D space}

The equations form a plane in $\R^3$. If column vectors independent, infinite linear combination of $\tb{b}$
exist in 3D space. If one vector is a scaled combination of another and the third is independent, then solutions lie
on a line. If all three are interdependent, the solution is on a line.


\section{Results}

\subsection{Step-by-step Example with OR}
Recall the two bit function OR which returns $1$ if there
is at least a single $1$ in the input and returns $0$ otherwise.
To understand Reichardt's formulation and the conversion
to Boyd's standard form, consider as inputs to our tool
the case with function $f: D \rightarrow E$ where
$D \subseteq {\{0,1\}}^2$ and $E =\{OR(x): x \in D \}$.
Let $F$ be the set of $(y,z)$ such that $f(y) \neq f(z)$. In this
case, $F = \{(00,01), (00,10), (00,11), (01,00), (10,00), (11,00)\}$

Let
\begin{align}
\X = \begin{blockarray}{ccccc}
\qquad & 00 & 01 & 10 & 11 \\
\begin{block}{c[cccc]}
  00 & X_{(00,00)} & X_{(00,01)} & X_{(00,10)} & X_{(00,11)}\\
  01 & X_{(01,00)} & X_{(01,01)} & X_{(01,10)} & X_{(01,11)}\\
  10 & X_{(10,00)} & X_{(10,01)} & X_{(10,10)} & X_{(10,11)}\\
  11 & X_{(11,00)} & X_{(11,01)} & X_{(11,10)} & X_{(11,11)}\\
\end{block}
\end{blockarray}. \nonumber
\end{align}

The objective function of the SDP is
\begin{align} \label{eq:reichardtObj}
    &\max_{y \in D} \sum_{j \in [n]}
    \bra{y,j}\X\ket{y,j} \\
    &= \max\{ \tr{X_{(00,00)}}, \tr{X_{(01,01)}}, \tr{X_{(10,10)}}, \tr{X_{(11,11)}} \} \nonumber
\end{align}
subject to constraints
\begin{align}
    \X \succcurlyeq 0  \nonumber
\end{align}
and
\begin{align}\label{specific_constraint}
    \forall (y,z) \in F \sum_{j \in [n]: y_j \ne z_j}
    \bra{y,j} \X \ket{z, j} = 1.
\end{align}
Observe that \cref{specific_constraint}
is equivalent to
\begin{align}
    \tr{X_{(00,01)}} &= \tr{X_{(00,10)}} = \tr{X_{(00,11)}} = 1 \nonumber \\
    \tr{X_{(01,00)}} &= \tr{X_{(10,00)}} = \tr{X_{(11,00)}} = 1. \nonumber
\end{align}

Our goal is to minimize $M$,
and thus $f_{\text{bound}}$ by
finding the optimal value of $\X$.
By running our SDP solver, we obtain
$\X$ with an objective function value of $\sqrt{2}$.
\begin{align}
    \X = \left[ \begin{array}{cc|cc|cc|cc}
    0.7 & 0 & 0 & 0 & 1 & 0 & 0.5 & 0\\
    0 & 0.7 & 0 & 1 & 0 & 0 & 0 & 0.5\\
    \hline
    0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\
    0 & 1 & 0 & \sqrt{2} & 0 & 0 & 0 & 0.7\\
    \hline
    1 & 0 & 0 & 0 & \sqrt{2} & 0 & 0.7 & 0\\
    0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\
    \hline
    0.5 & 0 & 0 & 0 & 0.7 & 0 & 0.6 & 0\\
    0 & 0.5 & 0 & 0.7 & 0 & 0 & 0 & 0.6\\
    \end{array}
\right] \nonumber
\end{align}
We use our solution to construct the input
vectors to the span program.
Consider $L$ such that $L^\dagger L = \X$.
Then construct $\bra{v_{x,i}}$
for all $x \in D$, $i \in [n]$.
With \cref{input_vectors},
\begin{align}
I &= \left[\begin{array}{cc}
    \bra{1}\bra{v_{00,1}} & \bra{1}\bra{v_{00,2}} \\
    \bra{1}\bra{v_{01,1}} & \bra{0}\bra{v_{01,2}} \\
    \bra{0}\bra{v_{10,1}} & \bra{1}\bra{v_{10,2}}\\
    \bra{0}\bra{v_{11,1}} & \bra{0}\bra{v_{11,2}}\\
\end{array} \right] \nonumber \\
&= \left[\begin{array}{c|c|c|c}
    0 \cdots 0 & \bra{v_{00,1}} & 0 \cdots 0 & \bra{v_{00,2}} \\
    0 \cdots 0 & \bra{v_{01,1}} & \bra{v_{01,2}} & 0 \cdots 0\\
    \bra{v_{10,1}} & 0 \cdots 0 & 0 \cdots 0 & \bra{v_{10,2}}\\
    \bra{v_{11,1}} & 0 \cdots 0 & \bra{v_{11,2}} & 0 \cdots 0\\
\end{array} \right] \nonumber
\end{align}
The columns of $I$ are the columns
used in the span program.
If we remove rows of $I$ corresponding to elements
$ x \in D$ such that $f(x) = 1$
and columns of $I$ that are the
all zero vector, we obtain an equivalent span program where
\begin{align}
I &= \left[\begin{array}{cccc}
   0 & 1 & 0 & 1 \\
\end{array} \right] \nonumber
\end{align}
and $\tau = 1$.
The left two partitions
correspond to the first bit in the
input, and the right two partitions correspond
to the second bit of the input. For each
input bit, the relevant block of $I$
is broken into left and right halves, corresponding
to whether the bit is $0$ (left) or $1$ (right).
It is clear from this matrix that if either input bit is
$1$ the function will return true and otherwise
will return false.

\subsection{Accuracy with OR and Parity}

\begin{figure}[H]
\centering
\includegraphics[scale=.5]{or_complexity}
\caption{The proven analytical optimal query complexity
and calculated empirical optimal query complexity by
size of input bitstring for OR.}
\label{fig:or_complexity}
\end{figure}

We verify that the implementation works correctly
with results for OR on various input sizes.
The optimal quantum query complexity of OR
is $\sqrt{n}$ and, as demonstrated in
\cref{fig:or_complexity}, the empirical results match the analytical
to the thousandth place.

\begin{figure}[H]
\centering
\includegraphics[scale=.5]{parity_complexity}
\caption{The proven analytical optimal query complexity
and calculated empirical optimal query complexity by
size of input bitstring for parity.}
\label{fig:parity_complexity}
\end{figure}

The parity function, determining whether
there are an even number of 1's an input
bitstring, is known to have optimal
quantum query complexity linear to the number of bits.
The implementation also correctly
calculates the complexity of parity to the thousandth
place as demonstrated in \cref{fig:parity_complexity}.

The results for OR and parity demonstrate
that the implementation accurately calculates
optimal quantum query complexity.

\subsection{Runtime with OR}

\begin{figure}[H]
\centering
\includegraphics[scale=.5]{or_runtime}
\caption{Runtime of SDP solver by size of input strings.}
\label{fig:or_runtime}
\end{figure}

Recall that the size of the input to the implementation
does not have to be all $2^n$ input bitstrings
for input size $n$. In fact,
by choosing different and smaller inputs,
we can improve runtime while maintaining accuracy.

In \cref{fig:or_runtime}, observe that the
runtime of the implementation on OR with all
inputs increases exponentially
with the input size $n$.
(Since the number of inputs grows exponentially,
it follows that the size of the problem grows
exponentially, too.) As a note, our
runtime results are presented as the
result of timing our program once. If we
were trying to make an argument about the
absolute runtime of our algorithm, it
would've been better to show a
distribution. We want to highlight the
asymptotic behavior of runtime which is
presented here and consistent between
runs of our program.
The bottleneck in the implementation is
the eigenvalue decomposition of the current
solution at each step of the iteration.

If we limit the inputs to the `worst-case`
bitstrings we observe much lower runtime, as shown in \ref{fig:or_runtime}.
It is in general very difficult to find the worst-case
inputs and OR is a special case.
When searching for at least one 1, the most
difficult bitstrings to search are those with a single 1.
So, by supplying only the worst-case inputs that
have at most one 1, we can maintain the same level of accuracy while
greatly reducing runtime.
We know of no other Boolean functions with
similarly intuitive worst-case sets of inputs.


% declare document class and geometry
\documentclass[12pt]{article} % use larger type; default would be 10pt
\usepackage[margin=1in]{geometry} % handle page geometry

% import packages and commands
\input{../header2.tex}

\title{Phys 220A -- Classical Mechanics -- HW05}
\author{UCLA, Fall 2014}
\date{\formatdate{7}{11}{2014}} % Activate to display a given date or no date (if empty),
         % otherwise the current date is printed

\begin{document}
\maketitle

\section*{Problem 1 (15 pts)}
\textit{
This problem concerns the Runge-Lanz vector and its algebra under Poisson brackets. The Runge-Lenz vector is defined as
\begin{eqn}
\v{K} = \frac{1}{m} \v p \times \v L - \frac{\v r}{r}.
\end{eqn}
}

\begin{enumproblem}

% part A
\item \textit{
Show that the following Poisson bracket is given by
\begin{eqn}
\cbr{L_i, K_j} = \epsilon_{ijk} K_k.
\end{eqn}
}

% part B
\item \textit{
Show that
\begin{eqn}
\cbr{K_i, K_j} = - \frac{2}{m} \left( \frac{p^2}{2m} - \frac{1}{r} \right) \epsilon_{ijk} L_k.
\end{eqn}
}

% part C
\item \textit{
Show that for a Hamiltonian of the form
\begin{eqn}
H = \frac{p^2}{2m} + V(r)
\end{eqn}
one has
\begin{eqn}
[\v K, H] = \frac{\v r \times (\v p \times \v r)}{m} \left( \frac{-1 + r^2 V'(r)}{r^3} \right).
\end{eqn}
}

\end{enumproblem}

\section*{Problem 2 (15 pts)}
\textit{
Consider a mass point which moves on a wire which rotates with constant angular velocity.
}

\begin{enumproblem}

% part A
\item \textit{
By transforming to general coordinates which rotate with the wire show that the Lagrangian of the system is given by
\begin{eqn}
L = \frac{m}{2} (\dot r^2 + \omega^2 r^2).
\end{eqn}
}

% part B
\item \textit{
Calculate the Hamiltonian of the system. Does it change with time ?
}

% part C
\item \textit{
Calculate the kinetic energy (which is the total energy) of the mass point in terms of the generalized coordinate $r$. Is it the same as the Hamiltonian and is it conserved ?
}

\end{enumproblem}

\section*{Problem 3 (15 pts)}
\textit{
Consider a transformation of a Lagrangian to an equivalent Lagrangian, i.e.
\begin{eqn}
L(q, \dot q, t) \rightarrow L(q, \dot q, t) + \od{}{t} \Lambda(q,t).
\end{eqn}
}

\begin{enumproblem}

% part A
\item \textit{
How do the coordinates and canonical momenta change ?
}

% part B
\item \textit{
How does the Hamiltonian change
}

% part C
\item \textit{
Consider this change as a generalized coordinate change $(q, p) \rightarrow (q', p')$. Show that the Poisson braced for the new coordinates will have the same form as the old one.
}

\end{enumproblem}

\section*{Problem 4 (10 pts)}
\textit{
Consider a dynamical flow which is defined on $\R^n$ with $\v x = (x_1, \dots x_n)$ and a vector field $\v v(\v x)$ as follows
\begin{eqn}
\od{}{t} \v x(t) = \v v(\v x(t)).
\end{eqn}
}

\begin{enumproblem}

% part A
\item \textit{
Solving the differential equations and finding $\v x(t)$ defines a map from $\R^n \rightarrow \R^n$. Show that this map is volume preserving if the vector field $\v v$ is divergence free, i.e.
\begin{eqn}
\sum_i \pd{v_i}{x_i} = 0.
\end{eqn}
}

% part B
\item \textit{
Show that by setting $\v x = \set{q_1, \dots, q_n, p_1, \dots, p_n}$ and
\begin{eqn}
v^i = \sum_j J^i_j \pd{H}{{x^j}}, \qquad
J = \pmat{0 & 1_n \\ -1_n & 0},
\end{eqn}
the vector field $\v v$ is divergence free if Hamilton's equations hold. Hence you have proved Liouville's theorem.
}

\end{enumproblem}

\end{document}


\chapter{Dynamic Programming}

So far we have considered two major strategies in algorithms design:
greedy, in which we repeatedly take the local optimum choice; and
divide and conquer, in which we divide the greater problem into
similar subproblems and recurse.

There may be problems for which these strategies are suboptimal.  In
which case, we have a third strategy which may be of use: dynamic
programming.  This strategy divides the problem into sub problems, but
rather than recursing on each sub problem individually, we identify
easy to compute base cases from which we can build towards the
solution to the larger problem, storing the results of previous
computations to use in later computations.  This technique differs
from a similar technique called ``memoization'' which computes
recursively top-down, whereas dynamic programming begins at the base
case(s) and works up.

Dynamic programming has these important steps:

\begin{enumerate}
\item Determine structure of optimal solution
\item Set up recurrences for optimal solution
\item Solve recurrences bottom-up
\item Construct the optimal solution
\end{enumerate}

And the final step is most often neglected since we can typically add
some small amount of information to the process so we can trivially
reconstruct the optimal solution.

\section{Matrix Chain Multiplication}

Assuming knowledge of matrices and how they are multiplied.

The problem is to find the way to multiply $n$ matrices $A_1,...,A_n$
with dimensions $p_0,...,p_n$ (e.g. $A_i$ has dimensions $p_{i-1}
\cross p_i$) using the least number of calculations.  Notice that
multiplying $A_iA_{i+1}$ takes $p_{i-1}p_ip_{i+1}$ calculations.

We start by determining the structure of the optimal solution.
Observe that the optimal solution will necessarily involve splitting
$A_1A_2...A_n$ into two subproblems at some optimally chosen
$A_kA_{k+1}$ so we multiply $A_1...A_k$ then $A_k...A_n$ and then
multiply the results together.  If we define the function $m(i,j)$ to
be the minimum cost of multiplying $A_i...A_j$, then the cost of the
optimal solution is $m(1,n) = m(1,k) + m(k+1,n) + p_0p_kp_n$.

We then define the recurrence $m(i,i) = 0$ and $m(i,j) = min \{ m(i,k) +
m(k+1,j) + p_{i-1}p_kp_j \}$ for all $k$ such that $i < k \leq j$.

We then compute all $m(i,i)$ for $1 \leq i \leq n$, then all
$m(i,i+1)$, $m(i,i+2)$, ... until we have calculated $m(1,n)$.

The time complexity of this solution is

\[
  \summ{l=2}{n}\summ{i=1}{n-l+1}\summ{k=i}{i+l+2}\BigOh{1} = \BigOh{n^3}
\]

\section{Longest Common Subsequence}

For a string $S = (s_1,s_2,\dots,s_n)$, a subsequence of $S$ is string $S'$ such that every element of $S'$ is in $S$, and if an element $a \in S'$ comes before $b \in S'$, this also holds for $S$. The longest common subsequence of two strings $A=(a_1,a_2,\dots,a_n)$ and $B = (b_1,b_2,\dots,b_m)$, is the longest string $C = (c_1,c_2,\dots,c_k)$, such that $C$ is a subsequence of $A$ and $B$.

Let the substring $(s_i,s_{i+1},\dots,s_j)$ be denoted as $S(i,j)$. Further, let $LCS(i,j)$ to be the length of the LCS of $A(0,i)$ and $B(0,j)$. Assume that we have computed $C(1,k)$, and that it is unique. Then if we remove $c_k$ and everything after it from $B$ and $A$, $C(1,k-1)$ is now the LCS of our modified $A$ and $B$.

In general
$LCS(i,j) =$

\begin{math}
   \left\{
    \begin{array}{l l}
      \emptyset 		& \text{if $i=0$ or $j=0$}\\
      LCS(i-1,j)+1 	& \text{if $a_i=b_j$}\\
      					& \text{and $LCS(i-1,j)>LCS(i,j-1)$}\\
      LCS(i,j-1)+1 	& \text{if $a_i=b_j$}\\
      					& \text{and $LCS(i-1,j)<LCS(i,j-1)$}\\

    \end{array} \right.
\end{math}

To compute this, we compute $LCS(0,0)$, $LCS(0,1)$,\dots,$LCS(0,m)$; then $LCS(1,0)$,\dots,$LCS(1,m)$; \dots;$LCS(n,0)$,\dots,$LCS(n,m)$. At which point we have our solution, which is $LCS(n,m)$.

\section{Optimal Triangulation of a Convex Polygon}

% page 91 in John's notes

First some definitions.  A \emph{polygon} is a list of vertices
$(v_1,...,v_n)$ such that for any $v_i$, there exists an edge
$(v_i,v_{i+1})$ and also there exists an edge $(v_1,v_n)$.  A polygon
is said to be \emph{convex} if any line passing through the polygon
crosses the edges of the polygon at most twice.  A \emph{chord} is an
edge between two non-adjacent vertices in a polygon.  A
\emph{triangulation} is a set of chords which divide a polygon into
triangles.

The problem is to build a triangulation of a given convex polygon
which minimizes total edge length.  We define the function $w(a,b,c)$
to be the weight of the triangle $(v_a,v_b,v_c)$, which in this case
will be the length of the edges $(v_a,v_b)$, $(v_b,v_c)$, and
$(v_c,v_a)$.  We also define the function $t(a,b)$ to be the optimal
triangulation of points $(v_a,...,v_b)$.  We would like to solve
$t(1,n)$.

We start by defining the structure of an optimal solution.  Notice
that the optimal triangulation contains the triangle $(v_1,v_k,v_n)$
for some $k$.  The cost of this triangulation is $t(1,k) + t(k,n) +
w(1,k,n)$.

We then define the recurrence $t(i,i+1) = 0$ for all $i$, and $t(i,j)
= min \{ t(i,k) + t(k,j) + w(i,k,j) \}$ for all $k$ such that $i < k < j$.

We then compute all $t(i,i+1)$, then all $t(i,i+2)$, $t(i,i+3)$,
... until we have calculated $t(1,n)$.

\hypertarget{sec:floyd_warshall}{\section{All-Pairs Shortest Path
    (Floyd-Warshall)}}

Given a graph $G=(V,E)$ where edges have an associated weight $w(e)$
for $e \in E$, compute the shortest path between $u$ and $v$ for all
$u,v \in V$.

We start by defining the structure of an optimal solution.  Let us
decompose the problem into finding the optimal path between $v_i$ and
$v_j$.  All of the vertices on this path are contained in $\{
v_1,...,v_k \}$ except perhaps $v_i,v_j$.  Notice that if $k=0$ then
if there is an edge $(v_i,v_j) \in E$ then the distance is the weight
of that edge, and if there is no such edge then the weight is
infinite.  Otherwise, we take the minimum of either including or
excluding $v_k$ on the path.  This gives the recurrence:

\begin{math}
  d^k_{i,j} = \left\{
    \begin{array}{l l}
      weight((v_i,v_j)) & \text{if $k=0$ and} \\
                        & (v_i,v_j) \in E \\
      \infty            & \text{if $k=0$ and} \\
                        & (v_i,v_j) \not \in E \\
      min \{ d^{k-1}_{i,j},d^{k-1}_{i,k} + d^{k-1}_{k,j} \} & \text{if $k>0$}
    \end{array} \right.
\end{math}

We then compute $d^0_{i,j}$ for all $v_i,v_j \in V$, then all
$d^1_{i,j}$, $d^2_{i,j}$, ... until we have computed $d^{n-2}_{i,j}$.

Running time: $\BigOh{n^3}$.  Analysis is left as an exercise to the
reader.

\section{Knapsack}

Given a knapsack (or bag) which can carry $W$ units of weight, and $n$
items where item $i$ has weight $w_i$ and value $v_i$, what is the
most valuable list of items which can fit in the given bag?  Notice
that this may include multiples of a particular item.

We start by defining the structure of an optimal solution.  Let $p(w)$
be the value of the optimal packing of a bag which can carry $w$ units
of weight, and let $p_i(w)$ be the same but necessarily including item
$i$.

\begin{align*}
  p_i(w) &= p(w - w_i) + v_i & \\
  p(0)   &= 0 & \\
  p(w)   &= max \{ p(w - w_i) + v_i \} & \text{where $i : w_i \leq w$} \\
\end{align*}

We then calculate $p(0), p(1), ..., p(W)$.

The running time for this algorithm is $\BigOh{nW}$.  Notice that $W$
requires $\BigOh{logW}$ bits to represent.  Since the input to the
problem is $\BigOh{n + logW}$, and $W = 2^{logW}$, the time complexity
is $\BigOh{n2^{logW}}$ so this solution is exponential with respect to
the input.

\section{String Edit Distance}

We define the edit distance between two strings as the minimum number
of edits necessary to transform one string into another, where edits
are insertions, deletions, or replacements of a single character.

The problem is to compute this edit distance, given strings $X$ of
length $m$ and $Y$ of length $n$.

We start by defining the structure of an optimal solution.  Let us
define $E[i,j]$ as the minimal edit distance between $X[1..i]$ and
$Y[1..j]$.  We would like to know $E[m,n]$.

It should be easy to see that $E[0,0] = 0$ and $E[1,1]$ is either $0$
if $X[1] = Y[1]$ or $1$ otherwise.  $E[0,j] = j$ and $E[i,0] = i$.

This gives the recurrence:

\begin{math}
  E[i,j] = \left\{
    \begin{array}{l l}
      0 & \text{if } i=0,j=0 \\
      i & \text{if } j=0 \\
      j & \text{if } i=0 \\
      min \{ 1 + E[i-1,j], 1 + E[i,j-1], 1 + E[i-1,j-1] \} & \text{if } X[i] \neq Y[j] \\
      min \{ 1 + E[i-1,j], 1 + E[i,j-1], E[i-1,j-1] \} & \text{otherwise} \\
    \end{array} \right.
\end{math}

Then we calculate $E[i,j]$ from $i=0,j=0$ to $i=m,j=n$.


\subsection{Connected and separated sets}

Two subsets of \(X\) in topological space \(T\) are separated if each subset is disjoint from the other's closure.

So \([-1,0)\) and \((0,1)\) are separated.

\([-1,0]\) and \((0,1)\) are not separated.

Sets which are not separated are connected.


\subsection{Boolean functions}\label{subsec:boolean_functions}

\begin{definition}\label{def:boolean_value}
  Fix a two-element set \( \set{ T, F } \). We can think of \( T \) as a value denoting truth and \( F \) as denoting falsity. See \fullref{rem:mathematical_logic_conventions/propositional_constants} for notation conventions.

  There is a natural \hyperref[def:boolean_algebra]{Boolean algebra} structure on \( \set{ T, F } \) where \( T \) is the \hyperref[def:partially_ordered_set_extremal_points/top_and_bottom]{top} and \( F \) is the \hyperref[def:partially_ordered_set_extremal_points/top_and_bottom]{bottom} and the operations are defined in an obvious way.

  By \fullref{thm:binary_boolean_algebra}, all two-element Boolean algebras are isomorphic, so our choice of symbols does not matter much. We do sometimes identify \( T \) with \( 1 \) and \( F \) with \( 0 \) in the \hyperref[thm:finite_fields]{prime field} \( \BbbF_2 \). The latter is a Boolean algebra as discussed in \fullref{thm:f2_is_boolean_algebra}.
\end{definition}

\begin{definition}\label{def:boolean_function}
  We call functions from any set to \( \set{ T, F } \) (Boolean-valued) \term{predicates} and functions from \( \set{ T, F }^n \) to \( \set{ T, F } \) \term{Boolean functions}.
\end{definition}

\begin{remark}\label{rem:boolean_valued_functions_and_predicates}
  \hyperref[def:boolean_function]{Boolean-valued functions} and \hyperref[def:relation]{relations} represent the same concept. In particular, the relation \( R \subseteq A_1 \times \cdots \times A_n \) corresponds to a unique Boolean-valued function
  \begin{equation*}
    \begin{aligned}
      &f: X_1 \times \cdots \times X_n \to \set{ T, F } \\
      &f(x_1, \ldots, x_n) = \begin{cases}
        T, &(x_1, \ldots, x_n) \in R, \\
        F, &\T{otherwise}
      \end{cases}
    \end{aligned}
  \end{equation*}
  and vice versa.
\end{remark}

\begin{definition}\label{def:boolean_closure}
  Fix a set \( B \) of Boolean functions of arbitrary arities.

  The \term{closure} \( \cl{B} \) of \( B \) is defined \hyperref[rem:structural_recursion_and_induction]{recursively} as follows:
  \begin{itemize}
    \item If \( f \in B \), then \( f \in \cl{B} \)
    \item If \( f_k(x_1, \ldots, x_n) \in \cl{B} \) for \( k = 1, \ldots, m \) and if \( g(x_1, \ldots, x_m) \in \cl{B} \), then their \hyperref[def:multi_valued_function/superposition]{superposition}
    \begin{equation*}
      h(x_1, \ldots, x_n) \coloneqq g(f_1(x_1, \ldots, x_n), \ldots, f_m(x_1, \ldots, x_n))
    \end{equation*}
    is also in \( \cl{B} \).
  \end{itemize}

  We say that \( B \) is \term{closed} if \( \cl{B} = B \) and \term{complete} if \( \cl{B} \) is the set of all Boolean functions of arbitrary arity.

  If \( B \) is complete, then from \fullref{thm:functions_over_model_form_model} it follows that \( B \) is a Boolean algebra. This is used in \fullref{thm:lindenmaum_tarski_algebra_of_full_propositional_logic/bijection}.
\end{definition}

\begin{definition}\label{def:zhegalkin_polynomial}
  A \term{Zhegalkin polynomial} is a \hyperref[def:polynomial_algebra]{polynomial} in the \hyperref[thm:finite_fields]{prime field} \( \BbbF_2 \). Due to \fullref{thm:functions_over_prime_fields}, however, we restrict ourselves to polynomials with \hyperref[def:square_free]{square-free} monomials.

  For example, for every binary Boolean function there exist coefficients \( a, b, c, d \in \BbbF_2 \) such that
  \begin{equation}\label{eq:def:zhegalkin_polynomial/binary_polynomial}
    f(x, y) = axy \oplus bx \oplus cy \oplus d.
  \end{equation}
\end{definition}

\begin{definition}\label{def:standard_boolean_operators}
  Unlike \hyperref[def:function]{arbitrary functions}, \hyperref[def:boolean_function]{Boolean functions} only have a small finite number of possible values that can easily be enumerated.

  Out of the following binary operations, \( \vee \), \( \wedge \) and \( \overline{\anon} \) form the \hyperref[def:boolean_algebra]{Boolean algebra} structure on \( \BbbF_2 \) and \( \oplus \) and \( \wedge \) form the \hyperref[def:field]{field} structure on \( \BbbF_2 \). The operations \( \rightarrow \) and \( \leftrightarrow \) are also defined in any \hyperref[def:boolean_algebra]{Boolean algebra}.

  \begin{center}
    \begin{tabular}{c | c || c c | c c c c c c}
      \( x \) & \( \overline{x} \) & \( x \) & \( y \) & \( x \vee y \)             & \( x \oplus y \)    & \( x \wedge y \)        & \( x \rightarrow y \)   & \( x \leftrightarrow y \) \\
      \hline
              & not \( x \)        &         &         & \( x \) or \( y \)         & \( x \) xor \( y \) & \( x \) and \( y \)     & \( x \) implies \( y \) & \( x \) iff \( y \)       \\
      \hline
      \( F \) & \( T \)            & \( F \) & \( F \) & \( F \)                    & \( F \)             & \( F \)                 & \( T \)                 & \( T \)                   \\
      \( T \) & \( F \)            & \( F \) & \( T \) & \( F \)                    & \( T \)             & \( F \)                 & \( T \)                 & \( F \)                   \\
              &                    & \( T \) & \( F \) & \( F \)                    & \( T \)             & \( F \)                 & \( F \)                 & \( F \)                   \\
              &                    & \( T \) & \( T \) & \( T \)                    & \( F \)             & \( T \)                 & \( T \)                 & \( T \)                   \\
      \hline
              & \( x \oplus 1 \)   &         &         & \( xy \oplus x \oplus y \) & \( x \oplus y \)    & \( xy \)            & \( xy \oplus x \oplus 1 \) & \( x \oplus y \oplus 1 \)
    \end{tabular}
  \end{center}

  See \fullref{thm:boolean_equivalences} for direct consequences of these definitions.
\end{definition}

\begin{definition}\label{def:boolean_functions_in_f2}\mcite[I.1.\S6]{Яблонский1986}
  Fix a \hyperref[def:boolean_function]{Boolean function} \( f(x_1, \ldots, x_n) \) in the \hyperref[thm:finite_fields]{prime field} \( \BbbF_2 \).

  \begin{thmenum}
    \thmitem{def:boolean_function_in_f2/dual} Its \term{dual function} is
    \begin{equation*}
      \overline{f}(x_1, \ldots, x_n) \coloneqq \overline{f(\overline{x_1}, \ldots, \overline{x_n})}.
    \end{equation*}

    \thmitem{def:boolean_functions_in_f2/self_dual} \( f \) is \term{self-dual} if it is its own \hyperref[def:boolean_function_in_f2/dual]{dual}.

    \thmitem{def:boolean_functions_in_f2/truth_preserving} \( f \) is \term{truth-preserving} if \( f(T, \ldots, T) = T \).

    \thmitem{def:boolean_functions_in_f2/falsity_preserving} \( f \) is \term{falsity-preserving} if \( f(F, \ldots, F) = F \).

    \thmitem{def:boolean_functions_in_f2/monotone} \( f \) is \term{monotone} if, for any two tuples of arguments \( x_1, \ldots, x_n \in \BbbF_2 \) and \( y_1, \ldots, y_n \in \BbbF_2 \), the inequalities \( x_k \leq y_k \) for all \( k \in \set{ 1, \ldots, n } \) imply that
    \begin{equation*}
      f(x_1, \ldots, x_n) \leq f(y_1, \ldots, y_n).
    \end{equation*}

    \thmitem{def:boolean_functions_in_f2/linear} \( f \) is \term{linear} if its \hyperref[def:zhegalkin_polynomial]{Zhegalkin polynomial} is linear, i.e. has only monomials of degree \( 0 \) or \( 1 \). In the case of binary Boolean functions, this means that the coefficient \( a \) in \eqref{eq:def:zhegalkin_polynomial/binary_polynomial} is zero.
  \end{thmenum}
\end{definition}

\begin{theorem}[Post-Yablonsky completeness theorem]\label{thm:posts_completeness_theorem}\mcite[thm. I.1.7]{Яблонский1986}
  The family \( B \) of Boolean functions is \hyperref[def:boolean_closure]{complete} if and only if all of the following conditions are satisfied:
  \begin{thmenum}
    \thmitem{thm:posts_completeness_theorem/truth_preserving} \( B \) contains a function that is not \hyperref[def:boolean_functions_in_f2/truth_preserving]{truth-preserving}.
    \thmitem{thm:posts_completeness_theorem/falsity_preserving} \( B \) contains a function that is not \hyperref[def:boolean_functions_in_f2/falsity_preserving]{falsity-preserving}.
    \thmitem{thm:posts_completeness_theorem/self_dual} \( B \) contains a function that is not \hyperref[def:boolean_functions_in_f2/self_dual]{self-dual}.
    \thmitem{thm:posts_completeness_theorem/monotone} \( B \) contains a function that is not \hyperref[def:boolean_functions_in_f2/monotone]{monotone}.
    \thmitem{thm:posts_completeness_theorem/linear} \( B \) contains a function that is not \hyperref[def:boolean_functions_in_f2/linear]{linear}.
  \end{thmenum}
\end{theorem}

\begin{example}\label{ex:thm:posts_completeness_theorem}
  We give examples of complete sets of Boolean functions in \( \BbbF_2 \).

  \begin{thmenum}
    \thmitem{ex:thm:posts_completeness_theorem/and_or} The archetypic example of a complete set of Boolean functions is the triple \( \vee, \wedge, \overline{\anon} \) that forms the Boolean algebra structure on \( \BbbF_2 \).

    We verify that the conditions of \fullref{thm:posts_completeness_theorem} are satisfied:
    \begin{refenum}
      \refitem{thm:posts_completeness_theorem/truth_preserving} \( \overline{\anon} \) is not truth-preserving.
      \refitem{thm:posts_completeness_theorem/falsity_preserving} \( \overline{\anon} \) is not falsity-preserving.
      \refitem{thm:posts_completeness_theorem/self_dual} Neither \( \vee \) nor \( \wedge \) are self-dual. In fact, due to \fullref{thm:de_morgans_laws}, \( \wedge \) is the dual of \( \vee \) and vice versa.
      \refitem{thm:posts_completeness_theorem/monotone} \( \overline{\anon} \) is not monotone.
      \refitem{thm:posts_completeness_theorem/linear} Neither \( \vee \) nor \( \wedge \) have linear Zhegalkin polynomials.
    \end{refenum}

    Thus, \( \set{ \wedge, \vee, \overline{\anon} } \) is indeed a complete set of Boolean functions. Note that having both \( \vee \) and \( \wedge \) is redundant and we usually include both for symmetry. The families \( \set{ \wedge, \overline{\anon} } \) and \( \set{ \vee, \overline{\anon} } \) are both complete.

    This is utilized for \hyperref[def:cnf_and_dnf]{conjunctive and disjunctive normal forms}.

    \thmitem{ex:thm:posts_completeness_theorem/nand} We can go even further and have a single binary Boolean function generate all others. We will use the function
    \begin{equation}\label{eq:ex:thm:posts_completeness_theorem/nand}
      (x \uparrow y) \coloneqq \overline{x \wedge y} = xy \oplus 1.
    \end{equation}

    This operation is called \term{Sheffer's stroke} or \term{nand} (\enquote{not and}).

    We have
    \begin{equation*}
      \begin{array}{ccc}
        \overline{x} = (x \uparrow 1)
        &
        \T{and}
        &
        (x \wedge y) = \overline{x \uparrow y},
      \end{array}
    \end{equation*}
    which allows us to reduce the case to \fullref{ex:thm:posts_completeness_theorem/and_or}. We conclude that the singleton set \( \set{ \uparrow } \) is a complete set of Boolean operations.

    \thmitem{ex:thm:posts_completeness_theorem/conditional_negation} Another commonly used complete family is \( \set{ \rightarrow, \overline{\anon} } \).
    We verify that the conditions of \fullref{thm:posts_completeness_theorem} are satisfied:
    \begin{refenum}
      \refitem{thm:posts_completeness_theorem/truth_preserving} \( \overline{\anon} \) is not truth-preserving.
      \refitem{thm:posts_completeness_theorem/falsity_preserving} \( \rightarrow \) is not falsity-preserving because \( (F \rightarrow F) = T \).
      \refitem{thm:posts_completeness_theorem/self_dual} \( \rightarrow \) is not self-dual because \( \overline{\overline{x} \rightarrow \overline{y}} \reloset {\eqref{eq:thm:boolean_equivalences/contrapositive}} = (y \rightarrow x) \neq (x \rightarrow y) \).
      \refitem{thm:posts_completeness_theorem/monotone} \( \rightarrow \) is not monotone because \( F \rightarrow T = F \).
      \refitem{thm:posts_completeness_theorem/linear} \( \rightarrow \) doesn't have a linear Zhegalkin polynomial.
    \end{refenum}

    \thmitem{ex:thm:posts_completeness_theorem/conditional_bottom} Given the family \( \set{ \rightarrow, F } \), we can define
    \begin{equation*}
      \overline{x} \coloneqq (x \rightarrow F),
    \end{equation*}
    which shows that \( \set{ \rightarrow, F } \) is also a complete family.
  \end{thmenum}
\end{example}


%\chapter{Multigrid Methods}\label{chap:introMG}
In this chapter, we will introduce the multigrid
methods for numerical partial differential equations with
some notation in convolutional neural networks as discussed in
previous sections.
The main purpose of this section is to give three different
version of multigrid methods: recursion form, method of subspace
correction form and expansion form.

\input{4MSC/4ChTri-tenmatrix}

\input{4MSC/4ChLocality}

\section{Some basic components for multigrid methods}\label{sec:mg}
Let us first briefly describe a geometric multigrid method used to solve the
following boundary value problem
\begin{equation}
\label{laplace}
-\Delta u = f,  \mbox{ in } \Omega,\quad
 u=0  \mbox{ on } \partial\Omega,\quad
\Omega=(0,1)^2.
\end{equation}

We consider a continuous linear finite element discretization of
\eqref{laplace} on a nested sequence of grids of sizes $n_\ell\times
n_\ell$ with $n_{\ell}=2^{J-\ell+1} + 1$, as shown in the left part of
Fig. \ref{mgrid} and the corresponding sequence of finite
element spaces \eqref{Vk}.
%Here we need to notice that, $n_\ell = 2^{k_\ell} + 1$ for general PDEs grid
%with the above boundary condition. For general images, we can take them as
%discrete functions on grid with size $n_\ell = 2^{k_\ell}m$ with small $m = 1,3,\cdots$.
%Then generally speaking, the coarse grid size is $n_{\ell+1} = \frac{n_\ell}{2}=2^{k\ell - 1}m$.

Based on the grid $\mathcal T = \mathcal T_\ell$, the discretized system is
\begin{equation}
\label{laplace-h}
Au=f.
\end{equation}
Here,  $A:\mathbb R^{n\times n}\mapsto \mathbb R^{n\times n}$ is a tensor satisfying
\begin{equation}
\label{uniform-laplace}
(Au)_{i,j}=4u_{i,j}-u_{i+1,j}-u_{i-1,j}-u_{i,j+1}-u_{i,j-1},
\end{equation}
which holds for $1\le i,j \le n$ with zero padding.
Here we notice that, there exists a $3\times 3$ kernel as
\begin{equation}\label{eq:kernel-A}
K_A = \begin{pmatrix}
0 & -1 & 0 \\
-1 & 4 & -1 \\
0 & -1 & 0
\end{pmatrix},
\end{equation}
with
\begin{equation}\label{eq:convA}
Au = K_A \ast u.
\end{equation}
Where $\ast$ is the stander convolution operation with zero padding like \eqref{con1}.
We now briefly describe a simple multigrid method by a mixed use of the terminologies from
deep learning \cite{goodfellow2017deep} and multigrid methods.

The gradient descent method can be derived as following
\begin{itemize}
	\item $\min_{x\in \mathbb{R}^n} F(x)$ with $ F(x) =
          \frac{1}{2}u^TA u - f^Tu$
	\item Gradient:
	$$
	\nabla F= Au - f := - r.
	$$
	\item Given $u^k$, a new approximate solution is computed by two steps:
	$$
	r^k = f-Au^k,\quad u^{k+1} = u^k + \omega r^k.
	$$
\end{itemize}
Here we can clearly see that gradient descent method is just Jacobi method.
And the method can be written as
$$
u^{k+1} = u^k + \omega (f-Au^k).
$$
Denote $e^k=u-u^k$, then the error equation of the method is as follows
$$
e^{k+1} = (I -\omega A)e^k.
$$
the method converges if and only if $\rho(I-\omega A)< 1$, a sufficient condition is $\|I-\omega A\|< 1$ which means $\omega<\frac{1}{8}$.

The first main ingredient in GMG is a smoother.  A commonly used smoother is a
damped Jacobi with damped coefficient $\omega$ with $\omega \in (0,2)$,  which can be written as $S_{0}:\mathbb R^{n\times n}\mapsto
\mathbb R^{n\times n}$ satisfying
\begin{equation}
\label{jacobi1}
(S_{0}f)_{i,j}={\omega\over 4}f_{i,j},
\end{equation}
for equation \eqref{laplace-h} with initial guess zero.
If we apply the Jacobian iteration twice, then
$$
S_1(f) = S_{0} f + S(f - A(S_{0}f)),
$$
with element-wise form
\begin{equation}
\begin{aligned}
\label{jacobi2}
[S_1(f)]_{i,j} &={1\over 4}\omega(2-\omega)f_{i,j} + {\omega^2\over 16}(f_{i+1,j}+f_{i-1,j}+f_{i,j+1}+f_{i,j-1}).
\end{aligned}
\end{equation}
Then we have
\begin{equation}\label{eq:kernel-S}
K_{S_{0}} = {\omega \over 4},
\end{equation}
and
\begin{equation}\label{eq:kernel-S2}
K_{S_1} = \begin{pmatrix}
0 & \frac{\omega^2}{16} & 0 \\
\frac{\omega^2}{16} & {\omega(2-\omega) \over 4} & \frac{\omega^2}{16}  \\
0 & \frac{\omega^2}{16}  & 0
\end{pmatrix},
\end{equation}
such that
\begin{equation}\label{eq:convS}
S_{0}f = K_{S_{0}} \ast f \quad S_1 f = K_{S_1} \ast f.
\end{equation}
Similarly, we can define
$S^{\ell}: \mathbb{R}^{n_\ell \times n_\ell} \mapsto \mathbb{R}^{n_\ell \times n_\ell}$.

We use prolongation $P_{\ell+1}^\ell: R^{n_{\ell+1}\times n_{\ell+1}}\mapsto R^{n_{\ell}\times n_{\ell}}$
as defined in \eqref{mg-prolong} and restriction $R_{\ell}^{\ell+1} = (P_{\ell+1}^{\ell})^T$. Further more,
we use the following relationship to define coarse operation
\begin{equation}\label{eq:def_coarse}
A^{\ell+1}=R_{\ell}^{\ell+1} A^{\ell}P_{\ell+1}^{\ell} \quad (\ell = 1:J-1),
\end{equation}
with $A^1 = A$.

An important results in multilevel finite element methods is that
if we take the restriction as \eqref{eq:restriction} with prolongation as
the transposition of restriction, we will have $A^2$ is still a convolution operation
with $K_{A^2} = K_{A^1} = K_A$ as in \eqref{eq:kernel-A}.

\section{Recursion form of multigrid methods}
In almost multigrid literatures \cite{trottenberg2000multigrid, briggs2000a},
the multigrid algorithms are expressed in a recursion form. Also, in the implementation
for multigrid method, the recursion form is much easier for coding.
Without loss of generality, we list a general V-cycle or W-cycle multigrid forms
with recursion form with iterative scheme
$$
u = {\text{MG}}(u, f; 1, \nu, J).
$$
\begin{breakablealgorithm}%[!htb]
	\caption{$ u = {\text{MG}}(u, f; \ell, \nu, J)$}
	\label{alg:MG}
	\begin{algorithmic}
		\If{$\ell == J$}
		\State Coarsest level: exact solve
		$$
		u = [A^J]^{-1} f;
		$$
		\EndIf
		\State Presmoothing
		\For{$i = 1:\nu$}
		\State
		$$
		u \leftarrow u + S^{\ell} (f - A^\ell u).
		$$
		\EndFor
		\State Restriction
		$$
		r = R_\ell^{\ell+1} (f - A^\ell u).
		$$
		\State Coarse grid correction
		$$
		e = \text{MG}(0,r, \ell+1, \nu, J);
		$$
		\If{ W-cycle}
		\State
		$$
		e = \text{MG}(e, r, \ell+1,\nu J);
		$$
		\EndIf
		\State Prolongation
		$$
		u=u+P_{\ell+1}^\ell e;
		$$
		\State Pre-smoothing:
		\For{$i = 1:\nu$}
		\State
		\begin{equation}\label{eq:smoothing}
		u \leftarrow u + [S^{\ell}]^T (f - A^\ell u).
		\end{equation}
		\EndFor
	\end{algorithmic}
\end{breakablealgorithm}

\begin{remark}
	Here we need to notice that we use $\mathcal V_1$ for the finest
	space and $\mathcal V_J$ for the coarser space to consistent
	with the CNN terminology. However this is opposite to the general
	multigrid literatures \cite{xu1992iterative, xu2002method,
		xu2016algebraic, trottenberg2000multigrid, briggs2000a}.
\end{remark}

\section{Method of subspace correction form for multigrid methods}
That the multigrid methods can be interpreted as a
successive subspace correction method is a very surprising
discovery by Prof. Xu in \cite{xu1989theory, xu1992iterative}.
Here we give a simple introduction for this method.
One of the most important idea for this theory is that
we focus on the abstract function space and its duality not
just the represented vector space \cite{xu1992iterative, xu2017algebraic}.

We consider a sequence of spaces $\mathcal V_{1},\ldots,\mathcal V_{J}$.
These spaces, which will be known as {\it auxiliary spaces}, are not
necessarily subspace of $\mathcal  V$
(such as the finest space as mentioned in \S \ref{sec:CNNs} and above),
but each of them is related to the
original space $\mathcal  V$ by a linear operator
$$
\Pi_{j}: \mathcal V_{j} \mapsto \mathcal V.
$$
Our very basic assumption is that the following decomposition holds:
\begin{equation}
\label{aux-decomp}
\mathcal V=\sum_{i=1}^J\Pi_i \mathcal V_i.
\end{equation}
This means that for any $v\in \mathcal  V$, there exists $v_i\in \mathcal  V_i$ (which may
not be unique) such that
\begin{equation}
\label{aux-decomp0}
v=\sum_{i=1}^J \Pi_i v_i.
\end{equation}
Furthermore, we assume that each $\mathcal  V_i$ is equipped with an energy inner product
$a_i(\cdot,\cdot)$.

We define
$$
 A^i:  \mathcal  V_i\mapsto \mathcal V_i'
$$
by
$$
\langle A^i w_i,\phi_i \rangle= (w_i,\phi_i)_{A^i}:=a_i(w_i,\phi_i), \quad w_i,\phi_i\in V_i.
$$
Let $\Pi_i':  \mathcal  V' \mapsto \mathcal V_i'$ be the adjoint of $\Pi_i$:
$$
\langle \Pi_i'f, v_i\rangle=\langle f,\Pi_i v_i\rangle, \quad f\in \mathcal  V',  v_i\in \mathcal V_i.
$$
Let $P_i=\Pi_i^*: \mathcal V\mapsto \mathcal V_i$ be the adjoint of $\Pi_i$ with respect
to the A-inner products:
$$
(P_iu,v_i)_{A^i}=(u,\Pi_iv_i)_{A}, u\in \mathcal V, v_i\in \mathcal V_i.
$$

\begin{lemma}
	The following identity holds
	\begin{equation}
	\label{PiAAP}
	\Pi_i'A=A^iP_i.
	\end{equation}
\end{lemma}

If $u$ is the solution of \eqref{laplace-h}, by \eqref{PiAAP}, we have
\begin{equation}
\label{3.3a}A^iu_i=f_i
\end{equation}
where
$$
u_i=P_iu, \quad f_i=\Pi_i'f
$$
This equation may be regarded as the restriction of  \eqref{laplace-h} to
$\mathcal V_i$.
We assume that each such $A_i$ has an approximate inverse or preconditioner:
\begin{equation}
\label{Ri}
S^i: \mathcal V_i'\mapsto \mathcal V_i.
\end{equation}

This type of algorithm is similar to the Gauss-Seidel method.
We here make the correction in one subspace at a time by using the most
updated approximation of $u$.  More precisely, starting from
$v^{0}= u^{\text{old}}$ and correcting its residual in $\mathcal  V_1$ gives
$$
v^1=v^{0}+\Pi_1R_1\Pi_1'(f-Av^0).
$$
By correcting the new approximation $v^1$ in the next space $\mathcal V_2$,
we get
$$
v^2=v^1+\Pi_2R_2\Pi_2'(f-Av^1).
$$
Proceeding this way successively for all $\mathcal V_i$ leads to
\begin{breakablealgorithm}%[!htb]
	\caption{$u^T = MSC(u^0)$}
	\label{alg:MSC}
	\begin{algorithmic}
		\For{$t=0,1,\ldots T$  or till convergence}
		\State  Initialization in one cycle
		$$
		v \leftarrow u^t.
		$$
		\For {$i=1:K$}
		\State Correction in subspace $V_i$
		$$
		v \leftarrow v+\Pi_iS^i\Pi_i'(f-Av).
		$$
		\EndFor
		\State Update for one cycle
		$$
		u^{t+1} \leftarrow v.
		$$
		\EndFor
	\end{algorithmic}
\end{breakablealgorithm}

To prove that this above Algorithm \ref{alg:MSC} can be
equivalent to the original form of multigrid in Algorithm \ref{alg:MG}
is non-trivial. We can prove it by investigate its error propagation operator.

Let $T_i = \Pi_iR_i\Pi_i'A$. By \eqref{PiAAP}, $T_i=\Pi_iR_iA_iP_i.$ Note that $T_i:
\mathcal  V \mapsto \mathcal  V$ is symmetric with respect to $(\cdot, \cdot)_A$ and nonnegative
and that $T_i=\Pi_iP_i$ if $S^i=A_i^{-1}$.

If $u$ is the exact solution of \eqref{laplace-h}, then $f=Au$.  Let $v^i$ be
the $i-th$ iterate (with $v^0=u^k$) from Algorithm \ref{alg:MSC}, we have by
definition
$$
u-v^{i+1}=(I-T_i)(u-v^i),\quad i=1,\cdots,K.
$$
A successive application of this identity yields
\begin{equation}\label{error}
u-u^t=E_K(u-u^{t-1}),
\end{equation}
where
\begin{equation}\label{Em}
E_K=(I-T_K)(I-T_{K-1})\cdots(I-T_1).
\end{equation}

It is easy to see that Algorithm \ref{alg:MSC} is equivalent to
$$
u^t=u^{t-1}+B(f - Au^{t-1}), \quad t=1,2,\ldots,
$$
if $B: \mathcal V' \mapsto \mathcal V$ is given by
\begin{equation}
\label{SSC-B}
I-BA= (I-T_K)(I-T_{K-1})\cdots(I-T_1).
\end{equation}

With the same idea, Algorithm \ref{alg:MG} can also be write as
$$
u^t=\text{V-MG}(u^{t-1},f), \quad t=1,2,\ldots,
$$
with
$$
u^{t} = u^{t-1} + B^{\text{V-MG}}(f - Au^{t-1}),
$$
because of its linearity. Here $\text{V-MG}$ means the V-cycle multigrid
methods.
Then one can prove that $B: \mathcal V' \mapsto \mathcal V$ is given by
\begin{equation}
I-B^{\text{V-MG}}A= (I-T_K)(I-T_{K-1})\cdots(I-T_1),
\end{equation}
if $K = 2J-1$ with $\mathcal V_{2J-i} = \mathcal V_i$  and
$S^{2J-i} = [S^i]^T$ for $i = 1:J-1$.

There is a beautiful result by investigating multigrid methods
as the method of subspace correction which is so-called
X-Z identity \cite{xu2002method}.
\begin{theorem}[X-Z Identity($c_0$)]\label{th:xzidentityc0}
	Assume that $B$ is defined by Equation (\ref{SSC-B}). Then
	\begin{equation}\label{eq:xzidentityc0}
	\|I-BA\|_A^2=1-\frac{1}{1+c_0}=1-\frac{1}{c_1},
	\end{equation}
	where
	\begin{equation}\label{eq:xzc0}
	c_0=\sup_{\|v\|_A=1}\inf_{\sum
		\Pi_i v_i=v}\sum_{i=1}^J\|[S^i]^tw_i\|_{{\overline R_i}^{-1}}^{2}.
	\end{equation}
	with $w_i=A^iP_i\sum_{j\geq i} \Pi_j v_j-R_i^{-1}v_i$,
	and
	\begin{equation}\label{eq:xzc1}
	c_1=\sup_{\|v\|_A=1}\inf_{\sum \Pi_i v_i=v}\sum_{i=1}^J
	\left\|\overline R_i R_i^{-1} v_i +R_i^t w_i\right\|_{\overline R_i^{-1}}^2
	\end{equation}

	In particular, for $S^i=[A^i]^{-1}$, we have
	\begin{equation}
	\label{eq:c0p}
	c_0=\sup_{\|v\|_A=1}\inf_{\sum \Pi_i v_i=v}\sum_{i=1}^J\left\| P_i\sum_{j> i}v_j\right\|_{A^i}^{2}.
	\end{equation}
	and
	\begin{equation}\label{eq:c1p}
	c_1=\sup_{\|v\|_A=1}\inf_{\sum \Pi_i v_i=v}\sum_{i=1}^J
	\left\|P_i\left(\sum_{j\geq i}v_j\right)\right\|_{A}^2.
	\end{equation}
\end{theorem}

\section{Expansion form of multigrid methods}\label{sec:mg_expansion}
Despite the convenience of recursion form in implementation
of multigrid methods especially for some complex cycles, the computer can
only execute it with its expansion (unfolded) form step by step.

Here, we are going to use the smoother $S^\ell$, prolongation $P^{\ell}_{\ell+1}$, restriction $R_{\ell}^{\ell+1}$ and mapping
$A^\ell$ as given in \eqref{eq:def_coarse}  to formulate the following algorithm
as a major component of a multigrid algorithm.
\begin{breakablealgorithm}%[!htb]
	\caption{$(u^{\ell,\nu_\ell}: ~\ell = 1:J) = {\text{MG0}}(f; J,\nu_1, \cdots, \nu_J)$}
	\label{alg:L-Slash0}
	\begin{algorithmic}
		\State Set up
		$$
		f^1 = f, \quad u^{1,0}=0.
		$$
		\State Smoothing and restriction from fine to coarse level (nested)
		\For{$\ell = 1:J$}
		\State Pre-smoothing:
		\For{$i = 1:\nu_\ell$}
		\State
		\begin{equation}\label{eq:smoothing}
		u^{\ell,i} = u^{\ell,i-1} + S^{\ell} (f^\ell - A^\ell u^{\ell,i-1}).
		\end{equation}
		\EndFor
		\State Form restricted residual and set initial guess:
		$$
		u^{\ell+1,0} = 0, \quad f^{\ell+1} = R^{\ell+1}_\ell(f^\ell - A^{\ell} u^{\ell,\nu_\ell}).
		$$
		\EndFor
	\end{algorithmic}
\end{breakablealgorithm}

Using the above algorithm, there are different multigrid algorithms such as: $\backslash$-cycle, V-cycle and W-cycle.
Let us now only give one special form of multigrid algorithm as follows.
\begin{breakablealgorithm}%[!htb]
	\caption{$u = {\backslash\text{-MG}}(f; J,\nu_1, \cdots, \nu_J)$}
	\label{alg:L-Slash1}
	\begin{algorithmic}
		\State Call Algorithm \ref{alg:L-Slash0},
		$$
		(u^{\ell,\nu_\ell}: ~\ell = 1:J) = {\text{MG0}}(f; J,\nu_1, \cdots, \nu_J).
		$$
		\State Prolongation and restriction from coarse to fine level
		\For{$\ell = J-1:1$}
		\State
		$$
		u^{\ell,\nu_\ell} \leftarrow u^{\ell,\nu_\ell} + P_{\ell+1}^{\ell}u^{\ell+1, \nu_{\ell+1}}.
		$$
		\EndFor
		\State Output
		$$
		u = u^{1,\nu_\ell}.
		$$
	\end{algorithmic}
\end{breakablealgorithm}

No one would like to implement a multigrid algorithm with the above form
especially for some complex cycles. However, we will show that the fine to
coarse process in expansion form as in Algorithm \ref{alg:L-Slash0} has a deep
connection with the convolutional neural networks.


\subsection{Coarse grid correction and two grid method}
%\vspace{-1.8cm}
\begin{figure}[hpt]
\begin{center}
\includegraphics*[width=3in]{figures/twogrid.pdf}
%\vspace{-1.5cm}
\caption{Two level grids.}
\label{Interpolation}
\end{center}
\end{figure}

\begin{figure}[!htb]
\begin{center}
\includegraphics[width=3in]{pictures/manygr.pdf}
\end{center}
\caption{Multiple grids in one dimension
\label{fig:manygrids}}
\end{figure}

%\vspace{-0.2cm}
As we discussed earlier, although gradient descent iteration usually
converges very slowly, it does quickly smooth out the rough
component in the error. In other words, the error becomes smooth
after a few gradient descent iterations on a fine grid, but looks
rough when viewed on a coarser grid. Hence, a few gradient descent
iterations can further reduce the error on the coarse grid.
The main idea of two grid method or multigrid method is
to use the fact that a smooth function can be well
approximated on a coarser grid.

In summery, we can write the two grid method in terms of finite element (FE) functions as follows:
\begin{algorithm}\caption{A two grid method (in terms of FE functions)}\Label{alg:2grid:opezero}
Input $u^0$.
\begin{enumerate}
\item [{\bf Step 1:}] Apply $\nu_1$-times gradient descent iterations for
$$
\min_{v^1\in V_1} J(v^1)
$$
with initial guess $u^0$ to obtain $u^1\in V_1$.
\item [{\bf Step 2:}]  Apply $\nu_2$-times gradient descent iterations for
$$
\min_{v^2\in V_2} J(u^1+v^2)
$$
with zero initial guess to obtain $u^2\in V_2$.
\item [{\bf Step 3:}]  Update: $u=u^1+u^2$.
%\item [{\bf Step 4:}]  Stop if converge or $u^0\update u$ and continue from step 1.
\end{enumerate}
\end{algorithm}

\newpage

\subsubsection{Realization of step 1:}
\smallskip\hrule \smallskip

Step 1: Given $u^{1,0}\in V_1$, apply $\nu_1$-times gradient descent method for
$$
\min_{v^1\in V_1} J(v^1)
$$
with initial guess $u^{1,0}$ to obtain $u^1\in V_1$.
\smallskip\hrule \smallskip

Let $\displaystyle u^{1,0}=\sum_{i=1}^{n_1}\mu^0_i\phi_i^1,\quad v^1=\sum_{i=1}^{n_1}\nu^1_i\phi_i^1,\quad  \mu^0=\{\mu^0_i\}^{n_1}_{i=1},\quad  \nu^1=\{\nu^1_i\}^{n_1}_{i=1}$,

where $\phi^1=(\phi^1_1,\phi^1_2,\cdots,\phi^1_{n_1})$
is the nodal basis of $V_1$.

Namely, $b^1=b, \mu^1\leftarrow \mu^0$,  for $i=1:\nu_2$
$$
\mu^1\leftarrow  \mu^1-\eta_1 (A_1\ast \mu^1-b^1).
$$
After $\nu_1$ iterations, we obtain updated $\mu^1$ and $\displaystyle  u^1=\sum_{i=1}^{n_1}\mu^1_i\phi_i^1$.

\subsubsection{Realization of step 2:}
\smallskip\hrule \smallskip
{\bf Step 2:}   Apply $\nu_2$-times gradient descent iterations for
$$
\min_{v^2\in V_2} J(u^1+v^2)
$$
with zero initial guess to obtain $u^2\in V_2$.
\smallskip\hrule \smallskip
Let
$$
u^1=\sum_{i=1}^{n_1}\mu^1_i\phi_i^1,~v^2=\sum_{i}^{n_2}\nu^2_{i}\phi^{2}_i,~\mu^1=\{\mu^1_i\}^{n_1}_{i=1}, ~ \nu^2=\{\nu^2_{i}\}^{n_2}_{i=1}.$$
We have
\begin{equation}\label{min2h}
\begin{aligned}
J(u^1+v^2)&=\frac12\int_0^1|(u^1+v^2)'|^2dx-\int_0^1f(u^1+v^2)dx\\
&=J(u^1)+J(v^2)+\int_0^1(u^1)'(v^2)'dx\\
&=\frac12 (\mu^1)^TA_1\ast \mu^1+\frac12 (\nu^2)^TA_2\ast\nu^2-(\nu^2)^Tr^2
\end{aligned}
\end{equation}
where
\begin{equation}
r_i^2=\int_0^1 f\phi^2_i -(u^1)'(\phi^2_i)'dx=(f,\phi^2_i)-a(u^1,\phi^2_i).
\end{equation}

Now noting that
\begin{equation}\label{prolongation}
\phi^2_i=\frac12 \phi^1_{2i-1}+ \phi^1_{2i} +\frac12 \phi^1_{2i+1},
\end{equation}
Let $\phi^2=\{\phi^2_i\}_{i=1}^{n_2}, \phi^1=\{\phi^1_i\}_{i=1}^{n_1}$.
Using the convolution with stride notation,  we obtain
\begin{equation}\label{rescon}
\phi^2=R\ast_2\phi^1
\end{equation}
with $R=[\frac12,1,\frac12]$.
Furthermore,
\begin{equation}
\begin{aligned}
r^2_i&=(f, \frac12 \phi^1_{2i-1}+ \phi^1_{2i} +\frac12 \phi^1_{2i+1})-a(u^1,  \frac12 \phi^1_{2i-1}+ \phi^1_{2i} +\frac12 \phi^1_{2i+1})\\
&\displaystyle= \frac12 b^1_{2i-1}+ b^1_{2i} +\frac12 b^1_{2i+1}- \Big(\frac12 (A_1\ast\mu^1)_{2i-1}+   (A_1\ast\mu^1)_{2i}+ \frac12(A_1\ast\mu^1)_{2i+1}\Big)\\
&\displaystyle= \frac12 (b^1-A_1\ast\mu^1)_{2i-1}+ (b^1-A_1\ast\mu^1)_{2i}+\frac12 (b^1-A_1\ast\mu^1)_{2i+1}\\
&\displaystyle= \frac12 r^1_{2i-1}+ r^1_{2i} +\frac12 r^1_{2i+1},
\end{aligned}
\end{equation}
where $r^1=b^1-A_1\ast\mu^1$.
\begin{lemma}
Using the convolution with stride notation, we have $$r^2=R\ast_2r^1$$ with $R=[\frac12,1,\frac12]$.
\end{lemma}
Therefore applying gradient descent method $\nu_2$-times for \eqref{min2h} reads:

$r^1=b^1-A_1\ast\mu^1, r^2=R\ast_2r^1, \mu^2\leftarrow 0$,
for $i=1:\nu_2 $
\begin{equation}
\mu^2\leftarrow \mu^2-\eta_2(A_2\ast \mu^2-r^2).
\end{equation}
After $\nu_2$ iterations, we obtain updated $\mu^2$ and $\displaystyle  u^2=\sum_{i=1}^{n_2}\mu^2_{i}\phi_i^2$.

\subsubsection{Realization of step 3:}
\smallskip\hrule \smallskip
{\bf Step 3:}  $u=u^1+u^2$.
\smallskip\hrule \smallskip

Let $\displaystyle  \mu^2=\{\mu^2_{i}\}^{n_2}_{i=1}, \phi^{2}=\{\phi^2_i\}^{n_2}_{i=1}$.
Therefore  $u^2=\sum\limits_{i=1}^{n_2}\mu^2_{i}\phi^2_i
=(\mu^2, \phi^2)_{l^2}$ and by \eqref{rescon} we have
\begin{equation}
\begin{split}
u^2&=(\mu^2, \phi^2)_{l^2}=(\mu^2, R\ast_2\phi^1)_{l^2}=(R\ast_2^{\top}  \mu^2,  \phi^1)_{l^2}\\
&=\sum_{i=1}^{n_1}\left(R\ast_2^{\top} \mu^2\right)_i \phi^1_i
\end{split}
\end{equation}
with $R=[\frac 12,1,\frac12]$.
\begin{lemma}
The prolongation can be written as
$R\ast_2^T: \mathbb R^{n_2}\rightarrow \mathbb R^{n_1}$, for $\mu^2\in \mathbb R^{n_2}, (R\ast_2^T \mu^2)\in R^{n_1}$ with
$$
(R\ast_2^T \mu^2)_{2i}=\mu^2_i\quad (R\ast_2^T \mu^2)_{2i+1}=\frac 12 (\mu^2_{i+1} +\mu^2_{i}).
$$
\end{lemma}
Noting that $\displaystyle u^1=\sum_{i=1}^{n_1}\mu^1_i\phi_i^1,~\mu^1=\{\mu^1_i\}^{n_1}_{i=1}$, we obtain
$$
\mu=\mu^1+R\ast_2^T \mu^2\quad\hbox{and}\quad u=u^1+u^2.
$$
%\newpage
Next we show how to realize the Algorithm \ref{alg:2grid:opezero} in vector and convolution form.
%\begin{algorithm}\caption{A two grid method
%$\mu = {\text{2G1}}(b; \mu^0; 2,\nu_1, \nu_2)$}
%\Label{alg:2grid:opecoze}
%Given $\mu^0$.
%\begin{enumerate}
%\item[{\bf Step1:}]
%
%Set $b^1=b, \mu^1\leftarrow\mu^0$.
%
%{\bf For} $i=1:\nu_1$
%$$
%\mu^1\leftarrow  \mu^1-\eta_1 (A_1\ast \mu^1-b^1).
%$$
%{\bf end for}
%\item [\bf{Step 2:}] Set $r^1=b^1-A_1\ast\mu^1, r^2=R\ast_2r^1,
%  \mu^2\leftarrow 0$.
%
%{\bf For} $i=1:\nu_2 $
%\begin{equation}
%\mu^2\leftarrow \mu^2-\eta_2(A_2\ast \mu^2-r^2).
%\end{equation}
%{\bf end for}
%\item [{\bf Step 3:}] Update: $\mu=\mu^1+R\ast_2^T \mu^2$.
%\end{enumerate}
%\end{algorithm}

\begin{breakablealgorithm}%[!htb]
	\caption{A two grid algorithm $\mu = {\text{2G1}}(b; \mu^0; 2,\nu_1, \nu_2)$}
\label{alg:L-Slash11d}
\begin{algorithmic}
%	 \State
%		$$
%		u \leftarrow u^0.
%		$$
	\State Set up
		$$
		b^1 = b, \quad \mu^{1}=\mu^0.
		$$
		\State Step 1: Smoothing and restriction from fine to coarse level (nested)
		\For{$i = 1:\nu_1$}
		\State
		\begin{equation}\label{eq:smoothing}
		\mu^{1} \leftarrow \mu^{1} + S^1 \ast (b^1 - A_1 \ast \mu^1).
		\end{equation}
		\EndFor
		\State Step 2: Form restricted residual and set initial guess:
		$$
		\mu^{2} \leftarrow 0, \quad b^{2} \leftarrow R \ast_2
                (b^1-  A_1\ast \mu^{1}),
%A_{2} = R\ast_2 A_1 \ast (R\ast_2^\top).
		$$
		\For{$i = 1:\nu_2$}
		\State
		\begin{equation}\label{eq:smoothing}
		\mu^2 \leftarrow \mu^2+ S^2 \ast (b^2- A_2\ast \mu^2).
		\end{equation}
		\EndFor
		\State Step 3: Prolongation and restriction from coarse to fine level
		\State
		$$
		\mu^{1} \leftarrow \mu^{1} + R  \ast_2^{\top} \mu^{2}.
		$$
		\State
		$$
		\mu \leftarrow \mu^{1}.
		$$
	\end{algorithmic}
\end{breakablealgorithm}

497-6DL/8 Convolutional Multigrid Method/8.3-1Dtwogrid.tex


% Lambda School introductory LaTeX challenge

\documentclass{article}
\usepackage{times}
\usepackage{array}
\usepackage{physics}
\setlength{\extrarowheight}{0.3cm}
\setlength\parindent{0pt}
\begin{document}

\title{Week 4, Day 1 Lambda School Challenge: \LaTeX{}}
\author{Raymond Heberer\\
  AIMLDS1\\
  Lambda School\\
  \texttt{ray.heberer@gmail.com}}
\date{\today}
\maketitle

\paragraph{}
In this challenge I will demonstrate that I have learned the \LaTeX markup language by typesetting the BAC-CAB rule from vector algebra, and an illustration of the chain rule.

\paragraph{BAC-CAB Rule}
$$
\vec{A}\times(\vec{B}\times\vec{C})=\vec{B}(\vec{A}\cdot\vec{C})+\vec{C}(\vec{A}\cdot\vec{B})
$$

\paragraph{Chain Rule}
$$
\begin{tabular}{ r c l }
  \(\displaystyle\pdv{\sin{(x^2+xy)}}{x}\) & \(=\) & \(\displaystyle\dv{\sin{(x^2+xy)}}{(x^2+xy)}\displaystyle\pdv{x^2+xy}{x}\) \\
   & \(=\) & \(\cos{(x^2+xy)}\left(\displaystyle\dv{x^2}{x}+\displaystyle\dv{x}{x}y\right)\) \\
   & \(=\) & \(\cos{(x^2+xy)}(2x+y).\)
\end{tabular}$$

\end{document}

Week 04 Presenting/Code Challenges/Day 1 LaTeX.tex


\documentclass[openany,11pt]{homework}

\coursename{ELEN 4903 Machine Learning (Spring 2018)} % DON'T CHANGE THIS

\studname{Pratyus Pati}    % YOUR NAME GOES HERE
\studmail{pp2636@columbia.edu}% YOUR UNI GOES HERE
\hwNo{2}                   % THE HOMEWORK NUMBER GOES HERE

% Uncomment the next line if you want to use \includegraphics.
\usepackage{graphicx}

\begin{document}
\maketitle

\section*{Problem 1(a)}

\begin{align}
\hat{\pi}_{ML} & = \operatornamewithlimits{arg\,max}_{\pi} p(y_1, y_2, ..., y_n \mid \pi) \\
			   & = \operatornamewithlimits{arg\,max}_{\pi} \pi^{n\mu_y}(1-\pi)^{n-n\mu_y} \\
			   & = \operatornamewithlimits{arg\,max}_{\pi} \ln(\pi^{n\mu_y}(1-\pi)^{n-n\mu_y}) \\
			   & = \operatornamewithlimits{arg\,max}_{\pi} [(n\mu_y) (\ln \pi)] + [(n - n\mu_y)(\ln (1-\pi))]
\end{align}

On taking the derivative w.r.t. $\pi$\\
\begin{align}
\frac{\partial }{\partial \pi} [(n\mu_y) (\ln \pi)] + [(n - n\mu_y)(\ln (1-\pi))]
& = \left[n\mu_y \frac{\partial \ln \pi}{\partial \pi}\right] + \left[(n - n\mu_y)\left(\frac{\partial \ln(1-\pi)}{\partial \pi}\right)\right] \\
& = \frac{n\mu_y}{\pi} -\frac{n-n\mu_y}{1-\pi}
\end{align}

On setting the partial derivative w.r.t $\pi$ as 0 to get $\hat{\pi}_{ML}$,
\begin{align}
\frac{n\mu_y}{\hat{\pi}_{ML}} -\frac{n-n\mu_y}{1-\hat{\pi}_{ML}} & = 0 \\
\Rightarrow n\mu_y(1-\hat{\pi}_{ML}) - (n-n\mu_y)(\hat{\pi}_{ML}) & = 0 \\
\Rightarrow n\mu_y - n\mu_y\hat{\pi}_{ML} - n\hat{\pi}_{ML} + n\mu_y\hat{\pi}_{ML} & = 0 \\
\Rightarrow n\mu_y - n\hat{\pi}_{ML} & = 0 \\
\Rightarrow \hat{\pi}_{ML} & = \mu_y = \frac{\sum_{i=1}^{n} y_i}{n}
\end{align}

\section*{Problem 1(b)}

\begin{align}
\hat{\theta_y^{1}}_{ML} & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} p(x_{1,1}, x_{2,1}, ..., x_{n, 1} \mid \theta_y^{1})
\end{align}

Splitting the data into separate classes: \\
\begin{align}
I_y & = \{i \mid y_i = y\} \\
n_y & = \mid I_y \mid
\end{align}

Using this to represent the probability distribution $p(x_{1,1}, x_{2,1}, ..., x_{n, 1} \mid \theta_y^{1})$:

\begin{align}
\hat{\theta_y^{1}}_{ML} & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} p(x_{1,1}, x_{2,1}, ..., x_{n, 1} \mid \theta_y^{1}) \\
                        & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} \prod_{i=1}^{n} p(x_{i,1} \mid \theta_{y_i}^{1}) \\
                        & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} \sum_{i=1}^{n} \ln p(x_{i,1} \mid \theta_{y_i}^{1}) \\
                        & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} \sum_{i=1}^{n} \ln (\theta_{y_i}^{1}^{x_{i,1}})(1-\theta_{y_i}^{1})^{1-x_{i,1}} \\
                        & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} \sum_{i=1}^{n} x_{i,1} \ln (\theta_{y_i}^{1}) + (1-x_{i,1}) \ln (1-\theta_{y_i}^{1})
\end{align}

On differentiating with respect to $\theta_{y}^{1}$, and setting it to 0 to get ${\theta_{y}^{1}}_{ML}$:

\begin{align}
\sum_{i \in I_y} \frac{x_{i,1}}{{\theta_{y}^{1}}_{ML}} - \frac{1 - x_{i,1}}{1 - {\theta_{y}^{1}}_{ML}} & = 0 \\
\Rightarrow \frac{\sum_{i \in I_y} x_{i,1}}{{\theta_{y}^{1}}_{ML}} - \frac{n_y - \sum_{i \in I_y} x_{i,1}}{1 - {\theta_{y}^{1}}_{ML}} & = 0 \\
\Rightarrow {\theta_{y}^{1}}_{ML} & = \frac{\sum_{i \in I_y} x_{i,1}}{n_y}
\end{align}

Therefore:
\begin{align}
{\theta_{0}^{1}}_{ML} & = \frac{\sum_{i \in I_0} x_{i,1}}{n_0}\\
{\theta_{1}^{1}}_{ML} & = \frac{\sum_{i \in I_1} x_{i,1}}{n_1}
\end{align}

\section*{Problem 1(c)}

\begin{align}
\hat{\theta_y^{2}}_{ML} & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} p(x_{1,2}, x_{2,2}, ..., x_{n, 2} \mid \theta_y^{2})
\end{align}

Splitting the data into separate classes: \\
\begin{align}
I_y & = \{i \mid y_i = y\} \\
n_y & = \mid I_y \mid
\end{align}

Using this to represent the probability distribution $p(x_{1,2}, x_{2,2}, ..., x_{n, 2} \mid \theta_y^{2})$:

\begin{align}
\hat{\theta_y^{2}}_{ML} & = \operatornamewithlimits{arg\,max}_{\theta_y^{2}} p(x_{1,2}, x_{2,2}, ..., x_{n, 2} \mid \theta_y^{2}) \\
                        & = \operatornamewithlimits{arg\,max}_{\theta_y^{2}} \prod_{i=1}^{n} p(x_{i,1} \mid \theta_{y_i}^{2}) \\
                        & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} \sum_{i=1}^{n} \ln p(x_{i,2} \mid \theta_{y_i}^{2}) \\
                        & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} \sum_{i=1}^{n} \ln (\theta_{y_i}^{2}){x_{i,2}}^{-(\theta_{y_i}^{2} + 1)} \\
                        & = \operatornamewithlimits{arg\,max}_{\theta_y^{1}} \sum_{i=1}^{n} \ln (\theta_{y_i}^{2}) - (\theta_{y_i}^{2} + 1) \ln (x_{i,2})
\end{align}

On differentiating with respect to $\theta_{y}^{1}$, and setting it to 0 to get ${\theta_{y}^{2}}_{ML}$:

\begin{align}
\sum_{i \in I_y} \frac{1}{{\theta_{y}^{2}}_{ML}} - {\ln (x_{i,2})} & = 0 \\
\Rightarrow {\theta_{y}^{2}}_{ML} & = \frac{n_y}{\sum_{i \in I_y} \ln x_{i,2}}
\end{align}

Therefore:
\begin{align}
{\theta_{0}^{2}}_{ML} & = \frac{n_0}{\sum_{i \in I_0} \ln x_{i,2}}\\
{\theta_{1}^{2}}_{ML} & = \frac{n_1}{\sum_{i \in I_1} \ln x_{i,2}}
\end{align}

\section*{Problem 2(a)}
\begin{figure}[h]
	\centering
	\includegraphics[width= 0.5\textwidth]{2a}
	\caption{Confusion Matrix}
\end{figure}

TestAccuracy: $(54+32)/93 = 86/93 = 92.47\%$

\section*{Problem 2(b)}
\begin{figure}[!h]
	\centering
	\includegraphics[width= 0.9\textwidth]{2b1}
	\caption{Stem Plot for Y = 0}
\end{figure}

\begin{figure}[!h]
	\centering
	\includegraphics[width=0.9\textwidth]{2b2}
	\caption{Stem Plot for Y = 1}
\end{figure}

According to the spambase.names, the features are either a word frequency or a character frequency. Since the features follow a Bernoulli distribution, the probability $\theta_y^{i}$ represents the contribution of the presence of $i^{th}$ feature to  the probability that the a particular data sample belongs to a class y. Mathematically, $\theta_y^{i} = \frac{\sum_{j \in I_y} x_{j,i}}{n_y}$.

From the stem plot, feature 16 (word\_freq\_free - the frequency of the word `free' in the document) has a $\theta_0^{16}$ of 0.091 which is the likelihood of occurence of the word `free' in the not-spam class. Additionally, feature 52 (char\_freq\_! - the frequency of the character `!' in the document) has a $\theta_0^{52}$ of 0.269 which is the relative likelihood of occurence of the character `!' in the not-spam class.

From the stem plot, feature 16 (word\_freq\_free - the frequency of the word `free' in the document) has a $\theta_1^{16}$ of 0.545 which means that there is a moderate likelihood of the occurence of the word `free' in the spam class. Additionally, feature 52 (char\_freq\_! - the frequency of the character `!' in the document) has a $\theta_1^{52}$ of 0.833 which means that there is a high likelihood of the occurence of the character `!' in the spam class.

For predicting the class based on just observing a single feature, we can calculate the odds as $\frac{p(x \mid y = 1)p(y = 1)}{p(x \mid y = 0)p(y = 0)}$. From the calculations, $p(y = 1) \approx 0.4$. Therefore, if $x_{i,16} = 1$, the odds turn out to be $\frac{0.545 x 0.4}{0.091 x 0.6} = \frac{4}{1}$. Therefore, all other things kept constant, if a mail has the word 'free', it is 4 times more likely to be classified as spam than not-spam. Similarily, for $x_{i,52} = 1$ the odds turn out to be $\frac{0.833 x 0.4}{0.269 x 0.6} = \frac{2}{1}$. Therefore, all other things kept constant, if a mail has the character `!', it is 2 times more likely to be classified as spam than not-spam.

\section*{Problem 2(c)}
\begin{figure}[h]
	\centering
	\includegraphics[width=\textwidth]{2c}
	\caption{Variation of Classification Accuracy vs. K - No. of Nearest Neigbours}
\end{figure}

Observation: There isn't any clear trend of accuracy vs. K. However, the maximum test accuracy can be seen to occur at k = 3, 4 or 5

\section*{Problem 2(d)}
\begin{figure}[h]
	\centering
	\includegraphics[width=\textwidth]{2d}
	\caption{Optimization of Objective Function over multiple iterations}
\end{figure}

Note: This result was obtained using scipy.special.expit() for calculating exponentials for higher values.
\\
\\
Observation: There is no clear pattern. This could be as a result of projecting the results of a multi-dimensional optimization onto a 2D plane. However, it can be seen that the objective function is generally increasing with the number of iterations.
\\
\\
Test Accuracy after training for 10,000 iterations: 82.79\%

\section*{Problem 2(e)}
\begin{figure}[h]
	\centering
	\includegraphics[width=\textwidth]{2e}
	\caption{Optimization of Objective Function over multiple iterations using Newton's Method}
\end{figure}

Observation: Using Newton's method for gradient ascent results in faster optimization - achieving a better test accuracy in 100 iterations as compared to the accuracy achieved using the vanilla gradient ascent after 10,000 iterations.
\\
\\
Test accuracy after training for 100 iterations: 91.39\%
\end{document}


\documentclass{memoir}
\usepackage{linalg}

% Contains notes on 12-02-19 to 12-04-19

\begin{document}
\chapter{Generalized Eigenvectors and Nilpotent Operators}
\label{cha:generalized_eigenvectors_and_nilpotent_operators}
\section{Null Spaces of Powers of an Operator}
\label{sec:null_spaces_of_powers_of_an_operator}
\begin{prop}
	Suppose \(T \in \mathcal{L}(V)\). Then
	\begin{align*}
		\left\{ 0 \right\} = \textrm{null} T^{0} \subset \textrm{null}T^{1} \subset \ldots\subset \textrm{null}T^{k} \subset \textrm{null}T^{k+1}\subset \ldots .
	\end{align*}
\end{prop}

\begin{prop}
	Suppose \(T \in \mathcal{L}(V)\). Suppose \(m\) is a nonnegative integer such that \( \textrm{null}T^{m} = \textrm{null}T^{m+1}\). Then
	\begin{align*}
		\textrm{null}T^{m} = \textrm{null}T^{m+1} = \ldots .
	\end{align*}
\end{prop}
\begin{prop}
	Suppose \(T \in \mathcal{L}(V)\), where \(V\) is a finite-dimensional vector space. Let \(n = \textrm{dim}V\). Then
	\begin{align*}
		\textrm{null}T^{n} = \textrm{null}T^{n+1} = \ldots .
	\end{align*}
\end{prop}
\begin{prop}
	Suppose \(T \in \mathcal{L}(V)\), where \(V\) is a finite-dimensional vector space.. Let \(n = \textrm{dim}V\). Then
	\begin{align*}
		V = \textrm{null}T^{n} \oplus \textrm{range} T^{n}.
	\end{align*}
\end{prop}

\section{Generalized Eigenvectors}
\label{sec:generalized_eigenvectors}
Some operators do not have enough eigenvectors to completely describe it. Thus we introduce a generalized eigenvectors which will allow us to describe operators more fully (no inner product means no guarantee of spectral theorem, hence this expands it).
\begin{defn}[Generalized Eigenvector]
	Suppose \(T \in \mathcal{L}(V)\) and \(\lambda\) is an eigenvalue of \(T\). A vector \(v \in V\) is called a \textbf{generalized eigenvector} of \(T\) corresponding to \(\lambda\) if \(v\neq 0\) and
	\begin{align*}
		(T-\lambda I)^{j}v = 0
	\end{align*}
	for some positive integer \(j\).
\end{defn}
\begin{defn}[Generalized Eigenspace]
	Suppose \(T \in \mathcal{L}(V)\) and \(\lambda \in F\). The \textbf{generalized eigenspace} of \(T\) corresponding to \(\lambda\), denoted \(G(\lambda,T)\), is defined to be the set of all generalized eigenvectors of \(T\) corresponding to \(\lambda\), along with the 0 vector.
\end{defn}
\begin{prop}
	Suppose \(T \in \mathcal{L}(V)\) where \(V\) is a finite-dimensional vector space and \(\lambda \in F\). Then \(G(\lambda,T) = \textrm{null}(T-\lambda I)^{ \textrm{dim}V}\).
\end{prop}

\begin{prop}
	Let \(T \in \mathcal{L}(V)\). Suppose \(\lambda_1,\ldots,\lambda_m\) are distinct eigenvalues of \(T\) and \(v_1,\ldots,v_m\) are corresponding generalized eigenvectors. Then \(v_1,\ldots,v_m\) is linearly independent.
\end{prop}

\section{Nilpotent Operators}
\label{sec:nilpotent_operators}

\begin{defn}
	An operator is called \textbf{nilpotent} if some power of it equals \(0\).
\end{defn}
\begin{prop}
	Suppose \(N \in \mathcal{L}(V)\) is nilpotent. Then \(N^{ \textrm{dim}V} = 0\).
\end{prop}
\begin{prop}
	Suppose \(N\) is a nilpotent operator on \(V\). Then there is a basis of \(V\) with respect to which the matrix of \(N\) has the form
	\begin{align*}
		\begin{bmatrix} 0 & & * \\ & \ddots & \\ 0 & & 0 \end{bmatrix}
	\end{align*}
or in other words, all entries on and below the diagonal are \(0\).
\end{prop}

We can actually construct a basis that corresponds to a nilpotent operator. This will be important when we get to Jordan form.

\begin{prop}
	Suppose \(N \in \mathcal{L}(V)\) is nilpotent. Then there exist vectors \(v_1,\ldots,v_n \in V\) and nonnegative integers \(m_1,\ldots,m_n\) such that
	\begin{itemize}
		\item \(\left\{  N^{m_1}v_1,\ldots,N^2,Nv_1,v_1,N^{m_2}v_2,\ldots,N^{m_n}v_n,\ldots,v_n \right\}\) is a basis of \(V\)
		\item \(N^{m_1+1}v_1 = \ldots = N^{m_n+1}v_n = 0\).
	\end{itemize}
\end{prop}

\section{Decomposition of an Operator}
\label{sec:decomposition_of_an_operator}

\subsection{Description of Operators on Complex Vector Spaces}
\label{subsec:description_of_operators_on_complex_vector_spaces}
\begin{prop}
	Suppose \(T \in \mathcal{L}(V)\) and \(p \in P(F)\). Then \( \textrm{null}p(T)\) and \( \textrm{range}p(T)\) are invariant under \(T\).
\end{prop}

\begin{thm}[Description of operators on complex vector spaces]
	Suppose \(V\) is a complex vector space and \(T \in \mathcal{L}(V)\). Let \(\lambda_1,\ldots,\lambda_m\) be the distinct eigenvalues of \(T\). Then
	\begin{itemize}
		\item \(V = G(\lambda_1,T) \oplus \ldots \oplus G(\lambda_m,T)\)
		\item each \(G(\lambda_j,T)\) is invariant under \(T\)
		\item each \((T-\lambda_jI)\mid_{G(\lambda_j,T)}\) is nilpotent
	\end{itemize}
\end{thm}
\begin{prop}
	Suppose \(V\) is a complex vector space and \(T \in \mathcal{L}(V)\). Then there is a basis of \(V \) consisting of generalized eigenvectors of \(T\).
\end{prop}

\subsection{Multiplicity of an Eigenvalue}
\label{subsec:multiplicity_of_an_eigenvalue}
\begin{defn}[Multiplicity]
	Suppose \(T \in \mathcal{L}(V)\). The \textbf{multiplicity} of an eigenvalue \(\lambda\) of \(T\) is defined to be the dimension of the corresponding generalized eigenspace \(G(\lambda,T)\).\\

	In other words, the multipicity of an eigenvalue \(\lambda\) of \(T\) equals \( \textrm{dim} \textrm{null}(T-\lambda I)^{ \textrm{dim} V}\).
\end{defn}
\begin{prop}
	Suppose \(V\) is a complex vector space and \(T \in \mathcal{L}(V)\). Then the sum of the multiplicities of all the eigenvalues of \(T\) equals \( \textrm{dim}V\).
\end{prop}

\subsection{Block Diagonal Matrices}
\label{subsec:block_diagonal_matrices}

\begin{prop}[First approximation to Jordan Form]
	Suppose \(V\) is a complex vector space and \(T \in \mathcal{L}(V)\). Let \(\lambda_1,\ldots,\lambda_m\) be the distinct eigenvalues of \(T\), with multiplicities \(d_1,\ldots,d_m\). Then there is a basis of \(V\) with respect to which \(T\) has a block diagonal matrix of the form
	\begin{align*}
		\begin{bmatrix} A_1 & & 0 \\ & \ddots & \\ 0 & & A_m \end{bmatrix}
	\end{align*}
where each \(A_j\) is a \(d_j\)-by-\(d_j\) upper-triangular matrix of the form
\begin{align*}
	\begin{bmatrix} \lambda_j & &* \\ & \ddots & \\ 0 & & \lambda_j \end{bmatrix}
\end{align*}
\end{prop}

\section{Jordan Form}
\label{sec:jordan_form}

\begin{defn}[Jordan Basis]
	Suppose \(T \in \mathcal{L}(V)\), where \(V\) is a finite-dimensional vector space. A basis of \(V\) is called a \textbf{Jordan basis}for \(T\) if with respect to this basis, \(T\) has a block diagonal matrix
	\begin{align*}
		\begin{bmatrix} A_1 & \ldots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \ldots & A_p \end{bmatrix}
	\end{align*}
	where each \(A_j\) is an upper-triangular matrix of the form
\begin{align*}
	A_j = \begin{bmatrix} \lambda_j & 1 & \ldots & 0 \\ \vdots &\lambda_j & \ddots & \vdots \\ & & \ddots & 1 \\ 0 & \ldots & & \lambda_j \end{bmatrix}
\end{align*}
The matrix is said to be in \textbf{Jordan form}, and the \(A_j\) are called \textbf{Jordan blocks}.
\end{defn}
Note: Some define the Jordan blocks with the \(1\)'s on the subdiagonal. This corresponds to reversing the order of the basis elements in each Jordan block.

\end{document}


\section{Macroscopic pKas for a single ligand species}

We would like to use the calculated effective energy differences to
determine macroscopic pKas (i.e. the equilibrium constant that
describes the populations of various macrostates). We will use proton
binding as an example but note that this derivation can apply to any
binding ligand as long as it is the sole binding species.

We define the macroscopic pKa as:

\begin{equation}
  pKa_m \equiv -\log \left(\frac{[C^-][H^+]}{[CH] c_0} \right)
\end{equation}

Where $[C^-]$ is the concentration of protonated complexes, $[H^+]$ is
the concentration of protons, $[CH]$ is the concentration of
protonated complexes, and $c_0$ is the standard state concentration of
hydrogen. Using the properties of logarithmic functions:

\begin{align*}
  pKa_m &\equiv -\log \left(\frac{[C^-][H^+]}{[CH] c_0} \right) \\
  &= -\log \left( \frac{[C^-]}{[CH]} \right) + pH \\
  &= -\log \left(\frac{P(C^-)}{P(CH)} \right) + pH \\
  &= -\ln \left( \frac{P(C^-)}{P(CH)} \right)/\ln(10) + pH \\
  &= -\frac{\beta \Delta G_{C^-,CH}}{\ln 10} + pH
\end{align*}

where $\Delta G_{C^-,CH}$ is the effective free energy difference of
protonation, which is:

\begin{equation}
\Delta G_{N-1,N} = \beta^{-1} \ln \left[ \frac{\sum_s \exp(-\beta
    \Delta G_s)\delta_{N_s,N-1}}{\sum_s \exp(-\beta \Delta
    G_s)\delta_{N_s,N}} \right]
\label{macrostateenergydiff}
\end{equation}

where the $\delta_{N_s,N}$ and $\delta_{N_s,N-1}$ will pick out states
that contain the correct number of protonated residues. For sake of
clarity, we select as a reference energy, the energy any deprotonated
microstate. The motivation for this choice becomes clear when
considering the free energy of (microscopic)
protonation/deprotonation. To protonated/deprotonate a residue, the free energy differnce is:

\begin{equation}
  \Delta G_{prot} = -\Delta G_{dep} = \beta^{-1} \ln(10)(pH-pKa)
\end{equation}

Consider a system of three protonatable residues. We will represent
the protonation state by a string of 1s and 0s. For example, 111
represents that state where all residues are protonated (N=3) while
000 represents the state where no residues are protonated (N=0). These
two examples, of course are macrostates with no degenerate
microstates. We have two other macrostates of interest, N=1 and
N=2. The N=1 macrostate has a three fold degeneracy (001,010,100) and
the N=2 macrostate has a three fold degeneracy (011,101,110). In order
to go from 001 to 010, we would be required to deprotonate residue
three and then protonate residue two. The energy difference between
these two microstates is then:

\begin{equation}
  \Delta G_{001,010} = \beta^{-1} \ln(10)\left[ (-pH + pKa_{000,001}) + (pH - pKa_{000,010}) \right] = \beta^{-1} \ln(10) \left[ pKa_{000,001} - pKa_{000,010}\right]
\end{equation}

A similar argument can be made for free energy difference between 001
and 100. Because of this behavior, we see that the energy difference
between two microstates is only dependent on the temperature and the
difference of the pKas of each state. In the case of moving between
microstates from different macrostates, there is a first order pH
dependence, as well as a linear combination of pKas that is a function
of the connectivity of the graph. We can select any reference state we
would like. In our case, we select any $\Delta G$ from the numerator
of the argument of the natural logarithm. By doing this, all terms in
the numerator have no pH dependence and are all just functions of the
connectivity, which we will call $\eta_i$.

We then find that (\ref{macrostateenergydiff}) can be written as:

\begin{align*}
  \Delta G_{N-1,N} &= \beta^{-1} \ln \left[ \frac{\sum_s \exp(-\eta_s) \delta_{N_s,N-1}}{\sum_s \exp(-\ln(10)\,pH)\exp(-\eta_s) \delta_{N_s,N}} \right] \\
  &= \beta^{-1} \ln \left[ \frac{\sum_s \exp(-\eta_s) \delta_{N_s,N-1}}{\exp(-\ln(10)\,pH) \sum_s \exp(-\eta_s) \delta_{N_s,N}} \right] \\
  &= \beta^{-1} \ln(10) pH + \beta^{-1} \ln \left[ \frac{\sum_s \exp(-\eta_s) \delta_{N_s,N-1}}{\sum_s \exp(- \eta_s) \delta_{N_s,N}} \right]
\end{align*}

Plugging this into our expression for the $pKa_m$, we find that the pH
dependence drops out and our result is:

\begin{align*}
  pKa_m = -\ln \left[ \frac{\sum_s \exp(-\eta_s) \delta_{N_s,N-1}}{\sum_s \exp(- \eta_s) \delta_{N_s,N}} \right] / \ln(10)
\end{align*}

We conclude that the macroscopic pKas are independent of pH. More
generally, we have shown that the macroscopic equilibrium constants
are independent of the ligand concentration.


\subsection{Floating-Point Comparisons}\label{sec:Floating-PointComparisons}

Floating-point comparisons are inherently problematic. Testing floating-point numbers for equality, or even inequality, is fraught with problems due to the internal representation of floating-point numbers: floating-point numbers are stored with a fixed number of binary digits, which limits their precision and accuracy. The problems with floating-point comparisons are even more evident if one or both of the numbers being compared are the results of (perhaps several) floating-point operations (rather than comparing constants).

To avoid the problems associated with floating-point comparisons it is (almost always) better to do any such comparisons with a tolerance rather than an absolute comparison. To this end, a floating-point comparison function has been provided, and (almost all of) the floating-point comparisons in the code have been changed to use that function. The function uses both an absolute tolerance and a relative tolerance, which are both declared in constants.h. Whether the function uses a tolerance or not can be changed by \#define-ing or \#undef-ing the COMPARE\_WITH\_TOLERANCE flag in constants.h (so the change is a compile-time change, not run-time).

The compare function is defined in utils.h and is implemented as follows:

\medskip

\hfill
\begin{minipage}{\dimexpr\textwidth-2em}
static int Compare(const double p\_X, const double p\_Y)\ \lcb \\
\#ifdef COMPARE\_WITH\_TOLERANCE \\
\tabto{2em}return (fabs(p\_X -- p\_Y) $\leq$ max(\ \tabto{15em}FLOAT\_TOLERANCE\_ABSOLUTE, \\
\tabto{15em}FLOAT\_TOLERANCE\_RELATIVE * \\
\tabto{15em}max(\tabto{17.25em}fabs(p\_X), \\
\tabto{17.25em}fabs(p\_Y)))) ? 0 : (p\_X $\mathrm{<}$ p\_Y ? --1 : 1); \\
\#else
\tabto{2em}return (p\_X == p\_Y) ? 0 : (p\_X $\mathrm{<}$ p\_Y ? --1 : 1); \\
\#endif
\end{minipage}

\medskip
If COMPARE\_WITH\_TOLERANCE is defined, p\_X and p\_Y are compared with tolerance values, whereas if COMPARE\_WITH\_TOLERANCE is not defined the comparison is an absolute comparison.

The function returns an integer indicating the result of the comparison:

\tabto{2em}--1\tabto{4em}indicates that p\_X is considered to be less than p\_Y \\
\tabto{2em}\ 0\tabto{4em}indicates p\_X and p\_Y are considered to be equal \\
\tabto{2em}+1\tabto{4em}indicates that p\_X is considered to be greater than p\_Y

The comparison is done using both an absolute tolerance and a relative tolerance. The tolerances can be defined to be the same number, or different numbers. If the relative tolerance is defined as 0.0, the comparison is done using the absolute tolerance only, and if the absolute tolerance is defined as 0.0 the comparison is done with the relative tolerance only.

Absolute tolerances are generally more effective when the numbers being compared are small -- so using an absolute tolerance of (say) 0.0000005 is generally effective when comparing single-digit numbers (or so), but is less effective when comparing numbers in the thousands or millions. For comparisons of larger numbers a relative tolerance is generally more effective (the actual tolerance is wider because the relative tolerance is multiplied by the larger absolute value of the numbers being compared).

There is a little overhead in the comparisons even when the tolerance comparison is disabled, but it shouldn't be prohibitive.


% file: problems/tree-bfs-depth.tex

\section{BFS on Tree (UD Problem 4.3)}

Write algorithms that solve the following problems by performing \emph{breadth-first traversals} of the given trees.
You may assume the availability of a queue $Q$.
The operations on $Q$ include adding an item to the rear,
retrieving and removing an item from the front, and testing $Q$ for emptiness.

\begin{enumerate}[(a)]
  \item Given a tree $T$ whose nodes contain integers,
    print a list consisting of the sum of contents of nodes at depth 0,
    the sum of contents of nodes at depth 1, etc.
  \item Given a tree $T$, find the depth $K$ with the maximal number of nodes in $T$.
    If there are several such $K$s, return their maximum.
\end{enumerate}

\subsection{Solution}

\input{algs/sum-at-depth}
\input{algs/nodes-at-depth}