\lab{Gibbs Sampling and LDA}{Gibbs Sampling and LDA}
\objective{Understand the basic principles of implementing a Gibbs sampler. Apply this to Latent Dirichlet Allocation.}

\section*{Gibbs Sampling}
Gibbs sampling is an MCMC sampling method in which we construct a Markov chain by which to sample from a desired joint (conditional) distribution
\begin{equation*}
\mathbb{P}(x_{1},\cdots,x_{n} | \mathbf{y}).
\end{equation*}
Often it is difficult to sample from this high-dimensional joint distribution, while it may be easy to sample from the one dimensional
conditional distributions
\begin{equation*}
\mathbb{P}(x_{i} | \mathbf{x}_{-i}, \mathbf{y})
\end{equation*}
where $\mathbf{x}_{-i} = x_{1},\cdots,x_{i-1},x_{i+1},\cdots,x_{n}.$

\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Gibbs Sampler}{}
    \State \textrm{Randomly initialize } $x_1,x_2,\ldots,x_n$.
    \For{$k = 1, 2, 3, \ldots$}
        \For{$i = 1, 2, \ldots,n$}
            \State \textrm{Draw } $x \sim \mathbb{P}(x_{i} | \mathbf{x}_{-i}, \mathbf{y})$
            \State \textrm{Fix } $x_i = x$
        \EndFor
        \State $\mathbf{x}^{(k)}= (x_1,x_2,\ldots,x_n)$
    \EndFor
\EndProcedure
\end{algorithmic}
\caption{Basic Gibbs Sampling Process.}
\label{alg:gibbs}
\end{algorithm}
A Gibbs sampler proceeds according to Algorithm \ref{alg:gibbs}.
Each iteration of the outer for-loop is a \emph{sweep} of the Gibbs sampler, and the value of $\mathbf{x}^{(k)}$ after a sweep is a sample. This creates an irreducible, non-null recurrent, aperiodic Markov chain over the state space consisting of all possible $\mathbf{x}$. The unique invariant distribution for the chain is the desired joint distribution
\begin{equation*}
\mathbb{P}(x_{1},\cdots,x_{n} | \mathbf{y}).
\end{equation*}
Thus, after a burn-in period, our samples $\mathbf{x}^{(k)}$ are effectively samples from the desired distribution.

Consider the data set of $N$ scores from a calculus exam in the file \texttt{examscores.csv}. We believe that the spread of these exam scores can be
modeled with a normal distribution of mean $\mu$ and variance $\sigma^{2}$.
Because we are unsure of the true value of $\mu$ and $\sigma^2$, we take a Bayesian approach and place priors on each parameter to quantify this uncertainty:
\begin{align*}
\mu & \sim N(\mu_{0}, \sigma_{0}^{2})\quad &&\text{(a normal distribution)} \\
\sigma^{2} & \sim IG(\alpha, \beta) &&\text{(an inverse gamma distribution)}
\end{align*}
Letting $\mathbf{y} = (y_1,\ldots,y_N)$ be the set of exam scores, we would like to update our beliefs of $\mu$ and $\sigma^2$ by sampling from the posterior
distribution
\begin{equation*}
\mathbb{P}(\mu, \sigma^{2} | \mathbf{y}, \mu_{0}, \sigma_{0}^{2}, \alpha, \beta).
\end{equation*}
Sampling directly can be difficult. However, we \emph{can} easily sample from the following conditional distributions:
\begin{align*}
\mathbb{P}(\mu | \sigma^{2}, \mathbf{y}, \mu_{0}, \sigma_{0}^{2}, \alpha, \beta) & = \mathbb{P}(\mu | \sigma^{2}, \mathbf{y}, \mu_{0}, \sigma_{0}^{2})\\
\mathbb{P}(\sigma^{2} | \mu, \mathbf{y}, \mu_{0}, \sigma_{0}^{2}, \alpha, \beta) & = \mathbb{P}(\sigma^{2} | \mu, \mathbf{y}, \alpha, \beta)
\end{align*}
The reason for this is that these conditional distributions are \emph{conjugate} to the prior distributions, and hence are part of the same distributional
families as the priors. In particular, we have
\begin{align*}
\mathbb{P}(\mu | \sigma^{2}, \mathbf{y}, \mu_{0}, \sigma_{0}^{2}) &= N(\mu^*, (\sigma^*)^2)\\
\mathbb{P}(\sigma^{2} | \mu, \mathbf{y}, \alpha, \beta) &= IG(\alpha^*, \beta^*),
\end{align*}
where
\begin{align*}
(\sigma^*)^2 &= \left(\frac{1}{\sigma_0^2}+\frac{N}{\sigma^2}\right)^{-1}\\
\mu^* &= (\sigma^*)^2\left(\frac{\mu_0}{\sigma_0^2} + \frac{1}{\sigma^2}\sum_{i=1}^N y_i \right)\\
\alpha^* &= \alpha + \frac{N}{2}\\
\beta^* &= \beta + \frac{1}{2}\sum_{i=1}^N (y_i-\mu)^2
\end{align*}
We have thus set this up as a Gibbs sampling problem, where we just have to alternate between sampling $\mu$ and $\sigma^{2}$.
We can sample from a normal distribution and an inverse gamma distribution as follows:
\begin{lstlisting}
>>> from math import sqrt
>>> from scipy.stats import norm
>>> from scipy.stats import invgamma
>>> mu = 0. # the mean
>>> sigma2 = 9. # the variance
>>> normal_sample = norm.rvs(mu, scale=sqrt(sigma))
>>> alpha = 2.
>>> beta = 15.
>>> invgamma_sample = invgamma.rvs(alpha, scale=beta)
\end{lstlisting}
Note that when sampling from the normal distribution, we need to set the \li{scale} parameter to the standard deviation, \emph{not} the variance.

\begin{problem}
Implement a Gibbs sampler for the exam scores problem using the following function declaration.
\begin{lstlisting}
def gibbs(y, mu0, sigma02, alpha, beta, n_samples):
    """
    Assuming a likelihood and priors
        y_i    ~ N(mu, sigma2),
        mu     ~ N(mu0, sigma02),
        sigma2 ~ IG(alpha, beta),
    sample from the posterior distribution
        P(mu, sigma2 | y, mu0, sigma02, alpha, beta)
    using a gibbs sampler.

    Parameters
    ----------
    y : ndarray of shape (N,)
        The data
    mu0 : float
        The prior mean parameter for mu
    sigma02 : float > 0
        The prior variance parameter for mu
    alpha : float > 0
        The prior alpha parameter for sigma2
    beta : float > 0
        The prior beta parameter for sigma2
    n_samples : int
        The number of samples to draw

    Returns
    -------
    samples : ndarray of shape (n_samples,2)
        1st col = mu samples, 2nd col = sigma2 samples
    """
    pass
\end{lstlisting}
Test it with priors $\mu_{0}=80, \sigma_{0}^{2} = 16, \alpha = 3, \beta = 50$, collecting $1000$ samples. Plot your samples of $\mu$ and your samples of $\sigma^{2}$. How long did it take for each to converge? It should have been very quick.
\end{problem}

We'd like to look at the posterior marginal distributions for $\mu$ and $\sigma^2$.
To plot these from the samples, we will use a kernel density estimator.
If our samples of $\mu$ are called \li{mu_samples}, then we can do this as follows:
\begin{lstlisting}
>>> import numpy as np
>>> from scipy.stats import gaussian_kde
>>> import matplotlib.pyplot as plt
>>> mu_kernel = gaussian_kde(mu_samples)
>>> x_min = min(mu_samples) - 1
>>> x_max = max(mu_samples) + 1
>>> x = np.arange(x_min, x_max, step=0.1)
>>> plt.plot(x,mu_kernel(x))
>>> plt.show()
\end{lstlisting}

\begin{figure}
	\begin{subfigure}[b]{.49\textwidth}
		\includegraphics[width=\textwidth]{mu_posterior.pdf}
		\caption{Posterior distribution of $\mu$.}
	\end{subfigure}
	\begin{subfigure}[b]{.49\textwidth}
		\includegraphics[width=\textwidth]{sigma2_posterior.pdf}
		\caption{Posterior distribution of $\sigma^2$.}
	\end{subfigure}
\caption{Posterior marginal probability densities for $\mu$ and $\sigma^2$.}
\label{fig:post}
\end{figure}

\begin{problem}
Plot the kernel density estimators for the posterior distributions of $\mu$ and $\sigma^{2}$.
You should get plots similar to those in Figure \ref{fig:post}.
\end{problem}

Keep in mind that the above plots are of the posterior distributions of the \emph{parameters}, not of the scores. If we would like to compute the posterior distribution of a new exam score $\tilde{y}$ given our data $\mathbf{y}$ and prior parameters, we compute what is known as the \emph{posterior predictive distribution}:
\begin{equation*}
\mathbb{P}(\tilde{y} | \mathbf{y}, \lambda) = \int_{\Theta} \mathbb{P}(\tilde{y} | \Theta)\mathbb{P}(\Theta | \mathbf{y}, \lambda) d\Theta
\end{equation*}
where $\Theta$ denotes our parameters (in our case $\mu$ and $\sigma^{2}$) and $\lambda$ denotes our prior parameters (in our case $\mu_{0}, \sigma_{0}^{2}, \alpha,$ and $\beta$).

Rather than actually computing this integral for each possible $\tilde{y}$, we can do this by sampling scores from our parameter samples. In other words, sample
\begin{equation*}
\tilde{y}_{(t)} \sim N(\mu_{(t)}, \sigma_{(t)}^{2})
\end{equation*}
for each sample pair $\mu_{(t)}, \sigma_{(t)}^{2}$. Now we have essentially drawn samples from our posterior predictive distribution, and we can use a kernel density estimator to plot this distribution from the samples.

\begin{figure}
\includegraphics[width=\textwidth]{predictiveposterior.pdf}
\caption{Predictive posterior distribution of exam scores.}
\label{fig:predictive}
\end{figure}

\begin{problem}
Use your samples to draw samples from the posterior predictive distribution. Plot the kernel density estimator of your sampled scores.
It should resemble the plot in Figure \ref{fig:predictive}.
\end{problem}

\section*{Latent Dirichlet Allocation}
Gibbs sampling can be applied to an interesting problem in language processing: that of determining which topics are prevalent in a document.
Latent Dirichlet Allocation (LDA) is a generative model for a collection of text documents.
It supposes that there is some fixed vocabulary of terms (of length $V$) and $K$ different topics, each represented as a probability distribution $\phi_{k}$ over the vocabulary, each with a Dirichlet prior $\beta$.
With the vocabulary and topics chosen, the LDA model assumes that a set of $D$ documents is generated as follows.
The $m$-th document consists of $N_m$ words, and
a probability distribution $\theta_{m}$ over the topics is drawn from a Dirichlet distribution with parameter $\alpha$.
To generate each new word in the $m$-th document, we first draw a topic assignment $z_{m,n}$ from the categorical distribution $\theta_{m}$, and then we draw a word $w_{m,n}$ from the categorical distribution $\phi_{z_{m,n}}$. Throughout this implementation, we assume $\alpha$ and $\beta$ are scalars. In summary, we have
\begin{enumerate}
	\item Draw $\phi_{k} \sim \text{Dir}(\beta)$ for $1 \leq k \leq K$.
	\item For $1 \leq m \leq D$:
	\begin{enumerate}
        \item Draw $\theta_{m} \sim \text{Dir}(\alpha)$.
        \item Draw $z_{m,n} \sim \text{Cat}(\theta_{m})$ for $1 \leq n \leq N_{m}$.
	    \item Draw $w_{m,n} \sim \text{Cat}(\phi_{z_{m,n}})$ for $1 \leq n \leq N_{m}$.
	\end{enumerate}
\end{enumerate}
This is typically depicted with graphical plate notation as in Figure \ref{fig:ldaplates}.
\begin{figure}[h]
\centering
\begin{tikzpicture}[>=stealth', dot/.style=
	{circle,fill=black,minimum size=3pt,inner sep=0pt, outer sep=-1pt} ]

\node[draw,minimum height=3.2cm, minimum width=2.1cm](r1)[]{};
\node[draw,minimum height=5cm, minimum width=2.4cm, node distance=
	.4cm](r2)[above of=r1]{};

\node[node distance=1.3cm](Nm)[below of=r1]{$1 \le n \le N_m$};
\node[node distance=2.28cm](M)[below of=r2]{$1 \le m \le M$};

\node[node distance = 2.7cm](dummy3)[left of=r1]{};
\node[draw,minimum height=2cm, minimum width=2.1cm, node distance=
	.55cm](r3)[ below of =dummy3]{};
\node[node distance=.7cm](K)[below of=r3]{$1 \le k \le K$};

\node[node distance=.6cm](dummy)[above right of =Nm]{};
\node[circle, draw,  inner sep=1pt, fill=black!25!,node distance=.4cm]
	(w)[above of=dummy]{$w_{m,n}$};
\node[circle, draw,  inner sep=1pt, node distance=1.3cm](z)[above
	of=w]{$z_{m,n}$};
\node[circle, draw,  inner sep=1pt, node distance=1.3cm]
	(theta)[above of=z]{$\vec{\theta}_{m}$};

\node[node distance=1.2cm, inner sep=0pt](alpha)[above of=
	theta]{$\vec{\alpha}$};

\node[node distance=.6cm](dummy2)[above right of=K]{};
\node[node distance=.35cm, circle, inner sep=1pt, draw](phi)[above
	of=dummy2]{$\vec{\phi_k}$};
\node[node distance=1.4cm, inner sep=0pt](beta)[above of = phi]{$\vec{\beta}$};

\foreach \x/\y in {alpha/theta, theta/z, z/w, beta/phi, phi/w} \draw[->](\x)--(\y);

\end{tikzpicture}
\caption{Graphical plate notation for LDA text generation.}
\label{fig:ldaplates}
\end{figure}

In the plate model, only the variables $w_{m,n}$ are shaded, signifying that these are the only observations visible to us; the rest are latent variables. Our goal is to estimate each $\phi_{k}$ and each $\theta_{m}$. This will allow us to understand what each topic is, as well as understand how each document is distributed over the $K$ topics. We can estimate these well if we know $z_{m,n}$ for each $m, n$, collectively referred to as $\mathbf{z}$. Thus, we need to sample
$\mathbf{Z}$ from the posterior distribution $\mathbb{P}(\mathbf{z} | \mathbf{w}, \alpha, \beta),$ where $\mathbf{w}$ is the collection words in the text corpus. Unsurprisingly, it is intractable to sample directly from the joint posterior distribution. However, letting $\mathbf{z}_{\neg (m,n)} = \mathbf{z}\setminus \{z_{m,n}\}$, the conditional posterior distributions
\[\mathbb{P}(z_{m,n} = k | \mathbf{z}_{\neg (m,n)}, \mathbf{w}, \alpha, \beta)\]
have nice, closed form solutions, making them easy to sample from.

These conditional distributions have the following form:
\begin{equation*}
\mathbb{P}(z_{m,n} = k | \mathbf{z}_{\neg (m,n)}, \mathbf{w}, \alpha, \beta) \propto \frac{(n_{(k,m,\cdot)}^{\neg (m,n)} + \alpha)(n_{(k, \cdot, w_{m,n})}^{\neg (m,n)} + \beta)}{n_{(k,\cdot,\cdot)}^{\neg (m,n)} + V \beta}
\end{equation*}
where
\begin{align*}
n_{(k,m,\cdot)} & = \mbox{ the number of words in document $m$ assigned to topic $k$} \\
n_{(k,\cdot,v)} & = \mbox{ the number of times term $v = w_{m,n}$ is assigned to topic $k$} \\
n_{(k,\cdot,\cdot)} & = \mbox{ the number of times topic $k$ is assigned in the corpus} \\
n_{(k,m,\cdot)}^{\neg (m,n)} & = n_{(k,m,\cdot)} - \indicator{z_{m,n} = k} \\
n_{(k,\cdot,v)}^{\neg (m,n)} & = n_{(k,\cdot,v)} - \indicator{z_{m,n} = k} \\
n_{(k,\cdot,\cdot)}^{\neg (m,n)} & = n_{(k,\cdot,\cdot)} - \indicator{z_{m,n} = k}
\end{align*}

Thus, if we simply keep track of these count matrices, then we can easily create a Gibbs sampler over the topic assignments. This is actually a particular class of samplers known as \emph{collapsed Gibbs samplers}, because we have collapsed the sampler by integrating out $\theta$ and $\phi$.

We have provided for you the structure of a Python object \li{LDACGS} with several methods. The object is already defined to have attributes \li{n\_topics}, \li{documents}, \li{vocab}, \li{alpha}, and \li{beta}, where \li{vocab} is a list of strings (terms), and documents is a list of dictionaries (a dictionary for each document). Each entry in dictionary $m$ is of the form $n : w$, where $w$ is the index in \li{vocab} of the $n^{th}$ word in document $m$.

Throughout this lab we will guide you through writing several more methods in order to implement the Gibbs sampler. The first step is to initialize our assignments, and create the count matrices $n_{(k,m,\cdot)}, n_{(k,\cdot,v)}$ and vector $n_{(k,\cdot,\cdot)}$.

\begin{problem}
Complete the method \li{initialize}. By randomly assigning initial topics, fill in the count matrices and topic assignment dictionary. In this method, you will initialize the count matrices (among other things). Note that the notation
provided in the code is slightly different than that used above. Be sure to understand how the formulae above
connect with the code.
\end{problem}

The next method we need to write fully outlines a sweep of the Gibbs sampler.

\begin{problem}
Complete the method \li{\_sweep}, which needs to iterate through each word of each document. It should call on the method \li{\_conditional} to get the conditional distribution at each iteration.
\end{problem}

\begin{comment}
Take out this problem to make the lab easier.
We need to write the method to create the appropriate conditional distribution.

\begin{problem}
Complete the method \li{\_conditional}. It accepts arguments $m,w$ where $m$ is the document and $w$ is an index of \li{vocab}. Don't forget to normalize to ensure you are actually returning a distribution!
\end{problem}
\end{comment}
We are now prepared to write the full Gibbs sampler.

\begin{problem}
Complete the method \li{sample}. The argument \emph{filename} is the name and location of a .txt file, where each line is considered a document. The corpus is built by method \li{buildCorpus}, and stopwords are removed (if argument \emph{stopwords} is provided). Burn in the Gibbs sampler, computing and saving the log-likelihood with the method \li{\_loglikelihood}. After the burn in, iterate further, accumulating your count matrices, by adding \li{nzw} and \li{nmz} to \li{total\_nzw} and \li{total\_nmz} respectively, where you only add every \emph{sample\_rate}$^{th}$ iteration. Also save each log-likelihood.
\end{problem}

You should now have a working Gibbs sampler to perform LDA inference on a corpus. Let's test it out on Ronald Reagan's State of the Union addresses.

\begin{problem}
Create an \li{LDACGS} object with $20$ topics, letting \li{alpha} and \li{beta} be the default values. Load in the stop word list provided. Run the Gibbs sampler, with a burn in of $100$ iterations, accumulating $10$ samples, only keeping the results of every $10$ sweep. Plot the log-likelihoods. How long did it take to truly burn in?
\end{problem}

We can estimate the values of each $\phi_{k}$ and each $\theta_{m}$ as follows:

\begin{align*}
\widehat{\theta}_{m,k} & = \frac{n_{(k,m,\cdot)} + \alpha}{K \cdot \alpha + \sum_{k=1}^{K} n_{(k,m,\cdot)}} \\
\widehat{\phi}_{k,v} & = \frac{n_{(k,\cdot,v)} + \beta}{V \cdot \beta + \sum_{v=1}^{V} n_{(k,\cdot,v)}}
\end{align*}

We have provided methods \li{phi} and \li{theta} that do this for you. We often examine the topic-term distributions $\phi_{k}$ by looking at the $n$ terms with highest probability, where $n$ is small (say $10$ or $20$).  We have provided a method \li{topterms} which does this for you.

\begin{problem}
Using the methods described above, examine the topics for Reagan's addresses. As best as you can, come up with labels for each topic.
\end{problem}

We can use $\widehat{\theta}$ to find the paragraphs in Reagan's addresses that focus the most on each topic. The documents with highest values of $\widehat{\theta}_{k}$ are those most heavily focused on topic $k$.

\begin{problem}
In your above topic analysis, you should have found a topic about the Cold War and one about education. Find the five paragraphs in Reagan's addresses that most closely focus on each of these topics, according to the above method.
\end{problem}


\section{Dual Space and Dual Maps}
\begin{definition}
    Let $V$ be a vector space over $F$, we define the dual space $V^\ast$ of $V$ to be the set of linear maps from $V$ to $F$, i.e. $V^\ast=L(V,F)$.\\
    We call linear maps $V\to F$ as linear forms.
\end{definition}
\begin{example}
    1. The map $\operatorname{tr}:M_{n,n}(F)\to F$ via $A=(a_{ij})\mapsto\sum_ia_{ii}$ is a linear form, so $\operatorname{tr}\in (M_{n,n}(F))^\ast$.\\
    2. For a function $f:\in C^{\infty}([0,1],\mathbb R)$, we can define the map $T_f:C^{\infty}([0,1],\mathbb R)$ via
    $$T_f(\phi)=\int_0^1f(x)\phi(x)\,\mathrm dx$$
    An interesting thing is, if we are given all information about $T_f$, can we recover $f$?
    The answer is yes and is left as an exercise.
    This idea comes from quantum mechanics, where you only get the information about $T_f$ by physical measurements but you want information about $f$.
\end{example}
There is a natural way of finding a basis for the dual space.
\begin{lemma}
    Let $V$ be a vector space over $F$ with a finite basis $B=\{e_1,\ldots,e_n\}$, then there exists a basis for $V^\ast$ given by
    $B^\ast=\{\epsilon_1,\ldots,\epsilon_n\}$ where $\epsilon_j(\sum_ia_ie_i)=a_j$ for $1\le j\le n$.
\end{lemma}
\begin{definition}
    The basis $B^\ast$ in the preceding lemma is called the dual basis.
\end{definition}
\begin{remark}
    If we introduce the Kronecker delta
    $$\delta_{ij}=\begin{cases}
        1\text{, if $i=j$}\\
        0\text{, otherwise}
    \end{cases}$$
    then we can define $\epsilon_j$ by extending $\epsilon_j(e_i)=\delta_{ij}$.
\end{remark}
\begin{proof}
    If there is some $\lambda_j\in F$ such that $\sum_j\lambda_j\epsilon_j=0$, then in particular the evaluation of the left hand side at $e_i$ is zero for each $i$.
    But $\epsilon_j(e_i)=0$, so $\lambda_i=0$ for each $i$.\\
    To see this is spanning, one just observe that $\alpha=\sum_j\alpha(e_j)\epsilon_j$ for any linear form $\alpha$.
\end{proof}
\begin{corollary}
    If $V$ is finite dimensional, then $\dim V^\ast=\dim V$.
\end{corollary}
\begin{remark}
    It is convenient to think of $V^\ast$ as the space of row vectors of length $n$ over $F$.
    Indeed, if we have a basis $\{e_i\}$ of $V$ and the corresponding dual basis $\{\epsilon_i\}$ for $V^\ast$, then via calculation we can obtain $\alpha(x)=\sum_i\alpha_ix_i$ where $\alpha=\sum_i\alpha_i\epsilon_i$ and $x=\sum_ix_ie_i$.
\end{remark}
\begin{definition}
    If $U\subset V$ is a subset of the vector space $V$, then the annihilator of $U$ is
    $$U^\circ=\{\alpha\in V^\ast:\forall u\in U,\alpha(u)=0\}$$
\end{definition}
\begin{lemma}
    $U^\circ\le V^\ast$ and if $U\le V$ and $\dim V$ is finite, then $\dim V=\dim U+\dim U^\circ$.
\end{lemma}
\begin{proof}
    Quite obvious that $U^\circ\le V^\ast$.
    To see the identity, write $n=\dim V$ and extend a basis $\{e_1,\ldots,e_k\}$ of $U$ to a basis $\{e_1,\ldots,e_n\}$ of $V$.
    Let $\{\epsilon_1,\ldots,\epsilon_n\}$ be the dual basis.
    Then it becomes obvious that $U^\circ=\langle\{\epsilon_{k+1},\ldots,\epsilon_n\}\rangle$, which gives the identity.
\end{proof}
\begin{lemma}
    Let $V,W$ be vector spaces over $F$ and $\alpha\in L(V,W)$.
    Then the map $\alpha^\ast:W^\ast\to V^\ast$ sending $\epsilon$ to $\epsilon\circ\alpha$ is linear.
\end{lemma}
\begin{proof}
    Obvious.
\end{proof}
\begin{definition}
    This map $\alpha^\ast$ is called the dual map of $\alpha$.
\end{definition}
\begin{proposition}
    Let $V,W$ be finite dimensional vector spaces over $F$ with bases $B,C$.
    Let $B^\ast,C^\ast$ be the dual bases of $V^\ast,W^\ast$, then $[\alpha^\ast]_{C^\ast,B^\ast}=[\alpha]_{B,C}^\top$.
\end{proposition}
\begin{proof}
    Just write it down.
\end{proof}
\begin{lemma}
    Let $E,F$ be bases of $V$ and $P=[\operatorname{id}]_{F,E}$ be the change-of-basis matrix from $F$ to $E$.
    Let $E^\ast,F^\ast$ be the corresponding dual bases, then the change-of-basis matrix from $F^\ast$ to $E^\ast$ is $(P^{-1})^\top$.
\end{lemma}
\begin{proof}
    We have
    $$[\operatorname{id}]_{F^\ast,E^\ast}=[\operatorname{id}]^\top_{E,F}=(P^{-1})^\top$$
    as desired.
\end{proof}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 7}

Suppose $n = 16$.
Draw the recursion tree for Equation \ref{eq16}, including every node in the tree.
Since $n$ is finite, there are only a finite number of nodes in the tree.
You need to show every one of them.

\begin{equation}
T(n) = T(\frac{n}{4})+T(\frac{n}{2})+n^2
\label{eq16}
\end{equation}

\subsection*{Solution}

Recursion tree for Equation \ref{eq16} is depicted in Figure \ref{fig1}.

\begin{figure}[H]\centering
\begin{tikzpicture}[level/.style={sibling distance=60mm/#1}]
\node [circle,draw] (z){$T(16)$}
  child {
    node [circle,draw] (a) {$T(8)$}
    child {
      node [circle,draw] (c) {$T(4)$}
      child {
        node [circle,draw] (g) {$T(2)$}
        child {
          node [circle,draw] (k) {$T(1)$}
        }
      }
      child {
        node [circle,draw] (h) {$T(1)$}
      }
    }
    child {
      node [circle,draw] (d) {$T(2)$}
      child {
        node [circle,draw] (i) {$T(1)$}
      }
    }
  }
  child {
    node [circle,draw] (b) {$T(4)$}
    child {
      node [circle,draw] (e) {$T(2)$}
      child {
        node [circle,draw] (j) {$T(1)$}
      }
    }
    child {
      node [circle,draw] (f) {$T(1)$}
    }
  };
\end{tikzpicture}
\caption{Recursion tree for $T(n)=T(\frac{n}{2})+T(\frac{n}{4})+n^2$ when $n = 16$}\label{fig1}
\end{figure}


\section{The Fundamental Theorem of Line Integrals}\label{sec:FundamentalTheoremLineIntegrals}

One way to write the Fundamental Theorem of Calculus
(\ref{thm:fundamental_theorem_I}) is:
$$\int_a^b f'(x)\,dx = f(b)-f(a).$$
That is, to compute the integral of a derivative $f'$
we need only compute the values of $f$ at the endpoints. Something
similar is true for line integrals of a certain form.

\begin{theorem}{Fundamental Theorem of Line Integrals}{FTLineIntegrals}
Suppose a curve $C$ is
given by the vector function ${\bf r}(t)$, with ${\bf a}={\bf r}(a)$
and ${\bf b}={\bf r}(b)$. Then
$$\int_C \nabla f\cdot d{\bf r} = f({\bf b})-f({\bf a}),$$
provided that $\bf r$ is sufficiently nice.
\end{theorem}
\begin{proof}
We write ${\bf r}=\langle x(t),y(t),z(t)\rangle$, so
that ${\bf r}'=\langle x'(t),y'(t),z'(t)\rangle$. Also,
we know that $\nabla f=\langle f_x,f_y,f_z\rangle$. Then
$$\int_C \nabla f\cdot d{\bf r} =
\int_a^b \langle f_x,f_y,f_z\rangle\cdot\langle
x'(t),y'(t),z'(t)\rangle\,dt=
\int_a^b f_x x'+f_y y'+f_z z' \,dt.$$
By the chain rule (see Section~\ref{sec:multivariable chain rules})
$f_x x'+f_y y'+f_z z'=df/dt$, where $f$ in this context means
$f(x(t),y(t),z(t))$, a function of $t$. In other words, all we have is
$$\int_a^b f'(t)\,dt=f(b)-f(a).$$
In this context,
$f(a)=f(x(a),y(a),z(a))$.
Since ${\bf a}={\bf r}(a)=\langle x(a),y(a),z(a)\rangle$, we can
write $f(a)=f({\bf a})$---this is a bit of a cheat, since we are
simultaneously using $f$ to mean $f(t)$ and $f(x,y,z)$, and since
$f(x(a),y(a),z(a))$ is not technically the same as
$f(\langle x(a),y(a),z(a)\rangle)$,
but the
concepts are clear and the different uses are compatible. Doing the
same for $b$, we get
$$\int_C \nabla f\cdot d{\bf r} = \int_a^b f'(t)\,dt=f(b)-f(a)=f({\bf
  b})-f({\bf a}).$$
\end{proof}

This theorem, like the Fundamental Theorem of Calculus, says roughly
that if we integrate a ``derivative-like function'' ($f'$ or $\nabla
f$) the result depends only on the values of the original function ($f$)
at the endpoints.

If a vector field $\bf F$ is the gradient of a function, ${\bf
  F}=\nabla f$, we say that $\bf F$ is a \dfont{conservative vector field}.
If $\bf F$ is a conservative force field, then the integral for work,
$\int_C {\bf F}\cdot d{\bf r}$, is in the form required by the
Fundamental Theorem of Line Integrals. This means that in a
conservative force field, the amount of work required to move an
object from point $\bf a$ to point $\bf b$ depends only on those
points, not on the path taken between them.

\begin{example}{Work Done}{}
An object moves in the force field
$${\bf F}=
\left
\langle {-x\over (x^2+y^2+z^2)^{3/2}},{-y\over (x^2+y^2+z^2)^{3/2}},{-z\over
  (x^2+y^2+z^2)^{3/2}}\right\rangle,$$
along the curve ${\bf r}=\langle 1+t,t^3,t\cos(\pi t)\rangle$ as $t$
ranges from 0 to 1. Find the work done by the force on the object.
\end{example}
\begin{solution}
The straightforward way to do this involves substituting the
components of ${\bf r}$ into $\bf F$, forming the dot product ${\bf
  F}\cdot{\bf r}'$, and then trying to compute the integral, but this
integral is extraordinarily messy, perhaps impossible to compute. But
since ${\bf F}=\nabla (1/\sqrt{x^2+y^2+z^2})$ we need only substitute:
$$\int_C {\bf F}\cdot d{\bf r}=
\left.{1\over \sqrt{x^2+y^2+z^2}}\right|_{(1,0,0)}^{(2,1,-1)}=
{1\over\sqrt6}-1.
$$
\end{solution}

Another immediate consequence of the Fundamental Theorem involves
\dfont{closed paths}. A path $C$ is closed if it
forms a loop, so that traveling over the $C$ curve brings you back to
the starting point. If $C$ is a closed path, we can integrate around
it starting at any point $\bf a$; since the starting and ending points are the
same,
$$\int_C \nabla f\cdot d{\bf r}=f({\bf a})-f({\bf a})=0.$$
For example, in a gravitational field (an inverse square law field)
the amount of work required to move an object around a closed path is
zero. Of course, it's only the \emph{net} amount of work that is
zero. It may well take a great deal of work to get from point $\bf a$
to point $\bf b$, but then the return trip will ``produce'' work. For
example, it takes work to pump water from a lower to a higher elevation,
but if you then let gravity pull the water back down, you can recover
work by running a water wheel or generator. (In the real world you
won't recover all the work because of various losses along the way.)

To make use of the Fundamental Theorem of Line Integrals, we need to
be able to spot conservative vector fields $\bf F$ and to compute
$f$ so that ${\bf F}=\nabla f$. Suppose that ${\bf F}=\langle
P,Q\rangle = \nabla f$. Then $P=f_x$ and $Q=f_y$, and provided that
$f$ is sufficiently nice, we know from Clairaut's Theorem
(\ref{thm:clairaut}) that $P_y=f_{xy}=f_{yx}=Q_x$. If we compute
$P_y$ and $Q_x$ and find that they are not equal, then $\bf F$ is not
conservative. If $P_y=Q_x$, then, again provided that $\bf F$ is
sufficiently nice, we can be assured that $\bf F$ is conservative.
Ultimately, what's important is that we be able to find $f$; as this
amounts to finding anti-derivatives, we may not always succeed.

\begin{example}{}{}
Find an $f$ so that $\langle 3+2xy,x^2-3y^2\rangle = \nabla f$.
\end{example}
\begin{solution}
First, note that
$${\partial\over\partial y}(3+2xy)=2x\qquad\hbox{and}\qquad
{\partial\over\partial x}(x^2-3y^2)=2x,$$
so the desired $f$ does exist. This means that $f_x=3+2xy$, so that
$f=3x+x^2y+g(y)$; the first two terms are needed to get $3+2xy$, and
the $g(y)$ could be any function of $y$, as it would disappear upon
taking a derivative with respect to $x$. Likewise, since
$f_y=x^2-3y^2$, $f=x^2y-y^3+h(x)$. The question now becomes, is it
possible to find $g(y)$ and $h(x)$ so that
$$3x+x^2y+g(y)=x^2y-y^3+h(x),$$
and of course the answer is yes: $g(y)=-y^3$, $h(x)=3x$. Thus,
$f=3x+x^2y-y^3$.
\end{solution}

\label{page:test for conservative vector field}
We can test a vector field ${\bf F}=\langle P,Q,R\rangle$ in a similar
way. Suppose that $\langle P,Q,R\rangle=\langle f_x,f_y,f_z\rangle$. If we temporarily hold
$z$ constant, then $f(x,y,z)$ is a function of $x$ and $y$, and
by Clairaut's Theorem $P_y=f_{xy}=f_{yx}=Q_x$.
Likewise, holding $y$ constant implies $P_z=f_{xz}=f_{zx}=R_x$, and
with $x$ constant we get $Q_z=f_{yz}=f_{zy}=R_y$. Conversely, if we
find that $P_y=Q_x$, $P_z=R_x$, and $Q_z=R_y$ then $\bf F$ is
conservative.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:FundamentalTheoremLineIntegrals}}

\begin{enumialphparenastyle}

\begin{ex}
Find an $f$ so that $\nabla f=\langle 2x+y^2,2y+x^2\rangle$, or
explain why there is no such $f$.
\begin{sol}
	no $f$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle x^3,-y^4\rangle$,
or explain why there is no such $f$.
\begin{sol}
	$x^4/4-y^5/5$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle xe^y,ye^x \rangle$,
or explain why there is no such $f$.
\begin{sol}
	no $f$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle y\cos x,y\sin x \rangle$,
or explain why there is no such $f$.
\begin{sol}
	no $f$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle y\cos x,\sin x\rangle$,
or explain why there is no such $f$.
\begin{sol}
	$y\sin x$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle x^2y^3,xy^4\rangle$,
or explain why there is no such $f$.
\begin{sol}
	no $f$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle yz,xz,xy\rangle$,
or explain why there is no such $f$.
\begin{sol}
	$xyz$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds\int_C (10x^4 - 2xy^3)\,dx - 3x^2y^2\,dy$ where $C$ is
the part of the curve $x^5-5x^2y^2-7x^2=0$ from $(0,0)$ to
$(3,2)$.
\begin{sol}
	414
\end{sol}
\end{ex}

\begin{ex}
Let
${\bf F}=
\langle yz,xz,xy\rangle$.
Find the work done by this force field on an object that moves from
$(1,0,2)$ to $(1,2,3)$.
\begin{sol}
	$6$
\end{sol}
\end{ex}

\begin{ex}
Let
${\bf F}=
\langle e^y,xe^y+\sin z,y\cos z\rangle$.
Find the work done by this force field on an object that moves from
$(0,0,0)$ to $(1,-1,3)$.
\begin{sol}
	$1/e-\sin3$
\end{sol}
\end{ex}

\begin{ex}
Let
$${\bf F}=
\left
\langle {-x\over (x^2+y^2+z^2)^{3/2}},{-y\over (x^2+y^2+z^2)^{3/2}},{-z\over
  (x^2+y^2+z^2)^{3/2}}\right\rangle.$$
Find the work done by this force field on an object that moves from
$(1,1,1)$ to $(4,5,6)$.
\begin{sol}
	$1/\sqrt{77}-1/\sqrt3$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\section{Reduction}

\subsection{Notation}
\begin{itemize}
    \item $\Gamma\vdash \tt t\rhd u$ : $\tt t$ reduces to $\tt u$ in $\Gamma$ with
    one of the $\beta,\iota,\delta,\zeta$ reductions.
    \item $\Gamma\vdash \tt t\mathop{\rhd}\limits^* u$ : $\Gamma\vdash\tt t\rhd\cdots\rhd u$.
    \item $\tt u\equiv v$ : $\tt u$ and $\tt v$ are identical.
\end{itemize}

\subsection{Conversion Rules}
\begin{equation*}
    \Gamma\vdash\tt((\lambda x:T.\ t)\ u)\ \rhd_\beta\ t\{x/u\}
    \tag{\sc $\beta$-Conv}
\end{equation*}
\begin{equation*}
    \Gamma\vdash\tt x\ \rhd_\delta\ t\qquad \textit{if }(x:=t:T)\in\Gamma
    \tag{\sc $\delta$-Conv}
\end{equation*}
\begin{equation*}
\Gamma\vdash \tt let\ \ x:=u\ \ in\ \ t\ \rhd_\zeta\ t\{x/u\}
    \tag{\sc $\zeta$-Conv}
\end{equation*}
\begin{equation*}
    \tt case((c_{p_i}\ q_1\ \cdots\ q_r\ a_1\ \cdots\ a_m),P,f_1\ |\ \cdots\ |\ f_n)\ \rhd_\iota\ (f_i\ a_1\ \cdots\ a_m)
    \tag{\sc $\iota$-Conv}
\end{equation*}
\begin{equation*}
\frac{\Gamma\vdash \tt t:\forall x:T,U\qquad x\textit{ fresh in }t}
    {\tt t\ \rhd_\eta\ \lambda\tt x:T.\ (t\ x)}
    \tag{\sc $\eta$-Exp}
\end{equation*}

Here's an example for $\iota$-reduction:
\begin{center}
\begin{minipage}{0.6\textwidth}
\begin{minted}{coq}
match (S m) as s0 in nat return nat with
| S n => n
| O => O
end

(* $\iota$-reduction *)
(fun (n:nat) => n) m.
\end{minted}
\end{minipage}
\end{center}

\begin{Def}[Convertibility]
$\tt t_1$ and $\tt t_2$ are convertible iff there exists $\tt u_1$ and $\tt u_2$ such that
$\Gamma\vdash \tt t_1\mathop{\rhd}\limits^* u_1$ and $\Gamma\vdash \tt t_2\mathop{\rhd}\limits^* u_2$
and either $\tt u_1\equiv u_2$ or they are convertible up to $\eta$-expansion.
\end{Def}


\documentclass{article}
\usepackage{amsmath}
\usepackage[margin=1.0in]{geometry}
\usepackage{xcolor}

\begin{document}

\noindent
Does $\displaystyle \sum_{n=1}^\infty \frac{n^2}{2^n}$
diverge, converge absolutely, or converge conditionally?

\subsection*{Solution}

\begin{align*}
L&=\lim_{n \to \infty} \left|\frac{a_{n+1}}{a_n}\right|\\
&= \lim_{n \to \infty} \left| \frac{(n+1)^2}{2^{n+1}} \cdot \frac{2^n}{n^2}\right|\\
&= \lim_{n \to \infty} \left| \frac{n^2+2n+1}{2^n \cdot 2} \cdot \frac{2^n}{n^2}\right|\\
&= \lim_{n \to \infty} \left| \frac{n^2+2n+1}{2n^2}\right|\\
&= \lim_{n \to \infty} \frac{n^2+2n+1}{2n^2}\\
&= \lim_{n \to \infty} \frac{2n+2}{4n}\quad\text{by L'hopital}\\
&= \lim_{n \to \infty} \frac{2}{4}\quad\text{by another L'hopital}\\
&= \frac24=\frac12
\end{align*}

Since $L< 1$, the series $\displaystyle \sum_{n=1}^\infty \frac{n^2}{2^n}$ converges absolutely by the Ratio Test.

\subsection*{Solution 2 (outline)}

The Root Test will work, but it is a lot of work compared to the solution presented.

\end{document}%%%%%%%%%%%%%%%%%

\begin{align*}
L&=\lim_{n \to \infty} \sqrt[n]{|a_n|}\\
&= \lim_{n \to \infty} \sqrt[n]{\left| \right|}\\
\end{align*}

\begin{align*}
L&=\lim_{n \to \infty} \left|\frac{a_{n+1}}{a_n}\right|\\
&= \lim_{n \to \infty} \left| \right|\\
\end{align*}

\begin{align*}
\lim_{n \to \infty} a_n
&= \lim_{n \to \infty} \\
\end{align*}

Since $\sum |a_n| = \sum a_n$, the series $\displaystyle \sum_{n=1}^\infty AAAAAAAAAAAAAA$ converges absolutely.

Since $|r| < 1$, the series ...  converges by the Geometric Series Test.

Since $|r| \geq 1$, the series ...  diverges by the Geometric Series Test.

The function $f(x)=\frac{}{}$ is continuous, positive, and decreasing on $[1,\infty)$.

\subsection*{Solution}


\chapter{Math}

\begin{eqnarray*}
	f(x) = \sum_{i=0}^{n} \frac{a_i}{1+x} \\
	\textstyle f(x) = \textstyle \sum_{i=0}^{n} \frac{a_i}{1+x} \\
	\scriptstyle f(x) = \scriptstyle \sum_{i=0}^{n} \frac{a_i}{1+x} \\
	\scriptscriptstyle f(x) = \scriptscriptstyle \sum_{i=0}^{n} \frac{a_i}{1+x}
\end{eqnarray*}

\section{Inline math}
In this line is a beautiful formular $e^{i\pi} + 1 = 0$ \par
This is also inline but special \[e^{i\pi} + 1 = 0\] Do you see it?

\section{Formulars}
For not only mention a formular, you should use that format:

\begin{equation}
E=mc^2
\end{equation}

\section{Multiline}

For long equations
\begin{multline}
p(x) = 3x^6 + 14x^5y \\+ 590x^4y^2 + 19x^3y^3
- 12x^2y^4 - 12xy^5 + 2y^6 \\- a^3b^3 + a^3 - b^6
\end{multline}

Or multipart
\begin{equation}
\begin{split}
A & = \frac{\pi r^2}{2} \\
& = \frac{1}{2} \pi r^2
\end{split}
\end{equation}

Or on another way:

\begin{align}
x&=y           &  w &=z              &  a&=b+c\\
2x&=-y         &  3w&=\frac{1}{2}z   &  a&=b\\
-4 + 5x&=2+y   &  w+2&=-1+w          &  ab&=cb
\end{align}

Without alignment:

\begin{gather}
2x - 5y =  8 \\
3x^2 + 9y =  3a + c
\end{gather}

\section{spacing}
\begin{align*}
f(x) =& x^2\! +3x\! +2 \\
f(x) =& x^2+3x+2 \\
f(x) =& x^2\, +3x\, +2 \\
f(x) =& x^2\: +3x\: +2 \\
f(x) =& x^2\; +3x\; +2 \\
f(x) =& x^2\ +3x\ +2 \\
f(x) =& x^2\quad +3x\quad +2 \\
f(x) =& x^2\qquad +3x\qquad +2
\end{align*}

\section{Brackets}

\ldots need to be equalized even with invisible brackets

\begin{align}
y  = 1 + & \left(  \frac{1}{x} + \frac{1}{x^2} + \frac{1}{x^3} + \ldots \right. \\
& \quad \left. + \frac{1}{x^{n-1}} + \frac{1}{x^n} \right)
\end{align}

\section{Units}
\SI{24,38176541e4}{m.Pa} \\
\num{24,38176541e4} \\
\si{\kilo\gram\metre\per\square\second} \\
\si{\gram\per\cubic\centi\metre}        \\
\si{\square\volt\cubic\lumen\per\farad} \\
\si{\metre\squared\per\gray\cubic\lux}  \\
\si{\henry\second} \\
\SI[mode=text]{1.23}{J.mol^{-1}.K^{-1}}          \\
\SI{.23e7}{\candela}                              \\
\SI[per-mode=symbol]{1.99}[\$]{\per\kilogram}    \\
\SI[per-mode=fraction]{1,345}{\coulomb\per\mole} \\
\si[unit-color=blue]{\highlight{red}\kilogram\metre\per\second}
$p=\SI{24,381}{MPa}$

\section{list/Range}
\numlist{10;30;50;70} \\
\numrange{10}{30}\\
\SIlist{10;30;45}{\metre}\\
\SIrange{10}{30}{\metre} \\
\num[negative-color = red]{-15673}

\section{Angle}
\ang{10}    \\
\ang{12.3}  \\
\ang{4,5}   \\
\ang{1;2;3} \\
\ang{;;1}   \\
\ang{+10;;} \\
\ang{-0;1;}


\chapter{Quantitative Finance}
\label{ch:quantitative-finance}
\index{Quantitative Finance}

\newthought{This section deals with some applications of quantitative finance,
both computational and theoretical. }

\section{Equity Mathematical Models}
\index{Equity Mathematical Models}
Equities and finance are heavily influenced by random events (see \cite{fooledbyrandomness} for a detailed philosophical discussion). As such, the analysis of financial instruments lends itself to a stochastic process.

A {\em Geometric Brownian Motion}\index{Geometric Brownian Motion} (GBM) equation (a form of stochastic differential equation, or SDE known as the Black-Sholes-Merton equation\cite{pyfinanceoreilly}) can be used to approximate a random process over time. In its continuous form this looks like \cite{advancedquantcpp}:
\begin{equation}
ds = \mu Sdt+\sigma SdW
\end{equation}
This breaks the movement of a stock price down into two key effects:

\begin{itemize}
	\item a deterministic effect (the left of the plus sign)
	\item a stochastic effect (the right of the plus sign)
\end{itemize}
In the equation, $\mu$ is known as {\it drift}, $\sigma$ is {\it volatility}, $S$ is the stock price, $dt$ is the change in time and $dW$ is an increment in a {\it Weiner process}.

As equity markets are a discrete process, this equation must be transformed into a discrete equation.
\begin{equation}
S_{t+1}=S_t(1+r\Delta{}t+\sigma\varepsilon_t \sqrt{\Delta{}t})
\end{equation}
Here $\varepsilon$ is a sample from a gaussian distribution with zero mean and standard deviation of 1 (i.e. $N(0,1)$) and $r$ is the risk free rate of return. This equation can be solved iteratively for a given time period if $r$, $\sigma$, $\varepsilon$ and $S_0$ are provided.

Another formulation for pricing European Call options gives a similar result of the discretised SDE\cite{pyfinanceoreilly}:
\begin{equation}
S_t=S_{t-\Delta t}e^{(r-\frac{1}{2}\sigma^2)\Delta t+\sigma\sqrt{\Delta t}z_t}
\end{equation}
In this instance, $z_t$ represents the random variable. Similar formulations can be determined for foreign exchange:
\begin{equation}
X_{t+1} = X_t(1+(r_d-r_f)\Delta{}t+\sigma\varepsilon_t\sqrt{\Delta{}t})
\label{eq:foreignexchangemodel}
\end{equation}
where $r_d$ and $r_f$ are the domestic and foreign fisk free rates of return.

\section{Structural and Intensity Models}
\TODO See \cite{advancedquantcpp} chapter 2.

\section{Monte Carlo Simulation}
\index{Monte Carlo Simulation}
Monte Carlo simulation is a method of estimating a probabilistic outcome through a high quantity of simulations. In a quantitative finance context, we may wish to estimate the future price of an equity based on use of a GBM equation simulated $M$ times.

For instance the process could be as follows for calculating a call option derivative based on estimate the price of a stock using Monte Carlo simulation:

\begin{enumerate}
	\item Generate $M$ different ``trajectories'' for the stock using a GBM simulation from time $t=0$ to time $t=T$. This generates a set of $N$ price estimates for $M$ different simulations, with the notation:
	\begin{equation}
		\{S_i^j\}\qquad i = 0...N,\qquad j=1...M
	\end{equation}
	This produces a vector of $M$ values for $S_T$,
	\begin{equation}
		\{S_T^i\}\qquad i = 0...M
	\end{equation}

	\item We next compute the pay off for each stock value. This is given by
	\begin{equation}
		H(S_T^i)\qquad i=1...M
	\end{equation}
	Where
	\begin{equation}
		H(S_T)=max(S_T-K,0)
	\end{equation}
	and $K$ is the actual price of the equity or premium. The expected pay off can then be computed by the average of all pay offs:
	\begin{equation}
		E[H(S_T^i)]=\frac{1}{M}\Sigma_{i=1}^{M}H(S_T^i)
	\end{equation}

	\item The item should then be discounted to present value, by either applying a discount factor $DF_T$ or
	\begin{equation}
		\pi = e^{-rT}\times E[H(S_T)]
	\end{equation}

	Where $\pi$ is the value of the derivative.
\end{enumerate}

\section{Binomial Trees \cite{advancedquantcpp}}
\index{Binomial Trees}

This approach builds a tree of possible prices. At each stage, the underlying can be assumed to go up or down by a given amount. The amount of change up ($u$) or down ($d$) is described by
\begin{equation}
	u=e^{\sigma\sqrt{\Delta{}t}}
\end{equation}
\begin{equation}
	d=e^{-\sigma\sqrt{\Delta{}t}}
\end{equation}
The probability of an equity going up $p$ is
\begin{equation}
p=\frac{e^{r\Delta{}t}-d}{u-d}
\end{equation}
The probability of the underlying going down is $1-p$. The binomial tree is then built in the following phases:

\begin{enumerate}
	\item Construct a tree where each level corresponds to a time step in the simulation period from $t=0$ to $t=T$. For example in two simulation steps,
	\begin{equation}
		At = 0, S = S_0
	\end{equation}
	\begin{equation}
		At = t_1, S = uS_0,\qquad dS_0
	\end{equation}
	\begin{equation}
		At = t_2, S = u^2S_0,\qquad udS_0,\qquad udS_0,\qquad d^2S_0
	\end{equation}
	Note that the central value at the last period is shared between adjacent nodes, hence there are only three distinct estimates after $N=2$ steps.

	This produces a number of prices at each time step. The notation is based on the estimate number $k$ at time T. There are $N$ time steps
	\begin{equation}
		\{S_T^k\}\qquad k=1...N+1
	\end{equation}

	\item Once the tree has been built, the payoff $H(S_T^k)$ should be calculated for each $S_T^k$

	\item Finally the tree is traversed back up towards the root node, calculating the discounted weighted probability of each node. If we define a given node (not a leaf, i.e. $t\neq T$) it has two children, one for the up price (denoted $S_T^u$) and one for the down price, $S_T^d$. The value for the parent node is given by
	\begin{equation}
	V_{T-1}^k = e^{-r\Delta{}t}[pH(S_T^u) + (1-p)H(S_T^d)]
	\end{equation}
	in this case $V_T^l$ is shorthand for $H(S_T^k)$

	\item Once the tree has been traversed back to the top, the value of the derivative $\pi=V_1^1$
\end{enumerate}

\section{Finite Difference Method}
\index{Finite Difference Method}

The finite difference method is a method for discretising a differential equation.\cite{advancedquantcpp} In the quantitative finance approach, we want to discretise partial differential equations (PDEs). The finite differences method is based on the relationship:
\begin{equation}
f(x) = \frac{df}{dx} \approx \frac{\Delta f}{\Delta x} = \frac{f_{i+1}-f_i}{\Delta x}
\end{equation}
The most important PDE in finance is the {\em Black-Scholes PDE}
\index{Black-Scholes PDE}, which is given by:
\begin{equation}
\frac{\delta V}{\delta t} + \frac{1}{2}\sigma^2S^2\frac{\delta^2V}{\delta S^2}+rS\frac{\delta V}{\delta S}-rV=0
\label{eq:blackscholeseq}
\end{equation}
This is usually solved in the $S$ and $t$ axes, where $S\in [a,b]$ and $t\in [0,T]$. The domain of this equation is said to be
\begin{equation}
\Omega=\{(S,t)\forall S\in [a,b]\times t\in [0,T]\}
\end{equation}
In other words, as the finite difference method is solving some partial differential equation in $S$ and $t$, the solution space is the rectangular domain defined by the ranges of $S$ and $t$. For a European call,
\begin{equation}
	V(S,T) = max(S-K,0)
\end{equation}
The boundary conditions are $V(a,t)=0$ and $V(b,t)=S$. This equation can be transformed with some variable substitution so that
\begin{equation}
\frac{\delta u}{\delta\tau}=\frac{\delta^2 u}{\delta x}\qquad -\infty<x<\infty, \tau>0
\end{equation}
This is a dimensionless PDE with a new solution domain $\Omega=\{(x,\tau)\}$. The payoff relationship therefore becomes
\sidenote{Where $k=\frac{r}{0.5\times\sigma^2}$}
\begin{equation}
u(x,0) = max(e^{\frac{1}{2}(k+1)x}-e^{\frac{1}{2}(k-1)x},0)
\end{equation}
Using finite differences, the return can be described as\sidenote{Where $\alpha=\frac{\Delta\tau}{(\Delta x)^2}$}
\begin{equation}
u_{i,j+1}=\alpha u_{i+1,j}+(1-2\alpha)u_{i,j}+\alpha u_{i-1,j}
\end{equation}
This relationship can be solved iteratively, using the following steps:

\begin{enumerate}
	\item Discretise the domain into N space divisions of $dS$ and M time divisions of $dT$. Use these to determine the steps $\Delta\tau,\Delta x$.
	\item Use finite differences to approximate the derivatives
	\item Calculate the results of the equation iteratively for each time step
\end{enumerate}

For a worked example, see \cite{advancedquantcpp}, end of chapter 3.


\subsection{Prior and posterior predictive distributions}

\subsubsection{Prior predictive distribution}

Our prior predictive distribution for \(X\) depends on our prior for \(\theta \).

\(P(\mathbf x)=\int_\Theta P(\mathbf x|\theta)P(\theta )d\theta \)

\subsubsection{Posterior predictive distribution}

Once we have calculated \(P(\theta |X)\), we can calculate a posterior probability distribution for \(X\).

\(P(\mathbf x|\mathbf X)=\int_\Theta P(\mathbf x|\theta)P(\theta |\mathbf X)d\theta \)


\subsection{Defining sets by enumeration}

We can describe a set by the elements it contains.

\(s=\{a,b,c\}\)

This is a shorthand way of writing:

\(\forall x (x\in s \leftrightarrow (x=a\lor x=b \lor x=c))\)


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS622: Theory of Formal Languages
% Copyright 2014 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 5}

Let $L$ be a regular language over an alphabet $A$.
Prove that the set of all words $z \in A^*$ for which there exist $u,v \in L$ such that $u = zx$ and $v=yz$ for some $x,y \in A^*$ is a regular language.

\subsection*{Solution}

We assume $S_1$ as set of all words $z \in A^*$ for which there exist $u \in L$ such that $u=zx$ for some $x \in K \subseteq A^*$.
We can represent $S_1$ as shown in Equation \ref{Eq1}.

\begin{equation}\label{Eq1}
	S_1 = LK^{-1} = \{z\in A^* | \exists x\in K, zx\in L\}
\end{equation}

Also, we assume $S_2$ as set of all words $z \in A^*$ for which there exists $v \in L$ such that $v = yz$ for some $y \in K \subseteq A^*$.
We can represent $S_2$ as shown in Equation \ref{Eq2}.

\begin{equation}\label{Eq2}
	S_2 = K^{-1}L = \{z\in A^* | \exists y\in K, yz\in L\}
\end{equation}

It has been proven \cite{simovici1999theory} that for every language $K$, both the right and the left quotients $LK^{-1}$ and $K^{-1}L$ are regular, if $L$ is a regular language over alphabet $A$.
Based on this theorem, $S_1$ and $S_2$ would both be regular.

Since any word $z \in A^*$ for which there exist $u,v \in L$ such that $u = zx$ and $v=yz$ for some $x,y \in A^*$ should be an element of $S_1$ and $S_2$, set of all such words will be the intersection of $S_1$ and $S_2$.
As regular languages $\mathcal{R}$ are closed with respect to intersection, it is proven that the desired set is a regular language.


\addcontentsline{toc}{chapter}{Useful Formulae}
\chapter*{Useful Formula}
\begin{itemize}
	%%%%%%%%%%
	\item Equations of a Line:
\begin{center}\begin{tabular}{rl}
		point $(x_1,y_1)$, slope $m$ form:&$y-y_1 = m(x - x_1)$\\
		slope $m$, intercept $b$ form:&$y=mx+b$
	\end{tabular}\end{center}
	%%%%%%%%%%
	\item Quadratic Formula:
	If $ax^2+bx+c=0$,\,$a\ne0,\quad$ then $\displaystyle\quad x = \frac{-b\pm \sqrt{b^2-4ac}}{2a}$
	%%%%%%%%%%
	\item Sine Law: $\frac{\sin A}{a}= \frac{\sin B}{b}=\frac{\sin C}{c}$; \hspace{0.2cm}Cosine Law: $a^2=b^2+c^2-2bc\cos(A)$
\end{itemize}
{\hspace*{0.55cm}
		\renewcommand{\arraystretch}{1.2}
		\begin{tabular}{lrclcrcl}
			&\multicolumn{3}{l}{\phantom{aaaa}$\bullet$ Logarithms}& &\multicolumn{3}{c}{$\bullet$ Exponents}\\
			\cmidrule{1-4}\cmidrule{6-8}
			Law 1: &$ \log_x(ab) $&$=$&$ \log_x(a)+\log_x(b)  $&\hspace{1cm} &$x^a x^b$&$=$&$x^{a+b} $\\
			Law 2: &$ \log_x\left(\frac{a}{b}\right) $&$=$&$  \log_x(a)-\log_x(b) $& &$\dfrac{x^a}{x^b}$&$=$&$x^{a-b}  $\\
			Law 3: &$ \log_x(a^b) $&$=$&$ b\cdot\log_x(a)   $& &$(x^a)^b $&$=$&$x^{ab} $\\
			\cmidrule{1-4}\cmidrule{6-8}
			&$  \log_{x}\left(\frac{1}{x^a}\right) $&$=$&$-a   $& &$x^{-a} $&$=$&$\frac{1}{x^a} $\\
			&$ \log_{x}1  $&$=$&$ 0  $& &$x^0 $&$=$&$1 $\\
			&$ \log_x(x) $&$=$&$  1 $& &$x^1 $&$=$&$x $\\
			\bottomrule
			&$ \log_a(x) $&$=$&$y$\hspace{0.5cm} converts to& $\Leftrightarrow$ &$x $&$=$&$a^y $\\
			\bottomrule
		\end{tabular}\\
}
\newline\phantom{blank line here please}\\
{\hspace*{0.55cm}
	\renewcommand{\arraystretch}{1.2}
		\begin{tabular}{llll}
			\multicolumn{4}{c}{$\bullet$ Standard Derivatives}\\\midrule
			$f(x)$ & $f'(x)$  &&Notes\\ \midrule
			$A$&0&&$A$ is constant\\
			$x$&$1$&&power rule for $x^1$\\
			$Ax$&$A$&&$A$ is a constant multiple\\ \midrule
			$x^n$ & $nx^{n-1}$ &&power rule general form\\
			$e^x$ & $e^x$  && exponential\\
			$\ln(x)$ & $\frac{1}{x}$ &&logarithmic\\ \midrule
			$\sin(x)$ & $\phantom{-}\cos(x)$  &&trigonometric \\
			$\cos(x)$ & $-\sin(x)$ && \\
			$\tan(x)$ & $\phantom{-}\sec^2(x)$ && \\ \bottomrule
		\end{tabular}
}
\begin{itemize}
	%%%%%%%%%%
	\item Product Rule: $[f(x)g(x)]' = f'(x)g(x) + f(x)g'(x)$
	%%%%%%%%%%
	\item Quotient Rule: $\displaystyle \left[\frac{f(x)}{g(x)}\right]' = \frac{f'(x)g(x) - f(x)g'(x)}{[g(x)]^2}$
	%%%%%%%%%%
	\item Chain Rule: $f[g(x)]' = f'[g(x)] \cdot g'(x)$
	%%%%%%%%%%
\clearpage
\thispagestyle{empty}
		\renewcommand{\arraystretch}{1.2}
		\begin{tabular}{llcl}
			\multicolumn{4}{c}{$\bullet$ Standard Integrals}\\\midrule
			$f(x)$  &  $\int f(x)dx$&&Notes \\ \midrule
			$1$&$x+C$&&constant\\
			$A$&$Ax+C$&&$A$ is constant\\\midrule
			$x^n, n \neq -1$ & $\displaystyle\frac{x^{n+1}}{n+1}+C$&&power rule general form\\
			$e^x$ & $e^x+C$&&exponential\\
			$\frac{1}{x}$ & $\ln|x|+C$&&special case: $x^{-1}$\\
			$\ln x$&$x\ln |x|-x+C$&&natural logarithm: $\log_e$\\\midrule
			$\sin(x)$ & $-\cos(x)+C$&& trigonometric \\
			$\cos(x)$ & $\sin(x)+C$\\
			$\tan (x)$&$\ln |\sec x|+C$\\
			$\sec^2(x)$ & $\tan(x)+C$ \\ \bottomrule
		\end{tabular}
	%%%%%%%%%%
	\item Integration By Parts: $\displaystyle \int fg' = fg-\int f'g \qquad \text{or} \qquad \int udv = uv - \int vdu$
	%%%%%%%%%%
	\item Area between the curve $f(x)$ and the $x$-axis: $\displaystyle A=\int_{a}^{b} f(x)dx$\\
	%%%%%%%%%%
	\item Area between two curves $f(x)$ and $g(x)$: $\displaystyle A=\int_{a}^{b} \left[f(x)-g(x)\right]dx$\\
	%%%%%%%%%%
	\item Volume of revolution about the $x$-axis:
	$\displaystyle V= \int_{a}^{b} \pi y^2 dx = \pi\int_{a}^{b}  [f(x)]^2 dx$
	\item Volume of revolution about the $y$-axis:
	$\displaystyle V= \int_{a}^{b} \pi x^2 dy = \pi\int_{a}^{b}  [f(y)]^2 dy$
\end{itemize}


\subsubsection{Cosine}
\noindent
Let $a$ be a constant.\\
By definitions of a Laplace transform and an improper integral,
\begin{equation*}
\Laplace{\cos{(at)}} = \lim\limits_{n\to\infty}{\int_{0}^{n}{\cos{(at)}e^{-st}\mathrm{d}t}}
\end{equation*}
\begin{equation*}
	= \frac{1}{s^2 + a^2}\lim\limits_{n\to\infty}{\left[e^{-st}\left(a\sin{(at)} - s\cos{(at)}\right)\right]_{0}^{n}}
\end{equation*}
\begin{equation*}
	 = \frac{1}{s^2 + a^2}\left(\lim\limits_{n\to\infty}{\left(e^{-sn}\left(a\sin{sn} - s\cos{(an)}\right)\right)} - \left(e^{-s\cdot 0}\left(a\sin{(a\cdot 0)} - s\cos{(a\cdot 0)}\right)\right)\right)
\end{equation*}
Both $\sin$ and $\cos$ have maximum values of 1 and minimum values of -1, so we can way that the left part of the expression has a maximum value of at most $a + s$. For positive $s$, the exponential dominates and the expression goes to 0 in the limit.
\begin{equation*}
	 = \frac{1}{s^2 + a^2}\left(0 + s\right) \text{, } s > 0
\end{equation*}
\begin{equation*}
	 = \frac{s}{s^2 + a^2} \text{, } s > 0
\end{equation*}
So,
\begin{equation*}
	\Laplace{\cos{(at)}} = \frac{s}{s^2 + a^2} \text{, } s > 0
\end{equation*}


\chapter{Multigrid Methods and Residual Neural Networks}
We are going to investigate the connections and difference between one of the most efficient solvers in numerical PDEs ``Multigrid Methods" and one of the most successful models ``Residual Neural Networks" especially for the one of the most updated ResNet \cite{he2016identity} for computer vision.

\section{Multigrid methods}\label{cs:mg}
\subsection{Multigrid methods for numerical PDEs}

Let us first briefly describe a geometric multigrid method used to
solve the following boundary value problem:
% \begin{equation}
%   \label{laplace}\left\{
%   \begin{array}{rccl}
%     -\Delta u &=& f & \mbox{ in } \Omega,\\
%     \frac{\partial u}{\partial n} &=& 0 & \mbox{ on } \partial\Omega,\\
%   \end{array}
%   \right.\quad \Omega=(0,1)^2.
% \end{equation}
\begin{equation}
\label{laplace}
-\Delta u = f,  \mbox{ in } \Omega,\quad
u =0  \mbox{ on } \partial\Omega,\quad
\Omega=(0,1)^2.
\end{equation}

\subsection{Multigrid methods as network}
\input{6DL/v-cycle}

We consider a continuous linear finite element discretization of
\eqref{laplace} on a nested sequence of grids of sizes $n_\ell\times
n_\ell$ with $n_{\ell-1}=(n_\ell-1)/2$, as shown in the left part of
Fig. \ref{fig:V-cycle} and the corresponding sequence of finite
element spaces:
$$
V_1\supset V_2\supset\cdots\supset V_J.
$$
The discretized system is
\begin{equation}
\label{laplace-h}
Au=f.
\end{equation}
In correspondence with the two-dimensional grid, here we treat $f\in \mathbb R^{n\times n}$.  We need to find $u\in \mathbb R^{n\times n}$ satisfying \eqref{laplace-h}.
Here,  $A:\mathbb R^{n\times n}\mapsto \mathbb R^{n\times n}$ is a tensor satisfying
\begin{equation}
\label{uniform-laplace}
(Au)_{i,j}=4u_{i,j}-u_{i+1,j}-u_{i-1,j}-u_{i,j+1}-u_{i,j-1},
\end{equation}
which holds for $1\le i,j \le n$ whereas a slightly different formula holds for
other $i, j$. Here we notice that, there exist a $3\times 3$ kernel as
\begin{equation}\label{eq:kernel-A}
K_A = \begin{pmatrix}
0 & -1 & 0 \\
-1 & 4 & -1 \\
0 & -1 & 0
\end{pmatrix},
\end{equation}
with
$$
Au = K_A \ast u.
$$
Where $\ast$ is the stander convolution operation with zero padding like \cite{goodfellow2017deep}.
We now briefly describe a two-level multigrid method by
a mixed use of the terminologies from deep learning \cite{goodfellow2017deep} and multigrid methods.

The first main ingredient in GMG is a smoother.  A commonly used smoother is a
damped Jacobi,  which can be written as $S:\mathbb R^{n\times n}\mapsto
\mathbb R^{n\times n}$ satisfying
\begin{equation}
\label{jacobi1}
(Sf)_{i,j}={\omega\over 4}f_{i,j}.
\end{equation}
If we apply the Jacobian iteration twice, then
\begin{equation}
\label{jacobi2}
(S^2f)_{i,j}
={1\over 4}\omega(2-\omega)f_{i,j}+{\omega^2\over 16}(f_{i+1,j}+f_{i-1,j}+f_{i,j+1}+f_{i,j-1}).
\end{equation}
Then we have
\begin{equation}\label{eq:kernel-S}
K_S = {\omega \over 4},
\end{equation}
and
\begin{equation}\label{eq:kernel-S2}
K_{S^2} = \begin{pmatrix}
0 & \frac{\omega^2}{16} & 0 \\
\frac{\omega^2}{16} & {\omega(2-\omega) \over 4} & \frac{\omega^2}{16}  \\
0 & \frac{\omega^2}{16}  & 0
\end{pmatrix}
\end{equation}
Another ingredient is the restriction matrix $R: \mathbb R^{n_1\times
	n_1}\mapsto \mathbb R^{n_2\times n_2}$, which can be obtained by two
different steps. First, we carry out a convolution operation $\tilde
R: \mathbb R^{n_1\times n_1}\mapsto \mathbb R^{n_1\times n_1}$:
\begin{equation}
\label{linear-restriction}
(\tilde Rr)_{i,j}=r_{i,j}+{1\over 2}(r_{i-1,j}+r_{i+1,j}+r_{i,j-1}+r_{i,j+1}+r_{i-1,j-1}+r_{i+1,j+1}),
\end{equation}
Then, we define
\begin{equation}
\label{linear-restriction-stride}
(Rr)_{i,j}=(\tilde R r)_{2i-1,2j-1}, \quad 1\le i, j \le n_2.
\end{equation}
Using the terminology from deep learning \cite{goodfellow2017deep},
we note that \eqref{linear-restriction} can be written as a
convolution with a $3\times3$ kernel:
\begin{equation}
\label{convolution}
\tilde Rr=K*r,\quad K=\left ( \begin{array}{ccc}
0 &\frac{1}{2}&\frac{1}{2}\\
\frac{1}{2}& 1&\frac{1}{2}\\
\frac{1}{2}&\frac{1}{2}& 0
\end{array}\right ).
\end{equation}
Similarly, \eqref{jacobi1} is a convolution with a $1\times1$ kernel,
\eqref{jacobi2} is a convolution with a $3\times 3$ kernel, and
\eqref{linear-restriction} and \eqref{linear-restriction-stride} is a
convolution with a stride.

In general, we have restriction $R_{\ell-1}^\ell:\mathbb
R^{n_{\ell-1}\times n_{\ell-1}}\mapsto R^{n_{\ell}\times n_{\ell}}$ on
level $\ell$, and the corresponding prolongation is given by
$P_\ell^{\ell-1}=[R_{\ell-1}^\ell]^T$.  This restriction gives rise
to a restriction of $A=A_1$ on the coarse mesh:
$A_\ell=R_{\ell-1}^\ell A_{\ell-1}P_\ell^{\ell-1}: \mathbb
R^{n_\ell\times n_\ell}\mapsto \mathbb R^{n_\ell\times n_\ell}$
($\ell=2:J$).

Using the smoother $S$ and restriction $R$ above, a two-level
multigrid method for solving \eqref{laplace-h} can be written as
$u\leftarrow u_0, r\leftarrow f-Au$:
\begin{equation}
\label{2level}
u\leftarrow u+ Sr, \quad r\leftarrow f-Au, \quad u\leftarrow u+ B_2Rr,
\quad r\leftarrow f-Au, \quad u\leftarrow u+S R^Tr.
\end{equation}
In a two-level setting, we can take $B_2=A_2^{-1}$, or we can apply
\eqref{2level} recursively to obtain a multilevel V-cycle.     More
specifically, given $u_0\in V$,  this multilevel V-cycle can be
described as follows

From the fine to the coarse level,
\begin{equation}\label{multi-backslash}
r_\ell = F_\ell (R^\ell_{\ell - 1} r_{\ell-1}), \quad
w_\ell =[(I+S_{\ell}(I - A_{\ell} )]R^{\ell}_{\ell-1} r_{\ell-1}\quad \ell = 2:J,
\end{equation}
where
\begin{equation}\label{eq:backslash}
F_\ell (x) = x - A_\ell S_\ell x, \quad \ell = 1:J-1
\end{equation}
with
$$
r_0=f-Au_0, r_1 = F_1(r_0),  F_J(x) = A_J^{-1}x.
$$
From the coarse to the fine level,
\begin{equation}\label{multi-slash}
v_{\ell-1} = G_{\ell-1} (P^{\ell-1}_{\ell} v_{\ell}), \quad \ell = J:2,
\end{equation}
where
\begin{equation}\label{eq:slash}
G_{\ell-1} (x) = w_{\ell-1} + x - S_\ell A_\ell x
\end{equation}
with
$$
v_J = r_J = F_J(R^J_{J-1} r_{J-1}).
$$
An example of a $4$-level ($J=4$) V-cycle is illustrated in
Fig. \ref{fig:V-cycle}.

In a more general setting, \eqref{laplace-h} can be a general linear
algebraic system, with a slight abuse of notation here, such that
$u\in \mathbb R^N$ and $A: \mathbb R^N\mapsto \mathbb R^N$.  (In the
uniform grid case $N=n^2$.)  In this case, the AMG method can still be
written as in \eqref{2level}, but the smoother and restriction
matrices must be designed to maximize the algorithm efficiency.  In
multigrid methods, we almost always choose the prolongation
$P=R^T$. Thus the design of restriction is the same as the design of
prolongation.  Most of the existing AMG literature focuses on
designing an appropriate prolongation for a given smoother such as the
Jacobi iterative method.  In this proposal, we advance the field by
also considering the design of an appropriate smoother for a given
restriction.

\section{Convolutional neural networks}\label{sc:cnn}
Convolutional neural network, first introduced by LeCun
\cite{lecun1998gradient}, has been successfully applied to many
problems.  One of the most successful applications is for image
classification
\cite{lecun1998gradient,krizhevsky2012imagenet,simonyan2014very,he2016deep}.
Here,  for simplicity of exposition, we introduce CNN for image
de-noising
\cite{NIPS2008_3506,bengio2013generalized,alain2014regularized,xie2012image,zhang2017beyond}.
We emphasize that our introduction may not follow the standard
notation or terminology used in the deep learning literature. Instead
we borrow some notation and terminology from multigrid methods
for our descriptions.

Given an image $x \in \mathbb{R}^{n\times n}$ stained with
noise (often assumed to be Gaussian), our goal is to use a deep neural
network to find a de-noised image such that $ y=f(x, \Theta) $.  The
function $f$, which depends on a set of parameters $\Theta$ and can be
viewed as a vector in $\mathbb R^{\mathcal N}$, will be given
in terms of a convolutional neural network  briefly described
next.  The parameter $\Theta$ can be obtained, for example, by
solving the following least squares problem:
\begin{equation}  \label{min-LSQ}
\min_{\Theta\in\mathbb R^{\mathcal N}}\mathcal L(\Theta), \quad
\mathcal L(\Theta)={1\over N}\sum_{i=1}^N \mathcal L_i(\Theta),
\quad \mathcal L_i(\Theta)=
\|f(x_i,\Theta)-\bar x_i)\|_{F}^2.
\end{equation}
Here, $x_i \in \mathbb{R}^{n\times n}$ ($i=1:N$), often referred to as training
data, are images obtained by adding noise to the corresponding
``clean'' images $\bar x_i$.

An image can be viewed as a function defined on a uniform grid.  We
assume that this uniform grid can be coarsened in a uniform fashion,
as shown in Fig. \ref{fig:V-cycle}.  As in multigrid methods, we
designate these grids as multilevel.  Every grid
represents a different level.  This is in contrast with the concept of
``layers'' in CNN in the deep learning literature
\cite{goodfellow2017deep}.  In Fig. \ref{fig:V-cycle}, the number of
levels is $J=4$. A CNN involves operations among different levels of
grids together with layers of nonlinear operations within each level
of grid.

A key ingredient of an artificial neural network is the
activation function, which is always nonlinear.  A commonly used
activation function is ReLU defined as
\begin{equation}
\label{ReLU}
\sigma(t)=\max(0,t).
\end{equation}

\subsection{Operations from level to level}\label{ssc:level}
Given an image $x$, we first consider operations from fine to coarse
levels as follows:
\begin{itemize}
	\item Classical CNN model
\begin{equation}\label{multi-encode-classic}
f^{\ell} = F^{\ell}(R^{\ell}_{\ell -1} f^{\ell-1}), \quad \ell=2:J,
\end{equation}
	\item ResNet model
\begin{equation}\label{multi-encode}
f^{\ell} = R^{\ell}_{\ell -1} f^{\ell-1} + F^{\ell}(R^{\ell}_{\ell -1} f^{\ell-1}), \quad \ell=2:J,
\end{equation}
\end{itemize}
with $f^1 = F^1(x)$, $R^{\ell}_{\ell-1}: \mathbb R^{\hat
	c_{\ell-1}\times n_{\ell-1}\times n_{\ell-1}}\mapsto \mathbb
R^{c_\ell \times n_{\ell}\times n_{\ell}}$ is a restriction often
given by convolution with stride (or by pooling), and $F^\ell: \mathbb
R^{c_\ell \times n_{\ell}\times n_{\ell}}\mapsto \mathbb R^{\hat
	c_\ell \times n_{\ell}\times n_{\ell}}$ consists of a sequence of
convolution and nonlinear activation operations. \eqref{multi-encode}
is often referred as an encoding process by convolution.

Operations from coarse to fine levels can be defined similarly:
\begin{itemize}
	\item Classical CNN model
	\begin{equation}\label{multi-decode-classic}
	g^{\ell-1} = G^{\ell-1}(P^{\ell-1}_{\ell}g^{\ell}), \quad \ell=J:2,
	\end{equation}
	\item ResNet model
	\begin{equation}\label{multi-decode}
	g^{\ell-1} =P^{\ell-1}_{\ell}g^{\ell} + G^{\ell-1}(P^{\ell-1}_{\ell}g^{\ell}), \quad \ell=J:2,
	\end{equation}
\end{itemize}
with $g^{J} = f^J$.  Here, $P^{\ell-1}_{\ell}: \mathbb R^{c_{\ell}
	\times n_{\ell}\times n_{\ell}}\mapsto R^{\hat n_{\ell-1}\times
	n_{\ell-1}\times n_{\ell-1}}$ is a prolongation matrix of the same
size as the transpose of $R^{\ell}_{\ell-1}$ in \eqref{multi-encode}.
$G^{\ell-1}$ consists of a sequence of de-convolution and nonlinear
activations.  Further,  \eqref{multi-decode} is often known as a decoding
process by de-convolution.

\subsection{Multilayer operations within a given  level}\label{ssc:layer}

\subsubsection{Classical CNN model.}
For general CNN case, the $F^\ell$ and $G^\ell$ are defined for general CNN structure as: $F^\ell (y^{\ell,0}) = y^{\ell, m_\ell}$ with
\begin{equation}\label{eq:gcnnblock}
y^{\ell, j} = \sigma \circ \theta^{\ell,j}(y^{\ell,j-1}),
\quad j = 1:m_\ell,
\end{equation}
where $y^{\ell,0} =R^{\ell}_{\ell-1} f^{\ell-1}$. We can have a similar formula for $G^j$ as $y^{\ell,0} =
P^{\ell}_{\ell+1} g^{\ell+1}$.
Here,  one key point is that $\theta^{\ell,j}$
are convolutional operations in the following form:
\begin{equation}\label{eq:conv-1}
[\theta^{\ell,j}(y)]_{i} = \sum_k^{c^{\ell,j-1}}K^{\ell, j}_{i,k} \ast y_k + b^{\ell,j}_i \bm{1} \in \mathbb{R}^{n_\ell \times n_\ell}, \quad i = 1:c_{\ell,j},
\end{equation}
where $\ast$ denotes the standard convolution operation (see
\eqref{linear-restriction} and \eqref{convolution} for example), and
$c^{\ell,j}$ denotes the number of channels or multiple smoothed
(coarse) spaces in the $j$-th layer in level $\ell$.

\subsubsection{New ResNet model.}
We now describe a special CNN model using a special form of ResNet
developed in \cite{he2016deep,he2016identity};  that is,  we need to
specify how $F^\ell$ and $G^\ell$ are defined.  Specifically, we can
define $F^\ell (y^{\ell,0}) = y^{\ell, m_\ell} -  y^{\ell, 0}$, with
$y^{\ell,0} =R^{\ell}_{\ell-1} f^{\ell-1}$ and,
\begin{equation}\label{eq:resblock}
y^{\ell, j} = y^{\ell, j-1} + \mathcal{F}^{\ell, j}(y^{\ell,j-1}),
\quad j = 1:m_\ell.
\end{equation}
Here
\begin{equation}\label{eq:basic}
\mathcal{F}^{\ell,j}(y) = \xi^{\ell,j} \circ \sigma \circ \eta^{\ell,j}(y),
\end{equation}
as $\xi^{\ell,j}$ and $\eta^{\ell,j}$ are both general convolutions
with multi-channels. There are two observations for the basic structure \eqref{eq:basic}:
\begin{itemize}
	\item $ \xi \circ \sigma \circ \eta$ have the universal approximation for any continuous functions in compact sets for general fully connected deep neural networks.
	\item In the V-cycle method, especially for \eqref{eq:backslash} it is exact the case for \eqref{multi-encode} with $j=1$ for \eqref{eq:resblock}.
\end{itemize}

The same as general CNN models, we can have a similar formula for $G^j$ as $y^{\ell,0} =
P^{\ell}_{\ell+1} g^{\ell+1}$. This shows that, for this kind of model, we do residual for between both levels and layers as:
\medskip
\begin{table}[!htbp]
	\centering
	\footnotesize
	\begin{tabular}{|c|c|c|}
		\hline
		&level to level & layer to layer  \\
		\hline
		Iterative Formula  & $f^{\ell} = R^{\ell}_{\ell -1} f^{\ell-1} + F^{\ell}(R^{\ell}_{\ell -1} f^{\ell-1})$ & $y^{\ell, j} = y^{\ell, j-1} + \mathcal{F}^{\ell, j}(y^{\ell,j-1})$\\
%		\hline
%		Residual Form	 & $ F^{\ell}$ &$y^{\ell,1} = \mathcal{F}^{\ell, 1}(y^{\ell,0})$ \\
%		\hline
%		feature extractors &smoother (designed a priori) & convolution (trained) \\
%		\hline
%		grid transfer & restriction (designed a priori) & convolution with stride  (trained) \newline or pooling (given)\\
		\hline
	\end{tabular}
	\caption{\footnotesize A brief comparison between levels and layers.}
	\label{Level-Layes}
\end{table}

%So, now all the parameter as collected as
%\begin{equation}\label{para-level}
%	\Theta = \{\theta^1, \cdots, \theta^J, \theta^J, \theta^1\},
%\end{equation}
%as
%\begin{equation}\label{para-layer}
%	\theta^\ell = \{\eta^{\ell,0}, \xi^{\ell,0}, \cdots, \eta^{\ell,m_\ell}, \xi^{\ell,m_\ell}\}.
%\end{equation}

In comparing \eqref{multi-encode}-\eqref{multi-decode} for CNN with
\eqref{multi-backslash}-\eqref{eq:slash} for MG and the convolutions
\eqref{eq:conv-1} for CNN with the smoother \eqref{convolution} for
MG, we note that the CNN model described above is strikingly similar
to the structure of the V-cycle multigrid method as illustrated by
Fig.~\ref{fig:V-cycle}.

In our terminology, the number of layers in deep learning literature
is associated with the total number of applications of the activation
and is equal to $\sum_{\ell=1}^Jm_\ell$.  Within a given level, the
number of layers can be compared with the number of smoothings used in
multigrid methods.

%CNN used for purpose of de-noising has been studied extensively in the
%literature, including in
%\cite{NIPS2008_3506,bengio2013generalized,alain2014regularized,xie2012image,zhang2017beyond}.
%We also have performed some experiments of this kind by adding
%Gaussian noise to a MNIST dataset and then using the aforementioned CNN
%procedure for de-noising.  The results are remarkably good as observed
%in many existing studies referenced above, and,  as shown in
%Fig.~\ref{noise-denoise} in which the bottom images are the de-noised
%images of the ones above.

\section{Connections between multigrid method and CNN}
We now offer a comparison between geometric multigrid methods (GMG)
and convolutional neural networks (CNN).  It is this comparison
between these two distinctive methodologies that motivates much of the
research described in this proposal.  A brief summary of our comparison is
given in Table~\ref{MG-CNN}.
\medskip
\begin{table}[!htbp]
	\centering
	\footnotesize
	\begin{tabular}{|l|l|p{40mm}|c|}
		\hline
		&Multigrid &CNN  \\
		\hline
		goals  & linear system solver & image denoising or classification \\
		\hline
		structure	 &multilevel grids with multiple smoothings &multilevel grids with multiple convolutions and activations \\
		\hline
		feature extractors &smoother (designed a priori) & convolution (trained) \\
		\hline
		grid transfer & restriction (designed a priori) & convolution with stride  (trained) \newline or pooling (given)\\
		\hline
	\end{tabular}
	\caption{\footnotesize A brief comparison between MG and CNN.}
	\label{MG-CNN}
\end{table}

First we note that MG and CNN are both based on a hierarchy level of
``grids'' although the notion of a ``grid'' is not often discussed in
the CNN literature.   MG is a linear algorithm for solving linear system of equations
whereas CNN is a  nonlinear model designed for image de-noising or
classification.  On each level, MG uses a given method such as Jacobi or
Gauss-Seidel as a smoother, whereas CNN uses multiple ``channels'' of
convolutions.    The similarity here is that a smoother such as the damped
Jacobi can be interpreted as a convolution of the size $1\times 1$ or
$3\times 3$.  The difference between the two methods on this point is
that in MG the smoother is designed a priori
and usually only one smoother is used on each level, whereas in CNN,
the convolution is trained and multiple channels of convolutions are
used.   From fine to coarse (or vice-versa), restriction  matrices are used in both
methods, but again the difference remains: the restriction in MG is
designed a priori,  whereas in CNN the restriction is trained (as a
convolution with stride) or sometimes is given a priori (by an
operation called ``pooling'').

\subsection{V-cycle and U-net}
If we compare the V-cycle and new ResNet model, we may mention formula,
U-net \cite{ronneberger2015u} structure likes the V-cycle very much. Here we will give more comparison on those two structures.
%Let us show the U-net model with ResNet structure in mathematically.
\subsubsection{Segmentation problem}
Here we introduce some basic background about segmentation problem.....
\subsubsection{Classical U-net model}
The U-net was proposed firstly in \cite{ronneberger2015u} based on the classical CNN model as introduced in \S \ref{sc:cnn}. A mathematical expression of this model is:
\begin{itemize}
	\item Encode phase
	\begin{equation}\label{U-multi-encode-classic}
	f^{\ell} = F^{\ell}(R^{\ell}_{\ell -1} f^{\ell-1}), \quad \ell=2:J,
	\end{equation}
	for $f^1 = F^1(x)$ and
	$F^\ell (y^{\ell,0}) = y^{\ell, m_\ell}$ with
	\begin{equation}\label{u-eq:gcnnblock}
	y^{\ell, j} = \sigma \circ \theta^{\ell,j}(y^{\ell,j-1}),
	\quad j = 1:m_\ell,
	\end{equation}
	where $y^{\ell,0} =R^{\ell}_{\ell-1} f^{\ell-1}$.

	\item Decode phase
	\begin{equation}\label{U-multi-decode-classic}
	g^{\ell-1} = G^{\ell-1}([P^{\ell-1}_{\ell}g^{\ell},C^{\ell-1}f^{\ell-1}]), \quad \ell=J:2,
	\end{equation}
	for $g^{J} = f^J$ and $G^\ell (y^{\ell,0}) = y^{\ell, m_\ell}$ with
	\begin{equation}\label{eq:gcnnblock}
	y^{\ell, j} = \sigma \circ \theta^{\ell,j}(y^{\ell,j-1}),
	\quad j = 1:m_\ell,
	\end{equation}
	where $y^{\ell,0} = [P^{\ell-1}_{\ell}g^{\ell},C^{\ell-1}f^{\ell-1}]$. Here $[P^{\ell-1}_{\ell}g^{\ell},C^{\ell-1}f^{\ell-1}]$ means to collect the prolongated result $P^{\ell-1}_{\ell}g^{\ell}$ and the rescale cut for $f^{\ell-1}$ (the dimension reduction for segmentation results) as a new image with double size channels(One comes form $f$ one comes form $g$).
\end{itemize}

\subsubsection{Connections between V-cycle and U-net}
\begin{itemize}
	\item Structure of coarse and prolongation.
	\item Basic structure in universal approximation theory.
	\item PDE-based model expression.
\end{itemize}

\subsubsection{New U-net model motivated by V-cycle}
Now we proposed a new U-net model as:
\begin{itemize}
	\item Encode phase
	\begin{equation}\label{U-multi-encode}
	f^{\ell} = R^{\ell}_{\ell -1} f^{\ell-1} + F^{\ell}(R^{\ell}_{\ell -1} f^{\ell-1}), \quad \ell=2:J,
	\end{equation}
	for $f^1 = F^1(x)$.
	$F^\ell (y^{\ell,0}) = y^{\ell, m_\ell} -  y^{\ell, 0}$, with
	$y^{\ell,0} =R^{\ell}_{\ell-1} f^{\ell-1}$ and,
	\begin{equation}\label{u-eq:resblock}
	y^{\ell, j} = y^{\ell, j-1} + \mathcal{F}^{\ell, j}(y^{\ell,j-1}),
	\quad j = 1:m_\ell,
	\end{equation}
	for
	\begin{equation}\label{u-eq:basic-f}
	\mathcal{F}^{\ell,j}(y) = \xi^{\ell,j} \circ \sigma \circ \eta^{\ell,j}(y).
	\end{equation}

	\item Decode phase
	\begin{equation}\label{U-multi-decode}
	g^{\ell-1} = C^{\ell-1}f^{\ell-1} + P^{\ell-1}_{\ell}g^{\ell} + G^{\ell-1}(P^{\ell-1}_{\ell}g^{\ell}), \quad \ell=J:2,
	\end{equation}
	for $g^{J} = f^J$.
	$G^{\ell-1} (y^{\ell-1,0}) = y^{\ell-1, m_{\ell-1}} - y^{\ell-1,0}$ with
	$y^{\ell-1,0} = P^{\ell-1}_{\ell}g^{\ell}$
	\begin{equation}\label{u-eq:rcnnblock}
	y^{\ell-1, j} = y^{\ell-1, j-1} + \mathcal{G}^{\ell-1, j}(y^{\ell,j-1}),
	\quad j = 1:m_{\ell-1},
	\end{equation}
	for
	\begin{equation}\label{u-eq:basic-g}
	\mathcal{G}^{\ell-1,j}(y) = \phi^{\ell-1,j} \circ \sigma \circ \psi^{\ell-1,j}(y).
	\end{equation}
\end{itemize}

\section{Some general CNN structure motivated by V-cycle}
Here we will talk about more observations for V-cycle method writing as a ResNet model.


\section{3-D DP}

%%%%%%%%%%
\begin{frame}{3-D DP}
  \begin{exampleblock}{Floyd-Warshall algorithm}
    \begin{itemize}
      \item subproblem $\text{dist}[i,j,k]$: the length of the shortest path from $i$ to $j$ via only nodes $v_{1} \cdots v_{k}$
      \item goal: $\text{dist}[i,j,n], \forall i,j$
      \item question: Is $v_{k}$ in $\text{ShortestPath}[i,j,k]$?
      \item recurrence:
	\fignocaption{width = 0.40\textwidth}{fig/floyd-warshall.png}
	\[
	  \text{dist}[i,j,k] = \min \set{\text{dist}[i,j,k-1], \text{dist}[i,k,k-1] + \text{dist}[k,j,k-1]}
	\]
    \end{itemize}
  \end{exampleblock}
\end{frame}


\section{Approximation Error}
Although knowing these infinite Taylor series is nice, if we want to use them to find function values, we'll need to approximate.
This means only using some finite number of terms in the Taylor series for our approximation.
Because we're not using all the terms, we'll naturally have some truncation error.
We'd like to be able to say how big this truncation error is so we can be sure that our approximation is good enough.

\subsection{Alternating Series Estimation Theorem}
\begin{theorem}[Alternating Series Estimation Theorem]
	Let $s$ be a convergent, alternating (i.e the sign of each term alternates) series where the terms of $\abs{s}$ are strictly decreasing.
	Then the $n$th term truncation error is the same sign as and less than in absolute value the $(n+1)$th term.
\end{theorem}

\begin{example}
	Give a bound for the truncation error of using the first 10 terms of the Maclaurin series of $\ln{(1+x)}$ to approximate $\ln{2}$.
\end{example}
\begin{answer}
	The Maclaurin series for $\ln{(1+x)}$ is
	\begin{equation*}
		\ln{(1+x)} = x - \frac{x^2}{2} + \frac{x^3}{3} - \ldots = \sum_{k=0}^{\infty}{(-1)^k\frac{x^{k+1}}{k+1}}, \abs{x} \leq 1,
	\end{equation*}

	which is an alternating series.
	We see that to approximate $\ln{2}$, we'd use $x=1$, which is in the interval of convergence, so the series converges.
	The first missing term of the series is $1/11$.
	Let $s_{10}$ be the partial sum of the first 10 terms when $x=1$.
	By the Alternating Series Estimation Theorem,
	\begin{equation*}
		0 < \ln{2} - s_{10} < \frac{1}{11}.
	\end{equation*}
\end{answer}

\subsection{Taylor's Theorem}
Although the Alternating Series Estimation Theorem is useful for quickly bounding the error of alternating series, we'd like something that can apply more generally to all Taylor series.
\begin{theorem}[Taylor's Theorem]
	Let $f$ be a $k+1$ times differentiable function on an open interval $I$ containing $a$.
	Then for all $x$ in $I$,
	\begin{equation*}
		f(x) = f(a) + f^\prime(a)(x-a) + \frac{f^{\prime\prime}(a)}{2!}(x-a)^2 + \ldots + \frac{f^{(n)}(a)}{n!}(x-a)^n + R_n(x).
	\end{equation*}
	where
	\begin{equation*}
		\abs{R_n(x)} = \frac{\abs{f^{(n+1)}(c)}}{(n+1)!}\abs{x-a}^{n+1}
	\end{equation*}
	for some $c$ between $x$ and $a$.
\end{theorem}

We can use this theorem to find the maximum value of $\abs{R_n(x)}$ over some interval.
We can also be more precise about what it means for a Taylor series to converge to some function over some interval.
\begin{definition}
	Let $R_n(x)$ be the remainder of truncating after the degree $n$ term in the Taylor series for $f$ centered at $x=a$.
	If for all $x$ in some interval $I$ containing $a$,
	\begin{equation*}
		\lim_{n\to\infty}{R_n(x)} = 0,
	\end{equation*}
	then we say the Taylor series for $f$ at $x=a$ converges to $f$ on $I$.
\end{definition}

\begin{example}
	Show that the Maclaurin series for $\sin{x}$ converges to $\sin{x}$ for all real $x$.
\end{example}
\begin{answer}
	We need to find the remainder and show that in the limit it goes to 0 as $n$ grows large.
	By Taylor's Theorem,
	\begin{align*}
		\abs{R_n(x)} &= \frac{\abs{f^{(n+1)}(c)}}{(n+1)!}\abs{x-a}^{n+1} \\
		&\leq \frac{\abs{x}^{n+1}}{(n+1)!}.
	\end{align*}

	The numerator is an exponential function, while the denominator is a factorial function, so using $n^n$FEPL,
	\begin{align*}
		0 \leq \lim_{n\to\infty}{R_n(x)} &\leq \lim_{n\to\infty}{\frac{\abs{x}^{n+1}}{(n+1)!}} = 0. \\
		\lim_{n\to\infty}{R_n(x)} &= 0.
	\end{align*}

	So, the Maclaurin series converges to $\sin{x}$ for all real $x$.
\end{answer}

\subsubsection{Remainder Estimation Theorem}
Notice that we didn't have to actually find the value of $f^{(n+1)}(c)$.
We just had to find a suitable upper bound where the limit would still go to 0.
\begin{theorem}[Remainder Estimation Theorem]
	If there are positive constants $M$ and $r$ such that
	\begin{equation*}
		\abs{f^{(n+1)}(t)} \leq Mr^{(n+1)}
	\end{equation*}
	for all $t$ between $a$ and $x$, then $R_n(x)$ satisfies the inequality
	\begin{equation*}
		\abs{R_n(x)} \leq M\frac{r^{n+1}\abs{x-a}^{n+1}}{(n+1)!}.
	\end{equation*}
\end{theorem}

\begin{example}
	Give a maximum error bound for using $\ln{(1+x)} = x - x^2/2$ when $\abs{x} \leq 0.1$.
\end{example}
\begin{answer}
	We are using the second-order Taylor polynomial, so we need to find $R_2(x)$.
	By Taylor's Theorem,
	\begin{align*}
		\abs{R_2(x)} &= \frac{\abs{f^{(2+1)}(c)}}{(2+1)!}\abs{x-0}^{2+1}  = \frac{\abs{f^{(3)}(c)}}{3!}\abs{x}^3\\
		f^{(3)}(x) &= \dd{^3}{x^3}\ln{(1+x)} = \frac{2}{(1+x)^3} \\
	\end{align*}

	On $-0.1 \leq x \leq 0.1$, $\abs{f^{(3)}(x)}$ is maximal at $(-0.1, \frac{2000}{729})$.
	\begin{equation*}
		\abs{R_2(x)} \leq \frac{\frac{2000}{729}}{6}\abs{x}^3.
	\end{equation*}

	On $-0.1 \leq x \leq 0.1$, $\abs{x}^3$ is maximal at $(-0.1, \frac{1}{1000})$.
	\begin{equation*}
		\abs{R_2(x)} \leq \frac{\frac{2000}{729}}{6}\frac{1}{1000} = \frac{1}{2187} \approx 4.572 \times 10^{-4}.
	\end{equation*}

	So, the error of approximating $\ln{(1+x)}$ with $x-x^2/2$ when $-0.1 \leq x \leq 0.1$ is at most $4.572 \times 10^{-4}$.
\end{answer}


\subsection{Curvature}
Curvature is 1 divided by the radius of the circle that best approximates the curve at a point.
Tighter turns have smaller radii and higher curvature.
We can use $\hat{T}$ to find the curvature at a point on $\vec{r}(t)$.
\begin{equation*}
	\kappa(t) = \norm{\dd{\hat{T}}{s}} = \norm{\dd{\hat{T}}{t}\left(\dd{s}{t}\right)^{-1}} = \norm{\dd{\hat{T}}{t}}\frac{1}{v(t)}
\end{equation*}

\noindent
For example, let's find $\kappa(t)$ for the circle in the yz-plane: $\vec{r}(t)=\langle 7, R\sin{t}, R\cos{t} \rangle$.
\begin{align*}
	\vec{r^\prime}(t) &= \langle 0, R\cos{t}, -R\sin{t}\rangle
	\text{ and }
	v(t) = \sqrt{0^2+(R\cos{t})^2+(-R\sin{t})^2} = R \\
	\hat{T}(t) &= \frac{1}{R}\vec{r}(t) = \langle0,\cos{t},-\sin{t}\rangle \\
	\frac{\mathrm{d}\hat{T}}{\mathrm{d}t} &= \langle 0,\sin{t},-\cos{t}\rangle \\
	\norm{\frac{\mathrm{d}\hat{T}}{\mathrm{d}t}} &= 1 \\
	\kappa(t) &= \frac{1}{R}.
\end{align*}
This relationship is true for all circles.


\subsection{The pseudo-inverse}

The definition of the inverse  is that:

\(MM^{-1}=I\)

\(M^{-1}M=I\)

We also have:

\(MM^{-1}M=M\)

\(M^{-1}MM^{-1}=M^{-1}\)

\subsubsection{The inverse of a homomorphism}

Generally we don't have inverses of homomorphisms as the number of dimensions are different.

We can, however, find a matrix \(M^+\) which satisfies:

\(MM^+M=M\)

\(M^+MM^+=M^+\)

This is the pseudo-inverse.


\subsection{The Gradient \& C-Level Curves}
\noindent
Let $\vec{r}(t)$ be the C-level curve of $f(x, y)$.
\begin{align*}
f\circ\vec{r} &= C  \text{ and } \frac{\mathrm{d}}{\mathrm{d}t}(f\circ\vec{r}) = 0 \\
	&\implies \frac{\mathrm{d}}{\mathrm{d}t}(f\circ\vec{r}) = \nabla f\cdot\vec{r^\prime}(t) = 0 \\
	&\implies \nabla f\perp\vec{r^\prime}(t) \\
	&\implies \nabla f \text{ is perpendicular to the C-level curve of } f.
\end{align*}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.33\textwidth]{./Images/differentialMultivariableCalculus/grad_c_level.png}
	\caption{The gradient is perpendicular to C-level curves.}
\end{figure}


\subsection{Osculating Plane/Circle \& B-Hat $\left(\hat{B}\right)$}
\noindent
The osculating plane in the plane containing $\vec{r}(t)$, $\hat{T}$ and $\hat{N}$. It is only defined when $\hat{N}\neq 0$. This means straight lines do not have an osculating plane.
The osculating circle lives in the osculating plane, is centered at $\vec{r}(t) + \frac{\hat{N}(t)}{\kappa(t)}$, and has radius $\frac{1}{\kappa(t)}$.
The tangent line at $\vec{r}(t)$ is also tangent to the osculating circle because both points have the same curvature.

\begin{figure}[H]
	\label{osculating_plane_circle}
	\centering
	\includegraphics[width = 0.33\textwidth]{./vectorValuedFunctions/osculatingCircle.png}
	\includegraphics[width = 0.33\textwidth]{./vectorValuedFunctions/osculatingPlane.png}
	\caption{Osculating circle and plane}
\end{figure}

\noindent
The vector that is normal to the plane is
\begin{equation*}
	\hat{B}(t) = \hat{T}\times\hat{N},
\end{equation*}
which is called the ``binormal'' vector because it is perpendicular to both $\hat{T}$ and $\hat{N}$.
Together, $\hat{T}$, $\hat{N}$, and $\hat{B}$ form the Frenet Serret Frame, also called the TNB frame.

\begin{figure}[H]
	\label{osculating_plane_circle}
	\centering
	\includegraphics[width = 0.33\textwidth]{./vectorValuedFunctions/TNB1.png}
	\includegraphics[width = 0.5\textwidth]{./vectorValuedFunctions/TNB2.png}
	\caption{TNB frame}
\end{figure}

\noindent
We can write the equation for the osculating plane as
\begin{equation*}
	\hat{B}(t)\cdot\left(\langle x,y,z\rangle - \vec{r}(t)\right) = 0.
\end{equation*}


\documentclass{article}
\usepackage{amsmath}

\newcommand{\flow}{\rightleftharpoons}

\begin{document}
\title{Typesetting Mathematics in \LaTeX}
\author{Dr. Milaan Parmar \\ milaanparmar9@gmail.com}
\maketitle

\section{Introduction}
\LaTeX\ is extremely powerful when it comes to typesetting mathematics. It's one of the core strengths of this system.

\section{Displaying Mathematics}
There are two ways of displaying maths. One is \emph{inline} and the other is \emph{display} format -- in which the whole math sits on its own set of lines.

\subsection{Inline Mode}
We are going to insert a mathematics equation inline here using a pair of \$ signs:  $E=mc^2$    . As you can see, the display (such as line spacing) does not get messed up by the mathematics as it does with word processing softwares.

\subsection{Display Mode}
We can also display equations in their own set of lines. To do this, we can use the equation environment.

\begin{equation}\label{eq:emc}
E=mc^2
\end{equation}

As you can see, \LaTeX\ inserts the equation number automatically. We can refer to it using the \verb|\ref| command just as we referred to sections, figures and tables. (E.g. Equation~\ref{eq:emc}.) To get rid of the equation number, simply use the \emph{star variant} of the equation environment. (For this, you need the \texttt{amsmath} package.)

\begin{equation*}
E=mc^2
\end{equation*}

Alternatively, we can use the shorthand keys \verb|\[| and \verb|\]|

\[
E=mc^2
\]

\section{Mathematical Features}
\LaTeX\ has many builtin features and you can get many more easily. Here, we'll see some of these features:

Addition, subtraction, multiplication and division:

\[
x+2 - 25 \times 35 \div 98
\]

Superscripts and subscripts:

\[ x^2  \]
\[ x_{(i)} \]

Summation, union, intersection, big-union, integral:

\[ \sum_{i=1}^{n}{i^2} \]
\[ x \cup y \cap z \]
\[ \bigcup_{i=1}^{n}{x_i} \]
\[ \int_0^n{x^2} \]

Fractions, brackets, square root:

\[ \frac{x}{y} \]
\[ \frac{\sum_i{x^2}}{\int_0^n{x^2}} \]
\[ \sqrt{\frac{\sqrt{36}} {x^5}} \]

\[ 2 \times \left( \frac{34}{\frac{124}{356}}    \right)  \]

Greek letters:

\[
\alpha + \beta + \gamma^* + \Sigma + \Theta + 2_\epsilon
\]

Matrices and vectors. For this, you need to include the \texttt{amsmath} package and then use the \texttt{bmatrix} or \texttt{pmatrix} environment:

\[
\begin{pmatrix}
\frac{a}{44} & b \\
c & \sqrt{d}
\end{pmatrix}
\]

Accents:

\[ \hat{x} \]
\[ \hat{\imath} \]
\[ \dot{x} \]

See the \texttt{Math} menu in the IDE for other operations. You can refer to ``Short Math Guide for \LaTeX'' for a lot more examples.

\section{Using Symbols}
You might come across situations where you need to find new symbols. For this, you can refer to the ``The Comprehensive \LaTeX Symbols List''.

\[ x \rightleftharpoons  y \]

(Optional) Since this is a long command, we might want to create a shortcut using the \verb|\newcommand| command in the preamble. This also allows us to later change the symbol without having to change the equations.

\[ x \flow y \]

\end{document}


\chapter{Search}

\section{Binary Search}
\runinhead{Variants:}
\begin{enumerate}
\item get the idx equal or just lower (floor)
\item get the idx equal or just higher (ceil)
\item \pyinline{bisect_left}
\item \pyinline{bisect_right}
\end{enumerate}
\subsection{idx equal or just lower}
Binary search, get the idx of the element equal to or just lower than the target. The returned idx is the $A_{idx} \leq target$. It is possible to return $-1$. It is different from the \pyinline{bisect_lect}.

\runinhead{Core clues:}
\begin{enumerate}
\item To get ``equal'', \pyinline{return mid}.
\item To get ``just lower'', \pyinline{return lo-1}.
\end{enumerate}
$A_{idx} \leq target$.
\begin{python}
def bin_search(self, A, t, lo=0, hi=None):
    if hi is None: hi = len(A)

    while lo < hi:
        mid = (lo+hi)/2
        if A[mid] == t:  return mid
        elif A[mid] < t: lo = mid+1
        else:            hi = mid

    return lo-1
\end{python}
\subsection{idx equal or just higher}
$A_{idx} \geq target$.
\begin{python}
def bin_search(self, A, t, lo=0, hi=None):
    if hi is None: hi = len(A)

    while lo < hi:
        mid = (lo+hi)/2
        if A[mid] == t:  return mid
        elif A[mid] < t: lo = mid+1
        else:            hi = mid

    return lo
\end{python}
\subsection{bisect\_left}
Return the index where to insert item x in list A. So if t already appears in the list,
A.insert(t) will insert just before the \textit{leftmost} t already there.
\runinhead{Core clues:}
\begin{enumerate}
\item Move \pyinline{lo} if $A_{mid} < t$
\item Move \pyinline{hi} if $A_{mid} \geq t$
\end{enumerate}

\begin{python}
def bisect_left(A, t, lo=0, hi=None):
    if hi is None: hi = len(A)

    while lo < hi:
        mid = (lo+hi)/2
        if A[mid] < t: lo = mid+1
        else:          hi = mid

    return lo
\end{python}

\subsection{bisect\_right}
Return the index where to insert item x in list A. So if t already appears in the list, A.insert(t) will insert just after the \textit{rightmost} x already there.
\runinhead{Core clues:}
\begin{enumerate}
\item Move \pyinline{lo} if $A_{mid} \leq t$
\item Move \pyinline{hi} if $A_{mid} > t$
\end{enumerate}
\begin{python}
def bisect_right(A, t, lo=0, hi=None):
    if hi is None: hi = len(A)

    while lo < hi:
        mid = (lo+hi)/2
        if A[mid] <= t: lo = mid+1
        else:           hi = mid

    return lo
\end{python}
\section{Applications}
\subsection{Rotation}
\runinhead{Find Minimum in Rotated Sorted Array.} Three cases to consider:
\begin{enumerate}
\item Monotonous
\item Trough
\item Peak
\end{enumerate}

If the elements can be duplicated, need to detect and skip.
\begin{python}
def findMin(self, A):
    lo = 0
    hi = len(A)
    mini = sys.maxint
    while lo < hi:
        mid = (lo+hi)/2
        mini = min(mini, A[mid])
        if A[lo] == A[mid]:  # JUMP
            lo += 1
        elif A[lo] < A[mid] <= A[hi-1]:
            return min(mini, A[lo])
        elif A[lo] > A[mid] <= A[hi-1]:  # trough
            hi = mid
        else:  # peak
            lo = mid+1

    return mini
\end{python}
\section{Combinations}
\subsection{Extreme-value problems}\label{extremeValueProblem}
\runinhead{Longest increasing subsequence.} Array $A$.

Clues:
\begin{enumerate}
\item \pyinline{MIN}: \textit{min} of index \textit{last} value of LIS of a particular \textit{len}.
\item \pyinline{RET}: result table, store the $\pi$'s idx (predecessor); (optional, to build the LIS, no need if only needs to return the length of LIS)
\item \pyinline{bin_search}: For each currently scanning index \pyinline{i}, if it smaller (i.e. $\neg$ increasing), to maintain the \pyinline{MIN}, binary search to find the position to update the min value. The \pyinline{bin_search} need to find the element $\geq$ to \pyinline{A[i]}.
\end{enumerate}
\newpage
\begin{python}
def LIS(self, A):
    n = len(A)
    MIN = [-1 for _ in xrange(n+1)]

    l = 1
    for i in xrange(1, n):
        if A[i] > A[MIN[l]]:
            l += 1
            MIN[l] = i
        else:
            j = self.bin_search(MIN, A, A[i], 1, l+1)
            MIN[j] = i

    return l
\end{python}
If need to return the LIS itself.
\begin{python}
    for i in xrange(1, n):
        if A[i] > A[MIN[l]]:
            l += 1
            MIN[l] = i

            RET[i] = MIN[l-1]  # (RET)
        else:
            j = self.bin_search(MIN, A, A[i], 1, l+1)
            MIN[j] = i

            RET[i] = MIN[j-1] if j-1 >= 1 else -1  # (RET)

    # build the LIS (RET)
    cur = MIN[l]
    ret = []
    while True:
        ret.append(A[cur])
        if RET[cur] == -1: break
        cur = RET[cur]

    ret = ret[::-1]
    print ret
\end{python}

\section{High dimensional search}
\subsection{2D}
\runinhead{2D search matrix I.} $m\times n$ mat. Integers in each row are sorted from left to right. The first integer of each row is greater than the last integer of the previous row.
$$
\begin{bmatrix}
1 & 3 & 5 & 7 \\
10 & 11 & 16 & 20 \\
23 & 30 & 34 & 50 \\
\end{bmatrix}
$$

Row column search: starting at top right corner: $O(m+n)$.

Binary search: search rows and then search columns: $O(\log m + \log n)$.

\runinhead{2D search matrix II.} $m\times n$ mat. Integers in each row are sorted from
left to right. Integers in each column are sorted in ascending from top to bottom.
$$
\begin{bmatrix}
1&   4&  7& 11& 15 \\
2&   5&  8& 12& 19 \\
3&   6&  9& 16& 22 \\
10& 13& 14& 17& 24 \\
18& 21& 23& 26& 30 \\
\end{bmatrix}
$$

Row column search: starting at top right corner: $O(m+n)$.

Binary search: search rows and then search columns, but upper bound row and lower bound row:

$$O\big(\min(n\log m, m\log n)\big)$$