%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Probability and Math Revision}
\label{ap:revision-probability}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Random variable}
We call random variable X a measurable real-valued  function of possible outcomes ($\Omega$ ) defined on a sample space($E$).

\begin{equation}
	X: \Omega \rightarrow E
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Probability Density Function (PDF)}

Variable X  is a continuous random variable if if there is a function $f (x) $, that satisfies for a set $B = \{b \in \mathbb{R} | b_ {1} \leq b \leq b_ {2}\} $, defined for all $x = \{x \in \mathbb{R}| - \infty \leq x \leq + \infty\} $, having the property:
\begin{equation}
	P(X \in B) = \int_{b_{1}}^{b_{2}}f(x) dx
\end{equation}

Where $P$ is the probability function of the random variable $x$. $f(x)$ is called probability density function of the random variable $X$\cite{ross-probability}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Cumulative Distribution Function (CDF)}

The Cumulative Distribution Function of a real-valued random variable $X$, is a function $F (x) $ defined by\cite{ross-probability}:

\begin{equation}
F(x) = P(X \leq x) = \int_{- \infty}^{x}f(x) dx
\end{equation}

Where $f(x)$ is the probability density function (PDF) of $X$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Expected value, Mean, Variance and Standard Deviation}

Let $X$ be a constinuous real-valued random variable, and $f(x)$ be its  probability density function (PDF). Then the expected value of $X$ is defined by:

\begin{equation}
E[X] =  \int_{- \infty}^{+ \infty}xf(x) dx
\end{equation}

For a random variable normally distributed $X_{normal}$ the result of this definition is equals to its mean $\mu$ of the distribution.

\begin{equation}
E[X_{normal}] = \mu
\end{equation}

For an exponential distribution is equals to the inverse of its rate:

\begin{equation}
E[X_{exponential}] = \frac{1}{\lambda}
\end{equation}

The variance of a random variable $X$, denoted by $Var(X)$, is defined by:

\begin{equation}
var(X) =  E[X^2] - (E[x])^{2}
\end{equation}

For a random variable $X$ normally distributed, the variance is equal to its standard deviation\cite{ross-probability}:

\begin{equation}
var(X) = \sigma^{2}
\end{equation}

For a finite data $X = \{ x_1, x_2, ..., x_n \}$set we can estimate the mean and standard deviation using the follow equations:

\begin{equation}
\mu = \frac{1}{n}\sum_{n}^{i=1}X_i
\end{equation}

\begin{equation}
\sigma = \sqrt{\frac{1}{n}[(x_1 - \mu)^2 + (x_2 - \mu)^2 + ... + (x_n - \mu)^2]]}
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Stochastic Process}

A stochastic process of a random variable represented by $\{X(t)| t \in T\}$ is a collection of random variables. Since $t$ is often interpreted as time, $X (t) $ is usually referred as the state of the process at a given time $t$\cite{ross-probability}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Correlation (Pearson correlation coefficient)}

Letting $(X,Y)$ be a pai of real-valued random variables, the covariance is defined by:

\begin{equation}
cov(X, Y) = E[(X-E[X])(Y - E[Y])]
\end{equation}

And the Pearson's correlation coefficient is defined by:

\begin{equation}
cor(X, Y) = \frac{cov(X, Y)}{\sigma_X \sigma_Y}
\end{equation}

Where:

\begin{equation}
\sigma_X = \sqrt{  E[X^2] - E[x]^{2} }
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Autocorrelation of a finite time series}
The autocorrelation function measures the correlation between data samples $y_{t}$ and $y_{t + k}$, where $k =0, ..., K$, and the data sample $\{y\}$  is generated by a stochastic process.

According to \cite{book-time-series-analysis}, the autocorrelation for a lag $k$ is:

\begin{equation}
r_{k} = \frac{c_{k}}{c_{0}}
\end{equation}

where

\begin{equation}
c_{k} = \frac{1}{T - 1}\sum_{t = 1}^{T - k} (y_{t} - \bar{y})(y_{t+k} - \bar{y})
\end{equation}

and $c_{0}$ is the sample variance of the time series.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Self-similarity}

A self similar object has the property of looking "roughly" the same at any scale. Self-similar objects are described by the power law:

\begin{equation}
	N = s^{d}
\end{equation}

where
\begin{equation}
	d = \frac{\ln{N}}{\ln{s}}
\end{equation}

is the dimension of the scaling law, called  Hausdorff dimension\cite{web-self-similar}.

\begin{figure*}[ht!]
	\centering
	\includegraphics[height=2.5in]{figures/apA/tritrans}
	\caption{This is a classical example of a self-similar figure, caled Sierpinski triangle.}
	\label{fig:self-similar-figure-example}
\end{figure*}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Hurst Exponent}

For a time-series $X = \{ X_1, X_2, ..., X_n \}$, letting $m$ be the time series mean:

\begin{equation}
\mu = \frac{1}{n}\sum_{n}^{i=1}X_i
\end{equation}

We can calculate the adjusted series $Y$ by:

\begin{equation}
Y = \{ Y_t \} = \{ X_t - \mu \}
\end{equation}

for $t = 1, 2, ..., n $. We can calculate the  cumulative deviate series $Z$ by:

\begin{equation}
Z_t = \sum_{t}^{1}Y_i, \quad t = 1, 2, ..., n
\end{equation}

We can than calculate the time series range by:

\begin{equation}
R(n) = max(Z_1, Z_2, ..., Z_n) - min(Z_1, Z_2, ..., Z_n)
\end{equation}

And its standard deviation by:

\begin{equation}
S(n) = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(X_i -  \mu)^{2}}
\end{equation}

Letting $E[x]$ be the expected value of a real-valued random variable  $X$, and $C$ a constant, the Hurst Exponent $H$ is defined by\cite{hurst-def}:

\begin{equation}
E\left [  \frac{R(n)}{S(n)} \right ] = Cn^{H}, \quad n \to \infty
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Heavy-tailed distributions}

Heavy tailed distributions are probability distributions whose tails are not exponentially bounded.  A distribution of of a real-valued random variable $X$, with cumulative distribution F(x), is said to be heavy tailed, if it satisfies this condition for all $\lambda \in \mathbb{R}$:

\begin{equation}
	\lim_{x\to\infty} e^{\lambda x} (1 - F(x)) = \infty
\end{equation}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{\textit{QQplot} analysis}

QQplot is used to test if two data-sets comes from a common distribution\cite{web-qqplot}. In our study case, we used to compare empirical data with theoretical given by model approximations. We show down below the image presented in chapter~\ref{ch:literature-review} for reference.
Looking on how the dot plot behaves compared to the linear line, we can see how well the theoretical plot (the estimated data, on the horizontal axis) represents the actual values (sample data, vertical axis):

\begin{itemize}
\item \textbf{Light-tailed}: the samples still hold a slight heavy-tail effect compared to the estimated by the theoretical values.
\item \textbf{Heavy-tailed}: the samples have a predominant heavy-tail effect compared to the estimated by the theoretical values.
\item \textbf{Linear}: the samples match the theoretical values.
\item \textbf{Bimodal}: samples present a bimodal pattern.
\item \textbf{Left skew}: small values are under-represented by the model(~\ref{fig:qqplot-rl-skew}).
\item \textbf{Right skew}: larger values are under-represented by the model(~\ref{fig:qqplot-rl-skew}).
\end{itemize}

As an example, we created a \textit{QQplot} (figure ~\ref{fig:qq-cauchy}), where we used as samples randomly generated data, generated by a Cauchy, and theoretical values, normal random data. Comparing with the figure ~\ref{fig:qqplot-tutorial-ap}, we can identify a heavy tail behavior on the sample data.

\begin{figure*}[ht!]
	\centering
	\includegraphics[width=0.55\textwidth]{figures/ch2/qqplot-tutorial}
	\caption{How information about data samples can be extracted from \textit{QQplots}. Depending on the shape of the dot plot,  }
	\label{fig:qqplot-tutorial-ap}
\end{figure*}

\begin{figure*}[ht!]
	\centering
	\includegraphics[width=0.5\textwidth]{figures/apA/rl-skew}
	\caption{Shape of a distribution with right and left skew.}
	\label{fig:qqplot-rl-skew}
\end{figure*}

\begin{figure}[t!]
    \centering
    %\subfloat[]{
        \includegraphics[width=60mm]{figures/apA/qq-c1}
    %    \label{fig:qq-cauchy-1}
    %}
    \hspace{0mm}
    %\subfloat[]{
        \includegraphics[width=60mm]{figures/apA/qq-c2}
    %    \label{fig:qq-cauchy-2}
    %}
    \caption{QQplot of randomly generated data of a Cauchy process as samples and a normal process as theoretical. We can identify a heavy-tail behavior on the samples, compared to the theoretical.}
    \label{fig:qq-cauchy}
\end{figure}

To generate the plots in figure~\ref{fig:qq-cauchy}, we used Python and the libraries matplotlib and numpy. The code used is shown down below:

\begin{minted}{python}
import numpy as np
import matplotlib.pyplot as plt

nn = sorted(np.random.standard_normal(30))
cc = sorted(np.random.standard_cauchy(30))
nn_max = max(nn)
nn_min = min(nn)
cc_max = max(cc)
cc_min = min(cc)
yy = np.linspace(cc_min, cc_max, num=10)
xx = np.linspace(nn_min, nn_max, num=10)
fig, ax = plt.subplots()
ax.plot(nn, cc, 'bo', markersize=10.0)
ax.plot(xx, yy, 'r-', linewidth=4.0)
plt.xlabel('estimated (normal data)')
plt.ylabel('samples (cauchy data)')
plt.tick_params(labelsize=14)
plt.tight_layout()
plt.show()
\end{minted}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Akaike information criterion (AIC) and Bayesian information criterion (BIC)}

Suppose that we have an statistical model $M$ of some dataset ${\boldsymbol{x} = \{x_1, ..., x_n}\}$, with $n$ independent and identically distributed observations of a random variable $X$. This model can be expressed by a PDF $f(x| \boldsymbol{\theta})$, where $\boldsymbol{\theta}$ a vector of parameter of the PDF, $\boldsymbol{\theta} \in \mathbb{R}^{k}$ ($k$ is the number of parameters). The  likelihood function  of this model $M$ is given by:

\begin{equation}
L(\boldsymbol{\theta}|\boldsymbol{x} ) =  f(x_1|\boldsymbol{\theta})\cdot...\cdot f(x_n|\boldsymbol{\theta}) = \prod_{i = 1}^{n}f(x_i|\boldsymbol{\theta})
\end{equation}

Now, suppose we are trying to estimate the best statistical model, from a set ${M_1, ..., M_n}$, each one whit an estimated vector of parameters  ${\boldsymbol{\hat{\theta_1}}}, ..., {\boldsymbol{\hat{\theta_n}}}$. $AIC$ and $BIC$ are defined by:

\begin{equation}
\label{eq:aic-ap}
AIC = 2k - \ln(L(\boldsymbol{\hat{\theta}}|\boldsymbol{x}))
\end{equation}

\begin{equation}
\label{eq:bic-ap}
BIC = k\ln(n) - \ln(L(\boldsymbol{\hat{\theta}}|\boldsymbol{x}))
\end{equation}

In both cases, the preferred model $M_i$, is the one with the smaller value of $AIC_i$ or $BIC_i$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Gradient Descendent Algorithm}

Given a linear hypothesis for a dataset:

\begin{equation}
	\boldsymbol{h_{\theta}} = \boldsymbol{\theta}^{T}\boldsymbol{x}
\end{equation}

were $\boldsymbol{h_{\theta}}, \boldsymbol{\theta}, \boldsymbol{x} \in \mathbb{R}^{m}$. If $m = 2$ we will just have a simple linear equation $h_{\theta}(x) = \theta_{0} + \theta_{1}x$.

The goal of the gradient descendent is to minimize the cost function $J_\nabla(\boldsymbol{\theta})$, defined by:

\begin{equation}
	J_\nabla(\boldsymbol{\theta}) = \frac{1}{2m} \sum_{i = 1}^{m}  ( \boldsymbol{h_{\theta}}(x^{(i)} - y^{(i)} )^{2}
\end{equation}

To do this, we initialize a $\boldsymbol{\theta_{j}}$ vector (usually with zeros), and repeat this procedure, until $\boldsymbol{\theta_{j}}$ converges:

\begin{equation}
	\boldsymbol{\theta_{j + 1}} := \boldsymbol{\theta_{j}} - \alpha \frac{1}{m} \sum_{i = 1}^{m}  ( \boldsymbol{h_{\theta}}(x^{(i)} - y^{(i)} )x_{j}^{i}
\end{equation}

where $\alpha$ is the step value, typically a small positive number. All values of $\boldsymbol{\theta_{j}}$ must be updated simultaneously\cite{web-coursera}.


\lab{The Arnoldi Iteration}{The Arnoldi Iteration}
\label{lab:kry_arnoldi}

\objective{Use Krylov subspaces to find eigenvalues of extremely large matrices.}

One of the biggest difficulties in computational linear algebra is the amount of memory needed to store a large matrix and the amount of time needed to read its entries.
Methods using Krylov subspaces avoid this difficulty by studying how a matrix acts on vectors, making it unnecessary in many cases to create the matrix itself.

The \emph{Arnoldi iteration} is an algorithm for finding an orthonormal basis of a Krylov subspace.
One of its strengths is it can run on any linear operator without knowing the operator's underlying matrix representation.
The outputs of the Arnoldi algorithm can be used to approximate the eigenvalues of the matrix of the linear operator.

\section*{Krylov Subspaces} % =================================================

The order-$N$ Krylov subspace of $A$ generated by $\x$ is
\[
\mathcal{K}_n(A, \x) =\text{span} \{\x, A\x, A^2\x, \ldots, A^{n-1}\x\}.
\]
If the vectors $\{\x, A\x, A^2\x, \ldots, A^{n-1}\x\}$ are linearly independent, then they form a basis for $\mathcal{K}_n(A,\x)$.
However, this basis is usually far from orthogonal, and hence computations using this basis will likely be ill-conditioned.

\section*{The Arnoldi Iteration Algorithm} % ==================================

One way to find an orthonormal basis for $\mathcal{K}_n(A,\x)$ is to use the modified Gram-Schmidt algorithm from Lab \ref{lab:QRdecomp} on the set $\{\x, A\x, A^2\x, \ldots, A^{n-1}\x\}$.
The Arnold iteration does this more efficiently by integrating the creation of $\{\x, A\x, A^2\x, \ldots, A^{n-1}\x\}$ with the modified Gram-Schmidt algorithm. It returns an orthonormal basis for $\mathcal{K}_n(A,\x)$. This algorithm is described in Algorithm \ref{alg:arnoldi_iteration}.

\begin{comment}
Before discussing the specific uses of the Krylov subspace in linear systems and eigenvalue problems, let us address a very
practical concern: how can we best compute a basis for the Krylov subspace? The obvious answer is simply to
calculate the vectors $x, Ax, A^2x, \ldots, A^{N-1} x$, which we can accomplish using only matrix-vector multiplication.
Straightforward though this may be, there is a major problem: $A^n x$ tends to converge to a dominant eigenvector of $A$
as $n$ gets large, and consequently these vectors become nearly parallel. Thus, the basis $\{x, Ax, A^2x, \ldots, A^{N-1} x\}$
is far from orthogonal, and matrix computations associated with this basis will likely be ill-conditioned and prone to
numerical instability. To redress this problem, we may think to apply the Gram-Schmidt orthogonalization process to the
basis, obtaining an orthonormal basis for the Krylov subspace that enjoys much better numerical properties. This turns out to
be a useful thought, and is the basis for the Arnoldi iteration.

You may recall from lab \ref{lab:QRdecomp} that the Modified Gram-Schmidt algorithm allows us to find an increasing number of orthogonal
vectors but does not require that we run the algorithm to its completion to find a full basis.
Our goal is to find an orthonormal set of vectors $q_1,\ldots,q_N$ having the same span as $x, Ax, A^2x, \ldots, A^{N-1} x$.
We start
 things off by setting
\[
q_1 = \frac{x}{\|x\|_2}.
\]
Now, assuming we have obtained $q_1,\ldots,q_n$, we obtain $q_{n+1}$ by projecting $A q_n$ onto the previous vectors,
subtracting out these projections, and then normalizing. To make this more precise, let $h_{i,n} = \langle q_i, A q_n\rangle$
for $i = 1,\ldots, n$. Subtract out these projections by calculating
\[
p_{n+1} = A^n x - \sum_{i=1}^n h_{i,n}q_i.
\]
Define $h_{n+1,n} = \|p_{n+1}\|_2$, and normalize $p_{n+1}$ by calculating
\[
q_{n+1} = \frac{p_{n+1}}{h_{n+1,n}},
\]
our next basis vector. This procedure is outlined (in slightly more Python-friendly notation) in Algorithm \ref{alg:arnoldi_iteration}.

Perhaps you noticed a slight discrepancy between the Arnoldi iteration as described above, and the usual Gram-Schmidt procedure.
Specifically, you might have expected to compute $q_{n+1}$ by projecting $A^n x$, rather than $A q_n$, onto the previous vectors.
Thankfully, it is straightforward to show that our algorithm produces a valid orthonormal basis for the Krylov subspace,
despite this difference. And because of this detail, we do not need to compute and store the original Krylov basis
$x, Ax, \ldots, A^{N-1}x$.
Additionally, each iteration only requires one matrix-vector calculation, and the individual entries in $A$ are never
referenced or modified. Thus, even if the matrix $A$ is very large in theory, as long as we have a reasonably efficient
subroutine to calculate $Ax$ for any vector $x$, the Arnoldi iteration is computationally tractable.

This algorithm produces an orthonormal basis $q_1,\ldots,q_N$ for the order-$N$ Krylov subspace generated by $A$ and $x$, as well
as a collection of numbers $h_{i,j}$. If we define a matrix $H_N$ whose $i,j$'th entry is $h_{i,j}$ for $i \leq j+1$ and
is $0$ otherwise, we now have an upper Hessenberg matrix.
Recall that an upper Hessenberg matrix has the property that all entries below the first subdiagonal
are equal to zero. Any square matrix is unitarily similar to an upper Hessenberg matrix. Dealing with a Hessenberg matrix is
often more convenient than dealing with a general matrix, especially when it comes to finding eigenvalues or solving systems
of equations, since efficient algorithms designed for these types of matrices exist.
It turns out that there is a Hessenberg factorization of $A$, given by
\[
A  = QHQ^*,
\]
where $Q$ is a unitary matrix and $H$ is upper Hessenberg such that the first $N$ columns of $Q$ are $q_1,\ldots,q_N$, and
the upper left $N \times N$ submatrix of $H$ is equal to $H_N$. Hence, the Arnoldi iteration provides a connection between
the Krylov subspace and the Hessenberg factorization of a matrix. Each step in the Arnoldi iteration can be thought of as computing
another step in the Hessenberg reduction of $A$. Each $H_N$ is really just the $N \times N + 1$ upper-left block of $H$.
Solving eigenvalue problems or systems of equations for
a general square matrix can thus be reduced, via Arnoldi iteration, to solving these problems for a Hessenberg matrix,
a much easier task.

At this point, we can view the Arnoldi iteration as a means to compute an orthonormal basis for a Krylov subspace, or
alternatively, to compute a partial Hessenberg factorization of a matrix.
But in Lab \ref{lab:Canonical_Transformations}, we discussed how orthogonal transformations can be used to transform a matrix to Upper
Hessenberg form.
We were able to find the eigenvalues of such Upper Hessenberg matrices in Lab \ref{lab:EigSolve}.
So what have we really gained by this new approach?
These previous approaches were based on matrix-matrix multiplication and required us to manipulate individual entries of the matrix.
The Arnoldi iteration avoids this and relies only on our ability to calculate matrix-vector multiplication.
Further, our present approach will allow us to compute only a partial Hessenberg factorization. This is advantageous when, as
is often the case, the behavior and properties of a matrix can be well-approximated by only a small portion of its Hessenberg
form.
\end{comment}

\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Arnoldi}{$\b, A, k, tol$}
	\State $Q \gets \allocate{\size{\b}}{k+1}$			\Comment{Some initialization steps}
	\State $H \gets \zeros{ k+1}{ k}$
	\State $Q[:,0] \gets \b/\norm{\b}_2$
	\For{$j=0\ldots k-1$}							\Comment{Perform the actual iteration.}
		\State $Q[:,j+1] \gets AQ[:,j]$
		\For{$i=0\ldots j$}					\Comment{Modified Gram-Schmidt.}
			\State $H[i,j] \gets Q[:,i]\trp Q[:,j+1]$
			\State $Q[:,j+1] \gets Q[:,j+1] - H[i,j] Q[:,i]$
		\EndFor
		\State $H[j+1,j] \gets \norm{Q[:,j+1]}_2$			\Comment{Set subdiagonal element of $H$.}
            \If{$|H[j+1,j]|<tol$}					\Comment{Stop if $\norm{Q[:,j+1]}_2$ is too small.}
			\State \pseudoli{return} $H[:j+1,:j+1]$, $Q[:,:j+1]$
		\EndIf
		\State $Q[:,j+1] \gets Q[:,j+1]/H[j+1,j]$				\Comment{Normalize $\q_{j+1}$.}
	\EndFor
	\State \pseudoli{return} $H[:-1, :]$, $Q$			\Comment{Return $H_{k}$.}
\EndProcedure
\end{algorithmic}
\caption{The Arnoldi Iteration. This algorithm accepts a square matrix $A$ and starting vector $\b$. It iterates $k$ times or until the norm of the next vector in the iteration is less than $tol$.
The algorithm returns upper Hessenberg $H$ and orthonormal $Q$ such that $H = Q^{\mathsf{H}}AQ$.}
\label{alg:arnoldi_iteration}
\end{algorithm}

In Algorithm \ref{alg:arnoldi_iteration}, $k$ is the number of times we multiply by $A$.
This will result in an order-$k+1$ Krylov subspace.

Something perhaps unexpected happens in the Arnoldi iteration if the starting vector $\x$ is an eigenvector of $A$.
If the corresponding eigenvalue is $\lambda$, then by definition $\mathcal{K}_k(A, \x)=\text{span}\{\x, \lambda \x, \lambda^2\x, \ldots, \lambda^k \x\}$, which is equal to the span of $\x$.
Let us trace through Algorithm \ref{alg:arnoldi_iteration} in this case. We will use $\q_i$ to denote the $i^{th}$ column of $Q$.

In line 4 we normalize $\x$, setting $\q_1 = \x/\|\x\|$.
In line 6 we set $\q_2 = A\q_1 = \lambda \q_1$.
Then in line 8
\[
H_{1,1} = \langle \q_1, \q_2 \rangle = \langle \q_1, \lambda \q_1 \rangle = \lambda \langle \q_1, \q_1 \rangle = \lambda,
\]
so in line 9 we subtract $\lambda \q_1$ from $\q_2$, ending with $\q_2=0$.

The vector $\q_2$ is supposed to be the next vector in the orthonormal basis for $\mathcal{K}_k(A, \x)$, but since it is 0, it is not linearly independent of $\q_1$. In fact, $\q_1$ already spans $\mathcal{K}_k(A, \x)$.
Hence, when in line 11 we find that the norm of $\q_2$ is zero (or close to it, allowing for numerical error), we terminate the algorithm early, returning the $1\times 1$ matrix $H = H_{1, 1}=\lambda$ and the $n\times 1$ matrix $Q = \q_1$.

A similar phenomenon may occur if the starting vector $\x$ is contained in a proper invariant subspace of $A$.

\section*{Arnoldi Iteration on Linear Operators} % ============================

A major strength of the Arnoldi Iteration is that it can run on a linear operator, even without knowing the matrix representation of the operator. If $A_{mul}$ is some linear function, then we can modify the pseudocode above by replacing $AQ[:,j]$ with $A_{mul}(Q[:,j])$. This will make it possible to find the eigenvalues of an arbitrary linear transformation. We will use this method in the problem below.

\begin{problem}\label{prob:arnoldi}
Using Algorithm \ref{alg:arnoldi_iteration}, complete the following Python function that performs the Arnoldi iteration.
Write this function so that it can run on complex arrays.

\begin{lstlisting}
def arnoldi(b, Amul, k, tol=1e-8):
    """Perform `k' steps of the Arnoldi iteration on the linear operator
    defined by `Amul', starting with the vector 'b'.

    Inputs:
        b (ndarray): The starting vector for the iteration.
        Amul (function): A function handle that describes a linear operator.
        k (int): The number of times to perform the iteration.
        tol (float): Stop iterating if the next vector in the iteration has
            norm less than `tol'. Defaults to 1e-8.

    Returns:
        H_n (ndarray)
        Q_n (ndarray)
            The number n will equal k, unless the algorithm terminated early,
            in which case n will be less than k.

    Examples:
        >>> A = np.array([[1,0,0],[0,2,0],[0,0,3]])
        >>> Amul = lambda x: A.dot(x)
        >>> H, Q = arnoldi(np.array([1,1,1]), Amul, 3)
        >>> np.allclose(H, np.conjugate(Q.T).dot(A).dot(Q) )
        True

        >>> H, Q = arnoldi(np.array([1,0,0]), Amul, 3)
        >>> H
        array([[ 1.+0.j]])
        >>> np.conjugate(Q.T).dot(A).dot(Q)
        array([[ 1.+0.j]])
    """
\end{lstlisting}

Hints:
\begin{enumerate}
\item Since \li{H} and \li{Q} will eventually hold complex numbers, initialize them as complex arrays (e.g., \li{A = np.empty((3,3), dtype=np.complex128)}).
\item Remember to use complex inner products.
\item This function can be tested on a matrix A by passing in \li{A.dot} for \li{Amul}.
\end{enumerate}
\end{problem}

\section*{Finding Eigenvalues Using Arnoldi Iteration} % ======================

Let $A$ be an $n \times n$ matrix.
Let $Q_k$ be the matrix whose columns $\q_1, \ldots, \q_k$ are the orthonormal basis for $\mathcal{K}_m(A, \x)$ generated by the Arnoldi algorithm, and
let $H_k$ be the $k\times k$ upper Hessenburg matrix defined at the $k^{th}$ stage of the algorithm.
Then these matrices satisfy
\begin{equation}\label{equ:hqa}
H_k = Q_k^{\mathsf H} A Q_k.
\end{equation}
If $k<n$, then $H_k$ is a low-rank approximation to $A$.
We may use its eigenvalues as approximations to the eigenvalues of $A$.
The eigenvalues of $H_k$ are called \emph{Ritz values}, and in fact they converge quickly to the largest eigenvalues of $A$.

\begin{problem}\label{prob:ritz}
Finish the following function that computes the Ritz values of a matrix.

\begin{lstlisting}
def ritz(Amul, dim, k, iters):
    """Find `k' Ritz values of the linear operator defined by `Amul'.

    Inputs:
        Amul (function): A function describing a linear operator on R^(dim).
        dim (int): The dimension of the space on which `Amul' acts.
        k (int): The number of Ritz values to return.
        iters (int): The number of times to perform the Arnoldi iteration.
            Must be between `k' and `dim'.

    Returns:
        ((k,) ndarray): `k' Ritz values of the operator defined by `Amul.'
    """
\end{lstlisting}
\end{problem}

One application of the Arnoldi iteration is to find the eigenvalues of linear operators that are too large to store in memory.
For example, if an operator acts on $\mathbb{C}^{2^{20}}$, then its matrix representation contains $2^{40}$ complex values.
Storing such a matrix would require 64 terabytes of memory!

An example of such an operator is the Fast Fourier Transform, cited by SIAM as one of the top algorithms of the century \cite{Cipra2000}.
The Fast Fourier Transform is used ubiquitously in the field of signal processing.

\begin{problem}
\label{prob:fourier_eigs}
The four largest eigenvalues of the Fast Fourier Transform are known to be $\{ -\sqrt{N}, \sqrt{N}, -i\sqrt{N}, i\sqrt{N} \}$ where $N$ is the dimension of the space on which the transform acts.
Use your function \li{ritz()} from Problem \ref{prob:ritz} to approximate the eigenvalues of the Fast Fourier Transform.
Set \li{k} to be 10 and set \li{dim} to be $2^{20}$.
For the argument \li{Amul}, use the \li{fft} function from \li{scipy.fftpack}.
\end{problem}

The Arnoldi iteration for finding eigenvalues is implemented in a Fortran library called ARPACK.
SciPy interfaces with the Arnoldi iteration in this library via the function \li{scipy.sparse.linalg.eigs()}.
This function has many more options than the implementation we wrote in Problem \ref{prob:ritz}.
In this example, the keyword argument \li{k=5} specifies that we want five Ritz values.
Note that even though this function comes from the \li{sparse} library in SciPy, we can still call it on regular NumPy arrays.

\begin{lstlisting}
>>> B = np.random.rand(10000).reshape(100, 100)
>>> sp.sparse.linalg.eigs(B, k=5, return_eigenvectors=False)
array([ -1.15577072-2.59438308j,  -2.63675878-1.09571889j,
        -2.63675878+1.09571889j,  -3.00915592+0.j        ,  50.14472893+0.j ])
\end{lstlisting}

\subsection*{Convergence} % ---------------------------------------------------

The Arnoldi method for finding eigenvalues quickly converges to eigenvalues whose magnitude is distinctly larger than the rest.
For example, matrices with random entries tend to have one eigenvalue of distinctly greatest magnitude.
Convergence of the Ritz values for such a matrix is plotted in Figure \ref{fig:arnoldi_random_val_conv}.

However, Ritz values converge more slowly for matrices with random eigenvalues.
Figure \ref{fig:arnoldi_random_eig_conv} plots convergence of the Ritz values for a matrix with eigenvalues uniformly distributed in $[0,1)$.

\begin{figure}
\centering
\begin{subfigure}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rand_vals_conv.pdf}
    \caption{The blue line plots the error of the Ritz value of largest magnitude.
    This eigenvalue converges after fewer than 20 iterations}
    \label{fig:arnoldi_random_val_conv}
\end{subfigure}
\begin{subfigure}[b]{.49\textwidth}
    \centering
    \includegraphics[width=\textwidth]{figures/rand_eigs_conv.pdf}
    \caption{All Ritz values have roughly equivalent magnitude.
    They take from 150 to 250 iterations to converge. }
    \label{fig:arnoldi_random_eig_conv}
\end{subfigure}
\caption{These plots show the relative error of the Ritz values as approximations to the eigenvalues of a matrix.
The figure at left plots the largest 15 Ritz values for a $500\times 500$ matrix with random entries.
The figure at right plots the largest 15 Ritz values for a $500\times 500$ matrix with uniformly distributed eigenvalues.}
\end{figure}

% TODO: this problem may be too difficult.
\begin{problem}
Finish the following function to visualize the convergence of the Ritz values.

\begin{lstlisting}
def plot_ritz(A, n, iters):
    """Plot the relative error of the Ritz values of `A'. Use the number of
    iterations as the x-axis and the relative error of the Ritz values of H_k
    a approximations to the eigenvalues of A as the y-axis.

    Inputs:
        A (ndarray)
        n (int): The number of Ritz values to plot.
        iters (int): The number of times to perform the Arnoldi iteration.
    """
    \end{lstlisting}

If $\tilde{\x}$ is an an approximation to $\x$, then the \emph{absolute error} in the approximation is
\[
\frac{\|\x - \tilde{\x}\|}{\|\x\|}.
\]
Hint: The most difficult part of this problem is to identify which Ritz values correspond to which eigenvalues.
After finding the Ritz values (or eigenvalues) of largest magnitude, use \li{np.sort()} to put them in order.
Make sure that this order is preserved throughout your program.

It may help to use the following algorithm.
\begin{enumerate}
    \item Find $n$ eigenvalues of $A$ of largest magnitude. Store these in order.
    \item Create an empty array to store the relative errors. For every $k \in $\li{[1, iters)},
    \begin{enumerate}
        \item Compute $H_k$ with the Arnoldi iteration.
        \item Find $n$ eigenvalues of $A$ of largest magnitude. Note that for small $k$, the matrix $H_k$ may not have this many eigenvalues.
        \item Store the absolute error. Make sure that the errors are stored in the correct order. For small $k$, some entries in the row or column may not be used.
    \end{enumerate}
    \item Use array broadcasting to compute the absolute error.
    \item Iteratively plot the errors. Lines for distinct eigenvalues should start at different places on the x-axis.
\end{enumerate}

Run your function on these examples.
The plots should be fairly similar to Figures \ref{fig:arnoldi_random_eig_conv} and \ref{fig:arnoldi_random_val_conv}.

\begin{lstlisting}
>>> # A matrix with random entries
>>> A = np.random.rand(300, 300)
>>> plot_ritz(A, 10, 175)
>>>
>>> # A matrix with uniformly distributed eigenvalues
>>> D = np.diag(np.random.rand(300))
>>> B = A.dot( D.dot(la.inv(A)) )
>>> plot_ritz(B, 10, 175)
\end{lstlisting}

If your code takes too long to run, consider integrating your solutions to Problems \ref{prob:arnoldi} and \ref{prob:ritz} with the body of this function.
\end{problem}

% Once the eigenvalue lab has been rewritten, have them use their own solver to find the eigenvalues of the $H_k$.

% This problem is very ill-conditioned.
\begin{comment}
\begin{problem}
Finding the roots of a polynomial can be represented as an eigenvalue problem.
Finding the roots of a monic polynomial (a polynomial with leading coefficient 1) $p = c_0 + c_1 x + \dots + c_{n-1} x^{n-1} + x^n$ is equivalent to finding the eigenvalues of the matrix
\[C = \begin{bmatrix}
0 & 0 & \dots & 0 & -c_0 \\
1 & 0 & \dots & 0 & -c_1 \\
0 & 1 & \dots & 0 & -c_2 \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
0 & 0 & \dots & 1 & -c_{n-1} \end{bmatrix}\]
This matrix is called the companion matrix of the polynomial $p$.
As it happens, every matrix is similar to the companion matrix of its characteristic polynomial, but we won't use that fact here.

The following is a function that, given an array containing the coefficients $c_0, c_1, \dots, c_{n-1}$ for a monic polynomial $p$, performs matrix multiplication by the corresponding companion matrix.

\begin{lstlisting}
def companion_multiply(c, u):
    v = np.empty_like(u)
    v[0] = - c[0] * u[-1]
    v[1:] = u[:-1] - c[1:] * u[-1]
    return v
\end{lstlisting}

Use the Arnoldi iteration to estimate the five zeros of largest norm of a degree $1000$ monic polynomial with randomly chosen coefficients (the leading coefficient still needs to be 1).
Run $50$ steps of the Arnoldi iteration.
Compare your results with the roots of the polynomial computed using NumPy's \li{poly1d} class.
This computation can be done like this (where \li{c} is the array of random coefficients for the polynomial)

\begin{lstlisting}
p = np.poly1d([1] + list(c[::-1]))
roots = p.roots
# Now sort by absolute value from largest to smallest
roots = roots[np.absolute(roots).argsort()][::-1]
\end{lstlisting}

How close are the first few zeros of largest norm?
\end{problem}
\end{comment}

\section*{Lanczos Iteration (Optional)} % =====================================

The Lanczos iteration is a version of the Arnoldi iteration that is optimized to operate on symmetric matrices.
If A is symmetric, then \eqref{equ:hqa} shows that $H_k$ is symmetric and hence tridiagonal.
This leads to two simplifications of the Arnoldi algorithm.

First, we have $0=H_{k, n}=\langle \q_k, A\q_n \rangle$ for $k \leq n-2$; i.e., $A\q_n$ is orthogonal to $\q_1, \ldots, \q_{n-2}$.
Thus, if the goal is only to compute $H_k$ (say to find the Ritz values), then we only need to store the two most recently computed columns of $Q$.
Second, the data of $H_k$ can also be stored in two vectors, one containing the main diagonal and one containing the first subdiagonal of $H_k$.
(By symmetry, the first superdiagonal equals the first subdiagonal of $H_k$.)

The Lanczos iteration is found in Algorithm \ref{alg:lanczos_iteration}.

\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Lanczos}{$\b, A, k, tol$}
	\State $\q_0 \gets \zeros{\size{\b}}$								\Comment{Some initialization}
	\State $\q_1 \gets \b/\norm{\b}_2$
	\State $\x \gets \allocate{k}$
	\State $\y \gets \allocate{k}$
	\For{$i=0\ldots k-1$}									\Comment{Perform the iteration.}
		\State $\z \gets A\q_1$					\Comment{$\z$ is a temporary vector to store $\q_{i+1}$.}
		\State $\x[i] \gets \q_1\trp \z$				\Comment{$\q_1$ is used to store the previous $\q_i$.}
		\State $\z \gets \z - \x[i] \q_1 + \y[i-1] \q_0$				\Comment{$\q_0$ is used to store $\q_{i-1}$.}
		\State $\y[i] = \norm{\z}_2$						\Comment{Initialize $\y[i]$.}
		\If{$\y[i]<tol$}								\Comment{Stop if $\norm{ \q_{i+1}}_2$ is too small.}
			\State \pseudoli{return} $\x[: i+1]$, $\y[: i]$
		\EndIf
		\State $\z = \z/ \y[i]$
		\State $\q_0, \q_1 = \q_1, \z$						\Comment{Store new $\q_{i+1}$ and $\q_i$ on top of $\q_1$ and $\q_0$.}
	\EndFor
	\State \pseudoli{return} $\x$, $\y[: -1]$
\EndProcedure
\end{algorithmic}
\caption{The Lanczos Iteration. This algorithm operates on a vector $\b$ of length $n$ and an $n \times n$ symmetric matrix $A$. It iterates $k$ times or until the norm of the next vector in the iteration is less than $tol$. It returns two vectors $\x$ and $\y$ that respectively contain the main diagonal and first subdiagonal of the current Hessenberg approximation.}
\label{alg:lanczos_iteration}
\end{algorithm}

\begin{problem}
\label{prob:lanczos}
Implement Algorithm \ref{alg:lanczos_iteration} by completing the following function.
Write it so that it can operate on complex arrays.
\begin{lstlisting}
def lanczos(b, Amul, k, tol=1E-8):
    '''Perform `k' steps of the Lanczos iteration on the symmetric linear
    operator defined by `Amul', starting with the vector 'b'.

    INPUTS:
    b    - A NumPy array. The starting vector for the Lanczos iteration.
    Amul - A function handle. Should describe a symmetric linear operator.
    k    - Number of times to perform the Lanczos iteration.
    tol  - Stop iterating if the next vector in the Lanczos iteration has
          norm less than `tol'. Defaults to 1E-8.

    RETURN:
    Return (alpha, beta) where alpha and beta are the main diagonal and
    first subdiagonal of the tridiagonal matrix computed by the Lanczos
    iteration.
    '''
\end{lstlisting}
\end{problem}

As it is described in Algorithm \ref{alg:lanczos_iteration}, the Lanczos iteration is not stable.
Roundoff error may cause the $\q_i$ to be far from orthogonal.
In fact, it is possible for the $\q_i$ to be so adulterated by roundoff error that they are no longer linearly independent.
% If needed we could make a separate lab on the Lanczos iteration and the Implicitly Restarted Lanczos Method.
% There isn't time or space here for it though.

\begin{problem}
The following code performs multiplication by a tridiagonal symmetric matrix.

\begin{lstlisting}
def tri_mul(a, b, u):
   ''' Return Au where A is the tridiagonal symmetric matrix with main
   diagonal a and subdiagonal b.
   '''
    v = a * u
    v[:-1] += b * u[1:]
    v[1:] += b * u[:-1]
    return v
\end{lstlisting}

Let $A$ be a $1000\times 1000$ symmetric tridiagonal matrix with random values in its nonzero diagonals.
Use the function \li{lanczos()} from Problem \ref{prob:lanczos} with 100 iterations to estimate the 5 eigenvalues of $A$ of largest norm.
Compare these to the 5 largest true eigenvalues of $A$

If you do this problem for different vectors $a$ and $b$, you may notice that occasionally the largest Ritz value is repeated.
This happens because the vectors used in the Lanczos iteration may not be orthogonal.
These erroneous eigenvalues are called ``ghost eigenvalues."
%They generally converge to actual eigenvalues of the matrix and can make the multiplicity of an eigenvalue look higher than it really is.
\end{problem}

There are modified versions of the Lanczos iteration that are numerically stable.
One of these, the Implicitly Restarted Lanczos Method, is found in SciPy as the function \li{scipy.sparse.linalg.eigsh()}.

\begin{comment}
\begin{problem}
In Lab \ref{lab:MarkovGraph} we discussed how to find the Laplacian Matrix of a graph.
The second-smallest eigenvalue of a graph is known as the ``Fiedler Value" or the ``algebraic connectivity."
The algebraic connectivity of a graph is positive if the graph is connected and zero if it is not.
In general, the multiplicity of the eigenvalue $0$ in the Laplacian matrix of a graph is the number of connected components of that graph.
The following code constructs the Laplacian matrix of a graph composed of a line of nodes.
The matrix is stored in \li{dia_matrix} format.

\begin{lstlisting}
from scipy import sparse
m = 1000
d = np.ones(m)
d[1:-1] += np.ones(m-2)
l = sparse.diags([-np.ones(m-1), d, -np.ones(m-1)], [-1, 0, 1])
\end{lstlisting}

Use the \li{eigsh} function to verify that this graph is connected.
You should look at documentation for the \li{scipy.sparse} library and find the options you need to use.
For proper convergence you will want to leave the number of eigenvalues computed at its default value.

The following code constructs the Laplacian matrix the same graph as before except that a single edge has been removed.

\begin{lstlisting}
m = 1000
cut = 500
d = np.ones(m)
d[1:-1] += np.ones(m-2)
d1 = -np.ones(m-1)
d1[cut] = 0
d[[cut, cut+1]] =1
l = sparse.diags([d1, d, d1], [-1, 0, 1])
\end{lstlisting}

Verify that this graph is not connected.
\end{problem}
\end{comment}


\subsection{Arithmetic on complex numbers}

For each of these we have:

\(x=a+bi\)

\(y=c+di\)

Addition is defined as:

\(x+y=a+bi+c+di\)

\(x+y=(a+c)+(b+d)i\)

Subtraction is defined as:

\(x-y=a+bi-c-di\)

\(x-y=(a-c)+(b-d)i\)

Multiplication is defined as:

\(xy=(a+bi)(c+di)\)

\(xy=ac-bd+adi+bci\)

\(xy=(ac-bd)+(ad+bc)i\)

Division is defined as:

\(\dfrac{x}{y}=\dfrac{a+bi}{c+di}\)

\(\dfrac{x}{y}=\dfrac{(a+bi)(c-di)}{(c+di)(c-di)}\)

\(\dfrac{x}{y}=\dfrac{(ac+bd)+(bc-ad)i}{c^2+d^2}\)


\section{Multigrid methods for nonlinear problem}
In classification, the key problem can be reduced to find the
representation (feature) for high dimension image for classifying.
Here we propose to solve the next unbalanced nonlinear system
\begin{equation}\label{eq:rep}
L(u) = f,
\end{equation}
for finding the suitable feature representation $u \in \mathbb{R}^c$ for
image $f\in \mathbb{R}^{n}$. The rationality of system can be traced back to
the low-dimension assumption that natural image need to be concentrated
on a low-dimension manifold with respect to the pixel space.

In order to motivate how to deal with the nonlinear system \eqref{eq:rep}, we
consider a simple nonlinear model problem
\begin{equation}\label{nonlinear:poisson}
\left\{
\begin{aligned}
-\nabla\cdot(a(u)\nabla u)+c(u)&=f, \quad x\in \Omega,\\
u&=0\quad\hbox{on}\quad\partial \Omega.
\end{aligned}
\right.
\end{equation}
%\example $p$-Laplacian $\displaystyle \min_{u\in W^{1,p}()\Omega} \frac{1}{p} \int_\Omega |\nabla u|^p dx-\int_\Omega fudx$.

The weak formulation of \eqref{nonlinear:poisson} reads: Find $u\in H^1_0(\Omega)$, such that
$$
(a(u)\nabla u,\nabla v)+(c(u), v)=(f,v), \quad \forall v\in H^1_0(\Omega).
$$
Define
\begin{equation}
(L(u), v)=(a(u)\nabla u,\nabla v)+(c(u), v).
\end{equation}
Then we have
\begin{equation}
L(u)=f.
\end{equation}
Now the discretization problem reads: Find $u_h\in V_h$ such that
\begin{equation}
(a(u_h)\nabla u_h,\nabla v_h)+(c(u_h), v_h)=(f,v_h)\quad \forall v_h\in V_h.
\end{equation}
Again define
$$
(L_h(u_h), v_h)=(a(u_h)\nabla u_h,\nabla v_h)+(c(u_h), v_h).
$$
Then we obtain
\begin{equation}\label{non:system}
L_h(u_h)=f_h.
\end{equation}
If $a(u_h)$ is a constant, then \eqref{non:system} reduces to a linear system and we denote the
linear system as $A_hu_h=f_h$.

Recall the two grid method described in \eqref{eq:smoothing0}, \eqref{eq:formresidual}, \eqref{erreq}, \eqref{coarse:correc} and \eqref{eq:prolongation00}
for linear problem
\begin{equation}\label{linearA}
A_hu_h=f_h,
\end{equation}
reads as following three steps:
\begin{enumerate}
\item Fine grid smoothing:
%applying $m$ times gradient descent iterations to obtain $u^m$.
$$
u_h\update u_h+S_h(f_h-A_hu_h)
$$
such as gradient descent method.
\item Coarse grid correction: solving the residual equation restricted
on the coarse grid $\ct_{2h}$ to obtain
$$
        A_{2h}e_{2h}=Q_{2h}r^h
$$
with $r^h=f_h-A_h u_h$.
\item Update: $u_h\update u_h+e_{2h}$.
\end{enumerate}
Coarse grid correction for the linear case can be rewritten as:
$$
 A_{2h}e_{2h}=Q_{2h}(f_h-A_h u_h).
 $$
 For any $u_h^0$, we write the above equation as
 \begin{equation}\label{eq2}
Q_{2h}(A_{h}(u_h^0+e_{2h})-A_hu_h^0)=Q_{2h}(f_h-A_h u_h).
 \end{equation}
 Noting that $Q_{2h}A_{h}=A_{2h}P_{2h}$, where $P_{2h}$ is the energy projection
%from $V_h$
 into $V_{2h}$.
 Therefore \eqref{eq2} can be written as
% $$
% A_{2h}(P_{2h} u_h^0)+ A_{2h} e_{2h})- A_{2h} (P_{2h} u_h^0)=Q_{2h}(f_h-A_h u_h).
% $$
% and further
  \begin{equation}\label{eq4}
 A_{2h}(P_{2h} u_h^0+ e_{2h})=Q_{2h}(f_h-A_h u_h)+A_{2h} (P_{2h} u_h^0).
  \end{equation}
Now let
$$
u_{2h,0}=P_{2h} u_h^0 \quad \hbox{and}\quad u_{2h}=P_{2h} u_h^0+ e_{2h},
$$
 then solving $e_{2h}$ in  \eqref{eq4} is equivalent to solve $u_{2h}$  in the following equation
\begin{equation}\label{eq5}
 A_{2h}u_{2h}=Q_{2h}(f_h-A_h u_h)+A_{2h} (u_{2h,0}).
 \end{equation}
 In this case, noting that $e_{2h}=u_{2h}-P_{2h} u_h^0=u_{2h}-u_{2h,0}$, hence the step 3 means
 $$
\hbox{ Update:} \,\, u_h\update u_h+u_{2h}-u_{2h,0}.
 $$

 In summery, the two grid method for the linear system \eqref{linearA} can be rewritten as following three steps:
\begin{enumerate}
\item Fine grid smoothing:
%applying $m$ times gradient descent iterations to obtain $u^m$.
$$
u_h\update u_h+S_h(f_h-A_hu_h)
$$
such as gradient descent method.
\item Coarse grid correction: for any $u_h^0$, solving the residual equation restricted
on the coarse grid $\ct_{2h}$ to obtain
$$
   u_{2h,0}=P_{2h} u_h^0, \quad     A_{2h}u_{2h}=Q_{2h}(f_h-A_h u_h)+A_{2h} (u_{2h,0}).
$$
where $P_{2h}$ is the energy projection %from $V_h$
into $V_{2h}$.
\item Update: $u_h\update u_h+u_{2h}-u_{2h,0}$.
\end{enumerate}
Usually, after the fine grid smoothing, we choose $u_h^0$ as the solution updated from the fine grid smoothing.  And for nonlinear problems, we replace $P_{2h}u_h^0$ by $\Pi_h^{2h}u_h^0$, where $\Pi_h^{2h}$ is an interpolation or projection from $V_h$ to $V_{2h}$.
Now we apply the above two grid method to the nonlinear system \eqref{non:system},
namely $L_{h}(u_h)=f_h$, we obtain the two grid method for nonlinear system
 \begin{enumerate}
\item Fine grid smoothing:
%applying $m$ times gradient descent iterations to obtain $u^m$.
$$
u_h\update u_h+S_h(f_h-L_h(u_h))
$$
such as gradient descent method.
\item Coarse grid correction: solving the residual equation restricted
on the coarse grid $\ct_{2h}$ to obtain
$$
   u_{2h,0}=\Pi_h^{2h} u_h, \quad     L_{2h}(u_{2h})=Q_{2h}(f_h-L_h (u_h))+L_{2h} (u_{2h,0}).
$$
where $\Pi_h^{2h}$ is an interpolation or projection from $V_h$ to $V_{2h}$.
\item Update: $u_h\update u_h+u_{2h}-u_{2h,0}$.
\end{enumerate}

To solve the nonlinear system, we can try the multilevel ideas
with smoothing in fine level, and truncated it into coarse level by recursion.
One strategy to involve the multi-scale idea is to ``smoothing'' in the fine level,
and restrict it as a good approximation in the coarse level, this idea can
be found in many literatures especially for multigrid methods in optimization
\cite{tai2002global, nash2000a}.  So, there is a more general nonlinear multigrid named
scheme-fully approximation scheme (FAS) \cite{briggs2000a, trottenberg2000multigrid},
which can be considered as the generalization of linear multigrid Algorithm \ref{alg:L-Slash1}.
Here we show a FAS algorithm with V-cycle as
\begin{breakablealgorithm}
	\caption{$u = {\text{ Bslash-FAS}}(u^{1,0},f,J,m_1, \cdots, m_J)$}
	\label{alg:Slash-FAS}
	\begin{algorithmic}
		\State Initialization
		$$
		f^1 = f.
		$$
		\State Smoothing and restriction from fine to coarse level (nested)
		\For{$\ell = 1:J$}
		\State Nonlinear relaxation on level $\ell$:
		\For{$i = 1:m_j$}
		\State
		$$
		u^{\ell,i} = u^{\ell,i-1} +S_\ell^i(f^\ell - L^{\ell}(u^{\ell,i-1})).
		$$
		\EndFor
		\State Form the initial guess and right side term for level $\ell+1$:
		$$
		u^{\ell+1,0} = \Pi_\ell^{\ell+1}u^{\ell,m_\ell}, \quad
		f^{\ell+1} = R_\ell^{\ell+1} (f^\ell - L^{\ell}(u^{\ell,m_\ell})) + L^{\ell+1}( u^{\ell+1, 0}).
		$$
		\EndFor
		\State Prolongation and correction from coarse to fine level
		\For{$\ell = J-1:1$}

		\State Form error in coarse level
		$$
		e^{\ell+1} = u^{\ell+1, m_{\ell+1}} - u^{\ell+1,0}.
		$$
		\State Correction by using error in coarse level
		$$
		u^{\ell,m_\ell} \leftarrow u^{\ell,m_\ell} + P_{\ell+1}^{\ell}e^{\ell+1}.
		$$
		\EndFor
	\end{algorithmic}
\end{breakablealgorithm}
%For some cases, we choose $S_\ell^i=[\nabla F^{\ell}(u^{\ell,i-1})]^{-1}$.
If the problem in \eqref{eq:rep} is linear, then we have the next theorem to show that
this FAS scheme is consist with the classical multigrid methods for linear systems.
\begin{theorem}
	If $L(u)$  in \eqref{eq:rep}  is a linear operation.
	Then Algorithm \ref{alg:Slash-FAS} is equivalent to Algorithm \ref{alg:L-Slash1} with any choice of $\Pi_\ell^{\ell+1}$.
\end{theorem}

\section{A nonlinear BVP example}
 Consider the following boundary value problem
$$
-u''+2u^3=0 \quad(0< x < 1), \quad u(0)=\frac{1}{3}\hbox{~and~}u(1)=\frac{1}{4}.
$$
The analytical solution is  $u=\frac{1}{x+3}$. The weak formulation becomes
\begin{equation}
(u',v')+(2u^3,v)=0.
\end{equation}
By choosing $u_h=\sum \alpha_i \phi_i$, we have
\begin{equation}\label{eq:nolinear}
L_h({\bm \alpha}):=A{\bm \alpha}+2h{\bm \alpha}^3-b,
\end{equation}
where
$A =\frac{1}{h}
\begin{pmatrix}
2 & -1 &  &&\\
-1 & 2 & -1&&\\
   &\vdots&\vdots&\vdots&\\
   &&-1&2&-1\\
   &&&-1&2
\end{pmatrix}$ and
$b =\frac{1}{h}
\begin{pmatrix}
\frac{1}{3}\\
\vdots\\
\frac{1}{4}\end{pmatrix}$.
We use gradient descent method and nonlinear multigrid method to solve \eqref{eq:nolinear}. The smoother used in the
nonlinear multigrid method is two steps of gradient descent method:
$$
\bm \alpha^{n+1}=\bm \alpha^n-\eta L_h(\bm \alpha^n) -\eta L_h(\bm \alpha^n-\eta L_h(\bm \alpha^n))
$$
where $L_h$ is defined by \eqref{eq:nolinear} and $\eta$ is the learning rate.

\begin{table}\label{Table:nonliear}%[htdp]
\begin{center}
\begin{tabular}{|c||c|c|}
\hline \hline
Size of Unkowns  & Nonlinear Multigrid  ethod  & Gradient Descent method \\ \hline\hline %& V + PCG for $A$\\ \hline %\hline
15       &   13 steps  &  61.6 k steps \\ \hline %&\onslide<3->{\brown{5/0.003s}}  \\ \hline
31      &   14 steps   &  976.6 k steps  \\ \hline %&\onslide<3->{\brown{5/0.003s}}  \\ \hline
63         &  15 steps & $>$ 1000 k steps\\ \hline %& \onslide<5->{\brown{5/0.005s}} \\ \hline
127       & 16 steps & $>$ 1000 k steps \\ \hline %&  \onslide<7->{\brown{5/0.016s}} \\ \hline
255        & 16 steps & $>$ 1000 k steps \\ \hline %& \onslide<9->{\brown{5/0.06s}}  \\ \hline
511     &17 steps& $>$ 1000 k steps \\ \hline %& \onslide<11->{\brown{5/0.22s}} \\   \hline \hline
1023      &18 steps&$>$ 1000 k steps \\ \hline %& \onslide<11->{\brown{5/0.22s}} \\   \hline \hline
\end{tabular}
\caption{Number of iterations for $\| L_h({\bm \alpha}) \|/ \|b\| \leq 10^{-6}$}
\end{center}
\end{table}

From the above numerical results shown in Table \ref{Table:nonliear}, the nonlinear multigrid method is uniform with respect to the size of
unknowns and much faster than the gradient descent method.

\endinput
Following the FAS idea, we propose the next Algorithms \ref{alg:OD-FAS} to solve the \eqref{eq:rep}:
\begin{breakablealgorithm}
	\caption{$u = {\text{OD-FAS}}(f,J,m_1, \cdots, m_J)$}
	\label{alg:OD-FAS}
	\begin{algorithmic}
		\State Initialization
		$$
		u^{1,0} = f,
		$$
		\State Feature extraction between levels,
		\For{$\ell = 1:J$}
		\State Feature extraction during level $\ell$,
		\For{$i = 1:m_\ell$}
		\State
		$$
		u^{\ell,i} = u^{\ell,i-1} +[\epsilon I + \nabla F^{\ell}(u^{\ell,i-1})]^{-1} (f^\ell - F^{\ell}(u^{\ell,i-1})).
		$$
		\EndFor
		\If{$\ell = J$}
		\State Return $u^{J,m_J}$.
		\EndIf
		\State Form initialization and right hand term in coarse level:
		$$
		u^{\ell+1,0}= R_\ell^{\ell+1} u^{\ell,m_\ell}, \quad
		f^{\ell+1} = R_\ell^{\ell+1} (r^\ell - F^{\ell}(u^{\ell,m_\ell})) + F^{\ell+1}( u^{\ell+1, 0}).
		$$
		\EndFor
	\end{algorithmic}
\end{breakablealgorithm}

497-6DL/8 Convolutional Multigrid Method/8.14-NonlinearMG.tex


\chapter{DT Fourier Transform}

Recall the complex exponential $z^{n}$ for $z\in\mathbb{C}$ is the Eigenfunction of DT LTI systems. If we can decompose an input into a (possibly infinite) sum of such signals, we can easily determine the output using the superposition principle. In this section we consider the decomposition when the input is aperiodic, called the DT \emph{Fourier Transform} (DTFT).

In contrast to the DT Fourier series, in this case the complex exponent of the Eigenfunction becomes $z = e^{j\omega}$ a continuous variable, and the decomposition is an uncountably infinite sum (integral). This gives the input-output relationship for a stable DT LTI system as
\[
x[n] = \frac{1}{2\pi}\int\limits_{2\pi} X\left(e^{j\omega}\right) \, e^{j \omega n}\; d\omega \;\longrightarrow\; y[n] = \frac{1}{2\pi}\int\limits_{2\pi} H\left(e^{j\omega}\right) X\left(e^{j\omega}\right) \, e^{j \omega n}\; d\omega
\]
where $H\left(e^{j \omega}\right)$ are the Eigenvalues, again called the \emph{frequency response}. We now turn to determining under what circumstances the decomposition exists and how to find the function $X\left(e^{j\omega}\right)$.

\textbf{Note:} The notation $X\left(e^{j\omega}\right)$ can be confusing. It just emphasizes that $z \rightarrow e^{j\omega}$. The expressions are functions of the independent variable $\omega$.

\section{Analysis and Synthesis Equations}

Consider the Fourier series of $x[n]$, a periodically extended finite-length DT signal $\tilde{x}[n]$, e.g.

\begin{center}
  \includegraphics[scale=0.8]{graphics/dt-derivation.pdf}
\end{center}
where $\tilde{x}[n]$ is zero outside the range $[N_1,N_2]$. Since $x[n] = \tilde{x}[n]$ over the interval $-N_1$ to $N_2$
\[
a_k = \frac{1}{N}\sum\limits_{n = -N_1}^{N_2} \tilde{x}[n] e^{-j\frac{2\pi}{N}kn} = \frac{1}{N}\sum\limits_{n = -\infty}^{\infty} x[n] e^{-j\frac{2\pi}{N}kn}
\]
Define the function $X\left(e^{j\omega}\right) = \sum\limits_{n = -\infty}^{\infty} x[n] e^{-j\omega n}$, then
\[
a_k = \frac{1}{N} X\left(e^{jk\omega_0}\right)
\]
are samples of $X\left(e^{j\omega}\right)$ at locations that are multiples of $\omega_0 = \frac{2\pi}{N}$.
Substituting back into the synthesis equation
\[
\tilde{x}[n] = \sum\limits_{k = -N_1}^{N_2} a_k e^{j\frac{2\pi}{N}kn} = \sum\limits_{k = -N_1}^{N_2} \frac{1}{N} X\left(e^{jk\omega_0}\right) e^{jk\omega_0 n}
\]
Now note that $N = \frac{2\pi}{\omega_0}$ so that
\[
\tilde{x}[n] = \frac{1}{2\pi} \sum\limits_{k = -N_1}^{N_2} X\left(e^{jk\omega_0}\right) e^{jk\omega_0 n} \; \omega_0
\]
Now let $N \rightarrow \infty$.
\begin{align*}
  \lim_{N\rightarrow \infty} \tilde{x}[n] &= \lim_{N\rightarrow \infty} \frac{1}{2\pi} \sum\limits_{k = -N_1}^{N_2} X\left(e^{jk\omega_0}\right) e^{jk\omega_0 n} \; \omega_0\\
  x[n] &= \frac{1}{2\pi} \int_{2\pi} X\left(e^{j\omega}\right) e^{j\omega n} \; d\omega
\end{align*}
This is shown graphically in the figure below. As $N$ approaches infinity the sampling of the unit circle becomes infinite, and the summation approaches an integral.

\begin{center}
  \includegraphics[scale=0.9]{graphics/dt-fourier-limit.pdf}
\end{center}

This gives the \emph{DT Fourier Transform Pair}. The Analysis Equation or Forward Transform is:
\[
X\left(e^{j\omega}\right) = \sum\limits_{n = -\infty}^{\infty} x[n] e^{-j\omega n}
\]
Note $X\left(e^{j\omega}\right)$ must be a periodic function with period $2\pi$. The Synthesis Equation or Inverse Transform is:
\[
x[n] = \frac{1}{2\pi} \int_{2\pi} X\left(e^{j\omega}\right) e^{j\omega n} \; d\omega
\]
where the integral is over any $2\pi$ period of $X$.

\begin{example}
  Let $x[n] = \delta[n]$
  \begin{align*}
    X\left(e^{j\omega}\right) &= \sum\limits_{n = -\infty}^{\infty} x[n] e^{-j\omega n}\\
    &= \sum\limits_{n = -\infty}^{\infty} \delta[n] e^{-j\omega n}\\
    &= e^{-j\omega (0)}\\
    &= 1
  \end{align*}
  $\blacksquare$
\end{example}

\begin{example}

  Let $x[n] = \left( \gamma \right)^n\; u[n]$
  \begin{align*}
    X\left(e^{j\omega}\right) &= \sum\limits_{n = -\infty}^{\infty} x[n] e^{-j\omega n}\\
    &= \sum\limits_{n = 0}^{\infty} \left( \gamma \right)^n \; e^{-j\omega n}\\
    &= \sum\limits_{n = 0}^{\infty} \left( \gamma e^{-j\omega} \right)^n
  \end{align*}
  Using the geometric series $ \sum\limits_{n = 0}^{\infty} z^n = \frac{1}{1-z}$ for $|z| < 1$ gives:
  \[
  X\left(e^{j\omega}\right) = \sum\limits_{n = 0}^{\infty} \left( \gamma e^{-j\omega} \right)^n = \frac{1}{1-\gamma e^{-j\omega}} = \frac{e^{j\omega}}{e^{j\omega} - \gamma}
  \]
  If $\mid\gamma e^{-j\omega}\mid < 1$ or equivalently $\mid \gamma \mid < 1$.

  \[
  \left( \gamma \right)^n\; u[n] \; \stackrel{\mathcal{F}}{\longrightarrow} \; \frac{1}{1-\gamma e^{-j\omega}}
  \]
Below is a plot of the original signal and the magnitude and phase spectrum when $\gamma = \tfrac{1}{2}$.
  \begin{center}
    \includegraphics[scale=0.5]{graphics/dtft-example1-x.pdf}
  \end{center}

  \begin{center}
    \includegraphics[scale=0.5]{graphics/dtft-example1-Xmag.pdf}
    \includegraphics[scale=0.5]{graphics/dtft-example1-Xarg.pdf}
  \end{center}
  $\blacksquare$
\end{example}

\begin{example}
  Let
  \[
  X\left(e^{j\omega}\right) = \left\{ \begin{array}{lc}
    1 & |\omega -2\pi k| < \omega_c\\
    0 & \text{else}
  \end{array}
  \right.
  \; \text{for}\; k\in\mathbb{Z} \;\text{and}\; \omega_c < \pi
  \]

  \begin{align*}
    x[n] &= \frac{1}{2\pi} \int_{2\pi} X\left(e^{j\omega}\right) e^{j\omega n} \; d\omega\\
    &= \frac{1}{2\pi} \int\limits_{-\omega_c}^{\omega_c}  e^{j\omega n} \; d\omega\\
    &= \frac{1}{2\pi} \frac{1}{jn} e^{j\omega n} \Bigg|_{-\omega_c}^{\omega_c}\\
    &= \frac{1}{\pi n} \left( \frac{1}{2j} e^{j\omega_c n} - \frac{1}{2j} e^{-j\omega_c n} \right)\\
    &= \frac{1}{\pi n} \sin(\omega_c n)
  \end{align*}
  $\blacksquare$
\end{example}

\begin{example}
  Let
  \[
  X\left(e^{j\omega}\right) = \sum\limits_{k = -\infty}^{\infty} \delta(\omega-\omega_0 -2\pi k)
  \]
  for $-\pi < \omega_0 < \pi$

  \begin{center}
    \includegraphics[scale=1]{graphics/dtft-periodic-ex.pdf}
  \end{center}

  \begin{align*}
    x[n] &= \frac{1}{2\pi} \int_{2\pi} X\left(e^{j\omega}\right) e^{j\omega n} \; d\omega\\
    &= \frac{1}{2\pi} \int\limits_{-\pi}^{\pi} \delta(\omega-\omega_0)e^{j\omega n} \; d\omega\\
    &= \frac{1}{2\pi} e^{j\omega_0 n}
  \end{align*}
  $\blacksquare$
\end{example}

\section{Existence of the DT Fourier Transform}

The example of the exponential $x[n] = \left(\gamma\right)^n\,u[n]$ above showed that for the DT Fourier transform to exist, the Fourier (analysis) sum must exist. Similar to the CT Fourier transform, a mild conditions is a sufficient prerequisite for the Fourier transform of a signal $x[n]$ to exist: it must be  absolutely summable
\[
\sum\limits_{n = -\infty}^{\infty} |x[n]| < \infty
\]

This conditions is not necessary however, and we can extend the Fourier transform to a broader class of signals, if we allow delta functions in the transform, as in the sinusoidal examples above.

\section{Properties of the DT Fourier Transform}

There are several useful properties of the DT Fourier Transform that, when combined with a table of transforms (see Table 5.2, page 392 of OW), allow us to take the Fourier transform of  wide array of signals, and one, the convolution property, that allows us to determine the output of a system in the frequency domain easily. We state these here without proof in rough order of usefulness. See the course text for detailed derivations.

We use the following notation
\[
\mathcal{F}\left\{ x[n] \right\} = X\left(e^{j\omega}\right) = \sum\limits_{n = -\infty}^{\infty} x[n] e^{-j\omega n}
\]
\[
\mathcal{F}^{-1}\left\{ X\left(e^{j\omega}\right) \right\} = x[n] =  \frac{1}{2\pi} \int_{2\pi} X\left(e^{j\omega}\right) e^{j\omega n} \; d\omega
\]
\[
x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \;  X\left(e^{j\omega}\right)
\]
Important: $X\left(e^{j\omega}\right)$ is periodic in $2\pi$ such that
\[
X\left(e^{j(\omega + 2\pi k)}\right) =  X\left(e^{j\omega}\right) \;\text{for}\; k \in \mathbb{Z}
\]
\begin{itemize}
  \item Linearity Property.
  Let $x_1[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X_1\left(e^{j\omega}\right)$ and $x_2[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X_2\left(e^{j\omega}\right)$
  then for $a,b\in\mathbb{C}$
  \[
a x_1[n] + b x_2[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; a X_1\left(e^{j\omega}\right) + b X_2\left(e^{j\omega}\right)
\]
Example:
\begin{align*}
\mathcal{F}\left\{ 2\left( \frac{1}{2}\right)^nu[n] -5 \left( -\frac{1}{4}\right)^nu[n] \right\} &= \frac{2}{1-\frac{1}{2}e^{-j\omega}} - \frac{5}{1+\frac{1}{4}e^{-j\omega}}
\end{align*}
\item Time-shift Property.
  Let
  \[
  x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j\omega}\right)
  \]
  then
  \[
  x[n-n_0] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; e^{-j\omega n_0} X\left(e^{j\omega}\right)
  \]
  Example:
  \[
  \mathcal{F}\left\{ \delta[n-5] \right\} = e^{-j5\omega}
  \]
\item Frequency Shift Property.
  Let
  \[
  x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j\omega}\right)
  \]
  then
  \[
  e^{j\omega_0 n} x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j(\omega-\omega_0)}\right)
  \]
  Example:
  \[
  \mathcal{F}^{-1}\left\{ \frac{1}{1-\frac{1}{2}e^{-j\omega}e^{j\frac{\pi}{20}}} \right\} = e^{j\frac{\pi}{20} n} \left( \frac{1}{2}\right)^n u[n]
  \]
\item Conjugation Property.
  Let
  \[
  x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j\omega}\right)
  \]
  then
  \[
  x^*[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X*\left(e^{-j\omega}\right)
  \]
  Thus, if $x[n]$ is real $X\left(e^{j\omega}\right)$ has conjugate symmetry
  \[
  X\left(e^{-j\omega}\right) = X^*\left(e^{j\omega}\right)
  \]
  and the magnitude spectrum is an even function and the phase spectrum is an odd function.
\item Differencing and Accumulation Property.
  Let
  \[
  x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j\omega}\right)
  \]
  then
  \[
  x[n] - x[n-1] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j\omega}\right) - e^{-j\omega} X\left(e^{j\omega}\right) = \left(1-e^{-j\omega}\right)X\left(e^{j\omega}\right)
  \]
  and
  \[
  \sum\limits_{m = -\infty}^{n} x[m] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; \frac{1}{1-e^{-j\omega}}X\left(e^{j\omega}\right) + \pi X\left(e^{j0}\right)\sum\limits_{k = -\infty}^{\infty} \delta(\omega - 2\pi k)
  \]
\item Time Expansion Property.
  Let
  \[
  x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j\omega}\right)
  \]
  then
  \[
  x_{(k)}[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{jk\omega}\right)
  \]
  where
  \[
  x_{(k)}[n] = \left\{ \begin{array}{lc}
    x[n/k] & \text{if}\; n = \; \text{multiple of}\; k\\
    0 & \text{if}\; n \neq \; \text{multiple of}\; k\\
  \end{array}
\right.
\]
\begin{center}
  \includegraphics[scale=0.8]{graphics/dt=transform-property-expand.pdf}
\end{center}
\item Frequency Differentiation Property
  Let
  \[
  x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j\omega}\right)
  \]
  then
  \[
  n\, x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; j \frac{d}{d\omega} X\left(e^{j\omega}\right)
  \]
  Example:
  \begin{align*}
    \mathcal{F}\left\{ n\left( \frac{1}{8}\right)^n u[n] \right\} &= j \frac{d}{d\omega} \left\{\frac{1}{1-\frac{1}{8}e^{-j\omega}} \right\}\\
    &= j \frac{-\left(-\frac{1}{8} (-j)e^{-j\omega}\right)}{\left( 1-\frac{1}{8}e^{-j\omega} \right)^2}\\
    &= \frac{\frac{1}{8}e^{-j\omega}}{\left( 1-\frac{1}{8}e^{-j\omega} \right)^2}
  \end{align*}
\item Parseval's Relation.
  Let
  \[
  x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j\omega}\right)
  \]
  then
  \[
  \sum\limits_{n = -\infty}^{\infty} |x[n]|^2 \; \stackrel{\mathcal{F}}{\longleftrightarrow} \;
  \frac{1}{2\pi} \int_{2\pi} \left| X\left(e^{j\omega}\right) \right|^2 \; d\omega
  \]
  The energy is also the integral over one period of the DTFT magnitude squared.
\item Convolution Property.
  Recall for a DT LTI system with impulse response $h[n]$ the output is
  \[
  y[n] = h[n]*x[n]
  \]
  In the frequency domain this is equivalent to
  \[
  Y\left(e^{j\omega}\right) = H\left(e^{j\omega}\right) \, X\left(e^{j\omega}\right)
  \]
  As in CT systems, convolution in the discrete-time domain is equivalent to multiplication in the frequency domain. Example: suppose a DT system has impulse response
  \[
  h[n] = \left( \gamma_1^n + \gamma_2^n \right)\, u[n]
  \]
  and the input is $x[n] = n\, \gamma_3^n\, u[n]$ where $|\gamma_1| < 1$, $|\gamma_2| < 1$, $|\gamma_3| < 1$. The output in the frequency domain is
  \begin{align*}
    Y\left(e^{j\omega}\right) &= H\left(e^{j\omega}\right) \, X\left(e^{j\omega}\right)\\
    &= \left[ \frac{1}{1-\gamma_1 e^{-j\omega}} + \frac{1}{1-\gamma_2 e^{-j\omega}} \right] \frac{\gamma_3 e^{-j\omega}}{\left( 1-\gamma_3 e^{-j\omega} \right)^2}\\
    &= \frac{\gamma_3 e^{-j\omega}}{\left(1-\gamma_1 e^{-j\omega}\right)\left( 1-\gamma_3 e^{-j\omega} \right)^2} + \frac{\gamma_3 e^{-j\omega}}{\left(1-\gamma_2 e^{-j\omega}\right)\left( 1-\gamma_3 e^{-j\omega} \right)^2}
  \end{align*}

\item Multiplication (modulation) Property.
  Let
  \[
  x[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; X\left(e^{j\omega}\right)
  \]
  and
  \[
  y[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; Y\left(e^{j\omega}\right)
  \]
  then
  \[
  x[n]\,y[n] \; \stackrel{\mathcal{F}}{\longleftrightarrow} \; \frac{1}{2\pi} \int_{2\pi} X\left(e^{j\theta}\right)\, Y\left(e^{j(\omega-\theta)}\right)\; d\theta
  \]
\end{itemize}

\section{DT Fourier Transform of a Periodic Signal}

The DTFS allows us to write any periodic function with period $N$ as
\[
x[n] = \sum\limits_{k = N_0}^{N_0 + N -1} a_k e^{j\frac{2\pi}{N}kn}
\]
taking the DT Fourier Transform
\[
X\left(e^{j\omega}\right) = \sum\limits_{k = N_0}^{N_0 + N -1} a_k \mathcal{F}\left\{e^{j\frac{2\pi}{N}kn}\right\}
\]
Using the previously derived transform shows, similar to CT, the DT Fourier Transform of a periodic signal is
\[
X\left(e^{j\omega}\right) = \sum\limits_{k = -\infty}^{\infty} 2\pi a_k \delta\left(\omega - \frac{2\pi k}{N}\right)
\]

Example
  \[
  x[n] = \cos\left(\frac{2\pi}{10} n\right) = \frac{1}{2}e^{j\frac{2\pi}{10} n} + \frac{1}{2}e^{-j\frac{2\pi}{10} n}
  \]
  Using the previous transform
  \[
  X\left(e^{j\omega}\right) =  \sum\limits_{k = -\infty}^{\infty} \pi \delta\left(\omega - \frac{2\pi}{10} -2\pi k\right) + \pi \delta\left(\omega + \frac{2\pi}{10} -2\pi k\right)
  \]
  Which looks like
  \begin{center}
  \includegraphics[scale=1]{graphics/dtft-periodic-ex2.pdf}
\end{center}


\documentclass[notitlepage]{simple}

\author{Matt McCarthy}
\title{Completeness of $\RR$}
\date{Februrary 2016}

\begin{document}

\maketitle

\begin{thm*}
	The real numbers form a complete metric space.
\end{thm*}

\noindent Notation: We denote the set of non-negative real numbers as $\RR^+$.

We begin by providing a way to measure distances in a space.

\begin{definition}[Metric]
	Let $X$ be a set and $d:X\times X\rightarrow \RR^+$ be a map.
	We say $d$ is a \textit{metric} if and only if all of the following hold.
	\begin{enumerate}
		\item For all $x,y\in X$, $d(x,y)=0$ if and only if $x=y$.
		\item For all $x,y\in X$, $d(x,y)=d(y,x)$ (symmetric property).
		\item For all $x,y,z\in X$, $d(x,z)\leq d(x,y)+d(y,x)$ (triangle inequality).
	\end{enumerate}
	If $d$ is a metric on $X$, we say $(X,d)$ forms a \textit{metric space}.
\end{definition}

For example, $(\QQ,d)$ and $(\RR,d)$ where $d(x,y)=\abs{x-y}$ is a metric space,
and more generally, $(\RR^n, d_n)$ where $d_n$ is the Euclidean distance is also a metric space.
Now that we have a way to talk about distances in spaces we can talk about convergence of sequences in those spaces.

\begin{definition}[Convergent Sequence]
	Let $(X,d)$ be a metric space and let $\set{a_n}_{n\in\NN}$ be a sequence in $X$.
	We say $\set{a_n}_{n\in\NN}$ \textit{converges} to some $a\in X$, denoted $a_n\rightarrow a$ if and only if for all $\varepsilon>0$ there exists a $N_\varepsilon\in\NN$ such that $d(a_n,a)<\varepsilon$ for all $n\geq N_\varepsilon$.
\end{definition}

However, convergence is often too strong a condition to prove, and moreover convergence also requires a proposed limit.
Therefore, we introduce the notion of a \textit{Cauchy Sequence} where the terms get arbitrarily close to each other.

\begin{definition}[Cauchy Sequence]
	Let $(X,d)$ be a metric space and let $\set{a_n}_{n\in\NN}$ be a sequence in $X$.
	We say $\set{a_n}_{n\in\NN}$ is a \textit{Cauchy sequence} if and only if for any $\varepsilon>0$, there exists a $N_\varepsilon\in\NN$ such that $d(a_n,a_m)<\varepsilon$ for all $n,m\geq N_\varepsilon$.
\end{definition}

Note, however that Cauchy sequences are not always convergent in their space.
For example consider the sequence given by
\[
	a_n = \paren{1+\frac{1}{n}}^n.
\]
We know these terms are rational and thus $a_n$ is a $\QQ$-valued sequence.
Furthermore, the sequence is in fact Cauchy.
However, the limit of this sequence as $n$ approaches infinity is $e$ and thus, this Cauchy sequence does not converge in $\QQ$, but it does in $\RR$.

This leads us to a notion of \textit{completeness} in a metric space.

\begin{definition}[Complete Metric Space]
	Let $(X,d)$ be a metric space.
	Then we say $(X,d)$ is \textit{complete} if and only if every Cauchy sequence in $X$ converges in $X$.
\end{definition}
By this definition of completeness, $\QQ$ is not complete as shown by our prior example.
However, $\RR$ is complete.

Before we get into any theorems here, we need a notion of \textit{boundedness} for subsets of the real numbers.

\begin{definition}[Bounded]
	Let $S$ be a subset of the real numbers.
	Then $S$ is \textit{bounded} if and only if there exists an $M>0$ such that $S\subseteq [-M,M]$.
\end{definition}

We also need to invoke a famous theorem due to Bolzano and Weierstrass, which states that any bounded sequence in $\RR$ has a subsequence that is convergent in $\RR$.

\begin{thm}[Bolzano-Weierstrass]
	Let $\set{a_n}_{n\in\NN}$ be a bounded $\RR$-valued sequence.
	Then there exists a $\set{a_{n_k}}\subseteq\set{a_n}$ such that $a_{n_k}$ converges to some $a\in\RR$.
\end{thm}
\begin{proof}
	Let $\set{x_n}_{n\in\NN}$ be bounded.
	Then there exists an $M\in\NN$ such that $\set{x_n}_{n\in\NN}\subset[-M,M]\subset\RR$.
	Bisect $[-M,M]$ into $[-M,0],[0,M]$.
	At least one half has infinitely many sequence points, call this one $I_1$.
	Pick a sequence point $x_{n_1}\in I_1$.
	Bisect $I_1$ and call the half with infinitely many sequence points $I_2$.
	Pick a sequence point $x_{n_2}\in I_2$ such that the index $n_2>n_1$.
	Iterate this process, that is bisect $I_k$ and call the half with infinitely many points $I_{k+1}$.
	Then choose a sequence point $x_{n_{k+1}}\in I_{k+1}$ with the property that the index $n_{k+1}>n_k>\ldots > n_1$.
	Moreover, by nested interval property, $\bigcap_{k\in\NN} I_k\neq\emptyset$.
	Thus there exists at least one $x\in \bigcap_{k\in\NN} I_k$.
	We want to show that $x_{n_k}\rightarrow x$.
	Let $\varepsilon> 0$ be given.
	Choose a $K_\varepsilon\in\NN$ such that $2^{-K_\varepsilon}M< \varepsilon$.
	Note that for all $k\geq K_\varepsilon$, $2^{-k} M <  2^{-K_\varepsilon}M< \varepsilon$.
	Since $x\in \bigcap_{k\in\NN} I_k$, $x\in I_k$ for all $k\in\NN$.
	Thus for all $k\in\NN$, $|x_{n_k}-x|\leq 2^{-k}M$.
	Therefore, when we force $k\geq K_\varepsilon$ we have $I_k\subset I_{K_\varepsilon}$ and
	\[
	|x_{n_k}-x|\leq 2^{-k}M \leq 2^{-K_\varepsilon} M < \varepsilon.
	\]
	Thus, $\set{x_{n_k}}_{k\in\NN}$ is a subsequence of $\set{x_n}_{n\in\NN}$ that is convergent in $\RR$.
\end{proof}

Furthermore, we need the following lemma.

\begin{lemma}
	Any $\RR$-valued Cauchy sequence is bounded.
\end{lemma}
\begin{proof}
	Let $\varepsilon >0$ be given and let $\set{a_n}_{n\in\NN}$ be a Cauchy sequence in $\RR$.
	Therefore, there exists an $N_\varepsilon$ such that $|a_n-a_m|<\varepsilon$ for all $n,m\geq N_\varepsilon$.
	Therefore, for all $n\geq N_\varepsilon$, $|a_{N_\varepsilon}-a_n|<\varepsilon$.
	Thus for each $n\geq N_\varepsilon$, $a_n\in[a_{N_\varepsilon}-\varepsilon,a_{N_\varepsilon}+\varepsilon]$.
	Take $M_1=\max\set{a_{N_\varepsilon}-\varepsilon,a_{N_\varepsilon}+\varepsilon}$, which yields $\set{a_n}_{n=N_\varepsilon}^\infty\subset[-M_1,M_1]$.

	Furthermore, since $\set{a_n}_{n=1}^{N_\varepsilon-1}$ is finite, take $M_2=\max\set{|a_n|}_{n=1}^{N_\varepsilon-1}$.
	Thus, $\set{a_n}_{n=1}^{N_\varepsilon-1}\subset [-M_2,M_2]$.
	If we take $M=\max\set{M_1,M_2}$, then we have $\set{a_n}_{n\in\NN}\subset[-M,M]$.
	Therefore, the sequence is bounded.
\end{proof}

And now we provide the proof.

\begin{thm}
	The real numbers form a complete metric space.
\end{thm}
\begin{proof}
	Let $\set{a_n}_{n\in\NN}$ be Cauchy in $\RR$.
	Let $\varepsilon > 0$ be given.
	Since $\set{a_n}_{n\in\NN}$ is Cauchy there exists an $N_{\varepsilon/2}\in\NN$ such that $|a_n-a_m|<\varepsilon/2$ for all $n,m\geq N_{\varepsilon/2}$.
	Furthermore, $\set{a_n}_{n\in\NN}$ is bounded and thus by Bolzano-Weierstrass, there exists a $\set{a_{n_k}}_{k\in\NN}\subset\set{a_n}_{n\in\NN}$ such that $a_{n_k}$ converges to some $a\in\RR$.

	We claim that $a_n$ converges to $a$.
	Since $a_{n_k}$ converges to $a$, there exists a $K_{\varepsilon/2}\in\NN$ such that $|a_{n_k}-a|<\varepsilon/2$ for all $k\geq K_{\varepsilon/2}$.
	Take $N=\max\set{N_{\varepsilon/2},n_{K_{\varepsilon/2}}}$ and force $n,k$ large enough such that $n,n_k\geq N$.
	Consider $|a_n-a|$.
	\[
		|a_n-a|=|a_n-a_{n_k}+a_{n_k}-a|\leq |a_n-a_{n_k}|+|a_{n_k}-a|<\frac{\varepsilon}{2}+\frac{\varepsilon}{2}=\varepsilon
	\]
	Thus for any $\varepsilon >0$, there exists an $N\in\NN$ such that $|a_n-a|<\varepsilon$ for all $n\geq N$.
	Therefore $a_n$ converges to $a$ and $\RR$ is a complete metric space.
\end{proof}

\section{Bonus}

\begin{thm*}
	The set of infinite binary sequences is uncountable.
\end{thm*}

\begin{definition}[Countable Set]
	Let $X$ be a set.
	We say $X$ is \textit{countable} if and only if there exists a bijection between $X$ and a subset of $\NN$.
	If $X$ has a bijection with $\NN$ itself, then we say $X$ is \textit{countably infinite}.
\end{definition}

\begin{proof}
	Let $B$ represent the set of infinite binary sequences.
	Proceed via contradiction.
	Assume $B$ is countable.
	Then $B=\set{s_i}_{i\in\NN}$ where $s_i=(b_{i,1},b_{i,2},\ldots)$ with each $b_{i,j}\in\set{0,1}$.
	Define $\sim:\set{0,1}\rightarrow\set{0,1}$ by
	\[
		\sim x=
		\begin{cases}
			0 & x=1\\
			1 & x=0
		\end{cases}.
	\]
	We need to create a infinite binary sequence that is not equal to any $b_i$.
	Take $s=(\sim b_{1,1},\sim b_{2,2},\ldots)$.
	Obviously, $s\in B$.
	However, for each $i\in\NN$, $\sim b_{i,i}\neq b_{i,i}$ and thus $s\neq s_i$ for each $i\in\NN$.
	Therefore, $s\notin B$ which is a contradiction.
	Ergo, $B$ must be uncountable.
\end{proof}

\end{document}


\section*{Exercises}

\begin{ex}
  Find the determinant of the following matrices.

  \begin{enumerate}
  \item
    $A = \begin{mymatrix}{rr}
      1 & -34 \\
      0 & 2
    \end{mymatrix}$

  \item
    $B = \begin{mymatrix}{rrr}
      4 & 0 &  0 \\
      3 & -2 & 0 \\
      14 & 1 & 5
    \end{mymatrix}$

  \item
    $C = \begin{mymatrix}{rrrr}
      2 & 3 & 15 & 0 \\
      0 & 4 & 1 & 7 \\
      0 & 0 & -3 & 5 \\
      0 & 0 & 0 & 1
    \end{mymatrix}$
  \end{enumerate}

 \begin{sol}
   (a) $\det(A) = 2$. (b) $\det(B) = -40$. (c) $\det(C) = -24$.
 \end{sol}

\end{ex}


%
% Chapter 1.8
%

\section*{1.8 Continuity}

A function is continuous at a number \(a\) if
$$\lim_{x \to a}f(x) = f(a)$$\\
A function \(f\) is continuous from the right at a number \(a\) if
$$\lim_{x \to a^+}f(x)=f(a)$$
and \(f\) is continuous from the left at \(a\) of
$$\lim_{x \to a^-}f(x)=f(a)$$\\
A function \(f\) is continuous on an interval if it is continuous at every number in the interval.
\\\\
If \(f\) and \(g\) are continuous at \(a\) and if \(c\) is a constant, then the following functions are also continuous at \(a\).
\begin{enumerate}
    \item \(f + g\)
    \item \(f - g\)
    \item \(cf\)
    \item \(fg\)
    \item \(\frac{f}{g}\) if \(g(a) \neq 0\)
\end{enumerate}
Any polynomial, rational function, root function, or trigonometric function is continuous anywhere on its domain.

\subsection*{Intermediate Value Theorem}

Suppose that \(f\) is continuous on the closed interval \([a, b]\) and let \(N\) be any number between \(f(a)\) and \(f(b)\), where \(f(a) \neq f(b)\). Then there exists a number \(c\) in \((a, b)\) such that \(f(c)\) = \(N\).


\section{Definitions}

\subsection{Notation}

Functions
\[ \exists f : A \to B \]
means that we claim that there exists a function \(f\) mapping items of set \(A\)
to items of set \(B\). We define sets by enumerating their items using the
standard \( \set{} \) notation.

Booleans
\[ \Bool = \set{ \True, \False } \]
Natural numbers plus zero
\[ \mathbb{Z} = \set{ 0,1,2,... } \]
Tuples (Products)
\[ (X,Y) = \set{ (x,y) ; x \in X \land y \in Y } \]
\[ (X,Y,Z,...) = (X,(Y,(Z,...))) \]
For every tuple there exist two unique functions:
\[ \exists  \pi_1 : (X,Y) \to X \]
\[ \exists  \pi_2 : (X,Y) \to Y \]
We access fields of nested tuples by using composition over these functions.
Often we want to define tuples together with accessors functions. By
saying
\[ T = (a:X,b:Y,c:Z,...) \]
we mean
\[ T = (X,(Y,(Z,...))) \land \]
\[ \exists a : T \to X = \pi_1 \land \exists b : T \to Y = \pi_1 \circ \pi_2
\land \exists c : T \to Z = \pi_1 \circ \pi_2 \circ \pi_2 \land ... \]
Unions (Co-products)
\[ X|Y = \set{u | u \in X \lor u \in Y }\]
Note: X and Y has to be separable from each other, i.e. for any particular \( u
\) it should be possible to tell whether it comes from X or from Y.
\hfill \break \noindent
Optional
\[ \mathbb{O}[X] = X | \oslash \]
Lists
\[ \List[X] = \mathbb{O}[(X,\mathbb{O}[(X,...)])] \]
Dictionaries
\[ \Dict[K,V] = \List[(K,V)] \]
\[ \exists item : \Dict[K,V] \to K \to \Opt[V] \]
Note: we assume that \(item \) returns the last \( V \) element with matching \(
K \).
\hfill \break \noindent

\subsection{Core entities}

TODO


\section{Lengths of Curves}
If we're at some point $(x,f(x))$ and move $\d{x}$ units over, we'll be at a new point $(x+\d{x},f(x+\d{x}))$.
This point is $\d{x}$ units horizontally and $\d{y}=f(x+\d{x})-f(x)$ vertically away from $(x,f(x))$.
So, by the Pythagorean Theorem, the new point is $\d{s} = \sqrt{(\d{x})^2+(\d{y})^2}$ units away.

\begin{figure}[H]
	\label{arclength}
	\centering
	\includegraphics[width=0.66\textwidth]{./applications_integrals/arclength.png}
	\caption{\hyperref{}{}{}{Secant length approaches arc length}}
\end{figure}

As $\d{x}$ approaches 0, $\d{s}$, the length of the secant line, approaches the length of the curve.
If we summed all of these $\d{s}$'s in some interval we'd have the length of the curve on that interval.
\begin{align*}
	s &= \int_{a}^{b}{\d{s}} \\
	&= \int_{a}^{b}{\sqrt{(\d{x})^2+(\d{y})^2}} \\
	&= \int_{a}^{b}{\sqrt{(\d{x})^2\left(1+\left(\frac{\d{y}}{\d{x}}\right)^2\right)}} \\
	&= \int_{a}^{b}{\sqrt{1+\left(\frac{\d{y}}{\d{x}}\right)^2}\d{x}}.
\end{align*}

\begin{example}
	Show that the circumference of a circle with radius $r$ is $C=2\pi r$.
\end{example}
\begin{answer}
	Starting with the equation of a circle of radius $r$ and implicitly differentiating,
	\begin{align*}
		x^2 + y^2 &= r^2 \\
		2x + 2y\dd{y}{x} &= 0 \\
		\dd{y}{x} &= \frac{-x}{y} \\
		\left(\dd{y}{x}\right)^2 &= \frac{x^2}{y^2} = \frac{x^2}{r^2-x^2} \\
		C &= 2\int_{-r}^{r}{\sqrt{1+\frac{x^2}{r^2-x^2}}\d{x}} \\
		&\text{ (2 b/c we need to count upper and lower half)} \\
		&= 2\int_{-r}^{r}{\sqrt{\frac{r^2}{r^2-x^2}}\d{x}} \\
		&= 2\int_{-r}^{r}{r\sqrt{\frac{1}{r^2-x^2}}\d{x}} \\
		&= 2r\arcsin{\left(\frac{x}{r}\right)}\biggr\rvert_{-r}^{r} \\
		&= 2\pi r.
	\end{align*}
\end{answer}

Note that when deriving the arc length formula, we could have just as easily have divided by $(\d{y})^2$.
This would give us an equivalent arc length formula that's applicable when $x$ is a function of $y$.

\begin{example}
	Find the length of the curve $y=x^{1/3}$ from $(-8,2)$ to $(8,2)$.
\end{example}
\begin{answer}
	Rather than  tediously integrate the square root of a cube root if we set our bounds in terms of $x$, we can rewrite our equation and set out bounds in terms of $y$.
	\begin{align*}
		x &= y^3 \\
		\dd{x}{y} &= 3y^2 \\
		\left(\dd{x}{y}\right)^2 &= 9y^4 \\
		s &= \int_{-2}^{2}{\sqrt{1+9y^4}\d{y}} \\
		&\footnotemark\approx 17.261.
	\end{align*}
\end{answer}
\footnotetext{You shouldn't be expected to evaluate this integral analytically. Using a calculator to get a numerical answer is fine.}


\documentclass[twoside]{MATH77}
\usepackage{multicol}
\usepackage[fleqn,reqno,centertags]{amsmath}
\begin{document}
\begmath 3.3 Random Numbers: Exponential, Rayleigh, and Poisson

\silentfootnote{$^\copyright$1997 Calif. Inst. of Technology, \thisyear \ Math \`a la Carte, Inc.}

\subsection{Purpose}

Generate pseudorandom numbers from the exponential and Rayleigh
distributions and pseudorandom integers from the Poisson distribution.

\subsection{Usage}

\subsubsection{Generating exponential pseudorandom numbers}

The density function for the exponential distribution with mean and standard
deviation, $\mu $, has the value zero for $x < 0$ and $\mu^{-1} \exp (-x/\mu )$ for $%
x \geq 0$. The cumulative distribution function has the value zero for $x <
0 $ and $1- \exp (- x/\mu )$ for $x \geq 0$. If $u$ is a random variable
having the uniform distribution on [0,~1] then $x = - \mu \ \log u$ is a
random variable having the exponential distribution with mean and standard
deviation, $\mu .$

\paragraph{Program Prototype, Single Precision}

\begin{description}
\item[REAL]  \ {\bf SRANE, XMEAN, X}
\end{description}

Assign a value to XMEAN.
$$
\fbox{{\bf X = SRANE(XMEAN)}}
$$
\paragraph{Argument Definitions}

\begin{description}
\item[XMEAN]  \ [in] Specifies the mean and standard deviation of the
desired exponential distribution. Require XMEAN $>0.$

\item[SRANE]  \ [out] The function returns a nonnegative pseudorandom number
from the exponential distribution with mean and standard deviation equal to
XMEAN.
\end{description}

\subsubsection{Generating Rayleigh pseudorandom numbers}

The density function for the Rayleigh distribution with scaling parameter, $%
\alpha $, has the value zero for $x < 0$ and
\begin{equation*}
(x/\alpha ^2) \exp (-x^2/2\alpha ^2)
\end{equation*}
for $x \geq 0$. The cumulative distribution function has the value zero for $%
x < 0$ and
\begin{equation*}
1 - \exp(-x^2/2 \alpha ^2)
\end{equation*}
for $x \geq 0$. The mean and standard deviation of this distribution are
\begin{equation*}
\mu = \alpha \sqrt{\pi /2} \approx 1.25331\ \alpha
\end{equation*}
and
\begin{equation*}
\sigma = \alpha \sqrt{2 - \pi /2} \approx 0.655136\ \alpha
\end{equation*}
If $u$ is a random variable having the uniform distribution on [0,~1] then
\begin{equation*}
x = \alpha \sqrt{-2 \log u}
\end{equation*}
is a random variable having the Rayleigh distribution with scaling
parameter, $\alpha .$

\paragraph{Program Prototype, Single Precision}

\begin{description}
\item[REAL]  \ {\bf SRANR, ALPHA, X}
\end{description}

Assign a value to ALPHA.
$$
\fbox{{\bf X = SRANR(ALPHA)}}
$$
\paragraph{Argument Definitions}

\begin{description}
\item[ALPHA]  \ [in] Specifies the scaling of the desired Rayleigh
distribution. Require ALPHA $>0$. The distribution will have mean = ALPHA $%
\times \sqrt{\pi /2}$ and variance = ALPHA$^2\times (2-\pi /2).$

\item[SRANR]  \ [out] The function returns a nonnegative pseudorandom number
from the Rayleigh distribution with scaling parameter, ALPHA.
\end{description}

\subsubsection{Generating Poisson pseudorandom integers}

The Poisson distribution with mean and variance, $\mu $, is defined over
nonnegative integers. The nonnegative integer, $k$, occurs with probability $%
p_k$ given by%
\begin{equation*}
p_k=e^{-\mu }\frac{\mu ^k}{k!}
\end{equation*}
\paragraph{Program Prototype, Single Precision}

\begin{description}
\item[REAL]  \ {\bf XMEAN}

\item[INTEGER]  \ {\bf ISRANP, K}
\end{description}

Assign a value to XMEAN.
$$
\fbox{{\bf K = ISRANP(XMEAN)}}
$$

\paragraph{Argument Definitions}

\begin{description}
\item[XMEAN]  \ [in] Specifies the mean and variance of the desired Poisson
distribution. XMEAN must be positive and not so large that exp($-$XMEAN)
would underflow. For example if the underflow limit is $10^{-38}$, XMEAN
must not exceed~87. This subprogram requires more computing time for larger
values of XMEAN or if XMEAN is changed frequently. See Section D.

\item[ISRANP]  \ [out] The function returns a nonnegative pseudorandom
integer from the Poisson distribution with mean XMEAN.
\end{description}

\subsubsection{Modifications for Double Precision}

Change the names SRANE, SRANR, and ISRANP to DRANE, DRANR, and IDRANP
respectively, and change the REAL type statements above to
DOUBLE PRECISION. Note particularly that if either of the function names
DRANE or DRANR is used it must be typed DOUBLE PRECISION either explicitly
or via an IMPLICIT statement.

\subsection{Examples and Remarks}

The programs DRSRANE, DRISRANP, and DRSRANR demonstrate, respectively, the
use of SRANE, ISRANP, and SRANR. These programs use SSTAT1 and SSTAT2, or
ISSTA1 and ISSTA2 to compute and print statistics and a histogram based on a
sample of~10000 numbers each.

To fetch or set the seed used in the underlying pseudorandom integer
sequence use the subroutines described in Chapter~3.1.

\subsection{Functional Description}

\subparagraph{Method}

The exponential random number is computed as $x = -\mu \ \log u$ where $u$
is a random number from the uniform distribution on [0,~1].

The Rayleigh random number is computed as $x = \alpha \sqrt{-2 \log u}$
where $u$ is a random number from the uniform distribution on [0,~1].

The Poisson subprogram uses ideas from \cite{Snow:1968:A342}.  The method
begins by obtaining a random number, $u$, from the uniform distribution on
[0,~1].  Then the probabilities $p_0$, $p_1$, ..., defined above in
Section B.3, are summed until the sum reaches or exceeds $u$.  The index
of the last term in the sum is then returned as the Poisson random
integer.

To improve efficiency on the assumption that the subprogram may be
referenced many successive times with the value of XMEAN remaining
unchanged, the partial sums through at most the term $p_{84}$ are stored in
an internal array as they are computed. On subsequent references, if the
value of XMEAN has not been changed, previously computed partial sums can be
tested without the need to recompute them. The testing starts at the index
nearest to the value, XMEAN, since these indices have the highest
probabilities of being selected.

These subprograms obtain uniform pseudorandom numbers by calling SRANUA or
DRANUA, using the array in common block /RANCMS/ or /RANCMD/ as a buffer as
described in Chapter~3.1.

Values returned as double-precision random numbers will have random bits
throughout the word, however the quality of randomness should not be
expected to be as good in a low-order segment of the word as in a high-order
part.

\bibliography{math77}
\bibliographystyle{math77}

\subsection{Error Procedures and Restrictions}

In subprograms SRANE, DRANE, SRANR, and DRANR the input parameter should be
positive, however no test is made of this. The input parameter is simply
used as a multiplicative factor.

Subprogram ISRANP will issue an error message and return the value $-1$ if
XMEAN\ $\leq $ 0 or if XMEAN $\geq -0.5\times \log \,($ underflow limit ).

\subsection{Supporting Information}

\begin{tabular}{@{\bf}l@{\hspace{5pt}}l}
\bf Entry & \hspace{.35in} {\bf Required Files}\vspace{2pt} \\
DRANE & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
DRANE, ERFIN, ERMSG, RANPK1, RANPK2\rule[-5pt]{0pt}{8pt}}\\
DRANR & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
DRANR, ERFIN, ERMSG, RANPK1, RANPK2\rule[-5pt]{0pt}{8pt}}\\
IDRANP & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, IDRANP, RANPK1, RANPK2, DERM1, DERV1\rule[-5pt]{0pt}{8pt}}\\
ISRANP & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
AMACH, ERFIN, ERMSG, ISRANP, RANPK1, RANPK2, SERM1, SERV1\rule[-5pt]{0pt}{8pt}}\\
SRANE & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
ERFIN, ERMSG, RANPK1, RANPK2, SRANE\rule[-5pt]{0pt}{8pt}}\\
SRANR & \parbox[t]{2.7in}{\hyphenpenalty10000 \raggedright
ERFIN, ERMSG, RANPK1, RANPK2, SRANR}\\\end{tabular}

Based on subprograms written for JPL by Stephen L. Richie, Heliodyne Corp.,
and Wiley R. Bunton, JPL, 1969. Adapted to Fortran~77 for the JPL MATH77
library by C. L. Lawson and S. Y. Chiu, JPL, April~1987.

1991 November: Lawson reorganized and renamed common blocks.

\begcodenp
\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSRANE}\vspace{10pt}
\lstinputlisting{\codeloc{srane}}
\newpage

\vspace{30pt}\centerline{\bf \large ODSRANE}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{srane}}
\newpage

\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRSRANR}\vspace{10pt}
\lstinputlisting{\codeloc{sranr}}
\newpage

\vspace{30pt}\centerline{\bf \large ODSRANR}\vspace{10pt}
\lstset{language={}}
\lstinputlisting{\outputloc{sranr}}

\newpage
\lstset{language=[77]Fortran,showstringspaces=false}
\lstset{xleftmargin=.8in}

\centerline{\bf \large DRISRANP}\vspace{0pt}
\lstinputlisting{\codeloc{isranp}}

\vspace{20pt}\centerline{\bf \large ODISRANP}\vspace{5pt}
\lstset{language={}}
\lstinputlisting{\outputloc{isranp}}
\end{document}


\lab{Algorithms}{LU Decomposition}{LU Decomposition}
\label{lab:LUdecomp}
\objective{In this section we will find the REF and the LU decomposition.}

In linear algebra there are three elementary row operations: switching two rows, multiplying a row by a constant, and adding a multiple of one row to another row.
Each of these operations can, in theory, be done by left multiplication by a corresponding elementary matrix.
This approach is \emph{extremely} slow in practice.
In practice, it is much faster to perform these operations directly by modifying only the portions of an array that change as a result of the row operation.
The follow code shows how these modifications can be made in-place to an array.
\lstinputlisting[style=python,name=]{row_opers.py}

\section*{Programming Row Reduction}
Solving a linear system can be done most efficiently by using elementary row operations to reduce a matrix to \emph{row echelon form} (REF), as opposed to \emph{reduced row echelon form} (RREF).
Consider the following matrix:
\[
\begin{pmatrix}
4&5&6&3 \\
2&4&6&4 \\
7&8&0&5
\end{pmatrix}
\]
Using elementary row operations, we can reduce $A$ to REF as follows:
\begin{lstlisting}
: import numpy as np
: A = np.array([[4., 5., 6., 3.],[2., 4., 6., 4.],[7., 8., 0., 5.]])
array([[ 4.,  5.,  6.,  3.],
       [ 2.,  4.,  6.,  4.],
       [ 7.,  8.,  0.,  5.]])
: A[1] -= (A[1,0]/A[0,0]) * A[0]
: A[2] -= (A[2,0]/A[0,0]) * A[0]
: A[2,1:] -= (A[2,1]/A[1,1]) * A[1,1:]
: A
array([[ 4. ,  5. ,  6. ,  3. ],
       [ 0. ,  1.5,  3. ,  2.5],
       [ 0. ,  0. , -9. ,  1. ]])
\end{lstlisting}
The additional requirement is often added that the first nonzero entry of each row be 1.
Do not worry about that requirement here.
Notice that in our third row operation we were able to operate on only a portion of the third row because we knew that the first value would still be 0.
In this case it made little difference, but it is good to watch for things like this because they can save a great deal more time when working with larger matrices.

\begin{problem}
\label{prob:REF}
Write a Python function, which takes a matrix and reduces it to REF.
Assume that the matrix is invertible and ignore the possibility that a zero may appear on the main diagonal during row reduction.
\end{problem}

\section*{LU Decomposition}
Using row reduction we can reduce an invertible matrix $A$ to upper triangular form.
Say this can be done in $k$ row operations.
Let $U$ be the upper triangular form of $A$, so we have:
Hence, we have
\[
U = E_k \dots E_2 E_1 A.
\]
Since the elementary matrices are invertible, we also have
\[
(E_k \dots E_2 E_1)^{-1} U =  A.
\]
Then we define $L$ to be
\[
L = (E_k \dots E_2 E_1)^{-1}
\]
which is the same as
\[
L = E_1^{-1} E_2^{-1} \dots E_k^{-1}
\]
In either case, we have $L U = A$.

The inverses of elementary matrices are also elementary matrices. $L$ can be computed by applying a series of simple operations to an identity matrix.
As it turns out, when we are only doing type 3 row operations, each of the operations represented by right multiplication by these inverse matrices results in a change in a single entry of $L$.

In practice, the LU decomposition of an array $A$ can be computed like this:
\begin{itemize}
\item Make a copy $U$ of $A$.
\item Make an identity matrix $L$ of the same shape as $A$.
\item Iterate through the entries below the diagonal of $U$.
For each entry below the main diagonal of $U$ do the following:
	\begin{itemize}
	\item Set the corresponding entry of $L$ to the quotient of the current entry of $U$ and the entry of the main diagonal of $U$ located above the current entry.
	\item Perform the type 3 row operation to set the current entry of $U$ to 0.
		Remember to avoid computation involving columns that have already been processed.
	\end{itemize}
\item Return $L$ and $U$
\end{itemize}
In this case, we have ignored the possibility that a 0 may appear along the main diagonal during computation.
A full implementation of the LU decomposition would have to account for this possibility as well.

\section*{Why This Matters}
The LU decomposition is more efficient for solving linear systems than traditional row reduction.
It also can be applied to matrices with more than one column.
The $LU$ decomposition also allows quick computation of inverses and determinants.
For very large matrices, the LU decomposition can be performed without using any extra space.
$L$ can be stored above the main diagonal of the array and $U$ can be stored below it.
There is no need to store the main diagonal of $L$ since all its entries are ones.

\begin{problem}
\label{prob:LU}
Write a Python function takes as input an $n\times n$ matrix, performs the LU decomposition and returns $L$ and $U$.
To verify that it works, multiply $L$ and $U$ together and compare to $A$.
Assume that the matrix is invertible and ignore the possibility that a zero may appear on the main diagonal during row reduction.

Write another version of the function that modifies its input in place, storing $L$ below the main diagonal and $U$ in the rest of the array.
\end{problem}

\begin{problem}
\label{prob:lusolve}
Write a function that takes the LU decomposition computed by the second function you made in Problem \ref{prob:LU} and another array representing the right hand side of a linear system and modifies the second array in place so that it represents the solution to the linear system.
No changes to the array storing the LU decomposition are necessary.
\end{problem}

\begin{problem}
\label{prob:det}
Write a function which uses the solution to Problem \ref{prob:REF} to find the determinant of $A$.
Notice that the solution to Problem \ref{prob:REF} computes $U$ when applied to a square matrix.
\end{problem}

SciPy includes more complete implementations of the LU decomposition in the \li{linalg} module.
This LU implementation returns an additional array that shows any row swaps that were made during the decomposition.
The LU decomposition is calculated with \li{lu}.
\li{lu_factor} calculates the LU decomposition and stores it in a single array.
It returns a tuple of arrays, one which is the LU decomposition and the other of which represents the necessary permutation of the rows.
\li{lu_solve} takes the output from \li{lu_factor} and uses it to solve a linear system.
Using \li{lu_factor} and \li{lu_solve} is a good alternative to inverting large matrices since it is faster to compute an LU decomposition than it is to invert a matrix.

\section*{The Cholesky Decomposition (Optional)}

For certain circumstances, we have a more efficient alternative to the LU decomposition.
The Cholesky decomposition requires half the number of calculations and half the memory that the standard LU decomposition needs.
Furthermore, it is a numerically stable decomposition, which makes it all the more useful.
Because of the efficiency and numerical stability, Cholesky decomposition is used in solving least squares, optimization, and state estimation problems.
The Cholesky decomposition, however, is only applicable to Hermitian (for real matrices, Hermitian is equivalent to symmetric) positive definite matrices.
In fact, the Cholesky decomposition is an efficient way to test if a matrix is positive definite.
The Cholesky decomposition of a positive definite matrix is unique.
Think of the Cholesky decomposition as the matrix equivalent taking the square root of a positive real number.

The Cholesky decomposition of a $A$ is an lower-triangular matrix, $L$, such that
\begin{equation*}
 A = LL^*
\end{equation*}
Where $L^*$ is the conjugate transpose of $L$.
For real valued matrices, this is equivalent to $L^T$.

The entries of $L$ are calculated as follows.
\begin{align*}
&L_{i,j} = \frac{1}{L_{j,j}}\left(A_{i,j} -\sum_{k=1}^{j-1}{L_{i,k}L_{j,k}^*}\right) \mbox{ for $i>j$} \\ \\
&L_{i,i} = \sqrt{A_{i,i} - \sum_{k=1}^{i-1}{L_{i,k}L_{i,k}^*}}
\end{align*}
where $L^*$ denotes the conjugate transpose of $L$.

Notice that in this computation, current calculation will depend on previous calculations.
To calculate $L$ properly, you must start in the upper left corner and iterate down.

Note: when testing positive definite systems, an easy way to generate a random symmetric positive definite matrix is by generating a random array \li{A} and then computing \li{A.dot(A.T)}.

\begin{problem}
Write your own implementation of the Cholesky decomposition.
Test it using a random symmetric matrix (build a random square matrix $A$, then $A^TA$ will be positive definite).
Check the output of your function to ensure that it is functioning properly.
\end{problem}

\begin{problem}
Modify your previous answer so that it computes the Cholesky decomposition by modifying the array in place.
Make sure you set the portion of the array above the main diagonal to 0.
Then write a function that takes this reduced form of the array and uses it to solve a linear system by back substitution.
This should be nearly the same as Problem \ref{prob:lusolve}.
\end{problem}

the linalg module of SciPy also includes a cholesky decomposition that should be much faster than the one you just computed.
It works much like the LU decomposition.
\li{cho_factor} returns the cholesky decomposition of the array that can then be used in solving a linear system with \li{cho_solve}.