\chapter{Eigen-things}
This chapter will develop the theory of eigenvalues and eigenvectors,
the so-called ``Jordan canonical form''.
(Later on we will use it to
define the characteristic polynomial.)

\section{Why you should care}
We know that a square matrix $T$ is really just
a linear map from $V$ to $V$.
What's the simplest type of linear map?
It would just be multiplication by some scalar $\lambda$,
which would have associated matrix (in any basis!)
\[
	T =
	\begin{bmatrix}
		\lambda & 0 & \dots & 0 \\
		0 & \lambda & \dots & 0 \\
		\vdots & \vdots & \ddots & \vdots \\
		0 & 0 & \dots & \lambda
	\end{bmatrix}.
\]
That's perhaps \emph{too} simple, though.
If we had a fixed basis $e_1, \dots, e_n$
then another very ``simple'' operation
would just be scaling each basis element $e_i$ by $\lambda_i$,
i.e.\ a \vocab{diagonal matrix} of the form
\[
	T = \begin{bmatrix}
		\lambda_1 & 0 & \dots & 0 \\
		0 & \lambda_2 & \dots & 0 \\
		\vdots & \vdots & \ddots & \vdots \\
		0 & 0 & \dots & \lambda_n
	\end{bmatrix}.
\]
These maps are more general.
Indeed, you can, for example, compute $T^{100}$ in a heartbeat:
the map sends $e_1 \to \lambda_1^{100} e_1$.
(Try doing that with an arbitrary $n \times n$ matrix.)

Of course, most linear maps are probably not that nice.
Or are they?
\begin{example}
	[Getting lucky]
	Let $V$ be some two-dimensional vector space
	with $e_1$ and $e_2$ as basis elements.
	Let's consider a map $T \colon V \to V$
	by $e_1 \mapsto 2e_1$ and $e_2 \mapsto e_1+3e_2$,
	which you can even write concretely as
	\[ T = \begin{bmatrix}
		2 & 1 \\
		0 & 3
	\end{bmatrix} \quad\text{in basis $e_1$, $e_2$}. \]
	This doesn't look anywhere as nice until we realize we can rewrite it as
	\begin{align*}
		e_1 &\mapsto 2e_1 \\
		e_1+e_2 &\mapsto 3(e_1+e_2).
	\end{align*}
	So suppose we change to the basis $e_1$ and $e_1 + e_2$.
	Thus in the new basis,
	\[ T = \begin{bmatrix}
		2 & 0 \\
		0 & 3
	\end{bmatrix} \quad\text{in basis $e_1$, $e_1+e_2$}. \]
	So our completely random-looking map,
	under a suitable change of basis,
	looks like the very nice maps we described before!
\end{example}
In this chapter, we will be \emph{making} our luck,
and we will see that our better understanding of matrices
gives us the right way to think about this.

\section{Warning on assumptions}
Most theorems in this chapter only work for
\begin{itemize}
	\ii finite-dimensional vector spaces $V$,
	\ii over a field $k$ which is \emph{algebraically closed}.
\end{itemize}
On the other hand, the definitions work fine without
these assumptions.

\section{Eigenvectors and eigenvalues}
Let $k$ be a field and $V$ a vector space over it.
In the above example, we saw that there were two very nice
vectors, $e_1$ and $e_1+e_2$, for which $V$ did something very simple.
Naturally, these vectors have a name.
\begin{definition}
	Let $T \colon V \to V$ and $v \in V$ a \emph{nonzero} vector.
	We say that $v$ is an \vocab{eigenvector} if $T(v) = \lambda v$
	for some $\lambda \in k$ (possibly zero, but remember $v \neq 0$).
	The value $\lambda$ is called an \vocab{eigenvalue} of $T$.

	We will sometimes abbreviate
	``$v$ is an eigenvector with eigenvalue $\lambda$''
	to just ``$v$ is a $\lambda$-eigenvector''.
\end{definition}
Of course, no mention to a basis anywhere.

\begin{example}
	[An example of an eigenvector and eigenvalue]
	Consider the example earlier with
	$T = \begin{bmatrix} 2 & 1 \\ 0 & 3 \end{bmatrix}$.
	\begin{enumerate}[(a)]
		\ii Note that $e_1$ and $e_1 + e_2$ are
		$2$-eigenvectors and $3$-eigenvectors.
		\ii Of course, $5e_1$ is also an $2$-eigenvector.
		\ii And, $7e_1 + 7e_2$ is also a $3$-eigenvector.
	\end{enumerate}
\end{example}
So you can quickly see the following observation.
\begin{ques}
	Show that the $\lambda$-eigenvectors, together with $\{0\}$
	form a subspace.
\end{ques}
\begin{definition}
	For any $\lambda$, we define the $\lambda$-\vocab{eigenspace}
	as the set of $\lambda$-eigenvectors together with $0$.
\end{definition}
This lets us state succinctly that
``$2$ is an eigenvalue of $T$
with one-dimensional eigenspace spanned by $e_1$''.

Unfortunately, it's not exactly true that eigenvalues always exist.
\begin{example}[Eigenvalues need not exist]
	Let $V = \RR^2$ and let $T$ be the map
	which rotates a vector by $90\dg$
	around the origin.
	Then $T(v)$ is not a multiple of $v$ for any $v \in V$,
	other than the trivial $v=0$.
\end{example}

However, it is true if we replace $k$ with an
algebraically closed field\footnote{A field is \vocab{algebraically closed}
	if all its polynomials have roots,
	the archetypal example being $\CC$.}.
\begin{theorem}[Eigenvalues always exist over algebraically closed fields]
	Suppose $k$ is an \emph{algebraically closed} field.
	Let $V$ be a finite dimensional $k$-vector space.
	Then if $T \colon V \to V$ is a linear map,
	there exists an eigenvalue $\lambda \in k$.
\end{theorem}
\begin{proof}
	(From \cite{ref:axler})
	The idea behind this proof is to consider ``polynomials'' in $T$.
	For example, $2T^2-4T+5$ would be shorthand for $2T(T(v)) - 4T(v) + 5v$.
	In this way we can consider ``polynomials'' $P(T)$;
	this lets us tie in the ``algebraically closed'' condition.
	These polynomials behave nicely:
	\begin{ques}
		Show that $P(T)+Q(T) = (P+Q)(T)$ and $P(T) \circ Q(T) = (P \cdot Q)(T)$.
	\end{ques}

	Let $n = \dim V < \infty$ and fix any nonzero vector $v \in V$,
	and consider vectors $v$, $T(v)$, \dots, $T^n (v)$.
	There are $n+1$ of them,
	so they can't be linearly independent for dimension reasons;
	thus there is a nonzero polynomial $P$ such that $P(T)$
	is zero when applied to $v$.
	WLOG suppose $P$ is a monic polynomial,
	and thus $P(z) = (z-r_1)\dots(z-r_m)$ say.
	Then we get
	\[ 0 = (T - r_1 \id) \circ (T - r_2 \id) \circ \dots
		\circ (T - r_m \id)(v) \]
	(where $\id$ is the identity matrix).  This means at least one of
	$T - r_i \id$ is not injective, i.e.\ has a nontrivial kernel,
	which is the same as an eigenvector.
\end{proof}
So in general we like to consider algebraically closed fields.
This is not a big loss:
any real matrix can be interpreted as a complex matrix
whose entries just happen to be real, for example.

\section{The Jordan form}
So that you know exactly where I'm going,
here's the main theorem.
\begin{definition}
	A \vocab{Jordan block} is an $n \times n$ matrix of the following shape:
	\[
		\begin{bmatrix}
			\lambda & 1 & 0 & 0 & \dots & 0 & 0 \\
			0 & \lambda & 1 & 0 & \dots & 0 & 0 \\
			0 & 0 & \lambda & 1 & \dots & 0 & 0 \\
			0 & 0 & 0 & \lambda & \dots & 0 & 0 \\
			\vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\
			0 & 0 & 0 & 0 & \dots & \lambda & 1 \\
			0 & 0 & 0 & 0 & \dots & 0 & \lambda
		\end{bmatrix}.
	\]
	In other words, it has $\lambda$ on the diagonal,
	and $1$ above it.
	We allow $n = 1$,
	so $\begin{bmatrix} \lambda \end{bmatrix}$ is a Jordan block.
\end{definition}

\begin{theorem}
	[Jordan canonical form]
	Let $T \colon V \to V$ be a linear map
	of finite-dimensional vector spaces
	over an algebraically closed field $k$.
	Then we can choose a basis of $V$
	such that the matrix $T$ is ``block-diagonal''
	with each block being a Jordan block.

	Such a matrix is said to be in \vocab{Jordan form}.
	This form is unique up to rearranging the order of the blocks.
\end{theorem}
As an example, this means the matrix should look something like:
\[
	\begin{bmatrix}
		\lambda_1 & 1 \\
		0 & \lambda_1 \\
		&& \lambda_2 \\
		&&& \lambda_3 & 1 & 0 \\
		&&& 0 & \lambda_3 & 1 \\
		&&& 0 & 0 & \lambda_3 \\
		&&&&&& \ddots \\
		&&&&&&& \lambda_m & 1 \\
		&&&&&&& 0 & \lambda_m
	\end{bmatrix}
\]
\begin{ques}
	Check that diagonal matrices are the special case
	when each block is $1 \times 1$.
\end{ques}

What does this mean?
Basically, it means \emph{our dream is almost true}.
What happens is that $V$ can get broken down as a direct sum
\[ V = J_1 \oplus J_2 \oplus \dots \oplus J_m \]
and $T$ acts on each of these subspaces independently.
These subspaces correspond to the blocks in the matrix above.
In the simplest case, $\dim J_i = 1$,
so $J_i$ has a basis element $e$
for which $T(e) = \lambda_i e$;
in other words, we just have a simple eigenvalue.
But on occasion, the situation is not quite so simple,
and we have a block of size greater than $1$;
this leads to $1$'s just above the diagonals.

I'll explain later how to interpret the $1$'s,
when I make up the word \emph{descending staircase}.
For now, you should note that even if $\dim J_i \ge 2$,
we still have a basis element
which is an eigenvector with eigenvalue $\lambda_i$.

\begin{example}
	[A concrete example of Jordan form]
	Let $T : k^6 \to k^6$ and suppose $T$ is given by the matrix
	\[
		T = \begin{bmatrix}
			5 & 0 & 0 & 0 & 0 & 0 \\
			0 & 2 & 1 & 0 & 0 & 0 \\
			0 & 0 & 2 & 0 & 0 & 0 \\
			0 & 0 & 0 & 7 & 0 & 0 \\
			0 & 0 & 0 & 0 & 3 & 0 \\
			0 & 0 & 0 & 0 & 0 & 3 \\
		\end{bmatrix}.
	\]
	Reading the matrix, we can compute all the eigenvectors and eigenvalues:
	for any constants $a, b \in k$ we have
	\begin{align*}
		T(a \cdot e_1) &= 5a \cdot e_1 \\
		T(a \cdot e_2) &= 2a \cdot e_2 \\
		T(a \cdot e_4) &= 7a \cdot e_4 \\
		T(a \cdot e_5 + b \cdot e_6) &= 3\left[ a \cdot e_5 + b \cdot e_6 \right].
	\end{align*}
	The element $e_3$ on the other hand,
	is not an eigenvector since $T(e_3) = e_2 + 2e_3$.
\end{example}

%Here's the idea behind the proof.
%We would like to be able to break down $V$ directly into components as above,
%but some of the $\lambda_i$'s from the different Jordan blocks could coincide,
%and this turns out to give a technical difficulty for detecting them.
%So instead, we're going to break down $V$ first into subspaces
%based on the values of $\lambda$'s;
%these are called the \emph{generalized eigenspaces}.
%In other words, the generalized eigenspace of a given $\lambda$
%is just all the Jordan blocks which have eigenvalue $\lambda$.
%Only after that will we break the generalized eigenspace
%into the individual Jordan blocks.
%
%The sections below record this proof in the other order:
%the next part deals with breaking generalized
%eigenspaces into Jordan blocks,
%and the part after that is the one where we break down $V$ into
%generalized eigenspaces.

\section{Nilpotent maps}
Bear with me for a moment.  First, define:
\begin{definition}
	A map $T: V \to V$ is \vocab{nilpotent} if $T^m$ is the zero map for some integer $m$.
	(Here $T^m$ means ``$T$ applied $m$ times''.)
\end{definition}
What's an example of a nilpotent map?
\begin{example}
	[The ``descending staircase'']
        Let $V = k^{\oplus 3}$ have basis $e_1$, $e_2$, $e_3$.
	Then the map $T$ which sends
	\[ e_3 \mapsto e_2 \mapsto e_1 \mapsto 0 \]
	is nilpotent, since $T(e_1) = T^2(e_2) = T^3(e_3) = 0$,
	and hence $T^3(v) = 0$ for all $v \in V$.
\end{example}
The $3 \times 3$ descending staircase has matrix representation
\[ T = \begin{bmatrix}
		0 & 1 & 0 \\
		0 & 0 & 1 \\
		0 & 0 & 0
	\end{bmatrix}. \]
You'll notice this is a Jordan block.
\begin{exercise}
	Show that the descending staircase above
	has $0$ as its only eigenvalue.
\end{exercise}

That's a pretty nice example.
As another example, we can have multiple such staircases.
\begin{example}
	[Double staircase]
	Let $V = k^{\oplus 5}$ have basis $e_1$, $e_2$, $e_3$, $e_4$, $e_5$.
	Then the map
	\[ e_3 \mapsto e_2 \mapsto e_1 \mapsto 0 \text{ and }
		e_5 \mapsto e_4 \mapsto 0 \]
	is nilpotent.
\end{example}
Picture, with some zeros omitted for emphasis:
\[ T = \begin{bmatrix}
		0 & 1 & 0 &   &   \\
		0 & 0 & 1 &   &   \\
		0 & 0 & 0 &   &   \\
		  &   &   & 0 & 1 \\
		  &   &   & 0 & 0 \\
	\end{bmatrix}
\]
You can see this isn't really that different
from the previous example;
it's just the same idea repeated multiple times.
And in fact we now claim that \emph{all}
nilpotent maps have essentially that form.
\begin{theorem}
	[Nilpotent Jordan]
	Let $V$ be a finite-dimensional vector space
	over an algebraically closed field $k$.
	Let $T \colon V \to V$ be a nilpotent map.
	Then we can write $V = \bigoplus_{i=1}^m V_i$
	where each $V_i$ has a basis of the form
	$v_i$, $T(v_i)$, \dots, $T^{\dim V_i - 1}(v_i)$
	for some $v_i \in V_i$.
\end{theorem}
Hence:
\begin{moral}
	Every nilpotent map can be viewed as independent staircases.
\end{moral}
Each chain $v_i$, $T(v_i)$, $T(T(v_i))$, \dots is just one staircase.
The proof is given later, but first let me point out where this is going.

Here's the punch line.
Let's take the double staircase again.
Expressing it as a matrix gives, say
\[
	S = \begin{bmatrix}
		0 & 1 & 0 &   &   \\
		0 & 0 & 1 &   &   \\
		0 & 0 & 0 &   &   \\
		  &   &   & 0 & 1 \\
		  &   &   & 0 & 0
	\end{bmatrix}.
\]
Then we can compute
\[
	S + \lambda \id = \begin{bmatrix}
		\lambda & 1 & 0 &   &   \\
		0 & \lambda & 1 &   &   \\
		0 & 0 & \lambda &   &   \\
		  &   &   & \lambda & 1 \\
		  &   &   & 0 & \lambda
	\end{bmatrix}.
\]
It's a bunch of $\lambda$ Jordan blocks!
This gives us a plan to proceed: we need to break $V$ into
a bunch of subspaces such that $T - \lambda \id$ is nilpotent over each subspace.
Then Nilpotent Jordan will finish the job.

\section{Reducing to the nilpotent case}
\begin{definition}
	Let $T \colon V \to V$. A subspace $W \subseteq V$
	is called $T$-\vocab{invariant}
	if $T(w) \in W$ for any $w \in W$.
	In this way, $T$ can be thought of as a map $W \to W$.
\end{definition}
In this way, the Jordan form is a decomposition of $V$ into invariant subspaces.

Now I'm going to be cheap, and define:
\begin{definition}
	A map $T \colon V \to V$ is called \vocab{indecomposable}
	if it's impossible to write $V = W_1 \oplus W_2$
	where both $W_1$ and $W_2$ are nontrivial $T$-invariant spaces.
\end{definition}
Picture of a \emph{decomposable} map:
\[
	\begin{bmatrix}
		\multicolumn{2}{c|}{\multirow{2}{*}{$W_1$}} & 0 & 0 & 0  \\
		\multicolumn{2}{c|}{} & 0 & 0 & 0 \\ \hline
		0 & 0 & \multicolumn{3}{|c}{\multirow{3}{*}{$W_2$}} \\
		0 & 0 & \multicolumn{3}{|c}{} \\
		0 & 0 & \multicolumn{3}{|c}{}
	\end{bmatrix}
\]
As you might expect, we can break a space apart into ``indecomposable'' parts.
\begin{proposition}
	[Invariant subspace decomposition]
	Let $V$ be a finite-dimensional vector space.
	Given any map $T \colon V \to V$, we can write
	\[ V = V_1 \oplus V_2 \oplus \dots \oplus V_m \]
	where each $V_i$ is $T$-invariant,
	and for any $i$ the map $T \colon V_i \to V_i$ is indecomposable.
\end{proposition}
\begin{proof}
	Same as the proof that every integer is the product of primes.
	If $V$ is not decomposable, we are done.
	Otherwise, by definition write $V = W_1 \oplus W_2$
	and then repeat on each of $W_1$ and $W_2$.
\end{proof}

Incredibly, with just that we're almost done!
Consider a decomposition as above,
so that $T \colon V_1 \to V_1$ is an indecomposable map.
Then $T$ has an eigenvalue $\lambda_1$, so let $S = T - \lambda_1 \id$; hence $\ker S \neq \{0\}$.
\begin{ques}
	Show that $V_1$ is also $S$-invariant, so we can consider $S : V_1 \to V_1$.
\end{ques}
By \Cref{prob:endomorphism_eventual_lemma}, we have
\[ V_1 = \ker S^N \oplus \img S^N \]
for some $N$.
But we assumed $T$ was indecomposable,
so this can only happen if $\img S^N = \{0\}$ and $\ker S^N = V_1$
(since $\ker S^N$ contains our eigenvector).
Hence $S$ is nilpotent, so it's a collection of staircases.
In fact, since $T$ is indecomposable, there is only one staircase.
Hence $V_1$ is a Jordan block, as desired.

\section{(Optional) Proof of nilpotent Jordan}
The proof is just induction on $\dim V$.
Assume $\dim V \ge 1$, and let $W = T\im(V)$ be the image of $V$.
Since $T$ is nilpotent, we must have $W \subsetneq V$.
Moreover, if $W = \{0\}$ (i.e.\ $T$ is the zero map) then we're already done.
So assume $\{0\} \subsetneq W \subsetneq V$.

By the inductive hypothesis, we can select a good basis of $W$:
\begin{align*}
	\mathcal B' =
	\Big\{ & T(v_1), T(T(v_1)), T(T(T(v_1))), \dots \\
	& T(v_2), T(T(v_2)), T(T(T(v_2))), \dots \\
	& \dots, \\
	& T(v_\ell), T(T(v_\ell)), T(T(T(v_\ell))), \dots \Big\}
\end{align*}
for some $T(v_i) \in W$ (here we have taken advantage of the fact that each element of $W$ is itself of the form $T(v)$ for some $v$).

Also, note that there are exactly $\ell$ elements of $\mathcal B'$ which are in $\ker T$
(namely the last element of each of the $\ell$ staircases).
We can thus complete it to a basis $v_{\ell+1}, \dots, v_m$ (where $m = \dim \ker T$).
(In other words, the last element of each staircase plus the $m-\ell$ new ones are a basis for $\ker T$.)

Now consider
\begin{align*}
	\mathcal B =
	\Big\{ & v_1, T(v_1), T(T(v_1)), T(T(T(v_1))), \dots \\
	& v_2, T(v_2), T(T(v_2)), T(T(T(v_2))), \dots \\
	& \dots, \\
	& v_\ell, T(v_\ell), T(T(v_\ell)), T(T(T(v_\ell))), \dots \\
	& v_{\ell+1}, v_{\ell+2}, \dots, v_m \Big\}.
\end{align*}
\begin{ques}
Check that there are exactly $\ell + \dim W + (\dim \ker T - \ell) = \dim V$ elements.
\end{ques}
\begin{exercise}
	Show that all the elements are linearly independent.
	(Assume for contradiction there is some linear dependence,
	then take $T$ of both sides.)
\end{exercise}
Hence $\mathcal B$ is a basis of the desired form.

\section{Algebraic and geometric multiplicity}
\prototype{The matrix $T$ below.}
This is some convenient notation:
let's consider the matrix in Jordan form
\[
	T =
	\begin{bmatrix}
		7 & 1 \\
		0 & 7 \\
		& & 9 \\
		& & & 7 & 1 & 0 \\
		& & & 0 & 7 & 1 \\
		& & & 0 & 0 & 7
	\end{bmatrix}.
\]
We focus on the eigenvalue $7$,
which appears multiple times, so it is certainly ``repeated''.
However, there a two different senses in which you could say it is repeated.
\begin{itemize}
	\ii \emph{Algebraic}: You could say it is repeated five times,
	because it appears five times on the diagonal.
	\ii \emph{Geometric}: You could say it really only appears two times:
	because there are only two eigen\emph{vectors}
	with eigenvalue $7$, namely $e_1$ and $e_4$.

	Indeed, the vector $e_2$ for example has $T(e_2) = 7e_2 + e_1$,
	so it's not really an eigenvector!
	If you apply $T - 7\id$ to $e_2$ twice though,
	you do get zero.
\end{itemize}
\begin{ques}
	In this example,
	how many times do you need to apply $T - 7\id$ to $e_6$ to get zero?
\end{ques}
Both these notions are valid,
so we will name both.
To preserve generality,
we first state the ``intrinsic'' definition.
\begin{definition}
	Let $T \colon V \to V$ be a linear map and $\lambda$ a scalar.
	\begin{itemize}
		\ii The \vocab{geometric multiplicity}
		of $\lambda$ is the dimension $\dim V_\lambda$
		of the $\lambda$-eigenspace.

		\ii Define the \vocab{generalized eigenspace}
		$V^\lambda$ to be the subspace of $v$
		for which $(T-\lambda \id)^n(v) = 0$ for some $n \ge 1$.
		The \vocab{algebraic multiplicity} of $\lambda$ is the
		dimension $\dim V^\lambda$.
	\end{itemize}
	(Silly edge case: we allow ``multiplicity zero''
	if $\lambda$ is not an eigenvalue at all.)
\end{definition}
However in practice you should just count the Jordan blocks.
\begin{example}
	[An example of eigenspaces via Jordan form]
	Retain the matrix $T$ mentioned earlier and let $\lambda = 7$.
	\begin{itemize}
		\ii The eigenspace $V_\lambda$ has basis $e_1$ and $e_4$,
		so the geometric multiplicity is $2$.
		\ii The generalized eigenspace $V^\lambda$ has basis $e_1$, $e_2$,
		$e_4$, $e_5$, $e_6$ so the algebraic multiplicity is $5$.
	\end{itemize}
\end{example}

To be completely explicit, here is how you think of these in practice:
\begin{proposition}
	[Geometric and algebraic multiplicity vs Jordan blocks]
	Assume $T \colon V \to V$ is a linear map
	of finite-dimensional vector spaces,
	written in Jordan form.
	Let $\lambda$ be a scalar.
	Then
	\begin{itemize}
		\ii The geometric multiplicity of $\lambda$ is the number
		of Jordan blocks with eigenvalue $\lambda$;
		the eigenspace has one basis element per Jordan block.

		\ii The algebraic multiplicity of $\lambda$ is the
		sum of the dimensions of the Jordan blocks
		with eigenvalue $\lambda$;
		the eigenspace is the direct sum of the subspaces
		corresponding to those blocks.
	\end{itemize}
\end{proposition}

\begin{ques}
	Show that the geometric multiplicity
	is always less than or equal to the algebraic multiplicity.
\end{ques}

This actually gives us a tentative definition:
\begin{itemize}
	\ii The trace is the sum of the eigenvalues,
	counted with algebraic multiplicity.
	\ii The determinant is the product of the eigenvalues,
	counted with algebraic multiplicity.
\end{itemize}
This definition is okay,
but it has the disadvantage of requiring the ground
field to be algebraically closed.
It is also not the definition that is easiest
to work with computationally.
The next two chapters will give us a better definition.

\section\problemhead

\begin{problem}
	[Sum of algebraic multiplicities]
	Given a $2018$-dimensional complex vector space $V$
	and a map $T \colon V \to V$,
	what is the sum of the algebraic multiplicities
	of all eigenvalues of $T$?
	\begin{sol}
		It's just $\dim V = 2018$.
		After all, you are adding the dimensions of the Jordan blocks\dots
	\end{sol}
\end{problem}

\begin{problem}
	[The word ``diagonalizable'']
	A linear map $T \colon V \to V$ (where $\dim V$ is finite)
	is said to be \vocab{diagonalizable}
	if it has a basis $e_1$, \dots, $e_n$
	such that each $e_i$ is an eigenvector.
	\begin{enumerate}[(a)]
		\ii Explain the name ``diagonalizable''.
		\ii Suppose we are working over an algebraically closed field.
		Then show that that $T$ is diagonalizable if and only if
		for any $\lambda$, the geometric multiplicity of $\lambda$
		equals the algebraic multiplicity of $\lambda$.
	\end{enumerate}
	\begin{sol}
		(a): if you express $T$ as a matrix in such a basis,
		one gets a diagonal matrix.
		(b): this is just saying each Jordan block has dimension $1$,
		which is what we wanted.
		(We are implicitly using uniqueness of Jordan form here.)
	\end{sol}
\end{problem}

\begin{problem}
	[Switcharoo]
	Let $V$ be the $\CC$-vector space
	with basis $e_1$ and $e_2$.
	The map $T \colon V \to V$ sends $T(e_1) = e_2$ and $T(e_2) = e_1$.
	Determine the eigenspaces of $T$.
	\begin{sol}
		The $+1$ eigenspace is spanned by $e_1+e_2$.
		The $-1$ eigenspace is spanned by $e_1-e_2$.
	\end{sol}
\end{problem}

\begin{problem}
	[Writing a polynomial backwards]
	Define the complex vector space $V$
	of polynomials with degree at most $2$,
	say $V = \left\{ ax^2 + bx + c \mid a,b,c \in \CC \right\}$.
	Define $T \colon V \to V$ by
	\[ T(ax^2+bx+c) = cx^2+bx+a. \]
	Determine the eigenspaces of $T$.
	\begin{sol}
		The $+1$ eigenspace is spanned by $1+x^2$ and $x$.
		The $-1$ eigenspace is spanned by $1-x^2$.
	\end{sol}
\end{problem}

\begin{problem}
	[Differentiation of polynomials]
	Let $V = \RR[x]$ be the real vector space of all real polynomials.
	Note that $\frac{d}{dx} \colon V \to V$ is a linear map
	(for example it sends $x^3$ to $3x^2$).
	Which real numbers are eigenvalues of this map?
	\begin{hint}
		Only $0$ is.
		Look at degree.
	\end{hint}
	\begin{sol}
		Constant functions differentiate to zero,
		and these are the only $0$-eigenvectors.
		There can be no other eigenvectors,
		since if $\deg p > 0$ then $\deg p' = \deg p - 1$,
		so if $p'$ is a constant real multiple of $p$
		we must have $p' = 0$, ergo $p$ is constant.
	\end{sol}
\end{problem}

\begin{problem}
	[Differentiation of functions]
	Let $V$ be the real vector space of all
	infinitely differentiable functions $\RR \to \RR$.
	Note that $\frac{d}{dx} \colon V \to V$ is a linear map
	(for example it sends $\cos x$ to $-\sin x$).
	Which real numbers are eigenvalues of this map?
	\begin{hint}
		All of them are!
	\end{hint}
	\begin{sol}
		$e^{cx}$ is an example of a $c$-eigenvector for every $c$.
		If you know differential equations,
		these generate all examples!
	\end{sol}
\end{problem}


\documentclass[]{article}

%\usepackage{showframe} % To render a frame marking the margins

\usepackage{tikz}
\usetikzlibrary{
    angles,
    arrows.meta,
    automata, % to use \node[state]
    graphs,
    intersections,
    quotes,
    positioning
}
\usepackage{amsmath, amssymb} % for $\therefore e_{l} * e_{r} \implies e_{r}$

% epigraph for quote at start of section
\usepackage{epigraph}
\setlength\epigraphwidth{8cm}
\setlength\epigraphrule{0pt}
\renewcommand{\epigraphsize}{\small\itshape}

% for hyperlinks and URL links
\usepackage{hyperref}

\title{\LaTeX{} Experiments\\ Part III: PGF/TikZ}

\author{AeAeA}

\begin{document}

\maketitle

\vspace{20pt}

\begin{figure}[h] \centering
    \begin{tikzpicture}
        \draw[gray, thick, ->] (-1,2) -- (2,-4);
        \draw[gray, thick] (-1,-1) -- (2,2);
        \filldraw[black] (0,0) circle (2pt) node[anchor=west] {Intersection point};
    \end{tikzpicture}
\end{figure}

%==============================================================================
\section{TikZ ist {\itshape kein} Zeichenprogramm}

\epigraph
{Für meinen Vater, damit er noch viele schöne TEX-Graphiken erschaffen kann.}
{--- \textup{Till Tantau}, PGF Manual}

\begin{itemize}
    \item \url{https://en.wikipedia.org/wiki/PGF/TikZ}
    \item \url{https://www.ctan.org/pkg/pgf}
    \item \url{https://github.com/pgf-tikz/pgf}
    \item \href{http://cremeronline.com/LaTeX/minimaltikz.pdf}
               {Minimal introduction to TikZ (unofficial)}
    \item  It comes with very good documentation; the version 3.1.5b of the
           \href{http://mirrors.ctan.org/graphics/pgf/base/doc/pgfmanual.pdf}
                {PGF Manual} has over 1,300 pages (!) \ldots
    \item \ldots and an extensive collection of examples: \\
          \url{http://www.texample.net/tikz/}
    \item \url{https://en.wikibooks.org/wiki/LaTeX/PGF/TikZ}
    \item \url{https://www.overleaf.com/learn/latex/TikZ_package}
\end{itemize}

%--------------------------------------
\subsection{Basic elements: points, lines and paths}

\begin{tikzpicture}
    \draw (-2,0) -- (2,0);
    \filldraw [gray] (0,0) circle (2pt);
    \draw[very thick, <->] (-2,-2) .. controls (0,0) .. (2,-2);
    \draw (-2,2) .. controls (-1,0) and (1,0) .. (2,2);
\end{tikzpicture}

%--------------------------------------
\subsection{Basic geometric shapes: Circles, ellipses and polygons}

\begin{tikzpicture}
    \filldraw[color=red!60, fill=red!7, very thick] (-1,0) circle (1.5);
    \fill[blue!10] (2.5,0) ellipse (1.5 and 0.5);
    \fill[blue]    (2.5,0) circle (2pt);
    \draw (2.5,0) ellipse [x radius=1.5, y radius=0.5];
    \draw[ultra thick, ->] (6.5,0) arc (0:270:1);
\end{tikzpicture}

\vspace{20pt}

\begin{tikzpicture}
    \draw[blue, very thick] (0,0) rectangle (3,2);
    \draw[orange, ultra thick] (4,0) -- (6,0) -- (5.7,2) -- cycle;
\end{tikzpicture}

The code for the little "turned" ellipse
\tikz \draw[rotate=30] (0,0) ellipse [x radius=6pt,y radius=3pt]; is \\
\verb+\tikz \draw[rotate=30] (0,0) ellipse [x radius=6pt,y radius=3pt];+

%--------------------------------------
\subsection{Elliptical arc}
%  \usetikzlibrary {arrows.meta}
\begin{tikzpicture}[>=Stealth]
    \draw[<->>,very thick]
        (0,0) arc [start angle=0, end angle=315, x radius=1.75cm, y radius=1cm];
\end{tikzpicture}

%--------------------------------------
\subsection{Arrow tips}

\marginpar{\texttt{Stealth}\\
\tikz [>=Stealth] \draw[<<<<-,ultra thick] (0,0) -- (0.1pt,0);}

\epigraph
{Karl wonders whether such a military name for the arrow type is really necessary.
He is not really mollified when his son tells him that Microsoft’s PowerPoint
uses the same name. He decides to have his students discuss this at some point.}
{--- \textup{Till Tantau}, PGF Manual}

This is an example of \texttt{Stealth} arrow tip
\tikz [>=Stealth]
    \draw[<<-,ultra thick] (1,0) -- (1.5cm,10pt) -- (2cm,0pt) -- (2.5cm,10pt);
which is a “stealth-fighter-like”.

All arrow tips:\\
%  \usetikzlibrary {arrows.meta}
\begin{tikzpicture} [->,very thick]
    \draw[>=Arc Barb]      (0,0) -- (0,10pt);
    \draw[>=Bar]           (10pt,0) -- (10pt,10pt);
    \draw[>=Bracket]       (20pt,0) -- (20pt,10pt);
    \draw[>=Butt Cap]      (30pt,0) -- (30pt,10pt);
    \draw[>=Circle]        (40pt,0) -- (40pt,10pt);
    \draw[>=Diamond]       (50pt,0) -- (50pt,10pt);
    \draw[>=Ellipse]       (60pt,0) -- (60pt,10pt);
    \draw[>=Fast Round]    (70pt,0) -- (70pt,10pt);
    \draw[>=Fast Triangle] (80pt,0) -- (80pt,10pt);
    \draw[>=Hooks]         (90pt,0) -- (90pt,10pt);
    \draw[>=Implies]       (100pt,0) -- (100pt,10pt);
    \draw[>=Kite]          (110pt,0) -- (110pt,10pt);
    \draw[>=LaTeX]         (120pt,0) -- (120pt,10pt);
    \draw[>=Latex]         (130pt,0) -- (130pt,10pt);
    \draw[>=Parenthesis]   (140pt,0) -- (140pt,10pt);
    \draw[>=Rays]          (150pt,0) -- (150pt,10pt);
    \draw[>=Rectangle]     (160pt,0) -- (160pt,10pt);
    \draw[>=Round Cap]     (170pt,0) -- (170pt,10pt);
    \draw[>=Square]        (180pt,0) -- (180pt,10pt);
    \draw[>=Stealth]       (190pt,0) -- (190pt,10pt);
    \draw[>=Straight Barb] (200pt,0) -- (200pt,10pt);
    \draw[>=Tee Barb]      (210pt,0) -- (210pt,10pt);
    \draw[>=To]            (220pt,0) -- (220pt,10pt);
    \draw[>=Triangle]      (230pt,0) -- (230pt,10pt);
    \draw[>=Triangle Cap]  (240pt,0) -- (240pt,10pt);
    \draw[>=Turned Square] (250pt,0) -- (250pt,10pt);
\end{tikzpicture}

\marginpar{\tikz \draw[-To,red,ultra thick] (0,0) -- (0.1pt,0);}
(Almost) zero-length arrow:
\tikz \draw[-To,red,ultra thick] (0,0) -- (0.1pt,0);

\subsubsection{Arrow Tip Kind \texttt{Implies}}
\marginpar{$\therefore\\ e_{l} * e_{r} \implies e_{r}$}
This arrow tip makes only sense in conjunction with the double option:
attach it to a double line to get something
( \tikz \draw[double equal sign distance, -Implies] (0,0) -- (15pt,0); )
that looks like
\texttt{amsmath} \TeX’s \verb+\implies+ arrow ( $\implies$ ).
A typical use of this arrow tip is:\\
% \usetikzlibrary {arrows.meta,graphs}
\tikz \graph [
    clockwise=3, math nodes, edges = {double equal sign distance, -Implies}
] {
    "\alpha", "\beta", "\gamma";
    "\alpha" -> "\beta" -> "\gamma" -> "\alpha"
};

%--------------------------------------
\subsection{Path}

\tikz \draw[thick,rounded corners=8pt]
    (0,0) -- (0,2) -- (1,3.25) -- (2,2) -- (2,0) -- (0,2) -- (2,2) -- (0,0) -- (2,0);

%--------------------------------------
\subsection{Curved path}

\begin{tikzpicture}[scale=2]
    \filldraw[gray] (0,0) circle [radius=2pt]
                    (1,1) circle [radius=2pt]
                    (2,1) circle [radius=2pt]
                    (2,0) circle [radius=2pt];
    \draw[gray] (0,0) .. controls (1,1) and (2,1) .. (2,0);

    \filldraw[orange] (0.5,0.5) circle [radius=2pt]
                      (2,0.5) circle [radius=2pt];
    \draw[orange] (0,0) .. controls (0.5,0.5) and (2,0.5) .. (2,0);

    \filldraw[blue] (2,2) circle [radius=2pt];
    \draw[blue] (0,0) .. controls (2,2) .. (2,0);

    \filldraw[red] (3,2) circle [radius=2pt]
                   (-1,2) circle [radius=2pt];
    \draw[red] (0,0) .. controls (3,2) and (-1,2) .. (2,0);
\end{tikzpicture}

%--------------------------------------
\subsection{Grid}
\marginpar{\tikz \draw[step=2pt] (0,0) grid (10pt,10pt);}
The code \verb+\tikz \draw[step=2pt] (0,0) grid (10pt,10pt);+ produces
\tikz \draw[step=2pt] (0,0) grid (10pt,10pt);.

Here is a bigger grid:\\
\tikz{
    \fill[orange] (0,0) circle (2pt);
    \fill[red] (50pt,50pt) circle (2pt);
    \fill[blue] (100pt,100pt) circle (2pt);

    \draw[step=5pt,gray,very thin] (1pt,1pt) grid (99pt,99pt);

    \draw (50pt,0) -- (50pt,100pt); % vertical
    \draw (0,50pt) -- (100pt,50pt); % horizontal
}

%--------------------------------------
\subsection{Circle and curved path}
\marginpar{0.555}
0.555 is the magic number.\\
\begin{tikzpicture} [
    dark green/.style = {green!50!black}
]
    \draw[step=.5cm,gray,very thin] (-5.4,-5.4) grid (5.4,5.4);

    \draw (-5.5,0) -- (5.5,0);
    \draw (0,-5.5) -- (0,5.5);

    \draw[gray,thick] (0,0) circle [radius=5cm];

    % 2.777 = 5 * 0.555(5)

    \fill[blue] (-5,2.777) circle (1pt) (-2.777,5) circle (1pt);
    \draw[blue,->] (-5,0) .. controls (-5,2.777) and (-2.777,5) .. (0,5);

    \fill[orange] (3,5) circle (1pt) (5,3) circle (1pt);
    \draw[orange,->] (0,5) .. controls (3,5) and (5,3) .. (5,0);

    \fill[dark green] (5,-5) circle (1pt);
    \draw[dark green,->] (5,0) .. controls (5,-5) .. (0,-5);

    \fill[red] (-2,-5) circle (1pt) (-5,-2) circle (1pt);
    \draw[red,->] (0,-5) .. controls (-2,-5) and (-5,-2) .. (-5,0);
\end{tikzpicture}

%--------------------------------------
\subsection{Clipping a path}
Original drawing on the left, clipped (and 1.4 scaled) drawing on the right:\\
\begin{tikzpicture}
    \draw[step=.5cm,gray,very thin] (-1.4,-1.4) grid (1.4,1.4);
    \draw (-1.5,0) -- (1.5,0);
    \draw (0,-1.5) -- (0,1.5);
    \draw[thick] (0,0) circle (1);
    \draw (0.3,0) arc [start angle=0, end angle=30, radius=0.3];
\end{tikzpicture}
\begin{tikzpicture}[scale=1.4]
    \path[draw,clip] (0.5,0.5) circle (1);
    \draw[step=.5cm,gray,very thin] (-1.4,-1.4) grid (1.4,1.4);
    \draw (-1.5,0) -- (1.5,0);
    \draw (0,-1.5) -- (0,1.5);
    \draw[thick] (0,0) circle (1);
    \draw (0.3,0) arc [start angle=0, end angle=30, radius=0.3];
\end{tikzpicture}

%--------------------------------------
\subsection{Parabola and Sine}
\marginpar{
    \tikz \draw[x=1.57ex,y=1ex] (0,0) sin (1,1) cos (2,0) sin (3,-1) cos (4,0)
(0,1) cos (1,0) sin (2,-1) cos (3,0) sin (4,1);
}
Two parabolas
\tikz \draw[x=2ex,y=2ex] (-1,0) rectangle (1,1) (-1,1) parabola (0,0) parabola (1,1);
and a parabola with placed bend
\tikz \draw[x=2ex,y=2ex] (-1,0) rectangle (1,1) (-1,1) parabola bend (0,0) (1,1);

A parabola with the bend:
\tikz \draw[x=1pt,y=1pt] (0,0) parabola bend (4,16) (6,12);

A sine \tikz \draw[x=1ex,y=1ex] (0,0) sin (1.57,1); curve,
and a longer span of sine and cosine:
\tikz \draw[x=1.57ex,y=1ex] (0,0) sin (1,1) cos (2,0) sin (3,-1) cos (4,0)
                            (0,1) cos (1,0) sin (2,-1) cos (3,0) sin (4,1);

%--------------------------------------
\subsection{Closing the path}
The \texttt{--cycle} causes the current path to be closed (actually the current part of
the current path) by smoothly joining the first and last point. To appreciate
the difference, consider the following example:\\
\begin{tikzpicture}[line width=5pt]
    \draw (0,0) -- (1,0) -- (1,1) -- (0,0);
    \draw (2,0) -- (3,0) -- (3,1) -- cycle;
    \useasboundingbox (0,1.5); % make bounding box higher
\end{tikzpicture}

%--------------------------------------
\subsection{Shading}
\marginpar{\tikz \shade[ball color=green] (.5,.5) circle (.5);}
The default shading is a smooth transition from gray at the top
to white at the bottom:\\
\tikz \shade (0,0) rectangle (2,1)  (3,0.5) circle (.5cm);

To specify different colors, you can use options:\\
\begin{tikzpicture}[rounded corners,ultra thick]
    \shade[top color=yellow,bottom color=black] (0,0) rectangle +(2,1);
    \shade[left color=yellow,right color=black] (3,0) rectangle +(2,1);
    \shadedraw[inner color=yellow,outer color=black,draw=yellow] (6,0) rectangle +(2,1);
    \shade[ball color=green] (9,.5) circle (.5);
\end{tikzpicture}

%--------------------------------------
\subsection{Scoping}
\begin{tikzpicture}[ultra thick]
    \draw (0,0) -- (0,1);
    \begin{scope}[thin]
        \draw (1,0) -- (1,1);
        \draw (2,0) -- (2,1);
    \end{scope}
    \draw (3,0) -- (3,1);
\end{tikzpicture}

\newpage
%==============================================================================
\section{A picture for Karl}

%--------------------------------------
\subsection{Style}
\tikzset{help lines/.style=very thin}
\begin{tikzpicture}[
    Karl's grid/.style ={help lines,color=#1!50},
    Karl's grid/.default=blue
]

    \draw[Karl's grid]     (0,0) grid (1.5,2);
    \draw[Karl's grid=red] (2,0) grid (3.5,2);
\end{tikzpicture}

%--------------------------------------
\subsection{Adding Text}
\begin{tikzpicture}
    \draw (0,0) rectangle (2,2);
    \draw (0.5,0.5) node [fill=yellow!80!black]
                           {Text at \verb!node 1!}
         -- (1.5,1.5) node {Text at \verb!node 2!};
\end{tikzpicture}

If the label directly after the \verb+--+ and before the coordinate,
this places the label in the middle of the line, but the \verb+pos=+
options can be used to modify this. Also, options like \verb+near start+
and \verb+near end+ can be used to modify this position. You can also
position labels on curves and, by adding the \verb+sloped+ option,
have them rotated such that they match the line’s slope.
Here is an example:\\
\begin{tikzpicture}
    \draw (0,0) .. controls (6,1) and (9,1) ..
      node[near start,sloped,above] {near start}
      node {midway}
      node[very near end,sloped,below] {very near end} (12,0);
\end{tikzpicture}

%--------------------------------------
\subsection{Diagrams with nodes}

\begin{tikzpicture}[
    roundnode/.style   = {circle, draw=green!60!black, fill=green!5, very thick,
                          minimum size=20pt},
    squarednode/.style = {rectangle, draw=red!60, fill=red!5, very thick,
                          minimum size=15pt},
]
    %Nodes
    \node[squarednode]      (maintopic)                              {2};
    \node[roundnode]        (uppercircle)       [above=of maintopic] {1};
    \node[squarednode]      (rightsquare)       [right=of maintopic] {3};
    \node[roundnode]        (lowercircle)       [below=of maintopic] {4};

    %Lines
    \draw[->] (uppercircle.south) -- (maintopic.north);
    \draw[->] (maintopic.east) -- (rightsquare.west);
    \draw[->] (rightsquare.south) ..
              controls +(down:20pt) and +(right:20pt)
              .. (lowercircle.east);
\end{tikzpicture}

\subsubsection{Extracting Insights From Data diagram}
\begin{tikzpicture}[
    squarednode/.style = {rectangle, draw=orange!60!black, fill=orange!7,
                          very thick,
                          minimum size=20pt,
                          rounded corners=2pt},
    impliesarrow/.style = {double equal sign distance, -Implies}
]

    \node[squarednode] (DK) {Domain Knowledge};
    \node[squarednode] (SA) [below=of DK] {Schema \& Algorithms};
    \node[squarednode] (SP) [right=of SA] {Production System};
    \node[squarednode] (EA) [above=of SP] {Exploratory Analysis};

    \draw[impliesarrow] (DK.south) -- (SA.north);
    \draw[impliesarrow] (SA.east) -- (SP.west);
    \draw[impliesarrow] (SP.north) -- (EA.south);
    \draw[impliesarrow] (EA.west) -- (DK.east);
    % \draw[impliesarrow] (DA.north) .. controls +(up:30pt) and +(right:40pt) .. (DK.east);

\end{tikzpicture}

%--------------------------------------
\subsection{Specifying Coordinates}
To appreciate the difference between + and ++ consider the following example:\\
\verb|-- ++(1cm,0cm)  -- ++(0cm,1cm)  -- ++(-1cm,0cm) -- cycle|\\
\begin{tikzpicture}
    \def\rectanglepath{-- ++(1cm,0cm)  -- ++(0cm,1cm)  -- ++(-1cm,0cm) -- cycle}
    \draw (0,0) \rectanglepath;
    \draw (1.5,0) \rectanglepath;
\end{tikzpicture}

By comparison, when using a single +, the coordinates are different:\\
\verb|-- +(1cm,0cm)  -- +(1cm,1cm)  -- +(0cm,1cm) -- cycle|\\
\begin{tikzpicture}
    \def\rectanglepath{-- +(1cm,0cm)  -- +(1cm,1cm)  -- +(0cm,1cm) -- cycle}
    \draw (0,0) \rectanglepath;
    \draw (1.5,0) \rectanglepath;
\end{tikzpicture}

%--------------------------------------
\subsection{Transformations}
\tikz \draw (0,0) -- (0,0.5) [xshift=20pt] (0,0) -- (0,0.5);

\marginpar{
\begin{tikzpicture}[even odd rule,rounded corners=2pt,x=10pt,y=10pt]
    \filldraw[fill=yellow!80!black]   (0,0) rectangle (1,1)
                [shift={(0.5,0.5)}]   (0,0) rectangle (1,1)
                        [rotate=30] (-1,-1) rectangle (2,2)
                                      (0,0) circle (0.2)
                  [shift={(-1,-1)}]   (0,0) circle (0.4);
\end{tikzpicture}
}

Another way to display all options for arrow tips:\\
\begin{tikzpicture}
    %  \usetikzlibrary {arrows.meta}
    \def\arrowup{[->,very thick] (0,0) -- (0,10pt)}

    \draw                                   [xshift=-40pt] \arrowup;
    \draw[>=To,red]                         [xshift=-30pt] \arrowup;
    \draw[>=Computer Modern Rightarrow,red] [xshift=-20pt] \arrowup;
    \draw[>=Classical TikZ Rightarrow]      [xshift=-10pt] \arrowup;
    \draw (0,5pt) circle (5pt); % circle to mark Arc Barb
    \draw[>=Arc Barb]                    \arrowup;
    \draw[>=Bar]           [xshift=10pt] \arrowup;
    \draw[>=Bracket]       [xshift=20pt] \arrowup;
    \draw[>=Butt Cap]      [xshift=30pt] \arrowup;
    \draw[>=Circle]        [xshift=40pt] \arrowup;
    \draw[>=Diamond]       [xshift=50pt] \arrowup;
    \draw[>=Ellipse]       [xshift=60pt] \arrowup;
    \draw[>=Fast Round]    [xshift=70pt] \arrowup;
    \draw[>=Fast Triangle] [xshift=80pt] \arrowup;
    \draw[>=Hooks]         [xshift=90pt] \arrowup;
    \draw[>=Implies]       [xshift=100pt] \arrowup;
    \draw[>=Kite]          [xshift=110pt] \arrowup;
    \draw[>=LaTeX]         [xshift=120pt] \arrowup;
    \draw[>=Latex]         [xshift=130pt] \arrowup;
    \draw[>=Parenthesis]   [xshift=140pt] \arrowup;
    \draw[>=Rays]          [xshift=150pt] \arrowup;
    \draw[>=Rectangle]     [xshift=160pt] \arrowup;
    \draw[>=Round Cap]     [xshift=170pt] \arrowup;
    \draw[>=Square]        [xshift=180pt] \arrowup;
    \draw[>=Stealth]       [xshift=190pt] \arrowup;
    \draw[>=Straight Barb] [xshift=200pt] \arrowup;
    \draw[>=Tee Barb]      [xshift=210pt] \arrowup;
    \draw[>=To]            [xshift=220pt] \arrowup;
    \draw[>=Triangle]      [xshift=230pt] \arrowup;
    \draw[>=Triangle Cap]  [xshift=240pt] \arrowup;
    \draw[>=Turned Square] [xshift=250pt] \arrowup;
\end{tikzpicture}

\newpage
%--------------------------------------
\subsection{Repeating Things: For-Loops}

\foreach \i/\t in {
    1/Arc Barb, 2/Bar, 3/Bracket, 4/Butt Cap, 5/Circle,
    6/Diamond, 7/Ellipse, 8/Fast Round, 9/Fast Triangle,
    10/Hooks, 11/Implies, 12/Kite, 13/LaTeX, 14/Latex,
    15/Parenthesis, 16/Rays, 17/Rectangle, 18/Round Cap,
    19/Square, 20/Stealth, 21/Straight Barb, 22/Tee Barb,
    23/To, 24/Triangle, 25/Triangle Cap, 26/Turned Square}
    {\i. \tikz [>=\t] \draw[->,ultra thick] (0,0) -- (15pt,0); \t \\}

\foreach \x in {1,2,3} {$x =\x$, }

\tikz  \foreach \x in {-1,-0.5,...,1} \draw[shift={(\x,0)}] (0,-5pt) -- (0,5pt);

\tikz \foreach \x in {1,...,10} \draw (\x,0) circle (0.4);

\subsubsection{2D tables}
\begin{tikzpicture}
    \filldraw[fill=yellow!80!black] (6,1) circle (2pt);
    \foreach \x in {1,2,...,5,7,8,...,12}
        \foreach \y in {1,...,5}
        {
            \draw (\x,\y) +(-.5,-.5) rectangle ++(.5,.5);
            \draw (\x,\y) node{\x,\y};
        }
\end{tikzpicture}

%--------------------------------------
\subsection{Karl's picture}
\marginpar{\begin{tikzpicture}
    \coordinate (A) at (2,0);
    \coordinate (B) at (0,0);
    \coordinate (C) at (30:2);
    \draw (A) -- (B) -- (C)
        pic [draw=green!50!black, fill=green!20, angle radius=9mm, "$\alpha$"]
            {angle = A--B--C};
\end{tikzpicture}}

\begin{tikzpicture}
    [scale=2.5,line cap=round,
    % Styles
    axes/.style=,
    grid lines/.style={gray,very thin},
    important line/.style={very thick},
    information text/.style={rounded corners,fill=red!10,inner sep=1ex}]

    % Colors
    \colorlet{anglecolor}{green!50!black}
    \colorlet{sincolor}{red}
    \colorlet{tancolor}{orange!80!black}
    \colorlet{coscolor}{blue}

    % The graphic
    \draw[grid lines,step=.5] (-1.4,-1.4) grid (1.4,1.4);
    \draw (0,0) circle [radius=1cm];

    \begin{scope}[axes]
        \draw[->] (-1.5,0) -- (1.5,0) node[right] {$x$} coordinate(x axis);
        \draw[->] (0,-1.5) -- (0,1.5) node[above] {$y$} coordinate(y axis);

        \foreach \x/\xtext in {-1, -.5/-\frac{1}{2}, 1}
            \draw[xshift=\x cm] (0pt,1pt) -- (0pt,-1pt) node[below,fill=white] {$\xtext$};
        \foreach \y/\ytext in {-1, -.5/-\frac{1}{2}, .5/\frac{1}{2}, 1}
            \draw[yshift=\y cm] (1pt,0pt) -- (-1pt,0pt) node[left,fill=white] {$\ytext$};
    \end{scope}

    \filldraw[fill=green!20,draw=anglecolor]
        (0,0) -- (3mm,0pt) arc [start angle=0, end angle=30, radius=3mm];
    \draw (15:2mm) node[anglecolor] {$\alpha$};

    \draw[important line,sincolor]
        (30:1cm) -- node[left=1pt,fill=white] {$\sin \alpha$} (30:1cm |- x axis);

    \draw[important line,coscolor]
    (30:1cm |- x axis) -- node[below=2pt,fill=white] {$\cos \alpha$} (0,0);

    % to find tan(30) "geometrically", as an intersection of two lines:
    \path [name path=upward line] (1,0) -- (1,1);
    \path [name path=sloped line] (0,0) -- (30:1.5cm);
    \draw [name intersections={of=upward line and sloped line, by=t}]
          [very thick,orange] (1,0) -- node [right=1pt,fill=white]
          {$\displaystyle \tan \alpha \color{black}=
          \frac{{\color{red}\sin \alpha}}{\color{blue}\cos \alpha}$} (t);
    \draw (0,0) -- (t);

    % text area to the left:
    \draw[xshift=-1.55cm]
        node[left,text width=3cm,information text] {
            The {\color{anglecolor} angle $\alpha$} is $30^\circ$ in the
            example ($\pi/6$ in radians).
            The {\color{sincolor}sine of
            $\alpha$}, which is the height of the red line, is
            \[ {\color{sincolor} \sin \alpha} = 1/2. \]
            By the Theorem of Pythagoras ...
        };
\end{tikzpicture}

\newpage
%==============================================================================
\section{Nodes}

\begin{tikzpicture}
    \fill (0,0) circle (1pt);
    \path ( 0,2) node [shape=circle,draw] {}
          ( 0,1) node [shape=circle,draw] {}
          ( 0,0) node [shape=circle,draw] {}
          ( 1,1) node [shape=rectangle,draw] {}
          (-1,1) node [shape=rectangle,draw] {};
  \end{tikzpicture}

%--------------------------------------
\subsection{Placing Nodes Using the At Syntax}

\begin{tikzpicture} [
    place/.style      = {circle,draw=blue!50,fill=blue!20,thick},
    transition/.style = {rectangle,draw=black!50,fill=black!20,thick}
]
    \node at ( 0,2) [place] {};
    \node at ( 0,1) [place] {};
    \node at ( 0,0) [place] {};
    \node at ( 1,1) [transition] {};
    \node at (-1,1) [transition] {};
\end{tikzpicture}

%--------------------------------------
\subsection{Node size, name and style}

\begin{tikzpicture} [inner sep=2mm,
    place/.style={circle,draw=blue!50,fill=blue!20,thick,
                  inner sep=0pt,minimum size=6mm},
    transition/.style={rectangle,draw=black!50,fill=black!20,thick,
                  inner sep=0pt,minimum size=4mm}
]
    \node[place]      (waiting 1)      at ( 0,2) {};
    \node[place]      (critical 1)     at ( 0,1) {};
    \node[place]      (semaphore)      at ( 0,0) {};
    \node[transition] (leave critical) at ( 1,1) {};
    \node[transition] (enter critical) at (-1,1) {};
\end{tikzpicture}

%--------------------------------------
\subsection{Labels and pins}

\tikz [
    every label/.style={draw,red,label distance=5mm},
    pin distance=1.5cm
] \node [
    circle,draw,
    pin = right:X,
    pin = above:Y,
    pin = below left:Z,
    label = center:*,
    label = 60:$60^\circ$,
    label = 105:$105^\circ$,
    label = 175:$175^\circ$,
    label = -40:$-40^\circ$,
    label = -95:$-95^\circ$,
    label = -170:$-170^\circ$
] {my circle};

%--------------------------------------
\subsection{Node with tabular text}

\tikz \node [draw] {
    \begin{tabular}{cc}
        upper left & upper right\\
        lower left & lower right
    \end{tabular}
};

%--------------------------------------
\subsection{Gallery}

\begin{tikzpicture}
    [scale=.8,auto=left,every node/.style={circle,fill=blue!20}]
    \node (a) at (-1,-2) {a};
    \node (b) at ( 1,-2) {b};
    \node (c) at ( 2,-1) {c};
    \node (d) at ( 2, 1) {d};
    \node (e) at ( 1, 2) {e};
    \node (f) at (-1, 2) {f};
    \node (g) at (-2, 1) {g};
    \node (h) at (-2,-1) {h};

    \foreach \from/\to in {a/b,b/c,c/d,d/e,e/f,f/g,g/h,h/a}
        \draw [->] (\from) -- (\to) node[midway,fill=red!20] {\from--\to};
\end{tikzpicture}

\vspace{20pt}

\begin{tikzpicture}[auto]
    \draw[help lines,use as bounding box] (0,-.5) grid (4,5);
    \draw (0.5,0) .. controls (9,6) and (-5,6) .. (3.5,0)
        node foreach \pos in {0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1}
            [pos=\pos,swap,fill=red!20] {\pos}
        node foreach \pos in {0.025,0.2,0.4,0.6,0.8,0.975}
            [pos=\pos,fill=blue!20] {\pos};
\end{tikzpicture}

\vspace{20pt}

% \usetikzlibrary {automata}
\begin{tikzpicture}[shorten >=1pt,node distance=2cm,auto]
    \draw[help lines] (0,0) grid (3,2);

    \node[state] (q_0) {$q_0$};
    \node[state] (q_1) [above right of=q_0] {$q_1$};
    \node[state] (q_2) [below right of=q_0] {$q_2$};
    \node[state] (q_3) [below right of=q_1] {$q_3$};

    \path[->] (q_0) edge              node        {0} (q_1)
                    edge              node [swap] {1} (q_2)
              (q_1) edge              node        {1} (q_3)
                    edge [loop above] node        {0} ()
              (q_2) edge              node [swap] {0} (q_3)
                    edge [loop below] node        {1} ();
\end{tikzpicture}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Markov chain Monte Carlo}
\begin{frame}{MCMC: The best bad method you have ever seen}
Markov chain Monte Carlo (MCMC) methods are a broad class of stochastic algorithms to compute integrals.

Suppose you are confronted with the following question: what is the ratio between the circumference of inscribed circle and its diameter?
You are \textbf{not} allowed to use any Geometry.
\begin{figure}
\includegraphics[scale=0.85]{figures/pi_MC.png}
\end{figure}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{First, a warning}
\begin{quote}
 ``Monte Carlo is an extremely bad method; it should be used only when all alternative methods are worse.''
\end{quote}
Alan Sokal (1955-) in \textit{Monte Carlo Methods in Statistical Mechanics: Foundations and New Algorithms} (1996, pg. 1).
\begin{figure}
\includegraphics[scale=0.25]{figures/tiger.jpg}
\caption{MCMC is, in a way, like a captive tiger...}
\end{figure}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Also...}
 Repeat after me,
 \begin{idea}[Bayesian MCMC is not a thing]
 \begin{center}
  {\Huge There is no such thing as ``Bayesian'' MCMC.}
 \end{center}

  MCMC is a numerical method for computing integrals.
  It does not care whether you are a Bayesian, frequentist, \textit{flamenguista} or \textit{corintiana}.
 \end{idea}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Computing integrals}
Technically, for a probability space $(X, \mathcal{F}, P)$, for $f : X \to \mathbb{R}$, we want to compute
$$
\mu_f = E_P[f] = \int_{X} f\,dP.
$$
When $P$ is absolutely continuous with respect to the Lebesgue measure, we have
$$
\mu_f= \int_{X} f(x)p(x)\,dx,
$$
as is usually written in introductory textbooks.

A ``natural'' approach to obtain an estimator of  $\mu_f$ is
$$
\hat{\mu}_{f, N}^{\text{MC}} = \frac{1}{N} \sum_{n = 1}^{N} f(x_{n}),
$$
with $x_1, \ldots, x_N \sim P$.
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{A central (limit) theorem}
Define
$$
\text{MC-SE}_{N}[f]
= \sqrt{ \frac{ \text{Var}_{P}[f]}{N} }.
$$
Then
$$
\lim_{N \rightarrow \infty}
\frac{ \hat{\mu}_{f,N}^{\text{MC}} - \mathbb{E}_{P}[f] }
{ \text{MC-SE}_{N}[f] }
\sim \text{Normal}(0, 1),
$$
\begin{idea}[MCMC-CLT needs to hold]
 A key insight is that MCMC only trustworthy when a central limit theorem holds.
 This means $f$ needs to be $2+\epsilon$-integrable with respect to $P$.
 Look out for $\text{MC-SE}$, too.
 It is important to quantify ``the probable error of the mean''\footnote{A ``pun'' with William Gosset's (1876--1937) paper: Student. (1908). The probable error of a mean. Biometrika, 1-25.}, as it were.
\end{idea}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Diagnostics}
\begin{idea}[Diagnose your MCMC!]
Perhaps as important as learning how to run an MCMC is to learn to \textbf{diagnose} it.
This means detecting failure to converge to $P$ and/or poor statistical performance.
\end{idea}
When running $K$ chains,  the between sample variance can be written as
\begin{equation*}
\label{eq:Between}
 B = \frac{N}{K-1} \sum_{k = 1}^K \left(\bar{x}_k - \bar{\bar{x}}\right)^2,
\end{equation*}
where $\bar{x}_k = N^{-1}\sum_{n = 1}^N x_k^{(n)}$ and $\bar{\bar{x}} = K^{-1}\sum_{k=1}^K\bar{x}_k$.
Now we can define the within variance as
\begin{equation*}
W =  K^{-1}\sum_{k = 1}^K s_k^2 \: \text{and} \: s_k^2  = (N-1)^{-1} \sum_{n = 1}^N \left(x_k^{(n)} - \bar{x}_k\right)^2
\end{equation*}
Finally we can define the~\textbf{potential scale reduction factor} (PSRF)~\citep{Gelman1992}:
\begin{equation*}
 \label{eq:PRSF}
 \hat{R} = \sqrt{\frac{ (N-1)W +  B }{NW}}.
\end{equation*}
At convergence, $\hat{R} < 1.1$, providing a univariate measure of convergence across chains (for a given parameter).
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{More diagnostics}
One of the things we are interested in is \textit{statistical} performance, i.e., how precise the estimator $\hat{\mu}_{f,N}^{\text{MC}}$ is.
To measure that, we can compute the \textbf{effective sample size}:
\begin{equation*}
 \text{ESS} = \frac{N}{1 + 2\sum_{t=1}^\infty \rho_t},
\end{equation*}
where $\rho_t$ is the \textbf{autocorrelation} at lag $t$, $t=1, 2, \ldots$.
A good rule of thumb\footnote{Assuming approximate normality. Calculation stolen from~\url{https://www.biorxiv.org/content/10.1101/2021.05.04.442586v1.full.pdf}} is that if one wants to have a an standard error which is 1\% of the width of the 95\% interval of the true distribution is to have $\text{ESS} \geq 625$:
\begin{align*}
 \frac{\sigma}{\sqrt{N}} &\leq \frac{\sigma}{\sqrt{\text{ESS}}},\\
 0.01 \times 4 \times \sigma &\leq \frac{\sigma}{\sqrt{\text{ESS}}},\\
 &\implies\\
 \text{ESS} &\geq 625,
\end{align*}
where $\sigma = \sqrt{\text{Var}_{P}[f]}$.
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Even more diagnostics}
\begin{figure}
\includegraphics[scale=0.25]{figures/traceplots.png}
\end{figure}
\begin{idea}[No one diagnostic is enough]
 Use multiple diagnostic metrics, always.
 Every MCMC diagnostic out there has blind spots; using multiple simultaneously increases the chances those blind spots are covered.
\end{idea}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Scaling with dimension}
\begin{figure}
\includegraphics[scale=0.35]{figures/concentration_measure_volume.pdf}
\end{figure}
Taken from~\url{https://mc-stan.org/users/documentation/case-studies/curse-dims.html}.
\begin{idea}[The higher the dimension, the more structure you need]
 As dimension increases, things start to get pretty lonely pretty fast for a particle.
 The only way to counteract this ``thinning'' is to introduce more structure.
 This is the intuitive basis for the success of gradient-based methods such as MALA\footnote{Metropolis-adjusted Langevin algorithm} and HMC\footnote{Hamiltonian (or Hybrid) Monte Carlo.}.
\end{idea}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Take home}
\begin{itemize}
 \item MCMC allows us to make inferences about huge models in Science and Engineering;
 \item MCMC is a terrible method, which nevertheless is our best shot at computing high-dimensional integrals;
 \item One has to make sure a CLT holds;
 \item One has to verify diagnostics to ensure no convergence/performance problems are present;
 \item No one diagnostic is enough.
\end{itemize}
\end{frame}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Recommended reading}
\begin{itemize}
  \item[\faBook] \cite{Robert2007}, Ch. 6\footnote{The Bayesian Choice by Christian Robert (2007, 2nd edition).}.
  \item[\faBook] \url{https://betanalpha.github.io/assets/case_studies/markov_chain_monte_carlo.html}
 \end{itemize}
\end{frame}


\chapter{Numerical Algorithms} \label{Ch:NumericalAlgorithms}

\section{Lagrange Interpolation}

Below we describe the algorithm used for lagrange interpolation.
This includes specifying how points  from the known data are chosen
for interpolation, and how interpolation is performed for multiple
function values (i.e. position and velocity) at the desired
interpolation point.

Assume we have $m$ functions to interpolate and that each function
is known at $\ell$ values of the independent variable $x$ (in
ephemeris interpolation this is time) as illustrated in
Table~\ref{Table:InterpolationData}. Given some point, say $x'$, we
desire the interpolated values of the functions, or $\mathbf{f}'
\approx \mathbf{P}(x')$ where $P$ is the Lagrange interpolating
polynomial.

%
\begin{table}[ht] \centering
\caption{Example of Data for interpolation}
 \begin{tabular}{ccccc}  \hline \hline
         x & $\mathbf{f}_1$ & $\mathbf{f}_2$ & \dots & $\mathbf{f}_m$ \\ \hline
         $x_1$ & $f_1(x_1)$ &  $f_2(x_1)$ &  \dots & $f_m(x_1)$\\
         $x_2$ & $f_1(x_2)$ &  $f_2(x_2)$ &  \dots &$f_m(x_2)$\\
         $x_3$ & $f_1(x_3)$ &  $f_2(x_3)$ &  \dots &$f_m(x_3)$\\
         $x_4$ & $f_1(x_4)$ &  $f_2(x_4)$ &  \dots &$f_m(x_4)$\\
         \vdots   & \vdots &   \vdots &  $\ddots$ & \vdots\\
         $x_{\ell-1}$ & $f_1(x_{\ell-1})$ &  $f_2(x_{\ell-1})$ & \dots & $f_m(x_{\ell-1})$\\
         $x_{\ell}$ & $f_1(x_\ell)$ &  $f_2(x_\ell)$ & \dots & $f_m(x_\ell)$\\
         \hline \hline
         \label{Table:InterpolationData}
 \end{tabular}
 \end{table}
 %

Before proceeding we define the following variables:
%
\begin{center}
    \begin{minipage}[t]{5.0 in}
        \begin{tabbing}[htbp!]
            123456 \= dummy line \kill
            $\mathbf{x}$ \> $m_x$ x 1 Known values of the independent variable (monotonically increasing or decreasing) \\
            $x'$ \>  Value of independent variable a the required interpolation point\\
            $\mathbf{f}_i$ \> $m_x$ x $m_f$ Array of dependent variable values for the $i^{th}$ function\\
            $\mathbf{f}'$ \> 1 x $m_f$ Array of interpolated function values at $x'$\\
            $n$ \> Order of interpolation \\
            $m_x$  \>  Number of points in $\mathbf{x}$ and $\mathbf{f}_i$ \\
            $m_f$  \>  Number of functions to be interpolated \\
        \end{tabbing}
    \end{minipage}
\end{center}

For interpolation to be feasible, several conditions must be
satisfied.  First, $x'$ must lie between the minimum and maximum
value of $\mathbf{x}$, or at the boundaries ( otherwise we would be
extrapolating ):
%
\begin{equation}
     x' \geq \mbox{min}(\mathbf{x})
\end{equation}
%
and
%
\begin{equation}
     x' \leq \mbox{max}(\mathbf{x})
\end{equation}
%
Second, there must be enough data points in points in $\mathbf{x}$
to support interpolation to the desired order.  If the requested
interpolation is of order $n$, then we require $n+1$ data points to
perform the interpolation.  Hence the following criteria must be
met:
%
\begin{equation}
    m_x \geq n + 1
\end{equation}
%

If the requested interpolation is feasible, we must choose the
subset of $\mathbf{x}$ to use for interpolating the data at $x'$.
Define $x(q)$ as the $q^{th}$ element of $\mathbf{x}$. Given $x'$,
we choose $q$ to minimize the difference between $x'$ and the mean
of $x(q)$ and $x(q + n)$, or:
%
\begin{equation}
    \underset{q}{\mbox{min }} \left| \frac{x(q+n) + x(q)}{2} - x'\right|  \hspace{.3 in} ( q \in 1,2,3,... m_x)
\end{equation}
%
Choosing $q$ in this way places $x'$ as near to the center of the
interpolation interval as possible.

The standard formula for the Lagrange interpolating polynomial for a
single function is
%
\begin{equation}
    P(x) =  \sum_{k=1}^n P_j(x) \label{Eq:LagrangeInterp}
\end{equation}
%
where $P_j(x)$ is given by
%
\begin{equation}
   P_j(x) = y_j\prod_{k=1,k \neq j}^n\frac{x - x_j}{x_j - x_k} \label{Eq:LagrangePj}
\end{equation}
%
Lagrange interpolation is an efficient method when interpolating
multiple data sets available at the same independent variable
points.  This is due to the fact that the product term in
Eq.~(\ref{Eq:LagrangePj}) is only a function of the values of the
independent variables and the desired interpolation point and not on
the function being interpolated.  As a result, we can evaluate the
product term one time and use it for many functions (which is what
is done when interpolating an ephemeris file).  The algorithm for
implementing Lagrangian interpolation for multiple functions is
shown below.

\begin{center}
\begin{minipage}{6 in}
\begin{small}
\begin{algorithm}[H]

    \KwIn{$\mathbf{x}, \mathbf{f},x',q,n$}
    %
    \KwOut{$\mathbf{f}'$}
    %
    $\mathbf{f}' = \mathbf{0}_{(1 \mbox{ x } m_f)}$\;
    %
    \For{ i = q to $q+n$}
    {
        $\mathbf{prod} = f(i,:)$ \% The $i^{th}$ row of $\mathbf{f}$ \;
        \For{ $j = q$ to $q+n$}
        {
            \If{ $ i \neq j$}
            {
            \% The product in Eq.~(\ref{Eq:LagrangePj})\;
                    $ \mathbf{prod} = \mathbf{prod} \cdot \displaystyle\frac{x' - x(j)}{x(i) - x(j)}$\;
            }
        }
       \% The summation in Eq.~(\ref{Eq:LagrangeInterp})\;
       $ \mathbf{f}'= \mathbf{f}' + \mathbf{prod}$\;
    }
    \hspace{.2 in}
    %
    \label{alg:LagrangeInterpolation}\caption{Algorithm for Lagrangian Interpolation}
    %
\end{algorithm}
\end{small}
\end{minipage}
\end{center}

\section{Quadratic Polynomial Interpolation }

We are given three data points defined by a vector of independent
variables
%
\begin{equation}
     \mathbf{x} = [\hspace{.05 in} x_1 \hspace{.05 in} x_2 \hspace{.05 in}
     x_3 \hspace{.05
     in}]^T
\end{equation}
%
and a vector of corresponding dependent variables
%
\begin{equation}
     \mathbf{y} = [\hspace{.05 in} y_1 \hspace{.05 in} y_2 \hspace{.05 in}
     y_3  \hspace{.05
     in}]^T
\end{equation}
%
we wish to find a quadratic polynomial that fits the data such that
%
\begin{equation}
   y = Ax^2 + Bx +C
\end{equation}
%
We begin by forming the system of linear equations
%
\begin{equation}
    \left(%
    \begin{array}{ccc}
       x_1^2 & x_1 & 1 \\
       x_2^2 & x_2 & 1 \\
       x_3^2 & x_3 & 1 \\
    \end{array}%
    \right)
    %
     \left(%
    \begin{array}{ccc}
       A \\
       B \\
       C \\
    \end{array}%
    \right)
    =
     \left(%
    \begin{array}{ccc}
       y_1 \\
       y_2 \\
       y_3 \\
    \end{array}%
    \right)
\end{equation}
%
We can solve for the coefficients using
%
\begin{equation}
    A = \frac{
    %  Numerator
    \left|%
    \begin{array}{ccc}
       y_1 & x_1 & 1 \\
       y_2 & x_2 & 1 \\
       y_3 & x_3 & 1 \\
    \end{array}%
    \right|
    }
    %  Denominator
    {    \left|%
    \begin{array}{ccc}
       x_1^2 & x_1 & 1 \\
       x_2^2 & x_2 & 1 \\
       x_3^2 & x_3 & 1 \\
    \end{array}%
    \right|}
\end{equation}
%
\begin{equation}
    B = \frac{
    %  Numerator
    \left|%
    \begin{array}{ccc}
     x_1^2 & y_1 & 1 \\
     x_2^2 & y_2 & 1 \\
     x_3^2 & y_3 & 1 \\
    \end{array}%
    \right|
    }
    %  Denominator
    {    \left|%
    \begin{array}{ccc}
       x_1^2 & x_1 & 1 \\
       x_2^2 & x_2 & 1 \\
       x_3^2 & x_3 & 1 \\
    \end{array}%
    \right|}
\end{equation}
%
\begin{equation}
    C = y_3 - A x_3^2 - B x_3
\end{equation}

\section{Cubic Spline (Not-a-Knot) Interpolation }

We are given five data points defined by a vector of independent
variables
%
\begin{equation}
     \mathbf{x} = [\hspace{.05 in} x_1 \hspace{.05 in} x_2 \hspace{.05 in}
     x_3 \hspace{.05 in} x_4 \hspace{.05 in} x_5 \hspace{.05
     in}]^T
\end{equation}
%
and a vector of corresponding dependent variables
%
\begin{equation}
     \mathbf{y} = [\hspace{.05 in} y_1 \hspace{.05 in} y_2 \hspace{.05 in}
     y_3 \hspace{.05 in} y_4 \hspace{.05 in} y_5 \hspace{.05
     in}]^T
\end{equation}
%
we wish to find the four cubic polynomials, $i = 1,2,3,4$, such that
%
\begin{equation}
    p_i = a_i(x - x_i)^3 + b_i(x - x_i)^2 + c_i(x - x_i) + d_i
\end{equation}
%
where the values for $x_i$ are known from the inputs.

To calculate the coefficients $a_i$, $b_i$, $c_i$, and $d_i$ we
start by calculating the eight quantities
%
\begin{eqnarray}
     h_i      &=& x_{i+1} - x_i\\
     \Delta_i &=& \displaystyle\frac{y_{i+1} - y_i}{h_i}
\end{eqnarray}
%
Next we solve the following system of linear equations
%
\begin{equation}
     \mathbf{A}\mathbf{S} = \mathbf{B}
\end{equation}
%
where the components of $\mathbf{A}$ are given by
%
\begin{eqnarray}
     A_{11} &=& 2h_2 + h_1 \\
     A_{12} &=& 2h_1 + h_2 \\
     A_{13} &=& 0 \\
     A_{21} &=& 0 \\
     A_{22} &=& h_3 + 2h_4 \\
     A_{23} &=& 2h_3 + h_4 \\
     A_{31} &=& \displaystyle\frac{h_2^2}{h_1 + h_2} \\
     A_{32} &=& \displaystyle\frac{h_1h_2}{( h_1 + h_2 )} + 2( h_2 + h_3 ) + \displaystyle\frac{h_3h_4}{( h_3 + h_4
     )}\\
     A_{33} &=& \displaystyle\frac{h_3^2}{h_3 + h_4}
\end{eqnarray}
%
the components of $\mathbf{B}$ are
%
\begin{eqnarray}
    B_{11} &=& 6( \Delta_2 - \Delta_1 )\\
    B_{21} &=& 6( \Delta_4 - \Delta_3 )\\
    B_{31} &=& 6( \Delta_3 - \Delta_2 )
\end{eqnarray}
%
and $\mathbf{S} = [\hspace{.05 in} S_1 \hspace{.05 in} S_3
\hspace{.05 in} S_5 \hspace{.05 in}]^T$. We can solve for the
components of $\mathbf{S}$ using Cramer's Rule as follows
%
\begin{equation}
    S_1 = \frac{
    %  Numerator
    \left|%
    \begin{array}{ccc}
       B_{11} & A_{12} & A_{13} \\
       B_{21} & A_{22} & A_{23} \\
       B_{31} & A_{32} & A_{33} \\
    \end{array}%
    \right|
    }
    %  Denominator
    {\mathbf{| \mathbf{A}|}}
\end{equation}
%
\begin{equation}
    S_3 = \frac{
    %  Numerator
    \left|%
    \begin{array}{ccc}
       A_{11} & B_{11} & A_{13} \\
       A_{21} & B_{21} & A_{23} \\
       A_{31} & B_{31} & A_{33} \\
    \end{array}%
    \right|
    }
    %  Denominator
    {\mathbf{| \mathbf{A}|}}
\end{equation}
%
\begin{equation}
     S_5 = \frac{B_{31} - A_{31} S_1 - A_{32} S_3}{A_{33}}
\end{equation}
%
Now we can calculate $S_2$ and $S_4$ using
%
\begin{equation}
     S_2 = \frac{ h_2 S_1 + h_1 S_3  }{  h_1 + h_2 };
\end{equation}
%
\begin{equation}
     S_4 = \frac{ h_4 S_3 + h_3 S_5  }{  h_3 + h_4 };
\end{equation}
%

Finally, the coefficients for the $i^{th}$ cubic polynomial are
given by
\begin{eqnarray}
   a_i &=&  \frac{ S_{i+1} - S_i   }{ 6  h_i}\\
   b_i &=&  \frac{S_i}{2}\\
   c_i &=&  \frac{ y_{i+1} - y_i  }{ h_i } - \frac{ 2 h_i S_i + h_i S_{i+1} }{  6
   }\\
   d_i &=&  y_i
\end{eqnarray}

\section{Root Location using Brent's Method}

NOTE: The implementation used in GMAT's even locators follows the Algorithm
in Numerical Recipes 3rd edition which slightlyl different than the algorithm presented below.

Brent's\cite{Brent:73} method is a root finding algorithm that takes
advantage of superlinear convergence properties of the secant method
and inverse quadratic interpolation, and the robustness of the
bisection method.  The algorithm is a modification of Dekker's
method which combines the bisection method and the secant method
\cite{Dekker:69}.

The algorithm requires as inputs to real numbers that bound the
desired root.

\begin{tabbing}
    12345678912345 \= Reynolds number based on length $s$ \kill
    $f$         \>  Function whose root is desired \\
    $a$         \>  Initial guess for independent variable ($a$ and $b$ must bound the desired root)\\
    $b$         \>  Initial guess for independent variable ($a$ and $b$ must bound the desired root) \\
    $tol$     \>  Convergence tolerance.\\
    $N$       \> Maximum number of iterations \\
    $x_o$     \>  Root value\\
    $i$         \> Number of iterations\\
    $\epsilon$ \> Machine precision\\
\end{tabbing}

\begin{center}
\begin{minipage}{6 in}
\begin{small}
\begin{algorithm}[H]

    \KwIn{$f,a,b,tol,N,$}
    %
    \KwOut{$x_o,i$}
    %
    $f_a = f(a)$; $f_b = f(b)$\;
    \textbf{if} $f_a \cdot f_b > 0$; error and return; \textbf{end} \;
    $c = a;$ $f_c = f_a;$ $d = b - a;$ $e = d$\;
    \If{ $|f_c| \leq |f_b|$}
    {
     $a = b;$ $b = c;$ $c = a;$
    $f_a = f_b;$ $f_b = f_c;$ $f_c = f_a;$
    }
    $t = 2\epsilon | b | + tol$\;
    $m = 0.5(c - b)$ \;
    $i = 1$\;
    \While{$ | m | >= t$ and $| f_b | >= t$ and $ i \leq N$}
        {
        \eIf{ $|e | \leq t $ or $ |f_a| <  |fb|$}
          {
           $d = m$; $ e = m$\;
          }{
           $s = f_b/f_a$\;  % line 40
           \eIf {$ a = c $}
              {
              $p = 2.0 \cdot m \cdot s$\;
              $q = 1.0 - s$\;
               }
              {
                $q = f_a/f_c$; $r = f_b/f_c$\;
                $p = s(2.0 \cdot m \cdot q (q-r) - (b - a)(r - 1.0))$\;
                $q = (q - 1.0)(r - 1.0)(s - 1.0)$\;
                }
          \eIf{ $p >= 0.0$ }{ %line 60
            $q = -q$\;
          }{
            $p = -p$\;  % line 70
          }
        $s = e$\;  % line 80
        $e = d$\;
        \eIf {($2 \cdot p < 3.0 \cdot m \cdot q- |t\cdot q|) $ or $ ( p \leq | 0.5 \cdot s\cdot q | )$}
           {
            $d = p/q$\;
         }{
            $d = m$; $e = m$\;
         }
       }
       $a  = b$; $f_a = f_b$\;
       \eIf { $|d| \geq  t$ }{
          $b = b + d$;
        }
        {
        \eIf {$ m \geq 0.0$ }{
           $ b = b + t$;
        }{
           $ b = b - t$;
         }
        }
       $ f_b = f(b)$\;
        \eIf { ( $f_b > 0.0$ and $f_c > 0$ ) or ( $f_b < 0.0$ and $f_c < 0$ )} {
            $c = a;$ $f_c = fa;$ $d = b - a;$ $e = d$\;
            }
         {
               \If{ $|f_c| \leq |f_b|$}
            {
             $a = b;$ $b = c;$ $c = a;$
            $f_a = f_b;$ $f_b = f_c;$ $f_c = f_a;$
            }
                 }
        $t = 2\epsilon |b| + tol$; % line 3
        $m    = 0.5(c - b)$;
        $ i = i + 1$;

    }
    $x_o = b$\;
    \hspace{.2 in}
    %
    \label{alg:BrentsMethod}\caption{Brent's Method for Root Finding}
    %
\end{algorithm}
\end{small}
\end{minipage}
\end{center}


\documentclass[12pt]{article}

\usepackage[top=1in, bottom=1in, left=1in, right=1in]{geometry} % see geometry.pdf on
\usepackage{graphicx}
\usepackage{amssymb,amsmath}
\usepackage[utf8]{inputenc}
\usepackage{listings}
\usepackage{color}
\geometry{letterpaper}
\linespread{1.1}% \geometry{landscape} % rotated page geometry

\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}

\lstdefinestyle{mystyle}{
	backgroundcolor=\color{backcolour},
	commentstyle=\color{codegreen},
	keywordstyle=\color{magenta},
	numberstyle=\tiny\color{codegray},
	stringstyle=\color{codepurple},
	basicstyle=\footnotesize,
	breakatwhitespace=false,
	breaklines=true,
	captionpos=b,
	keepspaces=true,
	numbers=left,
	numbersep=5pt,
	showspaces=false,
	showstringspaces=false,
	showtabs=false,
	tabsize=2
}

\lstset{style=mystyle}

\title{IEOR 160 Course Project}
\date{20 Oct. 2017}
\author{Qingan Zhao \\ SID: 3033030808}

\begin{document}

\maketitle
\newcommand{\ud}{\mathrm d} %COMMENT: for infinitesimal dx = \ud x and dy = \ud y TODO: DO YOURSELF all replacement
\renewcommand\theequation{\arabic{equation}}
\renewcommand{\figurename}{Fig.}
\renewcommand\thesection{Problem \arabic{section}}
\renewcommand\thesubsection{\alph{subsection} )}

\begin{large}
\noindent\textbf{All plots and relevant code are generated in MATLAB.}
\end{large}

\section{}

This problem is solved using Gradient method with the backtracking line search ($\alpha$ = 1 and $\beta$ = 0.6).\\

\noindent Code:

\lstinputlisting[language=MATLAB]{P1.m}

\bigskip\noindent The minimum value of the function is \textbf{2.722814}, where $x$=\textbf{(0.122218, -0.556434)}.\\

\noindent The plot of $f(x^{(k)})$ versus $k$ for $k=0,1,2,...,50$ is shown in Figure 1.\\\\\\

\begin{figure}[!htbp]
	\centering
	\includegraphics[width=10cm]{figures/fig1.eps}
	\caption{$f(x^{(k)})$ versus $k$ for $k=0,1,2,...,50$ (Gradient method)}
\end{figure}

\noindent The trajectory of points $x^{(0)}$, $x^{(1)}$, ..., $x^{(50)}$ is shown in Figure 2.

\begin{figure}[!htbp]
	\centering
	\includegraphics[width=10cm]{figures/fig2.eps}
	\caption{The trajectory of points $x^{(0)}$, $x^{(1)}$, ..., $x^{(50)}$ (Gradient method)}
\end{figure}

\section{}

This problem is solved using Newton's method with the backtracking line search ($\alpha$ = 1 and $\beta$ = 0.6).\\

\noindent Code:

\lstinputlisting[language=MATLAB]{P2.m}

\bigskip\noindent The minimum value of the function is \textbf{2.722814}, where $x$=\textbf{(0.122219, -0.556434)}.\\

\noindent The plot of $f(x^{(k)})$ versus $k$ for $k=0,1,2,...,50$ is shown in Figure 3.

\begin{figure}[!htbp]
	\centering
	\includegraphics[width=10cm]{figures/fig3.eps}
	\caption{$f(x^{(k)})$ versus $k$ for $k=0,1,2,...,50$ (Newton's method)}
\end{figure}

\noindent The trajectory of points $x^{(0)}$, $x^{(1)}$, ..., $x^{(50)}$ is shown in Figure 4.

\begin{figure}[!htbp]
	\centering
	\includegraphics[width=10cm]{figures/fig4.eps}
	\caption{The trajectory of points $x^{(0)}$, $x^{(1)}$, ..., $x^{(50)}$ (Newton's method)}
\end{figure}

\section{}

This problem is solved using Newton's method with the backtracking line search ($\alpha$ = 0.4 and $\beta$ = 0.4).\\

\noindent The function in this problem is much more complex than the previous one. Hence, to speed up the code, let's first calculate the gradient and hessian by hand instead of using $gradient()$ and $hessian()$ functions in MATLAB. Assume the $log()$ in the problem is the logarithm to the base 10.\\

\noindent Another thing needs to mention is that the code is not efficient enough to run with 500 variables, so let's just run with 100 variables instead.\\

\noindent Code:\\

\noindent\textbf{(1)} The $f$ function (the function in the problem)

\lstinputlisting[language=MATLAB]{f.m}
\ \\

\bigskip\noindent\textbf{(2)}  The gradient function

\lstinputlisting[language=MATLAB]{grad_fun.m}

\bigskip\noindent\textbf{(3)}  The hessian function

\lstinputlisting[language=MATLAB]{hess_fun.m}

\bigskip\noindent\textbf{(4)}  Main script

\lstinputlisting[language=MATLAB]{p3.m}

\bigskip\noindent The minimum value of the function is \textbf{-204.012227.}\\\\\\

\noindent The plot of $f(x^{(k)})$ versus $k$ for $k=0,1,2,...,300$ is shown in Figure 5.

\begin{figure}[!htbp]
	\centering
	\includegraphics[width=10cm]{figures/fig5.eps}
	\caption{$f(x^{(k)})$ versus $k$ for $k=0,1,2,...,300$ (Newton's method)}
\end{figure}

\end{document}


\section{Reversal}
\label{sec:reversal}

%\paragraph{Involution} -> Allows a page break because there is no
% text after \section.

\textbf{Involution}\index{stack!reversal!involution}\quad Sometimes a
proof requires some lemma to be devised. Let us consider the
definition of a function
\fun{rev\(_0\)/1}\label{def:rev0}\index{stack!reversal!definition}\index{rev0@\fun{rev\(_0\)/1}}
reversing a stack:
\begin{equation*}
\begin{array}{@{}r@{\;}l@{\;}lr@{\;}l@{\;}l@{}}
  \fun{cat}(\el,t)\index{cat@\fun{cat/2}}
& \xrightarrow{\smash{\alpha}} & t;
& \fun{rev}_0(\el)
& \xrightarrow{\smash{\gamma}} & \el;\\
  \fun{cat}(\cons{x}{s},t)
& \xrightarrow{\smash{\beta}} & \cons{x}{\fun{cat}(s,t)}.
& \fun{rev}_0(\cons{x}{s})
& \xrightarrow{\smash{\delta}} & \fun{cat}(\fun{rev}_0(s),[x]).
\end{array}
\end{equation*}
An evaluation is shown with abstract syntax trees in
\fig~\ref{fig:rev0_321}.
\begin{figure}[!b]
\centering
\includegraphics[bb=69 620 328 721]{rev0_321_0}
\includegraphics[bb=64 626 320 721]{rev0_321_1}
\includegraphics[bb=69 627 375 721]{rev0_321_2}
\includegraphics[bb=69 627 375 721]{rev0_321_3}
\caption{\(\fun{rev}_0([3,2,1]) \twoheadrightarrow [1,2,3]\)
\label{fig:rev0_321}\index{stack!reversal!example}}
\end{figure}
Let \(\pred{Inv}{s}\)\index{Inv@\predName{Inv}|(} be the property
\(\fun{rev}_0(\fun{rev}_0(s)) \equiv s\), that is, the function
\fun{rev\(_0\)/1}\index{rev0@\fun{rev\(_0\)/1}} is an
\emph{involution}\index{stack!reversal!involution}. In order to prove
\(\forall s \in S.\pred{Inv}{s}\), the induction principle on the
structure of~\(s\)\index{induction!example|(} requires that we
establish
\begin{itemize*}

  \item the basis \(\pred{Inv}{\el}\);

  \item the inductive step \(\forall s \in S.\pred{Inv}{s}
    \Rightarrow \forall x \in T.\pred{Inv}{\cons{x}{s}}\).

\end{itemize*}
The basis is quickly found: \(\fun{rev}_0(\fun{rev}_0(\el))
\xrightarrow{\smash{\gamma}} \fun{rev}_0(\el)
\xrightarrow{\smash{\gamma}} \el\). The induction hypothesis is
\(\pred{Inv}{s}\) and we want to establish
\(\pred{Inv}{\cons{x}{s}}\)\index{Inv@\predName{Inv}|)}, for
any~\(x\). If we commence head\hyp{}on with
\(\fun{rev}_0(\fun{rev}_0(\cons{x}{s})) \xrightarrow{\smash{\delta}}
\fun{rev}_0(\fun{cat}(\fun{rev}_0(s),[x]))\), we are stuck. But the
term to rewrite involves both \fun{rev$_0$/1} and \fun{cat/2}, hence
spurring us to conceive a lemma where the stumbling pattern
\(\fun{cat}(\fun{rev}_0(\dots),\dots)\) occurs and is equivalent to a
simpler term.\par\vskip\baselineskip

Let
\(\pred{CatRev}{s,t}\)\index{CatRev@\predName{CatRev}} \label{CatRev}
denote \(\fun{cat}(\fun{rev}_0(t),\fun{rev}_0(s)) \equiv
\fun{rev}_0(\fun{cat}(s,t))\)\index{cat@\fun{cat/2}}. In order to
prove it by induction on the structure of~\(s\), we need, for
all~\(t\),
\begin{itemize*}

  \item the basis \(\pred{CatRev}{\el,t}\);

  \item the inductive step \(\forall s,t \in S.\pred{CatRev}{s,t}
    \Rightarrow \forall x \in T.\pred{CatRev}{\cons{x}{s},t}\).

\end{itemize*}
The former is almost within reach:
\begin{align*}
  \fun{rev}_0(\fun{cat}(\el,t))
  & \smashedrightarrow{\alpha}
  \fun{rev}_0(t)\\
  & \leftrightsquigarrow \fun{cat}(\fun{rev}_0(t),\el)\\
  & \smashedleftarrow{\gamma}
    \fun{cat}(\fun{rev}_0(t),\fun{rev}_0(\el)).
\end{align*}
The missing part is filled by showing that
\((\leftrightsquigarrow)\)~is actually~\((\equiv)\).\par\vskip\baselineskip

Let \(\pred{CatNil}{s}\)\index{CatNil@\predName{CatNil}|(} be the
property \(\fun{cat}(s,\el) \equiv s\)\index{cat@\fun{cat/2}}. In
order to prove it by induction on the structure of~\(s\), we have to
prove
\begin{itemize}

  \item the basis \(\pred{CatNil}{\el}\);

  \item the inductive step \(\forall s \in S.\pred{CatNil}{s}
    \Rightarrow \forall x \in T.\pred{CatNil}{\cons{x}{s}}\).

\end{itemize}
The former is easy: \(\fun{cat}(\el,\el) \xrightarrow{\smash{\alpha}}
\el\)\index{cat@\fun{cat/2}}. The latter is not complicated either:
\(\fun{cat}(\cons{x}{s},\el) \xrightarrow{\smash{\beta}}
\cons{x}{\fun{cat}(s,\el)} \equiv \cons{x}{s}\), where the equivalence
is none other than the induction hypothesis
\(\pred{CatNil}{s}\)\index{CatNil@\predName{CatNil}|)}.\hfill\(\Box\)

%% Note that we could have proved instead \(\fun{cat}(s,\el)
%% \twoheadrightarrow s\), for all values~\(s\). The difference is that
%% \(\pred{CatNil}{s}\) holds even if~\(s\) is not a value, as all we
%% have to do is to replace \(\fun{cat}(\cons{x}{s},\el)
%% \xrightarrow{\smash{\beta}} \cons{x}{\fun{cat}(s,\el)}\) by the
%% equivalence \(\fun{cat}(\cons{x}{s},\el) \Rra{\beta}
%% \cons{x}{\fun{cat}(s,\el)}\). This principle will hold in all our
%% proofs: we assume that variables denote values and, if not, we simply
%% change \((\xrightarrow{\smash{\beta}})\) into \((\Rra{\beta})\), for
%% all rules~\(\beta\) in the proof. Our assumption perhaps improves
%% legibility.\par\vskip\baselineskip

\par\vskip\baselineskip

\noindent Assuming
\(\pred{CatRev}{s,t}\)\index{CatRev@\predName{CatRev}}, we must
establish \(\forall x \in T.\pred{CatRev}{\cons{x}{s},t}\):
\index{cat@\fun{cat/2}|(} \index{rev0@\fun{rev\(_0\)/1}|(}
\index{CatAssoc@\predName{CatAssoc}}
\begin{equation*}
\begin{array}{@{}r@{\;}l@{\;}l@{\;\;}r@{}}
  \fun{cat}(\fun{rev}_0(t),\fun{rev}_0(\cons{x}{s}))
& \xrightarrow{\smash{\delta}}
& \fun{cat}(\fun{rev}_0(t),\fun{cat}(\fun{rev}_0(s),[x]))\\
& \equiv
& \fun{cat}(\fun{cat}(\fun{rev}_0(t),\fun{rev}_0(s)),[x])
& (\predName{CatAssoc})\\
& \equiv
& \fun{cat}(\fun{rev}_0(\fun{cat}(s,t)),[x])
& (\pred{CatRev}{s,t})\\
& \xleftarrow{\smash{\delta}}
& \fun{rev}_0(\cons{x}{\fun{cat}(s,t)})\\
& \xleftarrow{\smash{\beta}}
& \fun{rev}_0(\fun{cat}(\cons{x}{s},t)).
& \multicolumn{1}{r@{}}{\Box}
\end{array}
\end{equation*}
Let us resume the proof of
\(\pred{Inv}{\cons{x}{s}}\):\index{Inv@\predName{Inv}}
\begin{equation*}
\begin{array}{@{}r@{\;}l@{\;}l@{\quad}r@{}}
  \fun{rev}_0(\fun{rev}_0(\cons{x}{s}))
& \xrightarrow{\smash{\delta}}
& \fun{rev}_0(\fun{cat}(\fun{rev}_0(s),[x]))\\
& \equiv
& \fun{cat}(\fun{rev}_0([x]),\fun{rev}_0(\fun{rev}_0(s)))
& (\pred{CatRev}{\fun{rev}_0(s),[x]})\\
& \equiv
& \fun{cat}(\fun{rev}_0([x]),s)
& (\pred{Inv}{s})\\
& \xrightarrow{\smash{\delta}}
& \fun{cat}(\fun{cat}(\fun{rev}_0(\el),[x]),s)\\
& \xrightarrow{\smash{\gamma}}
& \fun{cat}(\fun{cat}(\el,[x]),s)\\
& \xrightarrow{\smash{\alpha}}
& \fun{cat}([x],s)\\
& \xrightarrow{\smash{\beta}}
& \cons{x}{\fun{cat}(\el,s)}\\
& \xrightarrow{\smash{\alpha}} &
\cons{x}{s}.
& \hfill\Box
\end{array}
\end{equation*}
\index{rev0@\fun{rev\(_0\)/1}|)}\index{cat@\fun{cat/2}|)}
\index{CatRev@\predName{CatRev}}

\paragraph{Equivalence}

We may have two definitions meant to describe the same function, which
differ in complexity and/or efficiency. For instance,
\fun{rev\(_0\)/1}\index{rev0@\fun{rev\(_0\)/1}} was given an intuitive
definition, as we can clearly see in rule~\(\delta\) that the
item~\(x\), which is the top of the input, is intended to be located
at the bottom of the output. Unfortunately, this definition is
computationally inefficient, that is, it leads to a great deal of
rewrites relatively to the size of the input.

Let us assume that we also have an efficient definition for the stack
reversal, named \fun{rev/1}\index{rev@\fun{rev/1}}, which depends upon
an auxiliary function
\fun{rcat/2}\index{rcat@\fun{rcat/2}|(}\index{stack!reversal!$\sim$
  and catenation}\index{stack!reversal!efficient $\sim$}
(\emph{reverse and catenate}):
\begin{equation}
  \begin{array}{@{}r@{\;}l@{\;}l@{}}
    \fun{rev}(s) & \smashedrightarrow{\epsilon} & \fun{rcat}(s,\el).\\
    \fun{rcat}(\el,t) & \smashedrightarrow{\zeta} & t;\\
    \fun{rcat}(\cons{x}{s},t) & \smashedrightarrow{\eta} &
    \fun{rcat}(s,\cons{x}{t}).
  \end{array}
  \label{def:rev}
\end{equation}
An additional parameter introduced by \fun{rcat/2} accumulates partial
results, thus called an \emph{accumulator}\index{functional
  language!accumulator}. We can see it at work in
\fig~\vref{fig:rev_321}.
\begin{figure}
\begin{equation*}
\boxed{%
\begin{array}{r@{\;}l@{\;}l}
\fun{rev}([3,2,1])
& \xrightarrow{\smash{\epsilon}} & \fun{rcat}([3,2,1],\el)\\
& \xrightarrow{\smash{\eta}}     & \fun{rcat}([2,1],[3])\\
& \xrightarrow{\smash{\eta}}     & \fun{rcat}([1],[2,3])\\
& \xrightarrow{\smash{\eta}}     & \fun{rcat}(\el,[1,2,3])\\
& \xrightarrow{\smash{\zeta}}    & [1,2,3].
\end{array}}
\end{equation*}
\caption{\(\fun{rev}([3,2,1]) \twoheadrightarrow [1,2,3]\)
\label{fig:rev_321}\index{stack!reversal!example}}
\end{figure}
\index{rcat@\fun{rcat/2}|)}

Let us prove\label{EqRev} \(\pred{EqRev}{s} \colon
\fun{rev}_0(s) \equiv
\fun{rev}(s)\)\index{rev@\fun{rev/1}|(}\index{EqRev@\predName{EqRev}|(}
by
structural\index{stack!reversal!equivalence}\index{equivalence!proof
  of $\sim$} induction on~\(s\), namely,
\begin{itemize}

  \item the basis \(\pred{EqRev}{\el}\);

  \item the inductive step \(\forall s \in S.\pred{EqRev}{s}
    \Rightarrow \forall x \in T.\pred{EqRev}{\cons{x}{s}}\).

\end{itemize}
The former is easy: \(\fun{rev}_0(\el) \xrightarrow{\smash{\gamma}}
\el \xleftarrow{\smash{\zeta}} \fun{rcat}(\el,\el)
\xleftarrow{\smash{\epsilon}}
\fun{rev}(\el)\)\index{rev0@\fun{rev\(_0\)/1}}.

For the latter, let us rewrite \(\fun{rev}_0(\cons{x}{s})\) and
\(\fun{rev}(\cons{x}{s})\) so they converge:
\begin{equation*}
\begin{array}{@{}r@{\;}l@{\;}lr@{}}
  \fun{rev}_0(\cons{x}{s})
& \xrightarrow{\smash{\delta}}
& \fun{cat}(\fun{rev}_0(s),[x])\\
& \equiv
& \fun{cat}(\fun{rev}(s),[x])
& (\pred{EqRev}{s})\\
& \leftrightsquigarrow
& \fun{rcat}(s,[x])
& (\text{to be determined})\\
& \xleftarrow{\smash{\eta}}
& \fun{rcat}(\cons{x}{s},\el)\\
& \xleftarrow{\smash{\epsilon}}
& \fun{rev}(\cons{x}{s}).
\end{array}
\end{equation*}
The missing part is filled by showing \((\leftrightsquigarrow)\) to
be~\((\equiv)\) as follows.\index{rev@\fun{rev/1}|)}

\par\vskip\baselineskip

Let
\(\pred{RevCat}{s,t}\)\index{RevCat@\predName{RevCat}}\label{RevCat}
be the property \(\fun{rcat}(s,t) \equiv
\fun{cat}(\fun{rev}(s),t)\).\index{rcat@\fun{rcat/2}}\index{cat@\fun{cat/2}|(}\index{rev@\fun{rev/1}}
Induction on the structure of~\(s\) requires the proofs of
\begin{itemize}

  \item the basis \(\forall t \in S.\pred{RevCat}{\el,t}\);

  \item the general case \(\forall s,t \in S.\pred{RevCat}{s,t}
    \Rightarrow \forall x \in T.\pred{RevCat}{\cons{x}{s},t}\).

\end{itemize}
First, for the establishment of the basis, we have:
\begin{align*}
  \fun{rcat}(\el,t)
  & \smashedrightarrow{\zeta} t\\
  & \smashedleftarrow{\alpha} \fun{cat}(\el,t)\\
  & \smashedleftarrow{\zeta} \fun{cat}(\fun{rcat}(\el,\el),t)\\
  & \smashedleftarrow{\epsilon} \fun{cat}(\fun{rev}(\el),t).
\end{align*}
Next, let us assume \(\pred{RevCat}{s,t}\) and prove \(\forall x \in
T.\pred{RevCat}{\cons{x}{s},t}\):
\begin{equation*}
\begin{array}{@{}r@{\;}l@{\;}l@{}r@{}}
  \fun{rcat}(\cons{x}{s},t)
& \xrightarrow{\smash{\eta}}
& \fun{rcat}(s,\cons{x}{t})\\
& \equiv
& \fun{cat}(\fun{rev}(s),\cons{x}{t})
& (\pred{RevCat}{s,\cons{x}{t}})\\
& \xleftarrow{\smash{\alpha}}
& \fun{cat}(\fun{rev}(s),\cons{x}{\fun{cat}(\el,t)})\\
& \xleftarrow{\smash{\beta}}
& \fun{cat}(\fun{rev}(s),\fun{cat}([x],t))\\
& \equiv
& \fun{cat}(\fun{cat}(\fun{rev}(s),[x]),t)
& (\pred{CatAssoc}{\fun{rev}(s),[x],t})\\
& \equiv
& \fun{cat}(\fun{rcat}(s,[x]),t)
& (\pred{RevCat}{s,[x]})\\
& \xleftarrow{\smash{\eta}}
& \fun{cat}(\fun{rcat}(\cons{x}{s},\el),t)\\
& \xleftarrow{\smash{\epsilon}}
& \fun{cat}(\fun{rev}(\cons{x}{s}),t).
& \multicolumn{1}{r@{}}{\Box}
\end{array}
\end{equation*}
Finally, we proved \(\forall s.\pred{EqRev}{s}\), that is,
\(\fun{rev/1} =
\fun{rev\(_0\)/1}\)\index{EqRev@\predName{EqRev}|)}\index{rev@\fun{rev/1}}\index{rev0@\fun{rev\(_0\)/1}}\index{cat@\fun{cat/2}|)}\index{induction!example|)}.\hfill\(\Box\)

\paragraph{Cost}
\label{cost:rev0}
\index{stack!reversal!cost}

The definition of \fun{rev\(_0\)/1}\index{rev0@\fun{rev\(_0\)/1}}
directly leads to the recurrences\index{rev0@$\C{\fun{rev}_0}{n}$}
\begin{equation*}
\C{\fun{rev}_0}{0} = 1,\qquad \C{\fun{rev}_0}{k+1} = 1 +
\C{\fun{rev}_0}{k} + \C{\fun{cat}}{k} = \C{\fun{rev}_0}{k} + k + 2,
\end{equation*}
because the length of \(\fun{rev}_0(s)\) is~\(k\) if the length
of~\(s\) is~\(k\), and we already know \(\C{\fun{cat}}{k} = k +
1\)\index{cat@$\C{\fun{cat}}{n}$} (page~\pageref{cost:cat}). We have
\begin{equation*}
\sum_{k=0}^{n-1}(\C{\fun{rev}_0}{k+1} - \C{\fun{rev}_0}{k}) =
\C{\fun{rev}_0}{n} - \C{\fun{rev}_0}{0} =
\sum_{k=0}^{n-1}(k+2) = 2n + \sum_{k=0}^{n-1}{k}.
\end{equation*}
The remaining sum is a classic of algebra:
\begin{equation*}
2 \cdot \sum_{k=0}^{n-1}{k} =
\sum_{k=0}^{n-1}{k} + \sum_{k=0}^{n-1}{k} =
\sum_{k=0}^{n-1}{k} + \sum_{k=0}^{n-1}(n-k-1) = n(n-1).
\end{equation*}
Consequently,
\begin{equation}
\sum_{k=0}^{n-1}{k} = \frac{n(n-1)}{2},\label{eq:sum_k}
\end{equation}
and we can finally conclude
\begin{equation*}
\C{\fun{rev}_0}{n} = \frac{1}{2}n^2 + \frac{3}{2}n + 1 \sim \frac{1}{2}n^2.
\end{equation*}
Another way to reach the result is to induce an \emph{evaluation
  trace}\index{functional language!evaluation!trace}. A trace is a
composition of rewrite rules, noted using the mathematical convention
for multiplication. From \fig~\vref{fig:rev0_321}, we draw the trace
\(\T{\fun{rev}_0}{n}\)\index{trace_rev0@$\T{\fun{rev}_0}{n}$} of the
evaluation of \(\fun{rev}_0(s)\)\index{rev0@\fun{rev\(_0\)/1}}, where
\(n\)~is the length of~\(s\):
\begin{equation*}
\T{\fun{rev}_0}{n} :=
\delta^n\gamma\alpha(\beta\alpha)\dots(\beta^{n-1}\alpha) =
\delta^n\gamma \prod_{k=0}^{n-1}{\beta^k\alpha}.
\end{equation*}
If we note
\(\len{\T{\fun{rev}_0}{n}}\)\index{trace_rev0@$\T{\fun{rev}_0}{n}$}
the length of~\(\T{\fun{rev}_0}{n}\), that is, the number of rule
applications it contains, we expect to have the equations \(\len{x} =
1\), for a rule~\(x\), and \(\len{x \cdot y} = \len{x} + \len{y}\),
for rules \(x\)~and~\(y\). By definition of the
cost:\index{rev0@$\C{\fun{rev}_0}{n}$}
\begin{align*}
\C{\fun{rev}_0}{n}
  &:= \len{\T{\fun{rev}_0}{n}}
    = \left\lvert\delta^n\gamma
        \prod_{k=0}^{n-1}{\beta^k\alpha}\right\lvert
    = \len{\delta^n\gamma} + \sum_{k=0}^{n-1}\len{\beta^k\alpha}\\
   &= (n+1) + \sum_{k=0}^{n-1}(k+1) = (n+1) + \sum_{k=1}^{n+1}k
    = \frac{1}{2}n^2 + \frac{3}{2}n + 1.
\end{align*}
The reason for this inefficiency can be seen in the fact that
rule~\clause{\delta} produces a series of calls to
\fun{cat/2}\index{cat@\fun{cat/2}} following the pattern
\begin{equation}
\fun{rev}_0(s) \twoheadrightarrow \fun{cat}(\fun{cat}(\dots
\fun{cat}(\el, [x_n]), \dots, [x_2]), [x_1]),
\label{eq:rev0}
\end{equation}
where \(s = [x_1, x_2, \dots, x_n]\). The cost of all these calls to
\fun{cat/2}\index{cat@\fun{cat/2}} is thus
\begin{equation*}
\abovedisplayskip=4pt
\belowdisplayskip=4pt
 1 + 2 + \dots + (n-1) = \tfrac{1}{2}n(n-1) \sim
\tfrac{1}{2}n^2,
\end{equation*}
because the cost of \(\fun{cat}(s,t)\)\index{cat@\fun{cat/2}} is \(1 +
\fun{len}(s)\), where
\begin{equation}
%\abovedisplayskip=4pt
%\belowdisplayskip=4pt
\begin{array}{r@{\;}c@{\;}l}
\fun{len}(\el) & \xrightarrow{\smash{a}} & 0;\\
\fun{len}(\cons{x}{s}) & \xrightarrow{\smash{b}} & 1 + \fun{len}(s).
\label{eq:len}
\end{array}
\end{equation}
The problem is not calling \fun{cat/2}, but the fact that the calls
are embedded in the most unfavourable configuration. Indeed, we proved
the associativity\index{stack!catenation!associativity} of
\fun{cat/2}\index{cat@\fun{cat/2}} \vpageref{proof:assoc_cat}, to wit,
\(\fun{cat}(\fun{cat}(s,t),u) \equiv
\fun{cat}(s,\fun{cat}(t,u))\).

\par\vskip\baselineskip

Let \(\Call{f(x)}\) be the cost of the call \(f(x)\). Then
\begin{align*}
\Call{\fun{cat}(\fun{cat}(s,t),u)}
  &= (\fun{len}(s) + 1) + (\fun{len}(\fun{cat}(s,t)) + 1)\\
  &= (\fun{len}(s) + 1) + (\fun{len}(s) + \fun{len}(t) + 1) & (\pred{LenCat}{s,t})\\
  &= 2 \cdot \fun{len}(s) + \fun{len}(t) + 2,
\end{align*}
assuming \(\pred{LenCat}{s,t} \colon \fun{len}(\fun{cat}(s,t)) =
\fun{len}(s) + \fun{len}(t)\). On the other hand:
\begin{align*}
  \Call{\fun{cat}(s,\fun{cat}(t,u))}
  &= (\fun{len}(t) + 1) + (\fun{len}(s) + 1)\\
  &= \fun{len}(s) + \fun{len}(t) + 2.
\end{align*}
From which we conclude:
\begin{equation}
%\abovedisplayskip=4pt
%\belowdisplayskip=4pt
\Call{\fun{cat}(\fun{cat}(s,t),u)} = \fun{len}(s) +
\Call{\fun{cat}(s,\fun{cat}(t,u))}.
\label{ineq:cat_assoc}
\end{equation}
The items of~\(s\) are being traversed twice, although one visit
suffices.

Yet another way to determine the cost of \fun{rev\(_0\)/1} consists in
first guessing that it is \emph{quadratic}\index{cost!quadratic
  $\sim$}, that is, \(\C{\fun{rev}_0}{n} = an^2 + bn +
c\)\index{rev0@$\C{\fun{rev}_0}{n}$}, where \(a\), \(b\) and~\(c\) are
unknowns.  Since there are three coefficients, we only need three
values of~\(\C{\fun{rev}_0}{n}\) to determine them, for instance
\(n=0, 1, 2\). Making some traces, we find \(\C{\fun{rev}_0}{0} = 1\),
\(\C{\fun{rev}_0}{1} = 3\) and \(\C{\fun{rev}_0}{2} = 6\), so we solve
\begin{equation*}
%\abovedisplayskip=4pt
%\belowdisplayskip=4pt
\C{\fun{rev}_0}{0} = c = 1,\quad
\C{\fun{rev}_0}{1} = a + b + c = 3,\quad
\C{\fun{rev}_0}{2} = a \cdot 2^2 + b \cdot 2 + c = 6.
\end{equation*}
We draw \(a = \myfrac1/2\), \(b = \myfrac3/2\) and~\(c = 1\), that is
\(\C{\fun{rev}_0}{n} = (n^2 + 3n +
2)/2\)\index{rev0@$\C{\fun{rev}_0}{n}$}. Since the assumption about
the quadratic behaviour could have been wrong, it is then important to
try other values with the newly found formula, for instance
\(\C{\fun{rev}_0}{4} = (4+1)(4+2)/2 = 15\), then compare with the cost
of \(\fun{rev}_0([1,2,3,4])\)\index{rev0@$\C{\fun{rev}_0}{n}$}, for
example. Here, the contents of the stack is irrelevant, only its
length matters.

After finding a formula for the cost using the empirical method above,
it is necessary to prove it for all values of~\(n\). Since the initial
equations are recurrent, the proof method of choice is
\emph{induction}.

Let \(\pred{Quad}{n}\)\index{Quad@\predName{Quad}} be the property
\(\C{\fun{rev}_0}{n} = (n^2 + 3n + 2)/2\). We already checked its
validity for some small values, here, \(n = 0, 1, 2\). Let us suppose
it is valid for some value of~\(n\) (induction hypothesis) and let us
prove \(\pred{Quad}{n+1}\). We already know \(\C{\fun{rev}_0}{n+1} =
\C{\fun{rev}_0}{n} + n + 2\)\index{rev0@$\C{\fun{rev}_0}{n}$}. The
induction hypothesis implies
\begin{equation*}
%\abovedisplayskip=4pt
%\belowdisplayskip=4pt
\C{\fun{rev}_0}{n+1} = (n^2 + 3n + 2)/2 + n + 2
                     = ((n+1)^2 + 3(n+1) + 2)/2,
\end{equation*}
which is \(\pred{Quad}{n+1}\)\index{Quad@\predName{Quad}}. Therefore,
the induction principle says that the cost we found experimentally is
always correct.

To draw the cost of \fun{rev/1}\index{rev@\fun{rev/1}}, it is
sufficient to notice that the first argument of
\fun{rcat/2}\index{rcat@\fun{rcat/2}} strictly decreases at each
rewrite, so an evaluation trace has the shape \(\epsilon\eta^n\zeta\),
so \(\C{\fun{rev}}{n} = n + 2\). The cost is
\emph{linear}\index{cost!linear $\sim$}, so
\fun{rev/1}\index{rev@\fun{rev/1}} must be used instead of
\fun{rev\(_0\)/1}\index{rev0@\fun{rev\(_0\)/1}} in all contexts.

%\newpage
\paragraph{Exercises}

\begin{enumerate}

  \item Prove \(\pred{LenRev}{s} \colon \fun{len}(\fun{rev}_0(s)) \equiv
  \fun{len}(s)\).

  \item Prove \(\pred{LenCat}{s,t} \colon \fun{len}(\fun{cat}(s,t)) \equiv
  \fun{len}(s) + \fun{len}(t)\).

  \item What is wrong with the proof of involution of
  \fun{rev/1}\index{rev@\fun{rev/1}} in section~3.4.9 of the book of
  \cite{CousineauMauny_1998}?

\end{enumerate}