\section{$\mathbf{A} = \mathbf{U\Sigma V}^{\dagger}$ Singular Value Decomposition}
\label{sec:svd}

The Singular Value Decomposition (SVD) exists for \underline{any} matrix $\mathbf{A}\in\mathbb{C}^{m\times n}$, and is a closely related alternative to the eigendecomposition (cf. section \ref{sec:diagonalization}) that works for non-square matrices. The contrast to diagonalization is that the eigenvectors and eigenvalues of $\mathbf{A}^{\dagger}\mathbf{A}$ and $\mathbf{A}\mathbf{A}^{\dagger}$ found, rather than to look for the eigensystem of $\mathbf{A}$ itself. The advantage is that $\mathbf{A}^{\dagger}\mathbf{A}$ and $\mathbf{A}\mathbf{A}^{\dagger}$ always have the convenient properties of being square, hermitian and positive semidefinite.

SVD comes up incessantly in the context of data analysis. In general, the decomposition has the form:

\begin{equation}
\mathbf{A} = \mathbf{U\Sigma V^{\dagger}}
\end{equation}

Where $\mathbf{U}$ and $\mathbf{V}$ are unitary, and $\Sigma$ is a diagonal matrix with real and positive entries $\sigma_i^2$ along the diagonal, so that $\sigma_1 \geq \sigma^2_2 \geq ...\geq \sigma^2_n$. $\mathbf{U}$ is the matrix of left singular vectors, which are the eigenvectors of $\mathbf{A A^{\dagger}}$. $\mathbf{V}$ is the matrix of right singular vectors, which are the eigenvectors of $\mathbf{A^{\dagger} A}$. The singular values are the square roots of the eigenvalues of $\mathbf{A^{\dagger}A}$ or, equivalently, $\mathbf{AA^{\dagger}}$. If the $\mathbf{A}$ happens to be square and symmetric ($\mathbf{A}=\mathbf{A^T}$, cf. section \ref{sec:hermitian}), then the singular values are simply the absolute values of the eigenvalues of $\mathbf{A}$. The right and left singular vectors of the matrix are orthonormal bases that respectively span the column and row space of $\mathbf{A}$. The number of singular values of $\mathbf{A}$ is the rank of $\mathbf{A}$.
\\
The singular values are always real and positive or zero, because $\mathbf{A^{\dagger} A}$ and $\mathbf{A A^{\dagger}}$ are  hermitian and positive semidefinite (cf. sections \ref{sec:hermitian}, \ref{sec:definite}). When the matrix $\mathbf{A}$ has purely real entries, then the left and right singular vectors are also real, which is not necessarily the case for complex $\mathbf{A}$.
\\

The SVD of a matrix is unique up to the sign columns in $\mathbf{U}$ and $\mathbf{V}$. That is, the SVD is valid under transforming $\mathbf{u}_i, \mathbf{v}_i \rightarrow -\mathbf{u}_i, -\mathbf{v}_i $ if $\mathbf{u}_i$ and $ \mathbf{v}_i$ are the $i$th vectors in $\mathbf{U}$ and $\mathbf{V}$. This has the consequence that the basis of singular vectors that is found using SVD does not remain consistently oriented in the presence or noise or the slow evolution of a system. This is addressed in greater length in section \ref{sec:datasvd} on performing SVD specifically on data matrices.

\subsection{Full and Economy SVDs}
While these properties are always true, unfortunately people use a range of conventions when it comes to the size of $\mathbf{U}$, $\mathbf{\Sigma}$ and $\mathbf{V}$. \\

The first convention is for $\mathbf{U}$ and $\mathbf{V}$ to be square, in which case they contain the full set of left and right singular vectors, and $\Sigma$ has dimension $m\times n$, with $0$ entries in rows $i>n$. This is known as the \textit{full SVD}.\\

\citeasnoun{friedman2001elements} uses the the convention where $\Sigma$ is square, i.e. $\mathbf{U}:\ m\times n$, $\mathbf{\Sigma}:\ n\times n$ and $\mathbf{V}:\ n\times n$. This means that $U$ does not contain the full set of $m$ left singular vectors. Note that, in this case, $\mathbf{U^{\dagger}}\mathbf{U} = \mathbf{I}$ but $\mathbf{U}\mathbf{U^{\dagger}} \neq \mathbf{I}$. \possessivecite{friedman2001elements} convention is of advantage in the context of data analysis, where the data matrix tends to be "tall and skinny" (i.e. $m>>n$), and only the first $n$ left singular vectors are relevant. Also, a letting $\Sigma$ be a square matrix significantly simplifies calculations. \possessivecite{friedman2001elements} is known as the \textit{economy} or \textit{compact} SVD.\\

The two different layouts are illustrated in Figure \ref{fig:full_vs_economy_svd}, which I brazenly copied from \citeasnoun{mathworkseigs}.

\begin{figure}
\centering
\includegraphics[scale=0.5]{full_vs_economy_svd.png}
\caption{Dimensions for Full vs. Economy SVDs}
\label{fig:full_vs_economy_svd}
\end{figure}

\subsection{Matrix Approximation}
The SVD enables the decomposition of the matrix into a sum:

\begin{equation}
\mathbf{A} = \sum_{i=1}^n \sigma_i (\mathbf{u}_i \otimes \mathbf{v}_i)
\end{equation}

Which may be truncated at some rank $r$, resulting in a rank-$r$ approximation to $\mathbf{A}$. This is known as the \textit{truncated singular value estimator} (TSVD):

\begin{equation}
\mathbf{\hat{A}} = \sum_{i=1}^r \sigma_i (\mathbf{u}_i \otimes \mathbf{v}_i)
\end{equation}

Where $\mathbf{u}_i$ is the $i$th left singular vector and $\mathbf{v}_i$ is the $i$th right singular vector. Following the Eckart-Young-Mirsky Theorem, taking the first $r \leq n$ terms of this series is the best rank-$r$ approximation to $\mathbf{A}$ under the Frobenius norm of the error, i.e.  $\underset{\mathrm{rank}(\mathbf{A}^{(r)})=r\leq n}{\argmin} ||\mathbf{A^r}-\mathbf{A}||_F$. The Frobenius norm essentially measures the elementwise mean squares error (MSE), cf. section \ref{sec:frobenius}).\\

In the context of data analysis, the reduced rank representation of a data matrix $\mathbf{A}$ acts as a filter in which the components that explain less of the variance in the dataset are removed, with the hope being that these correspond to noise. The intuition is that noise probably originates from a random process, so that components that correspond to noise have weak correlation with the entries of the dataset and therefore small singular values. On the other hand, the signal probably systematic, so that the components that correspond to the signal have larger singular value. Since TSVD minimizes the average mean squares error (AMSE) amounts to the rank-$r$ maximum likelihood estimator of the signal under the assumption of normally distributed noise with mean zero.

By virtue of being expressible in terms of fewer base vectors, a lower rank approximation is also a very effective compression method. For a more elaborate discussion of SVD in data analysis, see section \ref{sec:datasvd}. In a practical context, the question of selecting the appropriate rank of the approximation emerges. Section \ref{sec:truncation} covers a few approaches to this.

% Geometric interpretation
\subsection{Geometric Interpretation of SVD}
A common intuitive interpretation of $\mathbf{U}$, $\mathbf{\Sigma}$ and $\mathbf{V}$ is to see them as a decomposition of the action of the matrix $\mathbf{A}\in\mathbb{R}^{m\times n}$ into a rotation, a stretching and another rotation. Let $\{\mathbf{x} : |\mathbf{x}|=1 \}$ be the points on the surface of a unit sphere, then $\mathbf{Ax}$:

\begin{itemize}
\item $\mathbf{V}$ rotates the unit sphere, which has no effect. However, it expresses each $\mathbf{x}$ in terms of a different coordinate system (the right singular basis of $\mathbf{A}$).
\item $\mathbf{\Sigma}$ stretches the unit sphere into an ellipse with principal axes aligned to that coordinate system.
\item $\mathbf{U}$ rotates the ellipse.
\end{itemize}

In the above, "rotation" might include coordinate flips that are undone by $\mathbf{U}$ following $\mathbf{V}$. Alternatively, for a vector $\mathbf{v}\in\mathbf{R}^{n}$, the operation $\mathbf{Av}$:

\begin{itemize}
\item $\mathbf{V}$ is a change of basis matrix that expresses $\mathbf{x}$ in terms of a right eigenbasis of $\mathbf{A}$
\item $\mathbf{\Sigma}$ performs a stretching in that basis
\item $\mathbf{U}$ is a change of basis from the right eigenspace into the left eigenspace of $\mathbf{A}$.
\end{itemize}


\section*{Exercises}

\begin{ex} In the following, polar coordinates $(r,\theta) $ for a
point in the plane are given. Find the corresponding Cartesian coordinates.

\begin{enumerate}
\item $(2,\pi /4) $

\item $(-2,\pi /4) $

\item $(3,\pi /3) $

\item $(-3,\pi /3) $

\item $(2,5\pi /6) $

\item $(-2,11\pi /6) $

\item $(2,\pi /2) $

\item $(1,3\pi /2) $

\item $(-3,3\pi /4) $

\item $(3,5\pi /4) $

\item $(-2,\pi /6) $
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Consider the following Cartesian coordinates $(x,y)$. Find polar coordinates corresponding to these points.

\begin{enumerate}
\item $(-1,1) $

\item $(\sqrt{3},-1) $

\item $(0,2) $

\item $(-5,0) $

\item $(-2\sqrt{3},2) $

\item $(2,-2) $

\item $(-1,\sqrt{3}) $

\item $(-1,-\sqrt{3}) $
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} The following relations are written in terms of Cartesian coordinates $(x, y)$. Rewrite them in terms of polar coordinates, $(r, \theta)$.

\begin{enumerate}
\item $y=x^{2}$

\item $y=2x+6$

\item $x^{2}+y^{2}=4$

\item $x^{2}-y^{2}=1$
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Use a calculator or computer algebra system to graph the following
polar relations.

\begin{enumerate}
\item $r=1-\sin (2\theta) ,\theta \in \mat{0,2\pi} $

\item $r=\sin (4\theta) ,\theta \in \mat{0,2\pi} $

\item $r=\cos (3\theta) +\sin (2\theta) ,\theta
\in \mat{0,2\pi} $

\item $r=\theta ,\ \theta \in \mat{0,15} $
\end{enumerate}
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=1+\sin \theta $ for $\theta \in \mat{0,2\pi}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=2+\sin \theta $ for $\theta \in \mat{0,2\pi}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=1+2\sin \theta $ for $\theta \in \mat{0,2\pi}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=2+\sin (2\theta) $ for $\theta \in \mat{0,2\pi
}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=1+\sin (2\theta) $ for $\theta \in \mat{0,2\pi
}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Graph the polar equation $r=1+\sin (3\theta) $ for $\theta \in \mat{0,2\pi
}$.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} Describe how to solve for $r$ and $\theta $ in terms of $x$ and $y$ in polar
coordinates.
%\begin{sol}
%\end{sol}
\end{ex}

\begin{ex} This problem deals with parabolas, ellipses, and
hyperbolas and their equations. Let $l,e>0$ and consider
\begin{equation*}
r=\frac{l}{1\pm e\cos \theta }
\end{equation*}
Show that if $e=0$, the graph of this equation gives a circle. Show that if $0<e<1$, the graph is an ellipse, if $e=1$
it is a parabola and if $e>1$, it is a hyperbola.
%\begin{sol}
%\end{sol}
\end{ex}


\section*{Exercises}

\begin{ex}
  Find the basic solutions of each of the following homogeneous
  systems of equations.
  \begin{equation*}
    (a)~
    \begin{array}{r@{~}c@{~}l}
      2x + 3y + 4z &=& 0 \\
      x - 2y + z &=& 0 \\
      4x - y + 6z &=& 0
    \end{array}
    \quad
    (b)~
    \begin{array}{r@{~}c@{~}l}
       x -  y +  z &=& 0 \\
      -x - 2y - 4z &=& 0 \\
      2x +  y + 5z &=& 0
    \end{array}
    \quad
    (c)~
    \begin{array}{r@{~}c@{~}l}
      x +  y -  z + 2w &=& 0 \\
      x + 3y +  z + 6w &=& 0 \\
      x + 2y      + 4w &=& 0
    \end{array}
  \end{equation*}
\end{ex}

\begin{ex}
  Which of the following homogeneous systems of linear equations have
  non-trivial solutions?
  \begin{enumerate}
  \item[(a)] 4 equations in 3 variables, rank 3.
  \item[(b)] 3 equations in 4 variables, rank 3.
  \item[(c)] 4 equations in 3 variables, rank 2.
  \item[(d)] 3 equations in 4 variables, rank 2.
  \end{enumerate}

  \begin{sol}
    The systems (b), (c), and (d) have non-trivial solutions, because
    the rank is less than the number of variables.
  \end{sol}
\end{ex}

\begin{ex}
  My system of equations has a solution $(x,y,z) = (1,2,4)$. The
  associated homogeneous system has basic solutions $(x,y,z) =
  (1,0,1)$ and $(x,y,z)=(0,1,-1)$. What is the general solution of my
  system of equations?

  \begin{sol}
    The general solution is $(x,y,z) = (1,2,4)+s(1,0,1)+t(0,1,-1)$, or
    equivalently, $(x,y,z) = (1+s, 2+t, 4+s-t)$, where $s$ and $t$ are
    parameters.
  \end{sol}
\end{ex}


% based on example 8 in pythontex_gallery
% https://github.com/gpoore/pythontex/

\documentclass[12pt]{mpllatex}
\usepackage{examples}

\begin{document}

\section*{Step-by-step integration}

This is another nice example drawn from the Pythontex gallery, see
\ \url{https://github.com/gpoore/pythontex}.

It shows the step-by-step computations of a simple triple integral.

\vspace{12pt}

\begin{maple}
   # Define limits of integration
   x_max := 2:   y_max := 3:   z_max := 4:
   x_min := 0:   y_min := 0:   z_min := 0:

   ans := int(f(x,y,z), [x=x_min..x_max, y=y_min..y_max, z=z_min..z_max]):            # mpl(lhs.01,ans)

   f := (x,y,z) -> x*y + y*sin(z) + cos(x+y):

   ans := ''int''(''int''(''int''(f(x,y,z), x=x_min..x_max), y=y_min..y_max), z=z_min..z_max):    # mpl(rhs.01,ans)
   ans := ''int''(''int''(int(f(x,y,z), x=x_min..x_max), y=y_min..y_max), z=z_min..z_max):        # mpl(rhs.02,ans)
   ans := ''int''(int(int(f(x,y,z), x=x_min..x_max), y=y_min..y_max), z=z_min..z_max):            # mpl(rhs.03,ans)
   ans := int(int(int(f(x,y,z), x=x_min..x_max), y=y_min..y_max), z=z_min..z_max):                # mpl(rhs.04,ans)

   # And now, a numerical approximation
   ans := evalf[15](ans):                                                             # mpl(rhs.05,ans)

\end{maple}

\begin{minipage}[t]{0.65\textwidth}
\begin{align*}
   \mpl{lhs.01} &= \mpl{rhs.01}\\
                &= \mpl{rhs.02}\\
                &= \mpl{rhs.03}\\
                &= \mpl{rhs.04}\\
                &\approx \mpl{rhs.05}
\end{align*}
\end{minipage}
\hskip 1cm
\lower16pt\hbox{%
\begin{minipage}[t]{0.35\textwidth}
\begin{latex}
   \begin{align*}
      \mpl{lhs.01} &= \mpl{rhs.01}\\
                   &= \mpl{rhs.02}\\
                   &= \mpl{rhs.03}\\
                   &= \mpl{rhs.04}\\
                   &\approx \mpl{rhs.05}
   \end{align*}
\end{latex}
\end{minipage}}

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 1}

\begin{enumerate}[label=(\alph*)]
\item Use the substitution method to show that $T(n) = 8T(n/2) + n^3 = \mathcal{O}(n^3\log n)$.
Assume $T(2)=d$ where $d$ is a constant.
Use induction and do not skip stages.

\item Prove or disprove $f(n) = \Theta(f(n/2))$.
If true, show why.
Otherwise give a counter-example.
\end{enumerate}

\subsection*{Solution}

\begin{enumerate}[label=(\alph*)]
\item Proof is given by induction on $n$.
Using assumption $T(2) = d$, $T(1) = \frac{T(2)-1}{8} = \frac{d-1}{8} = e$ where $e$ is a constant.
\begin{enumerate}[label=\arabic*.]
\item \textit{Base Case}\\
We take the base step as $n = 1$.
In this case, there exist constants $f > 0$, $n_0 = 1$ where $f = e(\frac{d}{8}+\frac{d}{1})$ such that $T(1) \leq f \times n^3 \log n$ for all $n \geq n_0$.
Therefore, $T(1) = \mathcal{O}(n^3 \log n)$.
\item \textit{Inductive Hypothesis}\\
We form inductive hypothesis as $T(k)=\mathcal{O}(n^3 \log n)$.
In other words, that there are constants $c$ and $n'_0$ such that $T(k) \leq ck^3 \log k$ for $k \geq n'_0$.
\item \textit{Induction Step}\\
Using inductive hypothesis we will show $T(2k) = O(n^3\log n)$; in other words, that there are constants $c^\prime$ and $n''_0$ such that $T(2k) \leq c' k^3 \log k$ for all $k \geq n''_0$.

\begin{equation}\label{eq11}
T(2k) = 8 T(\frac{2k}{2}) + (2k)^3
\end{equation}
By Inductive hypothesis, Equation \ref{eq11} can be written as shown in Equation \ref{eq12}.
\begin{equation}\label{eq12}
\begin{aligned}
T(2k) &\leq 8c \big( (2k)^3 \log (2k) \big) + (2k)^3\\
&\leq (8c + 1) (2k)^3 \log (2k)\\
&\leq 8(8c+1) (k)^3 (\log (k) + \log 2)\\
&\leq 16(8c+1) (k^3 \log k)
\end{aligned}
\end{equation}
Therefore, by taking $c' = 16(8c+1)$ and $n''_0 = n'_0$ we have shown Equation \ref{eq13} holds for all $k \geq n''_0$.
\begin{equation}\label{eq13}
T(2k) \leq c' \big( (2k)^3 \log 2k \big)
\end{equation}
Hence, we have shown $T(2k) = \mathcal{O}(n^3\log n)$ and the induction is complete.
\end{enumerate}
\item The statement is not valid.
To disprove, $f(n)=2^n$ can be given as a counter-example.
What statement suggests is there are constants $C$ and $n_0$ such that $2^n < C(2^{n/2})$; in other words, $2^{n/2} < C$ for all $n > n_0$.
This is obviously not true since there is no limit in our choice of $n$.
\end{enumerate}


\section{Eigenvalues and Eigenvectors}
For a linear map $T:V\to V$ where $V$ is a real or complex vector space, a vector $\underline{v}\neq\underline{0}$ is called an eigenvector of $T$ with eigenvalue $\lambda$ if $T\underline{v}=\lambda\underline{v}$.
Note that this happens if and only if
$$A\underline{v}=\lambda\underline{v}\iff (A-\lambda I)\underline{v}=\underline{0}$$
where $A$ is the matrix of $T$.
So $\lambda$ is an eigenvalue if and only if $\det(A-\lambda I)=0$ if and only if it is a root of the polynomial
$$\chi_A(t)=\det(A-tI)$$
which is a degree $n$ polynomial.
\begin{definition}
    For $A$ an $n\times n$ matrix, the characteristic polynomial $\chi_A(t)$ is defined as $\det(A-tI)$.
\end{definition}
This gives us a way to find the eigenvalues by looking at the characteristic polynomial and to find the eigenvectors by looking at the kernel of $A-\lambda I$.
\begin{example}
    1. Let $V=\mathbb C^3$ and
    $$A=\begin{pmatrix}
        2&i\\
        -i&2
    \end{pmatrix}$$
    So $\chi_A(t)=(2-t)^2-1$, so $\lambda$ is $1$ or $3$.
    For the eigenvalue $\lambda=1$, we find that the eigenvectors are spanned by $(1,i)$, and for $\lambda=3$, the eigenvectors are spanned by $(1,-i)$.
    \\
    2. Let $V=\mathbb R^2$, consider the shear
    $$A=
    \begin{pmatrix}
        1&1\\
        0&1
    \end{pmatrix}$$
    So $\chi_A(t)=(1-t)^2$, so $\lambda=1$.
    The eigenvectors are then spanned by $(1,0)$
    \\
    3. Let $V=\mathbb C^2$, and
    $$U=\begin{pmatrix}
        \cos\theta&-\sin\theta\\
        \sin\theta&\cos\theta
    \end{pmatrix}$$
    So $\chi_U(t)=t^2-2t\cos\theta+1$, so $\lambda=e^{\pm i\theta}$, so the eigenvectors are spanned by $(1,\pm i)$ respectively.\\
    4. Consider $V=\mathbb C^n$ and $A=\operatorname{diag}(\lambda_1,\lambda_2,\ldots,\lambda_n)$, then the eigenvalues are $\lambda_i$ and the corresponding eigenvectors are spanned by $\underline{e_i}$.
\end{example}
\begin{proposition}
    1. There exists at least one eigenvalue.
    In fact, there exists $n$ eigenvalues counting multiplicity.\\
    2. The trace of the matrix equals the sum of the eigenvalues (counting multiplicity as well).\\
    3. $\det A=\chi_A(0)$.\\
    4. If $A$ is real, then non-real eigenvalues occur in conjugates.
\end{proposition}
\begin{proof}
    Trivial.
\end{proof}
\begin{definition}
    For an eigenvalue $\lambda$ of a matrix $A$, define the eigenspace $E_\lambda$ is the set of all $\underline{v}$ with $A\underline{v}=\lambda\underline{v}$, so $E_\lambda=\ker(A-\lambda I)$.
    The geometrical multiplicity of $\lambda$ is $m_\lambda=\operatorname{null}E_\lambda$, and the algebraic multiplicity $M_\lambda$ is the multiplicity of $\lambda$ as a root of $\chi_A(t)$.
    So $\chi_A(t)=(t-\lambda)f(t)$ for $f(\lambda)\neq 0$.
\end{definition}
\begin{proposition}\label{alg_ge_geom}
    $M_\lambda\ge m_\lambda$.
\end{proposition}
\begin{proof}
    Postponed to later.
\end{proof}
Note that the strict inequality can happen in many non-trivial cases.
\begin{example}
    1. Take
    $$\begin{pmatrix}
        -2&2&-3\\
        2&1&-6\\
        -1&-2&0
    \end{pmatrix}$$
    Then $\chi_A(t)=(5-t)(t+3)^2$, so $5,-3$ are the eigenvalues.
    By calculation we find that $E_5=\operatorname{span}\{(1,2,-1)^\top\}$ and $E_{-3}=\operatorname{span}\{(-2,1,0)^\top,(3,0,1)^\top\}$, so $M_5=m_5=1,M_{-3}=m_{-3}=2$.\\
    2. Take
    $$A=\begin{pmatrix}
        -3&-1&1\\
        -1&-3&1\\
        -2&-2&0
    \end{pmatrix}$$
    Then $\chi_A(t)=-(t+2)^3$, so the only eigenvalue is $-2$, but $E_{-2}$ only has dimension $2$, therefore $M_{-2}=3\neq 2=m_{-2}$.\\
    3. Consider the reflection across some plane through the origin.
    It reflects every point on the normal without changing direction and fixes every point on the plane.
    So the only eigenvalues would be $\pm 1$ and $M_{-1}=m_{-1}=1,M_1=m_1=2$.\\
    4. Consider the generic rotation through some normal.
    So the only real eigenvalue would be $1$ and $M_1=m_1=1$ since the transformation changes every vector's direction except those on the normal, which it fixes.
\end{example}
\begin{proposition}
    Let $\underline{v_1},\underline{v_2},\ldots,\underline{v_r}$ be eigenvectors of a matrix $A$ with eigenvalues $\lambda_1,\lambda_2,\ldots,\lambda_r$.
    If the eigenvalues are distinct, then the eigenvectors are linearly independent.
\end{proposition}
\begin{proof}
    Suppose $\underline{w}=\sum_ia_i\underline{v_i}$, then $(A-\lambda I)\underline{w}=\sum_ia_i(\lambda_j-\lambda)\underline{v}$.
    Let $\underline{w}=\underline{0}$, and if we can find a linear combination where not all $a_i$'s is $0$, then we pick such a linear combination $\underline{0}=\sum_ia_i\underline{v_i}$ such that the number of nonzero $a_i$'s is the least.
    WLOG in this choice $a_1\neq 0$, then $\underline{0}=(A+\lambda I)\underline{0}=\sum_ia_i(\lambda_j-\lambda_1)\underline{v}$, which contradicts the minimality of the number of nonzero $a_i$ of our choice.
\end{proof}
\begin{proof}[Alternative proof]
    Given a linear relation $\underline{0}=\sum_ia_i\underline{v_i}$, then for any $j$, the consider
    $$\underline{0}=\prod_{i\neq j}(A-\lambda_iI)\sum_{i=1}^ra_i\underline{v_i}=a_j\underline{v_j}\prod_{i\neq j}(\lambda_j-\lambda_i)$$
    So $a_j=0$.
\end{proof}
\subsection{Diagonalizability}
\begin{proposition}
    For an $n\times n$ matrix $A$ acting on $V=\mathbb R^n$ or $\mathbb C^n$, the followings are equivalent:\\
    1. There exists a basis for $V$ consisting of eigenvectors of $A$.\\
    2. There exists an $n\times n$ invertible matrix $P$ such that $P^{-1}AP=D$ where is a diagonal matrix whose entries are the eigenvalues.
\end{proposition}
If either of these conditions hold, then we say $A$ is diagonalizable.
\begin{proof}
    For any matrix $P$, $AP$ has columns $A\underline{C_i}^{(P)}$, and $DP$ has columns
    $$\lambda_i\underline{C_i}^{(P)}$$
    so what it is saying is that every column of $P$ is an eigenvector of $A$.
    So if there is a basis of eigenvectors, we can just take $P$ to be the matrix whose columns are the basis eigenvectors.
    Conversely, given such a matrix $P$, its columns are a basis for $V$.
\end{proof}
\begin{example}
    1. Consider
    $$A=
    \begin{pmatrix}
        1&1\\
        0&1
    \end{pmatrix}$$
    So we only have one eigenvector, hence $A$ is not diagonalizable.\\
    2. Consider
    $$U=
    \begin{pmatrix}
        \cos\theta&-\sin\theta\\
        \sin\theta&\cos\theta
    \end{pmatrix}$$
    So it does not have real eigenvalues, but it has $2$ complex eigenvalues with linearly independent eigenvectors, thus it is diagonalizable over $\mathbb C^n$.
    Indeed, we can let
    $$P=
    \begin{pmatrix}
        1&1\\
        -i&i
    \end{pmatrix}$$
    Then
    $$P^{-1}UP=\begin{pmatrix}
        e^{i\theta}&0\\
        0&e^{-i\theta}
    \end{pmatrix}$$
\end{example}
\begin{proposition}
    If $A$ has $n$ distinct eigenvalues, then it is disgonalizable.
\end{proposition}
\begin{proof}
    Different eigenvalues give linearly independent eigenvectors.
\end{proof}
\begin{proposition}
    $A$ is diagonalizable if and only if $M_\lambda=m_\lambda$ for each eigenvalue $\lambda$ of $A$.
\end{proposition}
\begin{proof}
    The union of the basis vectors for the eigenspaces would be a basis for the entire space since we would have $\sum_{\lambda}M_\lambda=\sum_{\lambda}m_\lambda=n$.
\end{proof}
\begin{example}
    1. Again use
    $$A=\begin{pmatrix}
        -2&2&-3\\
        2&1&-6\\
        -1&-2&0
    \end{pmatrix}$$
    We already know from previous example that $\lambda=5,3$ and $M_5=m_5=1,M_{-3}=m_{-3}=2$, so $A$ is diagonalizable.
    Indeed, we have
    $$P^{-1}AP=\begin{pmatrix}
        5&&\\
        &-3&\\
        &&-3
    \end{pmatrix},P=\begin{pmatrix}
        1&-2&3\\
        2&1&0\\
        -1&0&1
    \end{pmatrix}$$
    2. Use the previous example
    $$A=\begin{pmatrix}
        -3&-1&1\\
        -1&-3&1\\
        -2&-2&0
    \end{pmatrix}$$
    Then $\lambda=-2$ and $M_{-2}=3>m_{-2}=2$, so $A$ is not diagonalizable.
    Indeed, if it is, then there is some invertible $P$ with $A=P(-2I)P^{-1}=-2I$ which is a contradiction.
\end{example}
\begin{definition}
    Two $n\times n$ matrices $A,B$ are similar if there is some invertible $n\times n$ matrix $P$ such that $A=PBP^{-1}$.
\end{definition}
This is trivially an equivalence relation.
\begin{proposition}
    If $A,B$ are similar, then $\operatorname{tr}B=\operatorname{tr}A$ and $\det B=\det A$.
    Also $\chi_B=\chi_A$.
\end{proposition}
In fact, if two matrices are similar, then we could get from one to the other by a change of basis.
The results here then are obvious.
\begin{proof}
    $$\operatorname{tr}(B)=\operatorname{tr}(P^{-1}AP)=\operatorname{tr}(APP^{-1})=\operatorname{tr}(A)$$
    $$\det(B)=\det(P^{-1}AP)=\det(P)^{-1}\det(A)\det(P)=\det(A)$$
    $$\chi_A(t)=\det(A-tI)=\det(P^{-1})\det(A-tI)\det(P)=\det(B-tI)=\chi_B(t)$$
    As desired.
\end{proof}
Note that if $A$ is duagonalizable, then so is $B$ since they are all in the same equivalence class containing some diagonal matrix.
\begin{proof}[Proof of Theorem \ref{alg_ge_geom}]
    Choose an eigenvalue $\lambda$ of an $n\times n$ matrix $A$ with $m_\lambda=r$ and a basis $\underline{v_1},\ldots,\underline{v_r}$ for $E_{\lambda}$.
    Extend this basis to the entire space by adding vectors $\underline{w_{r+1}},\ldots,\underline{w_n}$.
    Consider the matrix $P$ with columns $\underline{C_i}(P)=\underline{v_i}$ for $i=1,\ldots,r$ and $\underline{C_a}(P)=w_a$ for $a=r+1,\ldots,n$, then $AP=PB$ where $B$ is of the form
    $$B=\left(\begin{array}{@{}c|c@{}}
        \lambda I&*\\
        \hline
        0&\hat{B}
    \end{array}\right)$$
    Note that $P^{-1}AP=B$, so $\chi_A(t)=\chi_B(t)=(\lambda-t)^r\det(\hat{B}-tI)$, so $M_\lambda\ge m_\lambda$.
\end{proof}
\subsection{Diagonalisation of Hermitian and Symmetric Matrices}
Observe that if $A$ is Hermitian, then $(A\underline{v})^\dagger\underline{w}=\underline{v}^\dagger A\underline{w}$ for any complex vectors $\underline{v},\underline{w}$.
\begin{proposition}
    Hermitian matrices have real eigenvalues and orthogonal eigenvectors for distinct eigenvalues.
    In addition, if the matrix is actually real (hence symmetric), then for each eigenvalue stated above, we can choose real eigenvectors.
\end{proposition}
\begin{proof}
    Let $A$ be a Hemitian matrix and $\lambda$ an eigenvalue of it with eigenvector $\underline{v}$, then
    $$\bar{\lambda}\underline{v}^\dagger\underline{v}=(A\underline{v})^\dagger\underline{v}=\underline{v}^\dagger A\underline{v}=\lambda\underline{v}^\dagger\underline{v}$$
    So $\lambda=\bar\lambda$, thus $\lambda$ is real.\\
    If $\lambda,\mu$ are distinct eigenvalues with eigenvectors $\underline{v},\underline{w}$, then
    $$\mu\underline{v}^\dagger\underline{w}=\underline{v}^\dagger(A\underline{w})=(A\underline{v})^\dagger\underline{w}=\lambda\underline{v}^\dagger\underline{w}\implies (\lambda-\mu)\underline{v}^\dagger\underline{w}=0$$
    So $\underline{v}^\dagger\underline{w}=0$ since $\mu\neq\lambda$, thus $\underline{v}\perp\underline{w}$.\\
    For the last part, just choose the real or imaginary part (at least one of them is nonzero) of the eigenvectors.
\end{proof}
\begin{proposition}[Gram-Schmidt Orthogonalization]
    Given a linearly independent set of vectors $\underline{w_1},\underline{w_2},\ldots,\underline{w_k}$ in $\mathbb C^n$, we can construct a set $\underline{u_1},\underline{u_2},\ldots,\underline{u_k}$ of orthonormal vectors such that $\operatorname{span}(\{\underline{u_i}\})=\operatorname{span}(\{w_i\})$.
\end{proposition}
\begin{proof}
    We do it recursively by
    $$\underline{u_1}=\frac{1}{\|\underline{v_1}\|}\underline{v_1},\forall j>1,\underline{w_j}^{(1)}=\underline{w_j}-(\underline{u_1}^\dagger \underline{w_j})\underline{u_1}$$
    and
    $$\underline{u_r}=\frac{1}{\|\underline{w_r}^{(r-1)}\|}\underline{w_r}^{(r-1)},\forall j>r,\underline{w_j}^{(r)}=\underline{w_j}^{(r-1)}-(\underline{u_r}^\dagger \underline{w_j}^{(r-1)})\underline{u_r}$$
    Which, as one can easily verify, is valid.
\end{proof}
Using Gram-Schmidt, we can convert any basis for some eigenspace to an orthonormal basis $\mathscr B_\lambda$.
If $\lambda_i$ are distinct eigenvalues of some Hermitian basis, then by choosing this orthonormal basis for all eigenspaces, we have an orthonormal set of eigenvectors.
\begin{example}
    1. Take
    $$A=\begin{pmatrix}
        2&i\\
        -i&2
    \end{pmatrix},A^\dagger=A$$
    So the eigenvalues are $1,3$ with eigenvectors $2^{-1/2}(1,i)^\top,2^{-1/2}(1,-i)^\top$.
    Note that these two eigenvectors is an orthonormal basis for $\mathbb R^2$.
    So we can choose
    $$P=\frac{1}{\sqrt{2}}\begin{pmatrix}
        1&1\\
        i&-i
    \end{pmatrix}$$
    Which is unitary and $P^{-1}AP=\operatorname{diag}(1,3)$.\\
    2. Consider
    $$A=\begin{pmatrix}
        0&1&1\\
        1&0&1\\
        1&1&0
    \end{pmatrix}$$
    which is real and symmetric thus Hermitian and have eigenvalues $-1,2$ where the algebraic multiplicity of $-1$ is $2$.
    By calculation we find that the geometric multiplicity of $-1$ is also $2$.
    We can find an orthonormal basis for $E_{-1}$ by the Gram-Schmidt process.
    For example we can have
    $$E_{-1}=\operatorname{span}\{2^{-1/2}(1,-1,0)^\top,6^{-1/2}(1,1,-2)^\top\}$$
    Also $E_{2}=\operatorname{span}\{3^{-1/2}(1,1,1)^\top\}$, thus we have found an orthonormal eigenbasis for $\mathbb R^3$.
    We can also find the corresponding (unitary hence orthogonal) $P$ which diagonalizes $A$.
\end{example}
\begin{theorem}
    Any $n\times n$ Hermitian matrix is diagonalizable by an unitary matrix.
    Equivalently, there is an orthonormal eigenbasis.
    In particular, for real and symmetric case, we can choose such that the orthonormal basis is real.
    \label{hermitian_diag}
\end{theorem}
\begin{proof}
    We have already established all of the theorem but the part of the matrix being diagonalizable, so it suffices to show that, which is sadly not examinable, but here goes:\\
    Consider a Hermitian $A=\mathbb C^n\to\mathbb C^n$.
    If $V$ is any subspace of $\mathbb C^n$ that is invariant under $A$, then for any $\underline{v_1},\underline{v_2}\in V$ we have $\underline{v_1}^\dagger (A\underline{v_2})=(A\underline{v_1})^\dagger\underline{v_2}$.
    We shall prove by induction on $m=\dim V\le n$ that such a subspace has an orthonormal basis of eigenvectors of $A$, which shall establish the result by taking $m=n$.
    As the initial step is trivial, we will show the induction step.\\
    Given $V$ as above, since we are working in $\mathbb C$, the characteristic equation always has a root, so there is some $\underline{v}\in V$ such that $A\underline{v}=\lambda\underline{v}$ for some $\lambda$.
    Let $W$ be the subspace $\{\underline{w}\in V:\underline{v}^\dagger\underline{w}=0\}$, then $\dim W=m-1<m$.
    Now
    $$\underline{w}\in W\implies \underline{v}^\dagger (A\underline{w})=(A\underline{v})^\dagger\underline{w}=\lambda\underline{v}^\dagger\underline{w}=0\implies A\underline{w}\in W$$
    So $W$ is invariant under $A$, so by induction hypothesis, there is an orthonormal basis for $W$ of eigenvectors of $A$, say $\{\underline{u_1},\ldots,\underline{u_{m-1}}\}$, then we deem $\underline{u_m}=|\underline{v}|^{-1}\underline{v}$, then $\{\underline{u_1},\ldots,\underline{u_m}\}$ gives an orthonormal basis of $V$, and the proof is done.
\end{proof}
\subsection{Quadratic Forms}
\begin{definition}
    A quadratic form is a function $\mathcal F:\mathbb R^n\to\mathbb R$ given by $\mathcal{F}(\underline{x})=\underline{x}^\top A\underline{x}$ where $A$ is a real symmetric $n\times n$ matrix.
    So $\mathcal F(\underline{x})=x_iA_{ij}x_j$.
\end{definition}
So by Theorem \ref{hermitian_diag}, we can diagonalize it by an orthogonal matrix $P$.
Then by letting $\underline{x'}=P^\top\underline{x}$, we have $\mathcal F(\underline{x})=\underline{x'}^\top D\underline{x'}$ where $D=\operatorname{diag}(\lambda_1,\lambda_2,\ldots,\lambda_n)$, so we sau that $\mathcal F$ is diagonalized.
Note that $\underline{x}=x_i'\underline{u_i}$ where $\underline{u_i}$ is the orthonormal eigenbasis.
Thus we can interpret what we have done as not changing the vector, but instead changing the basis in which we are working under.
This basis (and the corresponding coordinate axes) is called the principle axes of the quadratic form.
\begin{example}
    1. In $\mathbb R^2$, take
    $$A=\begin{pmatrix}
        \alpha&\beta\\
        \beta&\alpha
    \end{pmatrix}$$
    So we can diagonalize it by $\lambda_1=\alpha+\beta$, $\lambda_2=\alpha-\beta$ and $\underline{u_1}=2^{-1/2}(1,1)^\top,\underline{u_2}=2^{-1/2}(1,-1)^\top$
    Hence $\underline{x'}=(2^{-1/2}(x_1+x_2),2^{-1/2}(x_1-x_2))^\top$.
    For example, if we take $\alpha=3/2,\beta=-1/2$, then the quadratic form defines an ellipse.
    If we let $\beta=3/2,\alpha=-1/2$, then the quadratic from would be a hyperbola.\\
    2. In $\mathbb R^3$, by a suitable choice of basis, $\mathcal F(\underline{x})=\lambda_1x_1^2+\lambda_2x_2^2+\lambda_3^2$.
    If $\lambda_i>0$, the graph is an ellipsoid (a sphere stretched towards the directions of the axes).
    Otherwise, for example, the matrix that we have introduced in a former example
    $$A=\begin{pmatrix}
        0&1&1\\
        1&0&1\\
        1&1&0
    \end{pmatrix}$$
    has eigenvalues $-1,2$ where the algebraic multiplicity of $-1$ is $2$.
    So on the principle axes $\mathcal F_A(\underline{x})=-x_1^2-x_2^2+2x_3^2$.
    Hence it gives a hyperboloid (obtained by rotating a hyperbola).
\end{example}
It has important application when we use the Hessians as the quadratic Form.
\subsection{Cayley-Hamilton Theorem}
\begin{theorem}[Cayley-Hamilton]
    $\chi_A(A)=0$.
\end{theorem}
\begin{remark}
    If $c_0I+c_1A+\cdots+c_nA^n$, then $A(c_1I+C_2A+\cdots+c_nA^{n-1})=-c_0I$.
    So if $c_0=\det A\neq 0$, we have
    $$A^{-1}=-\frac{1}{c_0}(c_1I+C_2A+\cdots+c_nA^{n-1})$$
\end{remark}
\begin{proof}
    The general $2\times 2$ case is immediate.
    The case for diagonalizable matrices is also obvious.
    The general case can be deal with a continuity argument or using modules.
    A proof by basic algebra is also presented in lecture but is deemed not elegant enough by the author, hence is omitted.
\end{proof}


%---------------------------Radius Ratio----------------------------------
\section{Radius Ratio\label{s:tet-radius-ratio}}

This metric is commonly known as the radius ratio since it is the
normalized ratio of the radius of the inscribed sphere to the radius
of the circumsphere. Note that it is equal to the tetrahedral aspect
$\beta$ for positively-oriented tetrahedra.

The radius ratio is the quotient of these two radii normalized by $\frac{1}{3}$ so
that an equilateral tetrahedron has quality of 1:
\begin{eqnarray*}
q & = & \frac{R}{3 r} \nonumber \\
  & = & \frac { \left|
   \normvec{L_3}^2 \left( \vec L_2 \times \vec L_0 \right) +
   \normvec{L_2}^2 \left( \vec L_3 \times \vec L_0 \right) +
   \normvec{L_0}^2 \left( \vec L_3 \times \vec L_2 \right)
   \right| A}{108 V^2}.
\end{eqnarray*}

Note that if $|V| < DBL\_MIN$, we set $q = DBL\_MAX$.

\tetmetrictable{radius ratio}%
{$1$}%                  Dimension
{$[1,3]$}%              Acceptable range
{$[1,DBL\_MAX]$}%       Normal range
{$[1,DBL\_MAX]$}%       Full range
{$1$}%                  Equilateral tet
{\cite{par:93}}%        Citation
{v\_tet\_radius\_ratio}% Verdict function name


\subsection{Representing wave functions}

Wave function are of the form:

\(\cos(ax + b)\)

\(\sin(ax + b)\)

We can use the following identities:

\begin{itemize}
\item \(\cos(x)=\sin(x+\dfrac{\tau }{8})\)
\item \(\sin(-x)=-\sin(x)\)
\item \(\sin(a+b)=\sin(a)\cos(b)+\sin(b)\cos(a)\)
\end{itemize}

So we can write any function as:

\subsubsection{Using \(e\)}


\lab{Algorithms}{QR decomposition}{QR decomposition}
\label{lab:QRdecomp}

\objective{Understand how the QR algorithm works and write your own implementation.}

The QR decomposition is used to represent any matrix as the multiple of an orthogonal matrix and an upper triangular matrix. This decomposition is useful in computing least squares and is part of a common method for finding eigenvalues.

\section*{Review of Gram Schmidt}

\vspace{5mm}
\begin{theorem}[Gram-Schmidt Orthogonalization Process] Let
$\{\x_i\}_{i=1}^n$ be a basis for the inner product space $V$. Let
\[
\q_1 = \frac{\x_1}{\norm{\x_1}},
\]
and define $\q_2,\q_3,\ldots,\q_n$ recursively by
\[
\q_{k+1} = \frac{\x_{k+1} - \p_k}{\norm{\x_{k+1} - \p_k}},
\]
where
\[
\p_k = \sum^k_{j=1} \langle \x_{k+1}, \q_j\rangle \q_j
\]
is a projection of $\x_{k+1}$ onto the subspace $\mbox{Span}(\q_1,\q_2,\ldots,\q_k)$.  Then the set $\{\q_i\}_{i=1}^n$ is an orthonormal basis for $V$.
\end{theorem}
\vspace{5mm}

For the above algorithm, let $r_{k k} = \|\x_k - \p_{k-1}\|$ and
$r_{j k} = \langle \x_k, \q_j\rangle$.  Then
\begin{align*}
r_{1 1} \q_1 &= \x_1\\
r_{k k} \q_k &= \x_k - r_{1 k} \q_1 - r_{2 k} \q_2 - r_{3 k} \q_3 -
\ldots - r_{k-1, k} \q_{k-1},\quad k=2,\ldots,n.
\end{align*}
This can be written as
\begin{align*}
\x_1 &= r_{1 1} \q_1\\
\x_2 &= r_{1 2} \q_1 + r_{2 2} \q_2\\
\vdots \:\: &= \quad \vdots\\
\x_n &= r_{1 n} \q_1 + r_{2 n} \q_2 + \ldots + r_{n n} \q_{n},
\end{align*}
or in matrix form as
\[
\begin{pmatrix}
\vdots & \vdots & & \vdots\\
\x_1 & \x_2 & \cdots & \x_n\\
\vdots & \vdots & & \vdots
\end{pmatrix}
=
\begin{pmatrix}
\vdots & \vdots & & \vdots\\
\q_1 & \q_2 & \cdots & \q_n\\
\vdots & \vdots & & \vdots
\end{pmatrix}
\begin{pmatrix}
r_{1 1} & r_{1 2} & \cdots & r_{1 n}\\
0 & r_{2 2} & \cdots & r_{2 n}\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \cdots & r_{n n}
\end{pmatrix}.
\]
Hence if our original basis $\{\x_i\}_{i=1}^n$ correspond to column
vectors of a matrix $A$, we can likewise write the resulting
orthonormal basis $\{\q_i\}_{i=1}^n$ as a matrix $Q$ of column
vectors.  Then we have that $A = Q R$, where $R$ is the above
nonsingular upper-triangular $n\times n$ matrix.  This is the QR
Decomposition and is summarized by the following theorem:
\vspace{5mm}
\begin{theorem}
Let $A$ be an $m\times n$ matrix of rank $n$.  Then $A$ can be
factored into a product $Q R$, where $Q$ is an $m\times n$ matrix
with orthonormal columns and $R$ is a nonsingular $n \times n$ upper
triangular matrix.
\end{theorem}

There are three mode options available in SciPy's implementation of QR Decomposition.  We will be using the ``economic" option.
\begin{lstlisting}[style=python]
: import scipy as sp
: from scipy import linalg as la
: A = sp.randn(4,3)
: Q, R = la.qr(A, mode='economic')
: sp.dot(Q, R) == A     there will be some False entries
: sp.dot(Q, R) - A
: sp.dot(Q.T, Q)
\end{lstlisting}

In order to interpret the results correctly, we need to understand that the computer has limited precision (especially with floating point numbers).  This is why \li{sp.dot(Q, R)} is not exactly equal to A.  But subtracting the two yields numbers that are essentially zero.  This shows that indeed the product of $Q$ and $R$ is $A$.  Note also that $Q^T Q = I$.  This implies that the column vectors of $Q$ are orthonormal (why?).

\section*{Solving Least Squares Problems}

For large or ill-conditioned problems, the QR decomposition provides
a nice method for computing least squares solutions of
over-determined matrices.  Consider the problem $A x = b$.  Recall
that the least squares solution is $\widehat x = (A^T A)^{-1}A^T b$.
Alternatively, we write the linear system as
\[
Q R x = b.
\]
We then multiply both sides by $Q^T$, yielding
\[
R x = Q^T b.
\]
Then $\widehat x = R^{-1} Q^T b$.

\section*{Computational Remark}

Numerically, the Gram Schmidt process can have problems due to
finite precision arithmetic. Specifically, due to rounding errors,
the resulting basis may not be orthonormal. To combat this, we
actually carry out a slightly revised algorithm called Modified Gram
Schmidt.  To do this, we compute $\q_1$ as before.  We then project
it out of each of the remaining original vectors
$\x_2,\x_3,\ldots,\x_n$ via
\[
\x_k := \x_k - \langle \x_{k}, \q_1\rangle \q_1,\quad k=2,\ldots,n.
\]
Then we compute $\q_2$ to be the unit vector of $\x_2$, that is,
\[
\q_2 = \frac{\x_2}{\|\x_2\|}.
\]
We repeat by projecting out $\q_2$ from the remaining vectors
$\x_3,\x_4,\ldots,\x_n$.

\begin{problem}
\label{prob:QR}
Write your own implementation of the QR decomposition.  It should accept as input a matrix $A$ and computes its QR decomposition, returning the matrices $Q$ and $R$. Be sure to use the numerically stable Modified Gram Schmidt algorithm.
\end{problem}


\part{Inequalities}

\chapter{Piecewise Functions}

\section{Piecewise Functions}

Piecewise functions typically feature one or more points at which the function changes from one form to another. To graph a piecewise function, simply graph each piece and then restrict it to its designated domain. Pay special attention when plotting the breaking point (closed circle includes the point, open circle excludes the point). \\

\begin{exercise}\nonumber
	Graph the piecewise function given by \\
	\begin{align}
		y = \begin{cases}
			1   & x \le 0 \\
			4^x & x > 0
		\end{cases}
	\end{align}

	\begin{figure}[H]
		\centering
		\begin{tikzpicture}[scale=0.3,yscale=0.4]
			\draw[->] (-7,0) -- (7,0) node[right] {$ x $};
			\draw[->] (0,-18) -- (0,18) node[above] {$ y $};
		\end{tikzpicture}
	\end{figure}
\end{exercise}

\begin{exercise}\nonumber
	Graph the piecewise function given by \\
	\begin{align}
		y = \begin{cases}
			{1 \over 2}x + 3 & x < -2         \\
			0                & -2 \le x \le 2 \\
			x^2 - 1          & x > 2
		\end{cases}
	\end{align}

	\begin{figure}[H]
		\centering
		\begin{tikzpicture}[scale=0.5,yscale=0.3]
			\draw[->] (-10,0) -- (6,0) node[right] {$ x $};
			\draw[->] (0,-5) -- (0,20) node[above] {$ y $};
		\end{tikzpicture}
	\end{figure}
\end{exercise}

\newpage

\chapter{Absolute Value Functions}

\section{Absolute Value Functions}

A very special and common piecewise function is the absolute value function. \\

\begin{itemize}
	\item
	      $ y = |x| = \begin{cases} x & x \ge 0 \\ -x & x < 0 \end{cases} $ \\

	\item
	      $ x \in \mathbb R $ \\

	\item
	      $ y \ge 0 $ \\
\end{itemize}

\begin{figure}[H]
	\centering
	\begin{tikzpicture}[scale=0.8]
		\draw[->] (-4,0) -- (4,0) node[right] {$ x $};
		\draw[->] (0,-2) -- (0,4) node[above] {$ y $};
		\draw[very thick,color=red,domain=-3:0] plot (\x,{-\x});
		\draw[very thick,color=red,domain=0:3] plot (\x,{\x}) node[right] {$ y = |x| $};
	\end{tikzpicture}
	\caption{absolute value function}
\end{figure}

\begin{theorem}[Absolute Values]
	\begin{align}
		 & |ab| = |a| \cdot |b|                           \\
		 & \left|{a \over b}\right| = {|a| \over |b|}     \\
		 & if\ |a| \le b,\ then\ -b \le a \le b           \\
		 & if\ |a| \ge b,\ then\ a \ge b \ or \ a \le -b  \\
		 & (Triangle \ Inequality)\ |a + b| \le |a| + |b|
	\end{align}
\end{theorem}

\chapter{Inequalities Notation}

\section{Inequalities Notation}

When solving equation we may get a single answer, or a number of answers that satisfy the equation. \\

Consider $ 3x - 5 = 1 $, only one value satisfies this equation. \\

But if we consider $ x^2 - 1 = 3 $, more than one value satisfies this equation. \\

Inequalities notation like $ 1 \le x < 3 $, where the symbols $ \le $ and $ \ge $ indicate inclusion of an endpoint, and $ < $ and $ > $ indicate exclusion of an endpoint. \\

A second notation is interval notation, for example, $ x \in [1, 3) $, where a square (or closed) bracket indicates inclusion of an endpoint, and a round (or open) bracket indicates exclusion of an endpoint. \\

The infinity symbol $ \infty $ is always accompanied by round brackets. \\

\begin{exercise}\nonumber
	Write each of the following in interval notation. \\

	(a) $ 2 \le x \le 7 $ \\
	\\
	\\

	(b) $ x < 9 $ \\
	\\
	\\

	(c) $ -3 > x > 0 $ \\
	\\
	\\
\end{exercise}

\begin{exercise}\nonumber
	Write each of the following using inequalities. \\

	(a) $ x \in [3, 6) $ \\
				\\
				\\

				(b) $ x \in (-2, 4) $ \\
				\\
				\\

				(c) $ x \in (-\infty, -1] $ \\
	\\
	\\
\end{exercise}

It is possible to have ranges of values that are disjoint. We use the union symbol $ \cup $ to include all of the values in any of the disjoint ranges. For example, $ [-1, 4) \cup [7, 10) $ meas $ -1 \le x < 4 $ or $ 7 \le x < 10 $. \\

\begin{exercise}\nonumber
	Express each of the following in interval notation. \\

	(a) $ -3 \le x < {1 \over 2} $ or $ 4 < x < 7 $ \\
	\\
	\\

	(b) $ 1 \le x < 5 $ or $ 3 \le x < 7 $ \\
	\\
	\\

	(c) $ x \in [-2, 6) \bigcup (0, 5) $ \\
	\\
	\\
\end{exercise}

Intersection symbol $ \cap $ allows only the values that are common between intervals. For example, $ [-1, 6) \cap (2, 7) $ means $ (2, 6) $. \\

\begin{exercise}\nonumber
	Express each of the following in interval notation. \\

	(a) $ -2 < x \le 6 $ and $ 0 < x < 7 $ \\
	\\
	\\

	(b) $ x \in [0, 5] \bigcap [3, 5] $ \\
	\\
	\\

	(c) $ -4 < x < 0 $ and $ 3 < x < 7 $ \\
	\\
	\\
\end{exercise}

\section{Solving Inequalities}

When solving inequalities, there are a few rules that we must follow: \\

\begin{enumerate}
	\item
	      When it comes to addition, subtraction, multiplication, and division, what you do to one side of the inequality, you must do to the other. \\

	\item
	      If you multiply or divide by a negative quantity, you must flip the inequality. \\

	\item
	      If both sides are positive or both sides are negative, then you can take the reciprocal of both sides, but you must flip the inequality. \\
\end{enumerate}

\begin{exercise}\nonumber
	Find all values of x that satisfy the following. \\

	(a)
	\begin{align}
		-6x + 7 & \ge 8x \\
		\\
		\\
	\end{align} \\

	(b)
	\begin{align}
		-{5 \over 2} < 4 - 2x \le 1 \\
		\\
		\\
		\\
	\end{align} \\

	(c)
	\begin{align}
		5x^3 + 27 & > -13 \\
		\\
		\\
		\\
	\end{align} \\

	(d)
	\begin{align}
		3x^2 + 2 & < -4 \\
		\\
		\\
		\\
	\end{align} \\

	(e)
	\begin{align}
		\sqrt{x-1} & > 4 \\
		\\
		\\
	\end{align} \\

	(f)
	\begin{align}
		log_2(3x) & \le -3 \\
		\\
		\\
		\\
	\end{align}
\end{exercise}

\chapter{The Case Method}

\section{The Case Method}

Consider the following example: $ {x - 3 \over x - 1} < 10 $ \\

You might be tempted to cross multiply, but be careful! The quantity $ x - 1$ is not always positive. If we multiply by $ x - 1$, the inequality needs to flip for some values of $ x $. How might we deal with this? \\

\begin{enumerate}
	\item
	      Separate into 2 cases \\
	      \begin{table}[H]
		      \centering
		      \begin{tabular}{|c|c|} \hline
			      \textbf{Case 1} & \textbf{Case 2}                \\ \hline
			      $ \begin{array}{ccc} x - 1 & > & 0 \\ x & > & 1 \end{array} $
			                      & $ \begin{array}{ccc} x - 1 & < & 0 \\ x & < & 1 \end{array} $ \\ \hline
		      \end{tabular}
	      \end{table}

	\item
	      Solve the original problem under each assumption \\
	      \begin{table}[H]
		      \centering
		      \begin{tabular}{|c|c|} \hline
			      \textbf{Case 1} & \textbf{Case 2}                \\ \hline
			      $ \begin{array}{ccc} {x-3 \over x-1} & < & 10 \\ x - 3 & < & 10(x-1) \\ -9x & < & -7 \\x & > & {7 \over 9} \end{array} $
			                      & $ \begin{array}{ccc} {x-3 \over x-1} & < & 10 \\ x - 3 & > & 10(x-1) \\ -9x & > & -7 \\ x & < & {7 \over 9} \end{array} $ \\ \hline
		      \end{tabular}
	      \end{table}

	\item
	      Find all common points between assumption and solution \\
	      \begin{table}[H]
		      \centering
		      \begin{tabular}{|c|c|} \hline
			      \textbf{Case 1} & \textbf{Case 2}                \\ \hline
			      $ \begin{array}{ccc} x > 1 \ and \ x > {7 \over 9} \\ x \in (1, \infty) \end{array} $
			                      & $ \begin{array}{ccc} x < 1 \ and \ x < {7 \over 9} \\ x \in \left(-\infty, {7 \over 9}\right) \end{array} $ \\ \hline
		      \end{tabular}
	      \end{table}

	\item
	      Consolidate the 2 cases by taking the union \\
	      $$
		      x \in (1, \infty) \cup \left(-\infty, {7 \over 9}\right)
	      $$
\end{enumerate}

\begin{exercise}\nonumber
	Find all values of x such that $ {7x - 2 \over 1 - 2x} \ge 4 $. \\

	\begin{enumerate}
		\item
		      Separate into 2 cases \\
		      \begin{table}[H]
			      \centering
			      \begin{tabular}{|c|c|} \hline
				      \textbf{\hspace{2cm}Case 1\hspace{2cm}} & \textbf{\hspace{2cm}Case 2\hspace{2cm}} \\ \hline
			      \end{tabular}
		      \end{table}
		      \vspace{2cm}

		\item
		      Solve the original problem under each assumption \\
		      \begin{table}[H]
			      \centering
			      \begin{tabular}{|c|c|} \hline
				      \textbf{\hspace{2cm}Case 1\hspace{2cm}} & \textbf{\hspace{2cm}Case 2\hspace{2cm}} \\ \hline
			      \end{tabular}
		      \end{table}
		      \vspace{3cm}

		\item
		      Find all common points between assumption and solution \\
		      \begin{table}[H]
			      \centering
			      \begin{tabular}{|c|c|} \hline
				      \textbf{\hspace{2cm}Case 1\hspace{2cm}} & \textbf{\hspace{2cm}Case 2\hspace{2cm}} \\ \hline
			      \end{tabular}
		      \end{table}
		      \vspace{3cm}

		\item
		      Consolidate the 2 cases by taking the union \\
		      \\
	\end{enumerate}
\end{exercise}

\chapter{The Number Line Method}

\section{The Number Line Method}

Another method for solving inequalities uses the following basic logic: \\

\begin{align}
	\nonumber
	(+)(+) = + &  & {(+) \over (+)} = + \\
	\nonumber
	(-)(-) = + &  & {(-) \over (-)} = + \\
	\nonumber
	(+)(-) = - &  & {(+) \over (-)} = - \\
	\nonumber
	(-)(+) = - &  & {(-) \over (+)} = -
\end{align}

By manipulating expressions into factors that are multiplied and/or divided on one side of the inequality (with a zero appearing on the other side), we can simply consider the combinations of positive and negative factors to draw conclusions. \\

\begin{exercise}\nonumber
	Find all values of x that satisfy $ 3x^2 - 13x > -10 $. \\

	\begin{align}
		\\
		\\
		\\
		\\
	\end{align}

	\begin{figure}[H]
		\centering
		\begin{tikzpicture}
			\draw[->] (-5.5,0)--(5.5,0);
			\foreach \x in {-5,-4,...,5}
			\draw (\x,0.2)--(\x,0);
			\draw[very thick] (-2,0)--(-2,1)--(-6,1) node[above] {$ (-)(-) $};
			\draw[very thick] (2,0)--(2,1)--(6,1) node[above] {$ (+)(+) $};
		\end{tikzpicture}
	\end{figure}

	\vspace{1cm}
\end{exercise}

\begin{exercise}\nonumber
	Find all values of x that satisfy $ x - 2 \ge {4 \over x+1} $. \\

	\begin{align}
		\\
		\\
		\\
		\\
	\end{align}

	\begin{figure}[H]
		\centering
		\begin{tikzpicture}
			\draw[->] (-5.5,0)--(5.5,0);
			\foreach \x in {-5,-4,...,5}
			\draw (\x,0.2)--(\x,0);
			\draw[very thick] (-3,0)--(-3,1)--(-1,1)--(-1,0) node[below] {$ {(-)(+)} \over (-) $};
			\draw[very thick] (2,0)--(2,1)--(4,1)--(4,0) node[below] {$ {(+)(+)} \over (+) $};
		\end{tikzpicture}
	\end{figure}

	\vspace{1cm}
\end{exercise}


\lab{Gibbs Sampling and LDA}{Gibbs Sampling and LDA}
\objective{Understand the basic principles of implementing a Gibbs sampler. Apply this to Latent Dirichlet Allocation.}

\section*{Gibbs Sampling}
Gibbs sampling is an MCMC sampling method in which we construct a Markov chain by which to sample from a desired joint (conditional) distribution
\begin{equation*}
\mathbb{P}(x_{1},\cdots,x_{n} | \mathbf{y}).
\end{equation*}
Often it is difficult to sample from this high-dimensional joint distribution, while it may be easy to sample from the one dimensional
conditional distributions
\begin{equation*}
\mathbb{P}(x_{i} | \mathbf{x}_{-i}, \mathbf{y})
\end{equation*}
where $\mathbf{x}_{-i} = x_{1},\cdots,x_{i-1},x_{i+1},\cdots,x_{n}.$

\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Gibbs Sampler}{}
    \State \textrm{Randomly initialize } $x_1,x_2,\ldots,x_n$.
    \For{$k = 1, 2, 3, \ldots$}
        \For{$i = 1, 2, \ldots,n$}
            \State \textrm{Draw } $x \sim \mathbb{P}(x_{i} | \mathbf{x}_{-i}, \mathbf{y})$
            \State \textrm{Fix } $x_i = x$
        \EndFor
        \State $\mathbf{x}^{(k)}= (x_1,x_2,\ldots,x_n)$
    \EndFor
\EndProcedure
\end{algorithmic}
\caption{Basic Gibbs Sampling Process.}
\label{alg:gibbs}
\end{algorithm}
A Gibbs sampler proceeds according to Algorithm \ref{alg:gibbs}.
Each iteration of the outer for-loop is a \emph{sweep} of the Gibbs sampler, and the value of $\mathbf{x}^{(k)}$ after a sweep is a sample. This creates an irreducible, non-null recurrent, aperiodic Markov chain over the state space consisting of all possible $\mathbf{x}$. The unique invariant distribution for the chain is the desired joint distribution
\begin{equation*}
\mathbb{P}(x_{1},\cdots,x_{n} | \mathbf{y}).
\end{equation*}
Thus, after a burn-in period, our samples $\mathbf{x}^{(k)}$ are effectively samples from the desired distribution.

Consider the data set of $N$ scores from a calculus exam in the file \texttt{examscores.csv}. We believe that the spread of these exam scores can be
modeled with a normal distribution of mean $\mu$ and variance $\sigma^{2}$.
Because we are unsure of the true value of $\mu$ and $\sigma^2$, we take a Bayesian approach and place priors on each parameter to quantify this uncertainty:
\begin{align*}
\mu & \sim N(\mu_{0}, \sigma_{0}^{2})\quad &&\text{(a normal distribution)} \\
\sigma^{2} & \sim IG(\alpha, \beta) &&\text{(an inverse gamma distribution)}
\end{align*}
Letting $\mathbf{y} = (y_1,\ldots,y_N)$ be the set of exam scores, we would like to update our beliefs of $\mu$ and $\sigma^2$ by sampling from the posterior
distribution
\begin{equation*}
\mathbb{P}(\mu, \sigma^{2} | \mathbf{y}, \mu_{0}, \sigma_{0}^{2}, \alpha, \beta).
\end{equation*}
Sampling directly can be difficult. However, we \emph{can} easily sample from the following conditional distributions:
\begin{align*}
\mathbb{P}(\mu | \sigma^{2}, \mathbf{y}, \mu_{0}, \sigma_{0}^{2}, \alpha, \beta) & = \mathbb{P}(\mu | \sigma^{2}, \mathbf{y}, \mu_{0}, \sigma_{0}^{2})\\
\mathbb{P}(\sigma^{2} | \mu, \mathbf{y}, \mu_{0}, \sigma_{0}^{2}, \alpha, \beta) & = \mathbb{P}(\sigma^{2} | \mu, \mathbf{y}, \alpha, \beta)
\end{align*}
The reason for this is that these conditional distributions are \emph{conjugate} to the prior distributions, and hence are part of the same distributional
families as the priors. In particular, we have
\begin{align*}
\mathbb{P}(\mu | \sigma^{2}, \mathbf{y}, \mu_{0}, \sigma_{0}^{2}) &= N(\mu^*, (\sigma^*)^2)\\
\mathbb{P}(\sigma^{2} | \mu, \mathbf{y}, \alpha, \beta) &= IG(\alpha^*, \beta^*),
\end{align*}
where
\begin{align*}
(\sigma^*)^2 &= \left(\frac{1}{\sigma_0^2}+\frac{N}{\sigma^2}\right)^{-1}\\
\mu^* &= (\sigma^*)^2\left(\frac{\mu_0}{\sigma_0^2} + \frac{1}{\sigma^2}\sum_{i=1}^N y_i \right)\\
\alpha^* &= \alpha + \frac{N}{2}\\
\beta^* &= \beta + \frac{1}{2}\sum_{i=1}^N (y_i-\mu)^2
\end{align*}
We have thus set this up as a Gibbs sampling problem, where we just have to alternate between sampling $\mu$ and $\sigma^{2}$.
We can sample from a normal distribution and an inverse gamma distribution as follows:
\begin{lstlisting}
>>> from math import sqrt
>>> from scipy.stats import norm
>>> from scipy.stats import invgamma
>>> mu = 0. # the mean
>>> sigma2 = 9. # the variance
>>> normal_sample = norm.rvs(mu, scale=sqrt(sigma))
>>> alpha = 2.
>>> beta = 15.
>>> invgamma_sample = invgamma.rvs(alpha, scale=beta)
\end{lstlisting}
Note that when sampling from the normal distribution, we need to set the \li{scale} parameter to the standard deviation, \emph{not} the variance.

\begin{problem}
Implement a Gibbs sampler for the exam scores problem using the following function declaration.
\begin{lstlisting}
def gibbs(y, mu0, sigma02, alpha, beta, n_samples):
    """
    Assuming a likelihood and priors
        y_i    ~ N(mu, sigma2),
        mu     ~ N(mu0, sigma02),
        sigma2 ~ IG(alpha, beta),
    sample from the posterior distribution
        P(mu, sigma2 | y, mu0, sigma02, alpha, beta)
    using a gibbs sampler.

    Parameters
    ----------
    y : ndarray of shape (N,)
        The data
    mu0 : float
        The prior mean parameter for mu
    sigma02 : float > 0
        The prior variance parameter for mu
    alpha : float > 0
        The prior alpha parameter for sigma2
    beta : float > 0
        The prior beta parameter for sigma2
    n_samples : int
        The number of samples to draw

    Returns
    -------
    samples : ndarray of shape (n_samples,2)
        1st col = mu samples, 2nd col = sigma2 samples
    """
    pass
\end{lstlisting}
Test it with priors $\mu_{0}=80, \sigma_{0}^{2} = 16, \alpha = 3, \beta = 50$, collecting $1000$ samples. Plot your samples of $\mu$ and your samples of $\sigma^{2}$. How long did it take for each to converge? It should have been very quick.
\end{problem}

We'd like to look at the posterior marginal distributions for $\mu$ and $\sigma^2$.
To plot these from the samples, we will use a kernel density estimator.
If our samples of $\mu$ are called \li{mu_samples}, then we can do this as follows:
\begin{lstlisting}
>>> import numpy as np
>>> from scipy.stats import gaussian_kde
>>> import matplotlib.pyplot as plt
>>> mu_kernel = gaussian_kde(mu_samples)
>>> x_min = min(mu_samples) - 1
>>> x_max = max(mu_samples) + 1
>>> x = np.arange(x_min, x_max, step=0.1)
>>> plt.plot(x,mu_kernel(x))
>>> plt.show()
\end{lstlisting}

\begin{figure}
	\begin{subfigure}[b]{.49\textwidth}
		\includegraphics[width=\textwidth]{mu_posterior.pdf}
		\caption{Posterior distribution of $\mu$.}
	\end{subfigure}
	\begin{subfigure}[b]{.49\textwidth}
		\includegraphics[width=\textwidth]{sigma2_posterior.pdf}
		\caption{Posterior distribution of $\sigma^2$.}
	\end{subfigure}
\caption{Posterior marginal probability densities for $\mu$ and $\sigma^2$.}
\label{fig:post}
\end{figure}

\begin{problem}
Plot the kernel density estimators for the posterior distributions of $\mu$ and $\sigma^{2}$.
You should get plots similar to those in Figure \ref{fig:post}.
\end{problem}

Keep in mind that the above plots are of the posterior distributions of the \emph{parameters}, not of the scores. If we would like to compute the posterior distribution of a new exam score $\tilde{y}$ given our data $\mathbf{y}$ and prior parameters, we compute what is known as the \emph{posterior predictive distribution}:
\begin{equation*}
\mathbb{P}(\tilde{y} | \mathbf{y}, \lambda) = \int_{\Theta} \mathbb{P}(\tilde{y} | \Theta)\mathbb{P}(\Theta | \mathbf{y}, \lambda) d\Theta
\end{equation*}
where $\Theta$ denotes our parameters (in our case $\mu$ and $\sigma^{2}$) and $\lambda$ denotes our prior parameters (in our case $\mu_{0}, \sigma_{0}^{2}, \alpha,$ and $\beta$).

Rather than actually computing this integral for each possible $\tilde{y}$, we can do this by sampling scores from our parameter samples. In other words, sample
\begin{equation*}
\tilde{y}_{(t)} \sim N(\mu_{(t)}, \sigma_{(t)}^{2})
\end{equation*}
for each sample pair $\mu_{(t)}, \sigma_{(t)}^{2}$. Now we have essentially drawn samples from our posterior predictive distribution, and we can use a kernel density estimator to plot this distribution from the samples.

\begin{figure}
\includegraphics[width=\textwidth]{predictiveposterior.pdf}
\caption{Predictive posterior distribution of exam scores.}
\label{fig:predictive}
\end{figure}

\begin{problem}
Use your samples to draw samples from the posterior predictive distribution. Plot the kernel density estimator of your sampled scores.
It should resemble the plot in Figure \ref{fig:predictive}.
\end{problem}

\section*{Latent Dirichlet Allocation}
Gibbs sampling can be applied to an interesting problem in language processing: that of determining which topics are prevalent in a document.
Latent Dirichlet Allocation (LDA) is a generative model for a collection of text documents.
It supposes that there is some fixed vocabulary of terms (of length $V$) and $K$ different topics, each represented as a probability distribution $\phi_{k}$ over the vocabulary, each with a Dirichlet prior $\beta$.
With the vocabulary and topics chosen, the LDA model assumes that a set of $D$ documents is generated as follows.
The $m$-th document consists of $N_m$ words, and
a probability distribution $\theta_{m}$ over the topics is drawn from a Dirichlet distribution with parameter $\alpha$.
To generate each new word in the $m$-th document, we first draw a topic assignment $z_{m,n}$ from the categorical distribution $\theta_{m}$, and then we draw a word $w_{m,n}$ from the categorical distribution $\phi_{z_{m,n}}$. Throughout this implementation, we assume $\alpha$ and $\beta$ are scalars. In summary, we have
\begin{enumerate}
	\item Draw $\phi_{k} \sim \text{Dir}(\beta)$ for $1 \leq k \leq K$.
	\item For $1 \leq m \leq D$:
	\begin{enumerate}
        \item Draw $\theta_{m} \sim \text{Dir}(\alpha)$.
        \item Draw $z_{m,n} \sim \text{Cat}(\theta_{m})$ for $1 \leq n \leq N_{m}$.
	    \item Draw $w_{m,n} \sim \text{Cat}(\phi_{z_{m,n}})$ for $1 \leq n \leq N_{m}$.
	\end{enumerate}
\end{enumerate}
This is typically depicted with graphical plate notation as in Figure \ref{fig:ldaplates}.
\begin{figure}[h]
\centering
\begin{tikzpicture}[>=stealth', dot/.style=
	{circle,fill=black,minimum size=3pt,inner sep=0pt, outer sep=-1pt} ]

\node[draw,minimum height=3.2cm, minimum width=2.1cm](r1)[]{};
\node[draw,minimum height=5cm, minimum width=2.4cm, node distance=
	.4cm](r2)[above of=r1]{};

\node[node distance=1.3cm](Nm)[below of=r1]{$1 \le n \le N_m$};
\node[node distance=2.28cm](M)[below of=r2]{$1 \le m \le M$};

\node[node distance = 2.7cm](dummy3)[left of=r1]{};
\node[draw,minimum height=2cm, minimum width=2.1cm, node distance=
	.55cm](r3)[ below of =dummy3]{};
\node[node distance=.7cm](K)[below of=r3]{$1 \le k \le K$};

\node[node distance=.6cm](dummy)[above right of =Nm]{};
\node[circle, draw,  inner sep=1pt, fill=black!25!,node distance=.4cm]
	(w)[above of=dummy]{$w_{m,n}$};
\node[circle, draw,  inner sep=1pt, node distance=1.3cm](z)[above
	of=w]{$z_{m,n}$};
\node[circle, draw,  inner sep=1pt, node distance=1.3cm]
	(theta)[above of=z]{$\vec{\theta}_{m}$};

\node[node distance=1.2cm, inner sep=0pt](alpha)[above of=
	theta]{$\vec{\alpha}$};

\node[node distance=.6cm](dummy2)[above right of=K]{};
\node[node distance=.35cm, circle, inner sep=1pt, draw](phi)[above
	of=dummy2]{$\vec{\phi_k}$};
\node[node distance=1.4cm, inner sep=0pt](beta)[above of = phi]{$\vec{\beta}$};

\foreach \x/\y in {alpha/theta, theta/z, z/w, beta/phi, phi/w} \draw[->](\x)--(\y);

\end{tikzpicture}
\caption{Graphical plate notation for LDA text generation.}
\label{fig:ldaplates}
\end{figure}

In the plate model, only the variables $w_{m,n}$ are shaded, signifying that these are the only observations visible to us; the rest are latent variables. Our goal is to estimate each $\phi_{k}$ and each $\theta_{m}$. This will allow us to understand what each topic is, as well as understand how each document is distributed over the $K$ topics. We can estimate these well if we know $z_{m,n}$ for each $m, n$, collectively referred to as $\mathbf{z}$. Thus, we need to sample
$\mathbf{Z}$ from the posterior distribution $\mathbb{P}(\mathbf{z} | \mathbf{w}, \alpha, \beta),$ where $\mathbf{w}$ is the collection words in the text corpus. Unsurprisingly, it is intractable to sample directly from the joint posterior distribution. However, letting $\mathbf{z}_{\neg (m,n)} = \mathbf{z}\setminus \{z_{m,n}\}$, the conditional posterior distributions
\[\mathbb{P}(z_{m,n} = k | \mathbf{z}_{\neg (m,n)}, \mathbf{w}, \alpha, \beta)\]
have nice, closed form solutions, making them easy to sample from.

These conditional distributions have the following form:
\begin{equation*}
\mathbb{P}(z_{m,n} = k | \mathbf{z}_{\neg (m,n)}, \mathbf{w}, \alpha, \beta) \propto \frac{(n_{(k,m,\cdot)}^{\neg (m,n)} + \alpha)(n_{(k, \cdot, w_{m,n})}^{\neg (m,n)} + \beta)}{n_{(k,\cdot,\cdot)}^{\neg (m,n)} + V \beta}
\end{equation*}
where
\begin{align*}
n_{(k,m,\cdot)} & = \mbox{ the number of words in document $m$ assigned to topic $k$} \\
n_{(k,\cdot,v)} & = \mbox{ the number of times term $v = w_{m,n}$ is assigned to topic $k$} \\
n_{(k,\cdot,\cdot)} & = \mbox{ the number of times topic $k$ is assigned in the corpus} \\
n_{(k,m,\cdot)}^{\neg (m,n)} & = n_{(k,m,\cdot)} - \indicator{z_{m,n} = k} \\
n_{(k,\cdot,v)}^{\neg (m,n)} & = n_{(k,\cdot,v)} - \indicator{z_{m,n} = k} \\
n_{(k,\cdot,\cdot)}^{\neg (m,n)} & = n_{(k,\cdot,\cdot)} - \indicator{z_{m,n} = k}
\end{align*}

Thus, if we simply keep track of these count matrices, then we can easily create a Gibbs sampler over the topic assignments. This is actually a particular class of samplers known as \emph{collapsed Gibbs samplers}, because we have collapsed the sampler by integrating out $\theta$ and $\phi$.

We have provided for you the structure of a Python object \li{LDACGS} with several methods. The object is already defined to have attributes \li{n\_topics}, \li{documents}, \li{vocab}, \li{alpha}, and \li{beta}, where \li{vocab} is a list of strings (terms), and documents is a list of dictionaries (a dictionary for each document). Each entry in dictionary $m$ is of the form $n : w$, where $w$ is the index in \li{vocab} of the $n^{th}$ word in document $m$.

Throughout this lab we will guide you through writing several more methods in order to implement the Gibbs sampler. The first step is to initialize our assignments, and create the count matrices $n_{(k,m,\cdot)}, n_{(k,\cdot,v)}$ and vector $n_{(k,\cdot,\cdot)}$.

\begin{problem}
Complete the method \li{initialize}. By randomly assigning initial topics, fill in the count matrices and topic assignment dictionary. In this method, you will initialize the count matrices (among other things). Note that the notation
provided in the code is slightly different than that used above. Be sure to understand how the formulae above
connect with the code.
\end{problem}

The next method we need to write fully outlines a sweep of the Gibbs sampler.

\begin{problem}
Complete the method \li{\_sweep}, which needs to iterate through each word of each document. It should call on the method \li{\_conditional} to get the conditional distribution at each iteration.
\end{problem}

\begin{comment}
Take out this problem to make the lab easier.
We need to write the method to create the appropriate conditional distribution.

\begin{problem}
Complete the method \li{\_conditional}. It accepts arguments $m,w$ where $m$ is the document and $w$ is an index of \li{vocab}. Don't forget to normalize to ensure you are actually returning a distribution!
\end{problem}
\end{comment}
We are now prepared to write the full Gibbs sampler.

\begin{problem}
Complete the method \li{sample}. The argument \emph{filename} is the name and location of a .txt file, where each line is considered a document. The corpus is built by method \li{buildCorpus}, and stopwords are removed (if argument \emph{stopwords} is provided). Burn in the Gibbs sampler, computing and saving the log-likelihood with the method \li{\_loglikelihood}. After the burn in, iterate further, accumulating your count matrices, by adding \li{nzw} and \li{nmz} to \li{total\_nzw} and \li{total\_nmz} respectively, where you only add every \emph{sample\_rate}$^{th}$ iteration. Also save each log-likelihood.
\end{problem}

You should now have a working Gibbs sampler to perform LDA inference on a corpus. Let's test it out on Ronald Reagan's State of the Union addresses.

\begin{problem}
Create an \li{LDACGS} object with $20$ topics, letting \li{alpha} and \li{beta} be the default values. Load in the stop word list provided. Run the Gibbs sampler, with a burn in of $100$ iterations, accumulating $10$ samples, only keeping the results of every $10$ sweep. Plot the log-likelihoods. How long did it take to truly burn in?
\end{problem}

We can estimate the values of each $\phi_{k}$ and each $\theta_{m}$ as follows:

\begin{align*}
\widehat{\theta}_{m,k} & = \frac{n_{(k,m,\cdot)} + \alpha}{K \cdot \alpha + \sum_{k=1}^{K} n_{(k,m,\cdot)}} \\
\widehat{\phi}_{k,v} & = \frac{n_{(k,\cdot,v)} + \beta}{V \cdot \beta + \sum_{v=1}^{V} n_{(k,\cdot,v)}}
\end{align*}

We have provided methods \li{phi} and \li{theta} that do this for you. We often examine the topic-term distributions $\phi_{k}$ by looking at the $n$ terms with highest probability, where $n$ is small (say $10$ or $20$).  We have provided a method \li{topterms} which does this for you.

\begin{problem}
Using the methods described above, examine the topics for Reagan's addresses. As best as you can, come up with labels for each topic.
\end{problem}

We can use $\widehat{\theta}$ to find the paragraphs in Reagan's addresses that focus the most on each topic. The documents with highest values of $\widehat{\theta}_{k}$ are those most heavily focused on topic $k$.

\begin{problem}
In your above topic analysis, you should have found a topic about the Cold War and one about education. Find the five paragraphs in Reagan's addresses that most closely focus on each of these topics, according to the above method.
\end{problem}


\section{Dual Space and Dual Maps}
\begin{definition}
    Let $V$ be a vector space over $F$, we define the dual space $V^\ast$ of $V$ to be the set of linear maps from $V$ to $F$, i.e. $V^\ast=L(V,F)$.\\
    We call linear maps $V\to F$ as linear forms.
\end{definition}
\begin{example}
    1. The map $\operatorname{tr}:M_{n,n}(F)\to F$ via $A=(a_{ij})\mapsto\sum_ia_{ii}$ is a linear form, so $\operatorname{tr}\in (M_{n,n}(F))^\ast$.\\
    2. For a function $f:\in C^{\infty}([0,1],\mathbb R)$, we can define the map $T_f:C^{\infty}([0,1],\mathbb R)$ via
    $$T_f(\phi)=\int_0^1f(x)\phi(x)\,\mathrm dx$$
    An interesting thing is, if we are given all information about $T_f$, can we recover $f$?
    The answer is yes and is left as an exercise.
    This idea comes from quantum mechanics, where you only get the information about $T_f$ by physical measurements but you want information about $f$.
\end{example}
There is a natural way of finding a basis for the dual space.
\begin{lemma}
    Let $V$ be a vector space over $F$ with a finite basis $B=\{e_1,\ldots,e_n\}$, then there exists a basis for $V^\ast$ given by
    $B^\ast=\{\epsilon_1,\ldots,\epsilon_n\}$ where $\epsilon_j(\sum_ia_ie_i)=a_j$ for $1\le j\le n$.
\end{lemma}
\begin{definition}
    The basis $B^\ast$ in the preceding lemma is called the dual basis.
\end{definition}
\begin{remark}
    If we introduce the Kronecker delta
    $$\delta_{ij}=\begin{cases}
        1\text{, if $i=j$}\\
        0\text{, otherwise}
    \end{cases}$$
    then we can define $\epsilon_j$ by extending $\epsilon_j(e_i)=\delta_{ij}$.
\end{remark}
\begin{proof}
    If there is some $\lambda_j\in F$ such that $\sum_j\lambda_j\epsilon_j=0$, then in particular the evaluation of the left hand side at $e_i$ is zero for each $i$.
    But $\epsilon_j(e_i)=0$, so $\lambda_i=0$ for each $i$.\\
    To see this is spanning, one just observe that $\alpha=\sum_j\alpha(e_j)\epsilon_j$ for any linear form $\alpha$.
\end{proof}
\begin{corollary}
    If $V$ is finite dimensional, then $\dim V^\ast=\dim V$.
\end{corollary}
\begin{remark}
    It is convenient to think of $V^\ast$ as the space of row vectors of length $n$ over $F$.
    Indeed, if we have a basis $\{e_i\}$ of $V$ and the corresponding dual basis $\{\epsilon_i\}$ for $V^\ast$, then via calculation we can obtain $\alpha(x)=\sum_i\alpha_ix_i$ where $\alpha=\sum_i\alpha_i\epsilon_i$ and $x=\sum_ix_ie_i$.
\end{remark}
\begin{definition}
    If $U\subset V$ is a subset of the vector space $V$, then the annihilator of $U$ is
    $$U^\circ=\{\alpha\in V^\ast:\forall u\in U,\alpha(u)=0\}$$
\end{definition}
\begin{lemma}
    $U^\circ\le V^\ast$ and if $U\le V$ and $\dim V$ is finite, then $\dim V=\dim U+\dim U^\circ$.
\end{lemma}
\begin{proof}
    Quite obvious that $U^\circ\le V^\ast$.
    To see the identity, write $n=\dim V$ and extend a basis $\{e_1,\ldots,e_k\}$ of $U$ to a basis $\{e_1,\ldots,e_n\}$ of $V$.
    Let $\{\epsilon_1,\ldots,\epsilon_n\}$ be the dual basis.
    Then it becomes obvious that $U^\circ=\langle\{\epsilon_{k+1},\ldots,\epsilon_n\}\rangle$, which gives the identity.
\end{proof}
\begin{lemma}
    Let $V,W$ be vector spaces over $F$ and $\alpha\in L(V,W)$.
    Then the map $\alpha^\ast:W^\ast\to V^\ast$ sending $\epsilon$ to $\epsilon\circ\alpha$ is linear.
\end{lemma}
\begin{proof}
    Obvious.
\end{proof}
\begin{definition}
    This map $\alpha^\ast$ is called the dual map of $\alpha$.
\end{definition}
\begin{proposition}
    Let $V,W$ be finite dimensional vector spaces over $F$ with bases $B,C$.
    Let $B^\ast,C^\ast$ be the dual bases of $V^\ast,W^\ast$, then $[\alpha^\ast]_{C^\ast,B^\ast}=[\alpha]_{B,C}^\top$.
\end{proposition}
\begin{proof}
    Just write it down.
\end{proof}
\begin{lemma}
    Let $E,F$ be bases of $V$ and $P=[\operatorname{id}]_{F,E}$ be the change-of-basis matrix from $F$ to $E$.
    Let $E^\ast,F^\ast$ be the corresponding dual bases, then the change-of-basis matrix from $F^\ast$ to $E^\ast$ is $(P^{-1})^\top$.
\end{lemma}
\begin{proof}
    We have
    $$[\operatorname{id}]_{F^\ast,E^\ast}=[\operatorname{id}]^\top_{E,F}=(P^{-1})^\top$$
    as desired.
\end{proof}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 7}

Suppose $n = 16$.
Draw the recursion tree for Equation \ref{eq16}, including every node in the tree.
Since $n$ is finite, there are only a finite number of nodes in the tree.
You need to show every one of them.

\begin{equation}
T(n) = T(\frac{n}{4})+T(\frac{n}{2})+n^2
\label{eq16}
\end{equation}

\subsection*{Solution}

Recursion tree for Equation \ref{eq16} is depicted in Figure \ref{fig1}.

\begin{figure}[H]\centering
\begin{tikzpicture}[level/.style={sibling distance=60mm/#1}]
\node [circle,draw] (z){$T(16)$}
  child {
    node [circle,draw] (a) {$T(8)$}
    child {
      node [circle,draw] (c) {$T(4)$}
      child {
        node [circle,draw] (g) {$T(2)$}
        child {
          node [circle,draw] (k) {$T(1)$}
        }
      }
      child {
        node [circle,draw] (h) {$T(1)$}
      }
    }
    child {
      node [circle,draw] (d) {$T(2)$}
      child {
        node [circle,draw] (i) {$T(1)$}
      }
    }
  }
  child {
    node [circle,draw] (b) {$T(4)$}
    child {
      node [circle,draw] (e) {$T(2)$}
      child {
        node [circle,draw] (j) {$T(1)$}
      }
    }
    child {
      node [circle,draw] (f) {$T(1)$}
    }
  };
\end{tikzpicture}
\caption{Recursion tree for $T(n)=T(\frac{n}{2})+T(\frac{n}{4})+n^2$ when $n = 16$}\label{fig1}
\end{figure}


\section{The Fundamental Theorem of Line Integrals}\label{sec:FundamentalTheoremLineIntegrals}

One way to write the Fundamental Theorem of Calculus
(\ref{thm:fundamental_theorem_I}) is:
$$\int_a^b f'(x)\,dx = f(b)-f(a).$$
That is, to compute the integral of a derivative $f'$
we need only compute the values of $f$ at the endpoints. Something
similar is true for line integrals of a certain form.

\begin{theorem}{Fundamental Theorem of Line Integrals}{FTLineIntegrals}
Suppose a curve $C$ is
given by the vector function ${\bf r}(t)$, with ${\bf a}={\bf r}(a)$
and ${\bf b}={\bf r}(b)$. Then
$$\int_C \nabla f\cdot d{\bf r} = f({\bf b})-f({\bf a}),$$
provided that $\bf r$ is sufficiently nice.
\end{theorem}
\begin{proof}
We write ${\bf r}=\langle x(t),y(t),z(t)\rangle$, so
that ${\bf r}'=\langle x'(t),y'(t),z'(t)\rangle$. Also,
we know that $\nabla f=\langle f_x,f_y,f_z\rangle$. Then
$$\int_C \nabla f\cdot d{\bf r} =
\int_a^b \langle f_x,f_y,f_z\rangle\cdot\langle
x'(t),y'(t),z'(t)\rangle\,dt=
\int_a^b f_x x'+f_y y'+f_z z' \,dt.$$
By the chain rule (see Section~\ref{sec:multivariable chain rules})
$f_x x'+f_y y'+f_z z'=df/dt$, where $f$ in this context means
$f(x(t),y(t),z(t))$, a function of $t$. In other words, all we have is
$$\int_a^b f'(t)\,dt=f(b)-f(a).$$
In this context,
$f(a)=f(x(a),y(a),z(a))$.
Since ${\bf a}={\bf r}(a)=\langle x(a),y(a),z(a)\rangle$, we can
write $f(a)=f({\bf a})$---this is a bit of a cheat, since we are
simultaneously using $f$ to mean $f(t)$ and $f(x,y,z)$, and since
$f(x(a),y(a),z(a))$ is not technically the same as
$f(\langle x(a),y(a),z(a)\rangle)$,
but the
concepts are clear and the different uses are compatible. Doing the
same for $b$, we get
$$\int_C \nabla f\cdot d{\bf r} = \int_a^b f'(t)\,dt=f(b)-f(a)=f({\bf
  b})-f({\bf a}).$$
\end{proof}

This theorem, like the Fundamental Theorem of Calculus, says roughly
that if we integrate a ``derivative-like function'' ($f'$ or $\nabla
f$) the result depends only on the values of the original function ($f$)
at the endpoints.

If a vector field $\bf F$ is the gradient of a function, ${\bf
  F}=\nabla f$, we say that $\bf F$ is a \dfont{conservative vector field}.
If $\bf F$ is a conservative force field, then the integral for work,
$\int_C {\bf F}\cdot d{\bf r}$, is in the form required by the
Fundamental Theorem of Line Integrals. This means that in a
conservative force field, the amount of work required to move an
object from point $\bf a$ to point $\bf b$ depends only on those
points, not on the path taken between them.

\begin{example}{Work Done}{}
An object moves in the force field
$${\bf F}=
\left
\langle {-x\over (x^2+y^2+z^2)^{3/2}},{-y\over (x^2+y^2+z^2)^{3/2}},{-z\over
  (x^2+y^2+z^2)^{3/2}}\right\rangle,$$
along the curve ${\bf r}=\langle 1+t,t^3,t\cos(\pi t)\rangle$ as $t$
ranges from 0 to 1. Find the work done by the force on the object.
\end{example}
\begin{solution}
The straightforward way to do this involves substituting the
components of ${\bf r}$ into $\bf F$, forming the dot product ${\bf
  F}\cdot{\bf r}'$, and then trying to compute the integral, but this
integral is extraordinarily messy, perhaps impossible to compute. But
since ${\bf F}=\nabla (1/\sqrt{x^2+y^2+z^2})$ we need only substitute:
$$\int_C {\bf F}\cdot d{\bf r}=
\left.{1\over \sqrt{x^2+y^2+z^2}}\right|_{(1,0,0)}^{(2,1,-1)}=
{1\over\sqrt6}-1.
$$
\end{solution}

Another immediate consequence of the Fundamental Theorem involves
\dfont{closed paths}. A path $C$ is closed if it
forms a loop, so that traveling over the $C$ curve brings you back to
the starting point. If $C$ is a closed path, we can integrate around
it starting at any point $\bf a$; since the starting and ending points are the
same,
$$\int_C \nabla f\cdot d{\bf r}=f({\bf a})-f({\bf a})=0.$$
For example, in a gravitational field (an inverse square law field)
the amount of work required to move an object around a closed path is
zero. Of course, it's only the \emph{net} amount of work that is
zero. It may well take a great deal of work to get from point $\bf a$
to point $\bf b$, but then the return trip will ``produce'' work. For
example, it takes work to pump water from a lower to a higher elevation,
but if you then let gravity pull the water back down, you can recover
work by running a water wheel or generator. (In the real world you
won't recover all the work because of various losses along the way.)

To make use of the Fundamental Theorem of Line Integrals, we need to
be able to spot conservative vector fields $\bf F$ and to compute
$f$ so that ${\bf F}=\nabla f$. Suppose that ${\bf F}=\langle
P,Q\rangle = \nabla f$. Then $P=f_x$ and $Q=f_y$, and provided that
$f$ is sufficiently nice, we know from Clairaut's Theorem
(\ref{thm:clairaut}) that $P_y=f_{xy}=f_{yx}=Q_x$. If we compute
$P_y$ and $Q_x$ and find that they are not equal, then $\bf F$ is not
conservative. If $P_y=Q_x$, then, again provided that $\bf F$ is
sufficiently nice, we can be assured that $\bf F$ is conservative.
Ultimately, what's important is that we be able to find $f$; as this
amounts to finding anti-derivatives, we may not always succeed.

\begin{example}{}{}
Find an $f$ so that $\langle 3+2xy,x^2-3y^2\rangle = \nabla f$.
\end{example}
\begin{solution}
First, note that
$${\partial\over\partial y}(3+2xy)=2x\qquad\hbox{and}\qquad
{\partial\over\partial x}(x^2-3y^2)=2x,$$
so the desired $f$ does exist. This means that $f_x=3+2xy$, so that
$f=3x+x^2y+g(y)$; the first two terms are needed to get $3+2xy$, and
the $g(y)$ could be any function of $y$, as it would disappear upon
taking a derivative with respect to $x$. Likewise, since
$f_y=x^2-3y^2$, $f=x^2y-y^3+h(x)$. The question now becomes, is it
possible to find $g(y)$ and $h(x)$ so that
$$3x+x^2y+g(y)=x^2y-y^3+h(x),$$
and of course the answer is yes: $g(y)=-y^3$, $h(x)=3x$. Thus,
$f=3x+x^2y-y^3$.
\end{solution}

\label{page:test for conservative vector field}
We can test a vector field ${\bf F}=\langle P,Q,R\rangle$ in a similar
way. Suppose that $\langle P,Q,R\rangle=\langle f_x,f_y,f_z\rangle$. If we temporarily hold
$z$ constant, then $f(x,y,z)$ is a function of $x$ and $y$, and
by Clairaut's Theorem $P_y=f_{xy}=f_{yx}=Q_x$.
Likewise, holding $y$ constant implies $P_z=f_{xz}=f_{zx}=R_x$, and
with $x$ constant we get $Q_z=f_{yz}=f_{zy}=R_y$. Conversely, if we
find that $P_y=Q_x$, $P_z=R_x$, and $Q_z=R_y$ then $\bf F$ is
conservative.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:FundamentalTheoremLineIntegrals}}

\begin{enumialphparenastyle}

\begin{ex}
Find an $f$ so that $\nabla f=\langle 2x+y^2,2y+x^2\rangle$, or
explain why there is no such $f$.
\begin{sol}
	no $f$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle x^3,-y^4\rangle$,
or explain why there is no such $f$.
\begin{sol}
	$x^4/4-y^5/5$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle xe^y,ye^x \rangle$,
or explain why there is no such $f$.
\begin{sol}
	no $f$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle y\cos x,y\sin x \rangle$,
or explain why there is no such $f$.
\begin{sol}
	no $f$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle y\cos x,\sin x\rangle$,
or explain why there is no such $f$.
\begin{sol}
	$y\sin x$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle x^2y^3,xy^4\rangle$,
or explain why there is no such $f$.
\begin{sol}
	no $f$
\end{sol}
\end{ex}

\begin{ex}
Find an $f$ so that $\nabla f=\langle yz,xz,xy\rangle$,
or explain why there is no such $f$.
\begin{sol}
	$xyz$
\end{sol}
\end{ex}

\begin{ex}
Evaluate $\ds\int_C (10x^4 - 2xy^3)\,dx - 3x^2y^2\,dy$ where $C$ is
the part of the curve $x^5-5x^2y^2-7x^2=0$ from $(0,0)$ to
$(3,2)$.
\begin{sol}
	414
\end{sol}
\end{ex}

\begin{ex}
Let
${\bf F}=
\langle yz,xz,xy\rangle$.
Find the work done by this force field on an object that moves from
$(1,0,2)$ to $(1,2,3)$.
\begin{sol}
	$6$
\end{sol}
\end{ex}

\begin{ex}
Let
${\bf F}=
\langle e^y,xe^y+\sin z,y\cos z\rangle$.
Find the work done by this force field on an object that moves from
$(0,0,0)$ to $(1,-1,3)$.
\begin{sol}
	$1/e-\sin3$
\end{sol}
\end{ex}

\begin{ex}
Let
$${\bf F}=
\left
\langle {-x\over (x^2+y^2+z^2)^{3/2}},{-y\over (x^2+y^2+z^2)^{3/2}},{-z\over
  (x^2+y^2+z^2)^{3/2}}\right\rangle.$$
Find the work done by this force field on an object that moves from
$(1,1,1)$ to $(4,5,6)$.
\begin{sol}
	$1/\sqrt{77}-1/\sqrt3$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\section{Reduction}

\subsection{Notation}
\begin{itemize}
    \item $\Gamma\vdash \tt t\rhd u$ : $\tt t$ reduces to $\tt u$ in $\Gamma$ with
    one of the $\beta,\iota,\delta,\zeta$ reductions.
    \item $\Gamma\vdash \tt t\mathop{\rhd}\limits^* u$ : $\Gamma\vdash\tt t\rhd\cdots\rhd u$.
    \item $\tt u\equiv v$ : $\tt u$ and $\tt v$ are identical.
\end{itemize}

\subsection{Conversion Rules}
\begin{equation*}
    \Gamma\vdash\tt((\lambda x:T.\ t)\ u)\ \rhd_\beta\ t\{x/u\}
    \tag{\sc $\beta$-Conv}
\end{equation*}
\begin{equation*}
    \Gamma\vdash\tt x\ \rhd_\delta\ t\qquad \textit{if }(x:=t:T)\in\Gamma
    \tag{\sc $\delta$-Conv}
\end{equation*}
\begin{equation*}
\Gamma\vdash \tt let\ \ x:=u\ \ in\ \ t\ \rhd_\zeta\ t\{x/u\}
    \tag{\sc $\zeta$-Conv}
\end{equation*}
\begin{equation*}
    \tt case((c_{p_i}\ q_1\ \cdots\ q_r\ a_1\ \cdots\ a_m),P,f_1\ |\ \cdots\ |\ f_n)\ \rhd_\iota\ (f_i\ a_1\ \cdots\ a_m)
    \tag{\sc $\iota$-Conv}
\end{equation*}
\begin{equation*}
\frac{\Gamma\vdash \tt t:\forall x:T,U\qquad x\textit{ fresh in }t}
    {\tt t\ \rhd_\eta\ \lambda\tt x:T.\ (t\ x)}
    \tag{\sc $\eta$-Exp}
\end{equation*}

Here's an example for $\iota$-reduction:
\begin{center}
\begin{minipage}{0.6\textwidth}
\begin{minted}{coq}
match (S m) as s0 in nat return nat with
| S n => n
| O => O
end

(* $\iota$-reduction *)
(fun (n:nat) => n) m.
\end{minted}
\end{minipage}
\end{center}

\begin{Def}[Convertibility]
$\tt t_1$ and $\tt t_2$ are convertible iff there exists $\tt u_1$ and $\tt u_2$ such that
$\Gamma\vdash \tt t_1\mathop{\rhd}\limits^* u_1$ and $\Gamma\vdash \tt t_2\mathop{\rhd}\limits^* u_2$
and either $\tt u_1\equiv u_2$ or they are convertible up to $\eta$-expansion.
\end{Def}


\documentclass{article}
\usepackage{amsmath}
\usepackage[margin=1.0in]{geometry}
\usepackage{xcolor}

\begin{document}

\noindent
Does $\displaystyle \sum_{n=1}^\infty \frac{n^2}{2^n}$
diverge, converge absolutely, or converge conditionally?

\subsection*{Solution}

\begin{align*}
L&=\lim_{n \to \infty} \left|\frac{a_{n+1}}{a_n}\right|\\
&= \lim_{n \to \infty} \left| \frac{(n+1)^2}{2^{n+1}} \cdot \frac{2^n}{n^2}\right|\\
&= \lim_{n \to \infty} \left| \frac{n^2+2n+1}{2^n \cdot 2} \cdot \frac{2^n}{n^2}\right|\\
&= \lim_{n \to \infty} \left| \frac{n^2+2n+1}{2n^2}\right|\\
&= \lim_{n \to \infty} \frac{n^2+2n+1}{2n^2}\\
&= \lim_{n \to \infty} \frac{2n+2}{4n}\quad\text{by L'hopital}\\
&= \lim_{n \to \infty} \frac{2}{4}\quad\text{by another L'hopital}\\
&= \frac24=\frac12
\end{align*}

Since $L< 1$, the series $\displaystyle \sum_{n=1}^\infty \frac{n^2}{2^n}$ converges absolutely by the Ratio Test.

\subsection*{Solution 2 (outline)}

The Root Test will work, but it is a lot of work compared to the solution presented.

\end{document}%%%%%%%%%%%%%%%%%

\begin{align*}
L&=\lim_{n \to \infty} \sqrt[n]{|a_n|}\\
&= \lim_{n \to \infty} \sqrt[n]{\left| \right|}\\
\end{align*}

\begin{align*}
L&=\lim_{n \to \infty} \left|\frac{a_{n+1}}{a_n}\right|\\
&= \lim_{n \to \infty} \left| \right|\\
\end{align*}

\begin{align*}
\lim_{n \to \infty} a_n
&= \lim_{n \to \infty} \\
\end{align*}

Since $\sum |a_n| = \sum a_n$, the series $\displaystyle \sum_{n=1}^\infty AAAAAAAAAAAAAA$ converges absolutely.

Since $|r| < 1$, the series ...  converges by the Geometric Series Test.

Since $|r| \geq 1$, the series ...  diverges by the Geometric Series Test.

The function $f(x)=\frac{}{}$ is continuous, positive, and decreasing on $[1,\infty)$.

\subsection*{Solution}


\chapter{Math}

\begin{eqnarray*}
	f(x) = \sum_{i=0}^{n} \frac{a_i}{1+x} \\
	\textstyle f(x) = \textstyle \sum_{i=0}^{n} \frac{a_i}{1+x} \\
	\scriptstyle f(x) = \scriptstyle \sum_{i=0}^{n} \frac{a_i}{1+x} \\
	\scriptscriptstyle f(x) = \scriptscriptstyle \sum_{i=0}^{n} \frac{a_i}{1+x}
\end{eqnarray*}

\section{Inline math}
In this line is a beautiful formular $e^{i\pi} + 1 = 0$ \par
This is also inline but special \[e^{i\pi} + 1 = 0\] Do you see it?

\section{Formulars}
For not only mention a formular, you should use that format:

\begin{equation}
E=mc^2
\end{equation}

\section{Multiline}

For long equations
\begin{multline}
p(x) = 3x^6 + 14x^5y \\+ 590x^4y^2 + 19x^3y^3
- 12x^2y^4 - 12xy^5 + 2y^6 \\- a^3b^3 + a^3 - b^6
\end{multline}

Or multipart
\begin{equation}
\begin{split}
A & = \frac{\pi r^2}{2} \\
& = \frac{1}{2} \pi r^2
\end{split}
\end{equation}

Or on another way:

\begin{align}
x&=y           &  w &=z              &  a&=b+c\\
2x&=-y         &  3w&=\frac{1}{2}z   &  a&=b\\
-4 + 5x&=2+y   &  w+2&=-1+w          &  ab&=cb
\end{align}

Without alignment:

\begin{gather}
2x - 5y =  8 \\
3x^2 + 9y =  3a + c
\end{gather}

\section{spacing}
\begin{align*}
f(x) =& x^2\! +3x\! +2 \\
f(x) =& x^2+3x+2 \\
f(x) =& x^2\, +3x\, +2 \\
f(x) =& x^2\: +3x\: +2 \\
f(x) =& x^2\; +3x\; +2 \\
f(x) =& x^2\ +3x\ +2 \\
f(x) =& x^2\quad +3x\quad +2 \\
f(x) =& x^2\qquad +3x\qquad +2
\end{align*}

\section{Brackets}

\ldots need to be equalized even with invisible brackets

\begin{align}
y  = 1 + & \left(  \frac{1}{x} + \frac{1}{x^2} + \frac{1}{x^3} + \ldots \right. \\
& \quad \left. + \frac{1}{x^{n-1}} + \frac{1}{x^n} \right)
\end{align}

\section{Units}
\SI{24,38176541e4}{m.Pa} \\
\num{24,38176541e4} \\
\si{\kilo\gram\metre\per\square\second} \\
\si{\gram\per\cubic\centi\metre}        \\
\si{\square\volt\cubic\lumen\per\farad} \\
\si{\metre\squared\per\gray\cubic\lux}  \\
\si{\henry\second} \\
\SI[mode=text]{1.23}{J.mol^{-1}.K^{-1}}          \\
\SI{.23e7}{\candela}                              \\
\SI[per-mode=symbol]{1.99}[\$]{\per\kilogram}    \\
\SI[per-mode=fraction]{1,345}{\coulomb\per\mole} \\
\si[unit-color=blue]{\highlight{red}\kilogram\metre\per\second}
$p=\SI{24,381}{MPa}$

\section{list/Range}
\numlist{10;30;50;70} \\
\numrange{10}{30}\\
\SIlist{10;30;45}{\metre}\\
\SIrange{10}{30}{\metre} \\
\num[negative-color = red]{-15673}

\section{Angle}
\ang{10}    \\
\ang{12.3}  \\
\ang{4,5}   \\
\ang{1;2;3} \\
\ang{;;1}   \\
\ang{+10;;} \\
\ang{-0;1;}


\chapter{Quantitative Finance}
\label{ch:quantitative-finance}
\index{Quantitative Finance}

\newthought{This section deals with some applications of quantitative finance,
both computational and theoretical. }

\section{Equity Mathematical Models}
\index{Equity Mathematical Models}
Equities and finance are heavily influenced by random events (see \cite{fooledbyrandomness} for a detailed philosophical discussion). As such, the analysis of financial instruments lends itself to a stochastic process.

A {\em Geometric Brownian Motion}\index{Geometric Brownian Motion} (GBM) equation (a form of stochastic differential equation, or SDE known as the Black-Sholes-Merton equation\cite{pyfinanceoreilly}) can be used to approximate a random process over time. In its continuous form this looks like \cite{advancedquantcpp}:
\begin{equation}
ds = \mu Sdt+\sigma SdW
\end{equation}
This breaks the movement of a stock price down into two key effects:

\begin{itemize}
	\item a deterministic effect (the left of the plus sign)
	\item a stochastic effect (the right of the plus sign)
\end{itemize}
In the equation, $\mu$ is known as {\it drift}, $\sigma$ is {\it volatility}, $S$ is the stock price, $dt$ is the change in time and $dW$ is an increment in a {\it Weiner process}.

As equity markets are a discrete process, this equation must be transformed into a discrete equation.
\begin{equation}
S_{t+1}=S_t(1+r\Delta{}t+\sigma\varepsilon_t \sqrt{\Delta{}t})
\end{equation}
Here $\varepsilon$ is a sample from a gaussian distribution with zero mean and standard deviation of 1 (i.e. $N(0,1)$) and $r$ is the risk free rate of return. This equation can be solved iteratively for a given time period if $r$, $\sigma$, $\varepsilon$ and $S_0$ are provided.

Another formulation for pricing European Call options gives a similar result of the discretised SDE\cite{pyfinanceoreilly}:
\begin{equation}
S_t=S_{t-\Delta t}e^{(r-\frac{1}{2}\sigma^2)\Delta t+\sigma\sqrt{\Delta t}z_t}
\end{equation}
In this instance, $z_t$ represents the random variable. Similar formulations can be determined for foreign exchange:
\begin{equation}
X_{t+1} = X_t(1+(r_d-r_f)\Delta{}t+\sigma\varepsilon_t\sqrt{\Delta{}t})
\label{eq:foreignexchangemodel}
\end{equation}
where $r_d$ and $r_f$ are the domestic and foreign fisk free rates of return.

\section{Structural and Intensity Models}
\TODO See \cite{advancedquantcpp} chapter 2.

\section{Monte Carlo Simulation}
\index{Monte Carlo Simulation}
Monte Carlo simulation is a method of estimating a probabilistic outcome through a high quantity of simulations. In a quantitative finance context, we may wish to estimate the future price of an equity based on use of a GBM equation simulated $M$ times.

For instance the process could be as follows for calculating a call option derivative based on estimate the price of a stock using Monte Carlo simulation:

\begin{enumerate}
	\item Generate $M$ different ``trajectories'' for the stock using a GBM simulation from time $t=0$ to time $t=T$. This generates a set of $N$ price estimates for $M$ different simulations, with the notation:
	\begin{equation}
		\{S_i^j\}\qquad i = 0...N,\qquad j=1...M
	\end{equation}
	This produces a vector of $M$ values for $S_T$,
	\begin{equation}
		\{S_T^i\}\qquad i = 0...M
	\end{equation}

	\item We next compute the pay off for each stock value. This is given by
	\begin{equation}
		H(S_T^i)\qquad i=1...M
	\end{equation}
	Where
	\begin{equation}
		H(S_T)=max(S_T-K,0)
	\end{equation}
	and $K$ is the actual price of the equity or premium. The expected pay off can then be computed by the average of all pay offs:
	\begin{equation}
		E[H(S_T^i)]=\frac{1}{M}\Sigma_{i=1}^{M}H(S_T^i)
	\end{equation}

	\item The item should then be discounted to present value, by either applying a discount factor $DF_T$ or
	\begin{equation}
		\pi = e^{-rT}\times E[H(S_T)]
	\end{equation}

	Where $\pi$ is the value of the derivative.
\end{enumerate}

\section{Binomial Trees \cite{advancedquantcpp}}
\index{Binomial Trees}

This approach builds a tree of possible prices. At each stage, the underlying can be assumed to go up or down by a given amount. The amount of change up ($u$) or down ($d$) is described by
\begin{equation}
	u=e^{\sigma\sqrt{\Delta{}t}}
\end{equation}
\begin{equation}
	d=e^{-\sigma\sqrt{\Delta{}t}}
\end{equation}
The probability of an equity going up $p$ is
\begin{equation}
p=\frac{e^{r\Delta{}t}-d}{u-d}
\end{equation}
The probability of the underlying going down is $1-p$. The binomial tree is then built in the following phases:

\begin{enumerate}
	\item Construct a tree where each level corresponds to a time step in the simulation period from $t=0$ to $t=T$. For example in two simulation steps,
	\begin{equation}
		At = 0, S = S_0
	\end{equation}
	\begin{equation}
		At = t_1, S = uS_0,\qquad dS_0
	\end{equation}
	\begin{equation}
		At = t_2, S = u^2S_0,\qquad udS_0,\qquad udS_0,\qquad d^2S_0
	\end{equation}
	Note that the central value at the last period is shared between adjacent nodes, hence there are only three distinct estimates after $N=2$ steps.

	This produces a number of prices at each time step. The notation is based on the estimate number $k$ at time T. There are $N$ time steps
	\begin{equation}
		\{S_T^k\}\qquad k=1...N+1
	\end{equation}

	\item Once the tree has been built, the payoff $H(S_T^k)$ should be calculated for each $S_T^k$

	\item Finally the tree is traversed back up towards the root node, calculating the discounted weighted probability of each node. If we define a given node (not a leaf, i.e. $t\neq T$) it has two children, one for the up price (denoted $S_T^u$) and one for the down price, $S_T^d$. The value for the parent node is given by
	\begin{equation}
	V_{T-1}^k = e^{-r\Delta{}t}[pH(S_T^u) + (1-p)H(S_T^d)]
	\end{equation}
	in this case $V_T^l$ is shorthand for $H(S_T^k)$

	\item Once the tree has been traversed back to the top, the value of the derivative $\pi=V_1^1$
\end{enumerate}

\section{Finite Difference Method}
\index{Finite Difference Method}

The finite difference method is a method for discretising a differential equation.\cite{advancedquantcpp} In the quantitative finance approach, we want to discretise partial differential equations (PDEs). The finite differences method is based on the relationship:
\begin{equation}
f(x) = \frac{df}{dx} \approx \frac{\Delta f}{\Delta x} = \frac{f_{i+1}-f_i}{\Delta x}
\end{equation}
The most important PDE in finance is the {\em Black-Scholes PDE}
\index{Black-Scholes PDE}, which is given by:
\begin{equation}
\frac{\delta V}{\delta t} + \frac{1}{2}\sigma^2S^2\frac{\delta^2V}{\delta S^2}+rS\frac{\delta V}{\delta S}-rV=0
\label{eq:blackscholeseq}
\end{equation}
This is usually solved in the $S$ and $t$ axes, where $S\in [a,b]$ and $t\in [0,T]$. The domain of this equation is said to be
\begin{equation}
\Omega=\{(S,t)\forall S\in [a,b]\times t\in [0,T]\}
\end{equation}
In other words, as the finite difference method is solving some partial differential equation in $S$ and $t$, the solution space is the rectangular domain defined by the ranges of $S$ and $t$. For a European call,
\begin{equation}
	V(S,T) = max(S-K,0)
\end{equation}
The boundary conditions are $V(a,t)=0$ and $V(b,t)=S$. This equation can be transformed with some variable substitution so that
\begin{equation}
\frac{\delta u}{\delta\tau}=\frac{\delta^2 u}{\delta x}\qquad -\infty<x<\infty, \tau>0
\end{equation}
This is a dimensionless PDE with a new solution domain $\Omega=\{(x,\tau)\}$. The payoff relationship therefore becomes
\sidenote{Where $k=\frac{r}{0.5\times\sigma^2}$}
\begin{equation}
u(x,0) = max(e^{\frac{1}{2}(k+1)x}-e^{\frac{1}{2}(k-1)x},0)
\end{equation}
Using finite differences, the return can be described as\sidenote{Where $\alpha=\frac{\Delta\tau}{(\Delta x)^2}$}
\begin{equation}
u_{i,j+1}=\alpha u_{i+1,j}+(1-2\alpha)u_{i,j}+\alpha u_{i-1,j}
\end{equation}
This relationship can be solved iteratively, using the following steps:

\begin{enumerate}
	\item Discretise the domain into N space divisions of $dS$ and M time divisions of $dT$. Use these to determine the steps $\Delta\tau,\Delta x$.
	\item Use finite differences to approximate the derivatives
	\item Calculate the results of the equation iteratively for each time step
\end{enumerate}

For a worked example, see \cite{advancedquantcpp}, end of chapter 3.


\subsection{Prior and posterior predictive distributions}

\subsubsection{Prior predictive distribution}

Our prior predictive distribution for \(X\) depends on our prior for \(\theta \).

\(P(\mathbf x)=\int_\Theta P(\mathbf x|\theta)P(\theta )d\theta \)

\subsubsection{Posterior predictive distribution}

Once we have calculated \(P(\theta |X)\), we can calculate a posterior probability distribution for \(X\).

\(P(\mathbf x|\mathbf X)=\int_\Theta P(\mathbf x|\theta)P(\theta |\mathbf X)d\theta \)


\subsection{Defining sets by enumeration}

We can describe a set by the elements it contains.

\(s=\{a,b,c\}\)

This is a shorthand way of writing:

\(\forall x (x\in s \leftrightarrow (x=a\lor x=b \lor x=c))\)