\subsection{Hamilton Equations}

\noindent\textbf{Legendre transformation}

\noindent Single variables

% \begin{definition}
    The Legendre transformation of the convex function $f(x)$ is another function $g(p)$ defined as
    $g(p)=\max_{x}F(p,x)$, where $F(p,x)=px-f(x)$.
% \end{definition}

The function $F(p,x)$ gets the maximum with respect to $x$ only when $\partial F/\partial x =0$
which means $p = f'(x)$. So, the transformation of the function $x$ is
\begin{equation}
        g(p) = px-f(x),\ p=f'(x)
\end{equation}

The variable $x$ and $p$ are dual variables. The functions $f$ and $g$ are the Legendre transformation mutually and
the function $g(p)$ is also a convex function.

\noindent Multi-variables

For the multi-variable function $f(\mathbf{x}):\mathbf{x}\in \mathbb{R}^{n}\rightarrow \mathbb{R}$,
its Legendre transformation is defined as
\begin{equation}
        g(\mathbf{p}) = \mathbf{p}^{T}\mathbf{x}-f(\mathbf{x}),\ \mathbf{p}=f'(\mathbf{x})
\end{equation}

\noindent \textbf{Hamiltonian}
The Legendre transformation of the Lagrangian $L(\mathbf{q},\dot{\mathbf{q}},t)$ is
\begin{equation}
    H(\mathbf{p},\mathbf{q},t)=\mathbf{p}^{T}\dot{\mathbf{q}}-L(\mathbf{q},\dot{\mathbf{q}},t),\ \mathbf{p}=\frac{\partial L}{\partial \dot{\mathbf{q}}}
\end{equation}

$H(\mathbf{p},\mathbf{q},t)$ is the so-called Hamiltonian.
The Lagrange equation is equivalent to the following Hamilton equations
\begin{equation}
    \begin{aligned}
        &\dot{\mathbf{p}} = -\frac{\partial H}{\partial \mathbf{q}} \\
        &\dot{\mathbf{q}} = \frac{\partial H}{\partial \mathbf{q}}
    \end{aligned}
\end{equation}

Proof:
% \begin{proof}
    The total differential of $H$ is
    $$dH = \frac{\partial H}{\partial \mathbf{p}}d\mathbf{p} + \frac{\partial H}{\partial \mathbf{q}}d\mathbf{q} + \frac{\partial H}{\partial t}dt.$$
    The total differential of $\mathbf{p}^{T}\dot{\mathbf{q}}-L(\mathbf{q},\dot{\mathbf{q}},t)$ is
    $$dH = \dot{\mathbf{p}}^{T}d\mathbf{p} - \frac{\partial L}{\partial \mathbf{q}}d\mathbf{q} - \frac{\partial L}{\partial t}dt$$
    The two equations are equivalent, so
\begin{equation}
    \dot{\mathbf{q}}=\frac{\partial H}{\partial \mathbf{p}},\ \frac{\partial H}{\partial \mathbf{q}}=-\frac{\partial L}{\partial \mathbf{q}},\ \frac{\partial H}{\partial t}=-\frac{\partial H}{\partial t}.
\end{equation}
    Combining with the Lagrange equation and $\mathbf{p}=\frac{\partial L}{\partial \dot{\mathbf{q}}}$, one can obtain the Hamilton equations.
% \end{proof}

The Euler's homogeneous function theorem for quadric form $f(\mathbf{x})$ is
\begin{equation}
    \mathbf{x}^{T}\frac{\partial f}{\partial \mathbf{x}}=2f
\end{equation}

Assuming the kinematic energy $T$ is a quadric form with respect to $\dot{\mathbf{q}}$, one can obtain
\begin{equation}
    H(\mathbf{p},\mathbf{q},t)=\mathbf{p}^{T}\dot{\mathbf{q}}-L(\mathbf{q},\dot{\mathbf{q}},t)=\mathbf{p}^{T}\dot{\mathbf{q}}-T+U=\dot{\mathbf{q}}^{T}\frac{\partial T}{\partial \dot{\mathbf{q}}}-T+U=T+U.
\end{equation}
Therefore, the Hamiltonian is the total energy.


\section{Task 01a}
In this exercise I will solve the one dimensional Poisson equation with Dirichlet boundary conditions by rewriting it as a set of linear equations.

\begin{equation}
	-u''(x) = f(x),  x\in (0,1), u(0)=u(1)=0
\end{equation}

We can approximate the second order derivative of u with:

\begin{equation*}
   -\frac{v_{i+1}+v_{i-1}-2v_i}{h^2} = f_i  \hspace{0.5cm} \mathrm{for} \hspace{0.1cm} i=1,\dots, n,
\end{equation*}

This can be written as a set of linear equations almost directly:

\begin{equation*}
   -v_{i-1}+2v_i-v_{i+1} = f_i h^2  \hspace{0.5cm} \mathrm{for} \hspace{0.1cm} i=1,\dots, n,
\end{equation*}

\begin{equation*}
   -v_{0}+2v_1-v_{2} = f_1 h^2\\
\end{equation*}
\begin{equation*}
   -v_{1}+2v_2-v_{3} = f_2 h^2\\
\end{equation*}
\begin{equation*}
   -v_{2}+2v_3-v_{4} = f_3 h^2\\
\end{equation*}
\begin{equation*}
   -v_{n-2}+2v_{n-1}-v_{n} = f_{n-1} h^2\\
\end{equation*}
\begin{equation*}
   -v_{n-1}+2v_n-v_{n+1} = f_n h^2\\
\end{equation*}

$v_0$ and $v_{n+1}$ are 0, so the linear equations reduce to the following matrix form:
\[
\begin{bmatrix}
       2& -1& 0 &\dots   & \dots &0 \\
       -1 & 2 & -1 &0 &\dots &\dots \\
       0&-1 &2 & -1 & 0 & \dots \\
       & \dots   & \dots &\dots   &\dots & \dots \\
       0&\dots   &  &-1 &2& -1 \\
       0&\dots    &  & 0  &-1 & 2 \\
\end{bmatrix}
\begin{bmatrix}
	v_1 \\
	v_2 \\
	v_3 \\
	\vdots \\
	v_{n-1} \\
	v_n \\
\end{bmatrix} =
h^2
\begin{bmatrix}
	f_1 \\
	f_2 \\
	f_3 \\
	\vdots \\
	f_{n-1} \\
	f_n \\
\end{bmatrix}
\]

This is the same form of
\begin{equation*}
   \mathbf{A}\mathbf{v} = \tilde{\mathbf{b}},
\end{equation*}

\section{Task 01b}
In this exercise I want to find a general algorithm for solving this set of equations with different values for the matrix elements by using decomposition and forward substitution and backward substitution. Let the tridiagonal matrix have elements $a_i$, $b_i$ and $c_i$\\

For clarity replace $'\tilde{b}'$ with $'d'$
\begin{equation*}
   a_{i-1} v_{i-1} + b_i v_i + c_i v_{i+1} = d_i  \hspace{0.5cm} \mathrm{for} \hspace{0.1cm} i=1,\dots, n,
\end{equation*}

where
$v_0 = v_{n+1} = a_0 = a_{n} = c_n = 0$\\

\begin{gather*}
   b_1 v_1 + c_1 v_{2} = d_1\\
   a_{1} v_{1} + b_2 v_2 + c_2 v_{3} = d_2\\
   a_{2} v_{2} + b_3 v_3 + c_3 v_{4} = d_3\\
   \vdots\\
   a_{n-2} v_{n-2} + b_{n-1} v_{n-1} + c_{n-1} v_{n} = d_{n-1}\\
   a_{n-1} v_{n-1}+b_{n} v_{n} = d_{n}\\
\end{gather*}

Methodology: Reduce the tridiagonal matrix to an upper diagonal matrix by decomposition, forwards substitution and backward substitution.
Modifying equation 2 with equation 1 to eliminate $v_1$\\
Equation 2 * b1 - equation 1 *a1: \\

\[
\begin{bmatrix}
     b_1 & c_1     & 0     & \dots    & \dots   & \dots    & \| & d_1 \\
     a_1 & b_2     & c_2   & \dots    & \dots   & \dots    & \| & d_2 \\
         & a_2     & b_3   & c_3      & \dots   & \dots    & \| & d_3 \\
         & \dots   & \dots & \dots    & \dots   & \dots    & \| & \dots \\
         &         &       & a_{n-2}  & b_{n-1} & c_{n-1}  & \| & d_{n-1} \\
         &         &       &          & a_{n-1} & b_n      & \| & d_n \\
\end{bmatrix}
\]

Equation 1 times $a_1$ and equation 2 times $b_1$\\
\[
\begin{bmatrix}
     a_1 b_1 & a_1 c_1     & 0     & \dots    & \dots   & \dots    & \| & a_1 d_1 \\
     b_1 a_1 & b_1 b_2     & b_1 c_2   & \dots    & \dots   & \dots    & \| & b_1 d_2 \\
         & a_2     & b_3   & c_3      & \dots   & \dots    & \| & d_3 \\
         & \dots   & \dots & \dots    & \dots   & \dots    & \| & \dots \\
         &         &       & a_{n-2}  & b_{n-1} & c_{n-1}  & \| & d_{n-1} \\
         &         &       &          & a_{n-1} & b_n      & \| & d_n \\
\end{bmatrix}
\]

Equation 2 - equation 1\\
\[
\begin{bmatrix}
     b_1 & c_1                   & 0     & \dots    & \dots   & \dots    & \| & d_1 \\
     0   & b_1 b_2 - a_1 c_1     & b_1 c_2   & \dots    & \dots   & \dots    & \| & b_1 d_2 - a_1 d_1 \\
         & a_2                   & b_3   & c_3      & \dots   & \dots    & \| & d_3 \\
         & \dots                 & \dots & \dots    & \dots   & \dots    & \| & \dots \\
         &                       &       & a_{n-2}  & b_{n-1} & c_{n-1}  & \| & d_{n-1} \\
         &                       &       &          & a_{n-1} & b_n      & \| & d_n \\
\end{bmatrix}
\]

$\tilde{b_2} = b_1 b_2 - a_1 c_1$ \\
$\tilde{c_2} = b_1 c_2$ \\
$\tilde{d_2} = b_1 d_2 - a_1 d_1$ \\

\[
\begin{bmatrix}
     b_1 & c_1                   & 0     & \dots    & \dots   & \dots    & \| & d_1 \\
     0   & \tilde{b_2} & \tilde{c_2}   & \dots    & \dots   & \dots    & \| & \tilde{d_2} \\
         & a_2                   & b_3   & c_3      & \dots   & \dots    & \| & d_3 \\
         & \dots                 & \dots & \dots    & \dots   & \dots    & \| & \dots \\
         &                       &       & a_{n-2}  & b_{n-1} & c_{n-1}  & \| & d_{n-1} \\
         &                       &       &          & a_{n-1} & b_n      & \| & d_n \\
\end{bmatrix}
\]

Equation 2 times $a_2$ and equation 3 times $\tilde{b_2}$\\

\[
\begin{bmatrix}
     b_1 & c_1                   & 0     & \dots    & \dots   & \dots    & \| & d_1 \\
     0   & a_2 \tilde{b_2}       & a_2 \tilde{c_2}   & \dots    & \dots   & \dots    & \| & a_2 \tilde{d_2} \\
         & \tilde{b_2} a_2       & \tilde{b_2} b_3   & \tilde{b_2} c_3      & \dots   & \dots    & \| & \tilde{b_2} d_3 \\
         & \dots                 & \dots & \dots    & \dots   & \dots    & \| & \dots \\
         &                       &       & a_{n-2}  & b_{n-1} & c_{n-1}  & \| & d_{n-1} \\
         &                       &       &          & a_{n-1} & b_n      & \| & d_n \\
\end{bmatrix}
\]

Equation 3 -equation 2: \\

\[
\begin{bmatrix}
     b_1 & c_1                   & 0     & \dots    & \dots   & \dots    & \| & d_1 \\
     0   & a_2 \tilde{b_2} & a_2 \tilde{c_2}   & \dots    & \dots   & \dots    & \| & a_2 \tilde{d_2} \\
         & 0                   & \tilde{b_2} b_3 - a_2 \tilde{c_2}  & \tilde{b_2} c_3      & \dots   & \dots    & \| & \tilde{b_2} d_3 - a_2 \tilde{d_2}\\
         & \dots                 & \dots & \dots    & \dots   & \dots    & \| & \dots \\
         &                       &       & a_{n-2}  & b_{n-1} & c_{n-1}  & \| & d_{n-1} \\
         &                       &       &          & a_{n-1} & b_n      & \| & d_n \\
\end{bmatrix}
\]

$\tilde{b_3} = \tilde{b_2} b_3 - a_2 \tilde{c_2}$ \\
$\tilde{c_3} = \tilde{b_2} c_3$ \\
$\tilde{d_3} = \tilde{b_2} d_3 - a_2 \tilde{d_2}$ \\

By examining the procedure, the modified coefficients notated with tildes may be defined recursively:\\

\begin{gather*}
	\tilde{a_i} = 0 \\
	\tilde{b_1} = b_1 \\
	\tilde{b_i} = b_i \tilde{b_{i-1}} - a_{i-1} \tilde{c_{i-1}} \\
	\tilde{c_1} = c_1 \\
	\tilde{c_i} = \tilde{b_{i-1}} c_i \\
	\tilde{d_1} = d_1  \\
	\tilde{d_i} = \tilde{b_{i-1}} d_i - a_{i-1} \tilde{d_{i-1}} \\
\end{gather*}

To further hasten the solution process, $\tilde{d_i}$ may be divided out if there is no division by zero risk. The newer modified coefficients each notated with a hat will be:\\

\begin{gather*}
	\hat{a_i} = 0 \\
	\hat{b_i} = 1 \\
	\hat{c_1} = \frac{c_1}{b_1} \\
	\hat{c_i} = \frac{c_i}{b_i - a_{i-1} \hat{c_{i-1}}} \\
	\hat{d_1} = \frac{d_1}{b_1}  \\
	\hat{d_i} = \frac{ d_i - a_{i-1} \hat{d_{i-1}}}{b_i - a_{i-1} \hat{c_{i-1}}} \\
\end{gather*}

This gives the following system with the same unknowns and coefficients defined in terms of the original ones above: \\

$v_i + \hat{c_i} v_{i+1} = \hat{d_i}$

$v_n = \hat{d_n}$

The last equation involves only one unknown. Solving it in turn reduces the next last equation to one unknown. This backward substitution can be used to find all of the unknowns: \\
$v_n = \hat{d_n}$  \\
$v_i + \hat{c_i} v_{i+1} = \hat{d_i}$ for $i = n-1, n-2, \ldots, 1$ \\

Floating point operations: \\

For calculating $\hat{c_i}$ we need $3(n-2)+1$ flops. \\
For calculating $\hat{d_i}$ we need $5(n-1)+1$ flops. \\
For back substitution we need $2(n-1)+1$ flops. \\

However the denominator is the same for both $\hat{c_i}$ and $\hat{d_i}$. Thus the calculation of the denominator only has to be carried out once. Therefore we can subtract 2(n-2) flops from the total number of flops.

In total to solve the problem we need $7(n-1)+3(n-2)-2(n-2)+3 = 8(n-1)+2$ flops. \\

\FloatBarrier
\begin{figure}[!ht]
\centering
\FloatBarrier
\includegraphics[width=0.32\textwidth]{task1b_n10.png}
\includegraphics[width=0.32\textwidth]{task1b_n100.png}
\includegraphics[width=0.32\textwidth]{task1b_n1000.png}
\caption{Solving the Poisson equation numerically and exact with grid size n =10,100,1000}
\label{fig:Poisson}
\end{figure}
\FloatBarrier

From figure~\ref{fig:Poisson} I can see that the numerical solution approaches the exact solution for more grid points. \\

\section{Task 01c}

Assuming identical matrix elements along the diagonal and identical  but different values for the non-diagonal elements. This means that the $a_i =c_i =-1$ and $b_i =2$. Substituting this result into the solution we found in project 1b we get:
\begin{gather*}
	\hat{a_i} = 0 \\
	\hat{b_i} = 1 \\
	\hat{c_1} = -0.5 \\
	\hat{c_i} = \frac{-1}{2 + \hat{c_{i-1}}} \\
	\hat{d_1} = \frac{d_1}{2}  \\
	\hat{d_i} = \frac{d_i + \hat{d_{i-1}}}{2 + \hat{c_{i-1}}} \\
\end{gather*}

Floating point operations specific: \\

For calculating $\hat{c_i}$ we need $2(n-2)$ flops. \\
For calculating $\hat{d_i}$ we need $3(n-1)+1$ flops. \\
Back substitution does not change so it's the same is in 1b, we need $2(n-1)+1$ flops. \\

The denominator is the same for both $\hat{c_i}$ and $\hat{d_i}$. Thus the calculation of the denominator only has to be carried out once. Therefore we can subtract (n-2) flops from the total number of flops.

In total to solve the specific problem we need $5(n-1)+2(n-2)-(n-2)+2 = 6(n-1)+1$ flops. \\

Running the program to check for CPU time for both the general and the specific case I got very similar results the first time. General case gave  a time of: 7.62825565887. The specific case gave a time of
7.512299363. This is probably because pure Python code has more overhead than compiled languages like C++ or Fortran, minimizing the importance of flops. Therefore I tried to import numba, which compiles the code just in time. This resulted in a time for the general case of; 0.515605151976 seconds, and for the specific case:
0.309811743968 seconds. Comparing the flops to the running times, I get for Flops: $(9(n-1)+2)/(6(n-1)+1) \approx 1.5$ and for running times: $0.515605151976/0.309811743968 \approx 1.65$\\

\section{Task 01d}
In this exercise I want to check for truncation errors in the calculations.\\

Here I made a function that extracts the maximum relative error for each step length. The results of the calculations are shown in figure~\ref{fig:Error_poisson}.\\

\FloatBarrier
\begin{figure}[!ht]
\centering
\FloatBarrier
\includegraphics[width=0.45\textwidth]{error.png}

\caption{Computing the relative error of the numerical solution to the Poisson equation for grid sizes n =10, 100, 1000, 10000, 100000, 1000000, 10000000}
\label{fig:Error_poisson}
\end{figure}
\FloatBarrier

In the figure~\ref{fig:Error_poisson} I can see that the error declines until h reaches about $10^{-5}$. Decreasing the step size beyond this value results in increased truncation errors.\\

\section{Task 01e}
Flops of LU decomposition goes like $O(n^2)$ when trying to solve the set of linear equations.

Trying to run the LU decomposition for a matrix of size $10^{10}$ does not work because I have too little memory space.

\FloatBarrier
\begin{table}[!ht]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
n & tridiagonal specific & tridiagonal & LU \\
\hline
10 & 0.000204 & 0.000177 & 0.002522 \\
\hline
100 & 0.000212 & 0.000199 & 0.002558 \\
\hline
1000 & 0.000478 & 0.000557 & 0.299024 \\
\hline
2000 & 0.000293 & 0.000516 & 1.664050 \\
\hline

\label{Runtimes for various methods of decomposition}
\end{tabular}
\caption{Runtimes in seconds for various methods of decomposition}
\end{table}
\FloatBarrier

For tridiagonal specific flops goes like $O(6n)$. For tridiagonal flops goes like $O(9n)$. LU decomposition flops goes like $O(n^2)$. From the runtimes I can see in \ref{Runtimes for various methods of decomposition} that the two first grows linearly within errors, while the LU decomposition grows quadratically.\\

\section{Comments}
In general the exercise was useful, but I realize that I could have written the report better. I have just gotten started with github, so I attached the program code, like I have done in projects I have done in the past. I hand in the exercise a bit late, because I was abroad until recently after agreement with Morten. It would be great to get some feedback on some of the report outlines, and maybe a tip on where or how I could implement further tests in my program.


\subsection{The Power Method}

According to Golub \cite{golub00}, quoting Householder, the power
method has its origin at the work of Müntz in 1913
\cite{muntz1913}. The method is the simplest algorithm for solving the
eigenproblem; it basically consists in picking carefully a vector,
and then apply the matrix $A$ iteratively until it converges to an
eigenvector. And not to any eigenvector, but precisely to the one with
largest eigenvalue (in absolute value). The following pseudocode is
taken from Golub \cite{golub13}:

\begin{algorithm}
  \label{alg:power-method}
  \caption{The Power Method}
%
  \setstretch{1.5}
  \SetKwInOut{Input}{Input}
  \SetKwInOut{Output}{Output}
  \DontPrintSemicolon
%
    \Input{A unit vector $\vec{q_0} \in \R{n}$ and a symmetric matrix $A^{n
        \times n}$}
%
    \Output{The tuple $(\lambda_k,\vec{q_k})$ which is expected to
      approximate an eigenpair $(\lambda,\vec{q})$ of $A$}
%
    \For {$i = 1,2,\dots,k$}
    {
      $\vec{z_k} \gets A \vec{q_{k-1}}$ \;
      $\vec{q_k} \gets \dfrac{\vec{z_k}}{\norm{\vec{z_k}}_2}$ \;
      $\lambda_k \gets \trans{\vec{q_k}} A \vec{q_k}$ \;
    }
%
    return $(\lambda_k, \vec{q_k})$ \;
\end{algorithm}
\hfill

One immediate trick that is detected, is that we are not giving a
precise value for $k$, the number of iterations; this is because we do
not really know how many in advance, though we know how ``fast'' we
can reach convergence, more about this in a minute. Though the
algorithm looks trivial, a powerful theorem justifies why it
works. Golub mentions the  conditions which are required for its
convergence: the
maximum eigenvalue of $A$ must be unique (no repetition), and the
initial vector $q_0$ is not ``deficient''  (its component on the
direction of the eigenvector with maximum value must not be zero). A
proof of convergence/correctness can be consulted in \cite{golub13}. \\\

Golub also mentions the computable error bounds of this method.
The real eigenvalue and eigenvectors of $A$ will satisfy the equation
below:

\[
A \vec{q} = \lambda \vec{q}
\]
\hfill

But accepting the fact that a computer will not product exactly the
eigenvalue nor the eigenvector, we can at least see how close we are
in meeting above condition. That is, we can calculate the error $\delta$:

\[
\norm{A \vec{q_k} - \lambda_k \vec{q_k}}_2 = \delta
\]
\hfill

Golub shows that there is an eigenvalue $\lambda$ that satisfies
$\abs{\lambda_k - \lambda} \le \sqrt{2}\delta$; which is a way to tell
that we can really approximate an actual eigenvalue, as long as we are
capable of reproducing its defining property with good accuracy (which
in turn, will depend on how many iterations we make). \\

Alright, so we know how to calculate one eigenpair; why not
calculating them all? We may be tempted now to recall the geometric
proof of SVD (see \cref{cha:svd-theory}), and consider the following
procedure for finding all the eigenpairs (assuming preconditions met):

\begin{enumerate}
\item Pick carefully initial vector.
\item Apply \cref{alg:power-method} to find the first
  eigenpair.
\item Obtain the hyperplane that is orthogonal to the first
  eigenvector found, and repeat recursively the procedure until
  we have all the desired eigenpairs\footnote{The third step is
    usually called ``deflation'' (see \cite{golub13}), when mentioned
    in the context of the matrix, as it is reduced to dimensions $(n-1)
    \times (n-1)$.}.
\end{enumerate}
\hfill

The problem with this procedure, also exposed by Golub in his proof of
correctness, is that the rate of convergence depends on
$\abs{\frac{\lambda_2}{\lambda_1}}^k$; where $\lambda_2$ is the second
largest eigenvalue in absolute value. Thus, unless there is a
considerable gap between first and second largest eigenvalues of $A$,
the Power Method will converge quite slowly. That makes it unsuitable
for practical purposes, at least in the standalone version we just
presented. Further sections will show how it can evolve to overcome
this limitation.


\chapter*{Part One: Matrix Factorization}
\addcontentsline{toc}{chapter}{Part One}

Note: I collaborated with Vivienne Zhang and Justin Yao.

The first part to code was the latent semantic analysis word representations. Latent semantic analysis uses singular value decomposition to represent words in a lower dimensional space than the number of documents in the corpus. In my case, I found that using the principle components matrix $\bm{U}$ alone provided significantly better results than weighting it with the diagonal matrix $\bm{\Sigma}$.

\begin{python}
    def learn_reps_lsa(matrix, rep_size):
    U, sigma, V = np.linalg.svd(matrix, full_matrices=True)
    return result = U[:, :rep_size]
\end{python}

The next piece of code is the term frequency-inverse document frequency (TF-IDF), which re-weights word counts in the term-document matrix to prevent frequent words from dominating the word representation. It penalizes ubiquitous words while allowing sparser data to appear in the representation.

\begin{python}
    def transform_tfidf(matrix):

    nwords, ndocs = matrix.shape
    vidf = np.vectorize(idf_helper)
    idfs = np.zeros(nwords)
    for i, _ in enumerate(idfs):
    idfs[i] = idf_helper(matrix[i,:], ndocs)
    tfidf = np.multiply(matrix, idfs[:, np.newaxis])
    return tfidf

    def idf_helper(row, ndocs):
    df = np.count_nonzero(row)
    idf = np.log(ndocs/df)
    return idf
\end{python}

The final code is the LSA featurizer, which projects word contexts into the LSA lower dimensional representation.

\begin{python}
    def lsa_featurizer(xs):
    lsa = reps_tfidf
    feats = xs @ lsa
    # normalize
    return feats / np.sqrt((feats ** 2).sum(axis=1, keepdims=True))
\end{python}

\subsection*{Theoretical Problem A}
The singular value decomposition of a matrix $\bm{M}$ is given by

\begin{equation*}
    \bm{M} = \bm{U \Sigma V}
\end{equation*}

Given the word co-occurence matrix $\bm{W_{tt}} = \bm{W_{td}W_{td}^T}$ we can solve as follows by applying the SVD to the matrix and its transpose:

\begin{align*}
    \bm{MM^T} & = \bm{U \Sigma V^T} \cdot \bm{V \Sigma U^T} \\
              & = \bm{U \Sigma^2 U^T}
\end{align*}

The final result is itself a singular decomposition, where the left singular matrix of $\bm{W_{tt}}$ is equal to the left singular matrix of the original term-document matrix $\bm{W_{td}}$.
If $\bm{U}$ is negative, then $\bm{U \Sigma^2 U^T}$ is positive, resulting in flipped signs compared to the original, therefore the left singular matrices are equivalent to each other's absolute values. This behavior is observed in my implementation of learn\_reps\_lsa, which simply truncates $\bm{U}$ along its second axis to achieve a lower dimensional word representation. This implies that word co-occurence relations can be incorporated while using $\bm{U}$ for my representation.

\subsection*{Experimental Problem B}

The nearest neighbors in representation space appear better for some words than others. For instance, their is a clear connection between dog and all of its top related words. Words associated with "the" are seemingly random; given that "the" is a filler word, associations are likely largely meaningless.

For this data set, using a smaller representation size and therefore a denser approximation of the term-document matrix appears to give slightly more meaningful associations. My LSA performance results match this assertion.

\begin{center}

    \begin{tabular}{ |p{3cm}|p{3cm}|p{3cm}|p{3cm}|  }

        \hline
        \multicolumn{4}{|c|}{Word Similarities r = 100}                 \\
        \hline
        the             & dog            & 3           & good           \\
        \hline
        lid 1.753       & foods 0.571    & per 0.884   & pretty 1.255   \\
        leaked 1.779    & pets 0.624     & 1 0.943     & bread 1.349    \\
        filter 1.779    & switched 0.742 & 2 0.992     & prepare 1.376  \\
        rest 1.786      & pet 0.794      & 4 1.061     & flaxseed 1.407 \\
        flavoring 1.789 & cheap 0.831    & ounce 1.100 & beat 1.457     \\
        \hline
    \end{tabular}
\end{center}

\begin{center}
    \begin{tabular}{ |p{3cm}|p{3cm}|p{4cm}|p{3cm}|  }
        \hline
        \multicolumn{4}{|c|}{Word Similarities r = 500}                     \\
        \hline
        the           & dog            & 3                 & good           \\
        \hline
        suspect 1.953 & happier 1.670  & omega 1.733       & gerber 1.873   \\
        flowers 1.961 & earlier 1.681  & vendor 1.739      & luck 1.885     \\
        leaked 1.966  & eats 1.702     & supermarket 1.747 & crazy 1.890    \\
        m 1.966       & stays 1.722    & nutty 1.755       & flaxseed 1.907 \\
        burn 1.967    & standard 1.727 & carries 1.797     & suspect 1.907  \\
        \hline
    \end{tabular}
\end{center}

\subsection*{Experimental Problem C}

Learned representations clearly help with the downstream classification task. Figure \ref{fig:fig} shows a parameter sweep of my three featurizers across representation size and training size. For the word featurizer, once the training size is sufficient it mostly platueaus, with a slight dip in the middle.
LSA featurizer performance drastically improves with a larger training size and a smaller representation space. The maximum values I obtained for each featurizer are as follows: LSA max: 0.812 Word max: 0.784 Combo max: 0.818.

\begin{figure}[ht]
    \begin{subfigure}{.5\textwidth}
        \centering
        % include first image
        \includegraphics[width=.8\linewidth]{resources/png/word_perf.png}
        \caption{Word Featurizer}
        \label{fig:sub-first}
    \end{subfigure}
    \begin{subfigure}{.5\textwidth}
        \centering
        % include second image
        \includegraphics[width=.8\linewidth]{resources/png/lsa_perf.png}
        \caption{LSA Featurizer}
        \label{fig:sub-second}
    \end{subfigure}
    \begin{subfigure}{.5\textwidth}
        \centering
        % include second image
        \includegraphics[width=.8\linewidth]{resources/png/combo_perf.png}
        \caption{Combo Featurizer}
        \label{fig:sub-third}
    \end{subfigure}
    \caption{Featurizer Training and Representation size feature sweep}
    \label{fig:fig}
\end{figure}

\chapter*{Part Two: Language Modeling}
\addcontentsline{toc}{chapter}{Part Two}

This section of the assignment involved implementing a Word2Vec model. My model uses CrossEntropy as its loss function for training, as we are training the model to predict binary labels for positive or negative reviews. CrossEntropy increases loss as a prediction is a further distance from the actual label, making it suited for teaching a model to label a review as good or bad.

For my model itself, I used the standard PyTorch embedding module to represent words. In my forward function, I sum the embedding of all context words for the CBOW model, put the embeddings through a linear layer, and then a log softmax.

I've observed some variance in my training. Somewhere a random number isn't seeded, so the finaly accuracy varies between different trainings in the range of ~.8 to .825.

\subsection*{Experimental Problem A}

The nearest neighbor space for Word2Vec is entirely unintuitive to me. The associated words for good seem reasonable, but every other example word seems somewhat random (particularly dog and barbecue). However Word2Vec appears to be my strongest featurizer, so perhaps the model is learning relationships that are not evident to me? In a production system I would worry it is cheating somehow and discovering idiosyncrasies hidden in my training set.

\begin{center}
    \begin{tabular}{ |p{3cm}|p{3cm}|p{4cm}|p{3cm}|  }
        \hline
        \multicolumn{4}{|c|}{Word2Vec Word Similarities context  = 2}    \\
        \hline
        the         & dog             & 3                 & good         \\
        \hline
        lots 1.668  & companies 1.686 & sip 1.701         & nice 1.660   \\
        fish 1.752  & double 1.707    & gingerbread 1.734 & sauces 1.697 \\
        betty 1.771 & barbeque 1.713  & pets 1.7434       & pure 1.698   \\
        alive 1.774 & times 1.717     & send 1.750        & great 1.704  \\
        my 1.775    & leak 1.726      & bears 1.752       & photo 1.732  \\
        \hline
    \end{tabular}
\end{center}
\subsection*{Experimental Problem B}
Word2Vec is consistently about 2\% more accurate than any of my other featurizers. The gains in performance over LSA are likely due to the fact that Word2Vec models context and has information about word ordering, which could be the difference between a positive and negative review.

\subsection*{Experimental Problem 3}
In terms of accuracy for my downstream task, two appears to be the magic number for my Word2Vec implementation. Using a context size of one tanks the model performance, and increasing it beyond two more gradually decreases performance.

Using a larger context size increases the distance from the center word that our model examines when learning. The relationships between words become less intuitive as the model is more likely to pick up on words farther away in the sentence. For instance, with a context size of six, dog has the following related words: junk, intake, photo, replace, and generally. One could imagine a review along the lines of "Generally I feed my little dog smaller kibbles" that is causing the adverb generally to associate with dog.

\chapter*{Part Three: Improving the Model}
\addcontentsline{toc}{chapter}{Part Three}

For my model improvements, I took multiple approaches. I spent a while tuning hyperparameters and I also implemented alternative TF-IDF weighting schemes.

Unfortunately there is some element of randomness to the training (somewhere a random call is not seeded), and I could not get perfectly deterministic results based on my hyperparameter tuning. For all models a higher training size correlated with better results.

For Word2Vec, I discovered the best performing parameters were generally a context window size of two, a representation size of 500, and five epochs of training. Reducing the training was the most impactful hyperparameter, improving the model by ~1.5\%, implying that the default of ten epochs is probably overfitting the training data. It also makes sense that Word2Vec benefits from a higher dimensional representation than LSA, because Word2Vec has more information in the form of an ordered context window. In terms of overall featurizer performance, my best featurizer is a concatenation of the provided combo featurizer and Word2Vec, with a performance of ~0.82.

For my LSA featurizer, I found the best performing parameter was to set the representation size to 100. A denser representation of the words (especially when context information is stripped and we're left purely with counts) results in a better fit on our regression. This implies there may be noise in our higher dimensional representations that a dense matrix eliminates.

Finally I implemented three alternative weightings schemes for both TF and IDF taken from our lecture notes (Figure \ref{fig2}).

\begin{figure}[ht]
    \begin{subfigure}{.5\textwidth}
        \centering
        % include first image
        \includegraphics[width=.8\linewidth]{resources/png/tf.png}
        \caption{Word Featurizer}

    \end{subfigure}
    \begin{subfigure}{.5\textwidth}
        \centering
        % include second image
        \includegraphics[width=.8\linewidth]{resources/png/idf.png}
        \caption{LSA Featurizer}

    \end{subfigure}
    \caption{Alternative TF-IDF weighting schemes}
    \label{fig2}
\end{figure}

My baseline LSA featurizer had a performance of 0.798. The table below shows a sweep of the different weighting schemes. Most of the weighting schemes failed to improve my results, with the exception being the term frequency combine with the probabilistic IDF, and the log norm combined with the smooth IDF. My best explanation for this is the first pair normalize document length, and perhaps longer reviews were given too much weight in the baseline. The second improved pair add a smoothing factor to account for sparseness in reviews, ensuring the TF-IDF matrix isn't mostly zeroes, and relatively boosting the importance of rare words. Perhaps some rarer but important words were below the noise threshold of the baseline.

\begin{table}[]
    \centering
    \begin{tabular}{|l|c|c|c|l}
        \cline{1-4}
                       & IDF smooth & IDF max & \begin{tabular}[c]{@{}c@{}}probabilistic\\ IDF\end{tabular} & \\ \cline{1-4}
        term frequency & 0.79       & 0.774   & 0.802                                                       & \\ \cline{1-4}
        log norm       & 0.802      & 0.764   & 0.766                                                       & \\ \cline{1-4}
        double norm    & 0.78       & 0.766   & 0.774                                                       & \\ \cline{1-4}
    \end{tabular}
\end{table}

\chapter*{Part Four: Skip-Gram Variant}
\addcontentsline{toc}{chapter}{Part Four}

\subsection*{Theoretical Problem A}
\begin{equation*}
    \sigma(\vec{w} \cdot \vec{c})=\frac{1}{1+e^{-\vec{w} \cdot \vec{c}}}
\end{equation*}

Substituting $x$ for $\vec{w} \cdot \vec{c}$

\begin{equation*}
    \frac{\partial \log \sigma(x)}{\partial x} = 1 - \frac{1}{e^{-x}+1}
\end{equation*}

\subsection*{Theoretical Problem B}

Given the local objective function

\begin{equation*}
    \ell(w, c)=\#(w, c) \cdot \log \sigma(\vec{w} \cdot \vec{c})+k \cdot \#(w) \cdot \frac{\#(c)}{|D|} \cdot \log \sigma(-\vec{w} \cdot \vec{c})
\end{equation*}

Again substituting $x$ for $\vec{w} \cdot \vec{c}$, and substituting $\lambda$ for the constant expression $k \cdot \#(w) \cdot \frac{\#(c)}{|D|}$ and $\alpha$ for constant term $\#(w,c)$

\begin{align*}
    \frac{\partial \ell(x)}{\partial x} & = \frac{\alpha}{e^x + 1} -\lambda \frac{e^x}{e^x + 1}
\end{align*}

Setting the derivative to $0$ we can rewrite $\partial \ell$ in the form

\begin{align*}
     & \frac{\alpha}{1 + e^x} = \frac{\lambda e^x}{1 + e^x}             \\
     & \frac{\alpha}{\lambda} = e^x                                     \\
     & x = \log(\frac{\alpha}{\lambda})                                 \\
     & =\log \left(\frac{\#(w, c)}{\#(w) \cdot \#(c) k} \cdot|D|\right)
\end{align*}

\subsection*{Theoretical Problem C}
Pointwise mutual information is the ratio of how often two events (in our case a word and its context) occur together versus how often they could be expected to occur if one assumes the events are independent. It is a useful index for showing quantitatively how strongly words are associated with each other.

The critical points of the skip-gram local objective function are the same as the PMI formulation except scaled by a factor of $\log(1/k)$ (or shifted by $log(k)$ if you pull the term outside the PMI). $k$ is the number of negative samples introduced to each context. When $k=1$, the skip-gram model trains our embedding vectors towards exactly the PMI.


\subsection{Directional derivative}

We have a function, \(f(\mathbf x)\).

Given a vector \(v\), we can identify by how much this scalar function changes as you move in that direction.

\(\nabla_v f(x):=\lim_{\delta \rightarrow 0}\dfrac{f(\mathbf x+\delta \mathbf v)-f(\mathbf x) }{\delta }\)

The directional derivative is the same dimension as underlying field.

\subsubsection{Other}

Differentiation of scalar field, \(d f\), can be defined as a vector field where grad is 0. can differ with orientation, scale


\documentclass{article}
\usepackage{color}
\usepackage{bigints}
\usepackage[italicdiff]{physics}
\color{white}
\definecolor{Red}{RGB}{124,10,2}
\begin{document}
\pagecolor{Red}
\title{Solution to Calculus with Calvin\textsc{\char13}s Problem}
\author{Jose Bedoya}
\maketitle
\section{Question}
{\LARGE
$$I=\bigintss_{0}^{1} \frac{\ln \left(x+1\right)\ln x}{x}\,dx$$
}
\section{Solution}
{\Large
Knowing,
$$\ln \left(x+1\right)=\sum_{n=1}^{\infty} \left(-1\right)^{n-1}\frac{x^n}{n}$$

\vspace{3mm}
We have
$$I=\sum_{n=1}^{\infty}\frac{\left(-1\right)^{n-1}}{n}\bigintsss_{0}^{1}x^{n-1}\ln x\,dx$$

\vspace{3mm}
Applying IBP, we get
$$I=\sum_{n=1}^{\infty}\frac{\left(-1\right)^{n-1}}{n}\left(\frac{x^n\ln x}{n}\Big|_0^1-\frac{1}{n}\bigintsss_{0}^{1}x^{n-1}\,dx\right)$$
$$=\sum_{n=1}^{\infty}\frac{\left(-1\right)^n}{n^3}$$

\vspace{4mm}
Let\textsc{\char13}s try to find a closed expression for this sum. We know:
$$\zeta\left(3\right)=\sum_{n=1}^{\infty}\frac{1}{n^3}$$

\vspace{3mm}
As you can see, our sum is equal to the even inverse cubes minus the odd ones. So, let\textsc{\char13}s find an expression for the odd ones
$$\sum_{n=1}^{\infty}\frac{1}{\left(2n+1\right)^3}=\sum_{n=1}^{\infty}\left(\frac{1}{n^3}-\frac{1}{\left(2n\right)^3}\right)$$
$$=\frac{7\zeta\left(3\right)}{8}$$

\vspace{3mm}
Therefore,
$$\sum_{n=1}^{\infty}\frac{\left(-1\right)^n}{n^3}=\sum_{n=1}^{\infty}\left(\frac{1}{\left(2n\right)^3}-\frac{1}{\left(2n+1\right)^3}\right)$$

\vspace{5mm}
Thus,
}
{\LARGE
$$I=\frac{-3\zeta\left(3\right)}{4}$$
}
\end{document}

Calculus/Jose Bedoya's questions/Integral with Riemann Zeta Function 2.tex


\section*{Ex.3.3}
\subsection*{Argue that if $h$ is strongly universal, then it is also universal.}

To be $c$-universal it must hold that $P(h(x) = h(y))\leq c/m$. To be strongly $c$-universal it must hold that $P(h(x) = h(y))\leq c^2/m^2$. Hence if we can show
$$\left(\frac{c}{m}\right)^2\leq\frac{c}{m}$$
the statement holds. As $\frac{c}{m}$ is the probability of a collision, the value must be in the interval [0,1]. Hence $\left(\frac{c}{m}\right)^2 < \frac{c}{m}$.


%-------------------------------Aspect Delta---------------------------------
\section{Aspect Delta}

Aspect $\delta$ is a dimensionless number defined as the smallest ratio of the
height of a vertex above its opposing triangle (see Figure~\ref{f:tet-height}) to
the square root of the area the triangle across all vertices of the tetrahedron.
\begin{figure}[bhp]
  \centering
  \includegraphics[width=1.5in]{tet-height}
  \caption{
    An illustration of the height $h_3$ of vertex 3.%
                                                          \label{f:tet-height}}
\end{figure}
In general, take $(i,j,k,\ell)$ to be a permutation of $\{0,1,2,3\}$
(i.e., $(i,j,k,\ell)\in\Sf$) and $\normvec{ L_{ab}}$ to be the length of the edge
connecting vertices $a$ and $b$.
Then aspect ratio $\delta$ may be written
\[
  q = \min_i\left\{C\frac{h_i}{\sqrt{A_{jk\ell}}}\right\}
\]
where $A_{jk\ell}$ is the area of the triangle opposite vertex $i$ and
$C = \frac{\sqrt[4]{108}}{4}\approx 0.805927$ chosen so that an equilateral tetrahedron has $q = 1$.
PATRAN~\cite{patran:03} also speaks of a ``normalized'' aspect ratio defined as
\[
  q_{alt} = 1 - q = 1 - \min_i\left\{C\frac{h_i}{\sqrt{A_{jk\ell}}}\right\}
\]
which is 0 for an equilateral tetrahedron.

\tetmetrictable{aspect $\delta$}%
{$1$}%                                                Dimension
{$[0.1,DBL\_MAX]$}%                                   Acceptable range
{$(0,DBL\_MAX]$}%                                     Normal range
{$[0,DBL\_MAX]$}%                                     Full range
{$1$}%                                                Equilateral tet
{\cite{patran:03}}%                                   Citation
{\nsup}%                                              Verdict function name


\section{Solving Poisson equation}

Poisson equation is relatively easy to solve in periodic
boundary condition using Fourier method (FFT). This situtation is different
from other discretization method, such as finite difference or Lagrange
function.

Poisson equation in $\mathbf{G}$-space:
\begin{equation}
-G^2 \tilde{V}_{\mathrm{Ha}}(\mathbf{G}) = -4\pi \tilde{\rho}(\mathbf{G})
\end{equation}

Hartree potential can be solve directly, for $\mathbf{G} \neq \mathbf{0}$:
\begin{equation}
\tilde{V}_{\mathrm{Ha}}(\mathbf{G}) = \frac{4\pi\tilde{\rho}(\mathbf{G})}{G^2}
\end{equation}

An example a program to solve Poisson equation is given in directory
{\tt poisson\_01}. In this program, a charge density is constructed
from difference between two Gaussian charge density. Total charge
(integrated charge density) is restricted to zero.
From this charge density, we calculate the electrostatic (Hartree) potential
by solving Poisson equation.

Function to generate vector {\tt dr}:

\begin{juliacode}
function gen_dr( r, center )
  Npoints = size(r)[2]
  dr = Array(Float64,Npoints)
  for ip=1:Npoints
    dx2 = ( r[1,ip] - center[1] )^2
    dy2 = ( r[2,ip] - center[2] )^2
    dz2 = ( r[3,ip] - center[3] )^2
    dr[ip] = sqrt( dx2 + dy2 + dz2 )
  end
  return dr
end
\end{juliacode}

Function to generate charge density:

\begin{juliacode}
function gen_rho( dr, σ1, σ2 )
  Npoints = size(dr)[1]
  rho = Array( Float64, Npoints )
  c1 = 2*σ1^2
  c2 = 2*σ2^2
  cc1 = sqrt(2*pi*σ1^2)^3
  cc2 = sqrt(2*pi*σ2^2)^3
  for ip=1:Npoints
    g1 = exp(-dr[ip]^2/c1)/cc1
    g2 = exp(-dr[ip]^2/c2)/cc2
    rho[ip] = g2 - g1
  end
  return rho
end
\end{juliacode}

Function to solve Poisson equation:

\begin{juliacode}
function solve_poisson( pw_grid::PWGrid, rho )
  Ω  = pw_grid.Ω
  G2 = pw_grid.G2
  Ns = pw_grid.Ns
  Npoints = pw_grid.Npoints
  ctmp = 4.0*pi*R_to_G( Ns, rho )
  for ip = 2:Npoints
    ctmp[ip] = ctmp[ip] / G2[ip]
  end
  ctmp[1] = 0.0
  phi = real( G_to_R( Ns, ctmp ) )
  return phi
end
\end{juliacode}


% mutually exclusive
\section{Mutually Exclusive Events, Disjoint Sets}
Two events $A$ and $B$ are mutually exclusive if $A\cap B= \emptyset$. That is, $A$ and $B$ are disjoint sets.

% independent
\section{Independent Events}
Interestingly, independence can, with the exception of specific symmetric cases, generally not be read off from a Venn diagram \cite{wasserman2013all}. A set of events $A_i$ is \textit{independent} if:

\begin{equation}
\mathbb{P}\left(\bigcap_i A_i \right) = \prod_i \mathbb{P}(A_i)
\end{equation}

\citeasnoun{wasserman2013all} writes independence using the coproduct symbol $A\coprod B$. Independence can be either assumed or it can be proven by verifying $\mathbb{P}(A\cap B) = \mathbb{P}(A)\mathbb{P}(B)$.

% conditional probability
\section{Conditional Probability}
Conditional probability $\mathbb{P}(A|B)$ "probability of $A$ given $B$" is the ratio of the probability measure applied to the subsets $A\cap B$ and $B$:

For $\mathbb{P}(B) > 0$:
\begin{equation}
\mathbb{P}(A|B) = \frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)}
\end{equation}

For dependent events, this can be interpreted as the fraction of $B$ that overlaps with $A$. For independent events, $\mathbb{P}(A|B) = \mathbb{P}(A)$.

% law of total expectation / tower rule
\section{Law of Total Probability}

Let $A_1,...,A_n$ be a partition of $\Sigma$ so that $\mathbb{P}(\bigcup_i A_i) = \mathbb{P}(\Omega) = 1$. Then:

\begin{equation}
\mathbb{P}(B) = \sum_i \mathbb{P}(B|A_i)\mathbb{P}(A_i)
\end{equation}

% Bayes Theorem
\section{Bayes' Theorem}
Let $A_1,...,A_n$ be a partition of $\Sigma$ so that $\mathbb{P}(\bigcup_i A_i) = \mathbb{P}(\Omega) = 1$ and $\mathbb{P}(A_i)>0$ for each $A_i$. Then:

\begin{equation}
\mathbb{P}(A_i|B) = \frac{\mathbb{P}(B|A_i)\mathbb{P}(A_i) }{\sum_j \mathbb{P}(B|A_j)\mathbb{P}(A_j) } = \frac{\mathbb{P}(B|A_i)\mathbb{P}(A_i) }{\mathbb{P}(B)}
\end{equation}

The theorem is so important that the different terms have names:

\begin{itemize}
\item $\mathbb{P}(A_i|B)$ is the posterior
\item $\mathbb{P}(B|A_i)$ is the likelihood
\item $\mathbb{P}(A_j)$ is the prior
\item $\mathbb{P}(B)$ is the evidence
\end{itemize}


\section{Change of Basis and Equivalent Matrices}
Consider vector spaces $V,W$ and $B=\{v_1,\ldots,v_n\}, B'=\{v_1',\ldots,v_n'\}$ bases of $V$, $C=\{w_1,\ldots,w_n\},C'=\{w_1',\ldots,w_n'\}$.
Let $\alpha:V\to W$ be a linear map.
We want to study the relationship between $[\alpha]_{B,C}$ and $[\alpha]_{B',C'}$.
\begin{definition}
    For a vector space $V$ with bases $B=\{v_1,\ldots,v_n\}, B'=\{v_1',\ldots,v_n'\}$, the change of basis matrix from $B'$ to $B$ is $P=(p_{ij})_{1\le i,j\le n}$ given by $p_{ij}=([v_j']_B)_i$.
\end{definition}
Indeed $P=[\operatorname{id}_V]_{B',B}$.
\begin{lemma}
    $[v]_B=P[v]_{B'}$.
\end{lemma}
\begin{proof}
    $P[v]_{B'}=[\operatorname{id}_V]_{B',B}[v]_{B'}=[\operatorname{id}_V(v)]_B=[v]_B$.
\end{proof}
\begin{remark}
    Let $P$ be the change of basis matrix from $B'$ to $B$, then $P$ is invertible and $P^{-1}$ is the change of basis matrix from $B'$ to $B$.
\end{remark}
So for the problem we stated at the start of this section, we write $P=[\operatorname{id}_V]_{B',B}$ and $Q=[\operatorname{id}_W]_{C',C}$, then
\begin{proposition}
    Let $A=[\alpha]_{B,C}$, $A'=[\alpha]_{B',C'}$ and $P,Q$ be as above, then $A'=Q^{-1}AP$.
\end{proposition}
\begin{proof}
    For any $v\in V$, we evaluate
    $$AP[v]_{B'}=[\alpha]_{B,C}[v]_B=[\alpha(v)]_C=Q[\alpha(v)]_{C'}=Q[\alpha]_{B',C'}[v]_B=QA'[v]_{B'}$$
    So $AP=QA'$, which means $A'=Q^{-1}AP$.
\end{proof}
\begin{definition}[Equivalent Matrices]
    Two matrices $A,A'\in M_{m,n}(F)$ are equivalent if $A'=Q^{-1}AP$ for some $Q\in M_{m,m},P\in M_{n,n}$ invertible.
\end{definition}
\begin{remark}
    As one can check, this defines an equivalent relation.
\end{remark}
\begin{proposition}\label{eqv_form}
    Let $V,W$ be vector spaces over $F$ and $\dim V=n,\dim W=m$.
    Let $\alpha:V\to W$ be linear.
    Then there exists bases $B$ of $V$ and $C$ of $W$ such that
    $$[\alpha]_{B,C}=\left( \begin{array}{c|c}
        I_r&0\\
        \hline
        0&0
    \end{array} \right)$$
    where $I_r$ is the identity matrix of dimension $r=n-n(\alpha)$.
\end{proposition}
\begin{proof}
    Fix a basis $v_{r+1},\ldots,v_n$ of $\ker\alpha$ and extend it to a basis $B=\{v_1,\ldots,v_r\}$.
    It is easy to see that $\alpha(v_1),\ldots,\alpha(v_r)$ gives a basis of $\operatorname{Im}\alpha$ as it is a spanning set that has the right size and that
    $$\sum_{i=1}^r\lambda_i\alpha(v_i)=0\implies \alpha\left( \sum_{i=1}^r\lambda_iv_i \right)=0\implies\sum_{i=1}^r\lambda_iv_i\in\ker\alpha\implies \forall i,\lambda_i=0$$
    Extend this to a basis $C$ of $W$, then $[\alpha]_{B,C}$ can be easily seen to have the right form.
\end{proof}
\begin{remark}
    This provides another proof of the Rank-Nullity Theorem.
\end{remark}
\begin{corollary}
    Any $m\times n$ matrix is equivalent to a matrix in the form illustrated in the preceding proposition.
\end{corollary}
\begin{proof}
    Immediate.
\end{proof}
\begin{definition}
    Let $A\in M_{m\times n}(F)$.
    The column rank $r(A)$ of $A$ is the dimension of the subspace spanned by the columns of $A$ in $F^n$.
    Similarly, the row rank of $A$ is the column rank of $A^T$.
\end{definition}
\begin{remark}
    If $\alpha$ is a linear map represented by $A$ with respect to some basis, then $r(\alpha)=r(A)$.
\end{remark}
\begin{proposition}
    Two matrices $A,A'$ of the same dimension are equivalent iff $r(A)=r(A')$.
\end{proposition}
\begin{proof}
    Direct consequence of Proposition \ref{eqv_form} and the preceding remark.
\end{proof}
\begin{theorem}
    $r(A)=r(A^T)$.
\end{theorem}
\begin{proof}
    Let $r=r(A)$, so there are some invertible $Q,P$ of the right sizes such that
    $$Q^{-1}AP=\left( \begin{array}{c|c}
        I_r&0\\
        \hline
        0&0
    \end{array} \right)$$
    which is an $m\times n$ matrix.
    But then
    $$P^\top A^\top (Q^{\top})^{-1}=P^\top A^\top(Q^{-1})^\top=(Q^{-1}AP)^\top=\left( \begin{array}{c|c}
        I_r&0\\
        \hline
        0&0
    \end{array} \right)$$
    as an $n\times m$ matrix.
    So $r(A^\top)=r=r(A)$.
\end{proof}
In the case where $\alpha:V\to V$ is an endomomrphism, the change of basis formula becomes $A'=P^{-1}AP$ where $P$ is the change of basis matrix and $A',A$ are the matrices of $\alpha$ in the two different bases.
This induces the following definition.
\begin{definition}
    Let $A,A'$ be square matrices.
    We say $A,A'$ are similar (or conjugate) if there is an $n\times n$ invertible square matrix $P$ such that $A'=P^{-1}AP$
\end{definition}
This notion is central to the study of diagonalisation and spectral theory.


\subsubsection{Integrals}
The definite integral of a function $f(x)$ from $x=a$ to $x=b$ where $a \leq b$ is the area between $f(x)$ and the x-axis bounded by the lines $x=a$ and $x=b$ where area above the x-axis is positive, and area below the x-axis is negative.
\begin{definition}
	\begin{equation*}
		\int_{a}^{b}{f(x) \mathrm{d}x} = \lim\limits_{h \to 0}{\sum_{n=1}^{\frac{b-a}{h}}}{f(a + (n-1)h) \cdot h}
	\end{equation*}
\end{definition}

We also define an indefinite integral, or antiderivative of $f(x)$, notated $F(x)$ where
\begin{equation*}
F'(x) = f(x) \implies \int{f(x)\mathrm{d}x} = F(x)
\end{equation*}
Note that there are infinitely many such functions $F$, since adding a constant to $F$ does not affect its derivative. To notate this, we add a constant $C$ to the indefinite integral. Given an initial condition for $f$, we can solve for $C$.\\

\noindent
Below are some properties of the integral. Let $f$ and $g$ be functions of $x$ and $p$, $a$, $b$, and $c$ where $a < b < c$, and $f$ and $g$ are continuous on the closed interval $[a,c]$.
\begin{enumerate}[label=]
	\item Linearity
	\begin{equation*}
		\int{(pf \pm g) \mathrm{d}x} = p\int{f \mathrm{d}x} \pm \int{g \mathrm{d}x}
	\end{equation*}
	\item Flipped Bounds
	\begin{equation*}
		\int_{a}^{b}{f \mathrm{d}x} = -\int_{b}^{a}{f \mathrm{d}x}
	\end{equation*}
	\item Union of Intervals
	\begin{equation*}
		\int_{a}^{b}{f \mathrm{d}x} + \int_{b}^{c}{f \mathrm{d}x} = \int_{a}^{c}{f \mathrm{d}x}
	\end{equation*}
	\item Power Rule
	\begin{equation*}
		\int{x^n \mathrm{d}x} = \frac{x^{n+1}}{n+1} + C \text{, }n \neq -1
	\end{equation*}
	\item U-Substitution
	\begin{equation*}
		\int{\left(f'\circ g\right) g' \mathrm{d}x} = f\circ g+ C
	\end{equation*}
	\item Integration by Parts
	\begin{equation*}
		\int{f' g \mathrm{d}x} = fg - \int{fg' \mathrm{d}x}
	\end{equation*}
	\item Fundamental Theorem of Calculus
	\begin{equation*}
		\dd{}{x}\int_{a}^{x}{f(s) \mathrm{d}s} = f(x)
	\end{equation*}
\end{enumerate}
Using the definition of the integral and the above rules, we can find the indefinite integral of some common functions.
\begin{enumerate}[label=]
	\item
	\begin{equation*}
	\int{\frac{1}{x} \mathrm{d}x} = \ln{\abs{x}} + C
	\end{equation*}
	\item
	\begin{equation*}
	\int{\sin{x} \mathrm{d}x} = -\cos{x} + C
	\end{equation*}
	\item
	\begin{equation*}
	\int{\cos{x} \mathrm{d}x} = \sin{x} + C
	\end{equation*}
	\item
	\begin{equation*}
	\int{\tan{x} \mathrm{d}x} = -\ln{\abs{\cos{x}}} + C
	\end{equation*}
\end{enumerate}


\section{Residue Theorem}
Today I want to talk about how to evaluate integrals using the residue theorem.

Let $z_0$ be an isolated singular point of $f(z)$.
Consider the value of the closed line integral $\oint_C f(z) dz$
around a simple closed curve $C$ surrounding $z_0$
but enclosing no other singularities.
The little circle means it's a closed line as opposed to an open line.
It means it comes back to where you started.
Let $f(z)$ be expanded in a Laurent series about $z=z_0$ that converges near
$z=z_0$.

Then,
\begin{align}
    f(z) = a_0 + a_1(z - z_0) + \cdots
    + \frac{b_1}{z - z_0} + \frac{b_2}{(z - z_0)^2} + \cdots.
\end{align}

\begin{question}
    What is converging near $z_0$?
\end{question}
It means converging in the neighbourhood around that point.
This is important because you could have several Laurent series about a point
that converge in different regions.
We want the series that converges near the point we're expanding about.

The point of the residue theorem is that there are an infinite number of $b$'s,
but $b_1$ is special.
Out of all the infinite terms, this $b_1$ is special.
This $b_1$ is called the \emph{residue}.
That one is special.

The terms in the $a$ series do not contribute to the integral,
because they are analytic.
The thing is, of the terms in the $b$ series,
only $b_1$ contributes.
This is because of this identity we see again and again.
\begin{align}
    \oint \frac{dz}{\left( z - z_0 \right)^n} =
    \begin{cases}
        2\pi i & \text{if } n =1\\
        0 & \text{otherwise}
    \end{cases}
\end{align}
It's easy to prove with a circle.
Just substitute $z=\rho e^{i\theta}$.
If the curve is not a circle,
first prove arbitrary curves give the same result
using Cauchy's theorem using a cut as we showed before.
Then if you know the results of the circle,
you know the result for arbitrary curves.
Anyway,
it's a homework problem and the solution will be provided at some point.

The point is,
that if you use this integral,
the $b_1$ is the only term that satisfies $n=1$.
All the other $b$s have an $n$ that is bigger than one and doesn't contribute.
The $a$'s don't contribute because they are analytic and Cauchy's theorem tells
you they don't contribute,
and of all the $b$'s only the $b_1$ contributes because of that formula.

So then, therefore
\begin{align}
    \oint_C f(z)\, dz = 2\pi b_1
\end{align}
and this $b_1$ is called the \emph{residue} of $f(z)$ at $z=z_0$.
It's the only term that survives this integration.
Any questions about all this?

So this particular result corresponds to when you have just one singular point
inside the contour $C$.
What if there are half a dozen,
what if there is more than one singularity inside that point?
Then what?

So we have to generalize this result.
We generalize to when there are multiple isolated singularities.
And we will see there is a very simple generalization of this thing.

Now let's say this is the region [picture]
and there are two isolated singular points enclosed by the contour $C$.
Let's call this point $z_1$ and this point $z_2$.
Now what you do is draw a circle around $z_1$
and another circle around $z_2$.
This contour $C$ goes counter clockwise.
Then we do what we did before.
We create a cut like this from $C$ to $C_1$ and a cut from $C$ to $C_2$.
Now the contour is deformed.
Follow the arrows and see you have a simply connected closed loop.

A couple of things I want you to realise.
This region in between $C$ and the circles,
that is simply connected.
If I put a rubber band anywhere,
I can shrink it to a point,
and I can apply Cauchy's theorem.

So around the deformed contour,
we have
\begin{align}
    \oint f(z)\, dz = 0
\end{align}
by Cauchy's theorem,
because we have cut out the singularities and it's analytic.
The contribution to the integral from the cut vanishes.
So then you can convince yourself using the logic very similar to our proof of
the Cauchy integral formula that
\begin{align}
    \oint_C f(z) \, &=
    \oint_{C_1} f(z)\,dz
    + \oint_{C_2} f(z)\, dz
\end{align}
In other words,
the contribution from the cuts vanishes,
but you can see that $C_1$ and $C_2$ are going clockwise,
but if you make everything counterclockwise,
the signs work out.
We are going around $C$, $C_1$ and $C_2$ counterclockwise
in the equation above.

And this generalizes no matter how many singularities you have.
It is the sum of the contour integral around each point.
But we just proved the integral around one singularity is just the residue times
$2\pi i$.
Each circle contains only one singular point
and we can immediately apply the result we just derived.
So it should be $2\pi i$ times the residue of each singular point.
And this is the residue theorem we're going to apply again and again.

So this is my claim.
This leads to the residue theorem
\begin{align}
    \oint_C f(z)\, dz &= 2\pi i\times
    \text{sum of residues of $f(z)$ from singular points inside $C$}
\end{align}
This is the thing that you guys have to remember.

\begin{question}
    That's the closed contour $C$?
\end{question}
Yes it's the closed contour $C$ that contains the isolated points.
Any questions?

The rest of this class and most of the rest of the next class will be giving
examples where we use the residue theorem to evaluate integrals.
This is a place where you use complex analysis a lot.
There are many integrals where the simplest way to do them is to use a contour
integral,
sometimes you know for example,
the most convenient way to express something is as a contour integral.
This result gets used a lot,
especially if you do theoretical physics,
you will see this over and over again.
Let's do some examples.

You should look at all the literature on ELMS
and try to work through the examples there.
Do several of them.
Unfortunately I can't cover the full range of possibilities.
As you'll see,
you have to be clever in most cases,
choose your contour cunningly,
hence it's helpful to do some examples so you can learn how to choose.
But time constraints.

Let's start with some very simple ones.
\begin{example}
    Integrate $f(z) = \sin(z)/z^4$ around the unit circle counterclockwise.
\end{example}
\begin{proof}[solution]
    This potentially has a pole at $z=0$.
    There's nowhere else where it might have a pole.
    The first thing to do is expand this in Taylor series.
    \begin{align}
        \frac{\sin(z)}{z^4} &=
        \frac{1}{z^4}\left\{
            z
            - \frac{z^3}{3!}
            + \frac{z^5}{5!}
            - \cdots
        \right\}
    \end{align}
    What is the order of the pole?
    It's order 3.
    What is the residue?
    It's $-1/6$.
    You can see this
    \begin{align}
        f(z) &=
        \frac{1}{z^3}
        - \frac{1}{6} \frac{1}{z}
        + \frac{z}{5!} - \cdots
    \end{align}
    Don't forget that negative sign,
    the residue is $-\frac{1}{6}$,
    not $\frac{1}{6}$.
    So by the residue theorem,
    \begin{align}
        \oint \frac{\sin z}{z^4}\,dz &=
        2\pi i \left( -\frac{1}{6} \right)\\
        &= -\frac{\pi i}{3}
    \end{align}
\end{proof}
Any questions?
OK let's do another example.

\begin{example}
    Integrate $\frac{4 - 3z}{z^2 - z}$
    around the circle $|z|=2$ counterclockwise.
\end{example}
\begin{proof}[solution]
    This is the circle of radius 2 centred at the origin.
    Look at this and tell me where it has poles.
    $z=0$ and $z=1$ where the denominator vanishes.
    The poles are where the denominator vanishes.
    And $0$ and $1$ are both inside the circle.
    And so we really have to use the residue theorem
    but first we need to find the residue at both $0$ and $1$.
    That's the procedure.

    The function has poles at $z=0$ and $z=1$.
    Near $z=0$,
    \begin{align}
        f(z) &=
        \frac{4 - 3z}{z(z - 1)}\\
        &= \frac{1}{z}\left\{
            \frac{4 - 3z}{z - 1}
        \right\}\\
        &= -\frac{4}{z} + \text{analytic terms near $z=0$}
    \end{align}
    The fraction in the bracket is analytic at $z=0$,
    and it's equal to $-4$.
    What's where it comes from.

    Near $z=1$,
    \begin{align}
        f(z) &=
        \frac{4 - 3z}{z(z - 1)}
        = \frac{1}{z - 1}\left\{
            \frac{4 - 3z}{z}
        \right\}
    \end{align}
    Near $z=1$, the things in the braces are analytic,
    so then the whole thing behaves like
    \begin{align}
        f(z) = \frac{1}{z - 1}
    \end{align}
    and hence $1$ is the residue.

    By the residue theorem,
    \begin{align}
        \oint f(z)\, dz &= 2\pi i\sum \text{residues}\\
        &= 2\pi i \left\{ -4 + 1 \right\}\\
        &= -6\pi i.
    \end{align}
    These are just some really simple examples to illustrate.
\end{proof}
Let's suppose this is $|z|=1/2$.
How would this change the answer?

If you draw it,
the circle would then have radius $1/2$,
and $z=1$ is outside the circle,
so then this $z=1$ residue wouldn't have contributed
and the result would be $-8\pi i$.
The boundary is not well defined for $|z|=1$.
You may have to take some limiting procedure,
and the answer may depend on how you take that limiting procedure.

Let's go on to the next problem.
\begin{example}
    Evaluate the definite integral
    \begin{align}
        I = \int_{0}^{2\pi} \frac{d\theta}{5 + 4\cos\theta}.
    \end{align}
\end{example}
\begin{proof}[Solution]
    Strictly speaking,
    you don't need complex analysis to do this integral,
    but it's a tough one.
    You'd have to use
    \begin{align}
        t = \tan\frac{\theta}{2}
    \end{align}
    but you would have had to know it.
    I'll show you it's easy with complex analysis.
    Here's what you do.
    There's all classes of problems where you integrate trig functions.
    The steps are going to be the same.

    Change variables to $z=e^{i\theta}$.
    Then $\theta$ goes from $0$ to $2\pi$
    around the unit circle in the complex plane.
    As $\theta$ goes from $0$ to $2\pi$,
    we go around the unit circle in the complex plane,
    which of course is $|z|=1$.

    So then, you have to change variables from $\theta$ to $z$.
    \begin{align}
        dz = i e^{i\theta} d\theta\qquad
        \implies\qquad
        \frac{1}{i} \frac{dz}{z} = d\theta.
    \end{align}
    And then
    \begin{align}
        \cos\theta = \frac{1}{2}\left(
            z + \frac{1}{z}
        \right).
    \end{align}
    To see this
    just use the Euler formula
    \begin{align}
        e^{i\theta} = \cos\theta + i\sin\theta.
    \end{align}
    So then the problem has reduced to evaluating
    \begin{align}
        I &=
        \oint \frac{1}{iz}
        \frac{1}{5 + 2\left( z + \frac{1}{z} \right)}\,dz\\
        &= \frac{1}{i}\oint \frac{dz}{2z^2 + 5z + 2}.
    \end{align}
    That's what you get when the dust settles.
    Now we have to figure out what the poles of this thing are.
    You have to find the roots first.
    I'm going to skip the step and claim it can be written like this.
    \begin{align}
        I &=
        \frac{1}{i}
        \oint
        \frac{dz}{(2z + 1)(z + 2)}.
    \end{align}
    If I draw this in the complex plane,
    it has a pole here at $z=-1/2$ and then it has a pole here at $z=-2$.
    And we want the contour integral of the unit circle
    centred at $z=0$.
    You can see that only one of the two poles is inside the circle.
    We're going counterclockwise because $\theta$ is increasing this way.
    $\theta$ is increasing from $0$ to $2\pi$ so we're going counterclockwise,
    which is good,
    so we don't have to put an extra sign in
    and just blindly apply the theorems.

    Only the pole at $z=-1/2$ contributes.
    Now let's calculate the residue.
    Let's look at this
    \begin{align}
        \frac{1}{(2z + 1)(z + 2)} &=
        \frac{1}{z + \frac{1}{2}}\left[
            \frac{1}{2(z + 2)}
        \right]\\
        &= \frac{1}{z + \frac{1}{2}}\left[
            \frac{1}{3}
        \right]
    \end{align}
    near $z=-1/2$,
    where we have evaluated the analytic part at $z=-1/2$.
    So $1/3$ is the residue,
    so
    \begin{align}
        I &= \frac{1}{i}2\pi i \frac{1}{3}\\
        &= \frac{2\pi}{3}
    \end{align}
\end{proof}

Any questions?
\begin{question}
    When evaluating the analytic part,
    can we ever just say this region is analytic and just plug in the value of
    $z$.
    Do we have to do the Taylor series?
\end{question}
You don't have to do the Taylor series,
only the first term matters.
This thing is non-zero,
but if it is zero,
the residue is zero.

We have to be a little bit careful.
Suppose if our function had a square in it
\begin{align}
    \frac{1}{(2z + 1)^2(z + 2)}
    = \frac{1}{\left( z + \frac{1}{2} \right)^2}\left[
    \frac{(z + \frac{1}{2})}{2(z + 2)}
    \right]
\end{align}
there is a higher order pole and you have to expand.
In physics problems you typically only have simple poles
and you don't have to expand.

You always want to make it a closed curve.

\begin{question}
    What happens if the integral is not from $0$ to $2\pi$?
\end{question}
Can we try $z=e^{i2\theta}$ for example if it's from $0$ to $\pi$.

\begin{question}
    Can you explain how you'd do it if it were a complex pole?
\end{question}
Let's say instead we had
\begin{align}
    \frac{1}{(2z + 1)^2 (z + 2)}
    = \frac{1}{\left(1 + \frac{1}{2} \right)^2}
    \frac{1}{4\left(z + 2 \right)}
\end{align}
Then you have to expand in a Taylor series,
so the coefficient of the next term in the Taylor series is what gives you the
residue.
In this situation with a higher order pole,
you need more terms in the Taylor series to work it out.

\begin{example}
    Evaluate
    \begin{align}
        I = \int_{-\infty}^{\infty}
        \frac{dx}{1 + x^2}
    \end{align}
\end{example}
\begin{proof}[Solution]
    If you were to use elementary methods,
    you would get
    \begin{align}
        \tan^{-1}(x)|_{-\infty}^{\inftyy}
        = \frac{\pi}{2}
        - \left( -\frac{\pi}{2} \right)
        = \pi.
    \end{align}
    Let's do it using complex analysis,
    just to show you the kind of contours you can integrate with.
    Consider the complex line integral
    \begin{align}
        \lim_{\rho\to\infty}
        \int_{-\rho}^{\rho}
        \frac{dz}{1 + z^2}
    \end{align}
    where the path is along the real line.
    This line integral is equal to $I$.
    Now let's draw this thing in the complex plane.
    Let's look at this integrand.
    It has poles where this $1+z^2$ vanishes.
    The denominator vanishes at $z=+i$ and $z=-i$.
    Right now the fact it has two poles is irrelevant for now.
    Consider this thing integrated not just along the real line,
    but back along a big semicircle in the upper half plane.

    Consider
    \begin{align}
        \oint \frac{dz}{1 + z^2}
    \end{align}
    integrated over the closed contour shown in the figure,
    which consists of a semicircle over the upper half plane,
    in addition to the integral over real line.

    This closed line integral we can evaluate using the residue theorem.
    Even though it has two poles,
    only $z=+i$ is inside the contour.
    Let's evaluate it.
    Close to $z=i$,
    \begin{align}
        \frac{1}{1 + z^2} = \frac{1}{(z + i)(z - i)}
        = \frac{1}{z - 1} \underbrace{\frac{1}{2i}}_{\text{residue}}
        + \text{analytic function}
    \end{align}
    Hence by the residue theorem, the contour integral is
    \begin{align}
        \oint \frac{dz}{1 + z^2} &=
        2\pi i \frac{1}{2i} = \pi.
    \end{align}
    We have evaluated this integral not over the real line,
    but over this closed contour which includes the real line,
    but also includes the contribution from this big semicircle.
    I'm going to show you the contribution from the big semicircle vanishes as
    you take $\rho\to\infty$.
    And so this closed integral is actually equal to the integral we want.
    That's going to be the logic here.

    We need to prove the line integral over the semicircle vanishes and then
    we're done.
    Let's prove that.
    \begin{align}
        \underbrace{\oint \frac{dz}{1 + z^2}}_{\pi} &=
        \underbrace{\int_{\text{real line}} \frac{dz}{1 + z^2}}_{I}
        + \underbrace{\int_{\text{semicircle}} \frac{dz}{1 + z^2}}_{?}
    \end{align}
    Set $z=\rho e^{i\theta}$.
    Then consider
    \begin{align}
        \int_{\text{semicircle}} \frac{dz}{1 + z^2} &=
        \int_{0}^{\pi} \frac{i\rho e^{i\theta}}{1 + \rho^2 e^{2i\theta}}\,
        d\theta
    \end{align}
    This has $\rho$ in the numerator and $\rho^2$ in the denominator,
    this should go to zero as $\rho\to\infty$.
    But you still have to be careful because it's complex,
    but you'll see this intuition is correct.
    It's just going to vanish.
    \begin{align}
        \left|
            \int_{\text{semicircle}} \frac{dz}{1 + z^2}
        \right|
        &=
        \left|
            \int_{0}^{\pi} \frac{i\rho e^{i\theta}}{1 + \rho^2 e^{2i\theta}}\,
            d\theta\\
        \right|
        &\le
        \int_{0}^{\pi}
        \left|
            \frac{i\rho e^{i\theta}}{1 + \rho^2 e^{2i\theta}}
        \right|
        \left|
            d\theta
        \right|\\
        &\le
        \int_{0}^{\pi}
        \frac{\rho}{\rho^2} |d\theta|\\
        &=
        \frac{\pi}{\rho}\\
        &\to 0
    \end{align}
    as $\rho\to\infty$.
    So in the end the contribution of this semicircle vanishes.
\end{proof}

This shows you can do an integral over the real line by doing an integral like
this with a contour like this.
This is one of the very standard contours where you have an integral over the
real line and make it a contour by closing the upper half plane,
or lower half plane.

\begin{question}
    What if you took the semicircle in the lower half plane?
\end{question}
Everything would work, but you're not going counterclockwise and you have to
deal with signs.
It still works though.
There's more than one way to do these problem.
Choose the most convenient one.
As you do more problems, you'll see.
I want you guys to convince yourselves that it can be closed in the negative
half plane and get the same result.

This contour works for this function works for a lot of functions,
but it also doesn't work for a lot of functions.
You could have used either up or down in this problem,
but we're going to look at at problem where you have to use one of them.
It's actually more common.

\begin{example}
    Evaluate
    \begin{align}
        I = \int_{0}^{\infty} \frac{\cos(z)}{1 + x^2}\,dx
    \end{align}
\end{example}
\begin{proof}[Solution]
    Notice that it's similar to before.
    Notice that the integrand is even, so this is equal to
    \begin{align}
        I &=
        \frac{1}{2}\int_{-\infty}^{\infty}
        \frac{\cos(x)}{1 + x^2}\,dx.
    \end{align}
    It's an even function so we can be a bit clever.
    We want to make this an integral over some line in the complex plane.
    There's more than one way to do this,
    but here's the trick.
    Write $\cos z = \Re e^{iz}$.
    Then write
    \begin{align}
        I &= \Re\left\{
        \frac{1}{2}
        \int_{-\infty}^{\infty}
        \frac{e^{iz}}{1 + z^2}\,
        dz
        \right\}
    \end{align}
    where the integral is over the real line in the complex plane.
    Now it's beginning to look a lot like the problem we just saw.
    You have a function integrating over the whole real line.
    It has singularities at exactly the same points $z=\pm i$.
    But can we use the same contour?

    The integrand is a bit different.
    Can we convince ourselves that the contribution from this semicircle
    vanishes?
    If it does, then we can use the same contour.
    It turns out it does vanish.

    Along that big semicircle, $z$ has a real and imaginary part
    \begin{align}
        z = \rho_R + i\rho_I
    \end{align}
    Unless you're at $+\rho$ or $-\rho$,
    the imaginary part is infinite.
    If the imaginary part is positive and infinite,
    $e^{iz}$ because $e$ raised to minus infinity,
    so it goes to zero exponentially quickly
    if you're in the upper half plane.
    That means this integrand vanishes incredibly quickly in the upper half
    plane.
    So we can use exactly the same contour as we used before
    and use the residue theorem.
    At the same time,
    we can't use a contour with a semicircle in the lower half plane,
    because it blows up incredibly quickly.
    Last time we had a choice of contours up or down,
    but here we don't have a choice,
    at least we don't have that choice,
    you have to choose to go in the upper half.
\end{proof}


\chapter{Bit Manipulation}
\section{Concepts}
\subsection{Basics}
\begin{enumerate}
\item Bit value: bit0, bit1.
\item BitSet/Bits
\item Bit position (bit interchangeably)
\item 32-bit signed range: $[-2^{31}, 2^{31}-1]$. $0$ is like positive number without complement.
\end{enumerate}
\subsection{Operations}
\runinhead{Mask.}
\begin{enumerate}
\item Masking to 1: to mask a single bit position, $bit\OR 1$
\item Masking to 0: to mask a single bit position, $bit\AND 0$
\item Querying a bit position value: to query a single bit position, $bit\AND 0010$
\item Toggling bit values: to toggle a single bit position, $bit\XOR 1$
\end{enumerate}}
This can be extended to do masking operations on multiple bits.

\runinhead{Check 2's power}
$$x\AND(x-1)$$

\runinhead{Rightmost bit set.} To get the rightmost bit, with the help of 2's complement:
\begin{enumerate}
\item Left extended with 1's:
$$x \XOR (-x)$$
\item Left extended with 0's:
$$x \AND (-x)$$
\end{enumerate}

\runinhead{Negation and index} We can use tilde notation for the index accessing a string or an array
\begin{lstlisting}
i  ~i
0  -1
1  -2
2  -3
3  -4
4  -5
5  -6
\end{lstlisting}
$$
\NOT i = -i+1
$$
To determine whether a string is palindrome:
\begin{python}
def is_palindrome(s):
  return all(s[i] == s[~i] for i in xrange(len(s)/2))
\end{python}
\section{Single Number}
\subsection{Three-time appearance}
Given an array of integers, every element appears three times except for one. Find that single one.

\rih{Using list.} Consider 4-bit numbers:
\begin{eqnarray*}
&& 0000 \\
&& 0001 \\
&& 0010 \\
&& ... \\
&& 1111
\end{eqnarray*}

Add (not $\&$) the bit values \textbf{vertically}, then result would be $abcd$ where $a, b, c, d$ can be any number, not just binary. $a, b, c, d$ can be divided by 3 if the all element appears three times. Until here, you can use a list to hold $a, b, c, d$. By mod 3, the single one that does not appear 3 times is found.

To generalize to 32-bit \pythoninline{int}, use a list of length 32.

\rih{Using bits.}
To further optimize the space, use bits (bit set) instead of list.
\begin{itemize}
\item Since all except one appears 3 times, we are only interested in $0, 1, 2$ (mod 3) count of bit1 appearances in a bit position.
\item We create 3 bit sets to represent $0, 1, 2$ appearances of all positions of bits.
\item For a bit, there is one and only one bit set containing bit1 in that bit position.
\item Transition among the 3 bit sets for every number:
$$
bitSet^{(i)} = (bitSet^{(i-1)}\AND num)\OR(bitSet^{(i)}\AND \NOT num)
$$
\end{itemize}

For $i$ appearances, the first part is the bit set \textbf{transited from} $(i-1)$ appearances, and the second part is the bit set \textbf{transited out} from itself.

Consider each single bit separately. For the $j$-th bit in $num$, if $num_j=1$, the first part indicates $bitSet^{(i-1)}$ will transit in (since transition); the 2nd part is always 0 (since transition out or initially 0). If $num_j=0$, the 1st part is always 0 (since no transition); the 2nd part indicates $bitSet^{(i)}$ will remain the same (since no transition).

\subsection{Two Numbers}
Given an array of numbers nums, in which exactly two elements appear only once and all the other elements appear exactly twice. Find the two elements that appear only once.

\begin{itemize}
\item Easily get: $x = a \XOR b$.
\item $a \neq b$; thus there are at least one 1-bit in $x$ is different.
\item Take an arbitrary 1 bit set in $x$, and such bit set can classify the elements in the array into two separate groups.
\end{itemize}

\section{Bitwise operators}
\runinhead{Comparison.} Write a method which finds the maximum of two numbers $a, b$. You should not use if- else or any other comparison operator
\\
Clues:
\begin{enumerate}
\item check the sign bit $s$ of $a-b$.
\item return $a-s*(a-b)$
\end{enumerate}
Codes:
\begin{java}
int getMax(int a, int b) {
    int c = a - b;
    int k = (c >> 31) & 0x1;
    int max = a - k * c;
    return max;
}

\end{java}
If consider overflow, it raises another level of difficulty.


\subsection{Ordinary Differential Equations}
\begin{definition}
	An ordinary differential equation is an equation involving a function of a single variable and its derivatives.
\end{definition}
\begin{example}
	\vspace{-\baselineskip}
	\begin{flalign*}
		& y' = y & \\
		& y' = y^2 - t \\
		& y'' + y' = ye^t
	\end{flalign*}
\end{example}
\subsection{Normal Forms}
\begin{definition}
	A first order equation $y' = f\p{t, y}$ is said to be in normal form. More generally, $y^{(n)} = f\p{t, y^{(1)}, y^{(2)}, \ldots, y^{(n-1)}}$ is in normal form.
\end{definition}
\begin{example}
The normal form of $y' + 2ty = 0$ is given by $y' = -2ty$.
\end{example}
\par\bigskip
Differential equations can have many solutions. In the above equation, all solutions are in the form $y\p{t} = Ce^{-t^2}$, so it is called the \textit{general solution} to the above ODE.
\begin{definition}
	The general solution is the family of solutions depending on some parameters that give all but a finite number of solutions.
\end{definition}

\subsection{Initial Value Problems}
\begin{definition}
	A first order ODE $y' = f\p{t, y}$ together with the initial value $y\p{t_0} = y_0$ is called an initial value problem (\textrm{abbreviated as IVP}).
\end{definition}
A solution to an initial value problem satisfies both the associated ODE and the initial condition.

From the above example, if we had the initial condition $y\p{0} = 2$, then the solution to the initial value problem would be $y\p{t} = 2e^{-t^2}$.

\pagebreak
\subsection{Interval of Existence}
\begin{definition}
	Suppose $y = y\p{t}$ satisfies an initial value problem with initial condition $y\p{t_0} = y_0$. Then the interval of existence is the largest interval containing $t = t_0$ on which $y\p{t}$ is defined.
\end{definition}
\begin{example}
	\vspace{-0.5\baselineskip}
	\begin{flalign*}
		&
		\begin{cases}
			y' = y^2 \\
			y\p{0} = 2
		\end{cases}
		&
	\end{flalign*}
	Solution: $\displaystyle y\p{t} = \frac{1}{\frac{1}{2} - t}$

	The interval of existence for this IVP is $\p{-\infty, \, \frac{1}{2}}$ (as opposed to the set $\mathbb{R} - \left\{ \frac{1}{2} \right\}$, which is not connected).
\end{example}

\subsection{Geometric Meaning}
Consider $y' = f\p{t, y}$ with $y\p{t_0} = y_0$. If $y\p{t}$ is a solution to the problem, then for any $t$, we must have $\underbrace{y'\p{t}}_{\mathclap{\substack{\text{Slope of } \\ \text{tangent line at }t.}}} = f\p{t, y\p{t}}$.

\subsection{Direction Fields}
\begin{definition}
	A direction field is a plot of line segments, where each one is centered at a point $\p{t, y}$ whose slope is given by $f\p{t, y}$.
\end{definition}
\textit{Example:} $y' = y$
\par
\begin{tikzpicture}
	\draw[<->, black] (-4, 0) -- (4, 0) node[pos = 1, right]{$t$};
	\draw[<->, black] (0, -2) -- (0, 3) node[pos = 1, above]{$y$};
	\foreach \x in {-3.5, -3, ..., 3.5}
		\foreach \y in {-1.5, -1, ..., 2.5}
		{
			\draw[\slopecolor] (\x, \y) -- +({atan(\y)}:-0.125);
			\draw[\slopecolor] (\x, \y) -- +({atan(\y)}:0.125);
		}
	\draw[<->, red, thick, domain = -3.5:1] plot (\x, {exp(\x)});
	\node[circle, inner sep = 1pt, fill = black, label = right:{$\p{0, 1}$}] at (0, 1) {};
\end{tikzpicture}
\par
These are useful for finding numerical solutions to equations we can't solve explicitly.

\subsection{Qualitative Methods}
\textit{Example:} $y' = 1 - y^2$
\par
\begin{figure}[H]
	\begin{minipage}[t]{0.5\textwidth}
		\centering
		\begin{tikzpicture}[scale = 0.75]
			\draw[<->, black] (-4, 0) -- (4, 0) node[pos = 1, right] {$t$};
			\draw[<->, black] (0, -3) -- (0, 3) node[pos = 1, above] {$y$};
			\foreach \x in {-3.5, -3, ..., 3.5}
				\foreach \y in {-2.5, -2, ..., 2.5}
				{
					\draw[\slopecolor] (\x, \y) -- +({atan(1 - \y * \y)}:-0.125);
					\draw[\slopecolor] (\x, \y) -- +({atan(1 - \y * \y)}:0.125);
				}
			\draw[<->, dashed, thick, red] (-4.25, 1) -- (4.25, 1) node[pos = 1, right] {$y\p{t} = 1$};
			\draw[<->, dashed, thick, red] (-4.25, -1) -- (4.25, -1) node[pos = 1, right] {$y\p{t} = -1$};
			\draw[thin] (5.2, 1.5) .. controls (5.2, 1.8) and (5.5, 2.1) .. (5.5, 2.5) node[pos = 1, above, align = center] {\small Equilibrium \\ \small solutions};
		\end{tikzpicture}
		\caption{Direction Field}
	\end{minipage}%
	\begin{minipage}[t]{0.5\textwidth}
		\centering
		\begin{tikzpicture}[scale = 1.5]
			\draw[<->, black] (-2, 0) -- (2, 0) node[pos = 1, right] {$y$};
			\draw[<->, black] (0, -1.5) -- (0, 1.5) node[pos = 1, above] {$y'$};
			\draw[thin, gray] (-1, -0.25) -- (-1, 0.25) node[pos = 0, below, black] {-1};
			\draw[thin, gray] (1, -0.25) -- (1, 0.25) node[pos = 0, below, black] {1};
			\draw[<->, red, thick] plot[domain = -1.5:1.5] (\x, 1 - \x * \x);
		\end{tikzpicture}
		\caption{Phase Portrait}
	\end{minipage}
\end{figure}
\begin{minipage}[t]{0.5\textwidth}
	Solution classifications:
	\begin{enumerate}[label = \arabic*.]
		\item Equilibrium solution: $y\p{t} = 1$, $y\p{t} = -1$
		\item $\phantom{|} y\p{0} \phantom{|} > 1$: \quad
			\begin{tikzpicture}[scale = 0.35, baseline = -0.8ex]
				\draw[red] (1.25, -0.75) .. controls (-1.3, -0.65) .. (-1.5, 0.5);
			\end{tikzpicture}
		\item $|y\p{0}| < 1$: \quad
			\begin{tikzpicture}[scale = 0.35, baseline = -0.8ex]
				\draw[red] (1.25, 0.5) .. controls (-0.25, 0.5) and (-0.25, -0.75) .. (-1.5, -0.75);
			\end{tikzpicture}
		\item $\phantom{|} y\p{0} \phantom{|} < 1$: \quad
			\begin{tikzpicture}[scale = 0.35, baseline = -0.8ex]
				\draw[red] (1.25, -0.75) .. controls (0.9, 0.5) .. (-1.5, 0.5);
			\end{tikzpicture}
	\end{enumerate}
\end{minipage}%
\begin{minipage}[t]{0.5\textwidth}
	\begin{tikzpicture}[baseline = (current bounding box.north)]
		\draw[<->, black] (-4, 0) -- (4, 0) node[pos = 1, right] {$t$};
		\draw[<->, black] (0, -3) -- (0, 3) node[pos = 1, above] {$y$};
		\foreach \x in {-3.5, -3, ..., 3.5}
			\foreach \y in {-2.5, -2, ..., 2.5}
			{
				\draw[\slopecolor] (\x, \y) -- +({atan(1 - \y * \y)}:-0.125);
				\draw[\slopecolor] (\x, \y) -- +({atan(1 - \y * \y)}:0.125);
			}
		\draw[<->, dashed, thick, red] (-4.25, 1) -- (4.25, 1) node[pos = 1, right] {};
		\draw[<->, dashed, thick, red] (-4.25, -1) -- (4.25, -1) node[pos = 1, right] {};
		\draw[<->, thick, red] plot[domain = -0.125:4] (\x, {(3 * exp(2 * \x) + 1) / (3 * exp(2 * \x) - 1)});
		\node[circle, inner sep = 1pt, fill = black, label = right:{$y\p{0} = 2$}] at (0, 2) {};
		\draw[<->, thick, red] plot[domain = -4:4] (\x, {(exp(2 * \x) - 1) / (exp(2 * \x) + 1)});
		\node[circle, inner sep = 1pt, fill = black, label = below right:{$y\p{0} = 0$}] at (0, 0) {};
		\draw[<->, thick, red] plot[domain = -4:0.125] (\x, {(exp(2 * \x) + 3) / (exp(2 * \x) - 3)});
		\node[circle, inner sep = 1pt, fill = black, label = right:{$y\p{0} = -2$}] at (0, -2) {};
	\end{tikzpicture}
\end{minipage}


\subsection{Factoring Polynomials}

We want to break up a polynomial like $f(x) = a_0 + a_1x^1 + \ldots a_nx^n$ into linear factors so that $f(x) = c(x-b_1)\cdot \ldots \cdot(x - b_n)$. This form makes it simple to see that the roots of $f$, solutions to $f(x) = 0$, are $x = b_1 \ldots b_n$.\\

For quadratics, $f(x) = ax^2 + bx + c$, there exists a simple formula that will give us both roots, the quadratic formula
\begin{equation*}
	x = \frac{-b \pm \sqrt{b^2-4ac}}{2a}.
\end{equation*}

We can see that when $b^2 - 4ac < 0$, like for $f(x) = x^2 + 5x + 10$, we will get complex roots $\alpha \pm \beta i$. For any polynomial, these roots come in pairs, so if $\alpha + \beta i$ is a root, then so is $\alpha - \beta i$. This means that every conjugate pair $\alpha \pm \beta i$ has a quadratic equation with those roots. Sometimes we will not factor quadratics with complex roots into linear terms.\\

Although there do exist explicit formulas for finding roots for cubic (degree 3) and quartic (degree 4) equations, they are too long and not useful enough to memorize. When working by hand, we instead use other tricks to find roots.\\

There are a few useful tricks that can help. If the polynomial doesn't have a constant term, then 0 is a root. If all the coefficients sum to 0, then 1 is a root. For certain polynomials with an even number of terms, like all cubics of the form $ax^3 + bx^2 + cax + cb$ we can factor out a term from the first two and last two terms to get $x^2(ax+b)+c(ax+b) = (ax+b)(x^2+c)$. For other polynomials, we might just try guessing and checking values. However, we need a more efficient way that works in general.\\

Since we are looking to find linear factors $f(x) = c(x-b_1)\cdot \ldots \cdot(x-b_n)$, we can see that the constant term in the polynomial is the product of the roots $b_1 \ldots b_n$. In fact, since the coefficients of polynomials are completely determined by the roots and the leading coefficient, all the coefficients are sums and products of roots. You might remember when factoring quadratics that the coefficient of $x$ term is the sum of the two roots. These rules are called Vieta's formulas.\\

So, if we have the constant term, we can check all of its integer factors to see if any are roots. For each root, we can divide, using a technique like synthetic division, to continue finding the rest of the roots. This method is especially useful on tests because the roots tend to be integers.

\input{./backgroundReview/algebraPreCalc/factoringPolynomials_example.tex}


\sec{Exact ODEs}
\subsection{Introduction}
\begin{defn}[Exact ODEs]
	A first order ODE $M(x, y) + N(x, y)y' = 0$ is called exact if there is a function $u(x, y)$ such that
	\[\dfrac{\partial u}{\partial x} = M \text{ and } \dfrac{\partial u}{\partial y} = N.\]
\end{defn}
\begin{mdframed}[style=boxstyle, frametitle={Identifying such an ODE}]
If the functions ($M$ and $N$) along with the domain are ``good enough'', then the ODE is exact if and only if
\[\dfrac{\partial M}{\partial y} = \dfrac{\partial N}{\partial x}.\]
\end{mdframed}
\newpage
\begin{mdframed}[style=boxstyle, frametitle={The Solution}]
Given any such scalar function $u(x, y)$ as mentioned above, the solution is then given by
\[u(x, y) = c.\]
Thus, the question now reduces to finding such a $u(x, y).$\\
This function can be found either via inspection or via the following method:
\begin{enumerate}[leftmargin = *, label = \Roman*:]
	\item Integrate $\dfrac{\partial u}{\partial x} = M(x, y)$ with respect to $x$ to get
	\[u(x, y) = \int M(x, y) dx + k(y),\]
	where $k(y)$ is a constant of integration.
	\item To determine $k(y),$ differentiate the above equation in Step I with respect to $y,$ to get:
	\[\frac{\partial u}{\partial y} = k'(y) + \frac{\partial}{\partial y}\left(\int M(x, y) dx\right).\]
	As the given ODE is exact, the LHS is simply $N(x, y).$ We rearrange this to get
	\[k'(y) = N(x, y) - \frac{\partial}{\partial y}\left(\int M(x, y) dx\right).\]
	This can now be used to determine $k(y)$ and hence, $u.$\\
	\emph{Remark.} Even though the RHS \emph{looks} like a function of $x$ and $y$ both, it will simplify to just a function of $y.$ This happens precisely because $u$ is exact to begin with.
\end{enumerate}
\end{mdframed}
\exercise{
	Solve the following ODEs.
	\begin{enumerate}[leftmargin=*]
		\item $(2x + y^2) + 2xy y' = 0.$
		\item $(y\cos x + 2xe^y) + (\sin x + x^2 e^y - 1)y' = 0.$
	\end{enumerate}
}
%
\newpage
\subsection{Integrating factors}
\begin{defn}[Integrating factor]
Consider the scenario where we have the ODE
\[M(x, y) + N(x, y)y' = 0\]
and $M_y \neq N_x,$ id est, the ODE is not exact.\\~\\
Sometimes, we may find a scalar function $\mu(x, y)$ such that
\[\mu(x, y)M(x, y) + \mu(x, y)N(x, y)y' = 0\]
is exact, id est,
\[(\mu M)_y = (\mu N)_x.\]
Such a function $\mu(x, y)$ is called an \defin{integrating factor} of the original ODE.
\end{defn}

\begin{mdframed}[style=boxstyle, frametitle={Actually solving it}]
In practice, we usually try to find an integrating factor which is only a function of $x$ (or $y$). In this case, we have $\mu_y = 0$ (or $\mu_x = 0,$ resp.).\\
In the respective cases, the equations simplify to:
\begin{enumerate}[leftmargin=*]
	\item $\dfrac{d \mu}{dx} = \left(\dfrac{M_y - N_x}{N}\right)\mu.$\\~\\
	Thus, our assumption that $\mu$ is just a function of $x$ is valid precisely when the term in the bracket is independent of $y.$\\
	Similarly, the other case gives us:
	\item $\dfrac{d \mu}{dy} = \left(\dfrac{N_x - M_y}{M}\right)\mu.$
\end{enumerate}
\end{mdframed}
\exercise{%
	Solve the following ODEs.
	\begin{enumerate}[leftmargin=*]
		\item $(8xy - 9y^2) + (2x^2 - 6xy)y' = 0.$
		\item $-y + xy' = 0.$
	\end{enumerate}
}


%!TEX root = ../TTK18-Summary.tex
\section{Bi-level programming in constrained control}
A bi-level program is an optimization problem that depends on the solution of another optimization problem, called ``upper level'' (UL) and ``lower level'' (LL) problems, respectively.
%
\begin{equation}\label{eq:bilevel-general}
  \begin{split}
    \min_x f\sub{UL}(x,z) \\
    G\sub{UI}(x,z) \leq 0 \\
    G\sub{UE}(x,z) = 0 \\
    z = \arg \min_z f\sub{LL}(x,z) \\
    G\sub{LI}(x,z) \leq 0 \\
    G\sub{LE}(x,z) = 0
  \end{split}
\end{equation}

\subsection{Restricting problem classes}
The formulation \eqref{eq:bilevel-general} is very general. Must restrict the problem in order to solve efficiently, by assuming:
%
\begin{itemize}
  \item LL problem is convex and regular (TODO: what is regular?),
  \item linear constraints, and
  \item linear or quadratic objective functions.
\end{itemize}
%
Then we can replace the LL problem with its KKT conditions. The Lagrangian function is
%
\begin{equation}
  \mathcal{L}(x,z,\lambda,\mu) = f\sub{LL}(x,z) + \lambda^T G\sub{LI}(x,z) + \mu^T G\sub{LE}(x,z)
\end{equation}
%
and the KKT conditions are
%
\begin{equation}\label{eq:kkt-lower-level}
  \begin{split}
    \nabla_z \mathcal{L}(x,z,\lambda,\mu) &= 0 \\
    G\sub{LI}(x,z) &\leq 0 \\
    G\sub{LE}(x,z) &= 0 \\
    \lambda &\geq 0 \\
    \lambda \hadamard G\sub{LI}(x,z) &= 0
  \end{split}
\end{equation}
%
where $\hadamard$ is the entrywise product. Line by line, the KKT conditions can be interpreted as
%
\begin{itemize}
  \item the Lagrangian does not change,
  \item the inequality constraints hold,
  \item the equality constraints hold,
  \item the Lagrange multipliers are nonnegative,
  \item each inequality constraint is either active, or its Lagrange multiplier is zero.
\end{itemize}
%
The overall problem is then
%
\begin{equation}\label{eq:bilevel-kkt}
  \begin{split}
    \min_{x,z,\lambda,\mu} f\sub{UL}(x,z) & \\
    G\sub{UI}(x,z) & \leq 0 \\
    G\sub{UE}(x,z) &= 0 \\
    \nabla_z \mathcal{L}(x,z,\lambda,\mu) &= 0 \\
    G\sub{LI}(x,z) &\leq 0 \\
    G\sub{LE}(x,z) &= 0 \\
    \lambda &\geq 0 \\
    \lambda \hadamard G\sub{LI}(x,z) &= 0 \\
  \end{split}
\end{equation}
%
which is solvable for smaller problems, using e.g. YALMIP.

\subsection{Big-M notation}
Big-M formulation replaces nonlinear complementarity constraints\footnote{In \eqref{eq:kkt-lower-level} and \eqref{eq:bilevel-kkt}, the complementarity constraint is $\lambda \hadamard G\sub{LI}(x,z) = 0$, because it is a product of two variables.} with linear constraints using binary variables $s \in \{0,1\}$ to indicate activeness for inequality constraints:
%
\begin{equation}\label{eq:big-M}
  \begin{split}
    G\sub{LI}(x,z) &\leq 0 \\
    G\sub{LI}(x,z) &\geq -M^u(1-s) \\
    \lambda        &\geq 0 \\
    \lambda        &\leq M^\lambda s
  \end{split}
\end{equation}
%
This notation fulfills the complementarity constraints, and given large enough $M^u$ and $M^\lambda$, the solution is unchanged. When $s = 1$ we get $G\sub{LI}(x,z) = 0$ (inequality constraint active), and when $s = 0$ we get $\lambda = 0$ (inequality constraint inactive).

The overall problem formulation becomes
%
\begin{equation}
  \begin{split}
    \min_{x,z,\lambda,\mu,s} f\sub{UL}(x,z) \\
    G\sub{UI}(x,z) & \leq 0 \\
    G\sub{UE}(x,z) &= 0 \\
    \nabla_z \mathcal{L}(x,z,\lambda,\mu) &= 0 \\
    G\sub{LI}(x,z) &\leq 0 \\
    G\sub{LE}(x,z) &= 0 \\
    \lambda &\geq 0 \\
    G\sub{LI}(x,z) &\geq -M^u(1-s) \\
    \lambda &\leq M^\lambda s \\
    s &\in \{0,1\}
  \end{split}
\end{equation}

Linear $f\sub{UL}$ gives a MILP\footnote{Mixed integer linear program.}, quadratic $f\sub{UL}$ gives a MIQP\footnote{Mixed integer quadratic program.}. Nonconvex, $np$-hard, but efficient software exists.

\subsection{Solving MILP/MIQP}
Some steps can be taken to make MILP/MIQPs easier to solve.

\subsubsection{Branch-and-bound}
A branch-and-bound solver partitions the search space into regions, and finds an upper bound $UB$ and lower bound $LB$ for the solution in each region. If
%
\begin{equation}
  LB_i > UB_j
\end{equation}
%
then we know the solution is not in $i$, and we discard that region.

You can also remove known symmetries from symmetric problems by adding extra constraints to make only one of the symmetric solutions feasible.

\subsubsection{Restrict combinations of binary variables}
The binary variables indicate which constraints are active, and some combinations of active constraints are known to be impossible. Linear constraints on the binary variables can remove these combinations from the search space.

\subsubsection{Use small $M^u$/$M^\lambda$}
For a correct analytical solution, $M^u$ and $M^\lambda$ must be sufficiently large. However, setting them too large gives numerical issues and an inaccurate solution.

A good $M^u$ can sometimes be found by solving a series of LPs, if the LL constraints are bounded.

$M^\lambda$ may need trial-and-error: If a value of $\lambda$ is constrained by $M^\lambda$, retry after increasing the corresponding element of $M^\lambda$.

This is made easier by setting $M^u$/$M^\lambda$ diagonal and positive.

\subsection{Approximate explicit MPC}
Explicit MPC means to solve the optimization problem of an MPC offline, so that the solution can be calculated quickly and explicitly online. Bilevel optimization can be used for this.
%
\begin{itemize}
  \item Find a set of points $V$ whose convex hull describes the feasible region of the MPC.
  \item Solve MPC problem for each such point in the state space.
  \item Partition the convex hull of $V$ into simplices with triangulation.
  \item Use linear interpolation in each simplex $S_i$ to define a piecewise affine state feedback
  \begin{equation}
    u = K^i x + k^i,\quad x \in S_i
  \end{equation}
  which will be feasible and continuous.
  \item Further subdivide triangulation if $\norm{u_0 - u}$ too large or stability cannot be proven.
  \begin{itemize}
    \item Can check difference from optimal input by solving a bilevel program for each simplex.
    \item If the error is too large, create a new point where the error is greatest.
    \item The bilevel program will be nonlinear ($f\sub{UL} = \max_x \norm{u_0 - u}_\infty$), but can be linearized using big M.
  \end{itemize}
  \item Can prove stability by showing that the value function is an LF for the closed loop system.
\end{itemize}

TTK18 Optimaliseringsbasert reguleringsdesign og analyse/tex/sec-bilevel.tex