\section{Systems of linear equations}
\subsection{Defintions}
Everything we learned so far about vectors and matrices can be used to solve and characterise a family of equations known as \emph{linear equations}. You're probably already very familiar with linear equations: they are equations in which the \emph{variables} appear directly, without any power or other functions acting on them. For example, the simple equation
\begin{equation}
	y = ax+b,
	\label{eq:linear_equation_2_vars}
\end{equation}
where $x,y$ are both variables and $a,b$ are both constant real numbers is a linear equation. \autoref{eq:linear_equation_2_vars} can be re-written as
\begin{equation}
	ax - y + b = 0,
	\label{eq:linear_equation_2_vars_rewritten}
\end{equation}
where now $a$ is the \emph{coefficient} of the variable $x$, while the variable $y$ has the coefficient $-1$ and $b$ is a so-called \emph{free coefficient}. In general, a linear equation of two variables has the form
\begin{equation}
	a_{0} + a_{x}x + a_{y}y = 0,
	\label{eq:general_linear_equation_2_vars}
\end{equation}
i.e. we changed the name of $a$ to $a_{x}$ and $b$ to $a_{0}$, and gave $y$ the coefficient $a_{y}$. We can also rename $x$ and $y$ to $x_{1}$ and $x_{2}$, respectively, and name their coefficients accordingly:
\begin{equation}
	a_{0} + a_{1}x_{1} + a_{2}x_{2} = 0.
	\label{eq:general_linear_equation_2_vars_renamed}
\end{equation}

The form shown in \autoref{eq:general_linear_equation_2_vars_renamed} can be easily expanded into $n$ variables:
\begin{equation}
	a_{0} + a_{1}x_{1} + a_{2}x_{2} + a_{3}x_{3} + \cdots + a_{n-1}x_{n-1} + a_{n}x_{n} = 0,
	\label{eq:general_linear_equation_n_vars}
\end{equation}
where $x_{1},x_{2},\dots,x_{n}$ are the variables of the equation, and $a_{0},a_{1},\dots,a_{n}$ are its coefficients. We say that $n$ is the \emph{order} (also: \emph{degree}) of the equation.

\begin{note}{Number set used for linear equations}{}
	As with other topics, in the context of this section both the variables and coefficients are all \textbf{real numbers}, however almost anything we discuss here can genrally be applied to complex numbers or other structures.
\end{note}

\begin{example}{Linear equations}{}
	The following is a linear equation of order $3$, using the variables $x,y,z$:
	\[
		3x+2y-z+4 = 0.
	\]
	The coefficients of the equation are
	\begin{align*}
		a_{0}&=4,\\
		a_{x}&=a_{1}=3,\\
		a_{y}&=a_{2}=2,\\
		a_{z}&=a_{3}=-1.
	\end{align*}

	Another linear equation of the same three variables is
	\[
		5x-2y+1 = 0.
	\]
	In this case the coefficient $a_{z}=a_{3}=0$. Depending on the context, this equation can be considered as either an equation of order $3$ or an equation of order $2$.
\end{example}

In $\Rs{2}$ linear equations represent a line, which doesn't necesserally go through the origin (and thus isn't necesserally a subspace of $\Rs{n}$). For a line to go through the origin, the free coefficient $a_{0}$ must equal zero (see \autoref{fig:linear_equations_2_vars}).

\begin{figure}
	\centering
	\begin{tikzpicture}
		\begin{axis}[
			vector plane,
			width=9cm, height=9cm,
			xmin=-1, xmax=6,
			ymin=-1, ymax=6,
			domain={-1:6},
		]
		\addplot[function, xred] {2*\x} node[pos=0.27, below, font=\Large, rotate=63.435] {$-2x+y=0$};
		\addplot[function, xblue] {-0.5*\x+5} node[pos=0.7, above, font=\Large, rotate=-30] {$\frac{1}{2}x+y+5=0$};
		\end{axis}
	\end{tikzpicture}
	\caption{Two linear equations represented as lines in $\Rs{2}$. Note how in the red equation the free coefficient is zero, and so the line goes through the origin.}
	\label{fig:linear_equations_2_vars}
\end{figure}

In $\Rs{3}$ linear equations represent planes. Much like with the lines in $\Rs{2}$, these planes don't necesserally go through the origin. The trend continues with increasing dimensions: in $\Rs{4}$ linear equations represent $3$-dimensional spaces, in $\Rs{5}$ linear equations represent $4$-dimensional spaces, etc. When the free coefficient is equal to zero, these spaces become subspaces of the respective $\Rs{n}$.

\begin{figure}
	\centering
	\begin{tikzpicture}
		% The trick with the labels is copied from https://tex.stackexchange.com/questions/212699/text-projection-onto-plane-in-3d-pgf-plots
		\def\h{58}
		\def\v{30}
		\begin{axis}[
			width=9cm, height=9cm,
			view={\h}{\v},
			]
			% Plots
			\pgfplotsset{plane/.style={surf, faceted color={#1!75!black!50}, fill={#1!20}, opacity=0.8, samples=6}}
			\addplot3[plane={xpurple}, domain=-4:0, y domain=-4:0, samples=3] {-1};
			\addplot3[plane={xpurple}, domain=-4:0, y domain=0:4, samples=3] {-1};
			\addplot3[plane={xorange}, domain=-4:4] {-0.5*\x};
			\addplot3[plane={xpurple}, domain=0:4, y domain=-4:0, samples=3] {-1};
			\addplot3[plane={xpurple}, domain=0:4, y domain=0:4, samples=3] {-1};
			\addplot3[mark=*] coordinates { (0,0,0) };

			% Labels
			\pgfmathparse{atan(tan(\h)*sin(\v))}
			\let\a=\pgfmathresult
			\pgfmathparse{atan(tan(90-\h)*sin(\v))}
			\let\b=\pgfmathresult
			\path (axis cs:-2.5,-1.5,1.25) -- (axis cs:-2.5,-0.5,1.25) node[color=xdarkorange, midway, above, sloped, xslant=tan(\a+\b+110), yscale=sin(\a+\b), font=\Huge] {$\frac{1}{2}x+z=0$};
			\path (axis cs:3.5,-1.5,-1) -- (axis cs:3.5,-0.5,-1) node[color=xdarkpurple, midway, above, sloped, xslant=tan(\a+\b+80), yscale=sin(\a+\b), font=\Huge] {$z+1=0$};
		\end{axis}
	\end{tikzpicture}
	\caption{Two intersecting planes in $\Rs{3}$ with their corresponding equations.}
	\label{fig:line_plane_R3}
\end{figure}

\subsection{Systems and matrix form}
A \emph{system of linear equations} is a set of linear equations using the same variables. For example, the three equations
\[
	\begin{cases}
		&2x-5y+4z+2=0\\
		&-3x-2y+1=0\\
		&5x+4z+-3=0\\
	\end{cases}
\]
form together a system of $3$ linear equations with $3$ variables ($x,y$ and $z$). Systems of linear equations can be written together in matrix form: in the above example, the system can be represented as the equation
\[
	\begin{bmatrix}
		2 & -5 & 4\\
		-3 & -2 & 0 \\
		5 & 0 & 4
	\end{bmatrix}\colvec{x;y;z} + \colvec{2;1;-3} = \colvec{0;0;0},
\]
since performing the matrix-vector product and vector addition yields back the system of equations. We call the matrix the \emph{coefficients matrix} of the equation.

\begin{note}{}{}
	In practice, many times the vector representing the free coefficients is moved to the right hand side of the equation. In the case of the above system this yields the simple equation
	\[
		\begin{bmatrix}
			2 & -5 & 4\\
			-3 & -2 & 0 \\
			5 & 0 & 4
		\end{bmatrix}\colvec{x;y;z} = \colvec{-2;-1;3}.
	\]
\end{note}

In the most general form, a system of $m$ equations in $n$ variavles $x_{1},x_{2},x_{3},\dots,x_{n}$ can be represented as the product of an $m\times n$ coefficient matrix and the variables vector, yielding the free-coefficient vector:

\vspace{1em}
\begin{equation}
	\begin{bNiceMatrix}[name=coeffMat_mxn]
		a_{11} & a_{12} & \cdots & a_{1n}\\
		a_{21} & a_{22} & \cdots & a_{2n}\\
		\vdots & \vdots & \Ddots & \vdots\\
		a_{m1} & a_{m2} & \cdots & a_{mn}
		\end{bNiceMatrix}\begin{bNiceMatrix}[name=varVec]x_{1}\\x_{2}\\x_{3}\\\vdots\\x_{n}\end{bNiceMatrix} = \begin{bNiceMatrix}[name=freeCoeffVec]b_{1}\\b_{2}\\\vdots\\b_{m}\end{bNiceMatrix},
	\label{eq:n_eqs_m_vars}
\end{equation}
\tikz[overlay, remember picture, blend mode=multiply]{
	% Highlights
	\node[mathl={xred!30}, fit=(coeffMat_mxn-1-1)(coeffMat_mxn-4-4)] {};
	\node[mathl={xblue!30}, fit=(varVec-1-1)(varVec-5-1)] {};
	\node[mathl={xgreen!30}, fit=(freeCoeffVec-1-1)(freeCoeffVec-4-1)] {};

	% Texts
	\tikzset{node distance=2cm and 2mm}
	\node[xred, below of=coeffMat_mxn-4-2, xshift=-5mm] (coeffMatTxt) {$\bm{m}\times \bm{n}$ coefficients};
	\node[xblue, below of=varVec-5-1] (varVecTxt) {$\bm{n}$ variables};
	\node[xgreen, below of=freeCoeffVec-4-1, xshift=15mm] (freeCoeffTxt) {$\bm{m}$ free coefficients};

	% Arrows
	\draw[vector, thin, xred] (coeffMatTxt) to [out=90, in=-90] ($(coeffMat_mxn-4-2)-(0,5mm)$);
	\draw[vector, thin, xblue] (varVecTxt) to [out=90, in=-90] ($(varVec-5-1)-(0,5mm)$);
	\draw[vector, thin, xgreen] (freeCoeffTxt) to [out=90, in=-90] ($(freeCoeffVec-4-1)-(0,5mm)$);
}

\vspace{5em}
which can be written succinctly as
\begin{equation}
	\colorbox{xred!30}{$A$}\colorbox{xblue!30}{$x$} = \colorbox{xgreen!30}{$b$}.
	\label{eq:system_equations_sussinct}
\end{equation}

\subsection{Solutions}
A system of linear equations can have one or more \emph{solutions}. A solution is a tuple
\[
	s=\left(s_{1},s_{2},\dots,s_{n}\right)
\]
such that if we substitute each $s_{i}$ into the respective variable $x_{i}$ all equation become \true{} statements.

\begin{example}{Solutions of a system of linear equations}{}
	The following linear system
	\[
		\begin{cases}
			&-4x+2y=0\\
			&x-y+3=0
		\end{cases}
	\]
	has the solution
	\[
		s = (-1,2).
	\]
	Indeed if we substitute $x=-1$ and $y=2$ into the system we get
	\[
		\begin{cases}
			&-4\cdot(-1)+2\cdot(2)=-4+4=0\ \Rightarrow\ \true{}\\
			&-1-2+3=-3+3=0\ \Rightarrow\ \true{}
		\end{cases}
	\]
\end{example}

In the graphical representation of linear equations, the solutions of a system are the points where the respective graphs representing the equation (line, plane, etc.) intersect.

\begin{example}{Solutions of a system of linear equations - graph}{sys_solution}
	The linear system from the previous example can be represented by the following graph:

	\vspace{2em}
	\centering
	\begin{tikzpicture}[every node/.style={font=\Large}]
		\begin{axis}[
			vector plane,
			width=9cm, height=9cm,
			xmin=-4, xmax=4,
			ymin=-4, ymax=4,
			domain={-4:4},
			major grid style={black!10},
		]
		\pgfmathsetmacro{\A}{-2}
		\pgfmathsetmacro{\B}{1}
		\addplot[function, xred] {\A*\x} node[pos=0.63, above, rotate={atan(\A)}] {$4x+2y=0$};
		\addplot[function, xblue] {\B*\x+3} node[pos=0.22, above, rotate={atan(\B)}] {$x-y-6=0$};
		\addplot[mark=*] coordinates { (-1,2) } node (s) {};
		\node[] (stxt) at (-3,3) {solution};
		\draw[-stealth, thick] (stxt.east) to [out=0, in=180] (s.west);
		\end{axis}
	\end{tikzpicture}
\end{example}

Not all systems have single solutions only, nor do all systems even have any solutions. For example, if we add to the system in \autoref{example:sys_solution} the equation
\[
	x-3y+3=0,
\]
the resulting system
\[
	\begin{cases}
		&-4x+2y=0\\
		&x-y+3=0\\
		&x-3y+3=0
	\end{cases}
\]
has no solutions (see \autoref{example:no_solution}).

\begin{example}{System with no solutions}{no_solution}
	When the system of linear equations from \autoref{example:sys_solution} is supplemented with the equation
	\[
		x-3y+3=0,
	\]
	it has no solution. However, any two equations of the system do have solutions (represented below as black points).

	\centering
	\begin{tikzpicture}[every node/.style={font=\Large}]
		\begin{axis}[
			vector plane,
			width=9cm, height=9cm,
			xmin=-4, xmax=4,
			ymin=-4, ymax=4,
			domain={-4:4},
			major grid style={black!10},
		]
		\pgfmathsetmacro{\A}{-2}
		\pgfmathsetmacro{\B}{1}
		\pgfmathsetmacro{\C}{1/3}
		\addplot[function, xred] {\A*\x} node[pos=0.63, above, rotate={atan(\A)}] {$4x+2y=0$};
		\addplot[function, xblue] {\B*\x+3} node[pos=0.22, above, rotate={atan(\B)}] {$x-y-6=0$};
		\addplot[function, xgreen] {\C*\x+1} node[pos=0.7, above, rotate={atan(\C)}] {$x-3y+3=0$};
		\addplot[only marks, mark=*] coordinates {
		  	(-1,2)
		  	(-3,0)
		  	(-0.4286,0.8571)
		} node (s) {};
		\end{axis}
	\end{tikzpicture}
\end{example}

Let us now explore when a set of linear equations in $\Rs{2}$ and $\Rs{3}$ has a single solution, infinitely many solutions or no solutions. In $\Rs{2}$, the lines representing two linear equations can be either parallel or non-parallel. If they are non-parallel then the two equations have a single solution - the intercept of both lines (as seen in the previous two examples). If the lines are parallel, there are two cases: either the two lines are identical, in which case there are infinitely many solutions (all the points on the line), or they are parallel yet distince, in which case there are no solutions to the system (see \autoref{fig:R2_num_solutions}).

\begin{figure}
	\centering
	\begin{subfigure}[c]{0.3\textwidth}
		\begin{center}
			\begin{tikzpicture}[every node/.style={font=\Large}]
				\begin{axis}[
					linear plane no ticks,
					]
					\addplot[function, xred] {-0.5*\x+1};
					\addplot[function, xgreen] {2*\x+3};
					% -0.5x+1 = 2x+3 ==> 2.5x=-2, x=-0.8, y=1.5
					\addplot[only marks, mark=*] coordinates { (-0.8,1.4) };
				\end{axis}
			\end{tikzpicture}
		\end{center}
		\caption{}
		\label{fig:}
	\end{subfigure}
	\begin{subfigure}[c]{0.3\textwidth}
		\begin{center}
			\begin{tikzpicture}[every node/.style={font=\Large}]
				\begin{axis}[
					linear plane no ticks,
					]
					\addplot[function, xred] {-0.5*\x-0.5};
					\addplot[function, dashed, xgreen] {-0.5*\x-0.5};
				\end{axis}
			\end{tikzpicture}
		\end{center}
		\caption{}
		\label{fig:R2_infinite_solutions}
	\end{subfigure}
	\begin{subfigure}[c]{0.3\textwidth}
		\begin{center}
			\begin{tikzpicture}[every node/.style={font=\Large}]
				\begin{axis}[
					linear plane no ticks,
					]
					\addplot[function, xred] {-0.5*\x-0.5};
					\addplot[function, xgreen] {-0.5*\x-1.5};
				\end{axis}
			\end{tikzpicture}
		\end{center}
		\caption{}
		\label{fig:}
	\end{subfigure}
	\caption{Three possible cases for two linear equations in $\Rs{2}$: (a) non-parallel and thus a single solution, (b) parallel and identical and thus infinitely many solutions, and (c) parallel but not identical and thus no solutions.}
	\label{fig:R2_num_solutions}
\end{figure}

In the case of more than two linear equations there can be, again, either a single solution, infinitely many solutions or no solutions. The difference is that in this case zero solutions can happen even when all of the lines representing the equations are non-parallel (see \autoref{fig:R2_num_solutions_multi}).

\begin{figure}
	\centering
	\begin{subfigure}[c]{0.3\textwidth}
		\begin{center}
			\begin{tikzpicture}[every node/.style={font=\Large}]
				\begin{axis}[
					linear plane no ticks,
					]
					\addplot[function, xred] {x};
					\addplot[function, xblue] {2*\x-2};
					\addplot[function, xgreen] {3*\x-4};
					\addplot[function, xpurple] {-\x+4};
					\addplot[function, xorange] {-2*\x+6};
					\addplot[only marks, mark=*] coordinates { (2,2) };
				\end{axis}
			\end{tikzpicture}
		\end{center}
		\caption{}
		\label{fig:}
	\end{subfigure}
	\begin{subfigure}[c]{0.3\textwidth}
		\begin{center}
			\begin{tikzpicture}[every node/.style={font=\Large}]
				\begin{axis}[
					linear plane no ticks,
					]
					\addplot[function, xred] {x};          % x-y=0
					\addplot[function, xblue] {2*\x+2};    % 2x-y=-2
					\addplot[function, xgreen] {3*\x-5};   % 3x-y=5
					\addplot[function, xpurple] {-\x+1};   % -x-y=-1
					\addplot[function, xorange] {-2*\x+3}; % -2x-y=-3
					\addplot[only marks, mark=*] coordinates {
							(-2.0,-2.0)
							(2.5,2.5)
							(0.5,0.5)
							(1.0,1.0)
							(7.0,16)
							(-0.334,1.334)
							(0.25,2.5)
							(1.5,-0.5)
							(1.6,-0.2)
							(2.0,-1.0)
					};
				\end{axis}
			\end{tikzpicture}
		\end{center}
		\caption{}
		\label{fig:}
	\end{subfigure}
	\begin{subfigure}[c]{0.3\textwidth}
		\begin{center}
			\begin{tikzpicture}[every node/.style={font=\Large}]
				\begin{axis}[
					linear plane no ticks,
					]
					\addplot[function, xred] {\x+2};
					\addplot[function, xblue] {\x+1};
					\addplot[function, xgreen] {\x};
					\addplot[function, xpurple] {\x-1};
					\addplot[function, xorange] {\x-2};
				\end{axis}
			\end{tikzpicture}
		\end{center}
		\caption{}
		\label{fig:}
	\end{subfigure}
	\caption{Three possible cases for the number of solutions of several linear equations in $\Rs{2}$: (a) non-parallel but all lines intercept at single point and thus the system has a solution, (b) non-parallel but no single interception point and thus no solution, and (c) parallel and thus no solutions. The case where all lines are identical and thus there are infinitely many solutions is ommited from the figure, and looks identical to \autoref{fig:R2_infinite_solutions}.}
	\label{fig:R2_num_solutions_multi}
\end{figure}

In $\Rs{3}$\ldots (TBW later because I'm lazy af)

\subsection{Finding solutions}
Several method to solve systems of linear equations were established over the years. One of the most well-known is \emph{Gauss elimination method}. The goal of this method is to use a set of pre-defined operations to bring the system into a form which is easy to solve. This form essentially exposes the rank of the coefficient matrix, from which one can deduce some important properties of the system such as the existance of a solution (or lack thereof), the degrees of freedom in the values which can be substituted into the system and more.

We start by describing the \emph{row-echelon form} of a matrix. For now we simply assume that such matrices exist, and later we will use a set of pre-defined operations to convert any matrix to this form. The following matrix is in row-echelon form:
\[
	A =
	\begin{bmatrix}
		1 & 3 & -5 & 7 &  0\\
		0 & 0 &  2 & 2 & -3\\
		0 & 0 &  0 & 1 &  2\\
		0 & 0 &  0 & 0 &  0\\
	\end{bmatrix}
\]

We note three properties of the above matrix:
\begin{itemize}
	\item The bottom row is all zeros.
	\item When looking at each row (left to right), all values are zero until we reach a non-zero element. For example, the first row starts with a non-zero element, but the second row has two non zero elements followed by the number $2$. These first non-zero elements are called \emph{leading coefficients} or \emph{pivots}, and each one is strictle to the \textbf{right} of the leading coefficient of the row above it.
	\item The leading coefficient of each row has only zeros below it in its column.
\end{itemize}
Any matrix that has the same three properies is said to be in its row-echelon form.

\begin{example}{Row-echelon form}{row-echelon}
	All of the matrices below are in their row-echelon form, and the leading zeros in each row are highlighted:

	\vspace{1em}
	\centering
	\begin{tabular}{cccc}
		$\begin{bNiceMatrix}[name=RE1]
			0 & 5 & 10\\
			0 & 0 & 0\\
			0 & 0 & 0\\
			0 & 0 & 0\\
		\end{bNiceMatrix}$ &
		$\begin{bNiceMatrix}[name=RE2]
			1 &  2 & 4\\
			0 & -2 & 2\\
			0 &  0 & 7\\
		\end{bNiceMatrix}$ &
		$\begin{bNiceMatrix}[name=RE3]
			0 & 0 & 3\\
			0 & 0 & 0\\
			0 & 0 & 0\\
		\end{bNiceMatrix}$ &
		$\begin{bNiceMatrix}[name=RE4]
			0 & 0 & 3 & -6 & 9 & -3\\
			0 & 0 & 0 &  0 & 2 &  6\\
		\end{bNiceMatrix}$\\
	\end{tabular}
	\tikz[overlay, remember picture, blend mode=multiply]{
		\node[mathl={xred!20}, fit=(RE1-1-2)] {};
		\foreach \pivot in {(RE2-1-1),(RE2-2-2),(RE2-3-3)}{
			\node[mathl={xblue!20}, fit=\pivot] {};
		}
		\node[mathl={xgreen!20}, fit=(RE3-1-3)] {};
		\foreach \pivot in {(RE4-1-3),(RE4-2-5)}{
			\node[mathl={xpurple!20}, fit=\pivot] {};
		}
	}
	\flushleft

	The following matrices are all \textbf{not} in their row-echelon form:

	\vspace{1em}
	\centering
	\begin{tabular}{cccc}
		$\begin{bmatrix}
			0 & 5 & 1\\
			0 & 0 & 0\\
			0 & 1 & 0\\
			0 & 0 & 0\\
		\end{bmatrix}$ &
		$\begin{bmatrix}
			1 &  2 & 4\\
			0 & -2 & 2\\
			3 &  0 & 7\\
		\end{bmatrix}$ &
		$\begin{bmatrix}
			0 & 0 & 0\\
			0 & 0 & 0\\
			0 & 0 & 3\\
		\end{bmatrix}$ &
		$\begin{bmatrix}
			0 & 0 & 7 & -3 & 4 & 4\\
			0 & 1 & 0 &  0 & 2 & 3\\
		\end{bmatrix}$\\
	\end{tabular}
\end{example}

Using the row-echelon form we define the \emph{reduced row-echelon form}. A matrix in a reduced row-echelon form is a matrix that is already in its row-echelon form, and in addition:
\begin{itemize}
	\item The leading coefficients are all equal to $1$ (and are thus called \emph{leading ones}).
	\item All the elements in the \textbf{column} of a leading one are equal to zero (except the leading one itself).
\end{itemize}

\begin{example}{Reduced row-echelon form}{}
	The following matrices are the reduced row-echelon forms of the row-echelon matrices in \autoref{example:row-echelon}, with theleading ones in each row highlighted:

	\vspace{1em}
	\centering
	\begin{tabular}{cccc}
		$\begin{bNiceMatrix}[name=RRE1]
			0 & 1 & 2\\
			0 & 0 & 0\\
			0 & 0 & 0\\
			0 & 0 & 0\\
		\end{bNiceMatrix}$ &
		$\begin{bNiceMatrix}[name=RRE2]
			1 &  0 & 0\\
			0 & -1 & 0\\
			0 &  0 & 1\\
		\end{bNiceMatrix}$ &
		$\begin{bNiceMatrix}[name=RRE3]
			0 & 0 & 1\\
			0 & 0 & 0\\
			0 & 0 & 0\\
		\end{bNiceMatrix}$ &
		$\begin{bNiceMatrix}[name=RRE4]
			0 & 0 & 1 & -2 & 0 & -10\\
			0 & 0 & 0 &  0 & 1 &   3\\
		\end{bNiceMatrix}$\\
	\end{tabular}
	\tikz[overlay, remember picture, blend mode=multiply]{
		\node[mathl={xred!20}, fit=(RRE1-1-2)] {};
		\foreach \pivot in {(RRE2-1-1),(RRE2-2-2),(RRE2-3-3)}{
			\node[mathl={xblue!20}, fit=\pivot] {};
		}
		\node[mathl={xgreen!20}, fit=(RRE3-1-3)] {};
		\foreach \pivot in {(RRE4-1-3),(RRE4-2-5)}{
			\node[mathl={xpurple!20}, fit=\pivot] {};
		}
	}
\end{example}

To bring a matrix to its reduced row-echelon form, we use a sequence of operations from the following set:
\begin{itemize}
	\item Scaling a row by a non-zero real number. For example:
		\[
			\begin{bNiceMatrix}[name=rreo1]
				 1 & 0 & -3 &  2\\
				-3 & 2 &  1 &  1\\
				 5 & 1 &  0 & -4\\
				 2 & 2 & -1 &  7\\
			 \end{bNiceMatrix} \xrightarrow[]{\colorbox{xblue!30}{$R_{2}$}\rightarrow3\colorbox{xblue!30}{$R_{2}$}}
			 \begin{bNiceMatrix}[name=rreo2]
				 1 & 0 & -3 &  2\\
				-9 & 6 &  3 &  3\\
				 5 & 1 &  0 & -4\\
				 2 & 2 & -1 &  7\\
			 \end{bNiceMatrix}
		\]
		here we take the 2nd row of the matrix (denoted as $R_{2}$) and scale it by $3$. The notation above the arrow tells us exactly that: $R_{2}$ is transformed into $2\times R_{2}$.

	\item Exchanging two rows. For example:
		\[
			\begin{bNiceMatrix}[name=rreo3]
				 1 & 0 & -3 &  2\\
				-9 & 6 &  3 &  3\\
				 5 & 1 &  0 & -4\\
				 2 & 2 & -1 &  7\\
			 \end{bNiceMatrix} \xrightarrow[]{\colorbox{xred!30}{$R_{2}$}\leftrightarrow \colorbox{xgreen!30}{$R_{3}$}}
			 \begin{bNiceMatrix}[name=rreo4]
				 1 & 0 & -3 &  2\\
				 5 & 1 &  0 & -4\\
				-9 & 6 &  3 &  3\\
				 0 & 2 &  6 &  3\\
			 \end{bNiceMatrix}
		\]

	\item Adding one scaled row to a different row (where the scalar is not zero). For example:
		\[
			\begin{bNiceMatrix}[name=rreo5]
				 1 & 0 & -3 &  2\\
				 5 & 1 &  0 & -4\\
				-9 & 6 &  3 &  3\\
				 2 & 2 & -1 &  7\\
			 \end{bNiceMatrix} \xrightarrow[]{\colorbox{xpurple!30}{$R_{4}$}\rightarrow \colorbox{xpurple!30}{$R_{4}$}-2\colorbox{xorange!30}{$R_{1}$}}
			 \begin{bNiceMatrix}[name=rreo6]
				 1 & 0 & -3 &  2\\
				 5 & 1 &  0 & -4\\
				-9 & 6 &  3 &  3\\
				 0 & 2 &  5 &  3\\
			 \end{bNiceMatrix}
		\]
		(note that $R_{1}$ itself is not changed by the operation)

\tikz[overlay, remember picture, blend mode=multiply]{
	% Scaling
	\node[mathl={xblue!30}, fit=(rreo1-2-1)(rreo1-2-4)] {};
	\node[mathl={xblue!30}, fit=(rreo2-2-1)(rreo2-2-4)] {};

	% Exchanging
	\node[mathl={xred!30}, fit=(rreo3-2-1)(rreo3-2-4)] {};
	\node[mathl={xred!30}, fit=(rreo4-3-1)(rreo4-3-4)] {};
	\node[mathl={xgreen!30}, fit=(rreo3-3-1)(rreo3-3-4)] {};
	\node[mathl={xgreen!30}, fit=(rreo4-2-1)(rreo4-2-4)] {};

	% Addition
	\node[mathl={xorange!30}, fit=(rreo5-1-1)(rreo5-1-4)] {};
	\node[mathl={xorange!30}, fit=(rreo6-1-1)(rreo6-1-4)] {};
	\node[mathl={xpurple!30}, fit=(rreo5-4-1)(rreo5-4-4)] {};
	\node[mathl={xpurple!30}, fit=(rreo6-4-1)(rreo6-4-4)] {};
}
\end{itemize}

\begin{example}{Row operations}{}
	Let us apply a sequence of 6 row opertions on a matrix until it reaches its reduced row-echelon form:

	\begin{align*}
		&
		\begin{bmatrix}
			 1 &  5 & 0 \\
			-2 &  3 & 1 \\
			-3 & -8 & 0 \\
		\end{bmatrix}
		\xrightarrow[]{R_{3}\to R_{3}+3R_{1}}
		\begin{bmatrix}
			 1 & 5 & 0 \\
			-2 & 3 & 1 \\
			 0 & 7 & 0 \\
		\end{bmatrix}
		\xrightarrow[]{R_{3}\to \frac{1}{7}R_{3}}
		\begin{bmatrix}
			 1 & 5 & 0 \\
			-2 & 3 & 1 \\
			 0 & 1 & 0 \\
		\end{bmatrix}
		\xrightarrow[]{R_{2}\leftrightarrow R_{3}}
		\\&
		\begin{bmatrix}
			 1 & 5 & 0 \\
			 0 & 1 & 0 \\
			-2 & 3 & 1 \\
		\end{bmatrix}
		\xrightarrow[]{R_{1}\to R_{1}-5R_{2}}
		\begin{bmatrix}
			 1 & 0 & 0 \\
			 0 & 1 & 0 \\
			-2 & 3 & 1 \\
		\end{bmatrix}
		\xrightarrow[]{R_{3}\to R_{3}+2R_{1}}
		\begin{bmatrix}
			 1 & 0 & 0 \\
			 0 & 1 & 0 \\
			 0 & 3 & 1 \\
		\end{bmatrix}
		\xrightarrow[]{R_{3}\to R_{3}-3R_{2}}
		\\&
		\begin{bmatrix}
			 1 & 0 & 0 \\
			 0 & 1 & 0 \\
			 0 & 0 & 1 \\
	    \end{bmatrix}=I_{3}.
	\end{align*}
\end{example}

To use row operations to solve a system of linear equations, we first write the \emph{augmented matrix} form of the system. This for looks as follows: suppose we have the system
\[
	\begin{cases}
		% x=-1, y=3, z=2
		&2x+y-3z  = -5\\
		&4x-2y+6z =  2\\
		&x+y-z    =  0
	\end{cases}
\]
Then the augmenten matrix of the system is
\[
	\begin{bNiceMatrixAug}
		2 &  1 & -3 & -5\\
		4 & -2 &  6 &  2\\
		1 &  1 & -1 &  0
	\end{bNiceMatrixAug},
\]
i.e. it is simpy the coefficients matrix \textit{augmented} with the free coefficients vector. We draw a vertical line where the matrix and vector were "stitched" together, to remind us which elements belong to what object.

To solve the system, we bring the augmented matrix to its reduced row-echelon form. In our case, this would be the matrix
\[
	\begin{bNiceMatrixAug}[name=AugM1]
		1 & 0 & 0 & -1\\
		0 & 1 & 0 &  3\\
		0 & 0 & 1 &  2
	\end{bNiceMatrixAug},
\]
which means that the solution to the system is
\[
	x=-1,\ y=3,\ z=2.
\]
\solvesym{AugM1-1-4}

\begin{example}{Solving a system of linear equations}{}
	Let us solve the following system of linear equations using the Gaussian elimination method:

	\[
		% x=4, y=0, z=3, w=-2
		\begin{cases}
		& -3x+6y+6z-2w = 10\\
		& 7x-5y+7z-6w = 61\\
		& 5x-5y-4z-5w = 18\\
		& -8x+2y+2z+7y = -40
		\end{cases}
	\]

	The augmented matrix for the system is
	\[
		A =
		\begin{bNiceMatrixAug}
			-3 &  6 &  6 & -2 &  10\\
			 7 & -5 &  7 & -6 &  61\\
			 5 & -5 & -4 & -5 &  18\\
			-8 &  2 &  2 &  7 & -40
		\end{bNiceMatrixAug}.
	\]

	Applying a sequence of row operations to $A$:

	\begin{align*}
		&
		\begin{bNiceMatrixAug}
			-3 &  6 &  6 & -2 &  10\\
			 7 & -5 &  7 & -6 &  61\\
			 5 & -5 & -4 & -5 &  18\\
			-8 &  2 &  2 &  7 & -40
		\end{bNiceMatrixAug}
		\xrightarrow[]{R_{1}\to R_{1}-3R_{4}}
		\begin{bNiceMatrixAug}
			21 &  0 &  0 & -23 & 130\\
			 7 & -5 &  7 &  -6 &  61\\
			 5 & -5 & -4 &  -5 &  18\\
			-8 &  2 &  2 &   7 & -40
		\end{bNiceMatrixAug}
	  \\[5mm]&
		\xrightarrow[]{R_{2}\to R_{2}-R_{3}}
		\begin{bNiceMatrixAug}
			21 &  0 &  0 & -23 & 130\\
			 2 &  0 & 11 &  -1 &  43\\
			 5 & -5 & -4 &  -5 &  18\\
			-8 &  2 &  2 &   7 & -40
		\end{bNiceMatrixAug}
		\xrightarrow[]{R_{3}\to R_{3}+2R_{4}}
		\begin{bNiceMatrixAug}
			21 &  0 &  0 & -23 & 130\\
			 2 &  0 & 11 &  -1 &  43\\
		   -11 & -1 &  0 &   9 & -62\\
			-8 &  2 &  2 &   7 & -40
		\end{bNiceMatrixAug}
	  \\[5mm]&
		\xrightarrow[]{R_{4}\to R_{4}+2R_{3}}
		\begin{bNiceMatrixAug}
			21 &  0 &  0 & -23 & 130\\
			 2 &  0 & 11 &  -1 &  43\\
		   -11 & -1 &  0 &   9 & -62\\
		   -30 &  0 &  2 &  25 & -164
		\end{bNiceMatrixAug}
		\xrightarrow[]{R_{4}\to R_{4}+15R_{2}}
		\begin{bNiceMatrixAug}
			21 &  0 &   0 & -23 & 130\\
			 2 &  0 &  11 &  -1 &  43\\
		   -11 & -1 &   0 &   9 & -62\\
		     0 &  0 & 167 &  10 & 481
		\end{bNiceMatrixAug}
	  \\[5mm]&
		\xrightarrow[]{R_{1}\to R_{1}-10.5R_{2}}
		\begin{bNiceMatrixAug}
			 0 &  0 & -115.5 & -12.5 & -321.5\\
			 2 &  0 &   11   &    -1 &   43\\
		   -11 & -1 &    0   &     9 &  -62\\
		     0 &  0 &  167   &    10 &   481
		\end{bNiceMatrixAug}
	  \\[5mm]&
		\xrightarrow[]{R_{1}\to R_{1}-1.25R_{4}}
		\begin{bNiceMatrixAug}
			 0 &  0 &  93.25 &  0 & 279.75\\
			 2 &  0 &  11    & -1 &  43\\
		   -11 & -1 &   0    &  9 & -62\\
		     0 &  0 & 167    & 10 &  481
		\end{bNiceMatrixAug}
	  \\[5mm]&
		\xrightarrow[]{R_{1}\to \frac{1}{93.25}R_{1}}
		\begin{bNiceMatrixAug}
			 0 &  0 &   1 &  0 &   3\\
			 2 &  0 &  11 & -1 &  43\\
		   -11 & -1 &   0 &  9 & -62\\
		     0 &  0 & 167 & 10 &  481
		\end{bNiceMatrixAug}
		\xrightarrow[]{R_{4}\to R_{4}-167R_{1}}
		\begin{bNiceMatrixAug}
			 0 &  0 &  1 &  0 &   3\\
			 2 &  0 & 11 & -1 &  43\\
		   -11 & -1 &  0 &  9 & -62\\
		     0 &  0 &  0 & 10 &  -20
		\end{bNiceMatrixAug}
	  \\[5mm]&
		\xrightarrow[]{R_{4}\to \frac{1}{10}R_{4}}
		\begin{bNiceMatrixAug}
			 0 &  0 &  1 &  0 &   3\\
			 2 &  0 & 11 & -1 &  43\\
		   -11 & -1 &  0 &  9 & -62\\
		     0 &  0 &  0 &  1 &  -2
		\end{bNiceMatrixAug}
		\xrightarrow[]{R_{3}\to R_{3}-9R_{4}}
		\begin{bNiceMatrixAug}
			 0 &  0 &  1 &  0 &   3\\
			 2 &  0 & 11 & -1 &  43\\
		   -11 & -1 &  0 &  0 & -44\\
		     0 &  0 &  0 &  1 &  -2
		\end{bNiceMatrixAug}
	  \\[5mm]&
		\xrightarrow[]{R_{2}\to R_{2}-11R_{1}}
		\begin{bNiceMatrixAug}
			 0 &  0 & 1 &  0 &   3\\
			 2 &  0 & 0 & -1 &  10\\
		   -11 & -1 & 0 &  0 & -44\\
		     0 &  0 & 0 &  1 &  -2
		\end{bNiceMatrixAug}
		\xrightarrow[]{R_{2}\to R_{2}-R_{4}}
		\begin{bNiceMatrixAug}
			 0 &  0 & 1 & 0 &   3\\
			 2 &  0 & 0 & 0 &   8\\
		   -11 & -1 & 0 & 0 & -44\\
		     0 &  0 & 0 & 1 &  -2
		\end{bNiceMatrixAug}
	  \\[5mm]&
		\xrightarrow[]{R_{2}\to \frac{1}{2}R_{2}}
		\begin{bNiceMatrixAug}
			 0 &  0 & 1 & 0 &   3\\
			 1 &  0 & 0 & 0 &   4\\
		   -11 & -1 & 0 & 0 & -44\\
		     0 &  0 & 0 & 1 &  -2
		\end{bNiceMatrixAug}
		\xrightarrow[]{R_{3}\to R_{3}+11R_{2}}
		\begin{bNiceMatrixAug}
			0 &  0 & 1 & 0 &  3\\
			1 &  0 & 0 & 0 &  4\\
			0 & -1 & 0 & 0 &  0\\
		    0 &  0 & 0 & 1 & -2
		\end{bNiceMatrixAug}
	  \\[5mm]&
		\xrightarrow[]{R_{3}\to -R_{3}}
		\begin{bNiceMatrixAug}
			0 & 0 & 1 & 0 &  3\\
			1 & 0 & 0 & 0 &  4\\
			0 & 1 & 0 & 0 &  0\\
		    0 & 0 & 0 & 1 & -2
		\end{bNiceMatrixAug}
		\xrightarrow[]{R_{1}\leftrightarrow R_{2},\ R_{2}\leftrightarrow R_{3}}
		\begin{bNiceMatrixAug}[name=rrefe]
			1 & 0 & 0 & 0 &  4\\
			0 & 1 & 0 & 0 &  0\\
			0 & 0 & 1 & 0 &  3\\
		    0 & 0 & 0 & 1 & -2
		\end{bNiceMatrixAug}.
	\end{align*}
	\tikz[overlay, remember picture]{
		\node[thick, draw=xred, rounded corners, fill=xred!20, right of=rrefe-3-5, text width=2.5cm, xshift=1.3cm, align=center] (rrefeTxt) {Reduced row echelon form};
		\draw[-stealth, thick, xred] (rrefeTxt.west) to ($(rrefe-3-5.east)+(2.5mm,0)$);
	}

	The solution of the system is therefore
	\[
		x=4,\ y=0,\ z=3,\ w=-2.
	\]

\end{example}


\def\Mu{\boldsymbol{\mu}}
\def\Lamb{\boldsymbol{\lambda}}

\lab{Interior Point 2: Quadratic Programs}{Interior Point 2: Quadratic Programs}
\objective{Interior point methods originated as an alternative to the Simplex method for solving linear optimization problems. However, they can also be adapted to treat convex optimization problems in general.
In this lab we implement a primal-dual Interior Point method for convex quadratic constrained optimization and explore applications in elastic membrane theory and finance.}

% Todo: change \geq in the constraints to the component-wise geq in the book.

\section*{Quadratic Optimization Problems}
%In this lab, we will explore an extension of Interior Point methods to a broader class of problems, namely quadratic constrained optimization problems.
A \emph{quadratic constrained optimization problem} differs from a linear constrained optimization problem only in that the objective function is quadratic rather than linear.
We can pose such a problem as follows:
\begin{align*}
\text{minimize }\qquad \frac{1}{2}&\x\trp Q\x + \c\trp \x\\
\text{subject to }\qquad &A\x \succeq \b,\\
&G\x = \mathbf{h}.
\end{align*}
% Unlike linear constrained optimization, the optimal point is not guaranteed to be one of the vertices of the feasible polytope (if the feasible region even is, indeed, a polytope).
% Thus, any attempt to extend the popular Simplex Algorithm to quadratic programs would require substantial adjustments, as that algorithm is crucially based on locating and searching through the vertices of the feasible region.
% Interior Point methods, however, generalize readily to the situation at hand.

We will restrict our attention to quadratic programs involving positive semidefinite quadratic terms (in general, indefinite quadratic objective functions admit many local minima, complicating matters considerably).
Such problems are called \emph{convex}, since the objective function is convex.
To simplify the exposition, we will also only allow inequality constraints (generalizing to include equality constraints is not difficult).
Thus, we have the problem
\begin{align*}
\text{minimize }\qquad \frac{1}{2}&\x\trp Q\x + \c\trp \x\\
\text{subject to }\qquad &A\x \succeq \b\\
\end{align*}
where $Q\in\mathbb{R}^{n\times n}$ is a positive semidefinite matrix, $A\in\mathbb{R}^{m\times n}$, $\x, \c \in \mathbb{R}^n$,
and $\b \in \mathbb{R}^m$.

%We begin by deriving the KKT conditions for this problem.
The Lagrangian function for this problem is:
\begin{align}
\mathcal{L}(\x,\Mu) = \frac{1}{2}\x\trp Q\x + \c\trp \x - \Mu\trp (A\x -\b),
\label{eq:lagrangian}
\end{align}
where $\Mu \in \mathbb{R}^m$ is the Lagrange multiplier.

\begin{comment} % Derivation overkill vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
We next take the gradient of the Lagrangian with respect to $x$ and set it equal to zero:
\begin{align*}
0 &= \nabla_x \mathcal{L}(x,\lambda)\\
&= Qx + c - A\trp \lambda.
\end{align*}
We next write the complementary slackness condition:
\[
(Ax - b)_i\lambda_i = 0, \qquad i=1,2,\ldots,m.
\]
We finish by listing the inequality constraints as well as the nonnegativity constraint for $\lambda$:
\begin{align*}
Ax - b &\geq 0,\\
\lambda &\geq 0.
\end{align*}
\end{comment} % Derivation overkill ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

% What we have now is a mixture of equations and inequalities.
% We want to express these conditions as a system of equations, so
We also introduce a nonnegative slack vector $\y\in\mathbb{R}^m$ to change the inequality
$A\x - \b \succeq \0$ into the equality $A\x - \b - \y = \0$.

\begin{comment}
\begin{align}
A\x - \b - \y = \0 &&\Longrightarrow &&\y = A\x - \b
\label{eq:equality}
\end{align}
\end{comment}

Then the complete set of KKT conditions are:
% we make this substitution into the complementary slackness conditions.
% We obtain the following statement of the KKT conditions:
\begin{align*}
Q\x - A\trp \Mu + \c &= \0,\\
A\x - \b - \y &= \0,\\
y_i\mu_i &= 0, \qquad i=1,2,\ldots,m,\\
\y,\Mu &\succeq \0.
\end{align*}

\section*{Quadratic Interior Point Method}

The Interior Point method we describe here is an adaptation of the method we used with linear programming.
%The basic intuition is the same as before: we start at some point in the interior of the feasible region, and we make a series of steps so that we approach the solution to the KKT conditions iteratively.
% Since the solution to the KKT conditions can be seen as the root of a system of equations, our first thought might be to use Newton's Method for root-finding.
% This will not work on its own, however, because the KKT conditions also require inequality constraints, which will be violated if we blindly apply Newton's Method.
% Hence, we make adjustments to our choice of search direction and step size, so that we respect the constraints while moving closer to the solution of the KKT conditions.
Define $Y = \text{diag}(y_1,y_2,\ldots,y_m)$, $M = \text{diag}(\mu_1,\mu_2,\ldots,\mu_m)$, and let $\e\in\mathbb{R}^m$ be a vector of all ones.
Then the roots of the function
\begin{align*}
F(\x,\y,\Mu) =
\begin{bmatrix}
Q\x-A\trp \Mu + c\\
A\x-\y-\b\\
YM\e
\end{bmatrix}
&= \0,\\
(\y,\Mu) &\succeq \0
\end{align*}
satisfy the KKT conditions.
The derivative matrix of this function is given by
\[
DF(\x,\y,\Mu) =
\begin{bmatrix}
Q & 0 & -A\trp \\
A & -I & 0\\
0 & M & Y
\end{bmatrix},
\]
and the duality measure $\nu$ for this problem is \[\nu = \frac{\y\trp \Mu}{m}.\]

\subsection*{Search Direction}

We calculate the search direction for this algorithm in the spirit of Newton's Method; this is the same way that we did in the linear programming case.
That is, we solve the system:
\begin{align}
DF(\x,\y,\Mu)
\begin{bmatrix}\triangle \x\\ \triangle \y\\ \triangle \Mu\end{bmatrix}
= - F(\x,\y,\Mu) +
\begin{bmatrix} \0 \\ \0 \\ \sigma\nu\e \end{bmatrix},
\label{eq:searchDirection}
\end{align}
where $\sigma\in [0,1)$ is the centering parameter.

\begin{problem}
Create a function \li{qInteriorPoint()}.
It should accept the arrays $Q, \c, A,$ and $\b$, a tuple of arrays \li{guess} giving initial estimates for $\x, \y,$ and $\Mu$ (this will be explained later), along with the keyword arguments \li{niter=20} and \li{tol=1e-16}.

In this function, calculate the search direction.
Create $F$ and $DF$ as described above, and calculate the search direction $(\triangle \x\trp , \triangle \y\trp , \triangle \Mu\trp )$ by solving Equation \ref{eq:searchDirection}.
Use $\sigma = \frac{1}{10}$ for the centering parameter.

(Hint: What are the dimensions of $F$ and $DF$?)
\end{problem}

\begin{comment} % Old method for search direction vvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Our goal is to produce a sequence of points that approach the solution to this system of equations, all the while
respecting the nonnegativity constraints.% $\y,\Mu \geq \0$.
We achieve this goal using largely the same approach as in our linear programming Interior Point method.
We first apply Newton's method to $F$, obtaining a Newton search direction $(\triangle x, \triangle y, \triangle \lambda)$
that solves the system
\begin{equation}
\begin{bmatrix}
Q & 0 & -A\trp \\
A & -I & 0\\
0 & \Lambda & \mathcal{Y}
\end{bmatrix}
\begin{bmatrix}
\triangle x\\
\triangle y\\
\triangle \lambda
\end{bmatrix}
=
\begin{bmatrix}
-Qx + A\trp \lambda - c\\
-Ax + y + b\\
-\Lambda\mathcal{Y}e
\end{bmatrix}.
\label{eq:affine}
\end{equation}
We may not be able to step far in this direction without violating the nonnegativity constraints, so we calculate
an improved direction by perturbing the system of equations. This is done by making several small calculations.

First calculate the \emph{duality measure}
\[
\nu = \frac{1}{m}y\trp \lambda,
\]
which tells us about how close we are to the optimal point (values closer to zero indicate better proximity to the optimizer).

Next, we calculate a maximal allowed step length in the Newton direction:
\[
\hat{\alpha} = \max \{\alpha \in (0,1] \, | \, (y,\lambda) + \alpha(\triangle y, \triangle \lambda) \geq 0\}.
\]
You can check that this value is given by the minimum of the values
\begin{align*}
\min\left(1, \min_{i : \triangle y_i < 0} - \frac{y_i}{\triangle y_i}\right),\\
\min\left(1, \min_{i : \triangle \lambda_i < 0} -\frac{\lambda_i}{\triangle \lambda_i}\right).
\end{align*}

Continuing on, we compute the Newton duality measure given by
\[
\hat{\nu} = \frac{1}{m}(y + \hat{\alpha}\triangle y)\trp (\lambda + \hat{\alpha}\triangle \lambda),
\]
and then the centering parameter
\[
\sigma = \left(\frac{\hat{\nu}}{\nu}\right)^3.
\]

Finally, we obtain our search direction $(\triangle x', \triangle y', \triangle \lambda')$ by solving the perturbed
system
\begin{equation}
\begin{bmatrix}
Q & 0 & -A\trp \\
A & -I & 0\\
0 & \Lambda & \mathcal{Y}
\end{bmatrix}
\begin{bmatrix}
\triangle x'\\
\triangle y'\\
\triangle \lambda'
\end{bmatrix}
=
\begin{bmatrix}
-Qx + A\trp \lambda - c\\
-Ax + y + b\\
-\Lambda\mathcal{Y}e - \triangle \Lambda\triangle\mathcal{Y}e + \sigma\nu e
\end{bmatrix}.
\label{eq:perturbed}
\end{equation}
\end{comment} % Old method for search direction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

\subsection*{Step Length}
Now that we have our search direction, we select a step length.
 We want to step nearly as far as possible
without violating the nonnegativity constraints.
However, we back off slightly from the maximum allowed step length
because an overly greedy step at one iteration may prevent a descent step at the next iteration.
Thus, we choose our step size
\[
\alpha = \max\{a \in (0,1] \, \mid \, \tau(\y,\Mu) +a(\triangle \y, \triangle \Mu) \succeq \0\},
\]
where $\tau \in (0,1)$ controls how much we back off from the maximal step length.
For now, choose $\tau = 0.95$.
In general, $\tau$ can be made to approach $1$ at each successive iteration.
This may speed up convergence in some cases.

% This is equivalent to the method of choosing a step direction used in the previous lab.
% In this case, however, we will use a single step length for all three of the parameters.
We wish to step nearly as far as possible without violating the problem’s constraints, as to remain in the interior of the feasible region.
First, we calculate the maximum allowable step lengths for $\mu$ and $\y$.

\begin{align*}
\beta_{\max} &= \min\{-\mu_i/\triangle \mu_i \mid \triangle \mu_i < 0 \}\\
\delta_{\max} &=\min\{-y_i/\triangle y_i \mid \triangle y_i < 0 \}
\end{align*}
% Since $\Mu,\y \geq \0$.
If all of the entries of $\triangle \mu$ are nonnegative, we let $\beta_{\max} = 1$.
Likewise, if all the entries of $\triangle \y$ are nonnegative, let $\delta_{\max}=1$.
Next, we back off from these maximum step lengths slightly:
\begin{align*}
\beta &= \min(1, \tau\beta_{\max})\\
\delta &= \min(1, \tau\delta_{\max})\\
\alpha &= \min(\beta, \delta)
\end{align*}
This $\alpha$ is our final step length.
Thus, the next point in the iteration is given by:
\[
(\x_{k+1}, \y_{k+1}, \Mu_{k+1}) = (\x_k, \y_k, \Mu_k) + \alpha(\triangle \x_k, \triangle \y_k, \triangle \Mu_k).
\]
This completes one iteration of the algorithm.

\begin{comment} % Put this back in?
\begin{info}
As with our Interior Point method for linear constrained optimization, the most expensive part of each iteration is solving the linear systems \ref{eq:affine} and \ref{eq:perturbed}.
Note, however, that these systems both have the same matrix on the left-hand side.
This allows us to factor the matrix just once per iteration, and use the factorization to solve both systems.
A more sophisticated implementation would likely split up these large systems of equations into a few smaller ones, and then use Cholesky-based factorizations.
To simplify matters, we suggest simply using as a first attempt an LU decomposition on the entire matrix.
\end{info}
\end{comment}

\subsection*{Initial Point}
The starting point $(\x_0, \y_0, \Mu_0)$ has an important effect on the convergence of the algorithm.
The code listed below will calculate an appropriate starting point:
\begin{lstlisting}
def startingPoint(G, c, A, b, guess):
    """
    Obtain an appropriate initial point for solving the QP
    .5 x\trp  Gx + x\trp  c s.t. Ax >= b.
    Parameters:
        G -- symmetric positive semidefinite matrix shape (n,n)
        c -- array of length n
        A -- constraint matrix shape (m,n)
        b -- array of length m
        guess -- a tuple of arrays (x, y, l) of lengths n, m, and m, resp.
    Returns:
        a tuple of arrays (x0, y0, l0) of lengths n, m, and m, resp.
    """
    m,n = A.shape
    x0, y0, l0 = guess

    # initialize linear system
    N = np.zeros((n+m+m, n+m+m))
    N[:n,:n] = G
    N[:n, n+m:] = -A.T
    N[n:n+m, :n] = A
    N[n:n+m, n:n+m] = -np.eye(m)
    N[n+m:, n:n+m] = np.diag(l0)
    N[n+m:, n+m:] = np.diag(y0)
    rhs = np.empty(n+m+m)
    rhs[:n] = -(G.dot(x0) - A.T.dot(l0)+c)
    rhs[n:n+m] = -(A.dot(x0) - y0 - b)
    rhs[n+m:] = -(y0*l0)

    sol = la.solve(N, rhs)
    dx = sol[:n]
    dy = sol[n:n+m]
    dl = sol[n+m:]

    y0 = np.maximum(1, np.abs(y0 + dy))
    l0 = np.maximum(1, np.abs(l0+dl))

    return x0, y0, l0
\end{lstlisting}
Notice that we still need to provide a tuple of arrays \li{guess} as an argument.
Do your best to provide a reasonable guess for the array $\x$, and we suggest setting $\y$ and $\Mu$ equal to arrays of ones.
We summarize the entire algorithm below.

\begin{algorithm}[H]
\begin{algorithmic}[1]
\Procedure{Interior Point Method for QP}{}
    \State \textrm{Choose initial point } $(\x_0, \y_0, \Mu_0)$.
    \While{$k <\ $\li{niters} and $\nu <\ $\li{tol}:}
        \State \textrm{Calculate the duality measure} $\nu$.
        \State \textrm{Solve \ref{eq:searchDirection} for the search direction} $(\triangle \x_k, \triangle \y_k, \triangle \Mu_k)$.
        \State \textrm{Calculate the step length} $\alpha$.
        \State $(\x_{k+1}, \y_{k+1}, \Mu_{k+1}) = (\x_k, \y_k, \Mu_k) + \alpha(\triangle \x_k, \triangle \y_k, \triangle \Mu_k).$
    \EndWhile
\EndProcedure
\end{algorithmic}
\label{alg:intPt2}
\end{algorithm}

\begin{problem}
Complete the implementation of \li{qInteriorPoint()}.
Return the optimal point $\x$ as well as the final objective function value.
% and \li{verbose}, a boolean value indicating whether or not to print the current objective function value ($ \frac{1}{2}x_k\trp Qx_k + c\trp x_k$) and duality measure ($\nu_k = \frac{1}{m}y_k\trp \lambda_k$) at each iteration.
% \end{problem}

Test your algorithm on the simple problem
\begin{align*}
\text{minimize }\qquad \frac{1}{2}x_1^2 + x_2^2 - &x_1x_2 - 2x_1 - 6x_2\\
\text{subject to }\qquad
-x_1-x_2 &\geq -2,\\
x_1-2x_2 &\geq -2,\\
-2x_1-x_2&\geq -3,\\
x_1, x_2 &\geq 0.
\end{align*}
In this case, we have for the objective function matrix $Q$ and vector $\c$,
\[
Q = \begin{bmatrix}
1 & -1\\
-1 & 2
\end{bmatrix},
\qquad
\c = \begin{bmatrix}
-2\\
-6
\end{bmatrix}.
\]
The constraint matrix $A$ and vector $\b$ are given by:
\[
A = \begin{bmatrix}
-1 & -1\\
1 & -2\\
-2 & -1\\
1 & 0\\
0 & 1
\end{bmatrix},
\qquad
\b = \begin{bmatrix}
-2\\
-2\\
-3\\
0\\
0
\end{bmatrix}.
\]
Use $\x = [.5, .5]$ as the initial guess.
The correct minimizer is $\left[\frac{2}{3}, \frac{4}{3}\right].$

(Hint: You may want to print out the duality measure $\nu$ to check the progress of the iteration).
\end{problem}
% We solve this problem with the following code:
% \begin{lstlisting}
% >>> # test out our algorithm
% >>> Q = np.array([[1,-1.],[-1,2]])
% >>> c = np.array([-2,-6.])
% >>> A = np.array([[-1, -1], [1, -2.], [-2, -1], [1, 0], [0,1]])
% >>> b = np.array([-2, -2, -3., 0, 0])
% >>> x = np.array([.5, .5])
% >>> y = np.ones(5)
% >>> mu = np.ones(5)
% >>> print qInteriorPoint(Q, c, A, b, (x,y,mu), niter=7, verbose=True)
% (array([ 0.66666668,  1.3333333 ]), -8.222222138159772)
% \end{lstlisting}
% Check that your function gives the same output.

\begin{info}
The Interior Point methods presented in this and the preceding labs are only special cases of the more general Interior Point algorithm.
The general version can be used to solve many convex optimization problems, provided that one can derive the corresponding KKT conditions and duality measure $\nu$.
\end{info}

\section*{Application: Optimal Elastic Membranes}
The properties of elastic membranes (stretchy materials like a thin rubber sheet) are of interest in certain fields of mathematics and various sciences.
A mathematical model for such materials can be used by biologists to study interfaces in cellular regions of an organism or by engineers to design tensile structures.
Often we can describe configurations of elastic membranes as a solution to an
optimization problem.
As a simple example, we will find the shape of a large circus tent by solving a quadratic constrained optimization problem using our Interior Point method.

Imagine a large circus tent held up by a few poles.
We can model the tent by a square two-dimensional grid, where each grid point has an associated number that gives the height of the tent at that point.
At each grid point containing a tent pole, the tent height is constrained to be at least as large as the height of the tent pole.
At all other grid points, the tent height is simply constrained to be greater than zero (ground height).
In Python, we can store a two-dimensional grid of values as a simple two-dimensional array.
We can then flatten this array to give a one-dimensional vector representation of the grid.
If we let $\x$ be a one-dimensional array giving the tent height at each grid point, and $L$ be the one-dimensional array giving the underlying tent pole structure (consisting mainly of zeros, except at the grid points that contain a tent pole), we have the linear constraint: \[\x \succeq L.\]

\begin{figure}
\includegraphics[width=\textwidth]{figures/tent.pdf}
\caption{Tent pole configuration (left) and optimal elastic tent (right).}
\label{fig:tent}
\end{figure}

The theory of elastic membranes claims that such materials tend to naturally minimize a quantity known as the \emph{Dirichlet energy}.
This quantity can be expressed as a quadratic function of the membrane.
Since we have modeled our tent with a discrete grid of values, this energy function has the form \[\frac{1}{2}\x\trp  H \x + \c\trp  \x,\] where $H$ is a particular positive semidefinite matrix closely related to Laplace's Equation, $\c$ is a vector whose entries are all equal to $-(n-1)^{-2}$, and $n$ is the side length of the grid.
Our circus tent is therefore given by the solution to the quadratic constrained optimization problem:
\begin{align*}
\text{minimize }\qquad &\frac{1}{2}\x\trp  H \x + \c\trp  \x\\
\text{subject to }\qquad &\x \succeq L.\\
\end{align*}
See Figure \ref{fig:tent} for an example of a tent pole configuration and the corresponding tent.

We provide the following function for producing the Dirichlet energy matrix $H$.
\begin{lstlisting}
from scipy.sparse import spdiags
def laplacian(n):
    """Construct the discrete Dirichlet energy matrix H for an n x n grid."""
    data = -1*np.ones((5, n**2))
    data[2,:] = 4
    data[1, n-1::n] = 0
    data[3, ::n] = 0
    diags = np.array([-n, -1, 0, 1, n])
    return spdiags(data, diags, n**2, n**2).toarray()
\end{lstlisting}
Now we initialize the tent pole configuration for a grid of side length $n$, as well as initial guesses for $\x$, $\y$, and $\Mu$.
\begin{lstlisting}
# Create the tent pole configuration.
>>> L = np.zeros((n,n))
>>> L[n//2-1:n//2+1,n//2-1:n//2+1] = .5
>>> m = [n//6-1, n//6, int(5*(n/6.))-1, int(5*(n/6.))]
>>> mask1, mask2 = np.meshgrid(m, m)
>>> L[mask1, mask2] = .3
>>> L = L.ravel()

# Set initial guesses.
>>> x = np.ones((n,n)).ravel()
>>> y = np.ones(n**2)
>>> mu = np.ones(n**2)
\end{lstlisting}
We leave it to you to initialize the vector $\c$, the constraint matrix $A$,
%(it's just the identity matrix of appropriate size)
and to initialize the matrix $H$ with the \li{laplacian()} function.
We can solve and plot the tent with the following code:
\begin{lstlisting}
>>> from matplotlib import pyplot as plt
>>> from mpl_toolkits.mplot3d import axes3d

# Calculate the solution.
>>> z = qInteriorPoint(H, c, A, L, (x,y,mu))[0].reshape((n,n))

# Plot the solution.
>>> domain = np.arange(n)
>>> X, Y = np.meshgrid(domain, domain)
>>> fig = plt.figure()
>>> ax1 = fig.add_subplot(111, projection='3d')
>>> ax1.plot_surface(X, Y, z,  rstride=1, cstride=1, color='r')
>>> plt.show()
\end{lstlisting}

\begin{problem}
Solve the circus tent problem with the tent pole configuration given above, for grid side length $n = 15$.
Plot your solution.
\end{problem}

\section*{Application: Markowitz Portfolio Optimization}
Suppose you have a certain amount of money saved up, with no intention of consuming it any time soon.
What will you do with this money?
If you hide it somewhere in your living quarters or on your person, it will lose value over time due to inflation, not to mention you run the risk of burglary or accidental loss.
A safer choice might be to put the money into a bank account.
That way, there is less risk of losing the money, plus you may even add to your savings through interest payments from the bank.
You could also consider purchasing bonds from the government or stocks from various companies, which come with their own sets of risks and returns.
Given all of these possibilities, how can you invest your money in such a way that maximizes the return (i.e. the wealth that you gain over the course of the investment) while still exercising caution and avoiding excessive risk?
Economist and Nobel laureate Harry Markowitz developed the mathematical underpinnings and answer to this question in his work on modern portfolio theory.

A \emph{portfolio} is a set of investments over a period of time.
Each investment is characterized by a financial asset (such as a stock or bond) together with the proportion of wealth allocated to the asset.
An asset is a random variable, and can be described as a sequence of values over time.
The variance or spread of these values is associated with the risk of the asset, and the percent change of the values over each time period is related to the return of the asset.
For our purposes, we will assume that each asset has a positive risk, i.e. there are no \emph{riskless} assets available.

Stated more precisely, our portfolio consists of $n$ risky assets together with an allocation vector
$\x = (x_1,\ldots,x_n)\trp $, where $x_i$ indicates the proportion of wealth we invest in asset $i$.
By definition, the vector $\x$ must satisfy \[\sum_{i=1}^n x_i = 1.\]
Suppose the $i$th asset has an expected rate of return $\mu_i$ and a standard deviation $\sigma_i$.
The total return on our portfolio, i.e. the expected percent change in our invested wealth over the investment period, is given by
\[\sum_{i=1}^n \mu_ix_i.\]
We define the risk of this portfolio in terms of the covariance matrix $Q$ of the $n$ assets: \[\sqrt{\x\trp  Q \x}.\]
The covariance matrix $Q$ is always positive semidefinite and captures the variance and correlations of the assets.

Given that we want our portfolio to have a prescribed return $R$, there are many possible allocation vectors $\x$ that make this possible.
It would be wise to choose the vector minimizing the risk.
We can state this as a quadratic program:
\begin{align*}
\text{minimize }\qquad &\frac{1}{2}\x\trp Q\x\\
\text{subject to }\qquad &\sum_{i=1}^n x_i = 1\\
&\sum_{i=1}^n \mu_ix_i = R.\\
\end{align*}
Note that we have slightly altered our objective function for convenience, as minimizing $\frac{1}{2}\x\trp Q\x$ is equivalent to minimizing $\sqrt{\x\trp Q\x}$.
The solution to this problem will give the portfolio with least risk having a return $R$.
Because the components of $\x$ are not constrained to be nonnegative, the solution may have some negative entries.
This indicates short selling those particular assets.
If we want to disallow short selling, we simply include nonnegativity constraints, stated in the following problem:
\begin{align*}
\text{minimize }\qquad &\frac{1}{2}\x\trp Q\x\\
\text{subject to }\qquad &\sum_{i=1}^n x_i = 1\\
&\sum_{i=1}^n \mu_ix_i = R\\
&\x \succeq \0.
\end{align*}

Each return value $R$ can be paired with its corresponding minimal risk $\sigma$.
If we plot these risk-return pairs on the risk-return plane, we obtain a hyperbola.
In general, the risk-return pair of any portfolio, optimal or not, will be found in the region bounded on the left by the hyperbola.
The positively-sloped portion of the hyperbola is known as the
\emph{efficient frontier}, since the points there correspond to optimal portfolios.
Portfolios with risk-return pairs that lie to the right of the efficient frontier are inefficient portfolios, since we could either increase the return while keeping the risk constant, or we could decrease the risk while keeping the return constant.
See Figure \ref{fig:frontier}.

\begin{figure}[H]
\includegraphics[width=0.75\textwidth]{figures/frontier.pdf}
\caption{Efficient frontier on the risk-return plane.}
\label{fig:frontier}
\end{figure}

One weakness of this model is that the risk and return of each asset is in general unknown.
After all, no one can predict the stock market with complete certainty.
There are various ways of estimating these values given past stock prices, and we take a very straightforward approach.
Suppose for each asset, we have $k$ previous return values of the asset. That is, for asset $i$, we have the data vector\[y^i = [y^i_1,\,\, \ldots, \,\,y^i_k]\trp .\]
We estimate the expected rate of return for asset $i$ by simply taking the average of $y_1,\ldots,y_k$, and we estimate the variance
of asset $i$ by taking the variance of the data.
We can estimate the covariance matrix for all assets by taking the covariance matrix of the vectors $y^1,\ldots,y^n$.
In this way, we obtain estimated values for each $\mu_i$ and $Q$.

\begin{problem}
The text file \li{portfolio.txt} contains historical stock data for several assets (U.S. bonds, gold, S\&P 500, etc).
In particular, the first column gives the years corresponding to the data, and the remaining eight columns give the historical returns
of eight assets over the course of these years.
Use this data to estimate the covariance matrix $Q$ as well as the expected rates of return $\mu_i$ for each asset.
Assuming that we want to guarantee an expected return of $R = 1.13$ for our portfolio, find the optimal portfolio both with and without short selling.

Since the problem contains both equality and inequality constraints, use the QP solver in CVXOPT rather than your \li{qInteriorPoint()} function.

Hint: Use \li{numpy.cov()} to compute Q.
\end{problem}


\subsection{Axiom of infinity}

The axiom of infinity states that:

\(\exists I (\varnothing \in I \land \forall x \in I((x\lor \{x\})\in I))\)

There exists a set, called the infinite set. This contains the empty set, and for all elements in \(I\) the set also contains the successor to it.

\subsubsection{Sequential function}

Let's define the sequential function:

\(s(n):=\{n\lor \{n\}\}\)

We can now rewrite the axiom of infinity as:

\(\exists \mathbb{N} (\varnothing \in \mathbb{N} \land \forall x \in \mathbb{N}(s(x)\in \mathbb{N}))\)

\subsubsection{Zero}

This set contains the null set: \(\varnothing \in \mathbb{N} \).

Zero is defined as the empty set.

\(0:=\{\}\)

\subsubsection{Natural numbers}

For all elements in the infinite set, there also exists another element in the infinite set: \(\forall x \in \mathbb{N}((x\lor \{x\})\in \mathbb{N}) \).

We then define all sequential numbers as the set of all preceding numbers. So:

\(1:=\{0\}=\{\{\}\}\)

\(2:=\{0,1\}=\{\{\},\{\{\}\}\}\)

\(3:=\{0,1,2\}=\{\{\},\{\{\}\},\{\{\},\{\{\}\}\}\}\)

\subsubsection{Existence of natural numbers}

Does each natural number exist? We know the infinite set exists, and we also know the axiom schema of specification:

Point is: For each set, all finite subsets exist. PROVE ELSEWHERE

\subsubsection{From infinite set to natural set}

We don’t know I is limited to natural numbers. Could contain urelements etc.

\subsubsection{More}

Infinite set axiom written using N. should be I

I could be superset of N, for example set of all natural numbers, and also the set containing the set containing 2.

Can extract N using axiom of specification

We need a way to define the set of natural numbers:

\(\forall n (n\in \mathbb{N}\leftrightarrow ([n=\emptyset \lor \exists k (n=k\lor \{k\})]\land ))\)

If we can can define N, we can show it exists from specicication

\(\forall x \exists s [P(x)\leftrightarrow (x\in s)]\)

\(\forall n \exists s [n\in N \leftrightarrow (n\in s)]\)


\section{Convolutional Neural Networks}
\begin{frame}{\insertsec}
    This type of networks are a bit different because instead of computing the activations by
    multiplying the previous ones with the weight matrix it convolutes the previous activations
    (denoted by $*$):

    \begin{align*}
        \bm{a}^{[l]} &= g(\bm{W}^{[l]}\cdot \bm{a}^{[l - 1]} + \bm{b}^{[l]}) \\
        \text{Becomes:} \\
        \bm{a}^{[l]} &= g(\bm{W}^{[l]} * \bm{a}^{[l-1]} + \bm{b}^{[l]})
    \end{align*}
\end{frame}

\begin{frame}{Convolution Operation}
    \begin{alertblock}{Important}
        What we call \textit{Convolution} in ML it's actually the \textit{Cross-Correlation} operation
        in signal analysis.
    \end{alertblock}

    \begin{align*}
        \left(
        \begin{array}{cccccc}
            10 & \cellcolor{green!20}10 & \cellcolor{green!20}10 & \cellcolor{green!20}0 & 0 & 0 \\
            10 & \cellcolor{green!20}10 & \cellcolor{green!20}10 & \cellcolor{green!20}0 & 0 & 0 \\
            10 & \cellcolor{green!20}10 & \cellcolor{green!20}10 & \cellcolor{green!20}0 & 0 & 0 \\
            10 & 10 & 10 & 0 & 0 & 0 \\
            10 & 10 & 10 & 0 & 0 & 0 \\
            10 & 10 & 10 & 0 & 0 & 0 \\
        \end{array}
        \right)
        *
        \left(
        \begin{array}{ccc}
            1 & 0 & -1 \\
            1 & 0 & -1 \\
            1 & 0 & -1 \\
        \end{array}
        \right)
        =
        \left(
        \begin{array}{cccc}
            0 & \cellcolor{green!20}30 & 30 & 0 \\
            0 & 30 & 30 & 0 \\
            0 & 30 & 30 & 0 \\
            0 & 30 & 30 & 0 \\
        \end{array}
        \right)
    \end{align*}
    $$
        10 \cdot 1 + 10 \cdot 1 + 10 \cdot 1 +
        10 \cdot 0 + 10 \cdot 0 + 10 \cdot 0 +
        0 \cdot -1 + 0 \cdot -1 + 0 \cdot -1 = 30
    $$
\end{frame}

\begin{frame}{Volume}
    With convolutions the Neural Network parameters ($w$) are the convolution matrix elements
    and so we do multiple convolutions to filter different features.
    \input{drawings/convolutional_block.tikz.tex}
\end{frame}

\begin{frame}{Pooling operation}
    We use pooling to reduce the volume width and height size
    \begin{block}{Max-Pooling}
        \begin{align*}
            \operatorname{max\,pool}
            \left(
            \begin{array}{cccc}
                \cellcolor{green!20} 0 & \cellcolor{green!20} 7 & 12 &  9 \\
                \cellcolor{green!20}15 & \cellcolor{green!20} 3 & 22 &  6 \\
                 2 & 12 &  8 &  3 \\
                 6 &  2 & 18 &  3 \\
            \end{array}
            \right)
            =
            \left(
            \begin{array}{cc}
                \cellcolor{green!20}15 & 22 \\
                12 & 18
            \end{array}
            \right)
        \end{align*}
    \end{block}

    \begin{block}{Avg-Pooling}
        \begin{align*}
            \operatorname{avg\,pool}
            \left(
            \begin{array}{cccc}
                \cellcolor{green!20} 0 & \cellcolor{green!20} 7 & 12 &  9 \\
                \cellcolor{green!20}15 & \cellcolor{green!20} 3 & 22 &  6 \\
                 2 & 12 &  8 &  3 \\
                 6 &  2 & 18 &  3 \\
            \end{array}
            \right)
            =
            \left(
            \begin{array}{cc}
                \cellcolor{green!20}6.25 & 12.25 \\
                5.50 & 8.00
            \end{array}
            \right)
        \end{align*}
    \end{block}
\end{frame}

\subsection{Residual Networks}
\begin{frame}{\insertsubsec}
    They are a type of network that can learn the identity function thus allowing us to do
    very deep networks.
    \input{drawings/resNet.tikz.tex}

    \begin{align*}
        \bm{a}^{[l+2]} =
        g(\underbrace{W^{[l+2]} \cdot \bm{a}^{[l + 1]} + \bm{b}^{[l + 2]}}
        _{0 \text{ if using regularization}} + \bm{a}^{[l]})
    \end{align*}
\end{frame}

\subsection{Inception block}
\begin{frame}{\insertsubsec}
    \begin{figure}[H]
        \includegraphics[width=\textwidth]{images/GoogLeNet3}
        \caption{GoogLeNet Inception Module}
    \end{figure}
\end{frame}

\subsection{Methods for improving CNN}
\begin{frame}{\insertsubsec}
    \begin{onlyenv}<1>
        \begin{block}{Transfer Learning}
            With transfer learning it's possible to download someone elses pre-trained weights,
            remove the softmax (output) layer and add your own and then only train this last layer.
            \begin{itemize}
                \item \textbf{Fine tuning}: continue to train the weights then adding your own data
                \item The exit values of the last layer can also be pre-computed and stored to disk
                to speed-up learning
            \end{itemize}
        \end{block}
    \end{onlyenv}
    \begin{onlyenv}<2>
        \begin{block}{Data augmentation}
            Apply different transformations to the image
            \begin{itemize}
                \item Mirroring
                \item Random cropping
                \item Rotation
                \item Shearing
                \item Color shifting
                \begin{itemize}
                    \item Distort the color channels
                    \item Apply different types of illumination, makes the algorithm more robust
                \end{itemize}
            \end{itemize}
        \end{block}
    \end{onlyenv}
\end{frame}