\documentclass{article}

\include{stddefs}
\include{imodefs}

\chapterno{3}

\begin{document}

\chapter{Matrices}\label{Chapter:Matrices}

Handling linear equations and keeping track of the unknowns can be a pain. At
a certain point one needs to simplify the notation. This is done introducing
matrices.

For example, the system of equations
\begin{equation}\label{equ2l}
\begin{matrix}
&&  &2y &+ &4z &= &-2\\
&3x &+ &2y &+ &7z &= &4
\end{matrix}
\end{equation}
can be represented by the rectangular array (matrix)
\begin{equation}\label{matrix1}
\begin{pmatrix}
0 & 2 & 4 & -2\\
3 & 2 & 7 & 4
\end{pmatrix}
\end{equation}
of numbers. Many of the operations we do to solve linear equations might as well be done on this array forgetting
about the unknowns.

\section{Matrices}

\subsection{Definitions}
A rectangular array of numbers is called a \index{matrix}\emph{matrix}. A matrix with $m$ \emph{rows} and $n$
\emph{columns} is called an $m\times n$ ($m$ by $n$) matrix.
The notation for an $m\times n$ matrix $A$ is
\begin{equation}\label{matr}
A =
\begin{pmatrix}
a_{11} & \cdots &a_{1j}& \cdots& a_{1 n} \\
\vdots & \ddots &\vdots & \ddots & \vdots\\
a_{i1} & \cdots &a_{ij}& \cdots& a_{i n} \\
\vdots & \ddots &\vdots & \ddots & \vdots\\
a_{m1} & \cdots &a_{mj}& \cdots& a_{m n}
\end{pmatrix},
\end{equation}
where $A_{ij} = a_{i j}$ denotes the number or \emph{entry} in the  $i$-th row and $j$-th column. If the matrix in \eqref{matrix1}
is denoted $A$, then it has $2$ rows and $4$ columns with $A_{14} = -2$.

Two matrices are equal if they have the same number of rows and columns and their entries are
identical.

Sage is built on top of python and has access to all its libraries. A
very useful (and famous) library for handling matrices is \texttt{numpy}. Here is how
the matrix in \eqref{matrix1} is entered in \texttt{numpy}.

\begin{sage}
import numpy as np

m = np.matrix( [[0, 2, 4, -2], [3, 2, 7, 4]] )
print(m)
\end{sage}

\begin{enumerate}

\item
  A matrix whose entries are all $0$ is called a zero matrix. It is denoted simply by $0$, when
  it is clear from the context what its numbers of rows and columns are.

\item
  A matrix is called  \emph{quadratic}\index{quadratic matrix} if it has an equal number of rows and columns.
  The first two matrices below are quadratic, whereas the third is not.
$$
\begin{pmatrix} 1 \end{pmatrix}, \qquad
\begin{pmatrix} 1 & 2 & 3\\ 4 & 5 & 6\\ 7 & 8 & 9\end{pmatrix}, \qquad
\begin{pmatrix} 0 & 1 & 0\\ 1 & 0 & 1\end{pmatrix}.
$$
\item
\label{diagonalmat}
The \emph{diagonal} in a matrix is defined as the entries in the matrix with the same row- and column indices.
Below we have a $3\times 4$ matrix with the diagonal elements marked
$$
\begin{pmatrix}
\color{red}{1} & 3 & 0 & 1\\
3 & \color{red}{2} & 1 & 5\\
1 & 0 & \color{red}{3} & 6
\end{pmatrix}.
$$
A matrix is called a \index{diagonal matrix}\emph{diagonal matrix}, if all its entries outside the
diagonal are $=0$. Below is an example of a square diagonal matrix
$$
\begin{pmatrix}
1 & 0 & 0\\
0 & 2 & 0\\
0 & 0 & 3
\end{pmatrix}.
$$
\item
A matrix is called a \index{row vector}\emph{row vector} if it has only one row. For example,
$$
\begin{pmatrix}
1 & 2 & 3
\end{pmatrix}
$$
is a row vector with three columns.
\item
A matrix is called a \index{column vector}\emph{column vector} if it has only one column.
For example,
$$
\begin{pmatrix}
1\\ 2 \\ 3
\end{pmatrix}
$$
is a column vector with three rows.
\item
  The rows in a matrix are called the  \emph{row vectors} of the matrix.
The $i$-th row in a matrix $A$ is denoted $A_i$.
The matrix $A$ in \eqref{matrix1} contains the row vectors
$$
A_1 = \begin{pmatrix}
0 & 2 & 4 & -2
\end{pmatrix}
\qquad\text{and}\qquad
A_2 = \begin{pmatrix}
3 & 2 & 7 & 4
\end{pmatrix}.
$$
\item
  The columns in a matrix are called the \emph{column vectors} of the matrix.
The $j$-th column in a matrix $A$ is denoted \footnote{$A^j$}{Not to be confused with powers of the matrix $A$ introduced later.}.
The matrix $A$ in \eqref{matrix1} contains the column vectors
$$
A^1 = \begin{pmatrix}
0 \\ 3
\end{pmatrix},\quad
A^2 =\begin{pmatrix}
2 \\ 2
\end{pmatrix},\quad
A^3 =\begin{pmatrix}
4 \\ 7
\end{pmatrix}\quad\text{and}\quad
A^4 = \begin{pmatrix}
-2 \\ 4
\end{pmatrix}.
$$

\item
  A row- or column vector is referred to as a \index{vector}\emph{vector}.
\item
  Even though we have used the notation $\RR^n$ for the $n$-th cartesian product of $\RR$, we
  will use $\RR^n$ henceforth to denote the set of column vectors with $n$ rows (entries). This
  definition is almost identical with the previous one, except that the tuple is formatted as
  a column vector.

  Illustrated by an example,
  $$
  \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix} \in \RR^3\qquad\text{instead of}\qquad (1, 2, 3)\in \RR^3.
  $$
\end{enumerate}

\section{Linear maps}\label{sectionLM}

In the first chapter we encountered a miniature version of a neural
network. Neural networks are generally incredibly complicated functions
from $\RR^n$ to $\RR^m$. The function $f:\RR^2\rightarrow \RR^2$
given by
$$
f\begin{pmatrix}
  x \\ y
\end{pmatrix} =
\begin{pmatrix}
  x^7 y + \cos(x y) e^{x^2 + y^2 -1}\\
  2 x y^2 - \sin(x + y) (x^3 + y^3)
\end{pmatrix},
$$
even though it looks complicated, is simple in comparison.

You probably agree that the function $g:\RR^2\rightarrow \RR^2$ given by
$$
g\begin{pmatrix}
  x \\ y
\end{pmatrix} =
\begin{pmatrix}
  2 x + 3 y\\
  3 x - 2 y
\end{pmatrix}
$$
is even simpler. This function (or map) is an example of a linear map.
In general, a \emph{linear map}
$f: \RR^n\rightarrow \RR^m$ has the form
$$
f\begin{pmatrix}
  x_1 \\ \vdots \\ x_n
\end{pmatrix} =
\begin{pmatrix}
  a_{11} x_1 + \cdots + a_{1 n} x_n\\
  \vdots \\
  a_{m1} x_1 + \cdots + a_{m n} x_n
\end{pmatrix},
$$
where $a_{11}, \dots, a_{mn}$ are $m n$ real numbers.

Using matrices we will use the notation
$$
\begin{pmatrix}
a_{11} & \cdots & a_{1n}\\
\vdots & \ddots & \vdots\\
a_{m1} & \cdots & a_{mn}
\end{pmatrix}
\begin{pmatrix} x_1 \\ \vdots \\ x_n\end{pmatrix} =
\begin{pmatrix}
  a_{11} x_1 + \cdots + a_{1 n} x_n\\
  \vdots \\
  a_{m1} x_1 + \cdots + a_{m n} x_n
\end{pmatrix}.
$$

In this way, we can write the map $f$ as
$$
f(v) = A v,
$$
where $A$ is the $m\times n$ matrix
$$
\begin{pmatrix}
a_{11} & \cdots & a_{1n}\\
\vdots & \ddots & \vdots\\
a_{m1} & \cdots & a_{mn}
\end{pmatrix}
$$
and $v$ is the vector
$$
\begin{pmatrix} x_1 \\ \vdots \\ x_n\end{pmatrix}
$$
in $\RR^n$.

Basically a linear map is a system of linear equations without the right hand side (including $=$).
In fact, we may write the system of linear equations in \eqref{equ2l} as
$$
\begin{pmatrix}
0 & 2 & 4\\
3 & 2 & 7
\end{pmatrix}
\begin{pmatrix}
  x \\ y \\ z
\end{pmatrix}
=
\begin{pmatrix}
  -2 \\ 4
\end{pmatrix}.
$$
\beginshex
Let $f: \RR^2\rightarrow \RR^2$ be the linear map given by the $2\times 2$ matrix
$$
\begin{pmatrix}
  1 & 2\\
  3 & 4
\end{pmatrix}.
$$
Does there exist $u\in \RR^2$, such that
$$
f(u) = \begin{pmatrix} 3 \\ 7 \end{pmatrix}?
$$
Quite generally, can we find $u\in \RR^2$, such that
$$
f(u) = \begin{pmatrix} b_1 \\ b_2 \end{pmatrix}?
$$
for arbitrary $b_1, b_2\in \RR$?
\endshex

\beginshex
Suppose you know that $f: \RR^n\rightarrow \RR^m$ is a linear map and
that you have a black box giving you output $f(v)\in \RR^m$ if you
supply the input $v\in \RR^n$. How would you find
the matrix defining $f$?
\endshex

\section{Matrix multiplication}

Suppose we are given two linear maps $f: \RR^2\rightarrow \RR^2$ and
$g:\RR^2\rightarrow \RR^2$. Then it turns out that the composition
$f\circ g: \RR^2\rightarrow \RR^2$ is also a linear map. A word of advice:
the computations below look large and intimidating. They are not. It
is important that you carry them out on your own. Do not look and copy or tell
yourself that it looks okay. Do the computations yourself and ask me
or fellow students if you get stuck.

Let us look at an example. Suppose that
$$
g\begin{pmatrix} x\\ y \end{pmatrix} =
\begin{pmatrix}
  2 & 3 \\
  -1 & -2
\end{pmatrix}
\begin{pmatrix}
  x \\ y
\end{pmatrix}\qquad
\text{and}
\qquad
f\begin{pmatrix} u\\ v \end{pmatrix} =
\begin{pmatrix}
  1 & 2 \\
  1 & -2
\end{pmatrix}
\begin{pmatrix}
  u \\ v
\end{pmatrix}.
$$
Then
\begin{align*}
(f\circ g)\begin{pmatrix} x\\ y \end{pmatrix} &=
f\left(g\begin{pmatrix} x\\ y \end{pmatrix}\right) =
f\left(\begin{pmatrix}
  2 & 3 \\
  -1 & -2
\end{pmatrix}
\begin{pmatrix}
  x \\ y
\end{pmatrix}\right)
=
\begin{pmatrix}
  1 & 2 \\
  1 & -2
\end{pmatrix}
      \left(
\begin{pmatrix}
  2 & 3 \\
  -1 & -2
\end{pmatrix}
\begin{pmatrix}
  x \\ y
\end{pmatrix}
  \right)\\
  \\
  &=
\begin{pmatrix}
  1 & 2 \\
  1 & -2
\end{pmatrix}
\begin{pmatrix}
  2 x + 3 y\\
  -x - 2y
\end{pmatrix} =
  \begin{pmatrix}
    - y \\
    4 x + 7 y
  \end{pmatrix}
  =
  \begin{pmatrix}
    0 & -1\\
    4 & 7
  \end{pmatrix}
  \begin{pmatrix}
    x \\ y
  \end{pmatrix}.
\end{align*}

In terms of the matrices of the linear maps, we write this as
\begin{equation}\label{moperation}
\begin{pmatrix}
1 & 2\\
1 & -2
\end{pmatrix}
\begin{pmatrix}
2 & 3\\
-1 & -2
\end{pmatrix}
=
\begin{pmatrix}
0 & -1\\
4 & 7
\end{pmatrix}
\end{equation}

There is nothing special about the numbers in this example. We might as well
do the computation in general: suppose that

$$
g\begin{pmatrix} x\\ y \end{pmatrix} =
\begin{pmatrix}
  b_{11} & b_{12} \\
  b_{21} & b_{22}
\end{pmatrix}
\begin{pmatrix}
  x \\ y
\end{pmatrix}\qquad
\text{and}
\qquad
f\begin{pmatrix} u\\ v \end{pmatrix} =
\begin{pmatrix}
  a_{11} & a_{12} \\
  a_{21} & a_{22}
\end{pmatrix}
\begin{pmatrix}
  u \\ v
\end{pmatrix}.
$$
Then
\begin{align*}
(f\circ g)\begin{pmatrix} x\\ y \end{pmatrix} &=
f\left(g\begin{pmatrix} x\\ y \end{pmatrix}\right) =
f\left(\begin{pmatrix}
  b_{11} & b_{12} \\
  b_{21} & b_{22}
\end{pmatrix}
\begin{pmatrix}
  x \\ y
\end{pmatrix}\right)
=
\begin{pmatrix}
  a_{11} & a_{12} \\
  a_{21} & a_{22}
\end{pmatrix}
      \left(
\begin{pmatrix}
  b_{11} & b_{12} \\
  b_{21} & b_{22}
\end{pmatrix}
\begin{pmatrix}
  x \\ y
\end{pmatrix}
  \right)\\
  \\
  &=
\begin{pmatrix}
  a_{11} & a_{12} \\
  a_{21} & a_{22}
\end{pmatrix}
\begin{pmatrix}
  b_{11} x + b_{12} y\\
  b_{21} x + b_{22} y
\end{pmatrix} =
  \begin{pmatrix}
a_{11} (b_{11} x + b_{12} y) + a_{12} (b_{21} x + b_{22} y)\\
a_{21} (b_{11} x + b_{12} y) + a_{22} (b_{21} x + b_{22} y)
\end{pmatrix}\\
  \\
  &=
  \begin{pmatrix}
(a_{11} b_{11} + a_{12} b_{21}) x + (a_{11}b_{12} + a_{12} b_{22}) y \\
(a_{21} b_{11} + a_{22} b_{21}) x + (a_{21} b_{12} + a_{22} b_{22}) y
\end{pmatrix}\\
  \\
  &=
  \begin{pmatrix}
a_{11} b_{11} + a_{12} b_{21} & a_{11}b_{12} + a_{12} b_{22} \\
a_{21} b_{11} + a_{22} b_{21} &  a_{21} b_{12} + a_{22} b_{22}
\end{pmatrix}
\begin{pmatrix}
  x \\ y
\end{pmatrix}.
\end{align*}

Again, in terms of the matrices of the linear maps, we write this as
\begin{equation}\label{matmult}
\begin{pmatrix}
a_{11} & a_{12}\\
\color{blue}{a_{21}} & \color{red}{a_{22}}
\end{pmatrix}
\begin{pmatrix}
\color{blue}{b_{11}} & b_{12}\\
\color{red}{b_{21}} & b_{22}
\end{pmatrix}
=
\begin{pmatrix}
a_{11} b_{11} + a_{12} b_{21} & a_{11} b_{12} + a_{12} b_{22}\\
\color{blue}{a_{21} b_{11}} + \color{red}{a_{22} b_{21}} & a_{21} b_{12} + a_{22} b_{22}
\end{pmatrix}
\end{equation}

The equation above is the formula for matrix multiplication for two $2\times 2$ matrices, precisely as
it was introduced
by \url{Cayley}{https://en.wikipedia.org/wiki/Arthur_Cayley} around $1857$.

Upon closer inspection (and colored in \eqref{matmult} for $i=2$ and $j= 1$), you will see that
the number in the $i$-th row and $j$-th column in the product matrix is the
\emph{row-column multiplication}\index{row-column multiplication} between the $i$-th row and
the $j$-th column in the two matrices:

\begin{frameit}
The \emph{row-column multiplication} between a row vector
$$
x = (x_1 x_2 \dots x_n)\qquad\text{and a column vector}\qquad
y = \begin{pmatrix}
y_1 \\ y_2 \\ \vdots \\ y_n
\end{pmatrix}
$$
with the same number of entries is defined as
$$
x y = x_1 y_1 + x_2 y_2 + \cdots + x_n y_n.
$$
\end{frameit}

\index{matrix multiplikation}
\begin{definition}[emph]\label{defmatmult}
  Let $A$ be an  $\color{blue}{m}\times \color{brown}{n}$ matrix and $B$ an $\color{brown}{n}\times \color{red}{r}$ matrix.
  Then the matrix product $A B$ is defined as the $\color{blue}{m}\times\color{red}{r}$ matrix $C$ given by the
  row-column multiplication
$$
C_{ij} = A_i B^j = A_{i1} B_{1j} + A_{i2} B_{2j} + \cdots + A_{in} B_{nj}
$$
for $1\leq i \leq m$ and $1\leq j \leq r$.
\end{definition}

\begin{frameit}
If $A$ is an  $m\times n$ matrix and $B$ is an  $r\times s$, then the matrix product $A B$ only makes sense if
$n = r$: the number of columns in $A$ must equal the number of rows in $B$.
\end{frameit}

\begin{quizexercise}[showhide]
\begin{quiz}
\question
Suppose that
$$
A = \begin{pmatrix} 1 & 0 & 0\\ 0 & 1 & 0 \end{pmatrix}, \quad
B = \begin{pmatrix} 1 & 0\\ 0 & 1\end{pmatrix}, \quad
C = \begin{pmatrix} 1 & 1 & 1\end{pmatrix}, \quad\text{and}\quad
D = \begin{pmatrix} 1 \\ 1 \\ 1\end{pmatrix}
$$
Which of the matrix products below make sense?
\answer{T}
$B A$
\answer{F}
$A B$
\answer{T}
$C D$
\answer{T}
$D C$
\answer{F}
$C A$
\answer{T}
$A D$
\end{quiz}
\end{quizexercise}

\begin{video}
  I have been told that my pronunciation of \emph{column} in the video below is wrong. In the area of the US, where I got my PhD, people for some reason had this (Irish?)
  \url{rare pronunciation}{https://en.wiktionary.org/wiki/column\#Pronunciation}.

\youtube{KVpBEynN3IM}
\end{video}

Using matrix product notation, the system of linear equations in \eqref{equ2l} can now
be written as
$$
\begin{pmatrix}
0 & 2 & 4\\
3 & 2 & 7
\end{pmatrix}
\begin{pmatrix} x \\ y \\ z \end{pmatrix} =
\begin{pmatrix} -2 \\ 4 \end{pmatrix}
$$
Here we multiply a $2\times 3$ with a $3\times 1$
matrix. The row-column multiplication gives the $2\times 1$ matrix
$$
\begin{pmatrix}
2 y + 4 z\\
3 x + 2 y + 7 z
\end{pmatrix}.
$$
This matrix must equal the $2\times 1$ matrix on the right hand side
for \eqref{equ2l} to be true.
This is in agreement with our convention for writing linear maps in
section \ref{sectionLM}.

\begin{quizexercise}[showhide]
\begin{quiz}
\question
Suppose that
$$
A =
\begin{pmatrix}
1 & 2 & 3\\
0 & 1 & 2\\
3 & x & 1
\end{pmatrix},
\quad
B =
\begin{pmatrix}
1& 1 & 1\\
2 & 2 & 2\\
0 & 1 & 1
\end{pmatrix}\quad\text{and}\quad
C = A B
$$
Which ones of the statements below are true?
\answer{F}
$C_{12} = 9$
\answer{T}
$C_{23} = 4$
\answer{T}
If $C_{32} = 4$, then $x = 0$.
\answer{F}
If $C_{31} = -1$, then $x=-1$.
\end{quiz}
\end{quizexercise}

\subsection{Matrix multiplication in \texttt{numpy}}

Matrix multiplication in \texttt{numpy} is represented by the function \texttt{dot}:

\begin{sage}
import numpy as np
A = np.matrix( [[1, 2], [1, -2]] )
B = np.matrix( [[2, 3], [-1, -2]] )
print("A =")
print(A)
print("B =")
print(B)
print("AB =")
print(np.dot(A, B))
\end{sage}

\subsection{The identity matrix}

\index{identity matrix}
The identity matrix $I_n$ of order $n$ is the $n\times n$ diagonal matrix with $1$
in the diagonal. Below is the identity matrix of order $5$.
$$
\begin{pmatrix}
1 & 0 & 0 & 0 & 0\\
0 & 1 & 0 & 0 & 0\\
0 & 0 & 1 & 0 & 0\\
0 & 0 & 0 & 1 & 0\\
0 & 0 & 0 & 0 & 1
\end{pmatrix}
$$

The identity matrix $I_n$ has the crucial property that
\begin{equation}\label{idmathident}
I_n A = A I_n = A
\end{equation}
for all $n\times n$ matrices $A$.

\begin{sage}
import numpy as np

I3 = np.identity(3)
A = np.matrix( [[1, 2, 3], [4, 5, 6], [7, 8, 9]] )
print("The identity matrix of order 3 is I3 = ")
print(I3)
print("The matrix A is ")
print(A)
print("The matrix product I3 A is")
print(np.dot(I3, A))
print("The matrix product A I3 is")
print(np.dot(A, I3))
\end{sage}

\beginshex
Prove that the two identities in \eqref{idmathident} are true for $n\times n$ matrices.
\endshex

\subsection{Examples of matrix multiplication}
Matrix multiplication is omnipresent in mathematics. Below we give an example,
which is a baby version of
Google's famous \url{page rank algorithm}{https://en.wikipedia.org/wiki/PageRank}.

\begin{example}\label{eksstokmatr}

  Suppose that $20$\% of the people living in the suburbs move to the big city and
  that $30$\% of the people living in the big city move to the suburbs per year.

  Aiming for a model using probabilities, let us be a bit more precise.

\begin{enumerate}[(i)]
\item
  If you live in the suburbs, the probability that you move to the big city is $0.2$,
\item
  If you live in the suburbs, the probability that you do not move is $0.8$.
\item
  If you live in the big city the probability that you move to the suburbs is $0.3$.
\item
  If you live in the big city the probability that you do not move is $0.7$.
\end{enumerate}
All of the above probabilities are per year and can be illustrated in the diagram below

\includegraphics{markov.svg}

We are interested in predicting, using this model, how many people live
in the big city and the suburbs given that we know how many people
live in the big city, $x_0$ and in the suburbs $y_0$ to begin with i.e.,
setting the time $t = 0$ (years).

How many people $x_1$ and $y_1$ live in the two places after the first year ($t=1$)?

The population of the big city will decrease by $30\%$, but there are newcomers amounting to
$20\%$ of the population in the suburbs. Therefore
$$
x_1 = 0.7 x_0 + 0.2 y_0.
$$
In the same way,
$$
y_1 = 0.3 x_0 + 0.8 y_0.
$$
Using matrix multiplication, these two equations can be written
$$
\begin{pmatrix} x_1 \\ y_1 \end{pmatrix} =
\begin{pmatrix}
0.7 & 0.2\\
0.3 & 0.8
\end{pmatrix}
\begin{pmatrix} x_0 \\ y_0 \end{pmatrix}.
$$
For $t=2$ years, we can repeat the procedure and the result becomes
\begin{align}\label{snyd}
\begin{pmatrix} x_2 \\ y_2 \end{pmatrix} &=
\begin{pmatrix}
0.7 & 0.2\\
0.3 & 0.8
\end{pmatrix}
\begin{pmatrix} x_1 \\ y_1 \end{pmatrix} =
\begin{pmatrix}
0.7 & 0.2\\
0.3 & 0.8
\end{pmatrix}
\left(\begin{pmatrix}
0.7 & 0.2\\
0.3 & 0.8
\end{pmatrix}
\begin{pmatrix} x_0 \\ y_0 \end{pmatrix}\right)\\
&=
\left(
\begin{pmatrix}
0.7 & 0.2\\
0.3 & 0.8
\end{pmatrix}
\begin{pmatrix}
0.7 & 0.2\\
0.3 & 0.8
\end{pmatrix}\right)
\begin{pmatrix} x_0 \\ y_0 \end{pmatrix} =
P^2 \begin{pmatrix} x_0 \\ y_0 \end{pmatrix},
\end{align}
where
\begin{equation}\label{stokmat}
P=\begin{pmatrix}
0.7 & 0.2\\
0.3 & 0.8
\end{pmatrix}.
\end{equation}
In general we have the formula

\begin{equation}[emph]\label{markovformel}
\begin{pmatrix} x_n \\ y_n \end{pmatrix} =
P^n
\begin{pmatrix} x_0 \\ y_0 \end{pmatrix},
\end{equation}

giving the distribution of the populations for $t = n$
years. Let us experiment a little:
\begin{align*}
P^2 &=
\begin{pmatrix}
0.55 & 0.3\\
0.45 & 0.7
\end{pmatrix}\\
P^3 = P P^2 &=
\begin{pmatrix}
0.475 & 0.35\\
0.525 & 0.65
\end{pmatrix}\\
P^4 = P P^3 &=
\begin{pmatrix}
0.4375 & 0.375\\
0.5625 & 0.625
\end{pmatrix}\\
&\vdots\\
P^{15} &=
\begin{pmatrix}
0.400018 & 0.399951\\
0.599982 & 0.600012
\end{pmatrix}\\
P^{16} &=
\begin{pmatrix}
0.400009 & 0.399994\\
0.599991 & 0.600006
\end{pmatrix}
\end{align*}

It seems that the distribution stabilizes around $40\%$ living
in the big city and $60\%$ living in the suburbs of the
original total population.

\begin{sage}
import numpy as np

P = np.matrix( [[0.7, 0.2], [0.3, 0.8]] )

print(P**16)
\end{sage}

The matrix $P$ is an example of a stochastic $2\times 2$ matrix.
In general, a square matrix is called a \emph{stochastic matrix} if its entries are
$\geq 0$ and the sum of the entries in its column vectors are $1$.
\end{example}

\begin{example}

  A simple example of the page rank algorithm is
  given in Example \ref{Googleprlineqs}. There you
  encountered the equations
  \begin{align*}
    T_2 &= T_1 + \tfrac{1}{2} T_4\\
    T_3 &= T_2\\
    T_4 &= T_3\\
    T_1 &= \tfrac{1}{2} T_4\\
    T_1 + T_2 + T_3 + T_4 &= 1.
  \end{align*}

  In terms of matrix multiplication the first four equations can be rewritten to
  $$
  \begin{pmatrix}
    0 & 0 & 0 & \tfrac{1}{2} \\[5pt]
    1 & 0 & 0 & \tfrac{1}{2}\\[5pt]
    0 & 1 & 0 & 0 \\[5pt]
    0 & 0 & 1 & 0
  \end{pmatrix}
  \begin{pmatrix}
    T_1 \\[5pt] T_2 \\[5pt] T_3 \\[5pt] T_4
  \end{pmatrix} =
  \begin{pmatrix}
    T_1 \\[5pt] T_2 \\[5pt] T_3 \\[5pt] T_4
  \end{pmatrix}.
  $$
  Putting
  $$
  P = \begin{pmatrix}
    0 & 0 & 0 & \tfrac{1}{2} \\[5pt]
    1 & 0 & 0 & \tfrac{1}{2}\\[5pt]
    0 & 1 & 0 & 0 \\[5pt]
    0 & 0 & 1 & 0
  \end{pmatrix}
  $$
  we get a stochastic matrix and  may again iterate and compute $P, P^2, P^3, \dots$.

\begin{sage}
import numpy as np
P = np.matrix( [[0, 0, 0, 0.5], [1, 0, 0, 0.5], [0, 1, 0, 0], [0, 0, 1, 0]] )

print(P**10)
print(P**20)
print(P**30)
print(P**40)
print(P**50)
\end{sage}

Is there a connection between the entries of $P^N$, where $N$ is very big and the solutions to the linear equations?
\end{example}

\beginshex
In the end of Example \ref{eksstokmatr} (above) a stochastic matrix is defined. Show that
the matrix product of two $n\times n$ stochastic matrices is a stochastic matrix.
\endshex

Below is an example, where  matrix multiplication occurs in networks.

\begin{example}
  Suppose we have five cities connected with roads as shown below

\includegraphics{bygraf.svg}

This network has a so called $5\times 5$ \emph{incidence matrix}, where city $i$ is associated with the
$i$-th row and $i$-th column. A $1$ in the matrix in the $(i, j)$ entry means that there is a road
from city $i$ to city $j$, whereas a $0$ means that city $i$ and city $j$ are not connected by a road:
$$
A = \begin{pmatrix}
0 & 1 & 1 & 0 & 0\\
1 & 0 & 1 & 1 & 0\\
1 & 1 & 0 & 1 & 1\\
0 & 1 & 1 & 0 & 1\\
0 & 0 & 1 & 1 & 0
\end{pmatrix}.
$$
Here
$$
A^2 =
\begin{pmatrix}
2 & 1 & 1 & 2 & 1 \\
1 & 3 & 2 & 1 & 2 \\
1 & 2 & 4 & 2 & 1 \\
2 & 1 & 2 & 3 & 1 \\
1 & 2 & 1 & 1 & 2
\end{pmatrix}\quad\text{and}\quad
A^3 =
\begin{pmatrix}
 2 & 5 & 6 & 3 & 3 \\
 5 & 4 & 7 & 7 & 3 \\
 6 & 7 & 6 & 7 & 6 \\
 3 & 7 & 7 & 4 & 5 \\
 3 & 3 & 6 & 5 & 2
\end{pmatrix}.
$$
What is the interpretation of $A^2, A^3$ and  $A^n$ in general?
It turns out that the entry $(i, j)$ in the matrix $A^n$ exactly is
the number of paths of length $n$ from city $i$ to city $j$.

For example, there are $3$ paths from city $1$ to city $5$ of length $3$
corresponding to the paths $1245, 1345, 1235$.  The $2$ paths from city $1$ to city $1$ of
length $3$ are $1231, 1321$ and the  $5$ paths of length $3$ from city $1$ to city
$2$ are $1342, 1242, 1312, 1212, 1232$.

\begin{hideinbutton}{A deeper explanation}
Suppose that we have a network with $m$ cities and incidence matrix $A$.

The general proof of the observations above in our special example, builds on the
fact that
a path of length $n$ from city $i$ to city $j$ has to end with a road from a neighboring city
$k$ to $j$.
For every one of these neighboring cities, we may count the number of paths of length $n-1$
from city $i$.
If $A^{n-1}_{gh}$ is the number of paths of length $n-1$ from city $g$ to city $h$,
then matrix multiplication tells us that
$$
A^n_{i j} = A^{n-1}_{i 1} A_{1 j} + \cdots + A^{n-1}_{i m} A_{m j}
$$
This number is exactly the number of paths of length $n$ from city $i$ to
city $j$, since
$A_{k j} = 1$ only when $k$ is a neighboring city to city $j$ (and $0$ otherwise).
\end{hideinbutton}
\end{example}

\section{Matrix arithmetic}\index{matrix arithmetic}

Matrix multiplication is very different from ordinary multiplication of numbers:
it is not \url{commutative}{https://en.wikipedia.org/wiki/Commutative_property}.
Consider the matrices
$$
A=
\begin{pmatrix}
0 & 1\\
0 & 0
\end{pmatrix}\qquad\text{and}\qquad
B =
\begin{pmatrix}
0 & 0\\
1 & 0
\end{pmatrix}.
$$
Then
$$
A B = \begin{pmatrix} 1 & 0 \\ 0 & 0\end{pmatrix}\qquad \text{and} \qquad
B A = \begin{pmatrix} 0 & 0 \\ 0 & 1\end{pmatrix}
$$
i.e.., $A B \neq B A$.

\begin{sage}
import numpy as np

A = np.matrix( [[1, 1], [0, 1]] )
B = np.matrix( [[1, 0], [1, 1]] )
print("A =")
print(A)
print("B =")
print(B)
print("AB=")
print(A*B)
print("BA=")
print(B*A)
\end{sage}

Addition of matrices is like ordinary addition, except that you add all the entries of
the involved matrices.

\subsection{Matrix addition}\index{addition of matrices}

Addition of two matrices with the same number of rows and columns is defined below.
$$
\begin{pmatrix}
a_{11} & \cdots & a_{1 n} \\
\vdots & \ddots & \vdots\\
a_{m1} & \cdots & a_{m n}
\end{pmatrix} +
\begin{pmatrix}
b_{11} & \cdots& b_{1 n} \\
\vdots & \ddots & \vdots\\
b_{m1} & \cdots & b_{m n}
\end{pmatrix}
=
\begin{pmatrix}
a_{11} + b_{11} & \cdots & a_{1 n} + b_{1n}\\
\vdots & \ddots  & \vdots\\
a_{m1}+b_{m1} & \cdots & a_{m n}+b_{mn}
\end{pmatrix}.
$$

The zero matrix is the ($m\times n$) matrix containing zero in all its entries. When its number of
rows and columns are clear from the context it is simply denoted by $0$. For $2\times 3$ matrices
for example, we write
$$
0 =
\begin{pmatrix}
  0 & 0 & 0\\
  0 & 0 & 0
\end{pmatrix}.
$$

\beginshex
  Given an example of a non-zero $2\times 2$ matrix, such that
  $$
  A^2 = 0.
  $$
\endshex

\subsection{Multiplication of a number and a matrix}\index{multiplication of matrix med number}

A matrix may be multiplied by a number $\lambda$ by multiplying each entry by the number:
$$
\lambda \begin{pmatrix}
a_{11} & \cdots & a_{1 n} \\
\vdots & \ddots & \vdots\\
a_{m1} & \cdots & a_{m n}
\end{pmatrix} =
\begin{pmatrix}
\lambda a_{11} & \cdots & \lambda a_{1 n} \\
\vdots & \ddots & \vdots\\
\lambda a_{m1} & \cdots & \lambda a_{m n}
\end{pmatrix}.
$$

\beginshex
Does there exists a number  $\lambda$, such that
$$
\lambda
\begin{pmatrix}
1 & 2 & 3\\
4 & 5 & 6
\end{pmatrix}
+
\begin{pmatrix}
0 & 0 & 0\\
0 & 0 & 2
\end{pmatrix}
=
\begin{pmatrix}
2 & 4 & 6\\
8 & 10 & 15
\end{pmatrix}?
$$
\endshex

\beginshex
Let $A$ be a $2\times 2$ matrix, such that
$$
A B = B A,
$$
for every other $2\times 2$ matrix $B$. Show that $A$ is a diagonal matrix of the form
$$
A =
\begin{pmatrix}
  a & 0\\
  0 & a
\end{pmatrix},
$$
where $a\in \RR$ i.e., $A = a I_2$.
\endshex

\subsection{The distributive law}

Ordinary numbers $a, b, c$ satisfy
$a (b + c) = a b + a c$. This rule also holds for
matrices and is called the distributive law (multiplication
is distributed over plus)

\begin{proposition}
Let $B$ and $C$ be $m\times n$ matrices, $A$ an $r\times m$ matrix and $D$ an $n\times s$ matrix. Then
$$
A ( B + C) = A B + A C\qquad\text{and}\qquad (B + C) D = B D + C D.
$$
\end{proposition}
\begin{proof}[showhide]
  Let us start by looking at $A(B+C) = A B + A C$.
Here it suffices to do the proof, when $A$ is a row vector and
$B, C$ column vectors, since
$$
(A (B+C))_{ij} = A_i (B+C)^j = A_i (B^j + C^j).
$$
For $(B + C) D = B D + C D$, we may reduce to the case, where
$B, C$ are row vectors and  $D$ a column vector, since
$$
((B+C) D)_{ij} = (B+C)_i D^j = (B_i + C_i) D^j.
$$
Both of these cases follow using the distributive law for ordinary numbers.
\end{proof}

\beginshex
Suppose that $A$ and $B$ are two $2\times 2$ matrices. Is it true that
$$
(A + B)^2 = A^2 + B^2 + 2 A B?
$$
What about
$$
(A + B) (A - B) = A^2 - B^2?
$$
\endshex

\subsection{The miraculous associative law}

It does not make sense to multiply three matrices $A, B$ and $C$. We have only defined
matrix multiplication for two matrices. There are two natural ways of evaluating
$A B C$:
$$
( A B ) C\qquad \text{and}\qquad A (B C).
$$

We can begin by multiplying $A$ by $B$ and then multiply $C$ from the right.
However, we may just as well start by multiplying $B$ by $C$ and then
multiply $A$ from the left.

It is in no way clear, that these two computations give the same result!

That this turns out to be true, is just one of many miracles in the universe (there is a
rather cool mathematical explanation, though, addressed in an exercise below).

\begin{theorem}\label{thmmatmultass}
Let $A$ be an $m\times n$ matrix, $B$ an $n\times r$ matrix and $C$ an $r\times s$ matrix. Then
$$
(A B) C = A (B C).
$$
\end{theorem}
\begin{proof}[showhide]
  We must prove that
$$
((A B) C)_{ij} = (A (B C))_{ij}
$$
for $1\leq i \leq m$ og $1\leq j \leq s$. The left hand side can be written
\begin{align}\label{eqleft}
(A B)_i C^j &= (A_i B^1, \dots, A_i B^r) C^j\\
            &= (A_i B^1) C_{1j} + (A_i B^2) C_{2j} + \cdots + (A_i B^r) C_{rj}.
\end{align}
The right hand side is
\begin{equation}\label{eqright}
A_i (B C)^j = A_i \begin{pmatrix} B_1 C^j \\ \vdots \\ B_n C^j\end{pmatrix}
            = A_{i1} (B_1 C^j) + \cdots + A_{in} (B_n C^j).
          \end{equation}

          Writing the row-column multiplications in \eqref{eqleft}, we get
\begin{align}\label{rowscomp}
&A_{i1} B_{11} C_{1j} + \cdots + A_{in} B_{n1} C_{1j} +\\
&A_{i1} B_{12} C_{2j} + \cdots + A_{in} B_{n2} C_{2j} +\\
&\vdots\\
&A_{i1} B_{1r} C_{rj} + \cdots + A_{in} B_{nr} C_{rj}.
\end{align}
          Writing the row-column multiplications in \eqref{eqright}, we get
\begin{align}\label{colscomp}
&A_{i1} B_{11} C_{1j} + \cdots + A_{i1} B_{1r} C_{rj} +\\
&A_{i2} B_{21} C_{1j} + \cdots + A_{i2} B_{2r} C_{rj} +\\
&\vdots\\
&A_{in} B_{n1} C_{1j} + \cdots + A_{in} B_{nr} C_{rj}.
\end{align}
The rows in the sum in  \eqref{rowscomp} correspond to the columns in the sum \eqref{colscomp}.
Therefore these sums are equal and $((A B) C)_{ij} = (A (B C))_{ij}$.
\end{proof}

\begin{frameit}
\begin{remark}
The associative law $(A B) C = A (B C)$ is true, but in computing $A B C$ there can be a (big)
difference in the number of multiplications in the two computations
$A (B C)$ and $(A B) C$ i.e., efficiency is not associative for
matrix multiplication. In the notation of Theorem \ref{thmmatmultass}, computing
$(A B) C$ requires
$$
m n r + m r s = m r (n + s)
$$
multiplications, whereas computing $A (B C)$ requires
$$
n r s  + m n s = n s (m + r)
$$
multiplications. If for example $m=10000, n = 10, r = 10000$ and $s = 10$, then computing
$(A B) C$ requires $2\cdot 10^9$ multiplications, whereas
computing $A (B C)$ requires $2\cdot 10^6$ multiplications!
\end{remark}
\end{frameit}

\beginshex
Verify the associative law for the three matrices
$$
A =
\begin{pmatrix}
  1 & 2\\
  3 & 4
\end{pmatrix}, \qquad
B =
\begin{pmatrix}
  1 & 2 & 3\\
  4 & 5 & 6
\end{pmatrix}\qquad\text{and}\qquad
C =
\begin{pmatrix}
  6 & 3\\
  5 & 2\\
  4 & 1
\end{pmatrix}
$$
by showing by explicit computation that
$$
(A B) C = A (B C).
$$
\endshex

\beginshex
There is in fact a high tech explanation that the associative law for matrices holds. An explanation
that makes the calculations in the above proof superfluous and shows the raw power of
abstract mathematics: suppose that
$f: \RR^n\rightarrow \RR^m, g:\RR^r\rightarrow \RR^n$ and $h:\RR^s\rightarrow \RR^r$ are linear maps.
Then $f\circ (g\circ h)$ and $(f\circ g)\circ h$ are both linear maps from $\RR^s\rightarrow \RR^m$, such that
$$
(f\circ (g\circ h))(x) = ((f\circ g)\circ h)(x) = f( g ( h (x)))
$$
for every $x\in \RR^s$. How does this relate to the associative law for matrix
multiplication?
\endshex

\section{The inverse matrix}

You are allowed to divide by a number provided it is $\neq 0$. Does it makes sense
to divide by matrices?

It does, but there are some matrices that correspond to the number $0$ that we are not allowed
to divide by.

\beginshex
Let $A, B$ and $C$ be $n\times n$ matrices. Show that
$$
B A = I_n
$$
and
$$
A C = I_n
$$
implies that $B = C$.
\endshex

\index{invertible matrix}\index{inverse matrix}

\begin{definition}
An $n\times n$ matrix $A$ is called invertible, if there exists an  $n\times n$ matrix $B$, such that
$$
A B = B A = I_n.
$$
In this case, $B$ is called the inverse matrix of $A$ and denoted $A^{-1}$.
\end{definition}

\beginshex
Show that a quadratic matrix with a column or row consisting entirely of zeros cannot
be invertible.
\endshex

\beginshex
Suppose that
$$
A = \begin{pmatrix}
  a & b\\
  c & d
  \end{pmatrix}
  $$
  with $D = a d - b c\neq 0$. Prove that $A$ is invertible with
  $$
  A^{-1} = \frac{1}{D}\begin{pmatrix} d & -b\\ -c & a \end{pmatrix}.
  $$
\endshex

\beginshex
When is a quadratic diagonal matrix invertible? Look first at the $2\times 2$ case:
$$
\begin{pmatrix}
  a & 0\\
  0 & d
\end{pmatrix}.
$$
\endshex

The inverse matrix can be computed in \texttt{numpy}:

\begin{sage}
import numpy as np

A = np.matrix( [[5, 3], [3, 2]] )
print("The inverse of")
print(A)
print("is")
print(A.I)

\end{sage}

The inverse matrix enters the picture when solving $n$ linear
equations with $n$ unknowns:
\begin{align*}
a_{11}x_1 + a_{12} x_2 + \cdots + a_{1n} x_n &= b_1\\
&\vdots\\
a_{n1} x_1 + a_{n2} x_2 + \cdots + a_{nn} x_n &= b_n
\end{align*}
can be rewritten using matrix notation as
$$
\begin{pmatrix}
a_{11} &  \cdots & a_{1 n} \\
\vdots & \ddots & \vdots\\
a_{n1} & \cdots & a_{n n}
\end{pmatrix}
\begin{pmatrix}
x_1 \\ \vdots \\ x_n
\end{pmatrix}
=
\begin{pmatrix}
b_1 \\ \vdots \\ b_n
\end{pmatrix}
$$
or more compactly as $A x = b$.

If $A$ is invertible, then the associative law gives the following:
\label{ainvsol}
\begin{frameit}
\begin{align*}
A x &= b \iff\\
 A^{-1}  \left(A x\right) &= A^{-1} b \iff \\
(A^{-1} A) x &= A^{-1} b \iff\\
I x &= A^{-1} b\iff\\
x &= A^{-1} b.\\
\end{align*}
\end{frameit}

The inverse matrix gives the solution to the linear equations $A x = b$ just by one matrix multiplication!

\begin{example}
  The system of linear equations
\begin{equation}\label{simplign}
\begin{matrix}
&5 x &+ &3 y &= &13\\
&3 x &+&2 y &= &8
\end{matrix}
\end{equation}
can be rewritten using matrix multiplication to
$$
A v = b,
$$
where
$$
A = \begin{pmatrix}
5& 3 \\
3 & 2
\end{pmatrix}, \qquad
v =
\begin{pmatrix} x \\ y \end{pmatrix}\qquad \text{and}\qquad
b = \begin{pmatrix} 13 \\ 8 \end{pmatrix}.
$$

Here $A$ is invertible and
$$
A^{-1} =
\begin{pmatrix}
2 & -3\\
-3 & 5
\end{pmatrix}.
$$
One simple matrix multiplication
$$
\begin{pmatrix} x \\ y \end{pmatrix} =
\begin{pmatrix}
2 & -3\\
-3 & 5
\end{pmatrix} \begin{pmatrix} 13 \\ 8 \end{pmatrix} =
\begin{pmatrix} 2 \\ 1 \end{pmatrix}
$$
shows the solution we expect from  \eqref{simplign}.
\end{example}

The product of two invertible matrices (when this makes sense)
is an invertible matrix. This is the content of the following result.

\label{prodinv}
\begin{proposition}
The product $A B$ of  two invertible matrices $A$ and $B$ is invertible and
$(A B)^{-1} = B^{-1} A^{-1}$.
\end{proposition}
\begin{proof}[showhide]
We must check that
$$
(B^{-1} A^{-1}) (A B) = I\qquad\text{and}\qquad A B (B^{-1} A^{-1}) = I.
$$
Let us check the first condition using the associative law:
\begin{align*}
(B^{-1} A^{-1}) (A B) &= ((B^{-1} A^{-1}) A) B\\
                    &= (B^{-1} (A^{-1} A)) B \\
                    &= (B^{-1} I) B = B^{-1} (I B) = B^{-1} B = I,
\end{align*}
where  $I$ denotes the identity matrix. The condition $A B (B^{-1} A^{-1}) = I$ is verified in the
same way.
\end{proof}

\beginshex

  We have defined a matrix $A$ to be invertible if there exists a matrix $B$, such that
  $A B = I$ and $B A = I$. Suppose that only $B A = I$. Can we then conclude that $A B = I$?

\endshex

\beginshex
Let
$$
N =
\begin{pmatrix}
  0 & 1 & 1 & 1\\
  0 & 0 & 1 & 1\\
  0 & 0 & 0 & 1\\
  0 & 0 & 0 & 0
\end{pmatrix}.
$$
Compute the powers $N^k$ for $k\geq 2$ i.e., $N^2, N^3, \dots$. Now let
$$
A = I + N,
$$
where $I = I_4$.
Show that $A$ is invertible, and
$$
A^{-1} = I - N + N^2 - N^3.
$$
Compute $A^{-1}$.

Do you see a way of generalizing this computation to
$n\times n$ matrices $N$ with a property shared by the
$4\times 4$ matrix above?

\endshex

\subsection{Well, how do I find the inverse of a matrix?}

Finding the inverse of a matrix or deciding that the matrix is not invertible is
a matter of solving systems of linear equations.

Given an $n\times n$ matrix $A$, we need to see if there exists an $n\times n$ matrix
$B$, such that
\begin{equation}\label{invdef}
A B = I,
\end{equation}
where $I = I_n$ is the identity matrix of order $n$. We can do this by
computing the columns of $B$. From the definition in \eqref{invdef}, the
$j$-th column $B^j$ of $B$ must satisfy
\begin{equation}\label{lineqinv}
A B^j = I^j.
\end{equation}
This follows from the definition of matrix multiplication!

The identity in \eqref{lineqinv} is a system of $n$ linear equations
in $n$ unknowns. The unknowns are the entries in the $j$-th column
$B^j$ of the inverse matrix $A^{-1}$ (if it exists).

\begin{example}\label{exinverse}
  Suppose that $A$ is a $2\times 2$ matrix. Then the inverse matrix
  $B$ (if it exists) can be computed from the systems of linear
  equations below.
  $$
  A B^1 = \begin{pmatrix} 1 \\ 0 \end{pmatrix}\qquad\text{and}\qquad A B^2 = \begin{pmatrix} 0 \\ 1 \end{pmatrix}.
  $$
  Writing
  $$
  \begin{pmatrix} x \\ y \end{pmatrix} = B^1\qquad\text{and}\qquad \begin{pmatrix} u \\ v\end{pmatrix} = B^2
  $$
  for the first and second columns, the systems of linear equations can be written as
$$
\begin{matrix}
  &A_{11} x &+  &A_{12} y &=  &1\\
  &A_{21} x &+  &A_{22} y &=  &0
\end{matrix}\qquad\text{and}\qquad
\begin{matrix}
  &A_{11} u &+  &A_{12} v &=  &0\\
  &A_{21} u &+  &A_{22} v &=  &1
\end{matrix},
$$
where
$$
B =
\begin{pmatrix}
  x & u\\
  y & v
\end{pmatrix}.
$$
A concrete example along with a useful way of keeping track of the computation is presented in the video below.

\begin{video}
\youtube{4BX3eW4qo3g}
\end{video}
\end{example}

\beginshex
Compute the inverse of the matrix
$$
A =
\begin{pmatrix}
  1 & 1 & 1\\
  1 & 2 & 1\\
  1 & 1 & 3
\end{pmatrix}
$$
by employing the method of solving linear equations above. Explain
the steps in your computation. You may find it useful to collect
inspiration from the video in Example \ref{exinverse}.
\endshex

\section{The transposed matrix}\index{transposed matrix}

The transpose of an $m\times n$ matrix $A$ is the $n\times m$ matrix $A^T$ given by
$$
A^T_{i j} = A_{j i}.
$$
As an example, we have
$$
\begin{pmatrix}
0 & 2 & 4 & -2\\
3 & 2 & 7 & 4
\end{pmatrix}^T =
\begin{pmatrix}
0 & 3\\
2 & 2\\
4 & 7\\
-2 & 4
\end{pmatrix}.
$$
Notice also that $(A^T)^T = A$ for an arbitrary matrix $A$.

\begin{proposition}\label{prop:transformel}
Let $A$ be an $m\times r$ matrix and $B$ an $r\times n$ matrix. Then
$$
(A B)^T = B^T A^T.
$$
\end{proposition}
\begin{proof}[showhide]
  By definition $(A B)^T_{i j} = (A B)_{j i}$. This entry is given
  by row-column multiplication of the $j$-th row in $A$ and
the $i$-th column in  $B$, which is the row-column multiplication of the
$i$-th row  in $B^T$ and the $j$-th column  in $A^T$.
\end{proof}

\beginshex
Let $A$ be a quadratic matrix. Prove that $A$ is invertible if and only if
$A^T$ is invertible.
\endshex

\beginshex In the sage window below, you are supposed to experiment a
bit by entering an arbitrary matrix $B$ and studying the quadratic
matrix $B B^T$. Is there anything special about this product?  Press
the \emph{Further explanation} button below the sage window to display
the rest of the exercise after(!) you have completed your
experimentation.

\begin{sage}
import numpy as np

B = np.matrix( [[1, 2, 3], [4, 5, 6], [7, 8, 9]] )
print("B is ")
print(B)
print("B times the transpose of B is ")
print(B*np.transpose(B))
\end{sage}

\begin{hideinbutton}{Further explanation}
  A quadratic matrix $A$ is called symmetric if $A = A^T$. Prove that
$$
B B^T
$$
is a symmetric matrix, where $B$ is an arbitrary matrix.
\end{hideinbutton}
\endshex

\section{Symmetric matrices}\label{Sectionsymmat}

A (quadratic) matrix $A$ is called symmetric if $A = A^T$. Visually,
this means that $A$ is symmetric around the diagonal like the
$3\times 3$ matrix
$$
\begin{pmatrix}
  1 & \cblue{2} & 3\\
  \cblue{2} & 5 & 4\\
  3 & 4 & 6
\end{pmatrix},
$$
but not like the $3\times 3$ matrix
$$
\begin{pmatrix}
  1 & \cblue{2} & 3\\
  \cred{4} & 5 & 6\\
  7 & 8 & 9
\end{pmatrix}.
$$

\beginshex
Show that
$$
B^T A B
$$
is a symmetric matrix, when $A$ is a symmetric matrix and
$B$ is an arbitrary matrix. Both matrices are assumed
quadratic of the same dimensions.
\endshex

If $A$ is a symmetric $n\times n$ matrix, we define the
function $f_A: \RR^n\rightarrow \RR$ given by
$$
f_A(v) = v^T A v.
$$

This definition is rather compact. You are encouraged to see the video below
for a specific example, when $A$ is a $2\times 2$ matrix.

\begin{video}\label{videoposdef}
  \youtube{YNwrLMJ8byM}
\end{video}

Inside set of the symmetric matrices we find two very important subsets
of matrices: the positive definite and the positive semi-definite
matrices. They correspond to positive and non-negative
real numbers.

\subsection{Positive definite matrices}

A symmetric matrix $A$ is called \emph{positive definite} if
$$
f_A(v) > 0
$$
for every $v\in \RR^n\setminus\{0\}$.

\beginshex
Give examples of (non-zero) $1\times 1$ and $2\times 2$ matrices that are
positive definite and ones that fail to be positive
definite.

When is a $2\times 2$ diagonal matrix positive definite?
\endshex

\beginshex
Let $A$ be a symmetric $n\times n$ matrix. Show that
$A$ is not positive definite if $A_{11} < 0$.
\endshex

\subsection{Positive semi-definite matrices}

A symmetric matrix $A$ is called \emph{positive semi-definite} if
$$
f_A(v) \geq 0
$$
for every $v\in \RR^n$.

From the definitions it follows that a positive definite matrix is
positive semi-definite.

\beginshex
Give an example of a non-zero matrix that is positive semi-definite,
but not positive definite.

When is a $2\times 2$ diagonal matrix positive semi-definite?
\endshex

\subsection{Symmetric reductions}

As you probably have noticed, it is rather straightforward to
see when a diagonal matrix is positive (semi)definite. For a
general symmetric matrix, one needs to transform to
an equivalent diagonal matrix. This is done
using the following result.

\begin{proposition}
  Let $A$ be a symmetric $n\times n$ matrix and $B$ an
  invertible $n\times n$ matrix. Then $A$ is
  positive (semi) definite if and only if
  $$
  B^T A B
  $$
  is positive (semi) definite.
\end{proposition}
  \begin{proof}[showhide]
    Every vector $v\in \RR^n$ is equal to $B u$ for a unique $u\in \RR^n$,
    since $B$ is invertible. Why? The upshot is that the equation
$$
v = B u
$$
can be solved by multiplying both sides by $B^{-1}$ giving
$$
v = B u \iff B^{-1} v = B^{-1} (B u) = (B^{-1} B) u = u.
$$

    So we get
    $$
    v^T A v = (B u)^T A (B u) = u^T (B^T A B) u.
    $$
    This computation shows that $A$ is positive (semi) definite if
    $B^T A B$ is positive semi-definite. The same
    reasoning with $u = B^{-1} v$ shows that $B^T A B$ is
    positive (semi) definite if $A$ is positive (semi) definite.

    Notice that it is important that $B v = 0$ only happens
    when $v=0$.
  \end{proof}

\beginshex
Let
$$
D =
\begin{pmatrix}
  d & 0\\
  0 & e
\end{pmatrix}
$$
be a diagonal matrix. What conditions must the diagonal
entries $d$ and $e$ satisfy in order for
$D$ to be positive definite?

Let
$$
A =
\begin{pmatrix}
  a & c\\
  c & b
\end{pmatrix}
$$
denote a symmetric $2\times 2$ matrix, where $a\neq 0$. Let
$$
B =
\begin{pmatrix}
  1 & -\frac{c}{a}\\
  0 & 1
\end{pmatrix}.
$$
Show that $B$ is invertible and compute
$$
B^T A B.
$$
Use this to show that $A$ is positive definite if and only if
$a>0$ and $a b - c^2 > 0$.

Let $f:\RR^2\rightarrow \RR$ be the function defined by
$$
f(x, y) = 2 x^2 + 3 y^2 + 4 x y.
$$
Show that $f(x, y)\geq 0$ for every $x, y\in \RR$.
\endshex

\end{document}


\subsection{Simulation}

The channel encoder that was skipped in section \ref{sec:step1} is now implemented as a low-density parity check code (LDPC) encoder. Soft and hard decoding have been implemented with a code rate of 1/2 and their performance is compared next.

The channel coding adds structured redundancy  to the information sent by the transmitter to be able to detect and correct errors at the receiver side. The LDPC code has the property that the check matrix is sparse, allowing a lower complexity of the implementation.

\subsubsection{Hard decoding}
\iffalse
\begin{figure}[H]
\centering
    \includegraphics{fig/HardLDPC.tex}
     \caption{Hard LDPC}
    \label{fig:HardLDPC}
\end{figure}
\fi

The hard decoding is based on the implementation of a Tanner graph. The influence of the maximal number of iterations on the BER curves is shown in Figure \ref{fig:HardLDPCbis} for a 16-QAM modulation. The conclusions can be generalized to any modulation.

\begin{figure}[H]
\centering
    \includegraphics[width=\textwidth]{fig/HardLDPCbis.tex}
     \caption{Effect on the BER curves of Hard LDPC decoding for a 16-QAM modulation}
    \label{fig:HardLDPCbis}
\end{figure}

For the hard decoding, the performance is worsened at low SNR. Indeed, a noise which is too important causes the code to add errors when it is trying to correct them. Increasing the maximal number of iterations allows a slight improvement in performance.

\subsubsection{Soft decoding}

In practice, hard decoding is never used as the soft decoding allows for much better performance. Instead on basing the decision on the received bits, it computes probablities based on the received symbols. The results of the simulations for a 2-PAM is shown in Figure \ref{fig:SoftLDPC}.

\begin{figure}[H]
\centering
    \includegraphics{fig/SoftLDPC.tex}
     \caption{Effect on the BER curves of Hard LDPC decoding for a 16-QAM modulation}
    \label{fig:SoftLDPC}
\end{figure}

The theoretical curve for the ideal communication chain as well as the curve for hard decoding are shown as references. The improvement is significant with soft decoding.

\subsection{Questions}

\subsubsection{Questions regarding the simulation}

\paragraph{When building the new BER curves, do you consider the uncoded or coded bit energy on	the x-axis?} \mbox{}

The coded bit energy is considered here.

\paragraph{How do you limit the number of decoder iterations?} \mbox{}

For the hard decoding, the iterations stop either when the maximum number of iterations is reached or when the syndrome is zero.

For the soft decoding,  stop either when the maximum number of iterations is reached or when there is no more detected errors.

\paragraph{Why is it much simpler to implement the soft decoder for BPSK or QPSK than for 16-QAM or 64-QAM?} \mbox{}

In the case of the 16-QAM or 64-QAM, the euclidean distance should be computed when decoding. This is not the case for BPSK and QPSK, where the decision can be made only by looking at the real and imaginary part of the received symbols.

\subsubsection{Questions regarding the communication system}

\paragraph{Demonstrate analytically that the parity check matrix is easily deduced from the generator matrix when the code is systematic.} \mbox{}

A code is systematic when the mapping is such that part of the code vector coincides with the message vector. In that case, the generator matrix has the form $\underline{\underline{G}} = \left[\underline{\underline{P}} | \underline{\underline{I}}\right]$, with $\underline{\underline{P}}$ the parity array and $\underline{\underline{I}}$ the identity matrix of the corresponding size.
The parity check matrix should be created so that its rows are orthogonal to the rows of the generator matrix:

\begin{equation*}
 \underline{\underline{G}} \cdot \underline{\underline{H}}^T = \underline{\underline{0}}
\end{equation*}

The solution to this equation is given by $\underline{\underline{H}} = \left[\underline{\underline{I}} | \underline{\underline{P}}^T\right]$. Indeed:

\begin{equation*}
	\underline{\underline{G}} \cdot \underline{\underline{H}}^T = \left[\underline{\underline{P}} | \underline{\underline{I}}\right] \cdot \left[\underline{\underline{I}} | \underline{\underline{P}}^T\right]^T =  \underline{\underline{P}} \oplus  \underline{\underline{P}} =0
\end{equation*}

\paragraph{Explain why we can apply linear combinations on the rows of the parity check matrix to produce an equivalent systematic code.} \mbox{}

The rows of the parity check matrix are the basis vectors of a subspace which is perpendicular to the subspace spanned by the generator matrix. A linear combination of these basis vectors will still be a basis for the same subspace and the resulting code will be equivalent.

\paragraph{Why is it especially important to have a sparse parity check matrix (even more important	than having a sparse generator matrix)?} \mbox{}

A sparse parity check matrix allows to significantly reduce the complexity of the decoder. Indeed, element $H_{ij}$ being equal to 1 results in a logical connection in the Tanner graph between the check-node $c_i$ and the variable-node $v_j$. Hence, reducing the number of 1's in $\underline{\underline{H}}$ reduces the number of exchanged messages and the number of computations.

\paragraph{Explain why the check nodes only use the information received from the other variable nodes when they reply to a variable node.} \mbox{}

The information exchanged between the check and variable nodes has to be statistically independent for hard and soft decoding. This means that only the information which is not related to the current node is used to compute the answer.


\section*{Exercises}

\begin{ex}
  Which of the following equations are linear?
  \begin{enumerate}
  \item[(a)] $2x-3y+4z = -10$
  \item[(b)] $2.123x_1 + 5.541x_2 - 9.101x_3 = 11.012$
  \item[(c)] $x^2+y^2+z^2 = 1$
  \item[(d)] $\frac{1}{\sqrt{2}}x+4^3y = \sin(\frac{\pi}{3})$
  \item[(e)] $x+yz = 3$
  \end{enumerate}

  \begin{sol}
    All of them except (c) and (e).
  \end{sol}
\end{ex}

\begin{ex}
  Consider the system of equations
  \begin{equation*}
    \begin{array}{c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c@{~}c}
      x&+&2y&+&3z&+&4w &=& 4 \\
      x&+&y&+&z&+&w &=& 2 \\
      x&+&2y&+&2z&+&w &=& 2.
    \end{array}
  \end{equation*}
  For each of the following tuples $(x,y,z,w)$ of real numbers,
  determine whether it is a solution of the first equation, second
  equation, and/or third equation. Which ones are solutions to the
  system of equations?

  \begin{center}
    (a) $(2,0,-2,2)$ \quad
    (b) $(2,2,-2,0)$ \quad
    (c) $(1,1,-1,1)$ \quad
    (d) $(3,0,-1,1)$ \quad
    (e) $(2,-2,2,0)$
  \end{center}

  \begin{sol}
    (a) is a solution to equations 1 and 2, (b) is a solution to
    equations 2 and 3, (c) is a solution to equations 1, 2, and 3,
    (d) is a solution to 1 and 3, and (e) is a solution to equations
    1, 2, and 3. Only (c) and (e) are solutions to the system of
    equations.
  \end{sol}
\end{ex}


\documentclass{article}

%-------------------------------------------------

\usepackage{fullpage}

\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amssymb}

%-------------------------------------------------
\begin{document}

\section*{Basics of Gaussian Process Priors}

We assume the values of a function are drawn from a single process, and assume a Gaussian distribution with a particular form for the covariance between values.
We then condition the values at sample points based on the values at observed points to obtain a ``posterior distribution'' for the sample points given the observed values.
This can be illustrated below with a simple example.

\subsection*{Simple example of conditioning}

Consider 2 values of a function: $f_a$ and $f_b$ corresponding to the ordinates $x_a$ and $x_b$.
We assume these values are correlated in some way, so that
\begin{equation}
\begin{matrix}
f_a \\
f_b
\end{matrix}
\sim
\mathcal{N} \left(
\mathrm{mean}=
\begin{bmatrix}
0 \\
0
\end{bmatrix}
,
\mathrm{covariance}=
\begin{bmatrix}
\sigma_{aa}^2 & \sigma^2_{ab} \\
\sigma_{ba}^2 & \sigma^2_{bb}
\end{bmatrix}
\right)
\end{equation}

The requirement for zero-mean is not stringent in practice.
Now, a feature of the Gaussian distribution is that we can immediately write down the marginal distributions as
\begin{equation}
f_a \sim \mathcal{N}\left(0, \sigma_{aa}^2\right)
\end{equation}
and so forth.
Now, to compute the conditional probability $p(f_b|f_a)$, we recognize
\begin{equation}
p(f_b|f_a) = \frac{p(f_a, f_b)}{p(f_a)}
\end{equation}
and the rest is just algebra.
We provide this in all it's gory detail here for pedagogy.

\begin{align}
p(f_a) & = \frac{1}{\sqrt{2\pi\sigma^2_{aa}}} \mathrm{exp}\left(-\frac{f_a^2}{2\sigma^2_{aa}} \right)
\end{align}
\begin{align}
p(f_a, f_b) & = \frac{1}{2\pi\sqrt{\sigma^2_{aa}\sigma^2_{bb} - \sigma^4_{ab}}} \mathrm{exp}\left( -\frac{1}{2} \begin{bmatrix} f_a & f_b \end{bmatrix} \frac{1}{\sigma^2_{aa}\sigma^2_{bb} - \sigma^4_{ab}}\begin{bmatrix} \sigma_bb^2 & -\sigma_{ab}^2 \\ -\sigma_{ab}^2 & \sigma_{aa}^2 \end{bmatrix} \begin{bmatrix} f_a \\ f_b \end{bmatrix} \right) \nonumber \\
            & = \frac{1}{2\pi\sqrt{\sigma^2_{aa}\sigma^2_{bb} - \sigma^4_{ab}}} \mathrm{exp}\left( -\frac{1}{2} \frac{f_a^2\sigma_{bb}^2 + f_b^2\sigma_{aa}^2 - 2f_a f_b \sigma_{ab}^2}{\sigma^2_{aa}\sigma^2_{bb} - \sigma^4_{ab}} \right)
\end{align}
\begin{align}
p(f_b|f_a) & = \frac{p(f_a, f_b)}{p(f_a)} \nonumber \\
           & = \sqrt{\frac{\sigma_{aa}^2}{2\pi\left(\sigma_{aa}^2\sigma_{bb}^2 - \sigma_{ab}^4\right)}}\mathrm{exp}\left( \frac{f_a^2}{2\sigma_{aa}^2} - \frac{1}{2} \frac{f_a^2\sigma_{bb}^2 + f_b^2\sigma_{aa}^2 - 2f_a f_b \sigma_{ab}^2}{\sigma^2_{aa}\sigma^2_{bb} - \sigma^4_{ab}}\right) \nonumber \\
           & = \sqrt{\frac{\sigma_{aa}^2}{2\pi\left(\sigma_{aa}^2\sigma_{bb}^2 - \sigma_{ab}^4\right)}}\mathrm{exp}\left( \frac{f_a^2}{2}\left(\frac{1}{\sigma_{aa}^2} - \frac{\sigma_{bb}^2}{\sigma_{aa}^2\sigma_{bb}^2-\sigma_{ab}^4}\right) - \frac{f_b^2 \sigma_{aa}^2 - 2 f_a f_b \sigma_{ab}^2}{2\left(\sigma_{aa}^2\sigma_{bb}^2 - \sigma_{ab}^4\right)}\right) \nonumber \\
           & = \sqrt{\frac{\sigma_{aa}^2}{2\pi\left(\sigma_{aa}^2\sigma_{bb}^2 - \sigma_{ab}^4\right)}}\mathrm{exp}\left( -\frac{f_a^2\sigma_{ab}^4}{2\sigma_{aa}^2\left(\sigma_{aa}^2\sigma_{bb}^2 - \sigma_{ab}^4\right)} - \frac{\sigma_{aa}^2}{2\left(\sigma_{aa}^2\sigma_{bb}^2 - \sigma_{ab}^4\right)}\left(f_b^2 - 2f_af_b\frac{\sigma_{ab}^2}{\sigma_{aa}^2}\right)\right) \nonumber \\
           & = \sqrt{\frac{\sigma_{aa}^2}{2\pi\left(\sigma_{aa}^2\sigma_{bb}^2 - \sigma_{ab}^4\right)}}\mathrm{exp}\left( - \frac{\sigma_{aa}^2}{2\left(\sigma_{aa}^2\sigma_{bb}^2 - \sigma_{ab}^4\right)} \left(f_b - f_a\frac{\sigma_{ab}^2}{\sigma_{aa}^2}\right)^2 \right)
\end{align}
and we see that
\begin{equation}
 f_b|f_a \sim \mathcal{N}\left(f_a\frac{\sigma_{ab}^2}{\sigma_{aa}^2}, \sigma_{bb}^2 - \frac{\sigma_{ab}^4}{\sigma_{aa}^2}\right)
\end{equation}
Thus, we can write conditional distributions for \textit{test points} ($f_b$) based on \textit{observed points} ($f_a$).

The last assumption needed is the form of the covariance assumed.
This is typically done by some measure of the distance in the ordinate via a kernel.
A common kernel is the \textit{squared exponential}
\begin{equation}
    \sigma_{ij}^2 = \sigma^2 \mathrm{exp}\left(\frac{(x_i - x_j)^2}{2l^2}\right)
\end{equation}
and we can incorporate the presence of noise in our observed samples by adding an additional term so that
\begin{equation}
    \sigma_{ij}^2 = \sigma^2 \mathrm{exp}\left(\frac{(x_i - x_j)^2}{2l^2}\right) + \sigma_{n}^2\delta_{ij}
\end{equation}
We note that this kernel introduces several \textit{hyper-parameters} which must be chosen in some way; there are common techniques for selecing optimal hyper-parameters based on the observed data.
They represent
\begin{itemize}
  \item{$\sigma^2$
    \begin{itemize}
      \item{The typical size of the covariance between different values of the function. This is is also the variance of each individual value of the function.}
    \end{itemize}
  }
  \item{$l^2$
    \begin{itemize}
      \item{The length scale over which correlations persist within the function. For $|x_i-x_j|\gg l$, the function values are essentially uncorrelated.}
    \end{itemize}
  }
  \item{$\sigma_n^2$
    \begin{itemize}
      \item{They typical size of the noise in each observation. We assume the noise is uncorrelated between different observations but drawn from the same distribution for each observation, thus the kronecker-$\delta$.}
    \end{itemize}
  }
\end{itemize}

\end{document}


\subsection{DenseNet}
\begin{itemize}
	\item Semi-iterative method for accelerating the residual correction iterative scheme,
	\begin{equation}\label{eq:multi}
	u^{i} = \sum_{j=0}^{i-1}  \alpha_{j}^i\left( u^{j} + B^{i}_j(f - A(u^j)) \right), \quad i=1:\nu,
	\end{equation}
	where $\alpha_j^i \ge 0$ and $\sum_{j=0}^{i-1}  \alpha_{j}^i = 1$.
	Letting the residual $r^j = f - A(u^{j})$ for $j=0:i$,
	the following iterative scheme for $r^j$ is implied by \eqref{eq:multi}
	\begin{equation}\label{eq:multi-res}
	r^{i} = \sum_{j=0}^{i-1}\alpha^i_j(I - AB^{i}_j)(r^j),
	\end{equation}
	because of the linearity of $A$. This scheme is analogous to the
	DenseNet \cite{huang2017densely} which will be discussed more in \S~\ref{sec:CNNs} and below.
	More discussion on semi-iterative method for linear system
	can be found in \cite{hackbusch1994iterative, golub2012matrix}.
\end{itemize}

The DenseNet \cite{huang2017densely} can be written as:
\begin{equation}\label{eq:densenet}
\begin{cases}
f^{1,0} &=R_{\rm max}\circ \sigma \circ \theta^0(f), \\
\text{\bf For} &\ell = 1:J \\
\quad &\text{\bf For} \quad i = 1:\nu_\ell \\
&f^{\ell,i} = \sigma \circ \theta^{\ell,i} (\tilde f^{\ell,i-1}), \quad i = 1:\nu_\ell ,\\
&\tilde f^{\ell,i} = [f^{\ell,0,}, \cdots, f^{\ell,i}] \quad i = 1:\nu_\ell  , \\
\quad &\text{\bf EndFor} \\
f^{\ell+1,0} &=  R_\ell^{\ell+1} (\tilde f^{\ell, \nu_\ell} ) , \\
\text{\bf End} &\\
H_0(f) &=  R_{\rm ave}(f^{L,\nu_\ell}). \\
\end{cases}
\end{equation}
Here $[f^{\ell,0,}, \cdots, f^{\ell,i}]$ means the collection of
all the previous output in $\ell$-th grids after $i$-th smoother.

The original setup in DenseNet is:
\begin{equation}
f^{\ell,0} \in \mathbb{R}^{n_\ell \times n_\ell \times k_0}, \quad f^{\ell,i} \in \mathbb{R}^{n_\ell \times n_\ell \times k}, \quad i = 1:\nu_\ell.
\end{equation}
Thus to say,
\begin{equation}
\theta^{\ell,i} = [[\theta^{\ell,i}]_0, [\theta^{\ell,i}]_{1}, \cdots, [\theta^{\ell,i}]_i]: \mathbb{R}^{n_\ell\times n_\ell \times (k_0 + ck)} \mapsto \mathbb{R}^{n_\ell\times n_\ell \times k} .
\end{equation}
This means that,
\begin{equation}
[\theta^{\ell,i}]_{0}:  \mathbb{R}^{n_\ell\times n_\ell \times k_0 } \mapsto \mathbb{R}^{n_\ell\times n_\ell \times k}, \quad [\theta^{\ell,i}]_{j}:  \mathbb{R}^{n_\ell\times n_\ell \times k } \mapsto \mathbb{R}^{n_\ell\times n_\ell \times k}, \quad j = 1:i.
\end{equation}
At last, we have
\begin{equation}
\theta^{\ell,i}(\tilde f^{\ell,i}) = \sum_{j=0}^i [\theta^{\ell,i}]_{j} [\tilde f^{\ell,i}]_j = \sum_{j=0}^i [\theta^{\ell,i}]_{j} f^{\ell,j}.
\end{equation}
Finally, we have that
\begin{equation}
[\theta^{\ell,i}]_{j},
\end{equation}
plays the role of $A$ in MgNet?

So, we can rewrite the above DenseNet with the above formula without $\tilde f$:
\begin{equation}\label{eq:densenet1}
\begin{cases}
f^{1,0} &=R_{\rm max}\circ \sigma \circ \theta^0(f), \\
\text{\bf For} &\ell = 1:J \\
\quad &\text{\bf For} \quad i = 1:\nu_\ell \\
&f^{\ell,i} = \sigma \left( \sum_{j=0}^{i-1} [\theta^{\ell,i}]_{j} f^{\ell,j} \right) ,\\
\quad &\text{\bf EndFor} \\
f^{\ell+1,0} &=  R_\ell^{\ell+1} ([f^{\ell,0,}, \cdots, f^{\ell,\nu_\ell}] ) , \\
\text{\bf End} &\\
H_0(f) &=  R_{\rm ave}(f^{L,\nu_\ell}). \\
\end{cases}
\end{equation}
So, similar idea in removing $i$ in $A^{\ell,i}$ in MgNet, we can remove $j$ in $\theta^{\ell,i}$ or
even remove $i$, this can reduce the number of parameters very efficient....

We can also adopt this idea into MgNet.
\begin{breakablealgorithm}
	\caption{$u^{J}={\rm Dense-MgNet}(f; J,\nu_1, \cdots, \nu_J)$}
	\label{alg:dense-mgnet}
	\begin{algorithmic}
		\State Initialization:  $f^1 =\theta(f)$, $u^{1,0} \leftarrow 0$
		%		\State Initialization $u^{1,0}$
		\For{$\ell = 1:J$}
		\For{$i = 1:\nu_\ell$}
		\State
		\begin{itemize}
			\item Standard Chebyshev iteration:
			\begin{equation}\label{chebyshev-mgnet}
			u^{\ell,i} \leftarrow u^{\ell,i-1} + B_{\ell,i}  ({f^\ell -  A^{\ell} (u^{\ell,i-1})}) + C_{\ell}(u^{\ell,i-1} - u^{\ell,i-2}).
			\end{equation}
			\item Expanded version:
			\begin{equation}\label{expanded-mgnet}
			u^{\ell,i} \leftarrow \sum_{j=0}^{i-1} \mathcal F^{\ell,i}(u^{\ell,j}, f^\ell),
			\end{equation}
			where $\mathcal F^{\ell,i}$ is defined by $A^\ell, B_{\ell,i}$ and $C_\ell$ with some special forms.(Parameter sharing in some sense.)
			\item Dense-MgNet:
			\begin{itemize}
				\item Extended form:
				\begin{equation}\label{extendes-mgnet}
				u^{\ell,i} \leftarrow \sum_{j=0}^{i-1} \mathcal F^{\ell,i}(u^{\ell,j}, f^\ell),
				\end{equation}
				where $\mathcal F^{\ell,i}$ is free to be trained with some special forms.(Removing parameter sharing...)
				\item $\tilde u^{\ell,i}$ form:
				\begin{equation}\label{dense-mgnet}
				u^{\ell,i} \leftarrow \mathcal D_{\ell,i}([\tilde u^{\ell,i-1}, f^{\ell}]).
				\end{equation}
				where
				\begin{equation}\label{eq:tilde-u}
				u^{\ell,i-1} = [u^{\ell,0}, \cdots, u^{\ell,i-1}],
				\end{equation}
				and $\mathcal D_{\ell,i}$ needs to be trained with some special forms.
			\end{itemize}

		\end{itemize}

		\EndFor
		\State Note $u^\ell = u^{\ell,\nu_\ell}$
		\begin{equation}
		\label{d-interpolation}
		u^{\ell+1,0} \leftarrow \Pi_\ell^{\ell+1}u^{\ell}
		\end{equation}
		\begin{equation}
		\label{d-restrict-f}
		f^{\ell+1} = R^{\ell+1}_\ell(f^\ell - A^\ell(u^{\ell})) + A^{\ell+1} (u^{\ell+1,0}).
		\end{equation}
		\EndFor
	\end{algorithmic}
\end{breakablealgorithm}

%What's more, the above ideas consider about the DenseNet structures (Chebyshev iteration)
%into the grids level for MgNet.


\section{The Knapsack Problem}

%%%%%%%%%%
\begin{frame}{The knapsack problem}
  \begin{exampleblock}{The change-making problem \pno{2.2.17 (b), 2.2.4 (subset sum)}}
    \begin{itemize}
      \item coins values: $x_{1} \dots x_{n}$
      \item amount: $v$
      \item possible to make change for $v$?
      \item without repetition
    \end{itemize}
  \end{exampleblock}

  \pause
  \begin{alertblock}{Trial and error.}
    \begin{itemize}
      \item subproblem $C[i]$: is it possible to make change for $v$ using only $x_{1} \cdots x_{n}$
      \item goal: $C[n]$
      \item question: using $x_{i}$ or not?
      \item recurrence:
	\[
	  C[i] = C[i-1] \lor \textcolor{red}{???}
	\]
    \end{itemize}
  \end{alertblock}
\end{frame}
%%%%%%%%%%
\begin{frame}{The knapsack problem}
  \begin{block}{Solution.}
    \begin{itemize}
      \item subproblem $C[i, w]$: is it possible to make change for $w$ using only $x_{1} \dots x_{n}$
      \item goal: $C[n,v]$
      \item question: using $x_{i}$ or not?
      \item recurrence:
	\[
	  C[i,w] = C[i-1, w] \lor (C[i-1, w-x_{i}] \textcolor{blue}{\land w \ge x_{i}})
	\]
      \item<2-> initialization:
	\begin{align*}
	  C[i,0] &= \text{true}  \\
	  C[0,w] &= \text{false}, \text{if } w > 0 \\
	  C[0,0] &= \text{true}
	\end{align*}
    \end{itemize}
  \end{block}
\end{frame}
%%%%%%%%%%
\begin{frame}{The knapsack problem}
  \begin{exampleblock}{The change-making problem \pno{2.2.17 (a)}}
    \begin{itemize}
      \item coins values: $x_{1} \dots x_{n}$
      \item amount: $v$
      \item possible to make change for $v$?
      \item unbounded repetition
    \end{itemize}
  \end{exampleblock}

  \begin{block}{Solution.}
    \begin{itemize}
      \item subproblem $C[i, w]$: is it possible to make change for $w$ using only $x_{1} \dots x_{n}$
      \item goal: $C[n,v]$
      \item question: using $x_{i}$ or not?
      \item recurrence:
	\[
	  C[i,w] = C[i-1, w] \lor (C[\textcolor{red}{i}, w-x_{i}] \land w \ge x_{i})
	\]
    \end{itemize}
  \end{block}
\end{frame}
%%%%%%%%%%
\begin{frame}{The knapsack problem}
  \begin{exampleblock}{The change-making problem \pno{2.2.17 (c)}}
    \begin{itemize}
      \item coins values: $x_{1} \dots x_{n}$
      \item amount: $v$
      \item possible to make change for $v$?
      \item $\le k$-coins
    \end{itemize}
  \end{exampleblock}

  \begin{block}{Solution.}
    \begin{itemize}
      \item subproblem $C[i,w,l]$: is it possible to make change for $w$ with $\le l$ coins of $x_{1} \dots x_{i}$
      \item goal: $C[n,v,k]$
    \end{itemize}
  \end{block}
\end{frame}
%%%%%%%%%%
\begin{frame}{The knapsack problem}
  \begin{block}{Solution.}
    \begin{itemize}
      \item question: using $x_{i}$ or not?
      \item recurrence:
	\[
	  C[i,w,l] = C[i-1,w,l] \lor \left( C[i,w-x_{i},l-1] \land w \ge x_{i} \right)
	\]
      \item<2-> initialization:
	\begin{align*}
	  C[0,0,l] &= \text{true} \\
	  C[0,w,l] &= \text{false}, \text{if } w > 0 \\
	  C[i,0,l] &= \text{true} \\
	  C[i,w,0] &= \text{false}, \text{if } w > 0
	\end{align*}
    \end{itemize}
  \end{block}
\end{frame}
%%%%%%%%%%


\section{Planar Graphs}
\label{title}

\subsection{Bipartite Graphs}
\label{Bipartite}

\subsection{Hamiltonian Cycles}
\label{ham-cycles}

\subsection{Planar Graphs}
\label{planar}

Euler’s Relationship for any connected planar graph

\[
n - m + r = 2
\]
(Th10.2, p190)

If $G$ is planar then order, i.e. number of vertices, $n \ge 3$, size $m$, then $m \le 3n - 6$

The boundary of every region must have at least 3 edges.

This is obvious for $n = 3$, so $n \ge 4$ and $m \ge 3$.
Suppose $G$ is planar and has $r$ regions.

\[ n - m + r = 2 \]

Let’s sum up all the arcs touching all the regions, $N$

\[ 3r \le N \]

Each arc can only touch two regions

\begin{align}
N &\le 2r \\
2 &= n - m + r \\
6 &= 3n -3m + 3r
\end{align}
but
\[
3r \le 2m
\]
So
\begin{align}
6 &\le 3n - 3m + 2m \\
6 &<= 3n -m \\
m &<= 3n - 6 \\
\end{align}

We can go on to say if $G$ is a graph of order at least 3 and size $m$, $m > 3n - 6$, $G$ is nonplanar. For $K_5$  that works.

$K_{3,3}$ it doesn’t.

Proof that $K_{3,3}$ is not planar:

Using Euler
$A = 9$, $N=6$, so $R$ must be 5.

Pick a region, its boundary must be a cycle and (it is bipartite) it must be an even cycle, at least 4, perhaps 6.

$5$ regions, $5*4$ arcs, each arc separates 2 regions so $(5*4/2) = 10$ arcs required \ldots but we only have 9.

Contradiction

\subsection{Ore's Theorem}

Let $G$ be a (finite and simple) graph with $n \ge 3$ vertices. We denote by $deg_v$ the degree of a vertex $v$ in $G$, i.e. the number of incident edges in $G$ to $v$. Then, Ore's theorem states that if

$deg_v + deg_w \ge n$ for every pair of distinct non-adjacent vertices $v$ and $w$ of $G$

then $G$ is Hamiltonian.

\subsubsection{Proof}

It is equivalent to show that every non-Hamiltonian graph G does not obey the condition.

 Accordingly, let $G$ be a graph on $n \ge 3$ vertices that is not Hamiltonian, and let $H$ be formed from $G$ by adding edges one at a time that do not create a Hamiltonian cycle, until no more edges can be added.

 Let $x$ and $y$ be any two non-adjacent vertices in $H$. Then adding edge $xy$ to $H$ would create at least one new Hamiltonian cycle, and the edges other than $xy$ in such a cycle must form a Hamiltonian path $v_1 v_2 ... v_n$ in $H$ with $x = v_1$ and $y = v_n$.

 For each index $i$ in the range $2 \le i \le n$, consider the two possible edges in $H$ from $v_1$ to $v_i$ and from $v_{i-1}$ to $v_n$. At most one of these two edges can be present in $H$, for otherwise the cycle $v_1 v_2 ... v_{i-1} v_n v_{n-1} ... v_i$ would be a Hamiltonian cycle.

 Thus, the total number of edges incident to either $v_1$ or $v_n$ is at most equal to the number of choices of $i$, which is $n-1$. Therefore, $H$ does not obey the property, which requires that this total number of edges $(deg_{v_1} + deg_{v_n})$ be greater than or equal to $n$.

 Since the vertex degrees in $G$ are at most equal to the degrees in $H$, it follows that $G$ also does not obey the property.

\footnote{Based on the proof outlined in \url{https://en.wikipedia.org/wiki/Ore\%27s_theorem}.}


\chapter{Heap}

\section{Introduction}
Heap-ordered. Binary heap is one of the implementations of Priority Queue (ADT). The core relationship of elements in the heap:
$A_{2i} \leq A_{i} \geq A_{2i+1}$.

\begin{figure}[hbtp]
\centering
\subfloat{\includegraphics[height=1.3in]{heap}}
\caption{Heap}
\label{fig:heap}
\end{figure}
\section{Operations}
Assume the root \textbf{starts} at $a[1]$ rather than $a[0]$.
\\
Basic operations:
\begin{enumerate}
\item sink()/ sift\_down() - recursive
\item swim()/ sift\_up() - recursive
\item build()/ heapify() - bottom-up sink()
\end{enumerate}
\subsection{Sink (sift\_down)}
Core clue: compare parent to the \textit{larger} child.
\begin{python}
def sink(self, idx):
    while 2*idx <= self.N:
        c = 2*idx
        if c+1 <= self.N and self.less(c, c+1):
            c += 1
        if not self.less(idx, c):
            return

        self.swap(idx, c)
        idx = c
\end{python}

\subsection{Swim (sift\_up)}
Core clue: compare child to its parent.
\begin{python}
def swim(self, idx):
    while idx > 1 and self.less(idx/2, idx):
        pi = idx/2
        self.swap(pi, idx)
        idx = pi
\end{python}
\subsection{Heapify}
Core clue: bottom-up sink().
\begin{python}
def heapify(self):
    for i in xrange(self.N/2, 0, -1):
        self.sink(i);
\end{python}
\runinhead{Complexity.} Heapifying \textbf{a sorted array} is the worst case for heap construction, because the root of each subheap considered sinks all the way to the bottom. The worst case complexity $\sim 2N$.

Building a heap is $O(N)$ rather than $O(N \lg N)$. Intuitively, the deeper the level, the more the nodes, but the less the level to sink down.

At most $\big\lceil\frac{n}{2^{h+1}}\big\rceil$ nodes of any height $h$.

Proof:
\begin{align*}
\because \sum_{i=0}^{+\infty} {ix^i} =\frac{x}{(1-x)^2} \\
\therefore \sum_{h=0}^{\lfloor\lg n\rfloor}{\Big\lceil\frac{n}{2^{h+1}}\Big\rceil
O(h)} &= O\Bigg(n\sum_{h=0}^{\lfloor\lg n\rfloor}{\frac{h}{2^h}}\Bigg) \\
&= O(n)
\end{align*}

\section{Implementation}
\subsection{General}
The self-implemented binary heap's index usually starts at 1 rather than 0.

The array representation of heap is in \textbf{level-order}.

The main reason that we can use an array to represent the heap-ordered tree in a binary heap is because the tree is \textbf{complete}.

Suppose that we represent a BST containing N keys using an array, with $a[0]$ empty, the root at $a[1]$. The two children of $a[k]$ will be at $a[2k]$ and $a[2k+1]$. Then, the length of the array might need to be as large as $2^N$.

It is possible to have 3-heap. A 3-heap is an array representation (using 1-based indexing) of a complete 3-way tree.
The children of $a[k]$ are $a[3k-1]$, $a[3k]$, and $a[3k+1]$.
\begin{figure}[hbtp]
\centering
\subfloat{\includegraphics[scale=.90]{heapRepr}}
\caption{Heap representation}
\label{fig:heap}
\end{figure}

\subsection{Python Heapq}
Python only has built in min-heap. To use max-heap, you can:
\begin{enumerate}
\item Invert the number: 1 becomes -1.
(usually the best solution)\item Wrap the data into another class and override \textbf{comparators}: \_\_cmp\_\_ or \_\_lt\_\_
\end{enumerate}

The following code presents the wrapping method:
\begin{python}
class HeapValue(object):
    def __init__(self, val):
        self.val = val
        self.deleted = False  # lazy delete

    def __cmp__(self, other):
        # Reverse order by height to get max-heap
        assert isinstance(other, Value)
        return other.val - self.val
\end{python}

Normally the deletion by value in Python is $O(n)$, to achieve $O(\lg n)$ we can use \textbf{lazy deletion}. Before take the top of the heap, we do the following:
\begin{python}
while heap and heap[0].deleted:
    heapq.heappop(heap)
\end{python}
\subsection{Java Priority Queue}
\begin{java}
// min-heap
PriorityQueue<Integer> pq = new PriorityQueue<>(
    (o1, o2) -> o1-o2
);

// max-heap
PriorityQueue<Integer> pq = new PriorityQueue<>(
    (o1, o2) -> o2-o1
);
\end{java}

\section{Derivatives}
\subsection{Heap of Linked Lists}
Maintain a heap of linked lists, pop the min head, and push the head's next back to the heap.


%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Conditional Expectation %
%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Conditional Expectation}

\begin{exercise}{8.1}{}

    Let $\CalG \coloneqq \{\emptyset, \Omega\}$, i.e. the trivial \SigmaAlgebra. Then if random variable $Y$ is $\CalG$-measurable, then $Y$ is constant.

\end{exercise}

\begin{lemma}{8.2}{}

    Let $Z$ be a $\CalG$-measurable random variable such that:

        \begin{align*}
            \int_{A} Z \, dP \geq 0 \iff E(\Indicator{A}Z) \geq 0,
        \end{align*}

    for any $A \in \CalG$, then $Z \geq 0$ (a.s.).

\end{lemma}

\begin{theorem}{8.6}{Properties of Conditional Expectations}

    Let $X$ be a random variable and $\CalG \subset \CalF$ be a \SigmaAlgebra. Then the following properties hold (under the given conditions):

        \begin{enumerate}[(i)]
            \setlength{\parskip}{0em}
            \item ``Adding/Dropping Conditional Expectation'':

                \begin{align*}
                    EX = E(E(X|\CalG));
                \end{align*}
            \item ``Tower Rule'': Let $\mathcal{H} \subset \CalF$ be a \SigmaAlgebra, such that $\mathcal{H}$ \emph{contains} $\CalG$, then:

                \begin{align*}
                    E(E(X|\mathcal{H})|\CalG) = E(X|\CalG);
                \end{align*}
            \item ``Pulling/Pushing Random Variables Through'': Let $Y$ be a random variable, such that $Y$ is $\CalG$-measurable \emph{and} $E|XY| < \infty$, then:

                \begin{align*}
                    E(XY|\CalG) = YE(X|\CalG);
                \end{align*}
            \item ``Independence of Conditional'': Let $X$ and $\CalG$ be independent, i.e. $\sigma(X)$ and $\CalG$ are independent, then:

                \begin{align*}
                    E(X|\CalG) = EX.
                \end{align*}
        \end{enumerate}

\end{theorem}


\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

% Main maths packages
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathrsfs}
\usepackage{mathtools}

\begin{document}

\section*{Boxed equation}

\begin{alignat*}{2}
	&& \sqrt{4 x^2} - 1 &= 0 \\
	&\Rightarrow & \sqrt{4} \sqrt{x^2} &= 1 \\
	&\Rightarrow & 2 \left|x\right| &= 1 \\
	&\Rightarrow & x &= \pm \frac{1}{2}
\end{alignat*}

The final result is

\begin{equation}
	\boxed{x = \pm \frac{1}{2}}
\end{equation}

\end{document}


\section{$\mathbf{X} = \mathbf{U\Sigma V^T}$ Eigenanalysis, Singular Value Decomposition (SVD)}
\label{sec:datasvd}

Singular Value Decomposition (SVD) is a matrix decomposition of the form $\mathbf{A}=\mathbf{U}\Sigma\mathbf{V}^T$, where $\mathbf{U}$ and $\mathbf{V}$ are both unitary (cf. section \ref{sec:svd}). The decomposition always exists for a general complex matrix $\mathbf{A}\in\mathbb{C}^{m\times n}$.
\\

If $\mathbf{X}\in\mathbb{R}^{N\times p}$ is a data matrix with $N$ samples in the rows and $p$ features in the columns, then SVD allows for the decomposition of the data into $p$ linearly independent components, ranked by their explained variance (i.e. their strength in the dataset). The basis of the decomposition turns out to be the same as for PCA (cf. section \ref{sec:pca}) except that it is arrived at slightly differently, because PCA involves diagonalizing the covariance matrix., nor are the singular values the same as the explained variances of the dataset. The decomposition of the dataset can be understood as follows. In the SVD of the data matrix:

\begin{equation}
\mathbf{X} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T
\end{equation}

The individual rows of $\mathbf{X}$, i.e. the individual data points, are expressed as:

\begin{equation}
\mathbf{x}^T_i = \sum_{j=1}^p u_{i,j}\sigma_j\mathbf{v}_j^T
\end{equation}

Where $\mathbf{x}_i$ is the $i$th data point. Hence the right singular vectors $\mathbf{v}_i$ are the normalized basis vectors, and the columns of $\mathbf{U}$ give the coefficients the basis expansion, together with the singular values $\sigma_i$, which give a measure of the global strength of the corresponding basis vector.

Truncating the expansion after the $r$th term results in a rank $r$ approximation of $\mathbf{X}$ known as the \textit{truncated singular value decomposition} (TSVD) estimator. According to the Eckart-Young Mirsky-Theorem, the estimator is the best rank-$r$ approximation under the Frobenius norm of the error (cf. \ref{sec:frobenius}), which is the average mean square error across $\mathbf{X}$.

\begin{equation}
\mathbf{\hat{X}} = \sum_{i=1}^r \sigma_i (\mathbf{u}_i \otimes \mathbf{v}_i)
\end{equation}

Since it minimizes the average mean square error, $\mathbf{\hat{X}}$ is the rank $r$ maximum likelihood approximation to $\mathbf{X}$ under the assumption of normally distributed noise. Optimal truncation is discussed in section \ref{sec:truncation}.

SVD is scalable to very large datasets and finds many applications in the wild, including page rank, facial recognition, recommendation algorithms, and others. Randomized SVD (cf. section \ref{sec:rsvd}) is an approximate method that gives an even faster speedup.

% eigenfaces
\subsection{Example: Eigenfaces and Facial Recognition}
One famous result are the so-called eigenfaces. The data matrix $\mathbf{X}\in\mathbb{R}^{N\times p}$ consists of $N$ pictures of faces that each have $p$ pixels. The right singular eigenvectors yield $p$ eigenfaces in terms of which any of the $N$ pictures can be expressed.

Below are the first eigenfaces extracted from the "Labeled Faces in the Wild" dataset, which includes 13233 portraits with 62x47=2914 pixels each. Running SVD on the data matrix $\mathbf{X} \in \mathbb{R}^{13233\times 2914}$ yields the eigenfaces shown in Figure \ref{fig:svd_eigenfaces}.

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{svd_eigenfaces.png}
    \caption{Left: The first 48 eigenfaces. The colorscale is consistent across the images. As can be expected, the eigenfaces seem to have an ordering from more general features, that are highly prevalent in the dataset, towards more specific features. Right: Five sample portraits from the dataset approximated using different numbers of eigenvectors. 2914 corresponds to the original image, which had 2914 pixels (degrees of freedom).}
    \label{fig:svd_eigenfaces}
\end{figure}

Facial recognition may be performed by projecting a new face into the space of eigenfaces and matching to the coefficients of a known subject. This could be done using the euclidean distance, or it could be done using a classifier. In case of classificiation, the dimensionality reduction that is enabled by approximating images with a smaller set of eigenvectors may be critical to making the problem tractable by overcoming the curse of dimensionality. When the data matrix is not centered (that is, the mean is not subtracted from it before performing the SVD), then the first right singular vector is vector close to the average row in the matrix.

% eigenbasis
\subsection{Tracking an Eigensystem over Time}
\label{sec:svd_tracking}

As discussed in section \ref{sec:svd}, the sign of the basis vectors can be flipped without affecting the validity of the singular value decomposition. That means that if the SVD is performed on some system may equally well return, say, a left-handed or a right-handed coordinate system. This becomes a problem when the results of repeated SVDs are supposed to be compared, for example to study the evolution of a system over time. Figure \ref{fig:svd_consistently_oriented} shows the eigenbasis of a bivariate Gaussian with principal axes slowly rotating over a 180-degree angle. The top shows how the eigenbases that is found flips back and forth, so that there is a double-trajectory corresponding to results with positive and negative sign on an eigenvector. The trajectory is not described by an injective function, which complicates analysis.

\subsubsection{Heuristic Method}
Say an SVD is performed on a slowly-evolving system at time $t=0$ and at time $t=1$. To compensate the sign flips, a heuristic method is to first match the basis vectors at $t=0$ to the basis vectors at $t=1$ using a distance metric that is immune to the sign of the vectors, for example the absolute value of the dot product. (This assumes that the vectors are similar enough at time $t=0$ and $t=1$ that the matching an be done unambiguously. For the higher-order singular vectors this might be a problem, because they have lower numerical certainty.) Once the vectors are matched, the sign of the inner product between the vectors at $t=0$ and $t=1$ may be compared, and the sign flipped accordingly.

This method works, but the problem is that the sign of the vectors is somewhat arbitrarily pinned relative to the result at $t=0$. That is, if, for example, the coordinate system found at $t=0$ was left-handed, then the time series of eigenbases will be adjusted to be left-handed. If the point of comparison had instead been the SVD performed at some other time $t=t'$, then one might have wound up with a right-handed coordinate system instead. It is desirable to find a consistent orientation.

\subsubsection{Consistent Method}

\citeasnoun{damask2019consistently} recently developed a method that can be used to find a consistently oriented basis. Consistent orientation in this case means, roughly, that the eigenbasis is always flipped to obey the convention of being right-handed. The method relies on reconstructing the rotations and reflections necessary to transform the eigenbasis in question to the natural basis $\mathbf{I}$ as reference. While rotations preserve the orientation of a basis, but reflections do not. When the reflections of an eigenbasis with respect to the natural basis are known, then they can be undone by flipping them back in place.

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{svd_consistently_oriented.png}
    \caption{Right singular vectors extracted using SVD, showing sign flips (top) and with a consistently oriented basis (bottom). The underlying data is a bivariate gaussian with principal axes undergoing a 180 degree rotation. With the consistently oriented basis, the singular vectors trace out a one-to-one trajectory that can be analyzed.}
    \label{fig:svd_consistently_oriented}
\end{figure}

\subsubsection{Rank Order Changes}

The singular vectors or eigenvectors of a system are labeled only in terms of their associated singular values or eigenvalues. In order to track a singular vector throughout rank order changes, it is necessary to figure out a way to attach a separate label, for example by looking at the "content" of a particular singular vector. How well that works has to be figured out in context, I am not currently aware of a generally valid solution.

% Bayesian SVD
\subsection{Bayesian SVD}
To do! Cf. https://ieeexplore.ieee.org/document/7336426

It's awful to be tied to the assumption of normality and it's awful to not know the uncertainty of my singular vectors!


%!TEX root = /Volumes/HD_2/Rudin/Rudin_DM.tex
\textit{
Put $K=[-1,1]$; define $\D_{K}$ as in section 1.46
(with $\R$ in place of $\R^{n}$).
Supose $\{f_{n}\}$ is a sequence of Lebesgue integrable functions such that
%
  $\Lambda\phi
    =
  \underset{n \to \infty}{\lim} \int_{\minus 1}^1 f_{n}(t)\phi(t)dt$
%
exists for every $\phi\in\D_{K}$.
Show that $\Lambda$ is a continuous linear functional on $\D_{K}$.
Show that there is a positive integer $p$ and a number $M<\infty$ such that
  \begin{align}
    \left\lvert
      \int_{\minus 1}^1 f_n (t)\phi (t) dt\
    \right\rvert
    \leq
    M \norm{\infty}{D^{p}}
  \nonumber
  \end{align}
for all $n$.
For example, if $f_{n}(t)=n^{3}t$ on $[\minus 1/n, 1/n]$ and $0$ elsewhere,
show that this can be done with $p=1$.
Construct an example where it can be done with $p=2$ but not with $p=1$.}
%
%
\renewcommand{\labelenumi}{(\roman{enumi})}%
%
\newline\newline\noindent
We will also consider the case $p=0$. Since all supports of %
%
  $\phi, \phi', \phi'', \dots, $ are in $K$, %
%
we make a specialization of the mean value theorem: %
%
\paragraph{Lemma}\label{2.3 Lemma}
\input{\ROOT/chapter_2/2_03/2_03_0_lemma.tex}
%: PROOF OF THE STATEMENT ----------------------------------------------------%
%/ FIRST PART ----------------------------------------------------------------%
%
\begin{proof}
\input{\ROOT/chapter_2/2_03/2_03_1_radon_measures.tex}
% SECOND PART -----------------------------------------------------------------
\input{\ROOT/chapter_2/2_03/2_03_2_uniform_bound.tex}
\input{\ROOT/chapter_2/2_03/2_03_3_example_1.tex}
\input{\ROOT/chapter_2/2_03/2_03_4_example_2.tex}
\end{proof}
%\renewcommand{\labelenumi}{\alph{enumi}.}%
\renewcommand{\labelenumi}{$(\textit{\alph{enumi}})$}%


\section{Theory}
\label{sec:theory}

The aim of the MinOver algorithm is to find the perceptron of maximum stability, i.e. to maximize the margin between the weight vector $\bm{\mathsf{w}}$ and the closest input example $\xi^\mu$. Formally, the stability $\kappa^\mu$ is defined in \cref{eq:perceptron-stability}:

\begin{equation}
    \kappa^\mu = \frac{\mathsf{\bm{w}} \cdotp \xi^\mu S^\mu_R}{\lvert \mathsf{\bm{w}} \rvert}
    \label{eq:perceptron-stability}
\end{equation}

During the training phase, the MinOver algorithm looks for the input vector with the lowest stability at the current time $t$ (\cref{eq:minimum_lookup}) and performs a Hebbian update using its value (\cref{eq:hebbian_update}):

\begin{equation}
    \kappa^{\mu(t)} = \min_\nu \left \{ \kappa^{\nu(t)} =  \frac{\mathsf{\bm{w}}(t) \cdotp \xi^\nu S^\nu_R}{\lvert \mathsf{\bm{w}}(t) \rvert} \right \}
    \label{eq:minimum_lookup}
\end{equation}

\begin{equation}
    \mathsf{\bm{w}}(t+1) = \mathsf{\bm{w}}(t) + \frac{1}{N} \xi^{\mu(t)} S^{\mu(t)}_R
    \label{eq:hebbian_update}
\end{equation}

The procedure is repeated for a certain number of epochs $n_{max}$ or until convergence, i.e. when $\mathsf{\bm{w}}(t+1) = \mathsf{\bm{w}}(t)$.

Like in the Rosenblatt perceptron, the output $S^\mu \in \{+1, -1\}$ for a given examples is computed by evaluating the sign of the dot product between the example $\xi^\mu$ and the weights vector $\mathsf{\bm{w}}$, as shown in \cref{eq:labels_definition}:

\begin{equation}
    S^\mu = sign(\mathsf{w} \cdotp \xi^\mu)
    \label{eq:labels_definition}
\end{equation}


\lab{Algorithm}{The Knapsack Problem}{The Knapsack Problem}
\label{lab:Knapsack}

\objective{This section teaches about NP-hard problems, particularly the knapsack problem.}

\section*{The Knapsack Problem}

\begin{figure}
\includegraphics[width=\textwidth]{Knapsack.png}
\caption{Visualization of the Knapsack problem TODO: Make a picture similar to this}
\end{figure}

You are given a knapsack that can only hold a certain weight.
You have a plethora of items that have value and weight.
You want to find the combination of items that have the most value with out exceeding the weight.
This is the 2 dimensional knapsack problem.
Will focus on the 0/1 knapsack problem, which states that each item must be put in entirely or not at all.

A three dimensional version the items would have volume and the knapsack can only hold an certain volume as well as weight.

This problem is has many different applications for decision making in a wide variety of fields.
Two examples are finding the least wasteful way to cut raw materials and the selection of capital investments in financial portfolios.
This problem often appears in problems involving resource allocation and financial constraints.

\begin{problem}
Write a function that solves the Knapsack problem by testing all possible combinations of items and choosing the combination with the most value that meets the weight constraint.
Only test it up to 15 objects.
\end{problem}

This way finds the optimal solution, but there is a problem.
Given $n$ objects the number of combinations is $2^n$.
The complexity of the problem grows very quickly.
For example: if you had 50 items, it would take more than 10 years to compute the optimal solution.
For all the following timing problems let your items have values between 1 and 100 and weights between 1 and $\frac{Capacity}{10}$.

\begin{problem}
Time the Knapsack problem for $11-20$ items with a carrying capacity of $10,000$.
Plot the times.
What is the complexity of the algorithm for increasing the number of items?

\begin{figure}[H]
\includegraphics[width=\textwidth]{naiveTime.pdf}
\caption{
Your graph should look similar to this.
As you can see as the number of items increases time increases exponentially}
\end{figure}
\end{problem}

\begin{problem}
Time that algorithm for the carrying capacities $10,000-90,000$ every multiple of $10,000$ with $15$ items.
Plot the times.
What is the complexity of the algorithm for increasing the weight?

\begin{figure}[H]
\includegraphics[width=\textwidth]{naiveWeight.pdf}
\caption{
Your graph should look like this.
The weight of the items has essentially no effect on the time.}
\end{figure}
\end{problem}

\section*{NP-Hard}

Any problem that is not polynomial time is known as NP-hard.
The Knapsack Problem is an NP-hard problem since it is $\mathcal{O}\left(2^n\right)$ where $n$ is the number of items that are available.
Using the branch and bound approach we can find the optimal solution in psuedo-polynomial time.

\section*{Branch and Bound method}

We calculate the optimal solution using only the first $i$ items for weights 0 through W (where W is the maximum weight the knapsack can hold).

(ALGORITHM DESCRIPTION NEEDS REVISION)
We do this by initializing the 0th row to be empty.
For the $i$'th row (where $i>0$) we go through $j$ from 1 through W.
If the ith element's weight is more than j then the optimal combination is the same as it was for i-1.
If ith element's weight is less than j, then we compare the combination at i-1 at weight j to the combination i-1 at the weight j-the ith element weight plus the ith element.
Whichever one has a higher value is the optimal combination using the first ith elements less than or equal to the weight of j.
You continue this until you have done all n elements.
The combination using n items with a weight of W is guaranteed to be the optimal combination.

This only does $W*n$ checks, so it is a lot faster.
This works by eliminating combinations that could not be the optimal solution.

\begin{problem}
Write a function that solves the Knapsack problem using the branch and bound method.
\end{problem}

\begin{problem}
Time the Knapsack problem for $11-20$ items with a carrying capacity of $10,000$.
Plot the times.
What is the complexity of the algorithm with respect to the number of items?
For the items, use the specifications given for the previous timing problems.

\begin{figure}[H]
\includegraphics[width=\textwidth]{dynamicTime.pdf}
\caption{
Your graph should look similar to this.
As the number of items increases time increases linearly}
\end{figure}
\end{problem}

\begin{problem}
Time the branch and bound algorithm for the carrying capacities $10,000-90,000$ every multiple of $10,000$ with $15$ items.
Plot the times.
What is the complexity of the algorithm with respect to weight?

\begin{figure}[H]
\includegraphics[width=\textwidth]{dynamicWeight.pdf}
\caption{
Your graph should look similar to this.
As the weight of items increases time increases linearly}
\end{figure}
\end{problem}


\title{Degrees of Freedom}
\author{
        Daniel Johnson \\
                Division of Applied Mathematics\\
        Brown University
}
\date{\today}

\documentclass[12pt]{article}

\usepackage{graphicx,amsfonts,amsbsy,bbm, amsmath}

\begin{document}
\maketitle
%\tableofcontents

%\begin{abstract}
%\end{abstract}

\section{Computing the Degrees of Freedom of a 3D Linkage}

We consider the linkage of rigid 2-dimensional polygons in $\mathbb{R}^3$ by means of ideal hinges located at the edges of the polygons. In this way, by treating intermediates of our various models as such a linkage, we can compute the non-trivial degrees of freedom (DoF) each intermediate has. Every linkage of this type has 6 trivial degrees of freedom: 3 corresponding to translational movement and 3 corresponding to rotational freedom. While these freedoms do change the orientation of the intermediate, they do not cause the faces to move relative to each other. For this reason, we label them as trivial degrees of freedom and focus on the calculation the remaining, non-trivial degrees of freedom. Since the concept of degrees of freedom exists in many diverse scientific fields, such as mechanical engineering and statistical physics, many different formal definitions of DoF are used in the literature. McCarthy defines degrees of freedom of a mechanical system as follows.
\begin{quote}
We derive formulas for the number of parameters needed to specify the configuration of a mechanism, in terms of the number of links and joints and the freedom of movement allowed at each joint. This number is the \textit{degrees of freedom} or \textit{mobility} of the mechanism. Changing the values of these parameters changes the configuration of the mechanism. Thus, if we view the set of all configuration available to a mechanism as a manifold, then the mobility of the mechanism is the dimension of this manifold.
\end{quote}
For our purposes, we define the \textbf{degrees of freedom} of a configuration to be the difference between the number of parameters needed to specify a configuration in an ambient parameter space $\Omega$ and the number of independent constraints imposed upon the configuration as a function of these parameters. Consequently, the non-trivial degrees of freedom is the number of degrees of freedom minus the six trivial degrees of freedom. If the ambiant parameter space is $\mathbb{R}^N$ and there are $M$ constraint equations given by $\varphi : \mathbb{R}^N \to \mathbb{R}^M$, we find the degrees of freedom to be $N - \operatorname{rank}(\varphi)$ and the non-trivial degrees of freedom to be $N - \operatorname{rank}(\varphi) - 6$. Here we need $\varphi$ to be a smooth map so that the rank is well defined. Furthermore, we define the \textbf{configuration space} to be the subset of parameter space $\{\omega \in \Omega : \varphi(\omega) = 0\}$ upon which the contraints are satisfied.

In our case, the location and orientation of each polygon in the linkage can be represented by a total of 6 parameters: 3 for location and 3 for roational orientation. Thus, the entire configuration of an intermediate $x$ can be represented in the parameter space $\mathbb{R}^{F_x\times6}$. With constraints corresponding to hinged connections we construct a function $\varphi$ which is zero if and only if the constraints are satisfied. Thus the configuration space is $\{z \in \mathbb{R}^{F_x\times6} : \varphi(z) =0\}$. As we will show below, the constraint equation $\varphi$ can be constructed as a polynomial which means that the configuration space (as the zero set of this polynomial) is an algebraic variety.

 PARAGRAPH ON DIFFERENT DOF VALS AT DIFFERENT CONFIGS OF SAME INTERMEDIATE We refer to the embedded configuration of $x$ such that $x$ is a subset of the original polyhedron as the \textbf{canonical configuration}.

To mathematically describe a particular configuration, we specify the locations of the vertices of each face. Thus, the configuration space can be described as a manifold embegged in $\mathbb{R}^{3\times N_x}$ where $N_x \doteq \sum_{f\in \left(\mathcal{P}\right)}s_f\mathbbm{1}_{f\subset x}$ and $s_f$ is the number of sides (and verticies) of face $f$. By parameterizing the configuration space as enmedding of the ambient space $\mathbb{R}^{3\times N_x}$, we must then identify the corresponding constaint equations as a function of points in ambient space. It is worth noting that while we represent each face with $3\times s_b$ coordinates, only 6 are required to speciafy a face's position and orientation if they are chosen carefully. However, this redundancy will be removed via the constraint equations. Notationally, we refer to the $k$th vertex of the $j$th face of $x$ as $v^{jk} = \left(v^{jk}_x,v^{jk}_y,v^{jk}_z\right)$. There are five fundamental types of constraint equations: base constraints, edge length constraints, vertex identification constraints, angle constraints, and 2D face constraint.

The base constaints are put in place to fix one of the intermediate's faces in space to remove the 6 trivial degrees of freedom from the calculation. If $f_b$ is the face we wish to designate as the base we have the following $3\times s_b$ constraint equations
\begin{align}
\psi_{base}^{k,x}\left(\mathbf{v}\right)& = v^{b,k}_x - c^{b,k}_x \\
\psi_{base}^{k,y}\left(\mathbf{v}\right)& = v^{b,k}_y - c^{b,k}_y \\
\psi_{base}^{k,z}\left(\mathbf{v}\right)& = v^{b,k}_z - c^{b,k}_z
\end{align}
for $k = 1,\dots,s_b$ and with $c^{b,k}_\bullet$ a known constant. The corresponding Jacobian calculation for each constraint equations is:
\[
\frac{\partial\psi_{base}^{k,\bullet}}{\partial v} =
  \begin{cases}
   1 & \text{if } v = v^{b,k}_\bullet \\
   0       & \text{else}
  \end{cases}
\]

Edge length constraints enforce that the lengths of the edges of each face in an intermediate cannot change. Since there are $s_j$ edges for each face $f_j$ of intermediate $x$, there are $N_x$ corresponsing edge constraints.
\begin{align}
\psi_{edge}^{j,k}\left(\mathbf{v}\right)& = \left|v^{j,k} - v^{j,k-1}\right|^2 - \ell_{j,k}^2 \\
& = \left(v_x^{j,k} - v_x^{j,k-1}\right)^2 +\left(v_y^{j,k} - v_y^{j,k-1}\right)^2 +\left(v_z^{j,k} - v_z^{j,k-1}\right)^2 - \ell_{j,k}^2
\end{align}
for all $j: f_j \subset x$ and with the convention that $v^{j,0} \doteq v^{j,s_j}$ and $\ell_{j,k}$ is a known constant. The resulting partial derivatives are
\[
	\frac{\partial\psi_{edge}^{j,k}}{\partial v} =
  	\begin{cases}
        	2\left(v^{j,k}_\bullet-v^{j,k-1}_\bullet\right) 	& \text{if } v = v^{j,k}_\bullet \\
   		-2\left(v^{j,k}_\bullet-v^{j,k-1}_\bullet\right) 	& \text{if } v = v^{j,k-1}_\bullet \\
   		0       & \text{else}
  	\end{cases}
\]
%--definiteion of 'base configuration'
%--representation of config space definition as system of constraint equations
%--solution of the constraint equations is a manifold except perhaps at some 'sp0ecial' configurations
%--define DoF as 6N - rank of the jacobian of the CEqs evaluated at the base config

%--go through computation of jacobian

\end{document}