\section{Linear Algebra and Calculus}
\subsection{General Notation}
A vector $x \in \mathbb{R}^n$ has $n$ entries, and $x_i \in \mathbb{R}$ is the
$i$-th entry:
\begin{equation}
  x=\left(\begin{array}{c}x_1\\x_2\\\vdots\\x_n\end{array}\right)\in\mathbb{R}^n
\end{equation}
We denote a matrix $A \in \mathbb{R}^{m \times n}$ with $m$ rows and $n$
columns, and $A_{ij} \in \mathbb{R}$ is the entry in the $i$-th row and
$j$-th column:
\begin{equation}
  A=\left(\begin{array}{ccc}A_{11}& \cdots&A_{1n}\\\vdots&& \vdots\\A_{m1}& \cdots&A_{mn}\end{array}\right)\in\mathbb{R}^{m\times n}
\end{equation}
Vectors can be viewed as a $n \times 1$ matrix.
\subsubsection{Indentity Matrix}
The identity matrix $I\in\mathbb{R}^{n\times n}$ is a square matrix with ones
along the diagonal and zero everywhere else:
\begin{equation}
  I=\left(\begin{array}{cccc}1&0& \cdots&0\\0& \ddots& \ddots& \vdots\\\vdots& \ddots& \ddots&0\\0& \cdots&0&1\end{array}\right)
\end{equation}
For all matrices $A \in \mathbb{R}^{n \times n}$ we have
$A \times I = I \times A = A$.
\subsubsection{Diagonal Matrix}
A diagonal matrix $D \in \mathbb{R}^{n \times n}$
is a square matrix with nonzero values along the diagonal and zero everywhere
else:
\begin{equation}
  D=\left(\begin{array}{cccc}d_1&0& \cdots&0\\0& \ddots& \ddots& \vdots\\\vdots& \ddots& \ddots&0\\0& \cdots&0&d_n\end{array}\right)
\end{equation}
The diagonal matrix $D$ is also written as $\textrm{diag}(d_1,\hdots,d_n)$.
\subsubsection{Orthogonal Matrix}
Two vectors $x, y \in \mathbb{R}^n$ are orthogonal if $x^T y = 0$. A vector
$x \in \mathbb{R}^n$ is normalized if $||x||_2 = 1$.
A square matrix $U\in \mathbb{R}^{n \times n}$ is orthogonal if all its
columns are orthogonal to each other and are normalized. Hence:
\begin{equation}
  U^TU = I = UU^T
\end{equation}
Hence the inverse of an orthogonal matrix is its transpose.m
\subsection{Matrix Operations}
\subsubsection{Vector-Vector Products}
Given two vectors $x$, $y \in \mathbb{R}^n$, the inner product is:
\begin{equation}
  x^T y = \sum_{i=1}^n x_i y_i \in \mathbb{R}
\end{equation}
The outer product for a vector $x \in \mathbb{R}^m$, $y \in \mathbb{R}^n$ is:
\begin{equation}
  xy^T=\left(\begin{array}{ccc}x_1y_1& \cdots&x_1y_n\\\vdots&\ddots& \vdots\\x_my_1& \cdots&x_my_n\end{array}\right)\in\mathbb{R}^{m\times n}
\end{equation}
\subsubsection{Vector-Matrix Products}
The product of a matrix $A \in \mathbb{R}^{m \times n}$ and vector
$x \in \mathbb{R}^n$ is a vector $y = Ax \in \mathbb{R}^m$. If we write $A$
by the rows, $Ax$ is expressed as:
\begin{equation}
  y = A x = \left( \begin{array} { c } { - \,\, a _ { 1 } ^ { T } \,\,- } \\ { - \,\,a _ { 2 } ^ { T }\,\, -} \\ { \vdots } \\ { - \,\, a _ { m } ^ { T } \,\,-} \end{array} \right) x = \left( \begin{array} { c } { a _ { 1 } ^ { T } x } \\ { a _ { 2 } ^ { T } x } \\ { \vdots } \\ { a _ { m } ^ { T } x } \end{array} \right)
\end{equation}
Here, the $i$-th entry of $y$ is the inner product of the $i$-th row of $A$ and
$x$, $y_i = a_i^T x$. If we write $A$ is column form:
\begin{equation}
  y = A x = \left( \begin{array} { c c c c } { | } & { | } & { } & { | } \\ { a _ { 1 } } & { a _ { 2 } } & { \cdots } & { a _ { n } } \\ { | } & { | } & { } & { | } \end{array} \right) \left( \begin{array} { c } { x _ { 1 } } \\ { x _ { 2 } } \\ { \vdots } \\ { x _ { n } } \end{array} \right) = a _ { 1 } x _ { 1 } + a _ { 2 } x _ { 2 } + \ldots +  a _ { n } x _ { n }
\end{equation}
Here, $y$ is a linear combination of the columns of $A$, where the coefficients
of the linear combination are given by the entries of $x$.
\subsubsection{Matrix-Matrix Products}
Given a matrix $A\in \mathbb{m \times n}$ and matrix $B \in \mathbb{n \times p}$,
we can define $C = AB$ as follows:
\begin{equation}
  C = AB = \left( \begin{array} { c c c c } { a _ { 1 } ^ { T } b _ { 1 } } & { a _ { 1 } ^ { T } b _ { 2 } } & { \cdots } & { a _ { 1 } ^ { T } b _ { p } } \\ { a _ { 2 } ^ { T } b _ { 1 } } & { a _ { 2 } ^ { T } b _ { 2 } } & { \cdots } & { a _ { 2 } ^ { T } b _ { p } } \\ { \vdots } & { \vdots } & { \ddots } & { \vdots } \\ { a _ { m } ^ { T } b _ { 1 } } & { a _ { m } ^ { T } b _ { 2 } } & { \cdots } & { a _ { m } ^ { T } b _ { p } } \end{array} \right)
\end{equation}
Hence, each $(i,j)$-th entry of $C$ is equal to the inner product of the $i$-th
row of $A$ and the $j$-th column of $B$. Compactly:
\begin{equation}
  C_{ij} = a_i^T b_j = \sum_{k=1}^n a_{ik} b_{kj}
\end{equation}
\subsubsection{The Transpose}
The transpose of a matrix $A\in \mathbb{R}^{m \times n}$ is
$A^T \in \mathbb{R}^{n \times m}$ matrix whose entries are:
\begin{equation}
  (A^T)_{ij} = A_{ji}
\end{equation}
Properties of the transpose:
\begin{enumerate}
  \item $(A^T)^T = A$
  \item $(AB)^T = B^T A^T$
  \item $(A + B)^T = A^T + B^T$
\end{enumerate}
\subsubsection{The Trace}
The trace of a square matrix $A \in \mathbb{R}^{n \times m}$ is denoted
$\operatorname{tr}(A)$. It is the sum of diagonal elements in the matrix:
\begin{equation}
  \operatorname{tr}A = \sum_{i=1}^n A_{ii}
\end{equation}
Properties of the trace:
\begin{enumerate}
  \item For $A \in \mathbb{R}^{n \times n}$, $\operatorname{tr}A=\operatorname{tr}A^T$
  \item For $A, B \in \mathbb{R}^{n \times n}$, $\operatorname{tr}(A+B)=\operatorname{tr}A+\operatorname{tr}B$
  \item For $A \in \mathbb{R}^{n \times n}$, $t\in \mathbb{R}$, $\operatorname{tr}(tA)=t \cdot \operatorname{tr}A$
  \item For $A, B$ such that $AB$ is square, $\operatorname{tr}AB = \operatorname{tr}BA$
  \item For $A,B,C$ such that $ABC$ is square, $\operatorname{tr}ABC=\operatorname{tr}BCA = \operatorname{tr}CAB$, and so on
\end{enumerate}
\subsubsection{The Inverse}
The inverse of a matrix $A$ is noted $A^{-1}$ and is the unique matrix
such that:
\begin{equation}
  A^{-1}A=I=AA^{-1}
\end{equation}
Not all square matrices are invertible. In addition, assuming
$A, B \in \mathbb{R}^{n \times n}$ are non-singular:
\begin{enumerate}
  \item $(A^{-1})^{-1} = A$
  \item $(AB)^{-1} = B^{-1}A^{-1}$
  \item $(A^{-1})^T = (A^T)^{-1}$
\end{enumerate}
\subsubsection{The Determinant}
The determinant of a square matrix $A\in\mathbb{R}^{n\times n}$, noted $|A|$
or $\textrm{det}(A)$ is expressed recursively in terms of
$A_{\backslash i, \backslash j}$, which is the matrix $A$ without
its $i$-th row and $j$-th column, as follows:
\begin{equation}
  \textrm{det}(A)=|A|=\sum_{j=1}^n(-1)^{i+j}A_{i,j}|A_{\backslash i,\backslash j}|
\end{equation}
Remark that $A$ is invertible if and only if $|A| \not= 0$. Also, $|AB|=|A||B|$
and $|A^T|=|A|$.
\subsection{Matrix Properties}
\subsubsection{Norms}
A norm of a vector $x$ is any function $f:\mathbb{R}^{n} \rightarrow \mathbb{R}$
that satisfies 4 properties:
\begin{enumerate}
  \item Non-negativity: For all $x \in \mathbb{R}^n$, $f(x) \geq 0$
  \item Definiteness: $f(x)=0$ if and only if $x=0$
  \item Homogeneity: For all $x \in \mathbb{R}^n$, $t \in \mathbb{R}$,
    $f(tx) = |t| f(x)$
  \item Triangle Inequality: For all $x, y \in \mathbb{R}^n$, $f(x+y) \leq f(x) + f(y)$
\end{enumerate}
However, most norms used come from the family of $\ell_p$ norms:
\begin{equation}
  \ell_p = ||x||_p = \left( \sum_{i=1}^n x_i^p \right)^{\frac{1}{p}}
\end{equation}
The $p$-norm is used in Holder's inequality. The Manhattan norm, used in
LASSO regularization, is:
\begin{equation}
  \ell_1 = ||x||_1 = \sum_{i=1}^n |x_i|
\end{equation}
The euclidean norm, $\ell_2$ is used in ridge regularization and
distance measures:
\begin{equation}
  \ell_2 = || x ||_2 = \sqrt{\sum_{i=1}^n x_i^2}
\end{equation}
Finally, the infinity norm is used in uniform convergence:
\begin{equation}
  \ell_{\infty} = || x ||_{\infty} = \underset{i}{\textrm{max }}|x_i|
\end{equation}
The Frobenius norm of a matrix is analogous to the $\ell_2$ norm of a vector:
\begin{equation}
  ||A||_F = \sqrt{\sum_{ij}A^2_{ij}}
\end{equation}
The dot product of two vectors can be expressed in terms of norms:
\begin{equation}
  x^T y = ||x||_2 ||y||_2 \cos \theta
\end{equation}
where $\theta$ is the angle between vectors $x$ and $y$.
\subsubsection{Linear Dependence and Rank}
A set of vectors $\{x_1, x_2, \hdots, x_n\} \subset \mathbb{R}^m$ is linearly
independent if no vector can be represented as a linear combination of the
remaining vectors. Conversely, if one vector in the set can be represented
as a linear combination of the remaining vectors, then the vectors are
said to be linearly dependent. Formally, for scalar values
$\alpha_1, \hdots, \alpha_{n-1} \in \mathbb{R}$:
\begin{equation}
  x_n = \sum_{i=1}^{n-1} \alpha_i x_i
\end{equation}
The rank of a given matrix $A$ is noted $\operatorname{rank}(A)$
and is the dimension of the vector space generated by its columns.
This is equivalent to the maximum number of linearly independent columns of
$A$. If $\operatorname{rank}(A) = \min(m,n)$, then $A$ is said to be full rank.m
\subsubsection{Span, Range, and Nullspace}
The span of a set of vectors $\{x_1, x_2, \hdots, x_n\}$ is the set of all
vectors that can be expressed as a linear combination of
$\{x_1, x_2, \hdots, x_n\}$. Formally,
\begin{equation}
  \operatorname { span } \left( \left\{ x _ { 1 } , \ldots x _ { n } \right\} \right) = \left\{ v : v = \sum _ { i = 1 } ^ { n } \alpha _ { i } x _ { i } , \quad \alpha _ { i } \in \mathbb { R } \right\}
\end{equation}
The projection of a vector $y \in \mathbb{R}^n$ onto the span of
$\{x_1, x_2, \hdots, x_n\}$ is the vector
$v \in \operatorname { span } \left( \left\{ x _ { 1 } , \ldots x _ { n } \right\} \right)$
such that $v$ is as close as possible to $y$ as measured by the euclidean norm:
\begin{equation}
  \operatorname { Proj } \left( y ; \left\{ x _ { 1 } , \ldots x _ { n } \right\} \right) = \operatorname { argmin } _ { v \in \operatorname { span } \left( \left\{ x _ { 1 } , \ldots , x _ { n } \right\} \right) } \| y - v \| _ { 2 }
\end{equation}
The range of a matrix $A\in \mathbb{R}^{m \times n}$ is the span of the columns
of $A$:
\begin{equation}
  \mathcal { R } ( A ) = \left\{ v \in \mathbb { R } ^ { m } : v = A x , x \in \mathbb { R } ^ { n } \right\}
\end{equation}
The nullspace of a matrix $A \in \mathbb{R}^{m \times n}$ is the set of all
vectors that equal $0$ when multiplied by $A$:
\begin{equation}
  \mathcal { N } ( A ) = \left\{ x \in \mathbb { R } ^ { n } : A x = 0 \right\}
\end{equation}
\subsubsection{Symmetric Matrices}
A square matrix $A \in \mathbb{R}^{n \times n}$ is symmetric if $A=A^T$. It
is anti-symmetric if $A=-A^T$. For any matrix $A\in \mathbb{R}^{n \times n}$
the matrix $A+A^T$ is symmetric and the matrix $A-A^T$ is anti-symmetric.
From this, any square matrix $A \in \mathbb{R}^{n \times n}$ can be represented
as a sum of a symmetric matrix and an anti-symmetric matrix:
\begin{equation}
  A=\underbrace{\frac{A+A^T}{2}}_{\textrm{Symmetric}}+\underbrace{\frac{A-A^T}{2}}_{\textrm{Antisymmetric}}
\end{equation}
\subsubsection{Positive Semidefinite Matrices}
Given a square matrix $A \in \mathbb{R}^{n \times n}$ and a vector
$x \in \mathbb{R}^n$, the scalar value $x^TAx$ is called the quadratic form:
\begin{equation}
  x^TAx = \sum_{i=1}^n \sum_{j=1}^n A_{ij} x_i x_j
\end{equation}
A symmetric matrix $A$ is:
\begin{enumerate}
  \item Positive definite (PD) if for all nonzero vectors, $x^TAx > 0$
  \item Positive semidefinite (PSD) if for all nonzero vectors, $x^TAx \geq 0$
  \item Negative definite (ND) if for all nonzero vectors, $x^TAx < 0$
  \item Negative semidefinite (NSD) if for all nonzero vectors, $x^TAx \leq 0$
  \item Indefinite if it is neiter PSD nor NSD, i.e. if there exists a $x_1, x_2$
    such that $x^T_1Ax_1 > 0$ and $x^T_2Ax_2 < 0$
\end{enumerate}
Given any matrix $A \in \mathbb{R}^{m \times n}$, the gram matrix $G = A^T A$
is always PSD. If $m \geq n$ and $A$ is full rank, then $G$ is PD.
\subsubsection{Eigendecomposition}
Given a square matrix $A \in \mathbb{R}^{n \times n}$, we say that
$\lambda \in \mathbb{C}$ is an eigenvalue of $A$ and $v \in \mathbb{C}$ is the
corresponding eigenvector if
\begin{equation}
  Av = \lambda v, \quad v \not= 0
\end{equation}
The trace of $A$ is equal to the sum of its eigenvalues:
\begin{equation}
  \operatorname{tr}A = \sum_{i=1}^n \lambda_i
\end{equation}
The determinant of $A$ is equal to the product of its eigenvalues:
\begin{equation}
  |A| = \prod_{i=1}^n \lambda_i
\end{equation}
Suppose matrix $A$ has $n$ linearly independent eigenvectors
$\{v_1, \hdots, v_n\}$ with corresponding eigenvalues
$\{\lambda_1, \hdots, \lambda_n\}$. Let $V$ be a matrix with one eigenvalue
per column: $V = \left[ v_1, \hdots, v_n \right]$. Let
$\Lambda =\textrm{diag}(\lambda_1,\hdots,\lambda_n)$. Then the eigendecomposition
of $A$ is given by:
\begin{equation}
  A = V \Lambda V^{-1}
\end{equation}
For when $A$ is symmetric, then the eigenvalues of $A$ are real, the
eigenvectors of $A$ are orthonormal, and hence $V$ is an orthogonal matrix
(henceforth renamed $U$). Hence $A = U \Lambda U^T$.
A matrix whose eigenvalues are all positive is called positive definite.
A matrix whose eigenvalues are all positive or zero valued is called positive
semidefinite. Likewise, if all eigenvalues are negative, the matrix is
negative definite, and if all eigenvalues are negative or zero valued,
it is negative semidefinite. In addition, assuming sorted eigenvalues,
for the following optimization problem:
\begin{equation}
  \max _ { x \in \mathbb { R } ^ { n } } x ^ { T } A x \quad \text { subject to } \| x \| _ { 2 } ^ { 2 } = 1
\end{equation}
The solution is $v_1$ the eigenvector corresponding to $\lambda_1$. For
the minimization problem:
\begin{equation}
  \min _ { x \in \mathbb { R } ^ { n } } x ^ { T } A x \quad \text { subject to } \| x \| _ { 2 } ^ { 2 } = 1
\end{equation}
the optimal solution for $x$ is $v_n$, the eigenvector corresponding to
eigenvalue $\lambda_n$.
\subsubsection{Singlular Value Decomposition}
The singular value decomposition provides a way to factorize a $m \times n$
matrix $A$ into singular vectors and singular values. It is defined as:
\begin{equation}
  A = U D V^T
\end{equation}
Suppose $A \in \mathbb{R}^{m \times n}$ matrix. Then
$U \in \mathbb{R}^{m \times m}$, $D \in \mathbb{R}^{m \times n}$,
$ V \in \mathbb{R}^{n \times n}$ matrices. The matricies $U$ and $V$ are
orthogonal, and matrix $D$ is diagonal. The elements along the diagonal of $D$
are known as the singular values of matrix $A$. The columns of $U$ are known
as the left-singular vectors while the columns of $V$ are the right-singular
vectors. The left-singular vectors of $A$ are the eigenvectors of $AA^T$.
The right-singular vectors of $A$ are the eigenvectors of $A^TA$. We can use
SVD to partially generalize matrix inversion to nonsquare matrices.
\subsubsection{The Moore-Penrose Pseudoinverse}
Matrix inversion is not defined for matrices that are not square.
Note that for nonsingular $A$:
\begin{equation}
  A A^{-1}A = A
\end{equation}
However, if the inverse is not defined, we seek to find a matrix $A^{+}$ such
that:
\begin{equation}
  A A^{+}A = A
\end{equation}
The moore-penrose pseudoinverse $A^+$ is defined as follows:
\begin{equation}
  A^+ = \lim_{\alpha \to 0} (A^TA + \alpha I)^{-1}A^T
\end{equation}
Practical algorithms use the singular value decomposition of $A$ such that:
\begin{equation}
  A^+ = V D^+ U^T
\end{equation}
where $U, D, V$ are from the SVD of $A$, and the pseudoinverse of $D^+$ is
obtained by taking the reciprocal of the nonzero diagonal elements of $D$.
If $A$ has more columns than rows, then using the pseudoinverse to solve a
linear equation $Ax=y$ provides one of many solutions, but provides $x = A^+ y$
with minimal euclidean norm $||x||_2$. When $A$ has more rows than columns,
the pseudoinverse gives us the $x$ for which $Ax$ is as close as possible
to $y$, i.e. minimizing $||Ax-y||_2$.
\subsection{Matrix Calculus}
\subsubsection{The Gradient}
Let $f:\mathbb{R}^{m\times n}\rightarrow\mathbb{R}$ be a function and
$A \in \mathbb{R}^{m \times n}$ be a matrix. The gradient of $f$ with respect to
$A$ is a $m \times n$ matrix noted as $\nabla_A f(A)$ such that:
\begin{equation}
  \nabla _ { A } f ( A ) \in \mathbb { R } ^ { m \times n } = \left(
  \begin{array} { c c c c } { \frac { \partial f ( A ) } { \partial A _ { 11 } } } & { \frac { \partial f ( A ) } { \partial A _ { 12 } } } & { \cdots } & { \frac { \partial f ( A ) } { \partial A _ { 1 } } } \\ { \frac { \partial f ( A ) } { \partial A _ { 21 } } } & { \frac { \partial f ( A ) } { \partial A _ { 22 } } } & { \cdots } & { \frac { \partial f ( A ) } { \partial A _ { 2 n } } } \\ { \vdots } & { \vdots } & { \ddots } & { \vdots } \\ { \frac { \partial f ( A ) } { \partial A _ { m 1 } } } & { \frac { \partial f ( A ) } { \partial A _ { m 2 } } } & { \cdots } & { \frac { \partial f ( A ) } { \partial A _ { m n } } } \end{array} \right)
\end{equation}
Or compactly for each $ij$ entry:
\begin{equation}
  \nabla _ { A } f ( A ) _ { i j } = \frac { \partial f ( A ) } { \partial A _ { i j } }
\end{equation}
However, the gradient of a vector $x \in \mathbb{R}^n$ is:
\begin{equation}
  \nabla _ { x } f ( x ) = \left( \begin{array} { c } { \frac { \partial f ( x ) } { \partial x _ { 1 } } } \\ { \frac { \partial f ( x ) } { \partial x _ { 2 } } } \\ { \vdots } \\ { \frac { \partial f ( x ) } { \partial x _ { n } } } \end{array} \right)
\end{equation}
\subsubsection{The Hessian}
Let $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ be a function and
$x \in \mathbb{R}^{n}$ be a vector. The hessian of $f$ with respect to
$x$ is a $n \times n$ symmetric matrix noted as $H = \nabla_x^2 f(x)$ such that:
\begin{equation}
  \nabla _ { x } ^ { 2 } f ( x ) \in \mathbb { R } ^ { n \times n } = \left( \begin{array} { c c c c } { \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { 1 } ^ { 2 } } } & { \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { 2 } } } & { \cdots } & { \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { 1 } \partial x _ { n } } } \\ { \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { 2 } \partial x _ { 1 } } } & { \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { 2 } ^ { 2 } } } & { \cdots } & { \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { 2 } \partial x _ { n } } } \\ { \vdots } & { \vdots } & { \ddots } & { \vdots } \\ { \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { n } \partial x _ { 1 } } } & { \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { n } \partial x _ { 2 } } } & { \cdots } & { \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { n } ^ { 2 } } } \end{array} \right)
\end{equation}
Or compactly:
\begin{equation}
  \nabla _ { x } ^ { 2 } f ( x ) _ { i j } = \frac { \partial ^ { 2 } f ( x ) } { \partial x _ { i } \partial x _ { j } }
\end{equation}
Note that the hessian is only defined when $f(x)$ is real-valued.
\subsubsection{Gradient Properties}
For matrices $A, B, C$ and vectors $x, b$:
\begin{enumerate}
  \item $\nabla_x b^Tx = b$
  \item $\nabla_x x^TAx = 2Ax$ (if $A$ symmetric)
  \item $\nabla^2_x x^TAx = 2A$ (if $A$ symmetric)
  \item $\nabla_A\textrm{tr}(AB)=B^T$
  \item $\nabla_{A^T}f(A)=\left(\nabla_Af(A)\right)^T$
  \item $\nabla_A\textrm{tr}(ABA^TC)=CAB+C^TAB^T$
  \item $\nabla_A|A|=|A|(A^{-1})^T$
\end{enumerate}


\subsection{More properties of inverses}

% ======================================================================

In this section, we will use elementary matrices to prove a useful
theorem about the inverse of a square matrix. We start with an
observation about the {\ef} of a right invertible matrix.

\begin{lemma}{{\Ef} of a right invertible matrix}{no-zero-row}
  Suppose that $A$ is right invertible. Then the {\rref} of $A$ does
  not have a row of zeros.
\end{lemma}

\begin{proof}
  Let $R$ be the {\rref} of $A$. Then by Theorem~\ref{thm:form-rua},
  we can write $R=UA$ for some invertible square matrix $U$. By
  assumption, we have $AB=I$, and therefore
  \begin{equation*}
    R(BU^{-1})
    ~=~
    (UA)(BU^{-1})
    ~=~
    U(AB)U^{-1}
    ~=~
    UIU^{-1}
    ~=~
    UU^{-1}
    ~=~
    I.
  \end{equation*}
  If $R$ had a row of zeros, then so would the product
  $R(BU^{-1})$. But since the identity matrix $I$ does not have a row
  of zeros, neither does $R$.
\end{proof}

\begin{theorem}{Right invertible square matrices are invertible}{right-square-left}
  Suppose $A$ and $B$ are square matrices such that $AB=I$. Then it
  follows that $BA=I$, and therefore $B=A^{-1}$. In particular, a
  square matrix is right invertible if and only if it is left
  invertible if and only if it is invertible.
\end{theorem}

\begin{proof}
  Assume $A$ and $B$ are square matrices such that $AB=I$. Let $R$ be
  the {\rref} of $A$. Then by Theorem~\ref{thm:form-rua}, we can write
  $R=UA$ where $U$ is an invertible matrix. Since $AB=I$, we know by
  Lemma~\ref{lem:no-zero-row} that $R$ does not have a row of
  zeros. Since $R$ is a square {\rref} with no row of zeros, each
  column must be a pivot column, and it follows that $R=I$. Hence,
  $UA=I$, and therefore $A$ is left invertible. Moreover, we have
  \begin{equation*}
    B ~=~ IB ~=~ (UA)B ~=~ U(AB) ~=~ UI ~=~ U,
  \end{equation*}
  and therefore $B=U$. It follows that $BA=UA=I$, as claimed.

  To prove the last claim, note that we just proved that for square
  matrices, $AB=I$ implies $BA=I$. Therefore, every right inverse of
  $A$ is also a left inverse, and therefore an inverse. But of course,
  we also have that $BA=I$ implies $AB=I$. This is just the same
  theorem, with the roles of $A$ and $B$ interchanged. Therefore,
  every left inverse of $A$ is also a right inverse.
\end{proof}

This theorem is very useful, because it allows us to test only one of
the products $AB$ or $BA$ in order to check that $B$ is the inverse of
$A$, saving us half of the work. It is important to stress, however,
that this only works for {\em square} matrices. As we saw in
Example~\ref{exa:right-inverse}, non-square matrices can be right
invertible without being left invertible, or vice versa.


\subsection{Variance}

\subsubsection{Definition}

The variance of a random variable is given by:

\(Var(x)=E((x-E(x))^2)\)

\(Var(x)=E(x^2+E(x)^2-2xE(x))\)

\(Var(x)=E(x^2)+E(E(x)^2)-E(2xE(x))\)

\(Var(x)=E(x^2)+E(x)^2-2E(x)^2\)

\(Var(x)=E(x^2)-E(x)^2\)

\subsubsection{Variance of a constant}

\(Var(c)=E(c^2)-E(c)^2\)

\(Var(c)= c^2-c^2\)

\(Var(c)=0\)

\subsubsection{Variance of multiple}

\(Var(cx)=E((cx)^2)-E(cx)^2\)

\(Var(cx)=E(c^2x^2)-[\sum_i cx P(x_i)]^2\)

\(Var(cx)=c^2E(x^2)-c^2[\sum_i x P(x_i)]^2\)

\(Var(cx)=c^2[E(x^2)- E(x)^2]\)

\(Var(cx)=c^2Var(x)\)

\subsubsection{Link between variance of expectation}

\(E(x)^2+Var(x)=E(x)^2+E((x-E(x))^2)\)

\(E(x)^2+Var(x)=E(x)^2+E(x^2+E(x)^2-2xE(x))\)

\(E(x)^2+Var(x)=E(x)^2+E(x^2)+E(E(x)^2)-E(2xE(x))\)

\(E(x)^2+Var(x)=E(x)^2+E(x^2)+E(x)^2-2E(x)E(x))\)

\(E(x)^2+Var(x)=E(x^2)\)

\subsubsection{Covariance}

\(Var(x+y)=E((x+y)^2)-E(x+y)^2\)

\(Var(x+y)=E(x^2+y^2+2xy)-E(x+y)^2\)

\(Var(x+y)=E(x^2)+E(y^2)+E(2xy)-E(x+y)^2\)

\(Var(x+y)=E(x^2)+E(y^2)+E(2xy)-[E(x)+E(y)]^2\)

\(Var(x+y)=E(x^2)+E(y^2)+E(2xy)-E(x)^2-E(y)^2-2E(x)E(y)]\)

\(Var(x+y)=[E(x^2)-E(x)^2]+[E(y^2)-E(y)^2]+E(2xy)-2E(x)E(y)\)

\(Var(x+y)=Var(x) +Var(y)+2[E(xy)-E(x)E(y)]\)

We then define:

\(Cov(x,y):=E(xy)-E(x)E(y)\)

Noting that:

\(Cov(x,x)=E(xx)-E(x)E(x)\)

\(Cov(x,x)=Var(x) \)

So:

\(Var(x+y)=Var(x)+Var(y)+2Cov(x,y)\)

\(Var(x+y)=Cov(x,x)+Cov(x,y)+Cov(y,x)+Cov(y,y) \)

\(Cov(x,c)=E(xc)-E(x)E(c)\)

\(Cov(x,c)=cE(x)-cE(x)\)

\(Cov(x,c)=0\)


\lab{Public Key Cryptography}{RSA}
\label{lab:RSA}

\objective{Implement the RSA cryptosystem as an example of public key cryptography.}

A \emph{public key cryptosystem} uses separate keys for encryption and decryption.
If Alice wishes to send Bob a message, she encrypts it using Bob's \emph{public key}, which is available to everyone.
Then Bob decrypts it with his \emph{private key}, which only he knows.
As long as it is difficult to find the private key from the public key, this is a secure system.

Public key cryptosystems are advantageous because for $n$ people to have secure communications with each other, they need exactly $n$ public-private key pairs.
A symmetric cryptosystem would require on the order of $n^2$ keys.
Also, they eliminate the need for secure key sharing algorithms like the Diffie-Hellman key exchange (see Lab \ref{lab:DiffieHellman}).

One of the oldest and most popular public key cryptosystems is RSA cryptography.\\

All material in this lab through the section ``Security of RSA'' is required.
After that, all sections and subsections are independent of each other and may be covered as desired.

\section*{The RSA system}
Suppose Alice wants to receive secret messages using RSA.
To do so, she first needs to generate a public key (for encryption) and a private key (for decryption).
Alice does this by choosing two prime numbers, $p$ and $q$, and setting $n=pq$.
Then she sets $\phi(n) = (p-1)(q-1)$.\footnote{
The function $\phi: \mathbb{Z} \rightarrow \mathbb{N}$ is called the \emph{Euler phi function}. In general $\phi(n)$ is the number of positive integers less than $n$ that are relatively prime to $n$.}
Alice chooses her encryption exponent to be some integer $e$ that is relatively prime to $\phi(n)$.
Then she uses the Extended Euclidean Algorithm to find $d'$ such that $ed' + \phi(n)x' = 1$, and adds or subtracts multiples of $\phi(n)$ until $d = d'+k\phi(n)$ is between 0 and $\phi(n)$.
Alice publishes her public key $(e, n)$ for the world to see and keeps her private key $(d,n)$ a secret.

Now imagine Bob wants to send Alice a message, which he represents as an integer $m<n$ (say, using the A=01, B=02 scheme).
Bob computes
\[
c \equiv m^e \pmod{n}
\]
and sends the ciphertext $c$ to Alice.

To decript the message, Alice computes
\[
m' \equiv c^d \pmod{n}.
\]
At this point, she uses the following theorem, which is easily proved with a combination of ring and group theory.
\begin{theorem}
For any integers $m$ and $n$ such that $n$ does not divide $m$, the following equality holds:
\[
m^{\phi(n)}\equiv 1 \pmod{n}.
\]
\end{theorem}
Then Alice concludes that
\[
m' = c^d \equiv m^{ed} = m^{\phi(n)(ek-x')+1} \equiv m \pmod{n}.
\]
Now Alice can read Bob's message.

As an example, let us encrypt and decrypt the message SECRET=190503180520.
First we define $p$, $q$, $n$, and $\phi(n)$.
\begin{lstlisting}
>>> p = 1000003
>>> q = 608609
>>> n = p*q
>>> phi_n = (p-1)*(q-1)
\end{lstlisting}

Now we choose an encryption exponent $e = 1234567891$ and compute $d = 589492582555$ using the Extended Euclidean Algorithm.
\begin{lstlisting}
>>> e = 1234567891
>>> d = 589492582555
\end{lstlisting}

Finally we are ready to encrypt the message.
Note that $m<n$. If this were not the case, we would need to break up $m$ into shorter pieces.
Also, we force $m$ to be a \li{long} integer so that the exponentiation operation does not overflow.
The function \li{pow(a, b, n)} computes $a^b \pmod{n}$.
\begin{lstlisting}
>>> m = long(190503180520)
>>> c = pow(m, e, n)
\end{lstlisting}

We decript the message by raising $c$ to the $d^{th}$ power.
\begin{lstlisting}
>>> m == pow(c, d, n)
True
\end{lstlisting}

\begin{problem}
Implement the RSA cryptosystem with three functions: one to generate keys, one to encrypt messages, and one to decrypt messages.
\begin{lstlisting}
def keys(p, q, e):
   '''Create a pair of RSA keys from primes `p' and `q' and encryption
   exponent `e'.

   Return [(e, n), (d, n)] where (e, n) is the public key and (d, n) is
   the private key.
   '''

def encrypt(message, e, n):
    '''Encrypt the `message' using the public key (e, n).'''

def decrypt(ciphertext):
    '''Decrypt the `ciphertext' using your private key.'''
\end{lstlisting}
\label{prob:rsa1}
\end{problem}
\subsection*{Security of RSA}
Suppose an enemy Eve wants to read the message that Bob sent to Alice.
Like Bob, she has access to Alice's public key $(e, n)$.
Let's assume she has also intercepted the ciphertext $c$.

One way for Eve to read Bob's message is to directly find $m$ such that $m^e \equiv c \pmod{n}$.
Such a computation is known as taking a \emph{discrete logarithm}.
When $n$ is very large, this computation is essentially impossible.

Another option is for Eve to compute $(d, n)$ and then find $c^d$.
However, computing $d$ means computing $\phi(n)$.
Computing $\phi(n)$ from $n$ directly requires factoring $n$.
When $n$ has hundreds of digits, finding its factors with known algorithms can take years.

\subsection*{Logistical considerations (Optional)}
The cryptosystem you wrote in Problem \ref{prob:rsa1} is not particularly easy to use, since you must convert your message to an integer and back again by hand.
The appendix to this lab contains some functions you can use to fix this problem.
The function \li{string_to_int()} turns any string into an integer (using a mapping more complicated than $A=01, B=02$), and the function \li{int_to_string()} changes it back again.

\begin{lstlisting}
>>> string_to_int('SECRET')
91556947314004
\end{lstlisting}

At this point we have a problem, because the message 91556947314004 is larger than $n=608610825827$.
We can only use RSA to encrypt messages that are smaller than $n.$
In fact, the function \li{string_size()} provided in the appendix tells us the maximum number of characters we can encrypt with this choice of $n$.

\begin{lstlisting}
>>> string_size(608610825827)
4
\end{lstlisting}

The function \li{partition()}, also in the appendix, will break our message into pieces of length 4.
We specify the ``fill value'' \li{x} that will be used to make all pieces the same length.

\begin{lstlisting}
>>> partition('SECRET', 4, 'x')
['SECR', 'ETxx']
\end{lstlisting}
Now we can proceed to encrypt and decrypt the strings \li{'SECR'} and \li{'ETxx'} as before.

\begin{problem}[Optional]
Modify your solution to Problem \ref{prob:rsa1}, making it simpler to use, by writing the following functions.
\begin{lstlisting}
def encrypt(message, e, n):
    '''Encrypt `message' using the public key (e, n).

    INPUTS:
    message - A string.
    (e, n)  - A pair of integers. Should be an RSA public key.

    OUTPUTS:
    Partition `message' into pieces of sufficiently small size, padding if
    necessary. Return a list of the encryptions of the pieces.
    '''

def decrypt(ciphertext):
    '''Decrypt `ciphertext' using your private key.

    INPUTS:
    ciphertext - A list of integers.

    OUTPUTS:
    Decrypt each integer in `cipertext' and return a single string.
    '''
\end{lstlisting}

You can test your cryptosystem on the example in this lab and also on this larger example:
\begin{itemize}
\item message: \emph{Simon sells seashells by the seashore.}
\item $p=83285677$, $q=2848968679$, $e=65593$
\end{itemize}
\label{rsa2}
\end{problem}

\section*{Digital Signatures (Optional)}
Suppose Bob wants Alice to be able to verify that \emph{he} sent her a message, not Eve.
Bob can do this using a \emph{digital signature}.

\subsection*{Method 1}
One way for Bob to digitally sign a message is to apply his \emph{private key} (his ``signature'') to the message and then apply Alice's public key.
When Bob applies his private key to a message, we say he \emph{signs} the message.
When Alice receives this message, she first applies her private key and then Bob's public key.
If the message makes sense, she knows it must be from Bob, since he is the only person with his private key.

Symbolically, let $e_B()$ and $d_B()$ be Bob's encryption and decryption functions (these are inverse functions), and let $e_A()$ and $d_A()$ be the corresponding functions for Alice.
Then Bob sends $e_A(d_B(m))$ to Alice, and Alice computes
\[e_B(d_A(e_A(d_B(m)))) = e_B(d_B(m)) = m.\]

This method requires twice as many RSA encryptions as sending an unsigned message.
Modular exponentiation can be an expensive operation, so if the message is long (and must be broken into several pieces) this is a serious slow-down.

\subsection*{Method 2: hashes}
A more efficient method (which also turns out to be more cryptographically secure) is to ``hash the message, and then sign the hash.''
A hash is a function that takes a string of any size to a fixed number of bytes.
A cryptographically secure hash satisfies two properties:
\begin{itemize}
\item If $m_1$ is different from $m_2$ then hash($m1$) is (almost certainly) different from hash($m2$).
\item Given hash($m$), it is essentially impossible to find $m$.
\end{itemize}
For example, the MD5 hash will map any string to 16 bytes (a character like \li{'a'} is one byte of data).\footnote{
The hash MD5 is not cryptographically secure, but is often used in non-cryptographic applications.
Because of its small digest size we use it as an example here.}
Signing the hash will be faster than signing the original message, if the original message is more than 16 characters.

In Python, hashes are implemented in the \li{hashlib} library.
Here we use MD5 to hash the string \li{'SECRET'}.
\begin{lstlisting}
>>> import hashlib
>>> hash = hashlib.md5('SECRET')
\end{lstlisting}
We view the hash with the \li{digest()} method.
\begin{lstlisting}
>>> hash.digest()
'D\xc7\xbeH"n\xba\xd5\xdc\xa8!ft\xca\xd6+'
\end{lstlisting}
The output is 16 bytes, some represented as characters (e.g. \li{'D'}) and some as two hexidecimal digits (e.g. \li{'\\xc7'}).
Here we separate the digest into its bytes.
\begin{lstlisting}
>>> [byte for byte in hash.digest()]
['D', '\xc7', '\xbe', 'H', '"', 'n', '\xba', '\xd5', '\xdc', '\xa8', '!', 'f', 't', '\xca', '\xd6', '+']
\end{lstlisting}

So when Bob sends Alice a message, he hashes the message and then signs the hash (applies his private key).
Then he encrypts the message with Alice's public key.
When Alice receives the message and signature, she decrypts the message with her private key.
After reading the message, she hashes it and compares the result to Bob's public key applied to the signature.
If they agree, then she knows the message came from Bob.

\begin{problem}
Rewrite your RSA cryptosystem to include digital signatures.
That is, write the following functions, using the same hash in each.
\begin{lstlisting}
def encrypt(message, e, n):
    '''Encrypt `message' using the public key (e, n) and sign it.

    INPUTS:
    message - A string.
    (e, n)  - A pair of integers. Should be an RSA public key.

    OUTPUTS:
    ciphertext - A list of the encrypted pieces of `message.'
    signatures - The hash of `message', signed with your private key.
    '''

def decrypt(ciphertext, e, n):
    '''Decrypt `ciphertext' using your private key and verify the
    signature matches the public key (e, n).

    INPUTS:
    ciphertext - A list of integers.
    (e, n)     - A public key.

    OUTPUTS:
    message  - The decrypted `ciphertext' as a single string.
    verified - A Boolean value indicating whether the digital signature
               was verified.
    '''
\end{lstlisting}
\end{problem}

\section*{Making RSA feasible (Optional)}

For RSA to be a reasonably fast cryptosystem, we need a fast algorithm for modular exponentiation.
This algorithm, known as \emph{binary exponentiation}, was discussed in Lab \ref{lab:DiffieHellman}.

In addition, the security of RSA depends on the use of large prime numbers for $p$ and $q$.
For RSA to be a practical cryptosystem, we need a fast way to find such numbers.

\subsection*{Finding large primes}
It is very slow to check that a large number is prime by finding its factors.
In fact, it is the difficulty of factoring that makes RSA secure.

However, there exist fast algorithms that determine when a number is ``probably prime.''
One such algorithm comes from a special case of Theorem \ref{thm:fermat} known as \emph{Fermat's Little Theorem}:
\[
a^{p-1} \equiv 1 \pmod{p}
\]
for all primes $p$ and integers $a$ that are not divisible by $p$.
Therefore, to check if a fixed number $p$ is likely a prime, we can compute $a^{p-1} \pmod{p}$ for several values of $a$.
If we ever get something different than 1, then we know that $p$ is composite.
The value of $a$ that proved $p$ was composite is called a \emph{witness number}.
In fact, the converse is true: if $a^{p-1} \equiv 1 \pmod{p}$ for every $a<p$ then $p$ is prime.

With a few exceptions, if $p$ is composite then more than half the integers in $[2, p-1]$ are witness numbers.
Thus, if we run the above test on $p$ just a few times and don't find a witness number, there is a high probability that $p$ is prime.
This test is called Fermat's test for primality.

In practice, a probabilistic algorithm like Fermat's test is used to identify numbers that have a high chance of being prime.
Then, deterministic algorithms are used to verify primality of a candidate.

\begin{problem}
Write the following function, implementing Fermat's test for primality.
\begin{lstlisting}
def is_prime(n):
    '''Use Fermat's test for primality to see if `n' is probably prime.

    INPUTS:
    n - An integer.

    Run Fermat's test on at most 5 numbers randomly chosen from [2, n-1].
    If a witness number is found, return the number of tries it took to
    find the witness number. If no witness number is found after 5 tries,
    return 0.
    '''
\end{lstlisting}
For most composite values of \li{n}, if you call \li{is_prime(n)} one hundred times, you should expect to get 0 at most five times.

However, some composite numbers do not follow this rule.
How many times do you have to call \li{is_prime()} on 340561 to get an answer of 0?
\label{prob:prime_confidence}
\end{problem}

\section*{PyCrypto: RSA in Python (Optional)}
A professional implementation of RSA is found in the PyCrypto module.

% TODO: edit this section. The PyCrypto documentation is extremely poor and I can't figure out how to use it.
\begin{comment}
RSA encryption in Python can be accomplished very easily with a library called PyCrypto.
This library contains many secure hash functions, random number generators, and encryption classes.
Many programs use PyCrypto for their security needs.
\begin{warn}
Make certain that you are using the latest version of PyCrypto.
Security software is updated often to fix security vulnerablilities and bugs.
The current version of PyCrypto at the time of writing is version 2.6.1.
\end{warn}

The RSA module in PyCrypto is located in the PublicKey module.
Let's try a small example using PyCrypto.
The library allows us to explicity construct a key, or generate a key automatically.
\begin{lstlisting}
>>> from Crypto.PublicKey import RSA
>>> from Crypto import Random
>>> keypair = RSA.generate(2048) #generate a 2048-bit RSA key
>>> publickey = keypair.publickey()
>>> share_this = publickey.exportkey()
\end{lstlisting}

The RSA encryption and decryption methods on these keys are textbook approaches.
However, to increase security, we will want to pad the messages so every message encrypted with a particular will become exactly as large (in bits) as the key itself.
A commonly used padding algorithm is implemented in PyCrypto in the \li{Crypto.Cipher.PKCS1_OAEP} module.
\begin{lstlisting}
>>> from Crypto.Cipher import PKCS1_OAEP as oaep

# generate a new key from the original RSA key.
# This key can encrypt and decrypt
>>> paddedkey = oaep.new(keypair)
>>> encrypted = paddedkey.encrypt('hello world')
>>> paddedkey.decrypt(encrypted)
'hello world'
\end{lstlisting}

To sign the message we use PyCrypto's \li{Crypto.Signature.PKCS1_PSS} module.
We first need to hash the message using a cryptographically secure hash, such as SHA.
Just as only private keys can decrypt, only private keys can sign.
Public keys can decrypt and verify.
\begin{lstlisting}
>>> from Crypto.Signautre import PKCS1_PSS as pss
>>> from Crypto.Hash import SHA
>>> sigkey = pss.new(keypair)
>>> mhash = SHA.new("Hello World")
>>> sigkey.sign
\end{lstlisting}

\begin{problem}[Group Project]
Split into several groups of at least three individuals.
Each group member should generate a private and public keypair using PyCrypto.
Share your public keys with everyone in the group.

You might find it useful to transmit your messages as strings.
Here are two functions that will read and write specially formatted strings for your messages.
\begin{lstlisting}
def print_msg(m_encrypted):
    #print the encrypted message as a specially formatted string
    out = ("----BEGIN MESSAGE----",
           m_encrypted[0],
           "----END MESSAGE----",
           "----BEGIN SIGNATURE----",
           m_encrypted[1],
           "----END SIGNATURE----")
    return '\n'.join(out)

def read_msg(m_string):
    d1 = m_string.find('----BEGIN MESSAGE----') + 21
    d2 = m_string.find('----END MESSAGE----', d1)

    d3 = m_string.find('----BEGIN SIGNATURE----', d2) + 23
    d4 = m_string.find('----END SIGNATURE----', d3)
    message  = m_string[d1:d2]
    sign = m_string[d3:d4]
    return message.strip(), sign.strip()
\end{lstlisting}

\begin{enumerate}
\item Encrypt and sign a message for someone in the group and send it to the entire group.
Attach the sender's name to the encrypted message (a claim that the message came from a particular sender).
In this exercise we will verify that the actual origin of the message is indeed the claimed origin.
Each group member should do the following:
\begin{enumerate}
\item Decrypt and verify the message addressed for you.
\item Attempt to decrypt and verify message addressed for someone else in the group.
\item Report the message and verified signature.
\end{enumerate}

\item Encrypt and sign a different message for someone in the group and send it, anonymously, to the entire group.
For example, if Bob encrypts a message and sends it to Alice, he should claim the message came from another individual.
Each group member should do the following:
\begin{enumerate}
\item Decrypt and verify the message address to you.
\item Report the message and verified signature.
\end{enumerate}
\end{enumerate}
\end{problem}
\end{comment}

\section*{Appendix: Helper Code}
Here are some functions that you can use to make your implementation of RSA more user-friendly (see the ``Logistical considerations'' section).
Note that you must import \li{izip_longest} from \li{itertools} before using some of these functions.

 \begin{lstlisting}
 from itertools import izip_longest

 def partition(iterable, n, fillvalue=None):
    '''Partition data into blocks of length `n', padding with `fillvalue' if needed. Return a list of the partitions.

    EXAMPLE:
    >>> partition('ABCDEFG', 3, 'x')
    ['ABC', 'DEF', 'Gxx']
    '''
    args = [iter(iterable)] * n
    pieces = izip_longest(fillvalue=fillvalue, *args)
    return [''.join(block) for block in pieces]

def string_size(n):
    '''Return the maximum number of characters that can be encoded with the public key (e, n).

    In other words, find the largest integer L, such that if `string' has at most L characters, then a2i(`string') will be less than `n.'
    '''
    L=0
    max_int = 0
    while max_int < n:
        max_int += sum([2**i for i in range(8*L, 8*L+8)])
        L += 1
    return L-1

def string_to_int(msg):
    '''Convert the string `msg' to an integer.
    '''
    # bytearray will give us the ASCII values for each character
    if not isinstance(msg, bytearray):
        msg = bytearray(msg)
    binmsg = []
    # convert each character to binary
    for c in msg:
        binmsg.append(bin(c)[2:].zfill(8))
    return int(''.join(binmsg), 2)

def int_to_string(msg):
    '''Convert the integer `msg' to a string.

    This function is the inverse of string_to_int().
    '''
    # convert to binary first
    binmsg = bin(msg)[2:]
    # we need to pad the message so length is divisible by 8
    binmsg = "0"*(8-(len(binmsg)%8)) + binmsg
    msg = bytearray()
    for block in partition(binmsg, 8):
        # convert block of 8 bits back to ASCII
        msg.append(int(block, 2))
    return str(msg)
 \end{lstlisting}


In this section we will show the finite difference equations needed to solve
\begin{align*}
\frac{\partial u}{\partial t}=\frac{1}{Re}\left(\frac{\partial^2 u}{\partial x^2}+\frac{\partial^2 u}{\partial y^2}\right),
\end{align*}
using ADI. Let $d_1=\frac{\Delta t}{2Re\Delta x^2}$ and $d_2=\frac{\Delta t}{2Re\Delta y^2}$.
\subsection{Step 1. Implicit in x-direction}
The differential equation discretized implicitly in the x-direction yields
\begin{align*}
\frac{u_{i+1/2,j}^{n+1/2}-u_{i+1/2,j}^n}{\Delta t/2}=\frac{1}{Re\Delta x^2}\left(u_{i+3/2,j}^{n+1/2}-2u_{i+1/2,j}^{n+1/2}+u_{i-1/2,j}^{n+1/2}\right)+\frac{1}{Re\Delta y^2}\left(u_{i+1/2,j+1}^{n}-2u_{i+1/2,j}^{n}+u_{i+1/2,j-1}^{n}\right).
\end{align*}
Using the definitions of $d_1$, $d_2$ and reorganizing the terms we obtain
\begin{align*}
-d_1u_{i-1/2,j}^{n+1/2}-(1+2d_1)u_{i+1/2,j}^{n+1/2}-d_1u_{i+3/2,j}^{n+1/2}=d_2u_{i+1/2,j-1}^n+(1-2d_2)u_{i+1/2,j}^n+d_2u_{i+1/2,j+1}^n,
\end{align*}
which gives us
\begin{align*}
a_{i+1/2,j}=-d_1,~~~~~~b_{i+1/2,j}=1+2d_1,~~~~~~c_{i+1/2,j}=-d_1,
\end{align*}
and
\begin{align*}
d_{i+1/2,j}=d_2u_{i+1/2,j-1}^n+(1-2d_2)u_{i+1/2,j}^n+d_2u_{i+1/2,j+1}^n.
\end{align*}
\subsubsection{Boundary Conditions}
To account for the left boundary we have imposed $a_{3/2,j}=0$ and
\begin{align*}
d_{3/2,j}=d_2u_{3/2,j-1}^n+(1-2d_2)u_{3/2,j}^n+d_2u_{3/2,j+1}^n+d_1u_{3/2,j},
\end{align*}
where $u_{3/2,j}$ is given by the boundary conditions

\[ u_{3/2,j}=\begin{cases}
      U_{in1}(y_j) & x\leq 0 \\
      0 & 100\leq x
   \end{cases}
\]

\subsection{Step 2. Implicit in y-direction}


In this section, we will indicate several important logical equivalences between the logical operators that we have introduced thus far.

Recall in the previous section that boolean expressions can be viewed as functions of each variable, so we deduced that  two expressions are logically equivalent if they have the same truth table. In fact, in some cases we can compare boolean expressions with different number of variables. For example, the logical expression $(p \land q) \lor (r \lor (\shortsim r))$ is logically equivalent to the logical expression $p \land q$. We can naturally view $p \land q$ as a function of the boolean variables $p$ and $q$, but we can also view this as a function of $p$, $q$, and $r$, where the result is independent of the value of $r$. So in this way we can compare logical expressions with varying or different boolean variables.

Now we will show a basic logical equivalence called the \textbf{contrapositive law}. The contrapositive law asserts for any logical variables $p$ and $q$ we have the equivalence
\[p \implies q \equiv (\shortsim p) \implies (\shortsim q).\]
To demonstrate that this equivalence is true it suffices to show that each entry in their respective truth tables are equal, as demonstrated below:

\begin{table}[h]
\centering
\begin{tabular}{|l|l|l|l|l|l|l|l|}
\hline
$p$ & $q$ & $p \implies q$ & & $p$ & $q$ & $(\shortsim q) \implies (\shortsim p)$ \\ \hline
$T$ & $T$ & $T$ & & $T$ & $T$ & $T$ \\ \hline
$T$ & $F$ & $T$ & & $T$ & $F$ & $T$ \\ \hline
$T$ & $F$ & $F$ & & $T$ & $F$ & $F$ \\ \hline
$F$ & $T$ & $T$ & & $F$ & $T$ & $T$ \\ \hline
\end{tabular}
\end{table}

The same approach is used when demonstrating each logical equivalence in the table below.

\begin{table}[h]
\centering
\begin{tabular}{|l|l|}
\hline
Logical Equivalence & Description \\ \hline
$p \land q \equiv q \land p$, $p \lor q \equiv q \lor p$ & Commutativity of $\land$ and $\lor$\\ \hline
$(p \land q) \land r \equiv p \land (q \land r)$ & Associativity of $\land$\\ \hline
$(p \lor q) \lor r \equiv p \lor (q \lor r)$ & Associativity of $\lor$ \\ \hline
$(p \lor q) \land r \equiv (p \land r) \lor (q \land r) $ & Distributivity of $\land$ over $\lor$ \\ \hline
$(p \land q) \lor r \equiv (p \lor r) \land (q \lor r) $ & Distributivity of $\lor$ over $\land$ \\ \hline
$p \land T \equiv p$ & $\land$ identity \\ \hline
$p \lor F \equiv p$ & $\lor$ identity \\ \hline
$\shortsim(\shortsim p) \equiv p$ & Double Negation \\ \hline
$p \land p \equiv p \lor p \equiv p$ & Idempotence of $\land$ and $\lor$ \\ \hline
$p \lor (p \land q) \equiv p \land (p \lor q) \equiv p$ & Absorption Property \\ \hline
$a \implies b \equiv (\shortsim b) \implies (\shortsim a)$ & Contrapositive Property \\ \hline
$a \implies b \equiv (\shortsim a) \lor b$ & Implication as Disjunction \\ \hline
$a \odot b \equiv (a \implies b) \land (b \implies a)$ & XNOR as Conjunction \\ \hline
$a \land (\shortsim a) \equiv F$, $b \lor (\shortsim b) \equiv T$ & Simplification Rules \\ \hline
\end{tabular}
\end{table}

Using logical equivalences like the ones in the table above, it is now possible to show that certain expressions are logically equivalent without explicitly constructing their truth tables, as the examples below show.

\begin{example}
In this example we will show that the expression
\[\left((q \lor z) \land (k \lor (\shortsim k))\right) \implies \left(((\shortsim q) \land (\shortsim z)) \lor p\right)\]
is logically equivalent to the simpler expression
\[(\shortsim(q \lor z)) \lor p.\]

To do this we will simplify the first expression using rules stated in the table.

\begin{align*}
& \left((q \lor z) \land (k \lor (\shortsim k))\right) \implies \left(((\shortsim q) \land (\shortsim z)) \lor p\right) \\
\equiv& \left((q \lor z) \land T\right) \implies \left(((\shortsim q) \land (\shortsim z)) \lor p\right) \\
\equiv& (q \lor z) \implies \left(((\shortsim q) \land (\shortsim z)) \lor p\right) \\
\equiv& (q \lor z) \implies \left(\shortsim(q \lor z) \lor p\right) \\
\equiv& (\shortsim(q \lor z) \lor \shortsim(q \lor z)) \lor p \\
\equiv& (\shortsim(q \lor z)) \lor p
\end{align*}

as desired.
\end{example}

\begin{example}
In this example we will simplify the expression
\[(k \land l) \lor ((k \land m) \land k \land (l \lor (\shortsim m))).\]

But not now because I am lazy.
\end{example}

In general, the rules above (and rules that can be derived from them) form the basis of a system where one can do calculations with logical variables. Now we will discuss some of these properties individually and make more comments about them.

\subsection{Commutativity and Associativity}

In general, a binary operator $\otimes$ is said to be commutative if $p \otimes q \equiv q \otimes p$, and associative if $(p \otimes q) \otimes r \equiv p \otimes (q \otimes r)$. As noted in the table above, the operators $\lor$ and $\land$ are associative and commutative.

Commutativity and Associativity are rather strong properties to have for any binary operator. This is because not all operators are associative or commutative. For example, the implies operator is not commutative, that is,
\[p \implies q \not\equiv q \implies p,\]
and the NAND operator is not associative, that is
\[p \uparrow (q \uparrow r) \not\equiv p \uparrow (q \uparrow r).\]
One can see this by constructing the truth table for each expression respectively.

\subsection{The Absorption Property}

The proper way to interpret the absorption properties in the table above are that their values are independent of the variable $q$ when evaluated. To see this, consider the expression $p \lor (p \land q)$. Evaluating $q$ as $F$ we get
\[p \lor (p \land F) \equiv p \lor F \equiv p\]
by other properties in the table. Similarly, evaluating $q$ as $T$ we get
\[p \lor (p \land T) \equiv p \lor p \equiv p.\]

\subsection{Implication as Disjunction}

The implication as disjunction property is the first example of the important idea of creating an expression given a truth table where all the variable assignments evaluate to true except for one. In this case, the implies operator evaluates $T$ unless $p \equiv T$ and $q \equiv F$. In this case, we can create an or expression $(\shortsim p) \lor q$ which we see has the same truth table as $p \implies q$. This idea of creating such an indicator function given any truth table later will be important when we study the conjunctive and disjunctive normal forms in a following section.

\subsection{Exercises}
\input{Ch1/op_cons_ex}


%
% Chapter 3.9
%

\section*{3.9 Antiderivatives}

A function \(F\) is called an \textbf{antiderivative} of \(f\) on an interval \(I\) if \(F'(x)=f(x)\) for all \(x\) in \(I\).

\subsubsection*{Theorem}

If \(F\) is an antiderivative of \(f\) on an interval \(I\), then the most general antiderivative of \(f\) on \(I\) is
$$ F(x) + C $$
where \(C\) is an arbitrary constant.

\subsection*{General Antiderivative Formulae}

$$ \text{The antiderivative of } \quad cf(x) \quad \text{ is } \quad cF(x) + C. $$
$$ \text{The antiderivative of } \quad f(x)+g(x) \quad \text{ is } \quad F(x) + G(x) + C. $$
$$ \text{The antiderivative of } \quad x^{n} (n \neq -1) \quad \text{ is } \quad \frac{x^{n+1}}{n+1} + C. $$
$$ \text{The antiderivative of } \quad \cos(x) \quad \text{ is } \quad \sin(x) + C. $$
$$ \text{The antiderivative of } \quad \sin(x) \quad \text{ is } \quad -\cos(x) + C. $$
$$ \text{The antiderivative of } \quad \sec^2(x) \quad \text{ is } \quad \tan(x) + C. $$
$$ \text{The antiderivative of } \quad \sec(x)\tan(x) \quad \text{ is } \quad \sec(x) + C. $$
\\
An equation that involves the derivatives of a function is called a \textbf{differential equation}.


\subsection{Definition and Notation}

Most roughly speaking, a set is a collection of objects. These objects can be numbers, people, or even other sets themselves. In general this collection need not be homogeneous (so for example, we can have a set which contains both a number and a person). For the purposes of this book, we will not even try to discuss a formal logical definition of a set (this gets very complicated). So for the rest of this book we can assume that a set is a collection of objects without any issues whatsoever.

In general, when we are describing sets as a collection of specific objects, we tend to use curly braces $\{\}$ to enclose these objects while delimiting them by commas.

\begin{example}
If we wanted to describe the set containing the numbers $1$, $2$, and $3$, we would do so as $\{1, 2, 3\}$. The set $\{1, \{1, 2\}\}$ is the set which contains the number $1$, and the set $\{1, 2\}$ containing $1$ and $2$.

The set $\{1, 1\}$ is the set containing the numbers $1$, \ldots and $1$? Since $1$ and $1$ are in fact the same this set is the same as $\{1\}$. We will make this example more formally clear later in the chapter.
\end{example}

In general describing sets like this gets awkward when the number of elements that the set contains gets very large. For example, it is very awkward to write out
\[\{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15\}\]
to describe the set of positive integers between $1$ and $15$ inclusive. There are a couple of things we can do. For most purposes if we instead wrote this set as
\[\{1, 2, 3, \dots, 14, 15\}\]
where most people would understand what the set was. Another thing we can do is use a variable, such as $A$, to denote the set of positive integers between $1$ and $15$ inclusive. The best convention is to combine these two notions. We can define
\[A = \{1, 2, 3, \dots, 14, 15\}.\]
In this formulation $A$ is defined as the set of positive integers between $1$ and $15$ inclusive. Once we have defined $A$ in this way we can freely use just the character ``$A$'' instead of $\{1, 2, 3, \dots, 14, 15\}$, which is a good way to write concisely.

In general, we can refer to any abstract set using a variable such as $A$, $B$, etc. (In general most people will use capital letters to refer to sets).

\begin{example}
The set $\mathbb{N}$, known as the set of \textbf{natural numbers}, denotes the set of positive integers. That is,
\[\mathbb{N} = \{1, 2, 3, 4, \dots\}.\]
The set $\mathbb{Z}$ (german for \textit{Zahlen}) denotes the set of all integers. That is,
\[\mathbb{Z} = \{\dots, -2, 1, 0, 1, 2, \dots\}.\]
In some texts people include $0$ in the natural numbers. This is entirely a matter of convention and not for any very deep reason. Sometimes it is convenient to include $0$.

The empty set, denoted $\varnothing$ is the set which does not contain any elements.
\end{example}

\subsection{Set Inclusion and Set Equality}

The first thing we describe in this section is notation to describe whether an object (usually denoted by a lowercase letter) is in a certain set. We use the $\in$ notation in order to do so. That is, we say that
\[x \in A\]
if the object $x$ is in the set $A$. Alternatively if an object $x$ is not in $A$ we write $x \not\in A$ instead.

\begin{example}
Let $A = \{1, 2, 3, \dots, 14, 15\}$. Then $6 \in A$.

The empty set $\varnothing$ has the property that $x \not\in A$ for any conceivable object $x$.
\end{example}

The notion of set containment, which is an inquiry about whether any object is contained in a set, leads naturally to a general notion of inquiring whether some set is contained inside another. The definition for this is pretty natural. Roughly speaking, a set $A$ is contained in a set $B$ only if all the objects contained in $A$ are also contained in $B$. We make this notion more formal now.

\begin{definition}
Suppose $A$ and $B$ are sets. Then we say that $A \subset B$ if we have for every $x \in A$ we have $x \in B$ also.
\end{definition}

\begin{example}
We have
\[\{1, 2\} \subset \{1, 2, 3\}.\]
\end{example}

A closely related concept to set inclusion is set equality. Below we define set equality. Roughly two sets are equal if they contain the same elements. We will define set equality then using the subset definition.
\begin{definition}
Suppose $A$ and $B$ are sets. Then $A = B$ if $A \subset B$ and $B \subset A$.
\end{definition}

This definition roughly encodes the notion of set equality. Since in the definition of subset, $A \subset B$ if all the elements of $A$ were in $B$. So two sets $A$ and $B$ are equal if all the elements of $A$ are in $B$, and all the elements of $B$ are in $A$. This raises some subtle, but not particularly deep observations.

\begin{example}
We claim that
\[\{1, 2, 3, 3\} = \{1, 2, 3\}.\]

This can be easily verified using the definition of set equality. However it is true that both sets do not ``look'' the same.
\end{example}

This subtlety with multiple elements in a set shows that we only need to consider sets where each element of the set is unique in that set, since such a set will always be equal to one with the same elements but maybe some of them duplicated. %%probably a terrible explanation.

\subsection{Common Set Operations}

In this section we describe common set operations and notations used to manipulate sets and create new sets. First we will explain why this is useful. In practice, when one is using sets, in general one considers sets with certain properties. That is one would like to consider perhaps certain objects with a certain property. In practice one would define these sets using \textbf{set builder notation}. The idea of defining such a set $S$ is encapsulated in the notation below.

\[S = \{\text{all the objects $x$ of some type} \colon \text{$x$ has some special property}\}.\]
Let's break down this notation further. With this notation, it is accepted that $S$ is defined to be the set of all objects $x$ such that $x$ has a special property. For example, the set
\[S = \{x \in \mathbb{Z} \colon |x| \leq 5\}\]
is the set of integers with absolute value less than or equal to $5$. In other words,
\[S = \{-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5\}.\]
In general set builder notation is very useful to create sets of objects with certain properties.

Not only do we want to describe objects with a certain property, but much of the time we would like to describe objects with more than one property or objects with at least one of the following few properties. These concepts are encapsulated by set intersection and set union, respectively.

\subsubsection{Set Intersection}

Suppose $E$ is the set of even numbers and $T$ is the set of multiples of $3$. Then if a number $n$ is divisible by $6$, then it is both an even number and a multiple of $3$. So $n$ is in $E$ and $T$ at the same time. Using notation already presented, one might say that $n \in E$ and $n \in T$. However, this is not the most concise way to write this. In order to express that an object is in two sets and the same time we introduce the notion of \textit{set intersection}.

\begin{definition}
Let $A$ and $B$ be sets. We define $A \cap B$ (read as ``$A$ intersect $B$'') to be the objects $n$ which are contained in both $A$ and $B$. A little more formally,
\[A \cap B = \{n \colon (n \in A) \land (n \in B)\}.\]
\end{definition}

In the example described above, the most concise way to write that $n$ is both even and a multiple of three would be $n \in E \cap T$.

\begin{definition}
Let $A$ and $B$ be sets. We say that $A$ and $B$ are \textbf{disjoint} if $A \cap B = \varnothing$. That is, $A$ and $B$ have no elements in common.
\end{definition}

\subsubsection{Set Union}
%%the first example I thought up was that of boys and girls in a schoolyard. But this might be politically incorrect in more than one way.
Similarly to how we would like to describe objects with multiple properties at the same time, in many cases it is useful to describe objects which have one of many properties, but not necessarily all of them at the same time.
Consider, for example, the set of all candy $C$ that some person has acquired one Halloween night. Any piece of candy satisfies the following condition. Either it contains chocolate or it does not. These are clearly mutually exclusive possibilities. Let $H$ denote the set of candies in the Halloween loot which contain chocolate, and let $J$ denote the set of candies in the Halloween loot that do not contain chocolate. Then we may observe the following.
\begin{itemize}
	\item Every candy $c$ in the Halloween loot is either in $H$ or in $J$, since a piece of candy in the loot either does or does not contain chocolate.
	\item $H \cap J = \varnothing$, because a piece of candy cannot both contain and not contain chocolate at the same time\footnote{Here we are assuming something known as the \textbf{law of the excluded middle}.}
\end{itemize}

As with set intersection, we would like some notation that would indicate that $c$, a piece of candy, is either in $H$ or in $J$. This is done using the set union notation, as defined below.
\begin{definition}
Let $A$ and $B$ be sets. We define $A \cup B$ (read as ``$A$ union $B$'') to be the objects $n$ which are contained in at least one of $A$ or $B$. A little more formally,
\[A \cup B = \{n \colon (n \in A) \lor (n \in B)\}.\]
\end{definition}
For instance, in the example above, we could write $c \in H \cup J$ to indicate that each piece of candy is in either $H$ or in $J$. This is not the most general use of set union because as we observed, the sets $H$ and $J$ were disjoint. As a simple example, one can verify that
\[\{1, 2, 3\} \cup \{2, 3, 4\} = \{1, 2, 3, 4\}.\]

\begin{definition}
Suppose that $S$ is a set and $A$ and $B$ are subsets of $S$. We say that $A$ and $B$ cover $S$ if
\[A \cup B = S.\]
\end{definition}

\subsubsection{Union and Intersection of More Than Two Sets}

In the above two parts we have defined set intersection and set union for two sets $A$ and $B$. As stated before, these notations are useful for describing objects that have many properties or at least one out of many given properties respectively. However, with our definitions we can only do this when ``many'' equals $2$. So now we define union and intersection for more than two sets. First, we define set union and intersection for arbitrarily many finite sets, and then for an arbitrary index set (which we will define later).

\begin{definition}
Suppose $n \geq 2$, and $A_1, A_2, \dots, A_n$ are sets. We define their intersection and union to be
\[A_1 \cap A_2 \cap \cdots \cap A_n = \{x \colon (x \in A_1) \land (x \in A_2) \land \cdots \land (x \in A_n)\}\]
and
\[A_1 \cup A_2 \cup \cdots \cup A_n = \{x \colon (x \in A_1) \lor (x \in A_2) \lor \cdots \lor (x \in A_n)\}\]

We can also write these intersections and unions in big union and intersection notation, which looks like this:
\[\bigcup_{i = 1}^nA_i = A_1 \cup A_2 \cup \cdots \cup A_n\]
and
\[\bigcap_{i = 1}^nA_i = A_1 \cap A_2 \cap \cdots \cap A_n.\]
\end{definition}

One might observe that in the case of chapter 1\footnote{this is temporary to indicate that i have not mentioned this issue in chapter 1 at all} that the definition of a union (resp. intersection) of more than two sets involves a logical formula that is not fully well defined namely a large connection of logical ands (resp. logical ors). As we have stated then this will not be a problem due to the associativity of the logical operators $\land$ and $\lor$. We will resolve this problem fully in Chapter 4. %note please label the chapters later.

Now we discuss the union of sets over an arbitrary index. Let $I$ be a set, possibly infinite. Suppose we have a collection of sets $A_i$ for each $i \in I$. In many cases we would like to discuss the union (or perhaps intersection) of these sets, denoted by
\[\bigcup_{i \in I}A_i, \bigcap_{i \in I}A_i\]

We will define these unions and intersections similarly to how we did it for finitely many sets.
\begin{definition}
Let $I$ be any set, and suppose that $A_i$ are sets defined for each $i \in I$. We will define the union and intersection of these sets as follows: We define
\[\bigcup_{i \in I}A_i = \{x \colon \text{$x \in A_i$ for some $i \in I$}\}\]
and
\[\bigcap_{i \in I}A_i = \{x \colon \text{$x \in A_i$ for all $i \in I$}\}.\]
\end{definition}

\begin{example}
Consider the set $I = \mathbb{N}$. We will define the set $A_n$ for each $n \in \mathbb{N}$ to be the set of positive multiples of $n$. That is,
\[A_n = \{n, 2n, 3n, 4n, \dots\}.\]
First we will determine
\[\bigcup_{n \in I}A_n.\] We claim this is just $\mathbb{N}$. Indeed, observe that since $A_n \subset \mathbb{N}$ for all $n \in I$, their union is a subset of $\mathbb{N}$ as well. For the other direction, we only need to observe that for each $n \in \mathbb{N}$, $n \in A_n$. So clearly by the definition of subset we have the other inclusion. %maybe explain more.

Now we will determine
\[\bigcap_{n \in I}A_n.\] We claim that this set is empty. Indeed, again we observe since $A_n$ are subsets of $\mathbb{N}$ we observe that their intersection is as well. Now we just need to show that no element of $\mathbb{N}$ is in the intersection (hence the intersection is empty). To see this, we only need to observe that any $n \in \mathbb{N}$ is not contained in $A_{n + 1}$. Hence no $k \in \mathbb{N}$ can be contained in all of the $A_n$ at the same time (which is the requirement for $k$ to be in the intersection.
\end{example}

As a remark, the indexing set $I = \mathbb{N}$ is so common that many people usually use the following notation for a union indexed by the natural numbers. In this case people usually write
\[\bigcup_{i = 1}^\infty A_i\] instead of
\[\bigcup_{i \in \mathbb{N}}A_i.\]

\begin{example}
Consider again the set $I = \mathbb{N}$. We will define a collection of sets $A_n$ for $n \in I$ which have the following properties:
\begin{itemize}
	\item \[\bigcup_{i = 1}^{\infty}A_n = \mathbb{N}\]
	\item $A_i \cap A_j = \varnothing$ when $i \neq j$,
	\item $A_i$ are infinite sets for each $I$.
\end{itemize}

I will work out this example later because I am lazy.
\end{example}


\subsection{Expected value}
\subsubsection{Definition}

For a random variable (or vector of random variables), \(x\), we define the expected value of \(f(x)\) as :

\(E[f(x)]:=\sum f(x_i) P(x_i)\)

The expected value of random variable \(x\) is therefore this where \(f(x)=x\).

\(E(x)=\sum_i x_i P(x_i)\)

\subsubsection{Linearity of expectation}

We can show that \(E(x+y)=E(x)+E(y)\):

\(E[x+y]=\sum_i \sum_j (x_i+y_j) P(x_i \land y_j)\)

\(E[x+y]=\sum_i \sum_j x_i [P(x_i \land y_j)]+\sum_i \sum_j [y_j P(x_i \land y_j)]\)

\(E[x+y]=\sum_i x_i \sum_j [P(x_i \land y_j)]+\sum_j y_j \sum_i [P(x_i \land y_j)]\)

\(E[x+y]=\sum_i x_i P(x_i)+\sum_j y_j P(y_j)\)

\(E[x+y]=E[x]+E[y]\)

\subsubsection{Expectations of multiples}

Expectations

\(E(cx)=\sum_i cx P(x_i)\)

\(E(cx)=c\sum_i x P(x_i)\)

\(E(cx)=cE(x)\)

\subsubsection{Expectations of constants}

\(E(c)=\sum_i c_i P(c_i)\)

\(E(c)= cP(c)\)

\(E(c)= c\)

\subsubsection{Conditional expectation}

If \(Y\) is a variable we are interested in understanding, and \(X\) is a vector of other variables, we can create a model for \(Y\) given \(X\).

This is the conditional expectation.

\(E[Y|X]\)

\(E[P(Y|X)Y]\)

In the continuous case this is

\(E(Y|X)=\int_{-\infty }^{\infty }yP(y|X)dy\)

We can then identify an error vector.

\(\epsilon :=Y-E(Y|X)\)

So:

\(Y=E(Y|X)+\epsilon \)

Here \(Y\) is called the dependent variable, and \(X\) is called the dependent variable.

\subsubsection{Iterated expectation}

\(E[E[Y]]=E[Y]\)

\(E[E[Y|X]=E[Y]\)


% Random SVD
\subsection{Randomized SVD}
\label{sec:rsvd}

Randomized SVD is a method to very efficient and effective way to perform approximate singular value decompositions using random matrix theory. The premise is a very large data matrix $\mathbf{X} \in \mathbb{R}^{n \times m}$ that has low \textit{intrinsic rank} $r$, so that $\mathbf{X} \approx \mathbf{U_r \Sigma_r V_r}^T$ will be a good approximation.

\subsubsection{Step 1: Sample Column Space of $\mathbf{X}$ with $\mathbf{P}$}
Given a random projection matrix $\mathbf{P}\in\mathbb{R}^{m \times r}$, the matrix:

\begin{equation}
\mathbf{Z} = \mathbf{XP}
\end{equation}

will, because of the properties of random matrices, likely have the same rank $r$  dominant column space as $\mathbf{X}$. SVD is usually calculated via the QR-decomposition of a matrix. Performing the QR decomposition of $\mathbf{Z}$ is much less laborious, because $\mathbf{Z}\in\mathbb{R}^{m \times r}$ is much smaller than $\mathbf{X}$. We obtain:

\begin{equation}
\mathbf{X} = \mathbf{QR}
\end{equation}

Where $\mathbf{Q}\in\mathbb{R}^{m \times r}$ and $\mathbf{R}\in\mathbb{R}^{r \times r}$.

\subsubsection{Step 2: Compute SVD on Projected $\mathbf{Y = Q^T X}$}
The next step consists of projecting $\mathbf{Z}$ into $\mathbf{Q}$:

\begin{equation}
\mathbf{Y} = \mathbf{Q}^T\mathbf{X}
\end{equation}

Resulting in the matrix $\mathbf{Y}\in\mathbb{R}^{r\times n}$. Performing the SVD on $\mathbf{Y}$ (which is, again, much smaller than $\mathbf{X}$), gives:

\begin{equation}
\mathbf{Y} = \mathbf{U}_Y\mathbf{\Sigma}_r\mathbf{V}_r^T
\end{equation}

Where $\mathbf{\Sigma}_r$ and $\mathbf{V}_r$ turn out to likely be the same as those that would be obtained from the SVD of $\mathbf{X}$ itself. In the last step, $\mathbf{U}_r$ is recovered using the mathrix $\mathbf{Q}$, via $\mathbf{U}_r = \mathbf{QU}_Y$.

Guaranteed error bounds for the low rank approximation obtained with this technique. There are two approaches to improve the accuracy of randomized SVD. The first is to oversample, by letting the projection matrix $\mathbf{P}$ have a rank larger than $r$. The second is to "sharpen" the singular value spectrum by using the matrix $\mathbf{(X X^T)^q X }$ instead of $\mathbf{X}$. Calculating the power results in a matrix with a much faster drop off in the singular values, but is much more computationally expensive. This technique is useful when the singular values of $\mathbf{X}$ decay only slowly. The column space of the power iteration $\mathbf{...X X^T X X^T X X^T X}$ has the same column space, so that the trick with the projection operation works here too, only that the dominant subspace will be emphasized.\\

\begin{python}
def rSVD(X,r,q,p):
    """
    Randomized SVD Code
    """
    # Step 1: Sample column space of X with P matrix
    ny = X.shape[1]
    P = np.random.randn(ny,r+p)
    Z = X @ P
    for k in range(q):
        Z = X @ (X.T @ Z)

    Q, R = np.linalg.qr(Z,mode='reduced')

    # Step 2: Compute SVD on projected Y = Q.T @ X
    Y = Q.T @ X
    UY, S, VT = np.linalg.svd(Y,full_matrices=False)
    U = Q @ UY

    return U, S, VT
\end{python}

Empirically, randomized SVD outperforms SVD significantly the lower the rank of the TSVD.

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{randomized_svd.png}
    \caption{Left: Time taken to perform conventional SVD (black line) and randomized SVD (colored lines) on the photo of the two young aristocrats which had a data matrix with rank $r=1277$. The different colors correspond to different numbers of power iterations performed ($q$). Right: Average mean square error (AMSE) of truncated SVD using matrices derived with randomized SVD compared with conventional SVD. Especially for small $r$, rSVD is significantly faster even after a few power iterations. For very small $r$, power iterations easily reduce the error introduced by the randomized method by several orders of magnitude, but at some point the method seems to increase the error. Perhaps this is due to numerical stability. I'm not sure. To somewhat account for random fluctuations, the curves shown are averaged over 20 trials.}
    \label{fig:svd_scree}
\end{figure}


\section{Linear dynamical systems}
\subsection{Linear dynamical systems}
$x_{t+1} = A_tx_t$ , t = 1,2,. . .\\
\textbullet $A_t$ are n × n dynamics matrices
\textbullet $(A_t)_{ij}(x_t)_j$
is contribution to $(x_{t+1})_i$
from $(x_t)_j$\\
\textbullet system is called time-invariant if $A_t = A$ doesn't depend on time\\
\textbullet can simulate evolution of xt using recursion $x_{t+1} = A_tx$\\
\textbullet linear dynamical system with input\\
$x_{t+1} = A_tx_t + B_tu_t + c_t$, t = 1,2,...\\
\thickspace -- $u_t$ is an input m-vector\\
\thickspace -- $B_t$ is n × m input matrix\\
\thickspace -- $c_t$ is offset\\
\textbf{K-Markov model}:\\
$x_{t+1} = A_1x_t + ... + A_Kx_{t-K+1}$, t = K,K + 1,...\\
\thickspace -- next state depends on current state and K - 1 previous states\\
\thickspace – also known as auto-regresssive model\\
\thickspace – for K = 1, this is the standard linear dynamical system $x_{t+1} = Ax_t$\\
\subsection{Population dynamics}
\textbullet $x_t \in R^{100}$ gives population distribution in year t = 1,...,T\\
\textbullet $(x_t)^i$ is the number of people with age i - 1 in year t (say, on January 1)\\
\textbullet birth rate $b \in R^{100}$, death (or mortality) rate $d \in R^{100}$\\
\textbullet $b_i$ is the number of births per person with age i - 1\\
\textbullet $d_i$ is the portion of those aged i - 1 who will die this year (we’ll take $d_{100} = 1$)
\textbullet let’s find next year’s population distribution $x_t+1$ (ignoring immigration)\\
\textbullet number of 0-year-olds next year is total births this year:\\
$(x_{t+1})_1 = b^Tx_t$\\
\textbullet number of i-year-olds next year is number of (i - 1)-year-olds this year, minus those who die:
$(x_{t+1})_{i+1} = (1 - d_i)(x_t)_i$ , i = 1,..,99\\
\textbullet $x_{t+1} = Ax_t$ , where\\
$A =
\begin{bmatrix}
    b_1 & b_2 &...& b_{99} & {b_{100}} \\
    1-d_1 & 0 &...& 0 & 0\\
    0 & 1-d_2 &...& 0 & 0\\
    . & . &...& . &.\\
    . & . &...& . &.\\
    0 & 0 &...& 1-d_{99} & 0
\end{bmatrix}
$
\subsection{Epidemic dynamics}
\textbf{SIR Model}\\
\textbullet 4-vector $x_t$ gives proportion of population in 4 infection states\\
-- \textit{Susceptible}: can acquire the disease the next day\\
-- \textit{Infected}: have the disease
-- \textit{Recovered}: had the disease, recovered, now immune\\
-- \textit{Deceased}: had the disease, and unfortunately died\\
\textbullet sometimes called SIR model\\
\textbullet e.g., $x_t = (0.75,0.10,0.10,0.05)$
\textbf{over each day},\\
\textbullet among susceptible population,\\
-- 5\% acquires the disease\\
-- 95\% remain susceptible
\textbullet among infected population,\\
-- 1\% dies\\
-- 10\% recovers with immunity\\
-- 4\% recover without immunity (i.e., become susceptible)\\
-- 85\% remain infected\\
\textbullet 100\% of immune and dead people remain in their state\\
\textbullet epidemic dynamics as linear dynamical system\\
$
x_{t+1} =
\begin{bmatrix}
    0.95 &  0.04 &  0 & 0 \\
    0.05 & 0.85 & 0 & 0\\
    0 & 0.10 & 1 & 0\\
    . & . &...& . &\\
    . & . &...& . &\\
    0 & 0.01 & 0 & 1
\end{bmatrix}x_t
$\\


\chapter{Graph}

A \emph{graph} is \(G = (V,E)\) where
\(V\) is the set of \emph{vertices} and \(E \subseteq V \times V\) is the set of \emph{edges}.

\(E\) is a relation.

\(G\) is a \emph{function} iff \(\forall a, b, c : a E b \wedge a E c \implies b=c\).

\(G\) is \emph{undirected} iff \(E\) is symmetric.

A \emph{path} is a sequence of edges.

Let \(P(a,b)\) mean ``there is a path from \(a\) to \(b\)'' or ``\(a\) has a path to \(b\)''.
Then \(P(a,b) = E(a,b) \vee \exists c (P(a,c) \wedge P(c,b))\).
We say that \(b\) is \emph{reachable} from \(a\) iff there is a path from \(a\) to \(b\).

\(G\) is \emph{cyclic} if it has a vertex that has a path to itself.
\(G\) is \emph{acyclic} iff it is not cyclic.

\(G\) is \emph{connected} iff \(\forall a,b \in V, a\neq b : P(a,b)\).
\(G\) is \emph{disconnected} iff it is not connected.

\(G\) is \emph{complete} iff \(\forall a,b \in V, a \neq b : E(a,b)\).

\(G_1 = (V_1,E_1)\) is a \emph{subgraph} of \(G_2 = (V_2,E_2)\)
iff \(G_2\) is a graph, \(V_1 \subseteq V_2\), and \(E_1 \subseteq E_2\).

A \emph{loop} is an edge of the form \((a,a)\).

The \emph{length} of a path is the number of edges in that path.

% https://en.wikipedia.org/wiki/Distance_(graph_theory)

The \emph{geodesic distance} from \(a\) to \(b\) is \(d(a,b)\),
the length of the shortest path from \(a\) to \(b\).

The natural metric space of a connected undirected graph \((V,E)\) is \((V,d)\) where
\(d(a,b)\) is the length of the shortest path from \(a\) to \(b\).

The \emph{indegree} of \(v\) is \(|\{ (a,v) : a E v \}|\).

The \emph{outdegree} of \(v\) is \(|\{ (v,a) : v E a \}|\).

A vertex is \emph{source} iff its indegree is zero.

A vertex is \emph{sink} iff its outdegree is zero.

The graph's \emph{fringe} of \(A\) is \(F(A) = \{ b ~|~ a \in A, ~ (a,b) \in E \}\).

Let \(F^{n+1}(A) = F(F^n(A))\).
Property: \(F(\emptyset) = \emptyset\).
Property: \(F(A \cup B) = F(A) \cup F(B)\).

A \emph{tree} is an undirected acyclic graph.


\subsection{\(L_p\) norms}

\(L_p\) norms can be used to measure the distance between two metrics.

If we have data points \(v\) and \(w\) the distance is:

\(||v-w||=(\sum_i |v_i-w_i|^p)^{\dfrac{1}{p}}\)

If \(p=2\) we have the Euclidian norm. If \(p=1\) we have the Manhatten norm.


\textbf{Solve the nonlinear boundary value problem}
\begin{align*}
y''=y'+\cos(y),~~y(0)=0,~~y(\pi)=1.
\end{align*}
\textbf{Use finite differences to discretize the problem and Newton’s method to solve the resulting system of equations. Estimate (numerically) the number of discretized points needed to obtain a solution that is accurate to 4 digits at $x = \pi/6$.}

We can express the previous differential equation as
\begin{align*}
y''-y'-\cos(y)=0,
\end{align*}
so that its roots will give us the solution. To do so we need to discretize the derivatives using finite differences in order to be able to use Newton's method. Then, the differential equation takes the linear form
\begin{align*}
y'' - y_x - cos(y) &= D_2y - D_1y - cos(y)\\
&=My-\cos(y) =0~,
\end{align*}
where the matrix $M = D_2 - D_1$. The matrices $D_1$ and $D_2$ will give us the discrete values of the first and second derivative, respectively, and are obtained using second order centered finite differences. For $D_1$,
\begin{align*}
&y' = \frac{1}{2(\Delta x)}
\begin{bmatrix}
-1 & 0 & 1 & ~&  \dots & 0 \\
0 & -1 & 0 & 1 & \dots & 0 \\
\vdots & ~&  \ddots & \ddots & \ddots & \vdots \\
0 & \dots & ~ & -1 & 0 & 1 \\
\end{bmatrix}y +
\begin{bmatrix}
0 \\
\vdots \\
0 \\
\frac{1}{2 \Delta x}
\end{bmatrix} = D_1y + g_1,
\end{align*}
where the column vector $g_1$ contains is found by imposing the boundary conditions. Note that the matrices are calculated in such a way that they give us the derivatives for the interior nodes, since the boundary nodes we know their values thanks to the boundary conditions. For $D_2$,
\begin{align*}
& y'' = \frac{1}{(\Delta x)^2} \begin{bmatrix}
1 & -2 & 1 & ~&  \dots & 0 \\
0 & 1 & -2 & 1 & \dots & 0 \\
\vdots & ~&  \ddots & \ddots & \ddots & \vdots \\
0 & \dots & ~ & 1 & -2 & 1 \\
\end{bmatrix}y + \begin{bmatrix}
0 \\
\vdots \\
0 \\
\frac{1}{( \Delta x)^2}
\end{bmatrix} = D_2y + g_2~,
\end{align*}
where the column vector $g_2$ contains is found by imposing the boundary conditions. We now define
\begin{align*}
\vec{F}(\vec{y}) = My - cos(y) + g_2 - g_1~,
\end{align*}
where $\vec{y}$ contains the interior nodes. The solution to our differential equations will be the vector $y$ that makes $\vec{F}(\vec{y})=0$. To calculate those roots we will use Newton's method. In order to be able to use it, we need to calculate the Jacobian,
\begin{align*}
J_{jk}=\frac{\partial}{y_j}F(y_k).
\end{align*}
First note that
\begin{align*}
\frac{\partial }{\partial y_j} \sum_{l=1}^{N}M_{kl}y_l = M_{kl}
\end{align*}
and
\begin{align*}
\frac{\partial }{\partial y_j} \cos(y_k) = \begin{dcases}
	-\sin(y_k) & \text{for}~k = j\\
		0 & \text{for}~j \neq k
\end{dcases}
\end{align*}
Hence, the Jacobian can be coded as
\begin{align*}
J(\vec{F}(\vec{y})) = M + \text{diag}(\sin(y)).
\end{align*}

We will first run the code in a while loop to find out how many nodes we need to have an $L_1$ norm of $y^{new}(\pi/6)-y^{old}(\pi/6)$ less than a $tol=10^{-4}$. Using 31 nodes I obtained an $L_1$ norm of $7.269\cdot 10^{-5}$. To be safe, since it is a low number of nodes and the code runs fast, we could run it for double number of elements $N$. Using then 61 nodes we obtain an $L_1$ norm of the error of $7.603\cdot 10^{-6}$. Using 61 nodes we have obtained the solution showed in the next figure

\begin{figure}[H]
\center{\includegraphics[scale=.75]{prob3sol.eps}}
\caption{Solution for 61 points}
\end{figure}

\subsection*{Matlab code for this problem}

\begin{verbatim}
%% Problem 3
close all
clc
format long
% Find an answer with sufficient level of convergence
tol=1e-4;
i=1;
N=6;
err=1;
yy=zeros(6+1,1);
while err>tol
    i=i+1;
    y_old=yy(N/6);
    N = i*6;
    xi=linspace(0,pi,N+1)';
    [F,J,y0]=FandJ(N);
    [yy,niter] = newton(F,J,y0,1e-14);
    y_new=yy(N/6);
    % Check solution
    err=max(abs(y_new-y_old));
end
% Results
nodes=N+1
err
% Run it for N=60
N=60;
xi=linspace(0,pi,N+1)';
[F,J,y0]=FandJ(N);
[yy,niter] = newton(F,J,y0,1e-14);
y_new=yy(N/6);
% Plot the solution
figure
plot(xi,[0; yy; 1],'linewidth',1.5)
hold on
plot(xi(N/6+1),y_new,'r.','markerSize',12)
grid on
leg1= legend('Solution', 'Value at $\pi/6$');
set(leg1,'interpreter','latex','FontSize',17);
set(gca,'fontsize',axisfontsize)
xlabel('$x$','interpreter','latex','fontsize',labelfontsize)
ylabel('$y$','interpreter','latex','fontsize',labelfontsize)
saveas(gcf,'Latex/FIGURES/prob3sol','epsc')
\end{verbatim}
\subsubsection*{Function to calculate F and J}
\begin{verbatim}
function [F,J,y0]=FandJ(N)
x = linspace(0,pi,N+1)';
h=pi/N;
xi = x(2:N);
y0 = xi/pi;

D1 = (gallery('tridiag',N+1,-1,0,1))/(2*h);
D1(1,:) = [];
D1(end,:) = [];
D1(:,1) = [];
D1(:,end) = [];

D2 = (gallery('tridiag',N+1,1,-2,1))/(h^2);
D2(1,:) = [];
D2(end,:) = [];
D2(:,1) = [];
D2(:,end) = [];

% g1 = [zeros(N-2,1)];
% g1(N-2) = (1/(2*h));
% g2 = [zeros(N-2,1)];
% g2(N-2) = (1/(h^2));
% e1 = g2 - g1;

M = D2 - D1;
e1 = [zeros(N-1,1)];
e1(N-1) = ((2-h)/(2*h^2));

F = @(y) (M*y)-cos(y)+e1;
J = @(y) M + diag(sin(y));
end
\end{verbatim}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS671: Machine Learning
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 3}

Let $f: \mathbb{R}^2 \rightarrow \mathbb{R}$ be the function defined by

\begin{equation}
f(x) = x_1^2 + 2x_1 x_2 - 10x_1 + 5x_2
\label{eq31}
\end{equation}

Is $f$ convex, concave or neither? Why?

\subsection*{Solution}

We use the fact that $f(x)$ is a twice-differentiable continuous function for any $x\in \mathbb{R}^2$.
Let $H_f$ be defined as Hessian matrix of the function $f$.
We use the theorem that function $f$ is convex, if and only if, $H_f(x)$ is positive semidefinite for all $x\in \mathbb{R}^2$, and $f$ is concave if and only if $H_f$ is negative semidefinite for all $x\in \mathbb{R}^2$.
Therefore, objective would be to obtain $H_f$ and verify whether it's positive semidefinite or negative semidefinite or neither.

By definition, $H_f$ can be obtained using second partial derivatives of $f(x)$ as shown in Eq. \ref{eq32}.

\begin{equation}
H_f(\mathbf{x})
= \begin{pmatrix}
\frac{\partial^2 f}{\partial^2 x_1} & \frac{\partial^2 f}{\partial x_1 \partial x_2}\\[0.6em]
\frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial^2 x_2}\\
\end{pmatrix}
\label{eq32}
\end{equation}

Using Eq. \ref{eq31}, partial derivatives of $H_f$ are obtained as given in Eq. \ref{eq33} and Eq. \ref{eq34}.

\begin{equation}
\frac{\partial f}{\partial x_1} = 2x_1 + 2x_2 - 10
\label{eq33}
\end{equation}
\begin{equation}
\frac{\partial f}{\partial x_2} = 2x_1 + 5
\label{eq34}
\end{equation}

Substituting Eq. \ref{eq33} and Eq. \ref{eq34} into Eq. \ref{eq32},

\begin{equation}
H_f(\mathbf{x}) = \begin{pmatrix}2 & 2\\2 & 0\\\end{pmatrix}
\end{equation}

For $H_f(\mathbf{x})$ to be positive semidefinite, eigenvalues of $H_f$ must be non-negative.
Similarly, for $H_f(\mathbf{x})$ to be negative semidefinite, eigenvalues of $H_f$ must be non-positive.
To obtain Eigenvalues of $H_f$, characteristic polynomial of $H_f$ is constructed as shown in Eq. \ref{eq35}.

\begin{equation}
A-\lambda I = \begin{pmatrix}2-\lambda & 2\\ 2 & -\lambda\\\end{pmatrix}\Rightarrow \text{det}(A-\lambda I) = (\lambda - 1)^2 -5
\label{eq35}
\end{equation}

As eigenvalues are solutions to Eq. \ref{eq35}, $H_f$ will have two eigenvalues $1+\sqrt{5} > 0$ and $1-\sqrt{5} < 0$.
Therefore, $H_f$ is neither positive semidefinite nor negative semidefinite.
Hence, function $f$ is neither convex nor concave.


\subsection{The optimisation problem}

Suppose all training samples $\z\train_t$ are gathered and divided over two data matrices $\Sm \in \reals^{C \cross N_S}$ and $\Nm \in \reals^{C \cross N_N}$, where $\Sm$ (for `signal') contains all $N_S$ samples of $\z\train_t$ where an SWR is present, and $\Nm$ (for `noise') contains all $N_N$ other samples. (These matrices can be easily constructed by concatenating segments from $\z\train_t$).

\Cref{eq:linear} then becomes, in vector notation:
%
\begin{align*}
\out_S &= \w^T \Sm \\
\out_N &= \w^T \Nm,
\end{align*}
where each element of the row vectors $\out_S$ and $\out_N$ is a filtered sample of $\Sm$ and $\Nm$, respectively. \Cref{fig:GEVec_principle} (right) shows the distribution of the values in two example data vectors $\out_S$ and $\out_N$.

We want to find the weight vector $\what$ that maximises the variance of $\out_S$ versus the variance of $\out_N$, i.e.
%
\begin{align}
\what &= \argmax_{\w} \frac{\Var(\out_S)}
                           {\Var(\out_N)}  \label{eq:argmax_SNR} \\[1em]
      &= \argmax_{\w}
         \frac{\frac{1}{N_S} \out_S \out_S^T}
              {\frac{1}{N_N} \out_N \out_N^T}   \nonumber \\[1em]
      &= \argmax_{\w}
         \frac{\frac{1}{N_S} \w^T \Sm \Sm^T \w}
              {\frac{1}{N_N} \w^T \Nm \Nm^T \w}  \nonumber
\end{align}

In this last equation, we recognize the empirical covariance matrices $\Rss$ and $\Rnn$, which are defined as:
%
\begin{align}
\Rss &= \frac{1}{N_S} \Sm \Sm^T \label{eq:covariance_S}\\
\Rnn &= \frac{1}{N_N} \Nm \Nm^T \label{eq:covariance_N}
\end{align}
%
$\Rss \in \reals^{C \cross C}$ and $\Rnn \in \reals^{C \cross C}$ are symmetric matrices, where each diagonal element yields the variance of a channel, and each off-diagonal element yields the covariance between a pair of channels.

The condition for the optimal weight vector, \cref{eq:argmax_SNR}, is thus equivalent to:
%
\begin{equation}
\label{eq:argmax_R}
\what = \argmax_{\w} \frac{\w^T \Rss \w}
                          {\w^T \Rnn \w}
\end{equation}
%
In \cref{apx:GEvecs-maximise-SNR}, we show that the solution $\what$ to this optimisation problem is the first so called ``generalized eigenvector'' of $(\Rss, \Rnn)$.


\section{Higher-Order Taylor Method}
\begin{defn}[Local Truncation Error]
The difference method
\begin{equation*}
  \begin{cases}
    \omega_0=\alpha & \\
    \omega_{i+1}=\omega_i+h\phi(t_i+\omega_i) & \text{for each $i=0,1,\cdots,N-1$}
  \end{cases}
\end{equation*}
has local truncation error
\begin{equation*}
  \tau_{i+1}(x)=\frac{y_{i+1}-\left(y_i+h\phi(t_i+y_i)\right)}{h}=\frac{y_{i+1}-y_i}{h}-\phi(t_i,y_i)
\end{equation*}
for each $i=0,1,\cdots,N-1$ where $y_i$ and $y_{i+1}$ denote the accuracy at a specific step, assuming that the method was exact at the previous step.
\end{defn}
Euler's method has $\tau_{i+1}=\frac{h}{2}y''(\xi_i)$, so the local truncation error in Euler's method is $O(h)$ .
\subsection{Taylor Method of Order $n$}
\begin{equation*}
  \begin{cases}
    \omega_0=\alpha & \\
    \omega_{i+1}=\omega_i+h\phi(t_i+\omega_i) & \text{for each $i=0,1,\cdots,N-1$}
  \end{cases}
\end{equation*}
where $T^{(n)}(t_i,\omega_i)=f(t_i,\omega_i)+\frac{h}{2}f'(t_i,\omega_i)+\cdots+\frac{h^{n-1}}{n!}f^{(n-1)}(t_i,\omega_i)$.
\begin{theo}
If Taylor's method of order $n$ is used to approximate the solution to
\[
\frac{\D y}{\D t}=f(t,y),\quad a\leq t\leq b,\quad y(a)=\alpha
\]
with step size $h$ and if $y\in C^{n+1}[a,b]$, then the local truncation error is $O(h^n)$.
\end{theo}
\begin{proof}
\begin{align*}
  y_{i+1} &= y_i+hf(t_i,y_i)+\cdots+\frac{h^n}{n!}f^{(n-1)}(t_i,y_i)+\frac{h^{n+1}}{(n+1)!}f^{(n)}(\xi_i,y(\xi_i)) \\
  \Rightarrow & \tau_{i+1}(h) = \frac{y_{i+1}-y_i}{h}-T^{(n)}(t_i,y_i)=\frac{h^n}{(n+1)!}f^{(n)}(\xi_i,y(\xi_i))
\end{align*}
for each $i=0,1,\cdots,N-1$. Since $y\in C^{n+1}[a,b]$, we have $y^{(n+1)}(t)=f^{(n)}(t,y(t))$ bounded on $[a,b]$ and $\tau_i(h)=O(h^n)$ for each $i=1,2,\cdots,N$.
\end{proof}


\subsection{Green's theorem}
Green's theorem tells us that

$$\oint P\,dx+Q\,dy=\int\!\!\!\int
\left({\partial Q\over\partial x}-{\partial P\over\partial y}\right)
dx\,dy$$

In other words, a line integral and a surface integral can yield
the same result.

Example 1.
The following exercise is from {\it Advanced Calculus}
by Wilfred Kaplan, p.~287.
Evaluate $\oint (2x^3-y^3)\,dx+(x^3+y^3)\,dy$ around the circle
$x^2+y^2=1$ using Green's theorem.

It turns out that Eigenmath cannot solve the double integral over
$x$ and $y$ directly.
Polar coordinates are used instead.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
P = 2x^3-y^3
Q = x^3+y^3
f = d(Q,x)-d(P,y)
x = r*cos(theta)
y = r*sin(theta)
defint(f*r,r,0,1,theta,0,2pi)
\end{Verbatim}

$\displaystyle \frac{3}{2}\pi$

The $defint$ integrand is $f{*}r$ because $r\,dr\,d\theta=dx\,dy$.

Now let us try computing the line integral side of Green's theorem
and see if we get the same result.
We need to use the trick of converting sine and cosine to exponentials
so that Eigenmath can find a solution.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
x = cos(t)
y = sin(t)
P = 2x^3-y^3
Q = x^3+y^3
f = P*d(x,t)+Q*d(y,t)
f = circexp(f)
defint(f,t,0,2pi)
\end{Verbatim}

$\displaystyle \frac{3}{2}\pi$

Example 2.
Compute both sides of Green's theorem for
$F=(1-y,x)$ over the disk $x^2+y^2\le4$.

First compute the line integral along the boundary of the disk.
Note that the radius of the disk is 2.

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
-- Line integral
P = 1-y
Q = x
x = 2*cos(t)
y = 2*sin(t)
defint(P*d(x,t)+Q*d(y,t),t,0,2pi)
\end{Verbatim}

$\displaystyle 8\pi$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
-- Surface integral
x = quote(x) --clear x
y = quote(y) --clear y
h = sqrt(4-x^2)
defint(d(Q,x)-d(P,y),y,-h,h,x,-2,2)
\end{Verbatim}

$\displaystyle 8\pi$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
-- Try computing the surface integral using polar coordinates.
f = d(Q,x)-d(P,y) --do before change of coordinates
x = r*cos(theta)
y = r*sin(theta)
defint(f*r,r,0,2,theta,0,2pi)
\end{Verbatim}

$\displaystyle 8\pi$

\begin{Verbatim}[formatcom=\color{blue},samepage=true]
defint(f*r,theta,0,2pi,r,0,2) --try integrating over theta first
\end{Verbatim}

$\displaystyle 8\pi$

In this case, Eigenmath solved both forms of the polar integral.
However, in cases where Eigenmath fails to solve a double integral, try
changing the order of integration.


\subsection{Conditional probability}

We define conditional probability

\(P(E_i|E_j):=\dfrac{P(E_i\land E_j)}{P(E_j)}\)

We can show this is between \(0\) and \(1\).

\(P(E_j)=P(E_i\land E_j)+P(\bar{E_i}\land E_j)\)

\(P(E_i|E_j):=\dfrac{P(E_i\land E_j)}{ P(E_i\land E_j)+P(\bar{E_i}\land E_j)}\)

We know:

\(P(x_i|y_j):=\dfrac{P(x_i \land y_j)}{P(y_j)}\)

\(P(y_j|x_i):=\dfrac{P(x_i \land y_j)}{P(x_i)}\)

So:

\(P(x_i|y_j)P(y_j)=P(y_j|x_i) P(x_i)\)

\(P(x_i|y_j)=\dfrac{P(y_j|x_i) P(x_i)}{P(y_j)}\)

Note that this is undefined when \(P(y_j)=0\)

Note that for the same event,

\(P(x_i|x_j)=\dfrac{P(x_i\land x_j)}{P(x_j)}\)

\(P(x_i|x_j)=0\)

For the same outcome:

\(P(x_i|x_i)=\dfrac{P(x_i\land x_i)}{P(x_i)}\)

\(P(x_i|x_i)=\dfrac{P(x_i)}{P(x_i)}\)

\(P(x_i|x_i)=1\)

\subsection{Bayes' theorem}

From the definition of conditional probability we know that:

\(P(E_i|E_j):=\dfrac{P(E_i\land E_j)}{P(E_j)}\)

\(P(E_j|E_i):=\dfrac{P(E_i\land E_j)}{P(E_i)}\)

So:

\(P(E_i\land E_j)=P(E_i|E_j)P(E_j)\)

\(P(E_i\land E_j)=P(E_j|E_i)P(E_i)\)

So:

\(P(E_i|E_j)P(E_j)=P(E_j|E_i)P(E_i)\)

\subsection{Independent variables}

Events are independent if:

\(P(E_i|E_j)=P(E_i)\)

Note that:

\(P(E_i\land E_j)=P(E_i|E_j)P(E_j)\)

And so for independent events:

\(P(E_i\land E_j)=P(E_i)P(E_j)\)

\subsection{Conjugate priors}

If the prior \(P(\theta)\) and the posterior \(P(\theta | X)\) are in the same family of distributions (eg both Gaussian), then the prior and posterior are conjugate distributions