\chapter{Relations}
\label{chapter:relations}
Nonetheless that function are used almost everywhere in mathematics, many
relations are not functional by their nature. For example, for any real
$a$, there are two solutions of $x^2 = a$ and there are zero solutions for
$a < 0$. To work with such situations, relations are used.

In order to define a relation we need to relax the definition of the graph of a
function (Section~\ref{section:graph}) by allowing more than one ``result'' and
by allowing zero ``results''. In other words we just say that any set $R
\subseteq X_1 \times \dots \times X_k$ is a $k$-ary \emph{relation} on $X_1$,
\dots, $X_k$. We also say that $x_1 \in X_1$, \dots, $x_k \in X_k$ are in the
relation $R$ iff $(x_1, \dots, x_k) \in R$. If $k = 2$ such a relation is
called a \emph{binary relation} and we write $x R y$ if $x$ and $y$ are in
the relation $R$. If $X_1 = \dots = X_k = X$, we say that $R$ is a $k$-ary
relation on $X$.

Note that $=$, $\le$, $\ge$, $<$, and $>$ define relations on $\R$ (or any
subset $S$ of $\R$). For example, if $S = \set{0, 1, 2}$, then $<$ defines
the relation $R = \set{(0, 1), (0, 2), (1, 2)}$.

Another widely used family of relations on $\Z$ can be defined as follows. Let
$n, a, b \in \Z$. If $n$ divides $a - b$, we say that
``$a$ equivalent to $b$ modulo $n$'' and denote it as $a \equiv b \pmod{n}$.
For example, $1$ and $4$ are equivalent modulo $3$ since $3$ divides
$1 - 4 = -3$.

\section{Equivalence Relations}
The definition of a relation is way too broad. Hence, quite often we consider
some types of relation. Probably the most interesting type of the relations is
equivalence relations.

\begin{definition}
  Let $R$ be a binary relation on a set $X$. We say that $R$ is an
  \emph{equivalence relation} if it satisfies the following conditions:
  \begin{description}
    \item[(reflexivity)] $x R x$ for any $x \in X$;
    \item[(symmetry)] $x R y$ iff $y R x$ for any $x, y \in X$;
    \item[(transitivity)] for any $x, y, z \in X$, if $x R y$ and $y R Z$,
      then $x R z$.
  \end{description}
\end{definition}
One may guess that the equivalence relation are mimicking $=$, so it is not a
surprise that $=$ is an equivalence relation.

The definition seems quite bizarre, however, all of you are already familiar
with another important example: you know that equivalent fractions represent the
same number. For example, $\frac{2}{4}$ is the same as $\frac{1}{2}$.
Let us consider this example more thorough, let $S$ be a set of symbols of the
form $\frac{x}{y}$ (note that it is not a set of numbers) where $x, y \in Z$
and $y \neq 0$. We define a binary relation $R$ on $S$ such that
$\frac{x}{y}$ and $\frac{z}{w}$ are in the relation $R$ iff $xw = zy$.
It is easy to prove that this relation is an equivalence relation.
\begin{description}
  \item[(reflexivity)] Let $\frac{a}{b} \in S$. Since $ab = ab$, we have that
    $\frac{a}{b} R \frac{a}{b}$.
  \item[(symmetry)] Let $\frac{a}{b}, \frac{c}{d} \in S$. Suppose that
    $\frac{a}{b} R \frac{c}{d}$, by the definition of $R$, it implies that
    $ac = db$. As a result, $\frac{c}{d} R \frac{a}{b}$.
  \item[(transitivity)] Let $\frac{a}{b}, \frac{c}{d}, \frac{e}{f} \in S$ with
    $\frac{a}{b} R \frac{c}{d}$ and $\frac{c}{d} R \frac{e}{f}$. Then
    $ad = cb$ and $cf = ed$. The first equality can be rewritten as
    $c = ad / b$.
    Hence, $adf / b = ed$ and $af = eb$ since $d \neq 0$. So $\frac{a}{b} R
    \frac{e}{f}$.
\end{description}

\subsection{Partitions}
Let $S$ be some set. We say that $\set{P_1, \dots, P_k}$ form a partition of
$S$ iff $P_1$, \dots, $P_k$ are pairwise disjoint and
$P_1 \cup \dots \cup P_k = S$; in other words, a partition is a way of dividing
a set into overlapping pieces.

\begin{exercise}
  Let $\set{P_1, \dots, P_k}$ be a partition of a set $S$ and $R$ be a binary
  relation of $S$ such that $a R b$ iff $a, b \in P_i$ for some
  $i \in \range{k}$. Show that $R$ is an equivalence relation.
\end{exercise}

This exercise shows that one may transform a partition of the set $S$ into an
equivalence relation on $S$. However, it is possible to do the opposite.
\begin{theorem}
\label{theorem:partition}
  Let $R$ be a binary equivalence relation on a set $S$. For any element
  $x \in S$, define $R_x = \set[x R y]{y \in S}$ (the set of all the elements
  of $S$ related to $x$) we call such a set the equivalence class of $x$.
  Then $\set[x \in S]{R_x}$ is a partition of $S$.
\end{theorem}

\begin{exercise}
  Prove Theorem~\ref{theorem:partition}.
\end{exercise}

\subsection{Modular Arithmetic}
The relation ``$\equiv \pmod{n}$'' is actively used in the number theory. One
of the important properties of this relation is that it is an equivalence
relation.
\nomenclature[U]{$a \equiv b \pmod{n}$}{says that $n$ divides $a - b$}
\begin{theorem}
  The relation $\equiv \pmod{n}$ is an equivalence relation.
\end{theorem}
\begin{proof}
  To prove this statement we need to prove all three properties: reflexivity,
  symmetry, and transitivity.
  \begin{description}
    \item[(reflexivity)] Note that for any integer $x$, $x - x = 0$ is divisible
      by any integer including $n$. Hence, $x \equiv x \pmod{n}$.
    \item[(symmetry)] Let us assume that $x \equiv y \pmod{n}$; i.e.,
      $x - y = kn$ for some integer $k$. Note that $y - x = (-k) n$, so
      $y \equiv x \pmod{n}$.
    \item[(transitivity)] finally, assume that $x \equiv y \pmod{n}$ and
      $y \equiv z \pmod{n}$; i.e. $x - y = k n$ and $y - z = \ell n$
      for some integers $k$ and $\ell$. It is easy to note that
      $x - z = (x - y) + (y - z) = (k + \ell) n$. As a result,
      $x \equiv z \pmod{n}$.
  \end{description}
  Thus, we proved that $\equiv \pmod{n}$ is an equivalence relation.
\end{proof}

Let $x \in \Z$; we denote by $r_{x, n}$ the equivalence class of $x$ with
respect to the relation $\equiv \pmod{n}$, we also denote by $\Z / n\Z$ the set
of all the equivalence classes with respect to the relation $\equiv \pmod{n}$.

Another important property of these relations is that they behave well with
respect to the arithmetic operations.
\begin{theorem}
  Let $x, y \in Z$ and $n \in \N$. Suppose that $a \in r_{x, n}$ and $b \in
  r_{y, n}$, then $(a + b) \in r_{x + y, n}$ and $ab \in r_{xy, n}$.
\end{theorem}

Using this theorem we may define arithmetic operations on the equivalence
classes with respect to the relation $\equiv \pmod{n}$.
Let $x, y \in \Z$ and $n \in \N$. Then
$r_{x, n} + r_{y, n} = \set[a \in r_{x, n}, b \in r_{y, n}]{a + b} =
r_{x + y, n}$ and
$r_{x, n} r_{y, n} = \set[a \in r_{x, n}, b \in r_{y, n}]{ab} = r_{xy, n}$.
Moreover, these operations have plenty of good properties.
\begin{exercise}
  Let $a, b, c \in \Z / n\Z$.
  Show that the following equalities are true:
  \begin{itemize}
    \item $a + (b + c) = (a + b) + c$,
    \item $a + r_{0, n} = a$ (thus we denote $r_{0, n}$ as $0$),
    \item $a r_{1, n} = a$ (thus we denote $r_{1, n}$ as $1$),
    \item there is a class $d \in \Z / n\Z$ such that $a + d = r_{0, n}$
      (thus we denote this $d$ as $-a$),
    \item $a + b = b + a$,
    \item $ab = ba$,
    \item $a(b + c) = ab + ac$,
  \end{itemize}
\end{exercise}

\section{Partial Orderings}
In the previous section we discussed a generalization of ``$=$''.
In this section we are going to give a way to analyze relations similar to
``$<$''.

\begin{definition}
  A binary relation $R$ on $S$ is a partial ordering if it satisfies the
  following constraints.
  \begin{description}
    \item[(reflexivity)]  $x R x$ for any $x \in S$;
    \item[(antisymmetry)] if $x R y$ and $y R x$, then $x = y$ for all
      $x, y \in S$;
    \item[(transitivity)] for any $x, y, z \in S$, if $x R y$ and $y R Z$, then
      $x R z$;
  \end{description}

  We say that a partial ordering $R$ on a set $S$ is total iff for any
  $x, y \in S$, either $x R y$ or $y R x$.
\end{definition}
\noindent Note that $\le$ defines a partial ordering on any $S \subseteq \R$;
moreover, it defines a total order.

Typically we use symbols similar to $\preceq$ to denote partial orderings and we
write $a \prec b$ to express that $a \preceq b$ and $a \neq b$.

Let $\divides$ be the relation on $\Z$ such that $d \divides n$ iff $d$
divides $n$.
\nomenclature[U]{$a \divides b$}{says that $a$ divides $b$}
\begin{theorem}
  The relation $\divides$ is a partial ordering of the set $\N$.
\end{theorem}
\begin{proof}
  To prove that this relation is a partial ordering we need to check all three
  properties.
  \begin{description}
    \item[(reflexivity)] Note that $x = 1 \cdot x$ for any integer $x$;
      hence, $x \divides x$ for any integer $x$.
    \item[(antisymmetry)] Assume that $x \divides y$ and $y \divides x$. Note
      that it means that $k x = y$ and $\ell y = x$ for some integers $k$ and
      $\ell$. Hence, $y = (k \cdot \ell) y$ which implies that
      $k \cdot \ell = 1$ and $k = \ell = 1$. Thus, $x = y$.
    \item[(transitivity)] finally, assume that $x \divides y$ and
    $y \divides z$; i.e., $k x = y$ and $\ell y = z$. As a result,
    $(k \cdot \ell) x = z$ and $x \divides z$.
  \end{description}
\end{proof}

\begin{exercise}
  Let $S$ be some set, show that $\subseteq$ defines a partial ordering on the
  set $2^S$.
\end{exercise}

\subsection{Topological Sorting}
Partial orderings are very useful for describing complex processes. Suppose
that some process consists of several tasks, $T$ denotes the set of these tasks.
Some tasks can be done only after some others e.g. when you cooking a salad you
need to wash vegetables before you chop them. If $x, y \in T$ be some tasks, $x
\preceq y$ if $x$ should be done before $y$ and this is a partial ordering.

In the applications this order is not a total order because some steps do not
depend on other steps being done first (you can chop tomatoes and chop
cucumbers in any order). However, if we need to create a schedule in which the
tasks should be done, we need to create a total ordering on $T$. Moreover, this
order should be compatible with the partial ordering. In other words, if $x
\preceq y$, then $x \preceq_t y$ for all $x, y \in T$, where $\preceq_t$ is the
total order. The technique of finding such a total ordering is called
\emph{topological sorting}.

\begin{theorem}
\label{theorem:topological-order}
  Let $S$ be a finite set and $\preceq$ be a partial order on $S$. Then there is
  a total order $\preceq_t$ on $S$ such that if $x \preceq y$, then
  $x \preceq_t y$ for all $x, y \in S$
\end{theorem}

This sorting can be done using the following procedure.
\begin{itemize}
  \item Initiate the set $S$ being equal to $T$
  \item Choose the minimal element of the set $S$ with respect to the ordering
    $\preceq$ (such an element exists since $S$ is a finite set, see
    Chapter~\ref{chapter:bijections-surjections-injections}).
    Add this element to the list, remove it from the set $S$, and repeat this
    step if $S \neq \emptyset$.
\end{itemize}

Let us consider the following example. In the left column we list the classes
and in the right column the prerequisite.
\begin{center}
    \begin{tabular}{l l}
        \toprule
        Courses & Prerequisite \\
        \midrule
        Math 20A \\
        Math 20B & Math 20A \\
        Math 20C & Math 20B \\
        Math 18 \\
        Math 109 & Math 20C, Math 18 \\
        Math 184A & Math 109 \\
        \bottomrule
    \end{tabular}
\end{center}
We need to find an order to take the courses.
\begin{enumerate}
  \item We start with
    \[
      S = \set{\text{Math 20A}, \text{Math 20B},
        \text{Math 20C}, \text{Math 18}, \text{Math 109}, \text{Math 184}}.
    \]
    There are two minimal elements: Math 20A and Math 18. Let us remove Math 18
    from $S$ and add it to the resulting list $R$.
  \item Now we have
    \[
      R = \text{Math 18}
    \]
    and
    \[
      S = \set{\text{Math 20A},
      \text{Math 20B}, \text{Math 20C}, \text{Math 109}, \text{Math 184}}.
    \]
    There is only one minimal element Math 20A. We remove it and add it to the
    list $R$.
  \item On this step
    \[
      R = \text{Math 18}, \text{Math 20A}
    \]
    and
    \[
      S = \set{
        \text{Math 20B}, \text{Math 20C}, \text{Math 109}, \text{Math 184}}.
    \]
    Again there is only one minimal element: Math 20B.
  \item
    \[
      R = \text{Math 18}, \text{Math 20A}, \text{Math 20B}
    \] and
    \[
      S = \set{\text{Math 20C}, \text{Math 109}, \text{Math 184}}.
    \]
    There is only one minimal element: Math 20C.
  \item
    \[
      R = \text{Math 18}, \text{Math 20A}, \text{Math 20B}, \text{Math 20C}
    \]
    and
    \[
      S = \set{\text{Math 109}, \text{Math 184}}.
    \]
    There is only one minimal element: Math 109.
  \item Finally,
    \[
      R = \text{Math 18}, \text{Math 20A}, \text{Math 20B}, \text{Math 20C},
      \text{Math 109}
    \]
    and
    \[
      S = \set{\text{Math 184}}.
    \]
    There is only one minimal element: Math 184A.
\end{enumerate}
As a result, the final list is
\[
  R = \text{Math 18}, \text{Math 20A}, \text{Math 20B}, \text{Math 20C},
  \text{Math 109}, \text{Math 184A}.
\]

\begin{chapterendexercises}
  \exercise[recommended] Show that the relation $\divides$ does not define a partial
    ordering on $\Z$.
  \exercise Let a relation $R$ be defined on the set of real numbers as
    follows: $x R y$ iff $2x + y = 3$. Show that it is antisymmetric.
  \exercise Are there any minimal elements in $\N$ with respect to
    $\divides$? Are there any maximal elements?
\end{chapterendexercises}


\section{$\mathbf{X}=\mathbf{W}\mathbf{H}$ Non-Negative Matrix Factorization}

Non-Negative Matrix Factorization (NNMF) is an approximate decomposition technique for positive data. It can be used, for example, for dimensionality reduction, blind source detection or topic extraction. The reputation is that it can often generate decompositions that are "sparse and meaningful".

Following the convention in \citeasnoun{gillis2014nonnegative}, it works by expressing a positive data matrix $\mathbf{X}\in\mathbb{R}^{p\times n}_+$ in terms of two smaller positive matrices $\mathbf{W}\in\mathbb{R}_+^{p\times r}$ and $\mathbf{H}\in \mathbb{R}^{r\times n}_+$. Here the $n$ columns of the data matrix are data points and the rows are the $p$ features. An excellent introduction to NNMF is found in \possessivecite{morningpaper2019nnmf} blog post and in \citeasnoun{gillis2014nonnegative}. Dimensionality reduction derives from the ability to adjust the rank of the decomposition via the dimension $r$. The decomposition is simply:

\begin{equation}
\mathbf{X} \approx \mathbf{WH}
\end{equation}

Where the error $||\mathbf{X} - \mathbf{WH}||_\alpha$ is minimized with respect to some norm (normally, the Frobenius norm with $\alpha=2$). The decomposition allows the interpretation of $\mathbf{W} \in \mathbb{R}^{p\times r}$ as a matrix with a number of $r$ $p$-dimensional strictly positive basis vectors and of $\mathbf{H} \in \mathbb{R}^{r\times n}$ as a matrix of coefficients that express the $n$ data points in terms of linear sums of the $r$ basis vectors with strictly positive coefficients.

One might ask: why NNMF when I can create linear decompositions of my data with SVD and PCA? After all, these techniques are well established and are known to give the optimal approximation under the Frobenius nurm. In my perception, the key difference is that NNMF solves the constrained problem of forcing both coefficients and basis vectors to be strictly positive. For many data types, negative values (word frequencies, pixel values) are unnatural, and so the results of NNMF are immediately more interpretable. NNMF is promising in particular in situations where a signal arises from the addition of a number of positive signals. A canonical example is hyperspectral imaging, where, assuming incoherent light, the measured spectrum is the linear sum of the spectra of the individual light sources. If all goes well, the basis vectors from NNMF of a hyperspectral image will be the source spectra. In the context of natural language processing, NNMF on a collection of documents can be interpreted as \textit{topics}. Figure \ref{fig:nnmf_eigenfaces} shows the NNMF-based decomposition and approximation of the "labeled faces in the wild" dataset, analogous to the SVD of the dataset shown in Figure \ref{fig:svd_eigenfaces}. The two are quite different. While for SVD, the faces look as if they are gradually coming into focus, for NNMF the faces are at first not recognizable. The portraits also become brighter in the plot as more basis components are added, because, in contrast to the SVD case, the composition is purely additive.

The drawback is that NNMF is computationally more difficult (possibly NP hard), nor is the matrix approximation optimal, nor is the decomposition unique.

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{nnmf_eigenfaces.png}
    \caption{Left: The first 34 basis faces stored in the $\mathbf{W}$ matrix extracted using NNMF under Frobenius norm from the "Labeled Faces in the Wild" Dataset. The basis images are normalized and shown on a log scale because they have very different contrast and brightness. The normalized images are then all shown on the same color scale. Right: Five sample portraits from the dataset approximated using different numbers of basis faces. The NNMF was done for a value of $r=300$. The original portraits had $2914$ pixels. In contrast to SVD, the basis vectors obtained through NNMF do not necessarily have an internal ordering in terms of how much of the variance in the dataset they explain.}
    \label{fig:nnmf_eigenfaces}
\end{figure}

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{nnmf_eigenfaces2.png}
    \caption{Left: The first 6 basis faces stored in the $\mathbf{W}$ matrix extracted using NNMF under Frobenius norm from the "Labeled Faces in the Wild" Dataset. Right: Sample images expressed in the reduced basis.}
    \label{fig:nnmf_eigenfaces2}
\end{figure}


\section{Basic Artificial Neural Network}
Artificial neural networks (ANNs) are biologically inspired computer programs designed to simulate the way in which the human brain processes information.
%This model is just a model for a class of parameterized function with some special function structure which can be presented simply using some simple network graph. This method is began from the approximation of neural network in human.
\subsection{basic element: Nonlinear active function with affine map}
Mc Culloch-Pitts Neuron, also known as M-P Neuron, is the earliest neural network that was discovered in 1943. In this model, the neurons are connected by connection weights, and the activation function is used in binary. The threshold is used to determine whether the neuron will fire or not.

M-P element(approximation of one neuron): this is a very simple example for interpolating a function $f: \mathbb{R}^m \to \mathbb{R}$, with the next definition:
\begin{equation}\label{eq:M-P}
f_{M-P}(x) = \sigma( w \cdot x + b)
\end{equation}
with $w  \in \mathbb{R}^{m} $ and $\sigma$ is called active function which can be chosen like:
\begin{equation}
\sigma (x) = \frac{1}{1 + e^{-x}},
\end{equation}
or
\begin{equation}
\sigma(x) = tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}.
\end{equation}
Basically, both these two functions are smooth and have two horizontal asymptotic lines, which can be seen as a smoothing approximation for
\begin{equation}
H(x) =
\begin{cases}
1  &\text{if}  ~x < 0, \\
0 &\text{if}~ x \le 0.
\end{cases}
\end{equation}
But now, the most commonly used activation function is the so-called ``Rectified Linear Unit'' (ReLU), which is defined by
\begin{equation}
{\rm ReLU}(x) = \max\{0,x\}.
\end{equation}
%We will talk about some basic stuff about variant of activation function in the next chapter and
It is also very important in the approximation theory for such as one hidden layer ANN networks.
This is often shown as the next picture:
\begin{figure}[!ht]
	\center{\includegraphics[width=10cm] {M-P.jpeg}}
	\caption{M-P neuron.}
\end{figure}

\subsection{neural network}
Now we want to use this simple basic element to construct some more complex model(from one neuron to neural network). A bionic but simple construction is to increase the basic element in both horizontal and vertical direction, which means the network would be like the one in Fig~\ref{fig:ANN}.
\begin{figure}[!ht]
	\center{\includegraphics[width=12cm,height=6cm] {ANN.png}}
	\caption{ANN}
	\label{fig:ANN}
\end{figure}

This is called fully connected feedforward neural network. Now we want to give a more compressive expression, we collect all the output in $k$-th level $[f^{k}]_l$, $l = 1,\cdots, n_k$.
So we can have the output in $k+1$-th level under this setting with
\begin{equation}
{f}^{k+1} = \theta^{k+1}\circ \sigma (f^{k})
\end{equation}
with
\begin{equation}
\theta^k(x) = w^kx+b \in\mathbb{R}^{n_{k+1}},
\end{equation}
where
\begin{equation}
w^k=
\begin{pmatrix}
w_1 \\
\vdots \\
w_{n_{k+1}}
\end{pmatrix},\qquad \mbox{ with }\qquad w_i \in \mathbb{R}^{n_k}.
\end{equation}

So, we get the final iterative definition with the last output and initial input of this network like:
\begin{equation}
\begin{cases}
f^{0} &= \theta^0(x), \\
f^{\ell} &= \theta^{\ell}\circ \sigma(f^{\ell-1}), \quad \ell = 1:J, \\
f(x;\Theta) &= f^J.
\end{cases}
\end{equation}
Here we note
\begin{equation}
\Theta = \{ (\theta^0, \cdots, \theta^J) \}.
\end{equation}

%\subsection{Interpolation by optimization}
%So the  interpolation process can be seen as a optimization problem in the special function class if the activation function for every elements and  the network structure parameter $\mathcal{N}_K$ are given:
%\begin{equation}
%\mathcal{F} = \{\bm{f}_K(W_K;\bm{x}) ~|~  \bm{f}_0 = (\bm{x}^{T},1)^{T}\},
%\end{equation}
%and the optimization problem can be seen as
%\begin{equation}
%%\mathop{\min}_{f \in \mathcal{F}}  \frac{1}{N}\sum_{i=1}^N \|f(X^i) - Y^i\|^2 =
%\mathop{\min}_{W_K \in T_3(\mathcal{N}_K)}  L(W_K) := \frac{1}{N}\sum_{i=1}^N \|\bm{f}_K(W_K; X^i) - Y^i\|^2.
%\end{equation}
%But $\mathcal{F}$ this is neither a linear space nor a convex set.

%\iffalse
\subsection{Back-Propagation}
Here we will talk about how to compute $\nabla_{\Theta} f(x;\Theta)$ by using chain rule which is called Back-Propagation (BP) algorithm in deep learning.
Thus we have
\begin{equation}
\frac{\partial {f}(x; \Theta)}{ \partial w^k} = \frac{\partial {f}^J}{\partial {f}^{J-1}} \cdot \frac{ \partial {f}^{J-1}}{\partial {f}^{J-2}} \cdots \frac{ \partial {f}^k}{\partial w^k}.
\end{equation}
We can see from the above that, we only need to compute those terms like:
\begin{equation}
\frac{ \partial {f}^k}{\partial w^{k}}  \quad \text{and} \quad \frac{\partial {f}^k}{\partial {f}^{k-1}}  \quad k = 1,2,\cdots,J.
\end{equation}
We have
\begin{equation}\label{eq:partial-f}
\frac{\partial {f}^k}{\partial {f}^{k-1}} =
w^k{\rm Diag}(\sigma'(f^{k-1})) ,
\end{equation}
and
\begin{equation}
\frac{ \partial {f}^k}{\partial w^{k}}  = \delta \otimes \sigma(f^{k-1}).
\end{equation}

In short, BP algorithm can be expressed as:
\begin{algorithm}[H]
	\begin{algorithmic}[1]
		\State {\bf{Input:}}  $X^i$ and $W_K$;
		\For{$k = K:-1:1$}
		\State Compute and save
		$$\frac{ \partial {f}^k}{\partial w^{k}}  \quad \text{and} \quad \frac{\partial {f}^k}{\partial {f}^{k-1}}$$.
		\State Compute
		$$\frac{ \partial {f}^J}{\partial w^{k}} = \frac{\partial {f}^J}{\partial {f}^{J-1}} \cdot \frac{ \partial {f}^{J-1}}{\partial {f}^{J-2}} \cdots \frac{ \partial {f}^k}{\partial w^{k}}$$
		\EndFor
	\end{algorithmic}
	\caption{Back-Propagation Algorithm}
\end{algorithm}

%\input{6DL/juncai_dnn}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Multiple Hypothesis Testing}

When you perform a hypothesis test, it has some probability, $\alpha$, of rejecting the null when, in fact, the null is true. For one test this is manageable. However, when you perform multiple tests, the chance quickly becomes very high that you will reject the null at least once when it's true (a false positive). If you test a lot of hypotheses and don't correct your significance level to account for that, you're likely going to end up with many, many false positives.

\subsection{An Experiment}

Imagine you take random samples of size $40$ in one of two ways, decided by the flip of a [fair] coin. You either (heads) take two groups of $20$ samples each, both from the green distribution (below) or you take one group of $20$ from the green distribution and the other $20$ from the black [dashed] distribution. The difference in means between the two distributions here is 0.5.

\begin{center}
\includegraphics[width=0.6\textwidth]{img/multiple-hypothesis-example-2a.png}
\end{center}

Imagine doing this $10000$ times and, for each run, performing a $T$-test between the two samples. If the $p$-value for that test is below the significance level, $\alpha$, reject the null. Now, measure the fraction of simulations for which the null is rejected and both samples were taken from the same distribution (false positives) and divide that by the total number of simulations where the null was rejected (true positives + false positives). That quotient is the \textbf{False Discovery Rate (FDR)}. Here's how it varies in this experiment with respect to $\alpha$ and effect size:

\begin{center}
\includegraphics[width=0.6\textwidth]{img/multiple-hypothesis-example-1.png}
\end{center}

\begin{mdframed}
\textbf{Question 2.9:} If the FDR decreases monotonically with decreasing significance level, why not just set $\alpha = 0.0000000001$ or something to minimize FDR?
\vspace{20mm}
\end{mdframed}

There are several algorithms that have been developed and proven to control FDR even though it's impossible to know what the true underlying distribution of effect sizes is. One is called the \textbf{Benjamini-Hochberg} method. Look that up and compare it to the \textbf{Bonferroni correction}, which controls - not the FDR - but the FWER (the probability that at least one significant result is a false positive).


\section{Problem Formulation}
\label{sec:problem_formulation}

Given a model $\mathcal{M}$ parameterized by $\bm{\omega}$ and a dataset $\mathcal{D} = \{(\mathbf{x}_i,\mathbf{y}_i)\}_{i=1}^{\mathbb{N}}$, the inference goal is to determine the posterior distribution $p(\bm{\omega} \mid \mathbf{X}, \mathbf{Y})$ through Bayes' rule. Since this posterior is generally intractable we consider the parameterized variational distribution $q(\bm{\omega};\bm{\theta})$ that is optimal for $\bm{\theta}^*$ in terms of KL Divergence.

$$
\bm{\theta}^* = \argmin_{\bm{\theta}\in\Theta} KL\left[q(\bm{\omega};\bm{\theta})||p(\bm{\omega} \mid \mathbf{X}, \mathbf{Y})\right]
$$

Given a prior, $p(\bm{\omega})$, we can can reformulate this as the evidence lower bound (ELBO):

\begin{equation}
\bm{\theta}^* = \argmax_{\bm{\theta}\in\Theta}\left\{\mathbb{E}_{\bm{\omega}\sim q}\big[\log p(\mathbf{Y} \mid \mathbf{X}, \bm{\omega})\big] - KL\big[q(\bm{\omega};\bm{\theta} || p(\bm{\omega})\big]\right\}
\end{equation}

The goal of this paper is to derive an explicit deterministic approximation of the first term, known as reconstruction term, and choose priors $p(\bm{\omega})$ empirically to increase robustness to the choice of variance parameters.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 2}

\begin{enumerate}[label=(\alph*)]
\item A researcher claimed that she discovered a comparison-based sorting algorithm that runs in $\mathcal{O}(n\log (\sqrt{n}))$.
Given the existence of an $\Omega(n\log n)$ lower bound for sorting, how can this be possible?

\item Let $S$ be an unsorted array of $n$ integers.
Give an algorithm that finds the pair $x,y \in S$ that maximizes $|x - y|$.
Your algorithm must run in $\mathcal{O}(n)$ worst-case time.

\item Explain why your algorithm is indeed $\mathcal{O}(n)$.
\end{enumerate}

\subsection*{Solution}

\begin{enumerate}[label=(\alph*)]
\item It suffices to show that a comparison-based sorting algorithm with runtime $\mathcal{O}(n\log \sqrt{n})$ does not necessarily violate the lower-bound runtime of sorting algorithms, in which case the researcher's claim will be justified.
Using fundamental properties of logarithms,
\begin{equation}\label{eq31}
\begin{aligned}
\mathcal{O}(n\log \sqrt{n}) &= \mathcal{O}(n\log n^{0.5})\\
&= \mathcal{O}(0.5 n \log n)\\
&= \mathcal{O}(n\log n)
\end{aligned}
\end{equation}
Therefore, researcher's proposed sorting algorithm is of the same lower-bound run-time and the claim is justified.
\item Expression $|x-y|$ is maximized when $x$ and $y$ are minimum and maximum elements of the unsorted array.
The problem could now be simplified as finding minimum and maximum of an unsorted array in $\mathcal{O}(n)$ runtime.
To achieve this goal, Algorithm \ref{alg1} shown below is proposed.
\begin{algorithm}[H]
\caption{\textsc{$|x-y|$-Maximizer}}\label{alg1}
\begin{algorithmic}[1]
\State $min \leftarrow +\infty$
\State $max \leftarrow -\infty$
\For {$i \leftarrow 1$ to $A.length$}
\If {$A[i] < min$}
\State min $\leftarrow A[i]$
\ElsIf {$A[i] > max$}
\State max $\leftarrow A[i]$
\EndIf
\EndFor
\State \Return $min$, $max$
\end{algorithmic}
\end{algorithm}
\item There are $n$ iterations of the for loop in each of which either $A[i] < min$ or $A[i] > max$ or none.
In first two cases there would be one assignment of cost $c$.
Therefore, the whole algorithm has a run time of $\mathcal{O}(nc+d)=\mathcal{O}(n)$, so algorithm is linear in $n$.
\end{enumerate}


\section{ICP Math}

Assume we have a model point cloud with points $p_i$ and surface
normals $n_i$ and we want to align the currently sensed point cloud
with points $q_i$ to the model by estimating the relative
transformation $(R_{mc}, t_{mc})$. The ICP point-to-plane distance
error function is given as:
\begin{align}
  \sum_i \| ( R_{mc} q_i + t_{mc} - p_i )^T n_i \|^2_2
\end{align}
Note that this already assumes a data association between model and current point cloud.
This association can be obtained efficiently in the case of RGBD frames
by projecting the current point cloud into the model camera frame and
forming the association by the pixel positions of the projected current point cloud.

\subsection{Solution using small angle assumption}
Locally, around the current relative transformation estimate $(R_{mc},
t_{mc})$ the cost function can be approximated by small perturbations $\xi$
in the tangent space $\se{3}$ around the current pose:
\begin{align}
  \sum_i \| ( R_{mc} \exp(\xi_w) q_i + t_{mc} + \xi_t - p_i )^T n_i \|^2_2
\end{align}

Slide 13 of
\url{{http://resources.mpi-inf.mpg.de/deformableShapeMatching/EG2011_Tutorial/slides/2.1%20Rigid%20ICP.pdf}} and
\url{https://www.comp.nus.edu.sg/~lowkl/publications/lowk_point-to-plane_icp_techrep.pdf}
show how to linearize this system assuming small rotations:
\begin{align}
  &\sum_i \| (  \exp(\xi_w) R_{mc} q_i + t_{mc} + \xi_t - p_i )^T n_i \|^2_2 \\
  =& \sum_i \| ( (I + [\xi_w]_\times) R_{mc} q_i + t_{mc} + \xi_t - p_i )^T n_i \|^2_2\\
  =& \sum_i \| (  [\xi_w]_\times R_{mc} q_i + R_{mc} q_i + t_{mc} + \xi_t - p_i )^T n_i \|^2_2\\
  =& \sum_i \| n_i^T (\xi_w \times (R_{mc} q_i)) + n_i^T R_{mc} q_i + n_i^T t_{mc} + n_i^T \xi_t -n_i^T p_i \|^2_2\\
  =& \sum_i \|  ( (R_{mc} q_i) \times n_i)^T \xi_w + n_i^T \xi_t + n_i^T ( R_{mc} q_i + t_{mc} - p_i) \|^2_2\\
  =& \|  A \xi - b \|^2_2  \,,\quad
  A = \begin{pmatrix}
    ((R_{mc} q_i) \times n_i)^T &  n_i^T \\
    \vdots & \vdots
  \end{pmatrix}\,,\;
  \xi = \begin{pmatrix} \xi_w \\ \xi_t \end{pmatrix}\,,\;
  b = \begin{pmatrix}
    n_i^T ( - R_{mc} q_i - t_{mc} + p_i) \\
    \vdots
  \end{pmatrix}
\end{align}
where we have used that the $\SO{3}$ exponential maps first order
expansion is $I + [\xi_w]_\times$ for small angular magnitude.

This can be solved in the standard approach using the pseudo inverse $A^TA$:
\begin{align}
  \xi = (A^TA)^{-1} A^Tb
\end{align}
For efficiency reasons $A$ and $b$ should never be constructed
directly. Instead we directly accumulate $A^TA$ and $A^Tb$:
\begin{align}
  A^TA = \begin{pmatrix}
    \sum_i A_{i0}^2 & \sum_i A_{i0} A_{i1} & \cdots \\
    \sum_i A_{i0} A_{i1} & \sum_i A_{i1}^2 & \cdots \\
      \vdots & \vdots & \ddots
  \end{pmatrix}
  &
  A^Tb = \begin{pmatrix}
    \sum_i A_{i0} b_{i} \\
    \sum_i A_{i1} b_{i} \\
      \vdots
  \end{pmatrix}
\end{align}
Due to symmetry we only need to accumulate $21$ values ($22$ if we want
the cost function value as well).

\subsection{Alternative using gradient descent}

The optimum perturbation $\xi$ to minimize the cost function around the
current pose estimate can be found by gradient descent in the direction of derivative with
respect to $\xi$:
\begin{align}
  & \deriv{}{\xi} \sum_i \| ( R_{mc} \exp(\xi_w) q_i + t_{mc} + \xi_t - p_i )^T n_i \|^2_2\\
  =&  \sum_i 2 ( R_{mc} \exp(\xi_w) q_i + t_{mc} + \xi_t - p_i )^T n_i  \deriv{}{\xi}
  \left[ n_i^T R_{mc} \exp(\xi_w) q_i   + n_i^T \xi_t
  \right]\\
  =&  \sum_i 2 ( R_{mc} \exp(\xi_w) q_i + t_{mc} + \xi_t - p_i )^T n_i
  \begin{pmatrix}
    - n_i^T [R_{mc} q_i]_\times  & n_i^T
  \end{pmatrix}
\end{align}


\section*{Exercises}

\begin{ex}
  Which of the following linear transformations $T:\R^3\to\R^3$ are
  orthogonal?
  \begin{enumerate}
  \item A rotation by 30 degrees about the $z$-axis.
  \item A reflection about the plane $x=y$.
  \item A scaling by a factor of $2$.
  \item A scaling by a factor of $-1$.
  \end{enumerate}
  \begin{sol}
    (a) Yes. (b) Yes. (c) No. (d) Yes.
  \end{sol}
\end{ex}

\begin{ex}
  Which of the following matrices define orthogonal transformations?
  Which ones define isometries?
  \begin{equation*}
    A = \begin{mymatrix}{rr}
      1 &  0 \\
      0 & -1 \\
    \end{mymatrix},\quad
    B = \frac{1}{2}\begin{mymatrix}{rr}
      1 & 1 \\
      -1 & 1 \\
    \end{mymatrix},\quad
    C = \frac{1}{\sqrt{5}}\begin{mymatrix}{rr}
      1 & -2 \\
      2 &  1 \\
    \end{mymatrix},\quad
    D = \frac{1}{4}\begin{mymatrix}{rr}
      1 &  2 \\
      2 &  2 \\
      2 & -1 \\
    \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    $A$ and $C$ are orthogonal (and therefore isometries). $D$ is an
    isometry but not orthogonal. $B$ is neither orthogonal nor an isometry.
  \end{sol}
\end{ex}

\begin{ex}
  Determine which of the following matrices are orthogonal by checking
  whether the columns form an orthonormal set.
  \begin{equation*}
    A = \frac{1}{3}
    \begin{mymatrix}{rrr}
      1  &  2 &  2 \\
      2  &  1 & -2 \\
      -2 &  2 & -1 \\
    \end{mymatrix},
    \quad
    B = \begin{mymatrix}{rrr}
      0 & 0 & 2 \\
      0 & 2 & 0 \\
      2 & 0 & 0 \\
    \end{mymatrix},
    \quad
    C = \frac{1}{3}
    \begin{mymatrix}{rrr}
      2  & -1 &  2 \\
      -1 &  2 & -2 \\
      2  & -2 &  1 \\
    \end{mymatrix},
    \quad
    D =
    \frac{1}{5}
    \begin{mymatrix}{rrr}
      3 &  4 & 0 \\
      4 & -3 & 0 \\
      0 &  0 & 5 \\
    \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    $A$ and $D$ are orthogonal, but $B$ and $C$ are not.
  \end{sol}
\end{ex}


\subsubsection{Method of Lagrange Multipliers}
\noindent
Given an objective function, $f(x,y)$ and a constraint equation $g(x,y) = k$, define $F(x,y,\lambda) = f(x,y) + \lambda(k-g(x,y))$.
The solution, $(x, y, \lambda)$, to $\nabla F = \vec{0}$ will be the solution to the constrained optimization problem.\\

\noindent
For example, let's maximize $f(x,y)=xy$ subject to $(x-1)^2 + (y-1)^2 = 1$.
\begin{equation*}
	F(x,y,\lambda) = xy + \lambda(1 - (x-1)^2 - (y-1)^2)
\end{equation*}
\begin{equation*}
	\nabla F = \langle y + 2\lambda(x-1), x+2\lambda(y-1), 1 - (x-1)^2 - (y-1)^2 = 1 \rangle = \vec{0}
\end{equation*}
\begin{equation*}
	\begin{cases}
		y - 2\lambda(x-1) = 0 \\
		x - 2\lambda(y-1) = 0 \\
		1 - (x-1)^2 - (y-1)^2 = 0
	\end{cases}
	\implies \begin{cases}
		y = 2\lambda(x-1) \\
		x = 2\lambda(y-1) \\
		(x-1)^2 + (y-1)^2 = 1
	\end{cases}
\end{equation*}
\begin{equation*}
	y = 2\lambda(2\lambda(y-1) - 1) = 4\lambda^2 y - 4\lambda^2 - 2\lambda = \frac{2\lambda}{2\lambda+1}
\end{equation*}
\begin{equation*}
	x = 2\lambda\left(\frac{2\lambda}{2\lambda+1} - 1\right) = \frac{2\lambda}{2\lambda+1} \implies x = y
\end{equation*}
\begin{equation*}
	2(x-1)^2 = 1 \implies x = y = 1 \pm \frac{1}{\sqrt{2}}
\end{equation*}
\begin{equation*}
	\implies \text{ Max/Min of } \frac{3 \pm 2\sqrt{2}}{2}.
\end{equation*}


\subsection{Calculating eigenvalues and eigenvectors using the characteristic polynomial}

The characteristic polynomial of a matrix is a polynomial whose roots are the eigenvalues of the matrix.

We know from the definition of eigenvalues and eigenvectors that:

\(Av=\lambda v\)

Note that

\(Av-\lambda v=0\)

\(Av-\lambda Iv=0\)

\((A-\lambda I)v=0\)

Trivially we see that \(v=0\) is a solution.

Otherwise matrix \(A-\lambda I\) must be non-invertible. That is:

\(Det(A-\lambda I)=0\)

\subsection{Calculating eigenvalues}

For example

\(A=\begin{bmatrix}2&1\\1 & 2\end{bmatrix}\)

\(A-\lambda I=\begin{bmatrix}2-\lambda &1\\1 & 2-\lambda \end{bmatrix}\)

\(Det(A-\lambda I)=(2-\lambda )(2-\lambda )-1\)

When this is \(0\).

\((2-\lambda )(2-\lambda )-1=0\)

\(\lambda =1,3\)

\subsection{Calculating eigenvectors}

You can plug this into the original problem.

For example

\(Av=3v\)

\(\begin{bmatrix}2&1\\1 & 2\end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix}=3\begin{bmatrix}x_1\\x_2\end{bmatrix}\)

As vectors can be defined at any point on the line, we normalise \(x_1=1\).

\(\begin{bmatrix}2&1\\1 & 2\end{bmatrix}\begin{bmatrix}1\\x_2\end{bmatrix}=\begin{bmatrix}3\\3x_2\end{bmatrix}\)

Here \(x_2=1\) and so the eigenvector corresponding to eigenvalue \(3\) is:

\(\begin{bmatrix}1\\1\end{bmatrix}\)


\section{Probability and Statistics}
\subsection{Basics}
The set of all possible outcomes of an experiment is known as the sample space
and denoted by $S$. Any subset $E$ of the sample space is known as an event.
An event is a set consisting of possible outcomes of the experiment.
\subsubsection{Axioms of Probability}
For each event $E$, we denote $P(E)$ as the probability of event $E$ occuring.
$P(E)$ satisfies the following properties:
\begin{enumerate}
  \item Every probability is between 0 and 1 included:
    \begin{equation} 0 \leq P(E) \leq 1 \end{equation}
  \item The probability that at least one event in the sample space will occur
  is $1$: \begin{equation} P(S) = 1 \end{equation}
  \item For any sequence of mutually exclusive events $E_1,\hdots,E_n$ we have:
    \begin{equation} P\left(\bigcup_{i=1}^nE_i\right)=\sum_{i=1}^nP(E_i) \end{equation}
\end{enumerate}
\subsubsection{Permutation}
A permutation is an arrangement of $r$ objects from a pool of $n$ objects, in
a given order. The number of such arrangements is given by $P(n, r)$:
\begin{equation} P(n, r) = \frac{n!}{(n-r)!} \end{equation}
\subsubsection{Combination}
A combination is an arrangement of $r$ objects from a pool of $n$ objects, where
order does not matter. The number of such arrangements is given by $C(n, r)$:
\begin{equation} C(n, r) = \frac{P(n, r)}{r!} = \frac{n!}{r! (n-r)!} \end{equation}
Note that for $0 \leq r \leq n$ we have $P(n, r) \geq C(n, r)$.
\subsection{Conditional Probability}
Let $B$ be an event with non-zero probability, the conditional proability of
any event $A$ given $B$ is:
\begin{equation} P(A | B) = \frac{P(A \cap B)}{P(B)} \end{equation}
\subsubsection{Bayes Rule}
For events $A, B$ such that $P(B) > 0$, we have:
\begin{equation} P(A | B) = \frac{P(B|A) P(A)}{P(B)} \end{equation}
From Bayes rule, we have:
\begin{equation} P(A \cap B) = P(A|B)P(B) = P(A)P(B|A) \end{equation}
Let $\{ A_i, i \in [1, n] \}$ be such that for all $i$, $A_i \not= \varnothing$.
We say that $\{A_i\}$ is a partition if we have:
\begin{equation} \forall i\neq j, A_i\cap A_j=\emptyset\quad\mbox{ and }\quad\bigcup_{i=1}^nA_i=S \end{equation}
Remark that for any event $B$ in the sample space, we have:
\begin{equation} P(B)=\sum_{i=1}^n P(B|A_i)P(A_i) \end{equation}
Let $\{ A_i, i \in [1, n] \}$ be a partition of the sample space, we can
extend bayes rule as:
\begin{equation} P(A_k|B)=\frac{P(B|A_k)P(A_k)}{\displaystyle\sum_{i=1}^nP(B|A_i)P(A_i)} \end{equation}
\subsubsection{Independence}
Two events $A$ and $B$ are independent if and only if we have:
\begin{equation} P(A \cap B) = P(A) P(B) \end{equation}
\subsection{Random Variables}
A random variable $X$ is a function that maps every element in a sample space
to a real line.
\subsubsection{Cumulative Distribution Function (CDF)}
The cumulative distribution function $F$, which is monotonically non-decreasing
and is such that $\lim_{x \to -\infty}F(X) = 0$ and $\lim_{x \to \infty}F(X) = 1$
is defined as:
\begin{equation} F(x) = P(X \leq x) \end{equation}
In addition:
\begin{equation} P(a < X \leq b) = F(b) - F(a) \end{equation}
\subsubsection{Probability Density Function (PDF)}
The probability density function is the derivative of the CDF.
It has the following properties:
\begin{enumerate}
  \item $f(x) \geq 0$
  \item $\int^{\infty}_{-\infty} f(x) = 1$
  \item $\int_{x \in A} f(x) dx = P(X \in A)$
\end{enumerate}
\subsubsection{Discrete PDF/CDF}
If $X$ is discrete, by denoting $f$ as the PDF and $F$ as the CDF, we have:
\begin{equation}
  F(X) = \sum_{x_i \leq x} P(X = x_i)
\end{equation}
\begin{equation}
  f(x_j) = P(X = x_j)
\end{equation}
And the following properties for the PDF:
\begin{equation}
  0 \leq f(x_j) \leq 1
\end{equation}
\begin{equation}
  \sum_j f(x_j) = 1
\end{equation}
\subsubsection{Continuous PDF/CDF}
If $X$ is continuous, by denoting $f$ as the PDF and $F$ as the CDF, we have:
\begin{equation}
  F(X) = \int_{-\infty}^x f(y) dy
\end{equation}
\begin{equation}
  f(x) = \frac{dF}{dx}
\end{equation}
And the following properties for the PDF:
\begin{equation}
  f(x) \geq 0
\end{equation}
\begin{equation}
  \int_{-\infty}^{\infty} f(x) dx = 1
\end{equation}
\subsubsection{Expectation}
The expected value of a random variable, also known as the mean value or
first moment, is denoted as $\mathbb { E }[X]$ or $\mu$. It is the value obtained by
averaging the results of a random variable.
We use $\mbox{(D)}$ for discrete, $\mbox{(C)}$ for continuous.
\begin{equation}
  \mbox{(D)}\quad \mathbb { E }[X]=\sum_{i=1}^nx_if(x_i)\quad\quad\mbox{and}\quad\mbox{(C)}\quad \mathbb { E }[X]=\int_{-\infty}^{+\infty}xf(x)dx
\end{equation}
The expected value of a function of a random variable $g(X)$ is:
\begin{equation}
  \mbox{(D)}\quad \mathbb { E }[g(X)]=\sum_{i=1}^ng(x_i)f(x_i)\quad\quad\mbox{and}\quad\mbox{(C)}\quad \mathbb { E }[g(X)]=\int_{-\infty}^{+\infty}g(x)f(x)dx
\end{equation}
The $k$-th moment, noted $\mathbb { E }[X^k]$ is the value of $X^k$ that we expect to observe
on average on infinitely many trials. The $k$-th moment is a case of the previous
definition with $g \,:\, X \mapsto X^k$.
\begin{equation}
  \mbox{(D)}\quad \mathbb { E }[X^k]=\sum_{i=1}^nx_i^kf(x_i) \quad\quad\mbox{and}\quad\mbox{(C)}\quad \mathbb { E }[X^k]=\int_{-\infty}^{+\infty}x^kf(x)dx
\end{equation}
A characteristic function $\psi(\omega)$ is derived from a probability density
function $f(x)$ and is defined as:
\begin{equation}
  \mbox{(D)}\quad \psi(\omega)=\sum_{i=1}^nf(x_i)e^{i\omega x_i} \quad\quad\mbox{and}\quad\mbox{(C)}\quad \psi(\omega)=\int_{-\infty}^{+\infty}f(x)e^{i\omega x}dx
\end{equation}
Remark that $e^{i\omega x} = \cos(\omega x)+ i \sin(\omega x)$. The $k$-th moment
can also be computed with the characteristic function as:
\begin{equation}
  \mathbb { E }[X^k]=\frac{1}{i^k}\left[\frac{\partial^k\psi}{\partial\omega^k}\right]_{\omega=0}
\end{equation}
\subsubsection{Variance and Standard Deviation}
The variance of a random variable, often noted $\mbox{Var}(X)$ or $\sigma^2$,
is a measure of the spread of its distribution function.
It is determined as follows:
\begin{equation}
  \mbox{Var}[X]=\mathbb { E }[(X-\mathbb { E }[X])^2]=\mathbb { E }[X^2]-\mathbb { E }[X]^2
\end{equation}
The standard deviation of a random variable, often noted $\sigma$, is a
measure of the spread of its distribution function which is
compatible with the units of the actual random variable.
It is determined as follows:
\begin{equation}
  \sigma = \sqrt{\mbox{Var}[X]}
\end{equation}
Note that the variance for any constant $a$ is $\mbox{Var}[a]= 0$, and
$\mbox{Var}[af(X)]=a^2\mbox{Var}[f(X)]$.
\subsection{Discrete Random Variables}
\subsubsection{Bernoulli}
For $X\sim Bernoulli(p)$, we define a binary event with a probability of $p$
for a true event, and a false event with probability of $q=1-p$.
\begin{equation}
  P(X=x) = \begin{cases} q=1-p & \textrm{if } x=0 \\ p & \textrm{if } x=1 \end{cases}
\end{equation}
It can also be expressed as:
\begin{equation}
  f(x; p) = p^k (1-p)^{1-k} \quad \mbox{for } k\in \{0, 1\}
\end{equation}
Other properties:
\begin{equation}
  \mathbb { E }[X] = p
\end{equation}
\begin{equation}
  \mbox{Var}[X] = pq = p (1-p)
\end{equation}
\begin{equation}
  \psi(\omega) = (1-p) + p e^{i\omega}
\end{equation}
\subsubsection{Binomial}
For $X\sim Binomial(n, p)$, the number of true events in $n$ independent
experiments, with true probability of $p$, false probability of $q=1-p$.
\begin{equation}
  \displaystyle P(X=x)=\displaystyle\binom{n}{x} p^x(1-p)^{n-x}
\end{equation}
Other properties:
\begin{equation}
  \mathbb { E }[X] = np
\end{equation}
\begin{equation}
  \mbox{Var}[X] = npq = np(1-p)
\end{equation}
\begin{equation}
  \psi(\omega) = (pe^{i\omega}+(1-p))^n
\end{equation}
\subsubsection{Geometric}
For $X\sim Geometric(p)$, is the number of experiments with true
probability of $p$ until the first true event (number of trials to get one success).
\begin{equation}
  \displaystyle P(X=x)=p(1-p)^{x-1}
\end{equation}
Other properties:
\begin{equation}
  \mathbb { E }[X] = \frac{1}{p}
\end{equation}
\begin{equation}
  \mbox{Var}[X] = \frac{1-p}{p^2}
\end{equation}
\begin{equation}
  \psi(\omega) = \frac{pe^{i \omega}}{1-(1-p)e^{i \omega}}
\end{equation}
\subsubsection{Poisson}
For $X\sim Poisson(\lambda)$, for $\lambda > 0$, a probability distribution
over the nonnegative integers
used for modeling the frequency of rare events.
\begin{equation}
  \displaystyle P(X=x)=\frac{\lambda^x}{x!}e^{-\lambda}
\end{equation}
Other properties:
\begin{equation}
  \mathbb { E }[X] = \lambda
\end{equation}
\begin{equation}
  \mbox{Var}[X] = \lambda
\end{equation}
\begin{equation}
  \psi(\omega) = e^{\lambda(e^{i\omega}-1)}
\end{equation}
\subsection{Continuous Random Variables}
\subsubsection{Uniform}
For $X\sim Uniform(a,b)$, we have equal probability density to every value
between $a$ and $b$.
\begin{equation}
  f(x) = \begin{cases} \frac{1}{b-a} & \textrm{if } a \leq x \leq b \\ 0 & \textrm{otherwise} \end{cases}
\end{equation}
Other properties:
\begin{equation}
  \mathbb { E }[X] = \frac{a+b}{2}
\end{equation}
\begin{equation}
  \mbox{Var}[X] = \frac{(b-a)^2}{12}
\end{equation}
\begin{equation}
  \psi(\omega) = \displaystyle\frac{e^{i\omega b}-e^{i\omega a}}{(b-a)i\omega}
\end{equation}
\subsubsection{Exponential}
For $X\sim Exponential(\lambda)$, $\lambda > 0$, is the
decaying probability density over the nonnegative reals.
\begin{equation}
  f(x) = \begin{cases} \lambda e^{-\lambda x} & \textrm{if } x \geq 0 \\ 0 & \textrm{otherwise} \end{cases}
\end{equation}
Other properties:
\begin{equation}
  \mathbb { E }[X] = \frac{1}{\lambda}
\end{equation}
\begin{equation}
  \mbox{Var}[X] = \frac{1}{\lambda^2}
\end{equation}
\begin{equation}
  \psi(\omega) = \displaystyle\frac{1}{1-\frac{i\omega}{\lambda}}
\end{equation}
\subsubsection{Gaussian (Normal)}
For $X\sim Normal(\mu, \sigma)$, denoted also $X \sim \mathcal{N}(\mu, \sigma)$.
\begin{equation}
  \displaystyle f(x)=\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2}
\end{equation}
Other properties:
\begin{equation}
  \mathbb { E }[X] = \mu
\end{equation}
\begin{equation}
  \mbox{Var}[X] = \sigma^2
\end{equation}
\begin{equation}
  \psi(\omega) = e^{i\omega\mu-\frac{1}{2}\omega^2\sigma^2}
\end{equation}
\subsection{Jointly Distributed Random Variables}
The joint probability distribution of two random variables $X$ and $Y$, denoted
as $f_{XY}$ is defined as:
\begin{equation}
  \mbox{(D)}\quad f_{XY}(x_i,y_j)=P(X=x_i\mbox{ and }Y=y_j)
\end{equation}
\begin{equation}
  \mbox{(C)}\quad f_{XY}(x,y)\Delta x\Delta y=P(x\leqslant X\leqslant x+\Delta x\mbox{ and }y\leqslant Y\leqslant y+\Delta y)
\end{equation}
Again, denote $\mbox{(D)}$ as the discrete case, and $\mbox{(C)}$ as the
continuous case.
\subsubsection{Marginal Density}
The marginal density for a random variable $X$ is:
\begin{equation}
  \mbox{(D)}\quad f_X(x_i)=\sum_{j}f_{XY}(x_i,y_j) \quad\quad\mbox{and}\quad\mbox{(C)}\quad f_X(x)=\int_{-\infty}^{+\infty}f_{XY}(x,y)dy
\end{equation}
\subsubsection{Cumulative Distribution}
The cumulative distribution $F_{XY}$ is:
\begin{equation}
  \mbox{(D)}\quad F_{XY}(x,y)=\sum_{x_i\leqslant x}\sum_{y_j\leqslant y}f_{XY}(x_i,y_j) \quad\quad\mbox{and}\quad\mbox{(C)}\quad F_{XY}(x,y)=\int_{-\infty}^x\int_{-\infty}^yf_{XY}(x',y')dx'dy'
\end{equation}
\subsubsection{Conditional Density}
The conditional density of $X$ with respect to $Y$, denoted $f_{X|Y}$ is
defined as:
\begin{equation}
  f_{X|Y} = \frac{f_{XY}(x, y)}{f_Y(y)}
\end{equation}
\subsubsection{Independence}
Two random variables $X$ and $Y$ are independent if:
\begin{equation}
  f_{XY}(x,y) = f_X(x)f_Y(y)
\end{equation}
\subsubsection{Expectation}
Given two random variables $X,Y$ and $g : \mathbb{R}^2 \to \mathbb{R}$ is a
function of these two variables. Then the expected value of $g$ is:
\begin{equation}
  \mbox{(D)}\quad\mathbb{E}[g(X, Y)] = \sum_i \sum_j g(x_i, y_i) f(x_i, y_i) \quad\quad\mbox{and}\quad\mbox{(C)}\quad\mathbb{E}[g(X, Y)] = \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} g(x_i, y_i) f(x_i, y_i) dy dx
\end{equation}
\subsubsection{Covariance}
The covariance of two random variables $X$ and $Y$, denoted $\sigma^2_{XY}$ or
as $\mbox{Cov}[X, Y]$ is defined as:
\begin{equation}
  \mbox{Cov}[X,Y]\triangleq\sigma_{XY}^2=\mathbb{E}[(X-\mu_X)(Y-\mu_Y)]=\mathbb{E}[XY]-\mu_X\mu_Y
\end{equation}
Where $\mu_X = \mathbb{E}[X]$ and $\mu_Y = \mathbb{E}[Y]$ respectively. If $X$
and $Y$ are independent, the covariance is $0$.
\begin{equation}
  \mbox{Var}[X + Y] = \mbox{Var}[X] + \mbox{Var}[Y] + 2 \mbox{Cov}[X, Y]
\end{equation}
\subsubsection{Correlation}
By noting $\sigma_X$, $\sigma_Y$ as the standard deviations of $X$ and $Y$, we
define the correlation between the random variables $X$ and $Y$ as $\rho_{XY}$:
\begin{equation}
  \rho_{XY} = \frac{\sigma_{XY}^2}{\sigma_{X}\sigma_{Y}}
\end{equation}
Correlation for $X, Y$ is $\rho_{XY} \in [-1, 1]$. If $X,Y$ independent, then
$\rho_{XY}=0$
\subsection{Parameter Estimation}

\subsubsection{Definitions}
\subsubsection{Bias}
\subsubsection{Mean and Central Limit Theorem}
\subsubsection{Variance}

\subsection{Probability Bounds and Inequalities}
This section looks at various bounds that define how likely a random variable
is to be close to its expectation.
\subsubsection{Markov}
Let $X \geq 0$ be a non-negative random variable. Then for all $a \geq 0$:
\begin{equation}
  P(X \geq a) \leq \frac{\mathbb{E}[X]}{a}
\end{equation}
\subsubsection{Chebyshev}
Let $X$ be any random variable with finite expected value $\mu = \mathbb{E}[X]$
and finite non-zero variance $\sigma^2$.
Then for all $k > 0$:
\begin{equation}
  P(|X - \mathbb{E}[X]| \geq k \sigma) \leq \frac{1}{k^2}
\end{equation}
\subsubsection{Chernoff}
Recall that the moment generating function for a random variable $X$ is:
\begin{equation}
  M_X(\lambda) := \mathbb{E}[e^{\lambda X}]
\end{equation}
Then the chernoff bound for a random variable $X$, obtained by applying the
markov inequality to $e^{\lambda X}$, for every $\lambda > 0$:
\begin{equation}
  P(X \geq a) = P(e^{\lambda X} \geq e^{\lambda a}) \leq \frac{\mathbb{E}[e^{\lambda X}]}{e^{\lambda a}}
\end{equation}
For the multiplicative chernoff bound, suppose $X_1, \hdots, X_n$ are independent
random variables taking values in $\{0, 1\}$. Let $X$ denote the sum, $\mu=\mathbb{E}[X]$
denote the sum's expected value. Then for any $0 < \delta < 1$,
\begin{equation}
  P(X > (1+\delta) \mu ) < \left( \frac{e^{\delta}}{(1+\delta)^{(1+\delta)}} \right)^{\mu} \leq e^{-\frac{\delta^2 \mu}{3}}
\end{equation}
\begin{equation}
  P(X < (1-\delta) \mu ) < \left( \frac{e^{-\delta}}{(1-\delta)^{(1-\delta)}} \right)^{\mu} \leq e^{-\frac{\delta^2 \mu}{2}}
\end{equation}
For $\delta \geq 0$,
\begin{equation}
  P(X \geq (1+\delta) \mu ) \leq e^{-\frac{\delta^2 \mu}{2+\delta}}
\end{equation}
\begin{equation}
  P(X \leq (1-\delta) \mu ) \leq e^{-\frac{\delta^2 \mu}{2}}
\end{equation}
\subsubsection{Hoeffding}
Let $X_1, \hdots, X_n$ be independent random variables such that $a_i \leq X_i \leq b_i$.
The sum of these variables $S_n = \sum_i X_i$, then the Hoeffding
inequality is:
\begin{equation}
  P\left(S_n - \mathbb{E}\left[S_n\right] \geq t \right) \leq \exp\left(-\frac{2t^2}{\sum_i(b_i-a_i)^2}\right)
\end{equation}
\begin{equation}
  P\left(\left|S_n - \mathbb{E}\left[S_n\right]\right| \geq t \right) \leq 2\exp\left(-\frac{2t^2}{\sum_i(b_i-a_i)^2}\right)
\end{equation}
The hoeffding lemma states that for a real-valued random variable $X$ with expected
value $\mathbb{E}[X] = 0$ and such that $a \leq X \leq b$, then for all
$\lambda \in \mathbb{R}$ we have,
\begin{equation}
  \mathbb{E}[e^{\lambda X}] \leq \exp\left(\frac{\lambda^2(b-a)^2}{8}\right)
\end{equation}


\begin{solution}
\textbf{Part a) }The error
\begin{align*}
\left|\frac{f(x+h)-f(x)}{h}-\frac{fl(f(x+h))-fl(f(x))}{h}\right|&=\left|\frac{f(x+h)-f(x)}{h}-\frac{f(x+h)(1+\delta_1)-f(x)(1+\delta_2)}{h}\right|\\
&=\left|-\frac{f(x+h)\delta_1-f(x)\delta_2}{h}\right|\\
&\leq\left|\frac{f(x+h)\delta_1}{h}\right|+\left|\frac{f(x)\delta_2}{h}\right|\\
&<\left|f(x+h)\right|\frac{\epsilon}{h}+\left|f(x)\right|\frac{\epsilon}{h}\\
&=\left(\left|f(x)+hf'(x)+\mathcal{O}(h^2)\right|+\left|f(x)\right|\right)\frac{\epsilon}{h}\\
&\leq \left(2\left|f(x)\right|+\left|hf'(x)\right|+\left|\mathcal{O}(h^2)\right|\right)\frac{\epsilon}{h}\sim\mathcal{O}\left(\frac{\epsilon}{h}\right),
\end{align*}
is proportional to $\epsilon/h$ in the worst case as $h\rightarrow 0$.

\textbf{Part b)} We just proved that the rounding error is proportional to $\epsilon/h$, then
\begin{align*}
e_{\delta}=k_1\frac{\epsilon}{h}.
\end{align*}
On the other hand, the truncation error is
\begin{align*}
e_T=\left|f'(x)-\frac{f(x+h)-f(x)}{h}\right|=k_2h,
\end{align*}
since it is a first order approximation. To minimize the total error
\begin{align*}
e=e_{\delta}+e_T=k_1\frac{\epsilon}{h}+k_2h,
\end{align*}
we just have to solve the optimization problem,
\begin{align*}
\frac{\partial e}{\partial h}=-k_1\frac{\epsilon}{h^2}+k_2=0,
\end{align*}
which gives us $h^*=\sqrt{\frac{k_1}{k_2}\epsilon}$. We check that it is indeed a minimum by checking that
\begin{align*}
\left.\frac{\partial^2e}{\partial h^2}\right|_{h=h^*}=\frac{2k_1\epsilon}{(h^*)^3}=\frac{2k_1\epsilon}{\left(\frac{k_1}{k_2}\epsilon\right)^{3/2}}>0.
\end{align*}

\textbf{Part c)} In equispaced finite difference methods of order $p$ the rounding error is
\begin{align*}
e_{\delta}&=\left|\sum_{j=0}^{p}\frac{w_j}{h}f_j-\sum_{j=0}^{p}\frac{w_j}{h}f_j(1+\delta_j)\right|\\
&=\left|-\sum_{j=0}^{p}\frac{w_j}{h}f_j\delta_j\right|\\
&<\left|\sum_{j=0}^{p}w_jf_j\right|\frac{\epsilon}{h}=k_1\frac{\epsilon}{h},
\end{align*}
and the truncation error of order $p$ is
\begin{align*}
e_T=k_2h^p.
\end{align*}
To minimize the total error
\begin{align*}
e=e_{\delta}+e_T=k_1\frac{\epsilon}{h}+k_2h^p,
\end{align*}
we just have to solve the optimization problem,
\begin{align*}
\frac{\partial e}{\partial h}=-k_1\frac{\epsilon}{h^2}+pk_2h^{p-1}=0,
\end{align*}
which gives us $h^*=\left(p\frac{k_1}{k_2}\epsilon\right)^{1/p+1}$. We check that it is indeed a minimum by checking that
\begin{align*}
\left.\frac{\partial^2e}{\partial h^2}\right|_{h=h^*}=\frac{2k_1\epsilon}{(h^*)^3}+p(p-1)k_2(h^*)^{p-2}=\frac{2k_1\epsilon}{\left(p\frac{k_1}{k_2}\epsilon\right)^{3/p+1}}+p(p-1)k_2\left(p\frac{k_1}{k_2}\epsilon\right)^{\frac{p-2}{p+1}}>0.
\end{align*}

\textbf{Part d)} We are given the function $\tan{(x)}$ and asked to approximate its derivative at $x_0=\pi/3$. We can calculate it analytically,
\begin{align*}
f'(x)=\frac{1}{\cos^2{x}},~~~~~f'(\pi/3)=4.
\end{align*}
With the same procedure as in the previous problem, we have derived the following approximations and the leading error terms:
\begin{itemize}
\item First order:
\begin{align*}
f'(x)=\frac{f(x+h)-f(x)}{h}-\frac{h}{2}f''(x)+\mathcal{O}(h^2)
\end{align*}
\item Second order:
\begin{align*}
f'(x)=\frac{f(x+h)-f(x-h)}{2h}-\frac{h^2}{6}f'''(x)+\mathcal{O}(h^3)
\end{align*}
\item Fourth order:
\begin{align*}
f'(x)=\frac{2}{3}\frac{f(x+h)-f(x-h)}{h}-\frac{1}{12}\frac{f(x+2h)-f(x-2h)}{h}+\frac{4h^4}{5!}f^{(V)}(x)+\mathcal{O}(h^5)
\end{align*}
\item Sixth order:
\begin{align*}
f'(x)=15\frac{f(x+h)-f(x-h)}{20h}-6\frac{f(x+2h)-f(x-2h)}{40h}+\frac{f(x+3h)-f(x-3h)}{60h}-\frac{h^6}{140}f^{(VII)}(x)+\mathcal{O}(h^7)
\end{align*}
\end{itemize}
In the next figure we can see, in logarithmic scale, the dependency of the truncation error on $h$ for every approximation. The lines are have slopes $1,2,4,6$ (from top to bottom) mathching its respective data. This proves the order of the truncation error for the four approximations considered. Note that the minimal error is achieved "sooner" (in the sense of not needing to reduce $h$ as much) for higher order approximations. In addition, we can check that once the rounding error is dominant, the dependency is the same for all the approximations. It cannot be any other way if we remember that $e_{\delta}\sim\frac{\epsilon}{h}$ regardless of the order of our approximation. We can check that this error would in fact have a slope of minus 1 in logarithmic scale, as we see in the figure.
\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.4]{IMAGES/problem3.eps}}
\caption{Error dependency on $h$ for the four different approximations.}
\end{figure}
\subsection*{Matlab code for this problem}
\begin{verbatim}
format long
x_0=pi/3;
f = @(x) tan(x);
fp = @(x) (1/cos(x))^2;
H=logspace(-1,-14,100);
for k = 1:length(H)
    h = H(k);
    df1(k) = (f(x_0+h)-f(x_0))/h;
    df2(k) = (f(x_0+h)-f(x_0-h))/(2*h);
    df4(k) = (-5/60*f(x_0+2*h)+2/3*f(x_0+h)-2/3*f(x_0-h)+5/60*f(x_0-2*h))/h;
    df6(k) = 0.1*(15*(f(x_0+h)-f(x_0-h))/(2*h)-6*(f(x_0+2*h)-f(x_0-2*h))/(4*h)
       +(f(x_0+3*h)-f(x_0-3*h))/(6*h));
end
figure('units','normalized','outerposition',[0 0 1 1])
loglog(H,abs(df1-fp(x_0)),'*',H,abs(df2-fp(x_0)),'*',H,abs(df4-fp(x_0)),'*',
	   H,abs(df6-fp(x_0)),'*',H,H/2,'--',H,H.^(2)/6,'--',H,H.^(4)/30,'--',H,H.^6/140,'--')
set(gca,'fontsize',14)
grid on
xlabel('$h$ (log scale)','fontsize',20,'interpreter','latex')
ylabel('Error (log scale)','fontsize',20,'interpreter','latex')
saveas(gcf,'IMAGES/problem3','epsc')
\end{verbatim}

\end{solution}


\section{Principal Component Analysis}
Principal Component Analysis is a dimension reduction technique that finds
the variance maximizing the directions onto which to project the data.
\subsection{Eigenvalues, Eigenvectors, and the Spectral Theorem}
Recall that for a given matrix $A\in \mathbb{R}^{n \times n}$, $\lambda$ is said
to be an eigenvalue of $A$ if there exists a vector
$z \in \mathbb{R}^n \backslash \{ 0 \}$, called an eigenvector, such that:
\begin{equation}
  Az=\lambda z
\end{equation}
The spectral theorem states that given matrix $A\in \mathbb{R}^{n \times n}$, if
$A$ is symmetric then $A$ is diagonalizable by a real orthogonal matrix
$U\in \mathbb{R}^{n \times n}$. Note
$\Lambda = \textrm{diag}(\lambda_1, \hdots, \lambda_n)$. Then $\exists \, \Lambda$
such that:
\begin{equation}
  A=U\Lambda U^T
\end{equation}
Note that the eigenvector associated with the largest eigenvalue is called the
principal eigenvector of matrix $A$.
\subsection{Algorithm}
The PCA procedure projects the data onto $k$ dimensions by maximizing the variance
of the data as follows:
\begin{enumerate}
  \item Normalize the data to have mean $0$, standard deviation $1$:
    \begin{equation} x^{(i)}\leftarrow\frac{x^{(i)}-\mu}{\sigma} \end{equation}
      where $\mu$ and $\sigma^2$ are:
      \begin{equation} \mu = \frac{1}{m}\sum_{i=1}^mx^{(i)} \end{equation}
      \begin{equation} \sigma^2=\frac{1}{m}\sum_{i=1}^m(x^{(i)}-\mu)^2 \end{equation}
  \item Compute covariance matrix $\Sigma$, which is symmetric with real eigenvalues.
    \begin{equation} \Sigma = \frac{1}{m}\sum_{i=1}^m x^{(i)}x^{(i)^T} \in \mathbb{R}^{n \times n} \end{equation}
  \item Compute $u_1, \hdots, u_k \in \mathbb{R}^{n}$ the $k$ orthogonal principal
    eigenvectors of $\Sigma$, i.e. the orthogonal eigenvectors of the $k$
    largest eigenvalues.
  \item Project the data on $\textrm{span}_\mathbb{R}(u_1,...,u_k)$ to create
    a vector $y^{(i)}$ from point $x^{(i)}$:
    \begin{equation} y^{(i)} = U^T x^{(i)} = \left( \begin{array} { c } { u _ { 1 } ^ { T } x ^ { ( i ) } } \\ { u _ { 2 } ^ { T } x ^ { ( i ) } } \\ { \vdots } \\ { u _ { k } ^ { T } x ^ { ( i ) } } \end{array} \right) \in \mathbb { R } ^ { k } \end{equation}
\end{enumerate}
This procedure maximizes the variance among all $k$-dimensional spaces.
\subsection{Algorithm: SVD}
Eigenvalue decomposition is deifned for square matricies, and using the SVD of a
data matrix $X$ is often used in practice.
\begin{enumerate}
  \item Zero-mean the data to have mean $0$:
    \begin{equation} x^{(i)}\leftarrow x^{(i)}-\mu \end{equation}
      where $\mu$ is:
      \begin{equation} \mu = \frac{1}{m}\sum_{i=1}^mx^{(i)} \end{equation}
  \item Computed the singular value decomposition of $X_{\mu}$, our zero-meaned
    data matrix:
    \begin{equation}
      X_{\mu} = USV
    \end{equation}
  \item Take the first $k$ columns of $U$ as our transform matrix, denoted $U_k$.
  \item Project the data to create a matrix $Y$ from data matrix $X_{\mu}$:
    \begin{equation} Y = X U_k\end{equation}
\end{enumerate}


\section{Optimization Functions}

The functions in this section are models with analytic optimal points.

\subsection{General}

\subsubsection{Beale's Function}
Beale's function has a two-dimensional input space and a single global/local minimum.
See \url{https://en.wikipedia.org/wiki/Test_functions_for_optimization}.

\begin{itemize}
  \item Function: $f(x,y) = (1.5-x+xy)^2+(2.25-x+xy^2)^2+(2.625-x+xy^3)^2$
  \item Domain: $-4.5 \leq x,y \leq 4.5$
  \item Global Minimum: $f(3,0.5)=0$
\end{itemize}

\subsubsection{Rosenbrock Function}
The Rosenbrock function can take a varying number of inputs.  For up to three inputs, a single global minimum
exists.  For four to seven inputs, there is one local minimum and one global maximum.
See \url{https://en.wikipedia.org/wiki/Rosenbrock_function}.

\begin{itemize}
  \item Function: $f(\vec x) = \sum_{i=1}^{n-1}\left[100\left(x_{i+1}-x_i^2\right)^2+\left(x_i-1)^2\right) \right]$
  \item Domain: $ -\infty \leq x_i \leq \infty \hspace{10pt} \forall \hspace{10pt} 1\leq i \leq n$
  \item Global Minimum: $f(1,1,\cdots,1,1)=0$
  \item Local minimum ($n\geq4$): near $f(-1,1,\cdots,1)$
\end{itemize}

\subsubsection{Goldstein-Price Function}
The Goldstein-Price function is a two-dimensional input function with a single global minimum.
See \url{https://en.wikipedia.org/wiki/Test_functions_for_optimization}.

\begin{itemize}
  \item Function:
    \begin{align}
      f(x,y) =& \left[1+(x+y+1)^2\left(19-14x+3x^2-14y+6xy+3y^2\right)\right] \\ \nonumber
        & \cdot\left[30+(2x-3y)^2(18-32x+12x^2+48y-36xy+27y^2)\right]
    \end{align}
  \item Domain: $-2 \leq x,y \leq 2$
  \item Global Minimum: $f(0,-1)=3$
\end{itemize}

\subsubsection{McCormick Function}
The McCormick function is a two-dimensional input function with a single global minimum.
See \url{https://en.wikipedia.org/wiki/Test_functions_for_optimization}.

\begin{itemize}
  \item Function: $f(x,y) = \sin(x+y) + (x-y)^2 - 1.5x + 2.5y + 1$
  \item Domain: $-1.5 \leq x \leq 4$, $-3 \leq y \leq 4$
  \item Global Minimum: $f(-0.54719,-1.54719) = -1.9133$
\end{itemize}

\subsubsection{2D Canyon}
The two-dimensional canyon offers a low region surrounded by higher walls.

\begin{itemize}
  \item Function: $f(x,y) = xy\cos(x+y)$
  \item Domain: $0 \leq x,y \leq \pi$
  \item Global Minimum: $f(1.8218,1.8218)=-2.90946$
\end{itemize}

\subsubsection{Time-Parabola}
This model features the sum of a parabola in both $x$ and $y$; however, the parabola in $y$ moves in time and
has a reduced magnitude as time increases. As such, the minimum is always found at $x=0$ and at $(t-y)=0$, with
$y$ values at low $t$ being more impactful than at later $t$, and $x$ as impactful as $y(t=0)$.

\begin{itemize}
  \item Function: $f(x,y,t) = x^2 + \sum_{t} (t-y)^2 \exp{-t}$
  \item Domain: $-10 \leq x,y,t \leq 10$
  \item Global Minimum: $f(0,y=t,t) = 0$
\end{itemize}

\subsection{Constrained}
\subsubsection{Mishra's Bird Function}
The Mishra bird function offers a constrained problem with multiple peaks, local minima, and one steep global
minimum.
See \url{https://en.wikipedia.org/wiki/Test_functions_for_optimization}.

\begin{itemize}
  \item Function: $f(x,y) = \sin(y)\exp[1-\cos(x)]^2 + \cos(x)\exp[1-\sin(y)]^2 + (x-y)^2$
  \item Constraint: $(x+5)^2 + (y+5)^2 < 25$
  \item Domain: $-10 \leq x \leq 0$, $-6.5 \leq y \leq 0$
  \item Global Minimum: $f(-3.1302468, -1.5821422) = -106.7645367$
\end{itemize}


\section{Bayesian Statistics}
Bayesian inference conceptually amounts to weighting the likelihood $L_n(\theta)$ by a prior knowledge we might have on $\theta$.
Given a statistical model we technically model our parameter $\theta$ as if it were a random variable. We therefore define the \textbf{prior distribution} (PDF):
\begin{align*}
\pi(\theta)
\end{align*}
Let $X_1,...,X_n$. We note $L_n(X_1,...,X_n|\theta)$ the joint probability distribution of $X_1,...,X_n$ conditioned on $\theta$ where $\theta \sim \pi$. This is exactly the likelihood from the frequentist approach.
\subsection{Bayes' formula}.
The {posterior distribution} verifies:
\begin{align*}
\forall \theta \in \Theta, \pi(\theta|X_1,...,X_n) \propto\\ \pi(\theta)L_n(X_1,...,X_n | \theta)
\end{align*}
The constant is the normalization factor to ensure the result is a proper distribution, and does not depend on $\theta$:
\begin{align*}
\pi(\theta|X_1,...,X_n) = \frac{\pi(\theta)L_n(X_1,...,X_n | \theta)}{\int_\Theta\pi(\theta)L_n(X_1,...,X_n | \theta)d\theta}
\end{align*}
We can often use an \textbf{improper prior}, i.e. a prior that is not a proper probability distribution (whose integral diverges), and still get a proper posterior. For example, the improper prior $\pi(\theta) = 1$ on $\Theta$ gives the likelihood as a posterior.
\subsection{Jeffreys Prior}
\begin{align*}
\pi_J(\theta) \propto \sqrt{det I(\theta)}
\end{align*}
where $I(\theta)$ is the Fisher information. This prior is \textbf{invariant by reparameterization}, which means that if we have $\eta = \phi(\theta)$, then the same prior gives us a probability distribution for $\eta$ verifying:
\begin{align*}
\tilde\pi_J(\eta) \propto \sqrt{det \tilde I(\eta)}
\end{align*}
The change of parameter follows the following formula:
\begin{align*}
\tilde\pi_J(\eta) = det(\nabla \phi^{-1}(\eta)) \pi_J(\phi^{-1}(\eta))
\end{align*}
\subsection{Bayesian confidence region}
Let $\alpha \in (0, 1)$. A *Bayesian confidence region with level $\alpha$* is a random subset $\mathcal{R} \subset \Theta$ depending on $X_1,...,X_n$ (and the prior $\pi$) such that:
\begin{align*}
P[\theta \in \mathcal{R} | X_1,...,X_n] \geq 1 - \alpha
\end{align*}
Bayesian confidence region and confidence interval are \textbf{distinct notions}.The Bayesian framework can be used to estimate the true underlying parameter. In that case, it is used to build a new class of estimators, based on the posterior distribution.
\subsection{Bayes estimator}

\textbf{posterior mean}:
\begin{align*}
\hat{\theta}_{(\pi)} = \int_\Theta\theta\pi(\theta | X_1,...,X_n)d\theta
\end{align*}
\textbf{Maximum a posteriori estimator (MAP):}
\begin{align*}
\hat{\theta}^{MAP}_{(\pi)} = argmax_{\theta\in\Theta}\pi(\theta | X_1,...,X_n)
\end{align*}
The MAP is equivalent to the MLE, if the prior is uniform.


\lab{Applications}{Regression on a Circle}{Statistics 2}
\label{LeastSquaresCircle}

\objective{This section will introduce a more advanced application of Least Squares: fitting data to an circle.}

\section*{Fitting data to a circle}

Recall that the equation of a circle, with radius $r$ centered at $(c_1,c_2)$, is given by
\begin{equation}
\label{circle}
(x-c_1)^2 + (y-c_2)^2 = r^2.
\end{equation}
Suppose we are given a set of data points closely forming a circle $\{(x_i,y_i)\}^n_{i=1}$.  The ``best'' fit is found via least squares by expanding \eqref{circle} to get
\[
2 c_1 x + 2 c_2 y + c_3 = x^2 + y^2,
\]
where $c_3 = r^2 - c_1^2 - c_2^2$.  Then we can write the linear system $A x = b$ as
\[
\begin{pmatrix}
2 x_1 & 2 y_1 & 1\\
2 x_2 & 2 y_2 & 1\\
\vdots & \vdots & \vdots \\
2 x_n & 2 y_n & 1
\end{pmatrix}
\begin{pmatrix}
c_1\\
c_2\\
c_3
\end{pmatrix}=
\begin{pmatrix}
x_1^2 + y_1^2\\
x_2^2 + y_2^2\\
\vdots\\
x_n^2 + y_n^2
\end{pmatrix},
\]
where the matrix $A$ and the vector $b$ are obtained by the given data and the unknown $x$ contains the information about the center and radius of the circle and is obtained by finding the least squares solution.

\section*{Example}

In this section, we fit the following points to a circle:
\begin{align*}
&(134,76),(104,146),(34,176),(-36,146),\\
&(-66,76),(-36,5),(34,-24),(104,5),(134,76)
\end{align*}

We enter them into Python as a $9\times 2$ array:
\begin{lstlisting}[style=python]
: P = sp.array([[134,76],[ 104,146],[ 34,176],[ -36,146],[ -66,76],[ -36,5],[ 34,-24],[ 104,5],[ 134,76]])
\end{lstlisting}
Then we can separate the $x$ and $y$ coordinates by the commands \li{P[:,0]} and \li{P[:,1]}, respectively.  Hence, we compute $A$ and $b$ by entering the following:
\begin{lstlisting}[style=python]
: A = sp.column_stack((2*P,sp.ones((9,1),dtype=sp.int_)))
: b = P[:,0]**2 + P[:,1]**2
\end{lstlisting}
Hence, we get the least squares solution
\begin{lstlisting}[style=python]
: x = sp.dot(sp.dot(la.inv(sp.dot(A.T,A)),A.T),b)
\end{lstlisting}
Then we find $c_1$, $c_2$, and $r$ by:
\begin{lstlisting}[style=python]
: c1 = x[0]
: c2 = x[1]
: c3 = x[2]
: r = sp.sqrt(c1**2 + c2**2 + c3)
\end{lstlisting}
We plot this by executing
\begin{lstlisting}[style=python]
: theta = sp.linspace(0,2*sp.pi,200)
: plt.plot(r*sp.cos(theta)+c1,r*sp.sin(theta)+c2,'-',P[:,0],P[:,1],'*')
: plt.show()
\end{lstlisting}

\begin{problem}
Download the file lab10.txt from the following link:
\url{http://www.math.byu.edu/~jeffh/teaching/m343h/lab10.txt}
You can load this datafile into Python by typing
\begin{lstlisting}[style=python]
: lab10 = sp.genfromtxt("lab10.txt")
\end{lstlisting}
Now the data is available in the matrix \li{lab10}.  This consists of two columns corresponding to the $x$ and $y$ values of a given data set.  Use least squares to find the center and radius of the circle that best fits the data.  Then plot the data points and the circle on the same graph.  Finish off the problem with a discussion of what you've learned.
\end{problem}

\begin{problem}
The general equation for an ellipse is:
\[
A(x-x_0)^2 + B(x-x_0)(y-y_0) + C(y-y_0)^2 = 1
\]

Write a program that uses least squares to fit data to an ellipse. One option to finding the center point $(x_0,y_0)$ is to use the mean function. Test the program on \li{lab10}. Also test it against sp.dot(lab10,sp.array([[2,0],[0,1]]) ) . Plot the result. How well does your function work?
\end{problem}


\subsection{Single equality constraint}

\subsubsection{Constrained optimisation}

Rather than maximise \(f(x)\), we want to maximise \(f(x)\) subject to \(g(x)=0\).

We write this, the Lagrangian, as:

\(\mathcal{L}(x,\lambda )=f(x)-\sum^m_k\lambda_k [g_k(x)-c_k]\)

We examine the stationary points for both vector \(x\) and \(\lambda \). By including the latter we ensure that these points are consistent with the constraints.

\subsubsection{Solving the Langrangian with one constraint}

Our function is:

\(\mathcal{L}(x, \lambda )=f(x)-\lambda [g(x)-c]\)

The first-order conditions are:

\(\mathcal{L}_{\lambda }= -[g(x)-c]\)

\(\mathcal{L}_{x_i}=\dfrac{\delta f}{\delta x_i}-\lambda \dfrac{\delta g}{\delta x_i}\)

The solution is stationary so:

\(\mathcal{L}_{x_i}=\dfrac{\delta f}{\delta x_i}-\lambda \dfrac{\delta g}{\delta x_i}=0\)

\(\lambda \dfrac{\delta g}{\delta x_i}=\dfrac{\delta f}{\delta x_i}\)

\(\lambda =\dfrac{\dfrac{\delta f}{\delta x_i}}{\dfrac{\delta g}{\delta x_i}}\)

Finally, we can use the following in practical applications.

\(\dfrac{\dfrac{\delta f}{\delta x_i}}{\dfrac{\delta g}{\delta x_i}}=\dfrac{\dfrac{\delta f}{\delta x_j}}{\dfrac{\delta g}{\delta x_j}}\)


\section{Exercises in Characteristic Functions}\label{S:xsCFs}% {S:xsExpectationsOfRVs} %S:xsMultivariateRVs
\begin{ExerciseList}
%CF
\Exercise
Let $X$ be a discrete random variable (RV) with probability mass function (PMF)
\[
f_X(x) =
\begin{cases}
\frac{1}{3} & \text{ if } x=0\\
\frac{1}{3} & \text{ if } x=1\\
\frac{1}{3} & \text{ if } x=2\\
0 & \text{ otherwise} \enspace .
\end{cases}
\]
\be
\item~Find the characteristic function (CF) of $X$
\item~Using the CF find $V(X)$, the variance of $X$. {Hint: $V(X)=E(X^2)-(E(X))^2$}
\ee

\Answer
~\\
\be
\item~The CF of the discrete RV $X$ is
\begin{align*}
\cf_X(t)
&= E(e^{\imath t X}) = \sum_{x\in\{0,1,2\}} e^{\imath t x} f_X(x)
= e^{\imath t \times 0} \times \frac{1}{3} + e^{\imath t \times 1} \times \frac{1}{3}+e^{\imath t \times 2} \times \frac{1}{3}\\
&= \frac{1}{3} \left(1+e^{\imath t} + e^{\imath 2 t}\right) \enspace .
\end{align*}

\item~To find $V(X)$ using $V(X)=E(X^2)-(E(X))^2$ we need the first two moments of $X$.  First, differentiate $\cf_X(t)$ w.r.t.~$t$
\begin{align*}
\frac{d}{dt}\cf_X(t)
= \frac{d}{dt} \left( \frac{1}{3} \left(1+e^{\imath t} + e^{\imath 2 t}\right) \right)
= \frac{1}{3} \left(0+ \imath e^{\imath t} + 2 \imath e^{\imath 2 t} \right)
= \frac{\imath}{3} \left(e^{\imath t} + 2 e^{\imath 2 t} \right)
\end{align*}
We get the $k$-th moment $E(X^k)$ by multiplying the $k$-th derivative of $\cf_X(t)$ evaluated at $t=0$ by $\frac{1}{\imath^k}$ as follows:
\begin{align*}
E(X)
= \frac{1}{\imath} \left[ \frac{d}{dt}\cf_X(t) \right]_{t=0}
= \frac{1}{\imath} \left[ \frac{\imath}{3} \left(e^{\imath t} + 2 e^{\imath 2 t} \right)\right]_{t=0}
= \frac{1}{\imath} \frac{\imath}{3} \left(e^0+2e^0\right) = \frac{1}{3}(1+2)=1 \enspace .
\end{align*}
\begin{align*}
E(X^2)
&= \frac{1}{\imath^2} \left[ \frac{d^2}{dt^2}\cf_X(t) \right]_{t=0}
= \frac{1}{\imath^2} \left[ \frac{d}{dt} \frac{\imath}{3} \left(e^{\imath t} + 2 e^{\imath 2 t} \right) \right]_{t=0}
= \frac{1}{\imath^2} \left[ \frac{\imath}{3} \left(\imath e^{\imath t} + 4 \imath e^{\imath 2 t} \right) \right]_{t=0}\\
& = \frac{1}{3} \left(e^0 + 4 e^0\right) = \frac{5}{3} \enspace .
\end{align*}
Finally,
\[
V(X) = E(X^2)-(E(X))^2 = \frac{5}{3} - 1^2 = \frac{5-3}{3}=\frac{2}{3} \enspace .
\]
\ee

\Exercise
Recall that the $\geometric(\theta)$ RV $X$ has the following PMF
\[
f_X(x;\theta) =
\begin{cases}
\theta (1-\theta)^x & \text{ if } x \in \{0,1,2,3,\ldots\}\\
0 & \text{ otherwise}
\end{cases}
\]
\be
\item~Find the CF of $X$. (Hint: the sum of the infinite geometric series $\sum_{x=0}^{\infty} a r^x = \frac{a}{1-r}$.)
\item~Using the CF find $E(X)$.
\ee
\Answer
~\\
\be
\item~
\[
\cf_X(t) = E \left(e^{\imath t X}\right) = \sum_{x=0}^{\infty} e^{\imath t x}\theta(1-\theta)^x
= \sum_{x=0}^{\infty} \theta \left(e^{\imath t} (1-\theta)\right)^x = \frac{\theta}{1-e^{\imath t} (1-\theta)} \enspace .
\]
The last equality is due to $\sum_{x=0}^{\infty} a r^x = \frac{a}{1-r}$ with $a=\theta$ and $r=e^{\imath t} (1-\theta)$.
\item~
\begin{align*}
E(X)
&= \frac{1}{\imath} \left[ \frac{d}{dt} \left( \frac{\theta}{1-e^{\imath t}(1-\theta)} \right) \right]_{t=0}
=  \frac{1}{\imath} \theta \left[ \frac{d}{dt} \left( \left(1-e^{\imath t}(1-\theta)\right)^{-1} \right) \right]_{t=0} \\
&= \frac{1}{\imath} \theta \left[ -\left(1-e^{\imath t} (1-\theta)\right)^{-2} \frac{d}{dt} \left( 1-e^{\imath t}(1-\theta)\right) \right]_{t=0}\\
&= \frac{1}{\imath} \theta \left[ -\left(1-e^{\imath t} (1-\theta)\right)^{-2} \left( 0-\imath e^{\imath t}(1-\theta)\right) \right]_{t=0}\\
&= \frac{1}{\imath} \theta \left( -\left(1-e^{0} (1-\theta)\right)^{-2} \left( -\imath e^{0}(1-\theta)\right) \right)\\
&= \frac{1}{\imath} \theta \left( -\left(1- 1+\theta\right)^{-2} \left( -\imath (1-\theta)\right) \right)\\
&=\frac{1-\theta}{\theta} \enspace .
\end{align*}
\ee

\Exercise
Let $X$ be the $\uniform(a,b)$ RV with the following probability density function (PDF)
\[
f_X(x; a,b) =
\begin{cases}
\frac{1}{b-a} & \text{ if } x \in [a,b] \\
0 & \text{ otherwise} \enspace.
\end{cases}
\]
Find the CF of $X$.
\Answer
~\\
\begin{align*}
\cf_X(t)
&= E\left( e^{\imath t X} \right)
= \int_{-\infty}^{\infty} e^{\imath t x} f_X(x;a,b) dx
= \int_{a}^{b} e^{\imath t x} \frac{1}{b-a} dx
= \frac{1}{b-a} \int_{a}^{b} e^{\imath t x} dx
= \frac{1}{b-a} \left[ \frac{e^{\imath t x}}{\imath t} \right]_{x=a}^b\\
&=  \frac{1}{(b-a)\imath t} \left( e^{\imath t b} - e^{\imath t a}\right) \enspace .
\end{align*}

\Exercise
%\vspace{0.5cm}

%A nice property of CFs we will use for some of the exercises below is the following.  If $X_1,X_2,\ldots,X_n$ are independent RVs and $a_1,a_2,\ldots,a_n$ are some constants, then the CF of the linear combination $Y=\sum_{i=1}^n a_i X_i$ is
%\begin{equation}\label{E:LinCombCFtut}
%\cf_Y (t) = \cf_{X_1} (a_1 t) \times \cf_{X_2} (a_2 t) \times \cdots \times \cf_{X_n} (a_n t) = \prod_{i=1}^n \cf_{X_i} (a_i t) \enspace .
%\end{equation}

Recall that the $\poisson(\lambda)$ RV has the following PMF
\[
f_X(x;\lambda) =
\begin{cases}
\frac{\lambda^x}{x!}e^{-\lambda} & \text{ if } x \in \{0,1,2,3,\ldots \}\\
0 & \text{ otherwise} \enspace .
\end{cases}
\]
{Hint: the power series of $e^{\alpha} = \sum_{x=0}^{\infty} \frac{\alpha^x}{x!}$.}
\be
\item~
Find the CF of $X$.
\item~
Find the variance of $X$ using its CF.
\ee
\Answer
~\\
\be
\item~
\[
\cf_X(t) = E\left(e^{\imath t X}\right)
= \sum_{x=0}^{\infty} e^{\imath t x} \frac{\lambda^x}{x!}e^{-\lambda}
= e^{-\lambda} \sum_{x=0}^{\infty} \frac{e^{\imath t x} \lambda^x}{x!}
= e^{-\lambda} \sum_{x=0}^{\infty} \frac{\left(\lambda e^{\imath t}\right)^x}{x!}
= e^{-\lambda} e^{\lambda e^{\imath t}} = e^{\lambda e^{\imath t} - \lambda} \enspace .
\]
The second-last equality above is using $e^{\alpha} = \sum_{x=0}^{\infty} \frac{\alpha^x}{x!}$ with $\alpha=\lambda e^{\imath t}$.
\item~
To find $V(X)$ using $\cf_X(t)$ we need $E(X)$ and $E(X^2)$.
\begin{align*}
E(X)
&= \frac{1}{\imath} \left[ \frac{d}{dt} \cf_X(t) \right]_{t=0}
= \frac{1}{\imath} \left[ \frac{d}{dt} e^{\lambda e^{\imath t} - \lambda} \right]_{t=0}
= \frac{1}{\imath} \left[ e^{\lambda e^{\imath t} - \lambda} \frac{d}{dt} \left(\lambda e^{\imath t} - \lambda \right) \right]_{t=0}\\
&= \frac{1}{\imath} \left[ e^{\lambda e^{\imath t} - \lambda} \lambda \imath e^{\imath t} \right]_{t=0}
= \frac{1}{\imath} \left( e^{\lambda - \lambda} \lambda \imath \right)
= \frac{1}{\imath} \left( \lambda \imath \right) = \lambda \enspace .
\end{align*}
\begin{align*}
E(X^2)
&= \frac{1}{\imath^2} \left[ \frac{d^2}{dt^2} \cf_X(t) \right]_{t=0}
= \frac{1}{\imath^2} \left[ \frac{d}{dt} \left( e^{\lambda e^{\imath t} - \lambda} \lambda \imath e^{\imath t} \right) \right]_{t=0}
= \frac{1}{\imath^2} \left[ \frac{d}{dt} \left( \lambda \imath e^{\lambda e^{\imath t} - \lambda + \imath t} \right) \right]_{t=0}\\
&= \frac{1}{\imath^2} \left[ \lambda \imath e^{\lambda e^{\imath t} - \lambda + \imath t} \frac{d}{dt} \left( \lambda e^{\imath t} - \lambda + \imath t \right) \right]_{t=0}
= \frac{1}{\imath^2} \left[ \lambda \imath e^{\lambda e^{\imath t} - \lambda + \imath t} \left( \lambda \imath e^{\imath t} - 0 + \imath \right) \right]_{t=0}\\
&= \frac{1}{\imath^2} \left( \lambda \imath e^{\lambda e^{0} - \lambda + 0} \left( \lambda \imath e^{0}  + \imath \right) \right)
= \frac{1}{\imath^2} \left( \lambda \imath \left( \lambda \imath  + \imath \right) \right)
= \frac{1}{\imath^2} \left( \lambda \imath^2 \left( \lambda  + 1 \right) \right)
= \lambda^2 + \lambda \enspace .
\end{align*}
Finally,
\[
V(X) = E(X^2) - (E(X))^2 = \lambda^2 + \lambda - \lambda^2 = \lambda \enspace .
\]
\ee

\Exercise
Let $X$ be a $\poisson(\lambda)$ RV and $Y$ be another $\poisson(\mu)$ RV.
Suppose $X$ and $Y$ are independent.
Use Eqn.~\eqref{E:LinCombCF} to first find the CF of the RV $W=X+Y$.
From the CF of $W$ try to identify what RV it is.

\Answer
~\\
\[
\cf_{W}(t) = \cf_{X+Y}(t) = \cf_X(t) \times \cf_Y(t)
= e^{\lambda e^{\imath t}-\lambda} \times e^{\mu e^{\imath t}-\mu}
= e^{\lambda e^{\imath t}-\lambda + \mu e^{\imath t}-\mu}
= e^{(\lambda+\mu) e^{\imath t} -(\lambda+\mu)} \enspace .
\]
So, $W$ is a $\poisson(\lambda+\mu)$ RV. Thus the sum of two independent Poisson RVs is also a Poisson RV with parameter given by the sum of the parameters of the two RVs being added.  The same idea generalizes to the sum of more than two Poisson RVs.

\Exercise
Recall from lecture that if $Y=a+bX$ for some constants $a$ and $b$ with $b\neq 0$ then $\cf_Y(t) = e^{\imath a t} \cf_X(bt)$ and that $\cf_Z(t)=e^{-t^2/2}$ if $Z$ is the $\normal(0,1)$ RV.
Using these facts find the CF of $-Z$, the RV obtained from $Z$ by simply switching its sign.
From the CF of $-Z$ identify what RV it is.
\Answer
~\\
We can use the facts by noting $Y=-Z = 0 + (-1) \times Z$, with $a=0$ and $b=-1$ in $Y=a+bX$ and get
\[
\cf_{Y}(t) = e^{\imath \times 0 \times t} \cf_Z(-1 \times t) = \cf_Z(-t) = e^{-((-1 \times t))^2/2} = e^{-t^2/2} = \cf_Z(t)
\]
Thus, $\cf_{-Z}(t) = \cf_Z(t)$ and therefore the distributions of $Z$ and $-Z$ are the same. This should make sense because by switching signs of a symmetric (about $0$) RV you have not changed its distribution!  Note: we are not saying $Z=-Z$ but just that their distibutions are the same, i.e., $F_Z(z) = F_{-Z}(z)$ for every $z \in \Rz$.

\end{ExerciseList}


\subsection{Pearson correlation coefficient}

The Pearson correlation coefficient is definited as the covariance normalised by the individual variances.

It is between \(-1\) (total negative linear correlation), \(0\) (no linear correlation) and \(1\) (total negative linear correlation).

\(p_{X,Y}=\dfrac{cov (X,Y)}{\sigma_X\sigma_Y}\)


\subsubsection{Euler's Identity}
\noindent
Let's see what happens when we look at the Maclaurin series for $e^{ix}$.
\begin{equation*}
	e^{ix} = 1 + (ix) + \frac{(ix)^2}{2!} + \frac{(ix)^3}{3!} + \frac{(ix)^4}{4!} + \frac{(ix)^5}{5!} \ldots
\end{equation*}
\begin{equation*}
	= 1 + ix - \frac{x^2}{2!} - i\frac{x^3}{3!} + \frac{x^4}{4!} + i\frac{x^5}{5!} - \ldots
\end{equation*}
\begin{equation*}
	= \left(1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \ldots \right) + i\left(x - \frac{x^3}{3!} + \frac{x^5}{5!} - \ldots \right)
\end{equation*}
The two expressions in parenthesis are exactly the Maclaurin series for $\cos{x}$ and $\sin{x}$. So,
\begin{equation*}
	e^{ix} = \cos{x} + i\sin{x}
\end{equation*}
In the case that $x = \pi$,
\begin{equation*}
	e^{i\pi} = \cos{\pi} + i\sin{\pi} = -1 + 0
\end{equation*}
So,
\begin{equation*}
	e^{i\pi} + 1 = 0
\end{equation*}


\subsubsection{Sphere Volume}
\noindent
Since a sphere of radius $R$ is spherically symmetric, it should be simple to find its volume using spherical coordinates.
\begin{align*}
	V &= \int_{0}^{R}{\int_{0}^{\pi}{\int_{0}^{2\pi}{\rho^2\sin{\phi}\mathrm{d}\theta}\mathrm{d}\phi}\mathrm{d}\rho}	\\
	&= \int_{0}^{R}{\rho^2\mathrm{d}\rho} \cdot \int_{0}^{\pi}{\sin{\phi}\mathrm{d}\phi} \cdot \int_{0}^{2\pi}{\mathrm{d}\theta} \\
	&= \frac{R^3}{3} \cdot 2 \cdot 2\pi \\
	&= \frac{4\pi}{3}R^3
\end{align*}


\section{Calculus}

\subsection{Derivative}

$d(f,x)$ returns the derivative of $f$ with respect to $x$.
The $x$ can be omitted for expressions in $x$.

{\color{blue}
\begin{verbatim}
d(x^2)
\end{verbatim}
}

\noindent
$2x$

\bigskip
\noindent
The following table summarizes the various ways to obtain multi-derivatives.

\begin{center}
\begin{tabular}{cllllll}
$\displaystyle{\frac{\partial^2f}{\partial x^2}}$ & & \verb$d(f,x,x)$ & & \verb$d(f,x,2)$ \\
\\
$\displaystyle{\frac{\partial^2f}{\partial x\,\partial y}}$ & & \verb$d(f,x,y)$ \\
\\
$\displaystyle{\frac{\partial^{m+n+\cdot\cdot\cdot} f}{\partial x^m\,\partial y^n\cdots}}$ & &
\verb$d(f,x,...,y,...)$ & & \verb$d(f,x,m,y,n,...)$ \\
\end{tabular}
\end{center}

\subsection{Gradient}

The gradient of $f$ is obtained by using a vector for $x$ in $d(f,x)$.

{\color{blue}
\begin{verbatim}
r = sqrt(x^2 + y^2)
d(r,(x,y))
\end{verbatim}
}

\noindent
$\begin{bmatrix}\frac{x}{(x^2+y^2)^{1/2}}\\ \frac{y}{(x^2+y^2)^{1/2}}\end{bmatrix}$

\bigskip
\noindent
The $f$ in $d(f,x)$ can be a tensor function.
Gradient raises the rank by one.

{\color{blue}
\begin{verbatim}
F = (x + 2 y,3 x + 4 y)
X = (x,y)
d(F,X)
\end{verbatim}
}

\noindent
$\begin{bmatrix}1&2\\3&4\end{bmatrix}$

\subsection{Template functions}

The function $f$ in $d(f)$ does not have to be defined.
It can be a template function with just a name and an argument list.
Eigenmath checks the argument list to figure out what to do.
For example, $d(f(x),x)$ evaluates to itself because $f$ depends on $x$.
However, $d(f(x),y)$ evaluates to zero because $f$ does not depend on $y$.

{\color{blue}
\begin{verbatim}
d(f(x),x)
\end{verbatim}
}

\noindent
$\operatorname{d}(f(x),x)$

{\color{blue}
\begin{verbatim}
d(f(x),y)
\end{verbatim}
}

\noindent
$0$

{\color{blue}
\begin{verbatim}
d(f(x,y),y)
\end{verbatim}
}

\noindent
$\operatorname{d}(f(x,y),y)$

{\color{blue}
\begin{verbatim}
d(f(),t)
\end{verbatim}
}

\noindent
$\operatorname{d}(f(),t)$

\bigskip
\noindent
As the final example shows, an empty argument list causes
$d(f)$ to always evaluate to itself, regardless
of the second argument.

\bigskip
\noindent
Template functions are useful for experimenting with differential forms.
For example, let us check the identity
$$\operatorname{div}(\operatorname{curl}{F})=0$$
for an arbitrary vector function $F$.

{\color{blue}
\begin{verbatim}
F = (F1(x,y,z),F2(x,y,z),F3(x,y,z))
curl(U) = (d(U[3],y) - d(U[2],z),d(U[1],z) - d(U[3],x),d(U[2],x) - d(U[1],y))
div(U) = d(U[1],x) + d(U[2],y) + d(U[3],z)
div(curl(F))
\end{verbatim}
}

\noindent
$0$


\section{Linear equations}
\subsection{Linear and affine functions}
\textbullet Superposition condition:$f(\alpha x + \beta y) = \alpha f(x) + \beta f(y)$\\
\textbullet Such an f is called Linear\\
\textbf{Matrix vector product function}:\\
\textbullet A is $m \times n$ matrix such that $f(x)=Ax$\\
\textbullet f is linear:  $f(\alpha x + \beta y)=A(\alpha x + \beta y)=\alpha f(x) + \beta f(y)$\\
\textbullet Converse is true: If $f:R^{n} \mapsto R^{m}$ is linear, then \\
$f(x)=f(x_{1}e_{1}+x_{2}e_{2}+...x_{n}e_{n})$\\
$=x_{1}f(e_{1})+x_{2}f(e_{2})+...x_{n}f(e_{n})$
$=Ax$ with $A = [f(e_{1})+f(e_{2})+...f(e_{n})]$
\newline \textbf{Affine Functions}: $f:R^{n} \mapsto R^{m}$ is affine if it is a linear function plus a constant i.e $f(x)=Ax+b$ same as $f(\alpha x + \beta y)=\alpha f(x)+\beta f(y)$ holds for all x, y and $\alpha,\beta$ such that $\alpha+\beta=1$
\newline A and b can be calculated as \\
$A=[f(e_{1})-f(0)\thickspace f(e_{2})-f(0) ... f(e_{n})-f(0)]] ; $\\
$b= f(0)$\\
\textbullet Affine functions sometimes incorrectly called linear functions
\subsection{Linear function models}
Price elasticity of demand
$\delta_i^{price}=(p_i^{new}-p_{i})/p_{i}$: fractional changes in prices
$\delta_i^{dem}=(d_i^{new}-d_{i})/d_{i}$: fractional change in demand
Price demand elasticity model: $\delta^{dem}=E\delta^{price}$\\
\textbf{Taylor series approximation}\\
\textbullet The (first-order) Taylor approximation of f near (or at) the point z:\\
\begin{scriptsize}
$\hat{f}(x)= f(z) + \frac{\partial f}{\partial x_1}(z)(x_1 - z_1) + ... +  \frac{\partial f}{\partial x_n}(z)(x_n - z_n)$
\end{scriptsize}\\
\textbullet in compact notation:  \\$\hat{f}(x) = f (z) + Df (z)(x - z)$
\subsection{Systems of linear equations}
\textbullet set (or system) of m linear equations in n variables $x_1,.., x_n$: \\
$A_{11}x_1 + A_{12}x_2 + ... + A_{1n}x_n = b_1$\\
$A_{21}x_1 + A_{22}x_2 + ... + A_{2n}x_n = b_2$\\
.\\
.\\
$A_{m1}x_1 + A_{m2}x_2 + ... + A_{mn}x_n = b_m$\\

\textbullet \textbf{systems of linear equations classified as}\\
\thickspace -- under-determined if m < n (A wide)\\
\thickspace -- square if m = n (A square)\\
\thickspace -- over-determined if m > n (A tall)\\


\subsection{Higher Order to System}
\noindent
Let's say we have the linear, $n^{th}$ order differential equation
\begin{equation*}
	a_ny^{(n)} + \ldots + a_0y = f
\end{equation*}
where $a_0, \ldots a_n$ and $f$ are functions.\\

\noindent
If we let $x_1 = y, x_2 = y', \ldots x_n = y^{(n-1)}$, then we have a system of equations,
\begin{equation*}
	\begin{cases}
		x_1' &= y' = x_2 \\
		\vdots \\
		x_{n-1}' &= y^{(n-1)} = x_n \\
		x_n' &= y^{(n)} = \frac{1}{a_n}\left(f - a_{n-1}x_n - \ldots - a_0x_1\right)
	\end{cases}
\end{equation*}
Rewriting in matrix form,
\begin{equation*}
	\vec{x}' = \begin{bmatrix}
		0 & 1 & 0 & \ldots & 0 \\
		0 & 0 & 1 & \ldots & 0 \\
		\vdots & \vdots	& \vdots & \ddots & \vdots \\
		0 & 0 & 0 & \ldots & 1 \\
		\frac{-a_0}{a_n} & \frac{-a_1}{a_n} & \frac{-a_2}{a_n} & \ldots & \frac{-a_{n-1}}{a_n}
	\end{bmatrix} \vec{x} + \begin{bmatrix}
		0 \\
		\vdots \\
		0 \\
		\frac{f}{a_n}
	\end{bmatrix}
\end{equation*}
The square matrix has characteristic polynomial $p(\lambda) = \frac{(-1)^{n}}{a_n}\left(a_n\lambda^n + a_{n-1}\lambda^{n-1} + \ldots + a_0\right)$ which has the same zeroes as the the auxiliary equation $a_nr^n + \ldots + a_0$.\\

\noindent
Note that if one starts with a higher order equation, converts it to a system, and the converts the system back to a higher order equation, the result is the original equation. This does not hold true when converting a system to an equation and back to a system. The two solutions will have the same eigenvalues but different eigenvectors.


\section{Expectation}

$\mathbb{E}\left[X\right]=\int_{-inf}^{+inf}{x \cdot f_X\left(x\right)\ \ dx}$\\

$\mathbb{E}\left[g\left(X\right)\right]=\int_{-inf}^{+inf}{g\left(x\right) \cdot f_X\left(x\right)dx}$\\

$\mathbb{E}\left[X\middle| Y=y\right]=\int_{-inf}^{+inf}{x \cdot { f}_{X|Y}\left(x|y\right)\ \ dx}$\\

Integration limits only have to be over the support of the pdf. Discrete r.v. same as continuous but with sums and pmfs.\\

Total expectation theorem:\\

$\mathbb{E}\left[X\right]=\int_{-inf}^{+inf}{f_Y\left(y\right)\cdot\mathbb{E}\left[X\middle| Y=y\right]dy}$\\

Law of iterated expectation:\\

$\mathbb E[Y] = \mathbb E[\mathbb E[Y\rvert {X}]]$\\

Expectation of constant $a$:\\

$\mathbb{E}[a] = a$\\

Product of \textbf{independent} r.vs $X$ and $Y$ :\\

$\mathbb{E}[X \cdot Y] = \mathbb{E}[X] \cdot \mathbb{E}[Y]$\\

Product of \textbf{dependent} r.vs $X$ and $Y$ :\\

$\mathbb{E}[X \cdot Y] \neq \mathbb{E}[X] \cdot \mathbb{E}[Y]$\\

$\mathbb{E}[X \cdot Y] = \mathbb{E}[\mathbb{E}[Y \cdot X|Y]] = \mathbb{E}[Y \cdot \mathbb{E}[X|Y]]$\\

Linearity of Expectation where $a$ and $c$ are given scalars:\\

$\mathbb{E}[aX + c Y] = a\mathbb{E}[X] + c\mathbb{E}[Y]$\\

If Variance of $X$ is known:\\

$\mathbb{E}[X^2] = var(X) - \mathbb{E}[X]$\\

\section{Variance}

Variance is the squared distance from the mean.\\

$Var(X)=\mathbb{E}[(X-\mathbb{E}(X))^2]$\\

$Var\left(X\right)=\mathbb{E}\left[X^2\right]-\left(\mathbb{E}\left[X\right]\right)^2$\\

Variance of a product with constant $a$:\\

$Var(aX)=a^2 Var\left(X\right)$\\

Variance of sum of two \textbf{dependent} r.v.:\\

$Var(X + Y)=Var(X)+Var(Y)+2Cov(X,Y)$\\

Variance of sum/difference of two \textbf{independent} r.v.:\\

$Var(X + Y)=Var(X)+Var(Y)$\\

$Var(X - Y)=Var(X)+Var(Y)$\\

\section{Covariance}

The Covariance is a measure of how much the values of each of two correlated random variables determine each other\\

$Cov(X,Y) = \mathbb E[(X - \mu _ X)(Y - \mu _ Y)]$ \\

$Cov(X,Y) = \mathbb E[XY] - \mathbb E[X]\mathbb E[Y] $\\

$ Cov(X,Y)= \displaystyle  \mathbb E[(X)(Y-\mu _ Y)]$\\

Possible notations:\\

$Cov(X,Y) = \sigma(X,Y) = \sigma_ {(X,Y)}$\\

Covariance is commutative:\\

$Cov(X,Y) = Cov(Y,X)$\\

Covariance with of r.v. with itself is variance:\\

$Cov(X,X) = \mathbb E[(X - \mu _ X)^2] = Var(X)$\\

Useful properties:\\

$Cov(aX + h,bY + c)= abCov(X,Y)$\\

$Cov(X,X + Y)= Var(X) + cov(X,Y)$\\

$\displaystyle  Cov(aX+ bY, Z) \displaystyle = aCov(X,Z) + bCov(Y,Z)$\\

If $Cov(X,Y) = 0$, we say that X and Y are uncorrelated. If $X$ and $Y$ are independent, their Covariance is zero. The converse is not always true. It is only true if $X$ and $Y$ form a gaussian vector, ie. any linear combination $\alpha X + \beta Y$ is gaussian for all $(\alpha,\beta) \in \mathbb{R}^2$ without $\{0,0\}$.

\section{correlation coefficient}

$\rho(X,Y) = \frac{Cov(X,Y)}{\sqrt{Var(X}Var(Y}$


\subsection{Divergence}

This takes a vector field and produces a scalar field.

It is the dot product of the vector field with the del operator.

\(div F = \nabla . F\)

Where \(\nabla =(\sum_{i=1}^n e_i\dfrac{\delta }{\delta x_i})\)

\(div F = \sum_{i=1}^n e_i\dfrac{\delta F_i}{\delta x_i}\)

\subsection{Divergence as net flow}

Divergence can be thought of as the net flow into a point.

For example, if we have a body of water, and a vector field as the velocity at any given point, then the divergence is \(0\) at all points.

This is because water is incompressible, and so there can be no net flows.

Areas which flow out are sources, while areas that flow inwards are sinks.

\subsection{Solenoidal vector fields}

If there is no divergence, then the vector field is called solenoidal.

\subsection{The Laplace operator}

Cross product of divergence with the gradient of the function.

\(\Delta f= \nabla . \nabla f\)

\(\Delta f= \sum_{i=1}^n \dfrac{\delta^2 f}{\delta x^2_i}\)


\subsection*{RA Ex. 1.3}
We consider a Monte Carlo algorithm $A$ for a problem $\Pi$ whose expected running time is at most $T(n)$ on any instance of size $n$ and that produces a correct solution with probability $\gamma(n)$. Furthermore, given a solution to $\Pi$, we can verify its correctness in time $t(n)$.

We wish to obtain a Las Vegas algorithm that always gives a correct answer to $\Pi$ and runs in expected time at most
$$
\frac{T(n) + t(n)}{\gamma(n)}
$$
This can be obtained, by repeating the Monte Carlo algorithm $A$, until a correct solution to $\Pi$ is produced.

We know, that the expected running time is at most $T(n)$ on any instance of size $n$, so the expected running time would also be $T(n)$ for each repetition in the new algorithm.
\\
Furthermore, for each repetition, we also need to verify that the solution is correct, which adds $t(n)$ to the running time of each repetition.

Since the running time for each repetition is $T(n) + t(n)$, we just need to determine, how many repetitions are needed to produce a correct solution to $\Pi$. We call this number $X$.
\\
To determine $X$, we first consider the following observations:
\\
- In each repetition, the solution produced is either correct, which we consider a success, or incorrect, which we consider a failure.
\\
- Each repetition is independent.
\\
- Each repetition has the same probability $\gamma(n)$ for producing a success (meaning a correct solution to $\Pi$).
\\
Hence, we can model the number of failures, before the one success, as a geometric distribution. Then we know from App. C.4 (C.32), that
$$
E\left[X\right] = \frac{1}{\gamma(n)}
$$
Since we have now determined $X$, we can show that the new Las Vegas algoritm runs in expected time at most
$$
E\left[\left(T(n) + t(n)\right) \cdot X \right]= \frac{T(n) + t(n)}{\gamma(n)}
$$


\subsection{Cardinality}

\subsubsection{Cardinality of finite sets}

The cardinality of a set \(s\) is shown as \(|s|\). It is the number of elements in the set. We define it formally below.

\subsubsection{Injectives, surjectives and bijectives}

Consider \(2\) sets. If there is an injective from \(a\) to \(b\) then for every element in \(a\) there is a unique element in \(b\).

If this injective exists then we say \(|a|\le |b|\).

Similarly, if there is a surjective, that is for every element in \(b\) there is a unique element in \(a\), then \(|a|\ge |b|\).

Therefore, if there is a bijection, \(|a|=|b|\), and if there is only an injective or a surjective then \(|a|<|b|\) or \(|a|>|b|\) respectively.

\subsubsection{Cardinality as a function}

Every set has a cardinality. As a result cardinality cannot be a well-defined function, for the same reason there is no set of all sets.

Cardinality functions can be defined on subsets.