The $n$-body problem is a classic one from the \textbf{Newtonian mechanics}: it consists in \textbf{resolving the gravity equations}.
However, this problem can be generalized (from the algorithmic point of view) and we can often find it in numerical simulation: this is why it is a good real case study.

At the problem beginning (the time $t$), for each body $i$, its position $q_{i}(t)$, its mass $m_i$ and its velocity $\vec{v_i}(t)$ are known.
The applied force between two bodies $i$ and $j$, at the $t$ time, is defined by:
\begin{equation}
\label{eq:force}
	\vec{f_{ij}}(t) = G.\frac{m_i.m_j}{||\vec{r_{ij}}||^2}.\frac{\vec{r_{ij}}}{||\vec{r_{ij}}||},
\end{equation}
with $G$ the gravitational constant ($G = 6,67384\times10^{-11} m^3.kg^{-1}.s^{-2}$) and $\vec{r_{ij}} = q_j(t) - q_i(t)$ the vector from body $i$ to body $j$.
The resulting force (alias the total force) for a given $i$ body, at the $t$ time, is defined by:
\begin{equation}
\label{eq:sumForces}
	\vec{F_i}(t) = \sum_{j \ne i}^{n} \vec{f_{ij}}(t) = G.m_i.\sum_{j \ne i}^{n}\frac{m_j.\vec{r_{ij}}}{||\vec{r_{ij}}||^3},
\end{equation}
with $n$ the number of bodies in space.
For the time integration, the acceleration is required. For a given body $i$, at the $t$ time, the acceleration characteristic is defined by:
\begin{equation}
\label{eq:acceleration}
	\vec{a_i}(t) = \frac{\vec{F_i}(t)}{m_i} = G.\sum_{j \ne i}^{n}\frac{m_j.\vec{r_{ij}}}{||\vec{r_{ij}}||^3}.
\end{equation}

\subsection{An approximation of the total force}

The total force $\vec{F_i}$ is given by Eq.~\ref{eq:sumForces}:
\begin{equation*}
	\vec{F_i}(t) = G.m_i.\sum_{j \ne i}^{n}\frac{m_j.\vec{r_{ij}}}{||\vec{r_{ij}}||^3}.
\end{equation*}
As bodies approach each other, \textbf{the force between them grows without bound}, which is an undesirable situation for numerical integration.
In astrophysical simulations, collisions between bodies are generally precluded; this is reasonable if the bodies represent galaxies that may pass right through each other.
Therefore, \textbf{a softening factor} $\epsilon^2 > 0$ is added, and the denominator is rewritten as follows:
\begin{equation}
\label{eq:sumForcesSoft}
	\vec{F_i}(t) \approx G.m_i.\sum_{j = 1}^{n}\frac{m_j.\vec{r_{ij}}}{(||\vec{r_{ij}}||^2 + \epsilon^2)^\frac{3}{2}}.
\end{equation}

Note the condition $j\ne i$ is no longer needed in the sum, because $\vec{f_{ii}} = 0$ when $\epsilon ^2 > 0$.
The softening factor models the interaction between two Plummer point masses: masses that behave as if they were spherical galaxies.
In effect, \textbf{the softening factor limits the magnitude of the force between the bodies}, which is desirable for numerical integration of the system state.

As before, we need to compute the acceleration in order to perform the integration over the time:
\begin{equation}
\label{eq:accelerationSoft}
	\vec{a_i}(t) = \frac{\vec{F_i}(t)}{m_i} \approx G.\sum_{j = 1}^{n}\frac{m_j.\vec{r_{ij}}}{(||\vec{r_{ij}}||^2 + \epsilon^2)^\frac{3}{2}}.
\end{equation}


\chapter{Probability}

\section{Shuffle}
Equal probability shuffle algorithm.

\subsection{Incorrect naive solution}
Swap current card $A_i$ with a random card from the entire deck $A$.
\begin{java}
for (int i = 0; i < N; i++) {
   int j = (int) Math.random()*N;
   swap(a[i], a[j]);
}
\end{java}
\begin{python}
def shuffle(A):
  n = len(A)
  for i in xrange(n):
    j = random.randrange(n)
    A[i], A[j] = A[j], A[i]
\end{python}
Consider 3 cards, the easiest proof that this algorithm does not produce a uniformly random permutation is that it generates $3^3=27$ possible plans (consider steps in plans, duplicated result included), but there are only 3! = 6 permutations. Since $27\%3 \neq 0$, there must be some permutation is that is picked too much, and some that is picked too little.
\subsection{Knuth Shuffle}
Knuth (aka Fisher-Yates) shuffling algorithm guarantees to rearrange the elements in uniformly random order.
\\
Core clues:
\begin{enumerate}
\item choose index uniformly $\in [i, N)$.
\item just like shuffling the poker card.
\end{enumerate}
\begin{java}
public void shuffle(Object[] a) {
    int N = a.length;
    for (int i = 0; i < N; i++) {
        // choose index uniformly in [i, N)
        int j = i + (int) (Math.random() * (N - i));
        swap(a[i], a[j]);
    }
}
\end{java}

\begin{python}
def shuffle(A):
  n = len(A)
  for i in xrange(n):
    j = random.randrange(i, n)
    A[i], A[j] = A[j], A[i]
\end{python}

\runinhead{Proof:}

$n!$ permutations.

\subsection{Random Maximum}
Find the index of maximum number in an array with a probability of $\frac{1}{\#maxima}$.

\runinhead{Code:}
\begin{python}
def random_max(A):
    maxa, maxa_idx, k = A[0], 0, 1
    for i in xrange(1, len(A)):
        if maxa < A[i]:
            maxa, maxa_idx, k = A[i], i, 1
        elif maxa == A[i]:
            k += 1
            if random.random() < float(1)/k:
                maxa_idx = i

    return maxa_idx

\end{python}

\runinhead{Proof:} Prove via mathematical induction.

That is, assuming it works for any array of size $N$, prove it works for any array of size $N+1$.

So, given an array of size $N+1$, think of it as a sub-array of size $N$ followed a new element at the end. By assumption, your algorithm uniformly selects one of the max elements of the sub-array... And then it behaves as follows:

If the new element is larger than the max of the sub-array, return that element. This is obviously correct.

If the new element is less than the max of the sub-array, return the result of the algorithm on the sub-array. Also obviously correct.

The only slightly tricky part is when the new element equals the max element of the sub-array. In this case, let the number of max elements in the sub-array be $k$. Then, by hypothesis, your algorithm selected one of them with probability $\frac{1}{k}$. By keeping that same element with probability $\frac{k}{k+1}$, you make the overall probability of selecting that same element equal $$\frac{1}{k} \cdot \frac{k}{k+1} = \frac{1}{k+1}$$, as desired. You also select the last element with the same probability.

To complete the inductive proof, just verify the algorithm works on an array of size 1

\section{Sampling}
\subsection{Reservoir Sampling}
Sample $k$ from $A$, where the length of $A$ is either very large or unknown or dynamic.
\begin{python}
def reservoir_sample(A, sz):
  R = A[:sz]  # sz for size

  for i in xrange(sz, len(A)):
    rv = random.randrange(0, i+1)
    if rv < sz: # rv for random variable
      R[rv] = A[i]
\end{python}

Until index $i$, with elements of length $i+1$ scanned, every element has a probability of $\frac{sz}{i+1}$ in the reservoir.

\runinhead{Random pick index.} Given an array of integers with possible duplicates, randomly output the index of a given target number.
\begin{python}
def pick(self, target):
    sz = 0
    ret = None
    for idx, val in enumerate(A):
        if val == target:
            sz += 1
            rv = random.randrange(0, sz)
            if rv == 0:
                ret = idx

    return ret
\end{python}

\section{Distribution}
\subsection{Geometric Distr}
$$
P(X=k) = (1-p)^{k-1}p
$$

Expected number of trials of get a specific outcome:
$$
E[T] = \frac{1}{p}
$$
, which is the mean of the Geometric Distr.
\subsection{Binomial Distr}
Notations:
$$
B(n, p)
$$
pmf:
$$
{n \choose k}\,p^{k}(1-p)^{n-k}
$$

\section{Expected Value}
\subsection{Dice value}
Expected value of rolling dice until getting a 3
\subsection{Coupon collector's problem}
Given n coupons, how many coupons do you expect you need to draw with replacement before having drawn each coupon at least once?

$$
E[T] = \Theta(n \lg n)
$$
, where $T$ is number of trial (i.e. time).

Let $T$ be the time to collect all. $t_i$ be the time to collect the $i$-th new different coupon. $p_i$ be the probability of collecting the $i$-th coupon after $i-1$ coupons have been collected. Observe that:
\begin{align*}
p_1 &= \frac{n}{n} \\
p_2 &= \frac{n-1}{n} \\
p_i &= \frac{n-i+1}{n}
\end{align*}
Thus,

\begin{align*}
E[T] &= \sum_{i=1}^n E[t_i] \\
&= \sum \frac{1}{p_i} \\
&= n(\frac{1}{1}+\frac{1}{2}+...+\frac{1}{n})
\end{align*}

\runinhead{Dice.} How many times must you roll a die until each side has appeared?


\subsection{Runge's phenomenon}
A natural way to approximate a given function on any interval $[a,b]$ is to use an $n$-degree polynomial $p_n(x)$ by $n+1$ equispaced  points, namely
$$
x_i=a+{b-a\over n},\quad i=0,1,2,\cdots,n.
$$
By Weierstrass' theorem, we expect a more accurate reconstruction of $f(x)$ by using more points. But this is not always true as shown in the following example.
%Consider the case where one desires to interpolate through $n+1$ equispaced points of a function $f(x)$ using the n-degree polynomial $p_n(x)$ that passes through those points. Naturally, one might expect from Weierstrass' theorem that using more points would lead to a more accurate reconstruction of $f(x)$. However, this particular set of polynomial functions $p_n(x)$ is not guaranteed to have the property of uniform convergence; the theorem only states that a set of polynomial functions exists, without providing a general method of finding one.

Consider the Runge function (a scaled version of the Witch of Agnesi)
$$
f(x)=\frac{1}{1+25x^{2}}.
$$
Runge found that if this function is interpolated at equidistant points $x_i$ between $-1$ and $1$ such that:
$$
x_{i}={\frac{2i}{n}}-1,\quad i\in \left\{0,1,\dots ,n\right\}
$$
with a polynomial $p_n(x)$ of degree $\leq n$, the resulting interpolation oscillates toward the ends of the interval, i.e. close to $-1$ and $1$. It can even be proven that the interpolation error increases (without bound) when the degree of the polynomial is increased:

$$\lim_{{n\rightarrow \infty }}\left(\max_{{-1\leq x\leq 1}}|f(x)-p_{n}(x)|\right)=+\infty.$$
This shows that high-degree polynomial interpolation at equidistant points can be troublesome.

\begin{figure}
	\begin{center}
		\includegraphics[scale=0.1]{6DL/pic/Runge.jpeg}
		\caption{Runge's phenomenon: Runge function $f(x)=\frac{1}{1+25x^{2}}$ and its polynomial interpolation $p_n(x)$.}
	\end{center}
\end{figure}

The experiment shows that  the polynomials $p_n(x)$ produced in this manner may in fact diverge away from $f(x)$ as $n$ increases. This typically occurs in an oscillating pattern that magnifies near the ends of the interpolation points. This phenomenon is attributed to Runge.

Thus, this particular set of polynomial functions $p_n(x)$ is not guaranteed to have the property of uniform convergence. In other words, Weierstrass' theorem guarantees the existence of the polynomial functions, but how to find such polynomials is not provided.


\section*{Problem 2: Locality-Sensitive Hashing (LSH) for Angle Similarity}

Suppose our set of $n$ points $D = \left\{p_1,\ldots,p_n\right\}$ are vectors in
$d$ dimensions. Our problem is: given a query point $q$ find a point $p \in D$,
which has a small angle with $q$. Recall that the angle between two vectors $a$
and $b$ is
$\cos^{-1}\left(\frac{a \cdot b}{\lVert a \rVert \lVert b \rVert} \right).$

As doing this exactly may be computationally expensive, let us try to do this
approximately with a fast algorithm. The approximate objective is as follows:
suppose there exists a point $p \in D$ which has angle less than $\theta$ with
$p$, then our goal is return a point with angle less than $c\theta$, where
$c > 1$.

Let us try to do this with LSH. Let us consider the a family of hash functions,
where $h(p) = \operatorname{sign}\left(u \cdot p\right)$ where we will sample
$u$ uniformly at random from a Gaussian (or from a unit sphere).

\begin{enumerate}
\item Provide an exact expression for
  $\mathbb{P}\left(h\left(p\right) = h\left(p^\prime\right)\right)$ based on
  some geometric relation between $p$ and $p^\prime$.

  \subsection*{Solution}

  Define
  \begin{equation}
    \operatorname{angle}\left(u, v\right) =
    \cos^{-1}\left(\frac{u \cdot v}{\lVert u \rVert \lVert v \rVert} \right),
  \end{equation}
  which is the angle between two vectors.

  Then,
  \begin{equation}
    \boxed{\mathbb{P}\left(h\left(p\right) = h\left(p^\prime\right)\right)
      = 1 - \frac{\operatorname{angle}\left(p,p^\prime\right)}{\pi}.}
  \end{equation}
\item Provide an expression for $P_1$ and $P_2$ in terms of $\theta$ and
  $c\theta$. Note that since we want a small angle, we should use:
  \begin{enumerate}
  \item If $\operatorname{angle}\left(p,p^\prime\right) < \theta$, then
    $\mathbb{P}\left(h\left(p\right) = h\left(p^\prime\right)\right) \geq P_1$.
  \item If $\operatorname{angle}\left(p,p^\prime\right) > c\theta$, then
    $\mathbb{P}\left(h\left(p\right) = h\left(p^\prime\right)\right) \leq P_2$.
  \end{enumerate}

  \subsection*{Solution}

  If $\operatorname{angle}\left(p,p^\prime\right) < \theta$, then
  \begin{align*}
    \mathbb{P}\left(h\left(p\right) = h\left(p^\prime\right)\right)
    &= 1 - \frac{\operatorname{angle}\left(p,p^\prime\right)}{\pi}
    \geq 1 - \frac{\theta}{\pi},
  \end{align*}
  so $\displaystyle\boxed{P_1 = 1 - \frac{\theta}{\pi}.}$

  If $\operatorname{angle}\left(p,p^\prime\right) > c\theta$, then
  \begin{align*}
    \mathbb{P}\left(h\left(p\right) = h\left(p^\prime\right)\right)
    &= 1 - \frac{\operatorname{angle}\left(p,p^\prime\right)}{\pi}
    \leq 1 - \frac{c\theta}{\pi},
  \end{align*}
  so $\displaystyle\boxed{P_2 = 1 - \frac{c\theta}{\pi}.}$
\item Provide expressions for query time for point $q$, the space to store the
  hash tables, and the construction time of our datastructure.

  \subsection*{Solution}

  Suppose we have $L$ hash functions. If we use the algorithm discussed in
  class, to query a point, we need to compute $L$ hashes. Then, up to $3$ times,
  we iterate through the buckets: for each bucket, we choose a point and check
  how close it is to $q$; if it is $c\theta$ close, we stop. The worst case is
  that we decide there exists no point that is $\theta$ close to $q$. In this
  case, we iterate through the $L$ buckets $3$ times, so the time complexity is
  $O(L)$.

  For the space needed to store the hash tables, we need to store $L$ bits for
  each point, so the space needed is $O(nL)$

  For construction, we need to compute $L$ hashes for each point, so the
  computational complexity is $O(nL)$ as well.
\end{enumerate}


\subsection{Multivariable Gaussian distribution}

\subsubsection{Definition}

For univariate:

\(x \sim N(\mu, \sigma^2 )\)

We define the multivariate gaussian distribution as the distribution where any linear combination of components are gaussian.

For multivariate:

\(X \sim N(\mu, \Sigma )\)

Where \(\mu \) is now a vector, and \(\Sigma \) is the covariance matrix.

Density function is :

\(f_x=\dfrac{1}{\sqrt {(2\pi )^n|\Sigma |}} e^{-\dfrac{1}{2}(x-\mu )^T\Sigma^{-1}(x-\mu)}\)

For normal gaussian it is:

\(f_x=\dfrac{1}{\sqrt {2\pi |\sigma^2}} e^{-\dfrac{1}{2\sigma^2}(x-\mu )^2)}\)

This is the same wher \(n=1\).

\subsubsection{Singular Gaussians}

Need det \(|\Sigma |\) and \(\Sigma^{-1}\). These rely on the covariance matrix not being degenerate.

If the covariance matrix is degenerate we can instead use the pseudo inverse, and the pseudo determinant.


% Cholesky Decomposition
\section{$\mathbf{A} = \mathbf{L}\mathbf{L}^{\dagger}$ Cholesky Decomposition}
\label{sec:cholesky}

The Cholesky Decomposition exists when a matrix is hermitian and positive-definite. It expresses the matrix $\mathbf{A}$ as:

\begin{equation}
\mathbf{A} = \mathbf{L}\mathbf{L^\dagger}
\end{equation}

Where $\mathbf{L}$ is a lower-triangular matrix with positive, real diagonal entries. When $\mathbf{A}$ is real, then so is $\mathbf{L}$. The Cholesky decomposition enables fast solution of a linear system, but it can also be used to create correlated random variables in Monte Carlo simulations.

\subsection{Creating Correlated Random Variables}
Let $\mathbf{u}_t$ be a vector of uncorrelated samples with mean 0 and 	standard deviation 1. If the covariance matrix of the system to be simulated is  $\mathbf{\Sigma}$ with Cholesky decomposition $\mathbf{\Sigma} = \mathbf{LL}^\dagger$, then the vector $\mathbf{v}_t = \mathbf{Lu}_t$ has the desired covariance.

\begin{figure}
\centering
\includegraphics[scale=0.5]{cholesky1.png}
\includegraphics[scale=0.5]{cholesky2.png}
\caption{Creating correlated random variables from uncorrelated random variables using the Cholesky decomposition of the covariance matrix. The 5 uncorrelated random variables are sampled from a standard normal distribution. It is difficult to see a difference between the correlated and uncorrelated random walks.}
\end{figure}


%
% Chapter 4.2
%

\section*{4.2 Definite Integral}

\subsection*{Definition of a Definite Integral}

If \(f\) is a function defined for \(a \leq x \leq b\), we divide the interval \([a, b]\) into \(n\) subintervals of equal width \(\Delta x=\frac{(b-a)}{n}\). We let \(x_0\) (\(=a\)), \(x_1\), \(x_2\), \ldots,\(x_n\) (\(=b\)) be the endpoints of these subintervals and we let \(x_1^*, x_2^*, \ldots, x_n^*\) be any sample points in these subintervals, so \(x_i^*\) lies in the \(i\)th subinterval \([x_{i-1}, x_i]\). Then the definite integral of \(f\) from \(a\) to \(b\) is
$$ \int_a^b f(x)dx=\lim_{n \to \infty} \sum_{i=1}^n f(x_i^*)\Delta x $$
provided that this limit exists and gives the same value for all possible choices of sample points. If it does exist, we say that \(f\) is \textbf{integrable} on \([a, b]\).

\subsection*{Precise Meaning of the Limit that Defines the Integral}

For every number \(\epsilon > 0\) there is an integer \(N\) such that
$$ \left|\int_a^b f(x)dx- \sum_{i=1}^n f(x_i^*)\Delta x\right|< \epsilon $$
for every integer \(n > N\) and for every choice of \(x_i^*\) in \([x_{i-1}, x_i]\).

\subsubsection*{Theorem}

If \(f\) is continuous on \([a, b]\), or if \(f\) has only a finite number of jump discontinuities, then \(f\) is integrable on \([a, b]\); that is, the definite integral \(\int_a^b f(x)dx\) exists.

\subsubsection*{Theorem}

If \(f\) is integrable on \([a, b]\), then
$$ \int_a^b f(x)dx=\lim_{x \to \infty} \sum_{i=1}^n f(x_i) \Delta x $$
where
$$ \Delta x=\frac{b-a}{n} \quad \text{ and } \quad x_i=a+i \Delta x $$

\subsection*{Formulae for Sums of Powers of Positive Integers}

$$ \sum_{i=1}^n i=\frac{n(n+1)}{2} $$
$$ \sum_{i=1}^n i^2=\frac{n(n+1)(2n+1)}{6} $$
$$ \sum_{i=1}^n i^3= {\left [ \frac{n(n+1)}{2} \right]}^2 $$

\subsection*{Formulae for Sigma Notation}

$$ \sum_{i=1}^n c=nc \quad \text{where } c \text{ is a constant} $$
$$ \sum_{i=1}^n ca_i=c \sum_{i=1}^n a_i \quad \text{where } c \text{ is a constant} $$
$$ \sum_{i=1}^n (a_i+b_i)=\sum_{i=1}^n a_i + \sum_{i=1}^n b_i $$
$$ \sum_{i=1}^n (a_i-b_i)=\sum_{i=1}^n a_i - \sum_{i=1}^n b_i $$

\subsection*{Midpoint Rule}

$$ \int_a^b f(x)dx \approx \sum_{i=1}^{n} f(\overline{x}_i) \Delta x = \Delta x [f(\overline{x}_1)+\cdots+f(\overline{x}_n)]$$
where
$$ \Delta x = \frac{b-a}{n} \text{ and } \overline{x}_i = \frac{1}{2}(x_{i-1} + x_i) \text{ = midpoint of } [x_{i-1},x_i] $$

\subsection*{Properties of the Definite Integral}

$$ \int_b^a f(x)dx = -\int_a^b f(x)dx $$
$$ \int_a^a f(x)dx = 0 $$
$$ \int_a^b c dx = c(b-a) \quad \text{where } c \text{ is any constant} $$
$$ \int_a^b [f(x)+g(x)] dx = \int_a^b f(x)dx + \int_a^b g(x)dx $$
$$ \int_a^b cf(x) dx = c\int_a^b f(x) dx \quad \text{where } c \text{ is any constant} $$
$$ \int_a^b [f(x)-g(x)] dx = \int_a^b f(x)dx - \int_a^b g(x)dx $$
$$ \int_a^c f(x) dx + \int_c^b f(x)dx = \int_a^b f(x)dx $$

\subsubsection*{Comparison Properties of the Definite Integral}

$$ \text{If } f(x) \geq 0 \text{ for } a \leq x \leq b \text{, then} \int_a^b f(x)dx \geq 0 $$
$$ \text{If } f(x) \geq g(x) \text{ for } a \leq x \leq b \text{, then} \int_a^b f(x)dx \geq \int_a^b g(x)dx $$
$$ \text{If } m \leq f(x) \leq M \text{ for } a \leq x \leq b \text{, then } m(b-a) \leq \int_a^b f(x)dx \leq M(b-a) $$


\section{Matrix examples}
\subsection{Geometric transformations}
\textbullet \textit{Scaling}: $y = Ax$ with $A = aI$ stretches a vector by the factor $|a|$ (or shrinks it when $|a| < 1$), and it flips the vector (reverses its direction) if $a < 0$\\
\textbullet \textit{Dilation}: $y = Dx$, where D is a diagonal matrix, $D = diag(d1,d2)$. Stretches the vector x by different factors along the two different axes. (Or shrinks, if $|d_i| < 1$, and flips, if $di < 0$.)\\
\textbullet \textit{Rotation Matrix}:
\[
y =
\begin{bmatrix}
    cos \theta & -sin \theta \\
    sin \theta & cos \theta
\end{bmatrix}x
\]
\textbullet \textit{Reflection}
Suppose that y is the vector obtained by reflecting x through the line that passes through the origin, inclined $\theta$ radians with respect to horizontal.
\[
y =
\begin{bmatrix}
    cos (2\theta) & sin(2\theta) \\
    sin (2\theta) & -cos (2\theta)
\end{bmatrix}x
\]
\subsection{Selectors}
An $m \times n$ selector matrix A is one in which each row is a unit vector (transposed):
\[
\begin{bmatrix}
    e^T_{k_1}\\
    .\\
    .\\
    e^T_{k_m}\\
\end{bmatrix}
\]
When it multiplies a vector, it simply copies the $k_i$th entry of x into the $i$th entry of $y = Ax$:\\
$y = (x_{k_1},x_{k_2},...,x_{k_m})$

\subsection{Incidence matrix}
\textbf{Directed graph}: A \textit{directed graph} consists of a set of \textit{vertices} (or nodes), labeled 1,...,n, and a set of \textit{directed edges} (or branches), labeled 1,...,m. \\
$
A_{ij} = \left\{
  \begin{array}{@{}ll@{}}
    1, & \text{edge j points to node i} \\
    -1, & \text{edge j points from node i}\\
    0, & \text{otherwise}
  \end{array}\right.
$\\
\subsection{Convolution}


An exponential family distribution $p(x | \theta)$
is parameterized by
natural parameter function $\eta(\theta)$,
sufficient statistics function $t(x)$,
and log-partition function  $\log Z(\eta(\theta))$,
such that
\begin{align*}
    p(x | \theta) = \exp{\left\langle\eta(\theta), t(x)\right\rangle - \log Z(\eta(\theta))}
\end{align*}

\section{Categorical distribution}
\label{sec:stats-categorical}
The categorical distribution is a distribution
over discrete sets (one-hot vectors $z$) in $R^K$,
and is parametrized by a $K$-simplex vector $\pi$.

\begin{align*}
    p(z | \pi) = \pi^Tz
\end{align*}
\subsection{Exponential family parametrization}
\begin{align*}
    \eta(\pi) &= \log \pi \\
    t(z) &= z \\
    \log Z(\eta(\pi)) &= \log\sum_{k = 1}^K \exp{\eta(\pi)_k} = 0\\
\end{align*}

\section{Dirichlet distribution}
\label{sec:stats-dirichlet}
The Dirichlet distribution is a distribution
over the $K$-simplex
and is parametrized by a vector $\alpha$.

\begin{align*}
    p(\pi | \alpha) = \frac{\Gamma\left(\sum_k \alpha_k\right)}{\prod_k\Gamma(\alpha_k)}\prod_{k = 1}^K \pi_k^{\alpha_k - 1}
\end{align*}

\subsection{Exponential family parametrization}
\begin{align*}
    \eta(\alpha) &= \alpha - 1\\
    t(\pi) &= \log \pi \\
    \log Z(\eta(\alpha)) &= \sum_{k = 1}^K \log \Gamma(\alpha_k) - \log \Gamma\left(\sum_{k = 1}^K \alpha_k\right)
\end{align*}

\section{Gaussian distribution}
\label{sec:stats-gaussian}

The Gaussian distribution is a distribution
distribution over vectors in $\R^d$,
and is parametrized by mean $\mu$ and covariance $\Sigma$.

\begin{align}
        p(x | \mu, \Sigma) = \frac{1}{(2\pi)^{\frac{d}{2}}|\Sigma|^{\frac{1}{2}}} \exp{(x - \mu)^T\Sigma^{-1}(x - \mu)}
\end{align}

\subsection{Exponential family parametrization}

\begin{align*}
    \eta(\mu, \Sigma) &= \begin{bmatrix}-\frac{1}{2}\Sigma^{-1} \\ \Sigma^{-1}\mu\end{bmatrix} \\
    t(x) &= \begin{bmatrix}xx^T \\ x\end{bmatrix} \\
    \log Z(\eta(\mu, \Sigma)) &= \frac{1}{2}\mu^T\Sigma^{-1}\mu + \frac{1}{2}\log |\Sigma| + \frac{d}{2}\log 2\pi
\end{align*}

\section{Inverse-Wishart (IW) distribution}
\label{sec:stats-iw}

The inverse-Wishart (IW) distribution is a
distribution over positive definite matrices in $\R^{d \times d}$
and is parametrized by scale matrix $\Psi$ and degrees of freedom $\nu$.

\begin{align}
        p(\Sigma | \Psi, \nu) = \frac{|\Psi|^{\frac{\nu}{2}}}{2^{\frac{\nu d}{2}}\Gamma_d(\frac{\nu}{2})}|\Sigma|^{\frac{-(\nu + d + 1)}{2}} \exp{-\frac{1}{2}\tr(\Psi\Sigma^{-1})}
\end{align}

\subsection{Exponential family parametrization}

\begin{align*}
    \eta(\Psi, \nu) &= \begin{bmatrix}\Psi \\ \nu + d + 1\end{bmatrix} \\
    t(\Sigma) &= \begin{bmatrix}-\frac{1}{2}\Sigma^{-1} \\ -\frac{1}{2}\log |\Sigma|\end{bmatrix} \\
    \log Z(\eta(\Psi, \nu)) &= -\frac{\nu}{2}\log |\Psi| + \frac{\nu d}{2}\log 2 + \log \Gamma_d\left(\frac{\nu}{2}\right)
\end{align*}

\section{Normal-inverse-Wishart (NIW) distribution}
\label{sec:stats-niw}

The Normal-inverse-Wishart (NIW) is a
distribution over the tuple of vectors in $\R^d$ and
positive definite matrices in $\R^{d \times d}$
and is parametrized by
scale matrix $\Psi$,
mean $\mu_0$,
scale $\kappa$,
and degrees of freedom $\nu$.

\begin{align}
        p(\mu, \Sigma | \Psi, \mu_0, \kappa, \nu) &= \IW(\Sigma|\Psi,\nu)\N(\mu|\mu_0,\frac{1}{\kappa}\Sigma) \\
\end{align}

\subsection{Exponential family parametrization}
\begin{align*}
    \eta(\Psi, \mu_0, \kappa, \nu) &= \begin{bmatrix}\Psi + \kappa\mu_0\mu_0^T\\\kappa \mu_0\\\kappa\\\nu + d + 1\end{bmatrix} \\
    t(\mu, \Sigma) &= \begin{bmatrix}-\frac{1}{2}\Sigma^{-1}\\\Sigma^{-1}\mu\\-\frac{1}{2}\mu^T\Sigma^{-1}\mu\\-\frac{1}{2}\log|\Sigma|\end{bmatrix} \\
    \log Z(\eta(\Psi, \nu)) &= -\frac{\nu}{2}\log |\Psi| + \frac{\nu d}{2}\log 2 + \log \Gamma_d\left(\frac{\nu}{2}\right) - \frac{d}{2} \log \kappa
\end{align*}

\section{Matrix normal (MN) distribution}
\label{sec:stats-mn}

The matrix normal (MN) is
a distribution over $\R^{m \times n}$
matrices, parametrized by
mean $M_0 \in \R^{m \times n}$,
row covariance $S \in \R^{m \times m}$,
and column covariance $V \in \R^{n \times n}$ .

\begin{align}
        p(M | M_0, S, V) &= \frac{1}{(2\pi)^{\frac{mn}{2}}|S|^{\frac{n}{2}}|V|^{\frac{m}{2}}} \exp{-\frac{1}{2}\tr\left(V^{-1}(M - M_0)^TS^{-1}(M - M_0)\right)}
\end{align}

\subsection{Exponential family parametrization}
\begin{align*}
    \eta(M_0, S, V) &= \begin{bmatrix}-\frac{1}{2}(V \otimes S)^{-1}\\(V \otimes S)^{-1}\vec(M_0)\end{bmatrix} \\
    t(M) &= \begin{bmatrix}\vec(M)\vec(M)^T\\\vec(M)\end{bmatrix} \\
    \log Z(\eta(M_0, S, V)) &= \frac{mn}{2}\log 2\pi + \frac{n}{2}\log|S| + \frac{m}{2}\log|V| + \frac{1}{2}\tr(V^{-1}M_0^TS^{-1}M_0)
\end{align*}

\section{Matrix normal inverse-Wishart (MNIW) distribution}
\label{sec:stats-mniw}

The matrix normal inverse Wishart (MNIW) is
a distribution over the tuple of $\R^{m \times n}$
matrices and $\R^{m \times m}$ covariance matrices, parametrized by
scale matrix $\Psi$,
mean $M_0 \in \R^{m \times n}$,
column covariance $V \in \R^{n \times n}$,
and degrees of freedom $\nu$.

\begin{align}
        p(\Sigma, M | \Psi, M_0, V, \nu) &= \IW(\Sigma|\Psi, \nu) \times \MN(M|M_0, \Sigma, V) \\
        &= \frac{|\Psi|^{\frac{\nu}{2}}}{2^{\frac{\nu m}{2}}\Gamma_m(\frac{\nu}{2})}|\Sigma|^{\frac{-(\nu + m + 1)}{2}} \exp {-\frac{1}{2}\tr(\Psi\Sigma^{-1})} \\
        &\times \frac{1}{(2\pi)^{\frac{mn}{2}}|\Sigma|^{\frac{n}{2}}|V|^{\frac{m}{2}}} \exp{-\frac{1}{2}\tr\left(V^{-1}(M - M_0)^T\Sigma^{-1}(M - M_0)\right)}
\end{align}

\subsection{Exponential family parametrization}
\begin{align*}
    \eta(\Psi, M_0, V, \nu) &= \begin{bmatrix}\Psi + M_0V^{-1}M_0^T\\M_0V^{-1}\\V^{-1}\\\nu + m + n + 1\end{bmatrix} \\
    t(\Sigma, M) &= \begin{bmatrix}-\frac{1}{2}\Sigma^{-1}\\\Sigma^{-1}M\\-\frac{1}{2}M^T\Sigma^{-1}M\\-\frac{1}{2}\log|\Sigma|\end{bmatrix} \\
    \log Z(\eta(\Psi, M_0, V, \nu)) &= \frac{m}{2}\log|V| + \frac{mn}{2}\log 2\pi - \frac{\nu}{2}\log |\Psi| + \frac{\nu m}{2}\log 2 + \log \Gamma_d\left(\frac{\nu}{2}\right)
\end{align*}


\section{Tikhonov Regularization}
\label{sec:tikhonov_regularization}
The goal of the optimization of the ESN output weights is to find a $\wmatr{out}$
such that for a given state $x_t$ we can produce a prediction of the next time step:
\begin{equation}
  y_t = \wmatr{out} x_t
\end{equation}

By collecting a number of states $ \matr{X} = (x_1, ..., x_T)$, that are produced by the inputs
$u_1, ..., u_T$, we can write
\begin{equation}
  \matr{D} = \wmatr{out} \matr{X},
\end{equation}
with $\matr{D} = (d_1, ..., d_T)$ as the conactenated desired outputs.
The optimal output weights can then be found by simply solving the overdetermined
system:
\begin{equation}
  \wmatr{out} = \matr{D} \XT (\matr{X} \XT)^{-1}
\end{equation}

With this simple least squares solution, the output weights are very prone to
overfitting, which is why a regularized version, also known as Tikhonov Regularization,
often yields more general results, that let the ESN perform much better in the
freely running mode.

Tikhonov Regularization minimizes the function $\Phi$:
\begin{equation}
  \label{eq:phi}
  \Phi(\wmatr{out}) = || \matr{D} - \wmatr{out} \matr{X} ||^2
                      + \beta^2 || \wmatr{out} ||^2,
\end{equation}
where the first term represents the \emph{misfit} of the outputs to the target
and the second term introduces a the regularization.
A larger coefficient $\beta$ will favor smaller output weights in the solution,
which prevents them from overfitting.

Eq.~\ref{eq:phi} can be written as:
\begin{equation}
  \Phi(\wmatr{out}) = \bigg | \bigg|
                      \vect{\matr{D}}{0}
                    - \wmatr{out} \vect{\matr{X}}{\beta \matr{I}}
                      \bigg | \bigg| ^2,
\end{equation}
which can be solved by:

\begin{align}
  \vect{\matr{D}}{0} &= \wmatr{out} \XbI \\
  \vect{\matr{D}}{0} \XbI^{\text{T}} &= \wmatr{out} \XbI \XbI^{\text{T}}\\
  \vect{\matr{D}}{0} \XbI^{\text{T}} \bigg[\XbI \XbI^{\text{T}}\bigg]^{-1} &= \wmatr{out} \\
  \wmatr{out} &= \matr{D} \XT (\matr{X}\XT + \beta^2 \matr{I})^{-1}
\end{align}


\subsection{Logistic function (AKA sigmoid, logit)}

\subsubsection{The function}

\(\sigma (z)=\dfrac{1}{1+e^{-z}}\)

The range of this activation is between \(0\) and \(1\).

\subsubsection{The derivative}

\(\sigma '(z)=\dfrac{e^{-z}}{(1+e^{-z})^2}\)

\(\sigma '(z)=\sigma (z)\dfrac{1+e^{-z}-1}{1+e^{-z}}\)

\(\sigma '(z)=\sigma (z)[1-\sigma (z)]\)

\subsubsection{Notes}


\section{OLS}

Given two random variables $X$ and $Y$, how can we predict the values of $Y$ given $X$?

Let us consider $(X_1, Y_1), \ldots, (X_n, Y_n) \sim^{iid } \mathbb{P}$ where $\mathbb{P}$ is an unknown joint distribution. $\mathbb{P}$ can be described entirely by:

\begin{align*}
g(X) = \int f(X, y)dy\\
h(Y|X=x) = \frac{f(x, Y)}{g(x)}
\end{align*}

where $f$ is the joint PDF, $g$ the marginal density of $X$ and $h$ the conditional density. What we are interested in is $h(Y|X)$.

\textbf{Regression function:} For a partial description, we can consider instead the conditional expection of $Y$ given $X=x$:

\begin{align*}
x \mapsto f(x) = \mathbb{E}[Y | X=x] = \int yh(y|x)dy
\end{align*}

We can also consider different descriptions of the distribution, like the median, quantiles or the variance.\\
\textbf{Linear regression:} trying to fit any function to $\mathbb{E}[Y | X=x]$ is a nonparametric problem; therefore, we restrict the problem to the tractable one of linear function:
\begin{align*}
f: x \mapsto a + bx
\end{align*}

\textbf{Theoretical linear regression:} let $X, Y$ be two random variables with two moments such as $\mathbb{V}[X] > 0$. The theoretical linear regression of $Y$ on $X$ is the line $a^{*} + b^{*}x$ where

\begin{align*}
(a^{*}, b^{*}) = argmin_{(a, b) \in \mathbb{R}^2}\mathbb{E}\left[(Y - a - bX)^2\right]
\end{align*}

Which gives:

\begin{align*}
b^{*} = \frac{Cov(X, Y)}{\mathbb{V}[X]}, \quad a^{*} = \mathbb{E}[Y] - b^{*} \mathbb{E}[X]
\end{align*}

\textbf{Noise:} we model the noise of $Y$ around the regression line by a random variable $\varepsilon = Y - a^{*} - b^{*} X$, such as:

\begin{align*}
\mathbb{E}[\varepsilon] = 0, \quad Cov(X, \varepsilon) = 0
\end{align*}

We have to estimate $a^{*}$ and $b^{*}$ from the data. We have $n$ random pairs $(X_1, Y_1), \ldots, (X_n, Y_n) \sim_{iid} (X, Y)$ such as:

\begin{align*}
Y_i = a^{*} + b^{*} X_i + \varepsilon_i
\end{align*}

The \textbf{Least Squares Estimator (LSE)} of $(a^{*}, b^{*})$ is the minimizer of the squared sum:

\begin{align*}
(\hat{a}_n, \hat{b}_n) = argmin_{(a, b) \in \mathbb{R}^2}\sum_{i=1}^n(Y_i - a - bX_i)^2
\end{align*}

The estimators are given by:
\begin{align*}
\hat{b}_n = \frac{\overline{XY} - \bar{X}\bar{Y}}{\overline{X^2} - \bar{X}^2}, \quad \hat{a}_n = \bar{Y} - \hat{b}_n \bar{X}
\end{align*}

The \textbf{Multivariate Regression} is given by:

\begin{align*}
Y_i = \sum_{j=1}^pX_i^{(j)}\beta_j^{*} + \varepsilon_i= \underbrace{X_i^\top}_{1 \times p}\underbrace{\beta^{*}}_{p \times 1} + \varepsilon_i
\end{align*}

We can assuming that the $X_i^{(1)}$ are 1 for the intercept.

\begin{itemize}
\item If $\beta^{*} = (a^{*}, b^{*}\top)^\top$, $\beta_1^{*} = a^{*}$ is the intercept.
\item the $\varepsilon_i$ is the noise, satisfying $Cov(X_i, \varepsilon_i) = 0$
\end{itemize}

The \textbf{Multivariate Least Squares Estimator (LSE)} of $\beta^{*}$ is the minimizer of the sum of square errors:
\begin{align*}
\hat{\beta} = argmin_{\beta \in \mathbb{R}^p}\sum_{i=1}^n(Y_i - X_i^\top\beta)^2
\end{align*}

\textbf{Matrix form:} we can rewrite these expressions. Let $Y = (Y_1, \ldots, Y_n)^\top \in \mathbb{R}^n$, and $\epsilon = (\varepsilon_1, \ldots, \varepsilon_n)^\top$.

Let
\begin{align*}
X = \begin{pmatrix} X_1^\top \\ \vdots \\ X_n^\top \end{pmatrix} \in \mathbb{R}^{n \times p}
\end{align*}

$X$ is called the **design matrix**. The regression is given by:

\begin{align*}
Y = X\beta^{*} + \epsilon
\end{align*}

and the LSE is given by:

\begin{align*}
\hat{\beta} = argmin_{\beta \in \mathbb{R}^p} \|Y - X\beta\|^2_2
\end{align*}

Let us suppose $n \geq p$ and $rank(X) = p$. If we write:

\begin{align*}
F(\beta)  = \|Y - X\beta\|^2_2 = (Y - X\beta)^\top(Y - X\beta)
\end{align*}

Then:

\begin{align*}
\nabla F(\beta) = 2 X^\top(Y - X\beta)
\end{align*}

\textbf{Least squares estimator}: setting $\nabla F(\beta) = 0$ gives us the expression of $\hat{\beta}$:

\begin{align*}
\hat{\beta} = (X^\top X)^{-1}X^\top Y
\end{align*}

**Geometric interpretation**: $X\hat{\beta}$ is the orthogonal projection of $Y$ onto the subspace spanned by the columns of $X$:

$$ X\hat{\beta} = PY$$

where $P = X(X^\top X)^{-1}X^\top$ is the expression of the projector.

**Statistic inference**: let us suppose that:

* The design matrix $X$ is deterministic and $rank(X) = p$.
* The model is **homoscedastic**: $\varepsilon_1, \ldots, \varepsilon_n$ are i.i.d.
* The noise is Gaussian: $\epsilon \sim N_n(0, \sigma^2I_n)$.

We therefore have:

$$Y \sim N_n(X\beta^{*}, \sigma^2 I_n)$$

Properties of the LSE:

$$\hat{\beta} \sim N_p(\beta^{*}, \sigma^2(X^\top X)^{-1})$$

The quadratic risk of $\hat{\beta}$ is given by:

$$\mathbb{E}\left[\|\hat{\beta} - \beta^{*}\|^2_2\right] = \sigma^2 Tr \left((X^\top X)^{-1}\right)$$

The prediction error is given by:

$$\mathbb{E}\left[\|Y - X\hat{\beta}\|^2_2\right] = \sigma^2(n - p)$$

The unbiased estimator of $\sigma^2$ is:

$$\hat{\sigma^2} = \frac{1}{n-p}\|Y - X\hat{\beta}\|^2_2 = \frac1{n-p}\sum_{i=1}^n\hat{\varepsilon}_i^2$$

By **Cochran's Theorem**:

$$ (n-p)\frac{\hat{\sigma^2}}{\sigma^2} \sim \chi^2_{n-p}, \quad \hat\beta \perp \hat{\sigma^2}$$

**Significance test**: let us test $H_0: \beta_j = 0$ against $H_1: \beta_j \neq 0$. Let us call

$$\gamma_j = \left((X^\top X)^{-1}\right)_{jj} > 0$$

then:

$$\frac{\hat{\beta}_j- \beta_j}{\sqrt{\hat{\sigma^2}\gamma_j}} \sim t_{n-p}$$

We can define the test statistic for our test:

$$T_n^{(j)} = \frac{\hat{\beta}_j}{\sqrt{\hat{\sigma^2}\gamma_j}}$$

The test with non-asymptotic level $\alpha$ is given by:

$$\psi_\alpha^{(j)} = \textbf{1}\{|T_n^{(j)}| > q_{\alpha/2}(t_{n-p})\}$$

**Bonferroni's test**: if we want to test the significance level of multiple tests at the same time, we cannot use the same level $\alpha$ for each of them. We must use a stricter test for each of them. Let us consider $S \subseteq \{1, \ldots, p\}$. Let us consider

$$H_0: \forall j \in S, \beta_j = 0, \quad H_1: \exists j \in S, \beta_j \neq 0$$

The *Bonferroni's test* with significance level $\alpha$ is given by:

$$\psi_\alpha^{(S)} = \max_{j \in S}\psi_{\alpha/K}^{(j)}$$

where $K = |S|$. The rejection region therefore is the union of all rejection regions:

$$R_\alpha^{(S)} = \bigcup_{j \in S}R_{\alpha/K}^{(j)}$$

This test has nonasymptotic level at most $\alpha$:

$$\P_{H_0}\left[R_\alpha^{(S)}\right] \leq \sum_{j\in S}\P_{H_0}\left[R_{\alpha/K}^{(j)}\right] = \alpha$$

This test also works for implicit testing (for example, $\beta_1 \geq \beta_2$).


% !TEX root = main.tex

\section{Events}\label{sec:events}

A set is a collection of distinct elements. If $\omega$ is an element of the set $A$, we denote this by $\omega\in A$.
The set containing no elements is called the \emph{empty set} and denoted by $\emptyset$.

\begin{definition}[Set relations]
Let $A$ and $B$ be sets.
\ben
\it $A$ is a \emph{subset} of $B$ if $\omega\in B$ for every $\omega\in A$. This is denoted by $A\subseteq B$.
\it $A$ is \emph{equal} to $B$ if $A\subseteq B$ and $B\subseteq A$. This is denoted by $A=B$.
\it $A$ is a \emph{proper subset} of $B$ if $A\subseteq B$ and $A\neq B$. This is denoted by $A\subset B$.
\it $A$ and $B$ are \emph{disjoint} (or \emph{mutually exclusive}) if $A\cap B=\emptyset$.
\een
\end{definition}

\begin{definition}[Set operations]
Let $\Omega$ be a set, and let $A$ and $B$ be subsets of $\Omega$.
\ben
\it The \emph{complement} of $A$ (relative to $\Omega$) is the set
$A^c=\{\omega\in\Omega:\omega\notin A\}$.
\it The \emph{union} of $A$ and $B$ is the set
$A\cup B = \{\omega\in\Omega: \omega\in A \text{ or }\omega\in B\}$.
\it The \emph{intersection} of $A$ and $B$ is the set
$A\cap B = \{\omega\in\Omega: \omega\in A \text{ and }\omega\in B\}$.
\een
\end{definition}

Table~\ref{tab:sets_and_logic} shows the connection between set theory and logic.
\begin{table}[ht]
\centering
\begin{tabular}{|c|c||c|c|c|} \hline
Set Theory 		& 			& Logic			&		& \\ \hline
Union			& $A\cup B$	& Disjunction 	& OR 	& $\lor$	\\
Intersection	& $A\cap B$	& Conjunction	& AND 	& $\land$\\
Complement		& $A^c$		& Negation		& NOT 	& $\lnot$	\\ \hline
\end{tabular}
\caption{Correspondence between set operations and logical connectives.\label{tab:sets_and_logic}}
\end{table}

%\begin{definition}[Set algebra]
%\ben
%\it Commutative property.
%\bit
%\it $A\cup B = B\cup A$,
%\it $A\cap B = B\cap A$.
%\eit
%\it Associative property.
%\bit
%\it $(A\cup B)\cup C = A\cup (B\cup C)$,
%\it $(A\cap B)\cap C = A\cap (B\cap C)$.
%\eit
%\it Distributive property.
%\bit
%\it $A\cup (B\cap C) = (A\cup B)\cap(A\cup C)$,
%\it $A\cap (B\cup C) = (A\cap B)\cup(A\cap C)$.
%\eit
%\een
%\end{definition}

\begin{definition}[Set algebra]
The union and intersection operations have the following properties.
\bit
\it The \emph{commutative} property: $A\cup B = B\cup A$ and $A\cap B = B\cap A$.
\it The \emph{associative} property: $(A\cup B)\cup C = A\cup (B\cup C)$ and $(A\cap B)\cap C = A\cap (B\cap C)$.
\it The \emph{distributive} property: $A\cup (B\cap C) = (A\cup B)\cap(A\cup C)$ and $A\cap (B\cup C) = (A\cap B)\cup(A\cap C)$.
\eit
Note that a statement such as $A\cup B\cap C$ is ambiguous.
\end{definition}

\begin{exercise}
\begin{questions}
\question
The \emph{set difference} $A\setminus B$ is the set containing those elements of $A$ that are not contained in $B$. Express $A\setminus B$ using only the intersection and complementation operations.
\question
Prove De Morgan's laws: $(A\cup B)^c = A^c\cap B^c$ and $(A\cap B)^c = A^c\cup B^c$.
\end{questions}
\end{exercise}

\subsection{Countable unions and intersections}
Let $A_1,A_2,\ldots$ be a countable family of sets. The union and intersection of this countable family are defined by
\begin{align*}
\medcup_{i=1}^{\infty} A_i
	& = \{\omega:\omega\in A_i\text{ for some } i=1,2,\ldots\} \\[2ex]
\medcap_{i=1}^{\infty} A_i
	& = \{\omega:\omega\in A_i\text{ for all } i=1,2,\ldots\}
\end{align*}

\begin{exercise}
Show that De Morgan's laws hold for countable unions and intersections:
\[
\big(\medcup_{i=1}^{\infty}A_i\big)^c = \medcap_{i=1}^{\infty}A_i^c
\quad\text{and}\quad
\big(\medcap_{i=1}^{\infty}A_i\big)^c = \medcup_{i=1}^{\infty}A_i^c.
\]
\end{exercise}

%%-------------------------------------------------
%\section{Sample spaces}\label{sec:sample_spaces}

\begin{definition}
\ben
\it A \emph{random experiment} is any process of observation or measurement whose result is uncertain.
\it The result of a random experiment is called the \emph{outcome}.
\it The \emph{sample space} of a random experiment is the set of all possible outcomes.
\it A \emph{random event} is a subset of the sample space.
\een
\end{definition}

Sample spaces can be \emph{finite}, \emph{countably infinite} or \emph{uncountable} sets.
\begin{example}
\ben
\it A coin is tossed $10$ times. The outcome is the total number of heads.
	\bit
	\it The sample space is a finite set: $\Omega = \{0,1,2,\ldots,10\}$.
	\it $A = \{1,3,5,7,9\}$ is the event that the total is odd.
	\eit
\it A coin is tossed repeatedly until the first head occurs. The outcome is the number of times the coin is tossed.
	\bit
	\it The sample space is a countably infinite set: $\Omega = \{1,2,3,\ldots\}$.
	\it $A = \{1,3,5,\ldots\}$ is the event that the number is odd.
	\it $B_n = \{n, n+1,n+2,\ldots\}$ is the event that the coin is tossed at least $n$ times.
	\eit
\it A spinning top is spun and comes to rest at a random angle relative to the horizontal axis.
	\bit
	\it The sample space is an uncountable set: $\Omega = [0,2\pi)$.
	\it $A = [0,\pi/2]$ is the event that the angle lies in the positive quadrant.
	\eit
\een
\end{example}

\begin{definition}
Let $\Omega$ be the sample space of some random experiment and let $A\subseteq\Omega$ be an event. Suppose we perform the experiment and observe the outcome $\omega$. If $\omega\in A$ we say that event \emph{$A$ occurs}, otherwise we say that \emph{$A$ does not occur}.
\end{definition}

\begin{example}
A coin is tossed $10$ times. Let the outcome of the experiment be the total number of heads, and let $A$ be the event that the total is odd.
\bit
\it If we observe a total of $5$ heads then $A$ occurs.
\it If we observe a total of $6$ heads, $A$ does not occur.
\eit
\end{example}

\begin{exercise}
Let $A$ and $B$ be two random events.
\ben
\it Show that if $A$ occurs and $A\subseteq B$ then $B$ also occurs.
\it Show that if $A$ occurs and $A\cap B=\emptyset$ then $B$ does not occur.
\een
\end{exercise}

% correspondence
Table~\ref{tab:sets_and_probability} shows the correspondence between terms used in set theory and those used in probability theory.
\begin{table}[ht]
\centering
\begin{tabular}{|l|l|l|} \hline
Notation 			& Set theory		& Probability theory \\ \hline
$\Omega$			& Universal set		& Sample space, certain event \\
$\omega\in\Omega$	& Element			& Elementary event, outcome \\
$A\subseteq\Omega$	& Subset			& Event $A$ \\
$A\subseteq B$		& Inclusion			& If $A$ occurs, then $B$ occurs \\
$A^c$				& Complement		& $A$ does not occur \\
$A\cap B$			& Intersection		& $A$ and $B$ both occur\\
$A\cup B$			& Union				& $A$ or $B$ (or both) occur \\
$A\setminus B$		& Set difference	& $A$ occurs, but $B$ does not \\
$\emptyset$			& Empty set			& Impossible event \\ \hline
\end{tabular}
\captionsetup{justification=centering}
\caption{Set theory and probability theory (Grimmett \& Stirzaker 2001).\label{tab:sets_and_probability}}
\end{table}


\section{Logistic equation}

\subsection{Why Logistic Equation}

\textbf{Logistic equation} is an \textit{Ordinary Differential Equation} that is
used to analyze the population change in a system. Before the theory of logistic
model is established, people were considering population using exponential
model (in which the rate of growth/decay is directly proportional to the current
population), and the differential equation that describes this is:

$$
\frac{dN}{dt}=rN
\footnote{$N$ stands for the current population, $t$ stands for time, and $r$
is the speed factor of the rate of change}
$$

In fact, a system may also have a limitation of resources for a species to live,
so the proposer Malthus believed that the population in a system may oscillate
and eventually will reach a number, which is known as the \textbf{Carrying
capacity}. Therefore, the differential equation of logistic model is:

$$
\frac{dN}{dt}=rN\left(\frac{K-N}K\right)
\footnote{$K$ is the carrying capacity}
$$

As you can see, when $N$ is greater than $K$, the population will decrease, and
if $N$ is less than $K$, the population will increase. Since the size of $N$
affects the sign of the derivative.

\subsection{Solve Logistic Equation}

Since we have known the origin and the meaning of this differential equation,
we can now figure a function out that can satisfy this population model along
with the equation.

Using Leibniz notation to represent derivatives can help us solve differential
equation by separating the differentiated variables:

$$dN=rN\left(\frac{K-N}k\right)dt$$

Since $r$ and $K$ are constants, we can leave $r$ alone and let the whole
polynomial related to $N$ be on the left side of the equation:

$$
\begin{aligned}
	\frac{dN}{N(K-N)}=\frac{r}Kdt \\
	\int\frac{dN}{N(K-N)}=\int\frac{r}Kdt
\end{aligned}
$$

The right part we can immediately come up with the answer $\frac{r}Kt+C$, but
the left part we need to cope with it longer:

$$
\begin{aligned}
	\because\int\frac{dN}{N(K-N)}=\int\frac{A}{N}dN+\frac{B}{K-N}dN \\
	\therefore A(K-N)+BN=1 \\
	\therefore AK-AN+BN=1 \\
	\therefore AK-N(A-B)=1 \\
\end{aligned}
$$

Since we know that $N$ is a variable and $K$ is a constant, we can quickly
conclude that the factor of the variable must be zero in order to let the result
be $1$:

$$
\begin{aligned}
	\because\begin{cases}
		AK=1 \\
		A-B=0
	\end{cases} \\
	\therefore A=B=\frac1K
\end{aligned}
$$

Therefore the integral of the left part is:

$$
\begin{aligned}
	\int\frac{dN}{N(K-N)}
	&=\frac1K\left(\int\frac{dN}{N}+\int\frac{dN}{K-N}\right) \\
	&=\frac1K\left(\ln(N)-\ln(K-N)+C_0\right) \\
	&=\frac1K\ln(\frac{N}{K-N})+C'
\end{aligned}
$$

Because both left and right parts of the equations are integrated, we can now
evaluate them to the original equation:

$$
\begin{aligned}
	\frac1K\ln(\frac{N}{K-N})+C'=\frac{r}Kt+C \\
	\ln(\frac{N}{K-N})=rt+C'' \\
	\frac{N}{K-N}=C_0e^{rt} \\
	N=C_0e^{rt}(K-N) \\
	N(1+C_0e^{rt})=KC_0e^{rt} \\
	N=\frac{KC_0e^{rt}}{1+C_0e^{rt}} \\
	N=N(t)=\frac{K}{1+C_1e^{-rt}}
\end{aligned}
$$

Finally we obtain the solution of this equation, which is called logistic
function. However this is not finished yet, the meaning of $C_1$ is not clear
enough. We can add a limiting condition where $N(0)=N_0$\footnote{To specify
the initial population of the system}:

$$
N(0)=\frac{K}{1+C_1e^{-r0}}=\frac{K}{1+C_1}
$$

We know that when $t=0$, $N$ shall be equal to the initial population $N_0$, so:

$$
\begin{aligned}
	N_0=\frac{K}{1+C_1} \\
	N_0(1+C_1)=K \\
	N_0C_1=K-N_0 \\
	C_1=\frac{K-N_0}{N_0}
\end{aligned}
$$

Let's plug $C_1$ in terms of $N_0$ to the function:

$$
\begin{aligned}
	N(t)
	&=\frac{K}{1+\frac{K-N_0}{N_0}e^{-rt}} \\
	&=\frac{KN_0}{N_0+(K-N_0)e^{-rt}}
\end{aligned}
$$

Finally we have the formula for logistic function:

$$
N(t)=\frac{KN_0}{N_0+(K-N_0)e^{-rt}}
$$


\chapter{Topology}

\section{Open set topology}

Let \(X\) be a set.
Let \(P(A) = 2^A\) be the power set of \(A\).
We say that \(T \subseteq P(P(X))\) is a \emph{topology} on \(X\) iff
\begin{enumerate*}[label={(\arabic*)}]
    \item \(\{ \emptyset, X \} \subseteq T\),
    \item \(T\) is closed under union, and
    \item \(T\) is closed under intersection.
\end{enumerate*}

A \emph{topological space} is \(S = (X,T)\) where \(X\) is a set and \(T\) is a topology on \(X\).
Each element of \(T\) is an \emph{open set} of \(S\).

\section{Topological manifold}

% https://en.wikipedia.org/wiki/Topological_manifold#Formal_definition

% topological manifold

% locally Euclidean

% Hausdorff space


\subsubsection{Gaussian Integral}
\noindent
Let's compute $\iint\limits_{D}{e^{-x^2-y^2}\mathrm{d}A}$ where $D$ is the unit disk.
\begin{equation*}
	A = \int_{-1}^{1}{\int_{-\sqrt{1 - x^2}}^{\sqrt{1 - x^2}}{e^{-x^2 - y^2}\mathrm{d}y}\mathrm{d}x} = \int_{0}^{1}{\int_{0}^{2\pi}{e^{-r^2}r\mathrm{d}\theta}\mathrm{d}r} = \int_{0}^{1}{re^{-r^2}\mathrm{d}r} \cdot \int_{0}^{2\pi}{\mathrm{d}\theta}.
\end{equation*}
Let $u = -r^2$, $\mathrm{d} u =-2r\mathrm{d}r$.\\
\begin{align*}
	&= \frac{1}{2}\int_{0}^{-1}{e^u\mathrm{d}u} \cdot 2\pi	\\
	&= \pi\left(1 - \frac{1}{e}\right).
\end{align*}
Now, let's have $D = \mathbb{R}^2$. This is the famous Gaussian Integral.
\begin{align*}
	&= \int_{0}^{\infty}{\int_{0}^{2\pi}{re^{-r^2}\mathrm{d}\theta}\mathrm{d}r} \\
	&= \int_{0}^{\infty}{re^{-r^2}\mathrm{d}r} \cdot \int_{0}^{2\pi}{\mathrm{d}\theta}.
\end{align*}
Let $ u =-r^2$, $\mathrm{d}u = -2r\mathrm{d}r$.
\begin{align*}
	&= -\pi\int_{0}^{-\infty}{e^{u}\mathrm{d}u} \\
	&= -\pi\left(\left(\lim_{a\to -\infty}{e^{a}}\right) - e^0\right) \\
	&= -\pi\left(0 - 1\right) \\
	&= \pi
\end{align*}


%!TEX root = sympy_tutorial.tex

%=======================================================================  matrices
\section{Linear algebra}
\label{sec:linear_algebra}

%\ifthenelse{\boolean{FORLA}}{

\small
\begin{verbatimtab}
from sympy import Matrix
\end{verbatimtab}
\normalsize

\noindent
A matrix $A \in \mathbb{R}^{m\times n}$ is a rectangular array of real numbers with $m$ rows and $n$ columns.
To specify a matrix $A$, we specify the values for its $mn$ components $a_{11}, a_{12}, \ldots, a_{mn}$
as a list of lists:

\small
\begin{verbatimtab}
>>> A = Matrix( [[ 2,-3,-8, 7],
                 [-2,-1, 2,-7],
                 [ 1, 0,-3, 6]] )
\end{verbatimtab}
\normalsize

\noindent
Use the square brackets to access the matrix elements or to obtain a submatrix:

\small
\begin{verbatimtab}
>>> A[0,1]            # row 0, col 1of A
-3
>>> A[0:2,0:3]        # top-left 2x3 submatrix of A
[ 2, -3, -8]
[-2, -1,  2]
\end{verbatimtab}
\normalsize

\noindent
Some commonly used matrices can be created with shortcut methods:

\small
\begin{verbatimtab}
>>> eye(2)            # 2x2 identity matrix
[1, 0]
[0, 1]
>>> zeros((2, 3))
[0, 0, 0]
[0, 0, 0]
\end{verbatimtab}
\normalsize

%TODO explain matrix concatenation operations:
%>>> M1.row_join(M2)
%[1  0  0  0  0  0  0]
%[                   ]
%[0  1  0  0  0  0  0]
%[                   ]
%[0  0  1  0  0  0  0]
%>>> M3 = zeros((4, 3))
%>>> M1.col_join(M3)

\noindent
Standard algebraic operations like
addition \texttt{+}, subtraction \texttt{-}, multiplication \texttt{*},
and exponentiation \texttt{**} work as expected for \texttt{Matrix} objects.
%
The \texttt{transpose} operation flips the matrix through its diagonal:

\small
\begin{verbatimtab}
>>> A.transpose()    # the same as A.T
[ 2, -2,  1]
[-3, -1,  0]
[-8,  2, -3]
[ 7, -7,  6]
\end{verbatimtab}
\normalsize

\noindent
Recall that the transpose is also used to convert row vectors into column vectors and vice versa.

\subsection{Row operations}
\label{matrices:row_operations}

\small
\begin{verbatimtab}
>>> M = eye(3)
>>> M[1,:] = M[1,:] + 3*M[0,:]
>>> M
[1, 0, 0]
[3, 1, 0]
[0, 0, 1] \end{verbatimtab}
\normalsize

The notation \texttt{M[i,:]} refers to entire rows of the matrix.
The first argument specifies the $0$-based row index,
for example the first row of~\texttt{M} is \texttt{M[0,:]}.
The code example above implements the row operation $R_2 \gets R_2 + 3R_1$.
To scale a row \texttt{i} by constant \texttt{c}, use the command \texttt{ M[i,:] = c*M[i,:]}.
To swap rows \texttt{i} and \texttt{j}, use can use the \texttt{Python} tuple-assignment syntax \texttt{ M[i,:], M[j,:] = M[j,:], M[i,:]}.
%The method \texttt{row\_op} takes two arguments as inputs:
%the first argument specifies the $0$-based index of the row you want to act on,
%while the second argument is a function of the form \texttt{f(val,j)}
%that describes how you want the value \texttt{val=M[i,j]} to be transformed.
% The above call to \texttt{row\_op} implements the row operation $R_2 \gets R_2 + 3R_1$.
%The expression \texttt{lambda  a,b: a+b} is the \texttt{Python} syntax for creating an anonymous function with arguments
%\texttt{a} and \texttt{b}, which computes their sum \texttt{a+b}.

\vspace{-2mm}

\subsection{Reduced row echelon form}
\label{matrices:reduced_row_echelon_form}

The Gauss--Jordan elimination procedure is a sequence of row operations you can perform
on any matrix to bring it to its \emph{reduced row echelon form} (RREF).
In \texttt{SymPy}, matrices have a \texttt{rref} method that computes their RREF:

\small
\begin{verbatimtab}
>>> A = Matrix( [[2,-3,-8, 7],
                 [-2,-1,2,-7],
                 [1 ,0,-3, 6]])
>>> A.rref()
([1, 0, 0,  0]  # RREF of A
 [0, 1, 0,  3]                    # locations of pivots
 [0, 0, 1, -2],                   [0, 1, 2]              )
\end{verbatimtab}
\normalsize

\noindent
Note the \texttt{rref} method returns a tuple of values:
the first value is the RREF of $A$,
while the second tells you the indices of the leading ones (also known as pivots) in the RREF of $A$.
To get just the RREF of $A$, select the $0$\textsuperscript{th} entry form the tuple: \texttt{A.rref()[0]}.

%
%\small
%\begin{verbatimtab}
%>>> Arref = A.rref()[0]
%>>> Arref
%[1, 0, 0,  0]
%[0, 1, 0,  3]
%[0, 0, 1, -2]
%\end{verbatimtab}
%\normalsize

\vspace{-2mm}

\subsection{Matrix fundamental spaces}
\label{matrices:matrix_fundamental_spaces}

Consider the matrix $A \in \mathbb{R}^{m\times n}$.
The fundamental spaces of a matrix are its column space $\mathcal{C}(A)$,
its null space $\mathcal{N}(A)$,
and its row space $\mathcal{R}(A)$.
These vector spaces are important when you consider the matrix product
$A\vec{x}=\vec{y}$ as ``applying'' the linear transformation $T_A:\mathbb{R}^n \to \mathbb{R}^m$
to an input vector $\vec{x} \in \mathbb{R}^n$ to produce the output vector $\vec{y} \in \mathbb{R}^m$.

\textbf{Linear transformations} $T_A:\mathbb{R}^n \to \mathbb{R}^m$ (vector functions)
\textbf{are equivalent to $m\times n$ matrices}.
This is one of the fundamental ideas in linear algebra.
You can think of $T_A$ as the abstract description of the transformation
and $A \in \mathbb{R}^{m\times n}$ as a concrete implementation of $T_A$.
By this equivalence,
the fundamental spaces of a matrix $A$
tell us facts about the domain and image of the linear transformation $T_A$.
The columns space $\mathcal{C}(A)$ is the same as the image space space $\textrm{Im}(T_A)$ (the set of all possible outputs).
The null space $\mathcal{N}(A)$ is the same as the kernel $\textrm{Ker}(T_A)$ (the set of inputs that $T_A$ maps to the zero vector).
The row space $\mathcal{R}(A)$ is the orthogonal complement of the null space.
Input vectors in the row space of $A$ are in one-to-one correspondence with the output vectors in the column space of $A$.

Okay, enough theory! Let's see how to compute the fundamental spaces of the matrix $A$ defined above.
The non-zero rows in the reduced row echelon form of $A$ are a basis for its row space:

\small
\begin{verbatimtab}
>>> [ A.rref()[0][r,:] for r in A.rref()[1] ]       # R(A)
[  [1, 0, 0, 0],  [0, 1, 0, 3],  [0, 0, 1, -2]   ]
\end{verbatimtab}
\normalsize

\noindent
The column space of $A$ is the span of the columns of $A$ that contain the pivots
in the reduced row echelon form of $A$:

\small
\begin{verbatimtab}
>>> [ A[:,c] for c in  A.rref()[1] ]                # C(A)
[  [ 2]    [-3]   [-8]
   [-2],   [-1],  [ 2]
   [ 1]    [ 0]   [-3]  ]
\end{verbatimtab}
\normalsize

\noindent
Note we took columns from the original matrix $A$ and not its RREF.

To find the null space of $A$, call its \texttt{nullspace} method:

\small
\begin{verbatimtab}
>>> A.nullspace()                                   # N(A)
[  [0, -3, 2, 1]  ]
\end{verbatimtab}
\normalsize
\subsection{Determinants}
\label{matrices:determinants}

The determinant of a matrix,
denoted $\det(A)$ or $|A|$,
is a particular way to multiply the entries of the matrix to produce a single number.

\small
\begin{verbatimtab}
>>> M = Matrix( [[1, 2, 3],
                 [2,-2, 4],
                 [2, 2, 5]] )
>>> M.det()
2
\end{verbatimtab}
\normalsize

\noindent
Determinants are used for all kinds of tasks:
to compute areas and volumes,
to solve systems of equations,
and to check whether a matrix is invertible or not.

\subsection{Matrix inverse}
\label{matrices:matrix_inverse}

For every invertible matrix $A$,
there exists an inverse matrix $A^{-1}$ which \emph{undoes} the effect of $A$.
The cumulative effect of the product of $A$ and $A^{-1}$ (in any order)
is the identity matrix: $AA^{-1}= A^{-1}A=\mathbbm{1}$.

\small
\begin{verbatimtab}
>>> A = Matrix( [[1,2],
                 [3,9]] )
>>> A.inv()
[ 3, -2/3]
[-1,  1/3]
>>> A.inv()*A
[1, 0]
[0, 1]
>>> A*A.inv()
[1, 0]
[0, 1]
\end{verbatimtab}
\normalsize

\noindent
The matrix inverse $A^{-1}$ plays the role of division by $A$.

\vspace{-3mm}

\subsection{Eigenvectors and eigenvalues}
\label{matrices:eigenvectors_and_eigenvalues}

\vspace{-1mm}

		When a matrix is multiplied by one of its eigenvectors the output
		is the same eigenvector multiplied by a constant $A\vec{e}_\lambda =\lambda\vec{e}_\lambda$.
		The constant $\lambda$ (the Greek letter \emph{lambda}) is called an \emph{eigenvalue} of $A$.
		%		and the vector is called an \emph{eigenvector}.
		%	\[
		%	 \Rightarrow
		%	 \quad
		%	 \left( A-\lambda \mathbbm{1}\right)	\vec{e}_\lambda 	= \vec{0}.
		%	\]
		%Thinking of matrices in term of their eigenvalues and eigenvectors is
		%a very powerful technique for describing their properties.
		%In particular

		To find the eigenvalues of a matrix,  start from the definition $A\vec{e}_\lambda =\lambda\vec{e}_\lambda$,
		insert the identity $\mathbbm{1}$,
		and rewrite it as a null-space problem:
		\[
			A\vec{e}_\lambda =\lambda\mathbbm{1}\vec{e}_\lambda
			\qquad
			\Rightarrow
			\qquad
			\left(A - \lambda\mathbbm{1}\right)\vec{e}_\lambda = \vec{0}.
		\]
		This equation will have a solution whenever $|A - \lambda\mathbbm{1}|=0$.\footnote{The invertible matrix theorem states
		that a matrix has a non-empty null space if and only if its determinant is zero.}
		%
		The eigenvalues of $A \in \mathbb{R}^{n \times n}$,
		denoted $\{ \lambda_1, \lambda_2, \ldots, \lambda_n \}$,
		are the roots of the \emph{characteristic polynomial} $p(\lambda)=|A - \lambda \mathbbm{1}|$.

\small
\begin{verbatimtab}
>>> A = Matrix( [[ 9, -2],
                 [-2,  6]] )
>>> A.eigenvals()   # same as  solve( det(A-eye(2)*x), x)
{5: 1, 10: 1}       # eigenvalues 5 and 10 with multiplicity 1
>>> A.eigenvects()
[(5, 1, [ 1]
        [ 2]  ),      (10, 1, [-2]
                              [ 1]  )]
\end{verbatimtab}
\normalsize

\noindent

		Certain matrices can be written entirely in terms of their eigenvectors and their eigenvalues.
		Consider the matrix $\Lambda$ (capital Greek \emph{L}) that has the eigenvalues of the matrix $A$ on the diagonal,
		and the matrix $Q$ constructed from the eigenvectors of~$A$ as columns:
		    \[
		     \Lambda =
		    \scriptscriptstyle
		     \begin{bmatrix}
		     \lambda_1	&  \cdots  &  0 \\
		     \vdots 	&  \ddots  &  0  \\
		     0  	&   0      &  \lambda_n
		     \end{bmatrix}\!,
		     \ \
		     {\textstyle Q} \:
		     =
		     \begin{bmatrix}
		     \big|  &  &\Huge| \\[1.2mm]
		     \vec{e}_{\lambda_1}  & \!  \cdots \! &  \large\vec{e}_{\lambda_n} \\[1.2mm]
		     \big|  &  & \Huge|
		     \end{bmatrix}\!,
		     \ \
		     {\textstyle
		     \textrm{then}
		     \ \
		     A =  Q\Lambda Q^{-1}.
		     }
		    \]

		Matrices that can be written this way are called \emph{diagonalizable}.
		%The matrix $A$ can be written as the product of three matrices
		%$A=Q\Lambda Q^{-1}$.
		%This is called the \emph{eigendecomposition} of $A$.
		%The matrix $Q$ contains the eigenvectors of $A$ as columns.
		%The matrix $\Lambda$ contains the eigenvalues of $A$ on its diagonal.
To \emph{diagonalize} a matrix $A$ is to find its $Q$ and $\Lambda$ matrices:

\small
\begin{verbatimtab}
>>> Q, L = A.diagonalize()
>>> Q               # the matrix of eigenvectors
[1,  -2]            # as columns
[2,   1]
>>> Q.inv()
[ 1/5, 2/5]
[-2/5, 1/5]
>>> L               # the matrix of eigenvalues
[5,  0]
[0, 10]
>>> Q*L*Q.inv()     # eigendecomposition of A
[ 9, -2]
[-2,  6]
>>> Q.inv()*A*Q     # obtain L from A and Q
[5,  0]
[0, 10]
\end{verbatimtab}
\normalsize

Not all matrices are diagonalizable.
You can check if a matrix is diagonalizable by calling its \texttt{is\_diagonalizable} method:

\small
\begin{verbatimtab}
>>> A.is_diagonalizable()
True
>>> B = Matrix( [[1, 3],
  	         [0, 1]] )
>>> B.is_diagonalizable()
False
>>> B.eigenvals()
{1: 2}              # eigenvalue 1 with multiplicity 2
>>> B.eigenvects()
[(1, 2,  [1]
         [0]  )]
\end{verbatimtab}
\normalsize

\noindent
The matrix $B$ is not diagonalizable because it doesn't have a full set of eigenvectors.
To diagonalize a $2\times 2$ matrix, we need two orthogonal eigenvectors but $B$ has only a single eigenvector.
Therefore, we can't construct the matrix of eigenvectors $Q$ (we're missing a column!)
and so $B$ is not diagonalizable.

Non-square matrices don't have eigenvectors and therefore don't have an eigendecomposition.
Instead, we can use the \emph{singular value decomposition} to break up a non-square matrix $A$ into
left singular vectors,
right singular vectors,
and a diagonal matrix of singular values.
Use the \texttt{singular\_values} method on any matrix to find its singular values.

%\subsection{QR decomposition}
%\label{matrices:qr_decomposition}

%It is possible to write a matrix $A$ as the product of an orthogonal matrix $Q$
%and an upper triangular matrix $R$.
%This is known as the QR-decomposition.

%\small
%\begin{verbatimtab}
%>>> A=Matrix( [[12,-51,4],
%               [6,167,-68],
%               [-4,24,-41]] )
%>>> Q,R = A.QRdecomposition()
%>>> Q
%[ 6/7, -69/175, -58/175]
%[ 3/7, 158/175,   6/175]
%[-2/7,    6/35,  -33/35]
%>>> Q*Q.T                  # verify Q is orthogonal
%[1, 0, 0]
%[0, 1, 0]
%[0, 0, 1]
%>>> R                      # and R is upper triangular
%[14,  21, -14]
%[ 0, 175, -70]
%[ 0,   0,  35]
%>>> Q*R                    # verify QR = A
%[12, -51,   4]
%[ 6, 167, -68]
%[-4,  24, -41]
%\end{verbatimtab}
%\normalsize
%
%\noindent
%Each \texttt{sympy} matrix is also equipped with
%\href{https://en.wikipedia.org/wiki/LU_decomposition}{\texttt{LUdecomposition}}
%and \href{https://en.wikipedia.org/wiki/Cholesky_decomposition}{\texttt{cholesky}} decomposition methods.


\subsection{Definition}

A sequence is an ordered list of terms.

These are commonly maps from natural numbers to real (or complex) numbers.

We can use \(a_i=f(i)\) to denote this.

If \(f(i)\) is defined on all \(i\in \mathbb{N}\) then the sequence is infinite. Otherwise it is finite.

If a sequence is defined on \(n\in \mathbb{N}\) and \(n\ne 0\) then the sequence must be defined on \(n-1\).

For example \(a_0, a_1, a_2,...\) is a sequence, but \(a_1,a_2,...\) is not.

\subsection{Monotone sequence}

A monotone sequence is one where each element is succeeded ordinally by the next entry.

For example:

\(<1,2,3,6,7>\) is monotone

\(<1,2,3,3,4>\) is not monotone

An increasing sequence is one where:

\(\forall m \in \mathbb{N} \forall n\in \mathbb{N} [m > n \leftrightarrow a_m \ge a_n]\)

A strictly increasing sequence is one where:

\(\forall m \in \mathbb{N} \forall n\in \mathbb{N} [m > n \leftrightarrow a_m > a_n]\)

A decreasing sequence is one where:

\(\forall m \in \mathbb{N} \forall n\in \mathbb{N} [m > n \leftrightarrow a_m \le a_n]\)

A strictly decreasing sequence is one where:

\(\forall m \in \mathbb{N} \forall n\in \mathbb{N} [m > n \leftrightarrow a_m < a_n]\)

All strictly decreasing sequences are decreasing, and all strictly increasing sequences are increasing.

A monotone sequence is one which is either increasing or decreasing.

\subsection{Subsequences}

A subsequence of a sequence is the original sequence with some elements of the original removed, not changing the order.

For example:

\(<1,3,5>\) is a subsequence of \(<2,1,3,4,7,5>\)


\subsection{Linear maps}

\subsubsection{Homomorphisms between vector spaces}

Homomorphisms map between algebras, preserving the underlying structure.

A homomorphism vetween vector space \(V\) and vector space \(W\) can be described as:

\(\hom (V, W)\)

Homomorphism between vector spaces must preserve the group-like structure of the vector space.

\(f(u+v)=f(u)+f(v)\)

The homomorphism must also preserve scalar multiplication.

\(f(\alpha v)=\alpha f(v)\)

A linear map (or function) is a map from one input to an output which preserves addition and scalar multiplication.

That is if function \(f\) is linear then:

\(f(aM+bN)=af(M)+bf(N)\)

\subsubsection{Alternative names for homomorphisms}

Vector spaces homomorphisms are also called linear maps or linear functions.


\section{Directional Statistics}

\subsection{Mean Direction}
The mean direction of a collection of $i$ vectors $\{\mathbf{x}\}_i$ is \cite{damask2019consistently}:

\begin{equation}
\left<\mathbf{x}\right> = \frac{\mathbf{x_s}}{||\mathbf{x_s}||_2}
\end{equation}

Where

\begin{equation}
\mathbf{x_s} = \sum_i \mathbf{x_i}
\end{equation}

The mean vector is not defined in case $||\mathbf{x_s}||_2 = 0$. The alternative way to calculate the mean direction might make use of angles but apparently that creates ambiguity with respect to the choice of a "zero" angle (cf. footnote in \citeasnoun{damask2019consistently}).

\subsection{Dispersion}

Dispersion is a measure of the variance on the direction of a set of vectors $\{\mathbf{x}\}_i$. For a system of vectors, for example an eigenbasis, there are common and differential modes of dispersion. One way of measuring directional dispersion is to look at the \textit{mean resultant length}:

\begin{equation}
\mu_r = \frac{||\mathbf{x_s}||}{N}\ \ \ 0\leq \mu_r \leq 1
\end{equation}

\textit{Circular variance} may be defined as $\sigma_c = 1-\mu_r$, but apparently there is an issue with generalization to higher dimensions. (TO DO)

An alternative is a model based approach, for example based on the von Mises - Fisher Distribution.

In the case of a basis, \citeasnoun{damask2019consistently} states that, in order to understand the drivers for variation, dispersion parameters should be calculated for each descending subspace of the basis, first including all of the eigenvectors, then excluding the first eigenvector, etc.


\subsection{Hessian matrix}

We can take a function and make a matrix of its second order partial derivatives.  This is the Hessian matrix, and it describes the local curvature of the function.

If the function \(f\) has \(n\) parameters, the Hessian matrix is \(n\times n\), and is defined as:

\(H_{ij}=\dfrac{\delta^2 f}{\delta x_i \delta x_j}\)

If the function is convex, then the Hessian matrix is positive semi-definite for all points, and vice versa.

If the function is concave, then the Hessian matrix is negative semi-definite for all points, and vice versa.

We can diagnose critical points by evaluating the Hessian matrix at those points.

If it is positive definite, it is a local minimum. If it is negative definite it is a local maximum. If there are both positive and negative eigenvalues it is a saddle point.


\section{Differentiability}
As we saw when looking at left and right hand derivatives, a derivative will fail to exist when the left and right hand derivatives differ.
However, there are several other cases where a function might not be differentiable at a point.
\begin{enumerate}
	\item \textbf{Corner: } This is the simplest case where both the one sided derivatives exist, are finite, but are different.
		An example would be $f(x)=\abs{x}$ at $x=0$.
	\item \textbf{Cusp: } This is sort of an extreme example of a corner where the one of the one sides derivatives approaches $-\infty$ and the other approaches $\infty$.
		An example would be $f(x)=\sqrt[3]{x^2}$ at $x=0$.
	\item \textbf{Vertical Tangent: } This is where the one sided derivatives exist and agree, but approach $-\infty$ or $\infty$.
		An example would be $f(x)=\sqrt[3]{x}$ at $x = 0$.
	\item \textbf{Discontinuity: } This is where one of the one sided derivatives doesn't exist.
		An example would be the unit step function at $x=0$, which is 1 for $x \geq 0$ and -1 for $x < 0$.
\end{enumerate}

\subsection{Differentiability Implies Local Linearity}
A function is locally linear at a point $a$ when it is differentiable at $a$ and closely resembles its own tangent line at $a$.
Visually, what this means is that as you ``zoom in'' on a locally linear point, the curve will look more and more like a line.

\begin{figure}[H]
	\label{locally_linear}
	\centering
	\includegraphics[width = 0.25\textwidth]{./derivatives/cos1.png}
	\includegraphics[width = 0.25\textwidth]{./derivatives/cos2.png}
	\includegraphics[width = 0.25\textwidth]{./derivatives/cos3.png}
	\caption{\hyperref{}{}{}{Zooming in on a $\cos$ curve}}
\end{figure}

\subsubsection{Numerical Approximation: Symmetric Difference Quotient}
We can use the derivative at a point and a very small value of $h$ to approximate the tangent slope.
However, we can get a better estimate with the same value of $h$ by using the symmetric difference quotient.
\begin{equation*}
	\frac{f(a+h)-f(a-h)}{2h}.
\end{equation*}

A calculator might prefer this formula because it tends to give better approximations.

\subsection{Differentiability Implies Continuity}
\begin{theorem}
	If $f^\prime(a)$ exists, then $f$ is continuous at $a$.
\end{theorem}

The converse is not necessarily true: a function can be continuous but not differentiable.

\begin{example}
	Determine whether or not the follow function is differentiable at $x=2$.
	\begin{equation*}
		f(x) = \begin{cases}
			x^2 + 1 & x \leq 2 \\
			4x-4 & x > 2
		\end{cases}.
	\end{equation*}
\end{example}
\begin{answer}
	Looking at the graph of this function, we see there is a jump discontinuity at $x=2$.
	If the function was differentiable at $x=2$, it would be continuous at $x=2$.
	Since it is not continuous at $x=2$, it must not be differentiable at $x=2$.
\end{answer}


\section{General Covariance}
Let $X \in V$ be a random variable living in an inner product space V with $E\left[ X\right] = 0$\newline
Define: The covariance of X is the quadratic form  $\operatorname{Cov}(X): V \rightarrow R $ defined by
$$
\operatorname{Cov}(X)(v)=E\left[\langle X, v\rangle^{2}\right]
$$
If V is finite dimensional and $e_{1}, \dots, e_{n}$ is an orthonormal matrix then
$$
\langle X, v\rangle=X^{\top} v=v^{\top} X
$$ and so
$$
\operatorname{Cov}(X)(v)=\mathbb{E}\left[v^{\top} X X^{\top} v\right]=v^{\top} \mathbb{E}\left[X X^{\top}\right] v
$$
Sometimes, we may define a scalar variance \newline
$$
\operatorname{Var}(X)=\mathbb{E}\left[\|X\|^{2}\right]=\operatorname{tr}\left(\mathbb{E}\left[X X^{\top}\right]\right)
$$

\subsection{Whitening}
Consider transforming the random variable X with a linear transformation matrix A. Then \newline
$$
Cov(A X)=\mathbb{E}\left[AXX^{\top} A^{\top}\right]=A \mathbb{E}\left[X X^{\top}\right] A^{\top}=A V A^{T}
$$
where $V= \mathbb{E}\left[X X^{\top}\right]$ is the coveriance of $X$.
So if we choose A s.t. \newline
$$
A VA^{\top}=I \Leftrightarrow V=A^{-1} A^{-T},
$$
then
$$
Cov(A X)= I,
$$
so $AX$ is "white noise".
Clearly there are many choice for $A$. Namely given an $A$, $OA$ also works for any orthonormal $O$.

\subsection{Batch Normalization}
\begin{itemize}
\item Calculating an appropriate A is generally competationally too expensive. The most efficient way to do it is likely a Cholesky factorization.
\item In BN, we instead choose A diagonal so that
$$
\operatorname{diag}\left(A \cup A^{T}\right)=(1,1, \ldots, 1)
$$
\item Of course the off-diagonal entries of $A \cup A^{T}$ will generally not be $O$.
\end{itemize}

\subsection{Central Limiting Theorem}
Let $X_{1}, X_{2}, \ldots, X_{n}$ be independent copies of $X\in \Omega$ with $E\left[ X\right] = 0$. Then $V=E\left[ XX^T\right] $.
Consider
$$
\bar{X}_{n}=\frac{1}{\sqrt{n}} \sum\limits_{i=1}^{n} X_{i}.
$$
As $n\rightarrow \infty$, $\bar{X}_{n}$ converges to a Gaussian distribution, namely,
$$
\lim_{n \rightarrow \infty} \bar{X}_{n}= {\rm Gaussian}(0, V)
$$
with distribution
$$
P(X)=\frac{1}{(\sqrt{2 n}\ {\rm det}(V))^{d}} e^{\frac{-x^{T}V^{-1}x}{2}}d x.
$$
%$\Omega \stackrel{X}{\longrightarrow} R^{d}$ pushforward measure
% \includepdf[pages=-,pagecommand={}]{497/handwritten_notes/MultidimensionalVariance.pdf}


\subsection{Frobenius norm}

If we have matrices \(A\) and \(A\) the distance is:

\(||A-B||=\sqrt {\sum_i \sum_j |a_{ij}-b_{ij}|^2}\)

This is a Euclidian norm.


\subsection{Momentum method and Averaged Gradient}

Another way of thinking about the momentum method is that we are simply replacing our gradient direction by an average of the previously encountered gradients. Specifically, the momentum method can be rewritten
\begin{equation}\label{Ave_grad1}
x_{t+1} = x_t - \frac{\eta_t}{1-\mu}[\nabla f]_t,
\end{equation}
where $[\nabla f]_t$ is an average of the past gradients with exponentially decaying weights given by
\begin{equation}\label{Ave_grad2}
[\nabla f]_t= (1-\mu)\sum_{i=0}^t \mu^{t-i} \nabla f(x_i).
\end{equation}
(Note that this isn't quite exactly the average, since it isn't normalized correctly.)
Here $\mu$ is the momentum parameter which determines the time scale of the averaging, i.e. how many previous gradients are taken into account. A typical value used in deep learning is $\mu = 0.9$, which corresponds to an average over around $10$ iterations.

The relationship between this formulation and the more standard formulation of momentum can be seen by noting the following recursive formula for $[\nabla f]_t$,
\begin{equation}
[\nabla f]_t = \mu[\nabla f]_{t-1} + (1-\mu)\nabla f(x_t).
\end{equation}

%Usually, the factor $(1-\mu)$ is absorbed into the step size $s_t$ in implementations.
It is important in this context to note that pushing $\mu$ closer to $1$ effectively increases the step size. This gives to PyTorch implementations below.

\begin{theorem}
	Given $v_0=0$ and random initial value $x_0$, the averaged gradient (\ref{Ave_grad1}) and (\ref{Ave_grad2}) are equivalent to
	\begin{align}
	&v_{t+1} = \mu v_t + \nabla f(x_t),\\
	&x_{t+1} = x_{t} - \eta_t v_{t+1},
	\end{align}
	for any $t\geq 0$.
\end{theorem}

\begin{algorithm}[H]\caption{SGD with mini-batch and momentum}
	\label{alg:batch-momentum-SGD}
	{\bf Input}: initialization $x_0$, learning rate $\eta_t$, momentum parameter $\mu$, $v_0=0$.

	{\bf For}: t = 0,1,2,$\dots$

	\begin{center}
		Randomly pick $B_t \subset \{1, 2, \cdots, N\}$ independently \\
		with probability $\frac{1}{\tbinom{N}{m}}$ and $\# B_t = m$.
	\end{center}
	\begin{align}\label{equ:sgd-iteration-momentum}
	&v_{t+1} = \mu v_t + g_t(x_t),\\
	&x_{t+1} = x_{t} - \eta_t v_{t+1}.
	\end{align}
	\quad where
	$$
	g_{t}(x_t) = \frac{1}{m} \sum_{i \in B_{t}}  \nabla f_i(x_t)
	$$
\end{algorithm}

\subsection{Derivation of momentum method}

The gradient descent method previously introduced can be viewed as a discretization of the gradient flow differential equation
\begin{equation}\label{gradient_flow}
 x'(t) = -\nabla f(x(t)).
\end{equation}

Gradient descent with momentum can be understood as a discretization of the damped Hamiltonian dynamics
\begin{equation}\label{hamiltonian}
 \begin{split}
  &v'(t) = -\alpha v(t) - \nabla f(x(t)) \\
  &x'(t) = v(t).
 \end{split}
\end{equation}
This can be thought of as describing the motion of a particle in a potential well given by $f(x)$ which experiences friction, the strength of which is determined by $\alpha$.

The relationship between this and the gradient flow dynamics can be seen by taking the high-friction limit $\alpha\rightarrow \infty$. In fact, rewriting the Hamiltonian dynamics \eqref{hamiltonian} as
\begin{equation}
 x''(t) = v'(t) = -\alpha x'(t) - \nabla f(x(t)),
\end{equation}
we make the time change $s = \alpha t$ to get
\begin{equation}
 \frac{1}{\alpha^2}x''(s) + x'(s) +\nabla f(x(s)) = 0.
\end{equation}
Taking the limit as $\alpha\rightarrow \infty$, we get the gradient flow equation \eqref{gradient_flow}. So we see that the gradient flow is a high-friction limit of the damped Hamiltonian flow \eqref{hamiltonian}.

It turns out that discretizing the Hamiltonian flow results in a method which converges much faster than gradient descent, provided we choose the friction parameter $\alpha$ correctly. To see this, it is instructive to analyze a strongly convex quadratic objective, from which we see that
the dynamics \eqref{gradient_flow} converges at an exponential rate
\begin{equation}
 f(x(t)) - f(x^*) = O(e^{-\lambda_{min} t}),
\end{equation}
where $\lambda_{min}$ is the smallest eigenvalue of the quadratic objective.

Choosing the optimal damping rate $\alpha = 2\sqrt{\lambda_{min}}$ (note that this makes the lowest frequency modes critically damped), the dynamics \eqref{hamiltonian} achieve an exponential convergence rate of
\begin{equation}
 f(x(t)) - f(x^*) = O(e^{-\sqrt{\lambda_{min}} t}).
\end{equation}

Combined with the fact that \eqref{hamiltonian} can be stably discretized with a step size $s = 0(\lambda_{max}^{-\frac{1}{2}})$, while \eqref{gradient_flow} can only be stably discretized with a step size $s = O(\lambda_{max}^{-1})$, shows that discretizing the Hamiltonian dynamics \eqref{hamiltonian} results in a method which convergences much faster than gradient descent (the number of iterations scales with $\sqrt{\frac{\lambda_{max}}{\lambda_{min}}}$ vs $\frac{\lambda_{max}}{\lambda_{min}}$.

Only certain special descretizations of \eqref{hamiltonian} achieve this improved convergence rate globally for convex functions. The most famous of these is Nesterov's accelerated gradient descent
\begin{equation}
 \begin{split}
  &x_{t+1} = y_t - \frac{1}{\lambda_{max}}\nabla f(y_t) \\
  &y_{t+1} = x_{t+1} + \frac{\sqrt{\lambda_{max}} - \sqrt{\lambda_{min}}}{\sqrt{\lambda_{max}} + \sqrt{\lambda_{min}}}(x_{t+1} - x_t).
 \end{split}
\end{equation}

However, for deep learning we often use the much simpler discretization
\begin{equation}
 \begin{split}
  &v_{t+1} = (1 - s\alpha) v_t + s\nabla f(x_t)\\
  &x_{t+1} = x_t - sv_t.
 \end{split}
\end{equation}
This often gives similar results since deep learning problems are non-convex and the theory of convex optimization doesn't apply anyway.

However, there is work which analyzes different versions of momentum (i.e. different discretizations of the Hamiltonian dynamics \eqref{hamiltonian}) for deep learning problems.

\includepdf{HandWrittenNotes/Momentum.pdf}


\subsection{Cobb-Douglas utility function}

\(U=A\sum_i X_i^{\alpha_i} \)


\subsection{Sample covariance}

We previously defined the population covariance as \(\sigma_{XY}=E[(X-\mu_X)^T(Y-\mu_Y)]\).

We define the sample covariance as \(\sigma_{XY}=\dfrac{1}{n}\sum_i(x_i-\bar x)(y_i-\bar y)\).

We can calculate this using matrices:

\(M=X-\bar x\)

\(N=Y-\bar y\)

\(\sigma_{XY}=\dfrac{1}{n}M^TN\).

\subsection{Sample correlation}

\(\rho_{XY}=\dfrac{\sigma_{XY}}{\sigma_X \sigma_Y}\)

\subsection{Covariance matrix}

If we have \(n\) variables we can have a \(n\times n \) matrix \(\Sigma \) where:

\(\Sigma_{ij} = \sigma_{ij}=\dfrac{1}{n}(X_i-\bar x_i)^T(X_j-\bar x_j)\)

\subsection{Centred covariance}

If \(\bar x = \bar y = 0\) then:

\(\sigma_{XY}=\dfrac{1}{n}X^TY\)

\subsection{Correlation matrix}

Here each entry is the correlation rather than the covariance.


\section{Additional Exercises}\label{sec:MoreFunctionExercises}

\Opensolutionfile{solutions}[ex]

\begin{enumialphparenastyle}

%%%%%%%%%%
\begin{ex}
If $f\left( x\right) =\dfrac{1}{x-1},$ then which
of the following is equal to $f\left( \dfrac{1}{x}\right) ?$
\begin{enumerate}
	\item	$f(x)$
	\item	$-f(x)$
	\item	$xf(x)$
	\item	$-xf(x)$
	\item	$\dfrac{f(x)}{x}$
	\item	$-\dfrac{f(x)}{x}$
\end{enumerate}
\begin{sol}
	(d)
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
If $f(x)=\dfrac{x}{x+3}$, then find
and simplify $\dfrac{f(x)-f(2)}{x-2}$.
\begin{sol}
	$3/\left[5(x+3)\right]$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
If $f(x)=x^2$, then find and
simplify $\dfrac{f(3+h)-f(3)}{h}$.
\begin{sol}
	$6+h$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
What is the domain of
\begin{enumerate}
	\item	$f(x)=\dfrac{\sqrt{x-2}}{x^2-9}$?
	\item	$g(x)=\dfrac{\sqrt[3]{x-2}}{x^2-9}$?
\end{enumerate}
\begin{sol}
\begin{enumerate}
	\item	$[2,3)\cup(3,\infty)$
	\item	$(-\infty,-3)\cup(-3,3)\cup(3,\infty)$
\end{enumerate}
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Suppose that $f(x)=x^3$ and $g(x)=x$. What is the domain of $\dfrac{f}{g}$?
\begin{sol}
	$\left\{x:x\neq 0\right\}$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Suppose that $f(x)=3x-4$. Find a
function $g$ such that $(g\circ f)(x)=5x+2$.
\begin{sol}
	$g(x)=(5x+26)/3$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Which of the following functions is one-to-one?
\begin{enumerate}
	\item	$f(x)=x^2+4x+3$
	\item	$g(x)=\vert x\vert+2$
	\item	$h(x)=\sqrt[3]{x+1}$
	\item	$F(x)=\cos x$, $-\pi\leq x\leq\pi$
	\item	$G(x)=e^x+e^{-x}$
\end{enumerate}
\begin{sol}
	(c)
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
What is the inverse of $f(x)=\ln\left(\dfrac{e^x}{e^x-1}\right)$? What is the domain of $f^{-1}$?
\begin{sol}
	$f^{-1}(x)=f(x)=\ln\left(\dfrac{e^x}{e^x-1}\right)$ and its domain is $(0,\infty)$.
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Solve the following equations.
\begin{enumerate}
	\item	$e^{2-x}=3$
	\item	$e^{x^2}=e^{4x-3}$
	\item	$\ln\left(1+\sqrt{x}\right)=2$
	\item	$\ln(x^2-3)=\ln 2+\ln x$
\end{enumerate}
\begin{sol}
\begin{enumerate}
	\item	$2-\ln 3$
	\item	1,3
	\item	$(e^2-1)^2$
	\item	3
\end{enumerate}
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Find the exact value of $\sin^{-1}\left(-\sqrt{2}/2\right)-\cos^{-1}\left(-\sqrt{2}/2\right)$.
\begin{sol}
	$-\pi$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Find $\sin^{-1}\left(\sin(23\pi/5)\right)$.
\begin{sol}
	$2\pi/5$
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
It can be proved that $f(x)=x^3+x+e^{x-1}$ is one-to-one. What is the value of $f^{-1}(3)$?
\begin{sol}
	1
\end{sol}
\end{ex}

%%%%%%%%%%
\begin{ex}
Sketch the graph of $f(x)=\left\{
\begin{array}{cc}
-x & \text{if }x\leq 0 \\
\tan ^{-1}x & \text{if }x>0%
\end{array}%
\right. $
\end{ex}

\end{enumialphparenastyle}


\subsection{Variance of OLS estimators}

\subsubsection{Variance-covariance matrix}

We know:

\(\hat \theta =(X^TX)^{-1}X^Ty\)

\(y=X\theta +\epsilon\)

Therefore:

\(\hat \theta =(X^TX)^{-1}X^T(X\theta +\epsilon)\)

\(\hat \theta =\theta +(X^TX)^{-1}X^T\epsilon\)

\(\hat \theta -\theta =(X^TX)^{-1}X^T\epsilon\)

\(Var [\hat \theta ]=E[(\hat \theta -\theta)(\hat \theta -\theta )^T]\)

\(Var [\hat \theta ]=E[(X^TX)^{-1}X^T\epsilon(X^TX^{-1}X^T\epsilon )^T]\)

\(Var [\hat \theta ]=E[(X^TX)^{-1}X^T\epsilon \epsilon^T X(X^TX)^{-1}]\)

\(Var [\hat \theta ]=(X^TX)^{-1}X^TE[\epsilon \epsilon^T ]X(X^TX)^{-1}\)

We write:

\(\Omega=E[\epsilon \epsilon^T]\)

\(Var [\hat \theta ]=(X^TX)^{-1}X^T\Omega X(X^TX)^{-1}\)

Depending on how we estimate \(\Omega\), we get different variance terms.

\subsubsection{Variance under IID}

If IID:

\(\Omega = I\sigma^2_{\epsilon }\)

\(Var [\hat \theta ]=(X^TX)^{-1}X^TI\sigma^2_{\epsilon } X(X^TX)^{-1}\)

\(Var [\hat \theta ]=\sigma^2_\epsilon (X^TX)^{-1}\)


% -*- root: Main.tex -*-
\section{SVD}
$\mathbf{A} = \mathbf{U} \mathbf{D} \mathbf{V}^\top = \sum_{k=1}^{\operatorname{rank}(\mathbf{A})} d_{k,k} u_k (v_k)^\top$\\
$\mathbf{A} \in \mathbb{R}^{N \times P}, \mathbf{U} \in \mathbb{R}^{N \times N}, \mathbf{D} \in \mathbb{R}^{N \times P}, \mathbf{V} \in \mathbb{R}^{P \times P}$\\
$\mathbf{U}^\top \mathbf{U} = I = \mathbf{V}^\top \mathbf{V}$ ($\mathbf{U}, \mathbf{V}$ orthogonal)\\
$\mathbf{U}$ columns are eigenvectors of $\mathbf{A} \mathbf{A}^\top$, $\mathbf{V}$ columns are eigenvectors of $\mathbf{A}^\top \mathbf{A}$, $\mathbf{D}$ diagonal elements are singular values.\\
$(\mathbf{D}^{-1})_{i,i} = \frac{1}{\mathbf{D}_{i, i}}$ (don't forget to transpose)

1. calculate $\mathbf{A}^\top \mathbf{A}$.\\
2. calculate eigenvalues of $\mathbf{A}^\top \mathbf{A}$, the square root of them, in descending order, are the diagonal elements of $\mathbf{D}$.\\
3. calculate eigenvectors of $\mathbf{A}^\top \mathbf{A}$ using the eigenvalues resulting in the columns of $\mathbf{V}$.\\
4. calculate the missing matrix: $\mathbf{U} = \mathbf{A} \mathbf{V} \mathbf{D}^{-1}$.\\
5. normalize each column of $\mathbf{U}$ and $\mathbf{V}$.

\subsection*{Low-Rank approximation}
Using only $K$ largest eigenvalues and corresponding eigenvectors. $\tilde{\mathbf{A}}_{i, j} = \sum_{k=1}^K \mathbf{U}_{i, k} \mathbf{D}_{k,k} \mathbf{V}_{j, k} = \sum_{k=1}^K \mathbf{U}_{i, k} \mathbf{D}_{k,k} (\mathbf{V}^\top)_{k, j}$.

\subsection*{Collaborative Filtering \textendash\ Prediction}
For $\mathbf{U}, \mathbf{V}$ (Movie-/User-embedding) use $k$-rank approx (fill up dimensions). Predict user preferences: $\mathbf{U_k} \mathbf{U_k^T} b$.

\subsection*{Echart-Young Theorem}
$\min_{rank(B)=K} ||A-B||_F^2 = ||A-A_k||_F^2 = \sum_{r=k+1}^{rank(A)} \sigma_r^2$
$\min_{rank(B)=K} ||A-B||_2 = ||A-A_k||_2 = \sigma_{k+1}$