\chapter{Binary Collatz Tree}
\label{ch:binary_tree}

\section{Some essentials on binary trees}
A binary tree is a rooted tree, where each node has at most two immediate successors. Those nodes, from which no edge goes out downward, are called leaves, the others are called internal nodes. In a full binary tree, all internal nodes have exactly two children \cite[p.~102]{Ref_Higham_2015}. Full binary trees have an odd number $2n+1$ of nodes. Of these $n+1$ are leaves and $n$ are inner nodes \cite[p.~134]{Ref_Kersting_Wakolbinger_2008}. Each node in a binary tree has a left subtree and a right subtree, which is why a binary tree is inherently recursive, since the left and right subtrees of the root are themselves binary trees \cite[p.~246-247]{Ref_Mazur_2010}. As it often pops up in combinatorial problems, the famous $n$-th Catalan number, named after the Belgian mathematician Eugène Catalan, comes in connection with binary trees into play. For $n\ge1$ it specifies the number of binary trees on $n$ vertices \cite[p.~247]{Ref_Mazur_2010}:
\[
B_n=\sum_{i=0}^{n-1}B_iB_{n-1-i}=\sum_{i=1}^{n}B_{i-1}B_{n-i}=\frac{1}{n+1}\binom{2n}{n}
\]

There is an interesting property that trees exhibit regarding abstract algebra. Let's have a look at the algebraic structure of magmas. Consider an element $x$ of a magma $(M,*)$ which is an iterated product of other elements in $M$. Such an element can be described by a planar (no edges cross each other) rooted binary tree whose $n$ leaves are labelled by these other elements $x_1,\ldots,x_n\in M$ \cite[p.~96]{Ref_Kalka_2016}.

Binary trees make well-suited data structures for storing information. With about $2^m$ data points (nodes), a search of a binary tree takes only about $m$ steps, compared to about $2^{m-1}$ steps which are required to search a simple list \cite[p.~84]{Ref_Benjamin_2009}.

\section{Transforming the Collatz tree into a binary tree}
Jan Kleinnijenhuis and Alissa M. Kleinnijenhuis \cite{Ref_Kleinnijenhuis_2020a} introduced a binary tree $T_{\ge0}$ by transforming the original Collatz tree $H_U$ into the Syracuse tree $H_{C,3}$, which in turn is transformed into the binary tree $T_{\ge0}$ as described next. The edges are changed according to the following procedure: whenever a parent node $w$ has edges to its child nodes $v_0,v_1,\ldots,v_n$, on the tree $H_{C,3}$, we draw an edge from $w$ to $v_0$, and edges from $v_i$ to $v_{i+1}$ for each $i=1,\ldots,n-1$, in the binary new tree. Note that the nodes $v_1,v_2,\ldots,v_n$ are sorted in increasing order of label $v_0<v_1<\ldots<v_n$, which is already given by \ref{eq:n_fold_right_sibling_k}. Figure~\ref{fig:bt3} and \ref{fig:bt3_rot} display that tree -- once in our standard layout and once reversed (from bottom to top).

\begin{figure}[H]
	\includegraphics[width=1.00\textwidth]{figures/bt_3_t0.png}
	\caption{The Collatz Tree transformed to the binary tree $T_{\ge0}$}
	\label{fig:bt3}
\end{figure}

\vspace{-2em}
\begin{figure}[H]
	\includegraphics[width=1.00\textwidth]{figures/bt_3_t0_rot.png}
	\caption{The binary tree $T_{\ge0}$ with \textit{bottom-to-top} layout orientation}
	\label{fig:bt3_rot}
\end{figure}

\begin{remark}
	To clarify the terminology, it should be mentioned that Jan and Alissa M. Kleinnijenhuis in their manuscripts \cite{Ref_Kleinnijenhuis_2020a}, \cite{Ref_Kleinnijenhuis_2020b} denote the original Collatz tree $T_C$ while we call it $H_U$. They denote the Syracuse Tree $T_T$ which in our nomenclature is referred to as $H_{C,3}$.
\end{remark}

Nodes that are highlighted orange in figures~\ref{fig:bt3},~\ref{fig:bt3_rot} are called \textit{prunable} and they are exactly those nodes resulting as output of the \textit{Rightward} function. For navigating within this binary tree, Jan Kleinnijenhuis and Alissa M. Kleinnijenhuis \cite{Ref_Kleinnijenhuis_2020a} defined an \textit{Upward} function $U(n)$ and a \textit{Rightward} function $R(n)$ as follows:

\begin{equation}
\label{eq:bintree_3_rightward_upward}
\setlength{\arraycolsep}{1.6em}
\begin{array}{cc}
U(n)=\begin{cases}
        4n+1	&	n\equiv 1\pmod 6\\
        16n+5	&	n\equiv 5\pmod 6
    \end{cases} &
R(n)=\begin{cases}
    \nicefrac{(2^2n-1)}{3}	&	n\equiv 1\pmod{18}\\
    \nicefrac{(2^3n-1)}{3}	&	n\equiv 5\pmod{18}\\
    \nicefrac{(2^4n-1)}{3}	&	n\equiv 7\pmod{18}\\
    \nicefrac{(2^1n-1)}{3}	&	n\equiv 11\pmod{18}\\
    \nicefrac{(2^2n-1)}{3}	&	n\equiv 13\pmod{18}\\
    \nicefrac{(2^1n-1)}{3}	&	n\equiv 17\pmod{18}
\end{cases}
\end{array}
\end{equation}

The domain and codomain of both functions consist of the two residue classes $[1]_6,[5]_6$, which form the multiplicative (cyclic) group $\mathbb{Z}^\ast_6=\{1,5\}=<5>$. Consequently, the domain and codomain exclude all integers divisible by $2$ and $3$, which is due to the fact that this binary tree (just like our tree $H_{C,3}$) does not contain even numbers and additionally all leaves -- namely those nodes labeled with an integer divisible by three -- were deleted. The function $U(n)$ is very similar to the function~\ref{eq:next_sibling_k3} and to the more general function~\ref{eq:n_fold_right_sibling_k} (when setting $n=1,k=3$) which both calculate the right-sibling of a given vertex. This is clear, since siblings (parallel) in $H_{C,3}$ are successors (serial) in the binary tree $T_{\ge0}$. In the end, for a node $v_0$ having a leaf as right-sibling in $H_{C,3}$, the function $U(v_0)$ is defined as $v_1=4v_0+1$ executed twice $v_1=4(4v_0+1)+1=16v_0+5$, because we must skip this leaf. Recall that all leafs in $H_{C,3}$ are excluded from the binary tree without exception. For any $n\in[5]_6$ it applies that $U(n)\equiv16n+5\equiv\boldsymbol{1}\bmod(6)$ since $6\mid16n+5-\boldsymbol{1}$ resulting in $6\mid16(5+k\cdot6)+5-\boldsymbol{1}$, see \ref{eq:congruence}, and analogously for any $n\in[1]_6$ it applies that $U(n)\equiv4n+1\equiv\boldsymbol{5}\bmod(6)$ since $6\mid4n+1-\boldsymbol{5}$ resulting in $6\mid4(1+k\cdot6)+1-\boldsymbol{5}$. Therefore executing the Upward function twice in a row leads unconditionally to $U^2(n)=16(4n+1)+5=4(16n+5)+1=64n+21$.

\begin{remark}
	While we displayed trees from top to down, it is sometimes usual to draw trees in a bottom-to-top fashion as Kleinnijenhuis \cite{Ref_Kleinnijenhuis_2020b} do. The Rightward function corresponds to what we call left-child and the Upward function relates to the right-child which is commonly used in the context of binary trees \cite[p. 246]{Ref_Mazur_2010}.
\end{remark}

Jan and Alissa M. Kleinnijenhuis \cite{Ref_Kleinnijenhuis_2020a} defined the set $N(T_C)=N(H_U)$ that contains the labels of all nodes, to which a path from the root in $H_U$ exists, in other words, this set contains all integers $n$ for which the orbit of $n$ under the (uncompressed) Collatz function~\ref{eq:func_collatz} converges to $1$. Furthermore they introduced $S_{\ge0}$ as the node set containing integers that are neither divisible by $2$ nor by $3$. The set $S_{-1}$ comprises on the contrary all numbers, which are divisible by $2$ or $3$. In order to comprehend the structure of these sets $S$, let us take a look at the following list showing which tree includes which node set, see also the ancillary files of \cite{Ref_Kleinnijenhuis_2020a}, \cite{Ref_Kleinnijenhuis_2020b}:

\[\arraycolsep=0.6em\def\arraystretch{1.4}
\begin{array}{llll}
\text{Original Collatz tree} & N(T_C)=N(H_U)&=&\mathbb{N^+} \hspace{0.6em}\text{if the Collatz conjecture holds}\\
\text{Syracuse tree} & N(T_T)=N(H_{C,3})&=&N(T_C)\setminus2\mathbb{N}\\
\text{Binary tree}\hspace{0.6em}T_{\ge0}& N(T_{\ge0})=S_{\ge0}&=&N(T_C)\setminus S_{-1}\hspace{2.1em}=S_{0}\cup S_{1}\cup S_{2}\ldots\\
\text{Binary tree}\hspace{0.6em}T_{\ge1}& N(T_{\ge1})=S_{\ge1}&=&N(T_C)\setminus \bigcup_{i=-1}^{0}S_i=S_{1}\cup S_{2}\cup S_{3}\ldots\\
\text{Binary tree}\hspace{0.6em}T_{\ge j} & N(T_{\ge j})=S_{\ge j}&=&N(T_C)\setminus \bigcup_{i=-1}^{j-1}S_i=\bigcup_{i=j}^{\infty}S_i
\end{array}
\]

\par\medskip
Let us describe these sets using multiplicative groups. The set $S_{\ge0}=\mathbb{Z}^\ast_6$ can be understood as the multiplicative group modulo $6$ and the set $S_{-1}=\mathbb{Z}/6\mathbb{Z}\setminus\mathbb{Z}^\ast_6=\{0,2,3,4\}$ as the set of all non-invertible elements (non-units) of $\mathbb{Z}/6\mathbb{Z}$.

The set $S_0$ consists of all nodes resulting as output of $R(n)$ within the binary tree $T_{\ge0}$. These are the orange highlighted nodes displayed by figures~\ref{fig:bt3},~\ref{fig:bt3_rot}. In other words, $S_0$ is the codomain of the function $R(n)$ operating on nodes within $T_{\ge0}$. The binary tree $T_{\ge0}$ can be transformed to a (pruned) binary tree $T_{\ge1}$. For this, the prunable nodes will be deleted and their neighbors reconnected. The upward neighbor of a pruned node will then be identified as pruning candidate for a later transformation of the resulting tree $T_{\ge1}$ to a more pruned tree $T_{\ge2}$.

The set $S_1$ contains all nodes that are (as per the description above) identified as pruning candidates for the next transformation of $T_{\ge1}$ to $T_{\ge2}$. After having transformed $T_{\ge1}$ to $T_{\ge2}$, the more pruned binary tree $T_{\ge2}$ contains nodes that are identified as pruning candidates for another upcoming transformation of $T_{\ge2}$ to $T_{\ge3}$ -- these nodes are elements of the set $S_2$. This pruning algorithm is repeatedly applied in the same pattern. And in this way we obtain the sets $S_1,S_2,S_3,\ldots$ and so forth. Generally, we can write these sets in the form $S_j=\{n\in N(T_{j-1})\mid U^{-j}(n)\in S_0\}$. Kleinnijenhuis found out that the codomain $\mathbb{N}^U$ of the Upward function contains $5$ residue classes modulo $96$, namely $\{5, 29, 53, 77, 85\}=\mathbb{N}^U$ and the codomain $\mathbb{N}^R$ of the Rightward function comprises $27$ residue classes modulo $96$, namely $\{1, 7, 11, 13, 17, 19, 23, 25, 31, 35, 37, 41, 43, 47, 49, 55, 59, 61, 65, 67, 71, 73, 79, 83, 89, 91, 95\}=\mathbb{N}^R$. The union of both sets $\mathbb{N}^U\cup\mathbb{N}^R$ forms the non-cyclic multiplicative group $\mathbb{Z}^\ast_{96}$, whose generating set is $\{5, 17, 31\}$ (see \cite{Ref_Lang_2017}, \cite{Ref_OESIS_A033949}). All elements of the Upward function's codomain have the same remainder $5$ when divided by $8$.

For each subset $X$ of a group $G$, the intersection over all subgroups (of $G$) that contain this subset $X$ is \cite[p.~34]{Ref_Karpfinger_Meyberg_2017}:
\[
\left<X\right>=\bigcap_{X\subseteq U\le G}U
\]

Firstly it applies $\left<X\right>\le G$ meaning that this intersection is again a subgroup of $G$. It is generated by the \textit{generating set} $X$ and it is the smallest subgroup of $G$ containing every element of $X$ \cite[p.~35]{Ref_Karpfinger_Meyberg_2017}. Secondly, $\left<X\right>\subseteq U$ for each subgroup $U$ (of $G$) containing $X$.  Thirdly, when there is only a single element $x$ in $X$, then $\left<X\right>$ is usually written as $\left<x\right>$ and in this case, $\left<x\right>$ is the cyclic subgroup of $G$ -- such situations we have already seen in section~\ref{sec:left_child_right_sibling_3}. Let us refer back to $\mathbb{Z}^\ast_{96}$. In this example, $\left<\{5,17,31\}\right>$ is the subgroup generated by $\{5,17,31\}$ and therefore every element of $\mathbb{Z}^\ast_{96}$ is of the form $5^l17^m31^n$ where $l\in\{0,1,\ldots,7\}$ because the element $5$ has order $8$, and similarly $m,n\in\{0,1\}$ since both elements $17$ and $31$ have order $2$. Non-cyclic groups can be cyclic decomposed, which is detailed by Gallian and Rusin \cite{Ref_Gallian_Rusin_1980} and Cheng \cite{Ref_Cheng_1989} using the concept of the external direct product \cite[p.~79]{Ref_Karpfinger_Meyberg_2017}, \cite[p.~156]{Ref_Gallian} and the internal direct product \cite[p.~80]{Ref_Karpfinger_Meyberg_2017}, \cite[p.~183]{Ref_Gallian}. A comprehensive table of cyclic decompositions of multiplicative non-cyclic groups of integers modulo $n$ up to $n=130$ is provided by Wolfdieter Lang \cite{Ref_Lang_2017}.

% http://mathonline.wikidot.com/the-internal-direct-product-of-two-groups

Let us take a closer look at the (cyclic) multiplicative group $\mathbb{Z}^\ast_{18}=\{1,5,7,11,13,17\}=\left<5\right>$ which has an order $ord(\mathbb{Z}^\ast_{18})=6$. Having the generator $5$ coprime to the modulus $18$, we obtain the congruence $5^{\phi(18)}\equiv1\pmod{18}$ in accordance with Euler's theorem \ref{eq:eulers_theorem}. This allows us to infer from $5^6\equiv5^{6(n+1)}\equiv5^j5^{6n+6-j}\equiv1\pmod{18}$ the congruences given by \ref{eq:homomorphism_congruences} (on the left).

If a natural number divides another, $m\mid n$, as in our case $3\mid18$, then for two integers $a,b$ the following implication holds, see \cite[p.~21]{Ref_Mueller-Stach_2011}:

\begin{equation}
	\label{eq:reduce_modulus}
	a\equiv b\pmod n\rightarrow a\equiv b\pmod m
\end{equation}

This means in our case $w\cdot5^{6n+6}\equiv 1\pmod{18}\rightarrow w\cdot5^{6n+6}\equiv 1\pmod 3$. Based on equation~\ref{eq:congruence_reduction} we can state that $a\equiv b\pmod m$ implies $(a+m)\equiv b\pmod m$. Let us set $a=w\cdot2^{2n+2}$ and $b=1$ and $m=3$, then we obtain $(w\cdot2^{2n+2}+3)\equiv1\pmod3$ and using a factor $i\in\mathbb{N}$ we obtain the more general congruence $(w\cdot2^{2n+2}+3i)\equiv1\pmod3$. As a consequence the congruences in \ref{eq:homomorphism_congruences} are true, while $w\cdot5^{6n+6}=w\cdot2^{2n+2}+3\cdot i$ or rather while $3\mid(5^{6n+6}-2^{2n+2})$. These conditions continue $3\mid(5^{6n+5}-2^{2n+3})$ and $3\mid(5^{6n+4}-2^{2n+4})$ and so forth.

\begin{equation}
\label{eq:homomorphism_congruences}
\begin{array}{llllll}
	j=0, & w\in [1]_{18} & \hspace{1em} w\cdot5^{6n+6}&\equiv1\pmod{18}&\hspace{4em}w\cdot2^{2n+2} &\equiv1\pmod{3}\\
	j=1, & w\in [5]_{18} & \hspace{1em} w\cdot5^{6n+5}&\equiv1\pmod{18}&\hspace{4em}w\cdot2^{2n+3} &\equiv1\pmod{3}\\
	j=2, & w\in [7]_{18} & \hspace{1em} w\cdot5^{6n+4}&\equiv1\pmod{18}&\hspace{4em}w\cdot2^{2n+4} &\equiv1\pmod{3}\\
	j=3, & w\in [17]_{18} & \hspace{1em} w\cdot5^{6n+3}&\equiv1\pmod{18}&\hspace{4em}w\cdot2^{2n+1} &\equiv1\pmod{3}\\
	j=4, & w\in [13]_{18} & \hspace{1em} w\cdot5^{6n+2}&\equiv1\pmod{18}&\hspace{4em}w\cdot2^{2n+2} &\equiv1\pmod{3}\\
	j=5, & w\in [11]_{18} & \hspace{1em} w\cdot5^{6n+1}&\equiv1\pmod{18}&\hspace{4em}w\cdot2^{2n+1} &\equiv1\pmod{3}
\end{array}
\end{equation}

Moreover, the map $f:\mathbb{Z}^\ast_n\rightarrow \mathbb{Z}^\ast_m$ with $f(r\bmod n)=r\bmod m$ is a homomorphism as long as $\gcd(r\bmod n,n)=1$ leads to $\gcd(r\bmod m,m)=1$. By Euclid, we know that in the case $r$ is coprime to $n$ then it is also coprime to every factor $m$ of $n$. That is why a homomorphism exist to the congruences shown in \ref{eq:homomorphism_congruences} (on the right).

Figure~\ref{fig:tree_transformations} shows the complete chain of tree transformations, beginning from the original Collatz tree, over the Syracuse tree to the binary tree and pruned ones.

% trim=left bottom right top
\begin{figure}[H]
	\includegraphics[trim=1.1cm 10cm 2.6cm 0.2cm,
	width=1.00\textwidth,page=1]{figures/tree_transformations.pdf}
	\caption{Transformation chain, beginning from the original Collatz tree up to pruned binary trees}
	\label{fig:tree_transformations}
\end{figure}


\subsection{Bilinear forms}\label{subsec:bilinear_forms}

\begin{definition}\label{def:bilinear_form}\mcite[249]{Knapp2016BasicAlgebra}
  Let \( M \) and \( N \) be left \( R \)-modules and \( L: M \times N \to R \) be a \hyperref[def:multilinear_function]{multilinear function}. We say that \( L \) is a \term{bilinear form}.

  If \( M = N \), we have the following additional types of bilinear forms:
  \begin{thmenum}
    \thmitem{def:bilinear_form/symmetric} If \( L \) is a \hyperref[def:multi_valued_function/symmetric]{symmetric function}, we say that is is a \term{symmetric bilinear form}

    \thmitem{def:bilinear_form/skew_symmetric} If for all \( x, y \in M \) instead of \( L(x, y) = L(y, x) \) we have \( L(x, y) = -L(y, x) \), we say that \( L \) is \term{skew-symmetric}.

    \thmitem{def:bilinear_form/alternating} If for all \( x \in M \) we have \( L(x, x) = 0 \), we say that \( L \) is \term{alternating}.
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:skew_symmetric_iff_alternating}
  Let \( 1 + 1 = 2 \) be a unit in the ring \( R \). Let \( M \) be a left module over \( R \). Then the bilinear form \( L: M \times M \to R \) is \hyperref[def:bilinear_form/alternating]{alternating} if and only if it is \hyperref[def:bilinear_form/skew_symmetric]{skew-symmetric}.

  Alternating implies skew-symmetric even if \( 2 \) is not invertible.
\end{proposition}
\begin{proof}
  \SufficiencySubProof Let \( L \) be alternating. Then
  \begin{balign*}
    L(x + y, x + y) & = L(x, x) + L(x, y) + L(y, x) + L(y, y) \\
    0               & = L(x, y) + L(y, x)                     \\
    L(x, y) = -L(y, x).
  \end{balign*}

  \NecessitySubProof Let \( L \) be skew-symmetric. Then
  \begin{equation*}
    L(x, x) = -L(x, x),
  \end{equation*}
  which implies that \( 2L(x, y) = 0 \). Hence, \( L \) is alternating if we are able to divide by \( 2 \).
\end{proof}

\begin{definition}\label{def:bilinear_form_radicals}\mcite[250]{Knapp2016BasicAlgebra}
  Let \( L: M \times N \to R \) be a bilinear form. We define its \term{left radical}
  \begin{equation*}
    \{ x \in M \colon \forall y \in N, \inprod x y = 0 \}
  \end{equation*}
  and \term{right radical}
  \begin{equation*}
    \{ y \in N \colon \forall x \in M, \inprod x y = 0 \}.
  \end{equation*}

  Note that if \( L \) is symmetric or skew-symmetric (which also implies \( M = N \)), the two are identical and we speak simply of the \term{radical} \( \sqrt L \).
\end{definition}

\begin{definition}\label{def:nondegenerate_bilinear_form}\mcite[249]{Knapp2016BasicAlgebra}
  We say that a bilinear form \( L: M \times N \to R \) is \term{nondegenerate} if both its left and right \hyperref[def:bilinear_form_radicals]{radicals} are nontrivial.
\end{definition}

\begin{theorem}\label{thm:bilinear_form_matrix_presentation}
  Fix a commutative unital ring \( R \) and a bilinear form \( L: R^n \times R^m \to R \). Then there exists a matrix \( A \in R^{n \times m} \) such that
  \begin{equation*}
    L(x, y) \coloneqq x^T A y.
  \end{equation*}

  This matrix is called the generalized \term{Gram matrix}.

  In particular, if \( L \) is \hyperref[def:multi_valued_function/symmetric]{symmetric}, so it \( A \).
\end{theorem}
\begin{proof}
  Denote by \( e_1, \ldots, e_n \) the basis of \( R^n \) and by \( f_1, \ldots, f_m \) the basis of \( R^m \).

  Define the matrix \( A = \{ a_{i,j} \}_{i,j=1}^{n,m} \) by
  \begin{equation*}
    a_{i,j} \coloneqq L(e_i, f_j).
  \end{equation*}

  Note that if \( n = m \) and if \( L \) is symmetric, then the matrix \( A \) is obviously symmetric too.

  For any fixed basis vector \( e_i, i = 1, \ldots, n \) of \( R^n \), we have
  \begin{equation*}
    L(e_i, y)
    =
    \sum_{j=1}^m y_i L(e_i, f_j)
    =
    y_i a_{(i,-)},
  \end{equation*}
  where \( a_{(i,-)} \) is the \( i \)-th row of \( A \).

  Thus, for an arbitrary \( x \in R^n \)
  \begin{equation*}
    L(x, y)
    =
    \sum_{i=1}^n x_i L(e_i, y)
    =
    \sum_{i=1}^n x_i (a_{(i,-)} y)
    =
    \left( \sum_{i=1}^n x_i a_{(i,-)} \right) y
    =
    x^T A y.
  \end{equation*}
\end{proof}

\begin{corollary}\label{thm:bilinear_forms_isomorphic_to_matrices}
  Fix a commutative unital ring \( R \). The vector space of bilinear forms of type \( R^n \times R^m \to R \) is isomorphic to the matrix space \( A \in R^{n \times m} \).
\end{corollary}

\begin{definition}\label{def:sesquilinear_form}\mcite[258]{Knapp2016BasicAlgebra}
  Let \( V \) be a complex vector space and let \( \overline V \) be its conjugate \hyperref[def:complex_conjucate_vector_space]{transpose}. We call the bilinear form \( L: V \times \overline V \to \BbbC \) a \term{sesquilinear form} (we say that \( L \) is \enquote{semilinear} in its second argument and \enquote{sesqui} means \enquote{one and a half} is Latin).

  Similar to \fullref{def:bilinear_form}, we have
  \begin{thmenum}
    \thmitem{def:sesquilinear_form/hermitian} If for all \( x, y \in V \) we have \( L(x, y) = \overline{L(y, x)} \), we say that \( L \) is \term{Hermitian}.

    \thmitem{def:sesquilinear_form/skew_hermitian} If for all \( x, y \in V \) we have \( L(x, y) = -\overline{L(y, x)} \), we say that \( L \) is \term{skew-Hermitian}.
  \end{thmenum}
\end{definition}

\begin{definition}\label{def:quadratic_form}
  If \( L: M \times M \to R \) be a bilinear \hyperref[def:bilinear_form]{form}, we call the function
  \begin{equation*}
    Q(x) \coloneqq L(x, x)
  \end{equation*}
  a \term{quadratic form} over \( M \).
\end{definition}

\begin{definition}\label{def:quadratic_form_definiteness}
  \todo{Define definiteness of quadratic forms}
\end{definition}

\begin{definition}\label{def:homogenous_function}
  Let \( M \) and \( N \) be left \( R \)-modules. We say that the function \( f: M \to N \) is homogeneous with degree \( n \) if for all \( t \in R \) and \( x \in M \) we have
  \begin{equation*}
    f(t x) = t^n f(x).
  \end{equation*}
\end{definition}

\begin{proposition}\label{thm:bilinear_forms_vs_to_quadratic_forms}
  A \term{quadratic form} \( Q: M \to R \) is a \hyperref[def:homogenous_function]{homogeneous function} of degree \( 2 \). In particular, \( Q(x) = Q(-x) \).
\end{proposition}
\begin{proof}
  Let \( L: M \times M \to R \) be the corresponding bilinear form. Then, by \fullref{def:semimodule/homomorphism/homogeneity},
  \begin{equation*}
    Q(tx) = L(tx, tx) = t^2 L(x, x) = t^2 Q(x).
  \end{equation*}
\end{proof}

\begin{proposition}\label{thm:polarization_identity}\mcite{nLab:polarization_identity}
  Let \( L: M \times M \to R \) be a bilinear \hyperref[def:bilinear_form]{form} and \( Q: M \to R \) be its associated quadratic \hyperref[def:quadratic_form]{form}. Then the \term{polarization identity} holds:
  \begin{equation}\label{thm:polarization_identity/polarization_identity}
    2 L(x, y) + 2 L(y, x) = Q(x + y) - Q(x - y)
  \end{equation}

  The similar looking, but slightly less useful parallelogram law also holds:
  \begin{equation}\label{thm:polarization_identity/parallelogram_law}
    2 Q(x) + 2 Q(y) = Q(x + y) + Q(x - y)
  \end{equation}

  If \( 2 = 1 + 1 \) is a unit in \( R \), we can \enquote{recover} from \( Q \) the bilinear form:
  \begin{equation}\label{thm:polarization_identity/symmetrization_definition}
    \hat L(x, y) \coloneqq \frac 1 2 \left[ Q(x + y) - Q(x) - Q(y) \right]
  \end{equation}

  The function \( \hat L \) is \hyperref[def:multi_valued_function/symmetric]{symmetric} and is called the \term{symmetrization} of \( L \). If \( L \) itself is symmetric, \( L = \hat L \).
\end{proposition}
\begin{proof}
  Identities \fullref{thm:polarization_identity/polarization_identity,thm:polarization_identity/parallelogram_law,thm:polarization_identity/symmetrization_definition} all follow from the bilinearity of \( L \), that is,
  \begin{equation*}
    Q(x \pm y)
    =
    L(x, x) \pm L(x, y) \pm L(y, x) + L(y, y)
    =
    [Q(x) + Q(y)] \pm [L(x, y) + L(y, x)].
  \end{equation*}
\end{proof}

\begin{definition}\label{def:orthogonality}
  Let \( U \) and \( V \) be vector spaces over \( \BbbK \) and let \( L: U \times V \to \BbbK \) be a nondegenerate bilinear form. We say that the vectors \( x \in U \) and \( y \in V \) are \term{orthogonal} with respect to \( L \) if
  \begin{equation*}
    L(x, y) = 0.
  \end{equation*}

  For every subspace \( U \subseteq V \) we define its \term{orthogonal complement} with respect to \( L \) as
  \begin{equation*}
    U^\perp \coloneqq \set{ x \in U \colon L(x, y) = 0 \T{for all} y \in V }
  \end{equation*}
  and analogously for submodules of \( V \).

  Let \( \mscrK \) be an index set and \( \seq{ x_k }_{k \in \mscrK} \subseteq U \), \( \seq{ y_k }_{k \in \mscrK} \subseteq V \) be two families of vectors indexed by \( \mscrK \). We say that these families form a \term{biorthogonal system} with respect to \( L \) if
  \begin{equation*}
    L(x_k, y_m) = 0 \text{ follows from } k \neq m
  \end{equation*}

  If \( U = V \), we usually consider \term{orthogonal systems} \( \seq{ x_k }_{k \in \mscrK} \subseteq V \) where
  \begin{equation*}
    L(x_k, x_m) = 0 \iff k \neq m
  \end{equation*}
\end{definition}

\begin{definition}\label{def:inner_product_space}
  An \term{inner product space} is a vector space \( V \) over \( F \) equipped with a positive \hyperref[def:quadratic_form_definiteness]{definite} \hyperref[def:bilinear_form/symmetric]{symmetric} bilinear form \( \inprod \cdot \cdot: V \times V \to F \).

  In the special case where \( F = \BbbC \), by convention we require \( V \) to instead be equipped with a positive definite \hyperref[def:sesquilinear_form/hermitian]{Hermitian} sesquilinear form instead.
\end{definition}

\begin{definition}\label{def:symplectic_vector_space}
  A \term{symplectic vector space} is a vector space \( V \) over \( F \) equipped with a \hyperref[def:bilinear_form/symmetric]{nondegenerate}  \hyperref[def:bilinear_form/alternating]{alternating} bilinear form \( \inprod \cdot \cdot: V \times V \to F \).
\end{definition}

\begin{lemma}\label{thm:inner_product_quadratic_form_is_positive_definite}
  Let \( V \) be a real or complex \hyperref[def:inner_product_space]{inner product space} with product \( \inprod \cdot \cdot \). The function \( Q(x) \coloneqq \inprod x x \) (which is not a quadratic form in the complex case) is positive definite.
\end{lemma}
\begin{proof}
  The real case is trivial. Assume that \( V \) is a complex vector space and that \( \inprod \cdot \cdot \) is Hermitian. This implies that \( \inprod x x = \overline{\inprod x x} \), thus \( \inprod x x \in \BbbR \). Furthermore, since the inner product is positive definite, we have \( Q(x) = \inprod x x \geq 0 \). Thus, \( Q \) is nonnegative real valued.

  Since \( \inprod \cdot \cdot \) is positive definite, so is \( Q \).
\end{proof}

\begin{theorem}[Cauchy-Bunyakovsky-Schwarz inequality]\label{thm:cauchy_bunyakovsky_schwarz_inequality}
  Let \( V \) be a real or complex \hyperref[def:inner_product_space]{inner product space} with product \( \inprod \cdot \cdot \). For every \( x, y \in V \) it holds that
  \begin{equation}\label{thm:cauchy_bunyakovsky_schwarz_inequality/inequality}
    {\abs{\inprod x y}}^2 \leq \inprod x x \inprod y y.
  \end{equation}

  Furthermore, equality is achieved if and only if \( x \) and \( y \) are linearly dependent.
\end{theorem}
\begin{proof}
  Note that we use this theorem to prove that the induced norm is a norm, so we cannot use the norm here. Associate with \( \inprod \cdot \cdot \) the function \( Q(x) \coloneqq \inprod x x \). By \fullref{thm:inner_product_quadratic_form_is_positive_definite}, \( Q \) is positive definite.

  Fix \( x, y \in V \) and \( t \in \BbbC \). If either vector is zero the statement is trivially true, so let both be nonzero. We have
  \begin{balign*}
    Q(x + ty)
     & =
    \inprod {x + ty} {x + ty}
    =    \\ &=
    Q(x) + \overline t \inprod x y + t \inprod y x + \abs{t}^2 Q(y)
    =    \\ &=
    Q(x) + 2\real t \overline{\inprod x y} + \abs{t}^2 Q(y)
  \end{balign*}

  Take \( t \coloneqq - \frac {\inprod x y} {Q(y)} \), so that
  \begin{equation*}
    Q(x + ty)
    =
    Q(x) - 2 \frac {\abs{\inprod x y}^2} {Q(y)} + \frac {\abs{\inprod x y}^2} {Q(y)}
    =
    Q(x) - \frac {\abs{\inprod x y}^2} {Q(y)}
  \end{equation*}

  Since \( Q(x + ty) \geq 0 \), it follows that
  \begin{balign*}
    Q(x) - \frac {\abs{\inprod x y}^2} {Q(y)} & \geq 0                  \\
    Q(x) Q(y)                               & \geq \abs{\inprod x y}^2.
  \end{balign*}

  If \( x \) and \( y \) are linearly dependent, equality obviously holds. Conversely, suppose that equality holds. This implies that
  \begin{equation*}
    Q(x + ty) = 0,
  \end{equation*}
  which by the positive definiteness of \( Q \) means that \( x = -ty \). Thus, \( x \) and \( y \) are linearly dependent.
\end{proof}

\begin{definition}\label{def:bilinear_form_induced_norm}
  Let \( V \) be a real or complex \hyperref[def:inner_product_space]{inner product space} with product \( \inprod \cdot \cdot \). We define its induced \hyperref[def:norm]{norm} as
  \begin{balign*}
     & \norm \cdot : V \to \BbbR_{\geq 0}    \\
     & \norm x \coloneqq \sqrt{\inprod x x}.
  \end{balign*}

  If \( V \) is a real inner product space, the induced norm is a square root of the induced quadratic \hyperref[def:quadratic_form]{form} of \( \inprod \cdot \cdot \).
\end{definition}
\begin{proof}
  We will only prove the complex case because the real case is identical, but slightly simpler.

  Note that \( \norm \cdot \) is well-defined (that is, positive definite) by \fullref{thm:inner_product_quadratic_form_is_positive_definite}.

  Now we will show that it is a norm.
  \SubProofOf{def:norm/N1} Follows from the positive definiteness of \( \inprod \cdot \cdot \)

  \SubProofOf{def:norm/N2} For \( t \in \BbbC \) and \( x \in V \) we have
  \begin{equation*}
    \norm{tx} = \sqrt{\inprod{tx} {tx}} = \abs{t} \sqrt{\inprod x x} = \abs t \norm x.
  \end{equation*}

  \SubProofOf{def:norm/N3} For \( x, y \in V \) we have
  \begin{balign*}
    \norm{x + y}^2
     & =
    \inprod{x + y} {x + y}
    =                                                            \\ &=
    \inprod x x + \inprod x y + \inprod y x + \inprod y y
    =                                                            \\ &=
    \norm{x}^2 + 2 \real \inprod x y + \norm{y}^2
    \leq                                                         \\ &\leq
    \norm{x}^2 + 2 \abs{\real \inprod x y} + \norm{y}^2
    \reloset {\ref{thm:cauchy_bunyakovsky_schwarz_inequality}} = \\ &=
    \norm{x}^2 + 2 \norm x \norm y + \norm{y}^2
    =
    (\norm{x} + \norm{y})^2
  \end{balign*}

  Therefore,
  \begin{equation*}
    \norm{x + y} \leq \norm x + \norm y.
  \end{equation*}
\end{proof}


\section{Correctness of the Bayes Filter Algorithm}\label{sec:ap:bayes-filter}
The current posterior distribution is calculated as it is shown in equation \ref{eq:ch-2:bel-derivation}.

\begin{equation}\label{eq:ch-2:bel-derivation}
p(x_t| z_{1:t}, u_{1:t}) = \frac{p(z_t | x_t, z_{1:t-1}, u_{1:t}) \: p(x_t | z_{1:t-1}, u_{1:t})}{p(z_t | z_{1:t-1}, u_{1:t})}
\end{equation}

Where,

\begin{itemize}
\item $p(x_t | z_{1:t-1}, u_{1:t})$ is the prior distribution. That is the information of state $x_t$ before seen the observation at time $t$.
\item $p(z_t | x_t, z_{1:t-1}, u_{1:t})$ is the likelihood model for the measurements. A causal, but noisy relationship\cite{Sarkka:bayesian-filters}.
\item $p(z_t | z_{1:t-1}, u_{1:t})$ is the normalization constant defined as $\eta$
\end{itemize}

Thus equation \ref{eq:ch-2:bel-derivation} can be summarized as follows:

\begin{equation} \label{eq:ch-2:bel-derivation-2}
p(x_t| z_{1:t}, u_{1:t}) = \eta \: p(z_t | x_t, z_{1:t-1}, u_{1:t}) \: p(x_t | z_{1:t-1}, u_{1:t})
\end{equation}

The prediction of observation $z_t$ based on the state $x_t$, the previous observation $z_{1:t-1}$ and the control action $u_{1:t}$ has a conditional independence regarding the previous observation and the control action because they do not aport any information while predicting $z_t$. Thus the likelihood model for the measurements can be simplified as follows:

\begin{equation}
p(z_t | x_t, z_{1:t-1}, u_{1:t}) = p(z_t | x_t)
\end{equation}

Therefore equation \ref{eq:ch-2:bel-derivation-2} reduces to:

\begin{equation} \label{eq:ch-2:bel-derivation-3}
p(x_t| z_{1:t}, u_{1:t}) = \eta \: p(z_t | x_t) \: p(x_t | z_{1:t-1}, u_{1:t})
\end{equation}

In equation \ref{eq:ch-2:expand-prior-here} the prior distribution is expanded.

\begin{equation}\label{eq:ch-2:expand-prior-here}
p(x_t | z_{1:t-1}, u_{1:t}) = \int p(x_t | x_{t-1}, u_t) \: p(x_{t-1} | z_{1:t-1}, u_{1:t-1}) dx_{t-1}\\
= \overline{bel}(x_t)
\end{equation}

Replacing equation \ref{eq:ch-2:expand-prior-here} in \ref{eq:ch-2:bel-derivation-3}, equation \ref{eq:ch-2:bel-derivation-4} is obtained which is calculated by the algorithm in list \ref{ch-2:algo:bayes-filter}, third line.

\begin{equation}\label{eq:ch-2:bel-derivation-4}
p(x_t| z_{1:t}, u_{1:t}) = \eta \: p(z_t | x_t) \: \overline{bel}(x_t)
\end{equation}


\subsubsection{The factorization properties}

Let us resume the discussion from last section, where we claim that
all we needed  was that the vectors of
the two bases had the property of $A\vec{v_i} =
\sigma_i\vec{u_i}$. Let us prove such claim, and show that if that
condition is met, then SVD factorization can be achieved. \\

\begin{theorem}[SVD Part 1: the factorization]
\label{thm:SVD1}
Let $A$ be a real matrix of $n \times m$,\text{if } $\exists$
orthonormal basis $\{\vec{v_1},\vec{v_2},\dots,\vec{v_n}\}$ for \R{n},
and another orthonormal $\{\vec{u_1},\vec{u_2},\dots,\vec{u_m}\}$ for
\R{m} which hold the following property:

\[
A\vec{v_i} = \sigma_i\vec{u_i}, \ds{}\forall i=1 \dots r, \ds{}\text{where
} r = rank(A).
\]
\\
Then we can factorize matrix $A$ as $U \Sigma \trans{V}$. \\

where

\begin{itemize}
\item $V$ is the orthogonal matrix formed by arranging vectors
\vec{v}\apos{s}
\item matrix $U$ is defined similarly for vectors
\vec{u}\apos{s}
\item The only non zero entries of diagonal matrix $\Sigma$, are
those $\Sigma_{ii} = \sigma_i > 0$ for $1 <= i <= r$.
\end{itemize}
\end{theorem}
\hfill

\begin{proof}
This \cref{thm:SVD1} is mentioned in \cite{kalman96}, though not
explicitly proved. Let us do it here, following the advice from
\cite{strang88}, that the trick is to think about a matrix
multiplication $A B$, as the result of matrix-vector products ($A
\vec{b_i}$), where the vectors are the columns of $B$. In our particular
case, the matrix-vector products we have are $A\vec{v_i}$; if we
arrange them as columns of a new matrix it would be equal to $A
V$. That is:

\[
\begin{bmatrix}
  A\vec{v_1} \mid A\vec{v_2} \mid \cdots \mid A\vec{v_n}
\end{bmatrix} =
A
\begin{bmatrix}
  \vec{v_1} \mid \vec{v_2} \mid \cdots \mid \vec{v_n}
\end{bmatrix} =
AV
\]
\hfill

Since we do not have product $AV$ in our target result, let us use the
fact that $V$ is orthogonal; which in particular implies that \inv{V}
= \trans{V}. That allows to focus on an target result, which involves
$AV$:

\[
A   = U \Sigma \trans{V} \ds{\iff}
A V = U \Sigma V \trans{V} \ds{\iff}
A V = U \Sigma
\]
\hfill

So we can focus in proving that $A V = U \Sigma$. Let us work
the left side first, which per our previous observation that
$A\vec{v_i}$ are the columns of matrix product $AV$, and per hypothesis
that $A\vec{v_i} = \sigma_i\vec{u_i}$ can be rewritten as follows:

\begin{align*}
  & A V &= \\
  & A \begin{bmatrix}
    \vec{v_1} \mid \vec{v_2} \mid \cdots \mid \vec{v_n}
  \end{bmatrix} &= \\
  & \begin{bmatrix}
      A\vec{v_1} \mid A\vec{v_2} \mid \cdots \mid A\vec{v_n}
    \end{bmatrix} &= \\
  & \begin{bmatrix}
      \sigma_1\vec{u_1} \mid \sigma_2\vec{u_2} \mid \dots \mid \sigma_r\vec{u_r}
      \mid A\vec{v_{r+1}} \mid A\vec{v_{r+2}} \mid \dots \mid A\vec{v_{n}}
    \end{bmatrix}
\end{align*}
\hfill

Let us now develop the left side $U \Sigma$ by thinking again in the
result, as formed by columns of the form $U \vec{\Sigma_i}$ (where
  \vec{\Sigma_i} is the $i$th column of diagonal matrix $\Sigma$):

\begin{align*}
  & U \Sigma &= \\
  & \begin{bmatrix}
      U\vec{\Sigma_1} \mid U\vec{\Sigma_2} \mid \cdots \mid U\vec{\Sigma_n}
    \end{bmatrix} &= \\
  & \begin{bmatrix}
      U\vec{\Sigma_1} \mid U\vec{\Sigma_2} \mid \cdots \mid U\vec{\Sigma_r} \mid
      U\vec{\Sigma_{r+1}} \mid U\vec{\Sigma_{r+2}} \mid \cdots \mid U\vec{\Sigma_{n}}
    \end{bmatrix} &= \\[1ex]
  & \begin{bmatrix}
      U\vec{\Sigma_1} \mid U\vec{\Sigma_2} \mid \cdots \mid U\vec{\Sigma_r} \mid
      \smash{\underbrace{U\vec{0} \mid U\vec{0} \mid \cdots
          \mid U\vec{0}}_{n - r}}
    \end{bmatrix} &= \\[2.5ex]
  & \begin{bmatrix}
      U\vec{\Sigma_1} \mid U\vec{\Sigma_2} \mid \cdots \mid U\vec{\Sigma_r} \mid
      \smash{\underbrace{\vec{0} \mid \vec{0} \mid \cdots \mid \vec{0}}_{n-r}}
    \end{bmatrix} &= \\[0.5ex]
\end{align*}

The last $n - r$ zero vectors were a consequence of the definition of
$\Sigma$, which only has non-zeroes on diagonal up to position
$r$. And the columns $U\vec{\Sigma_i}$ can be simplified further, as
each column vector \vec{\Sigma_i} has the only non-zero entry
$\sigma_i$ precisely at position $i$. Hence, only column $i$ of $U$
survives after multiplying it by \vec{\Sigma_i}, and the final effect
is just the multiplication by scalar $\sigma_i$:

\[
 U \Sigma =
\begin{bmatrix}
   U\vec{\Sigma_1} \mid U\vec{\Sigma_2} \mid \cdots \mid U\vec{\Sigma_r} \mid
   \smash{\underbrace{\vec{0} \mid \vec{0} \mid \cdots \mid \vec{0}}_{n-r}}
\end{bmatrix} =
\begin{bmatrix}
   \sigma_1\vec{u_1} \mid \sigma_2\vec{u_2} \mid \cdots \mid \sigma_1\vec{u_r} \mid
   \smash{\underbrace{\vec{0} \mid \vec{0} \mid \cdots \mid \vec{0}}_{n-r}}
\end{bmatrix}
\]
\hfill

If we put together the developments for each side, we are almost done:

\begin{align*}
  & A V &= \\[1.5ex]
  & \begin{bmatrix}
    \sigma_1\vec{u_1} \mid \sigma_2\vec{u_2} \mid \cdots \mid \sigma_r\vec{u_r} \mid
    \smash{\underbrace{A\vec{v_{r+1}} \mid A\vec{v_{r+2}} \mid \cdots \mid A\vec{v_{n}}}_{n-r}}
  \end{bmatrix}  &= \\[3.5ex]
  & \begin{bmatrix}
    \sigma_1\vec{u_1} \mid \sigma_2\vec{u_2} \mid \cdots \mid \sigma_1\vec{u_r} \mid
    \smash{\underbrace{\vec{0} \mid \vec{0} \mid \cdots \mid \vec{0}}_{n-r}}
  \end{bmatrix} &= \\[1.5ex]
  & U \Sigma
\end{align*}
\hfill

\end{proof}

It can be observed that the proof is not complete, though we are
almost done; in order to achieve an equality between $AV$ and
$U\Sigma$, the only missing part is that the last $n-r$ items on each
side are the same. This can be restated in an additional theorem: \\

\begin{restatable}[SVD Part2: basis of null space]{theorem}{svdtwo}
\label{thm:SVD2}
Assuming same definitions as \cref{thm:SVD1}, it must be the case that:

\[
A\vec{v_i} = \vec{0}, \textds{for} (r+1) \le i \le n
\]

which is equivalent to say that those vectors \vec{v_i}, belong to the
null space of $A$ (they actually form a basis of it).
\end{restatable}

We do not have yet the required machinery to proof
\cref{thm:SVD2}, but we will do it on the next section, when we
introduce the subspaces associated with each matrix $A$. \\


\subsection{Examples}

\begin{frame}{Universal Hashing}{Examples}
  \textbf{Negative example:}
  \begin{itemize}
    \item<2->
      The set of all {\color{MainA}$h$} for which
      {\color{MainA}$h_a(x) = (a \cdot x) \mod m$, for a $a \in \mathbb{U}$}
    \item<3->
      It is not {\color{MainA}$c$}-universal. \onslide<3-| handout: 0>{Why?}
    \item<4->
      If universal:
      {\color{MainA}
      \begin{displaymath}
        \forall x, y \hspace{1em} x \neq y\!: \; \frac{
          \vert \{h \in \mathbb{H}: h(x) = h(y)\} \vert
        }{
          \vert \mathbb{H} \vert
        } \leq c \cdot \frac{1}{m}
       \end{displaymath}}
    \vspace{-1em}
    \item<5->
      Which {\color{MainA}$x, y$} lead to a relative collision count bigger than {\color{MainA}$\frac{c}{m}$}?
  \end{itemize}
\end{frame}

%-------------------------------------------------------------------------------

\begin{frame}{Universal Hashing}{Examples}
  \textbf{Positive example:}
  \begin{itemize}
    \item<1->
      Let {\color{MainA}$p$} be a big prime number, {\color{MainA}$p > m$}
      and {\color{MainA}$p \geq \vert \mathbb{U} \vert$}
    \item<2->
      Let {\color{MainA}$\mathbb{H}$} be the set of all {\color{MainA}$h$}
      for which:
      {\color{MainA}
      \begin{align*}
        h_{a, b}(x) = ((a \cdot x + b) \mod p) \mod m,\\
        \hspace*{1.5em} \text{{\color{black}where }}
        1 \leq a < p, \; 0 \leq b < p
      \end{align*}}
    \item<3->
      This is {\color{MainA}$\approx1$-universal},
      see {\color{teal}Exercise 4.11} in Mehlhorn/Sanders
    \item<4->
      E.g.: {\color{MainA}$U=\{0,...,99\},$ \hspace{0.5em} $p=101,$
        \hspace{0.5em} $a=47,$ $b=5$}
   \item<5->
     Then {\color{MainA}$h(x) = ((47 \cdot x + 5) \mod 101) \mod m$}
   \item<6->
     Easy to implement but hard to proof
   \item<7->
     Exercise: show empirically that it is {\color{MainA}$2$}-universal
  \end{itemize}
\end{frame}

%-------------------------------------------------------------------------------

\begin{frame}{Universal Hashing}{Examples}
  \textbf{Positive example:}
  \begin{itemize}
    \item<1->
      The set of hash functions is {\color{MainA}$c$}-universal:
      {\color{MainA}
      \[h_{a}(x) = a \bullet x \mod m, \hspace*{1.5em} a \in \mathbb{U}\]}
    \item<2->
      \textbf{We define:}
      {\color{MainA}
      \begin{align*}
        a &= \sum_{0, \ldots, k-1} a_i \cdot m^i,
        \hspace*{1.5em} k = \mathrm{ceil}(\log_m \vert \mathbb{U} \vert)\\
        x &= \sum_{0, \ldots, k-1} x_i \cdot m^i
      \end{align*}}
    \item<3->
      \textbf{Intuitive}: scalar product with base {\color{MainA}$m$}
      {\color{MainA}
      \begin{align*}
        a \bullet x &= \sum_{0, \ldots, k-1} a_i \cdot x_i
      \end{align*}}
  \end{itemize}
\end{frame}

%-------------------------------------------------------------------------------

\begin{frame}{Universal Hashing}{Examples}
  \begin{example}[$\mathbb{U} = \{0, \dots, 999\}, \; m = 10, \; a = 348$]%
    With $a=348$: $a_2 = 3, \; a_1 = 4, \; a_0 = 8$
    \begin{align*}
      h_{348}(x) &= (a_2 \cdot x_2 + a_1 \cdot x_1 + a_0 \cdot x_0) \mod m\\
      {} &= (3 \, x_2 + 4 \, x_1 + 8 \, x_0) \mod 10
    \end{align*}
    With $x = 127$: $x_2 = 1, \; x_1 = 2, \; x_0 = 7$
    \begin{align*}
      h_{348}(127) &= (3 \cdot x_2 + 4 \cdot x_1 + 8 \cdot x_0) \mod 10\\
      {} &= (3 \cdot 1 + 4 \cdot 2 + 8 \cdot 7) \mod 10\\
      {} &= 7
    \end{align*}
  \end{example}
\end{frame}


\documentclass[../PHYS306Notes.tex]{subfiles}

\begin{document}
\subsection{Worksheet - Energy and the Hamiltonian}
\begin{p}
Show that if a coordinate transformation is "natural" (no explicit time-dependence, $\v{r} = \v{r}(q_1, \cdots, q_n)$) that the kinetic energy is a homogenous quadratic function of the velocities (a function is homogenous of degree $k$ if $f(\alpha x) = \alpha^kf(x)$. Find this quadratic function.
\end{p}
\begin{s}
Note that any function that is a power law will satisfy this property; it turns out that kinetic energy has this property (as we know). To solve the problem, we consider $\dot{\v{r}}_j$:
\[\dot{\v{r}}_j = \sum_{i=1}^n\dpd{\v{r}_j}{q_i}\dot{q}_i\]
To calculate $\dot{\v{r}}_j^2$, we multiply the two sums:
\[\dot{\v{r}}_j^2 = \left(\sum_i\dpd{\v{r}_j}{q_i}\dot{q}_i\right)\left(\sum_k\dpd{\v{r}_j}{q_k}\dot{q}_k\right)\]
This gives us:
\[T = \frac{1}{2}\sum_jm_j\dot{\v{r}}_j^2 = \frac{1}{2}\sum_{i,k}\dot{q}_i\dot{q}_j\left(\sum_j m_j\dpd{\v{r}_j}{q_i}\dpd{\v{r}_j}{q_k}\right)\]
This term on the right is a matrix which we can call $A_{ik}$. Once we sum over these components, we get the total energy. Now we can see that this is homogenous, quadratic in $\dot{\v{r}}_j$.
\end{s}

\begin{p}
Show then that the conjugate momenta have the form $p_i = \sum_j A_{ij}\dot{q}_j$, and thus that the hamiltonian $\HH = E$, the total energy. Thus, under the above conditions, symmetry under time translation is equivalent to conservation of energy.
\end{p}
\begin{s}
Using the result from above:
\[p_i = \dpd{\LL}{\dot{q}_i} = \dpd{T}{\dot{q}_i}  = \sum_j A_{ij}\dot{q}_j\]
Therefore:
\[\sum_i p_i\dot{q}_i = \sum_i\left(\sum_j A_{ij} \dot{q}_j\right)\dot{q}_i = \sum_{i,j}A_{ij} \dot{q}_i\dot{q}_j = 2T\]
Hence (using the defintion of $\HH$ from lecture):
\[\HH = 2T - \LL = 2T - (T - U) = T + U = E\]
Which is just the total energy.
\end{s}

\begin{p}
Show that the same derivation follows directly by using the property that the kinetic energy is a homogenous function of degree 2 in the generalized velocities (use Euler’s homogeneity theorem: $f$ is positively homogenous of degree $k$ if and only if $\v{x}\cdot\grad{f(\v{x}} = kf(\v{x})$.
\end{p}
\begin{s}
We have (letting $\v{x} = \dot{\v{q}}$:
\[\dot{\v{q}} \cdot \dpd{T}{\dot{\v{q}}} = \sum_i p_i \dot{q}_i = 2T\]
Where the last line follows from the homogeneity theorem.
\end{s}
\end{document}


\documentclass[12pt, a4paper, twoside,draft]{article}
\usepackage{amsmath}
\usepackage{framed}

\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\title{CASSANDRA (draft)\\ A framework for tensorial prediction \\ v0.1}

\date{\today}
% \date{\today} date coulde be today
% \date{25.12.00} or be a certain date
% \date{ } or there is no date
\begin{document}
\maketitle

\section{Abstract}
We present a method for tensorial prediction using a convolutional blind decomposition approach (filters convolved with activations) where the activation variables are shared between the input and the output.

\section{A framework for tensor evolution prediction using a tensor-based spectral convolutional method}

\subsection{Analysis Formulation - Multiple Tensors + Spectral Data fit}
Let $\mathbf{X} = [\mathbf{X}_1, \hdots, \mathbf{X}_D ]$ be a collection of $D$ tensors.
\begin{framed}
\textbf{Analysis formulation:} Identification of a common compositional structure (filters) among the $D$ tensors such that the $D$ tensors can be reconstructed, within a margin of error, by a finite sum of tensorial convolutions of the compositional structures (filters) and an activations of the compositional structures (activations).
\end{framed}
This can be achieved through the following optimization problem:
\begin{align}
f, \left[ \begin{matrix}h_1 \\ \vdots \\ h_D \end{matrix} \right] = \arg \min_{f,h} \left\Vert \left[ \begin{matrix}\mathbf{X}_1 \\ \vdots \\ \mathbf{X}_D \end{matrix} \right] - \sum_k f_k \ast  \left[ \begin{matrix}h_{1,k} \\ \vdots \\ h_{D,k} \end{matrix} \right] \right\Vert,
\end{align}
where $f_j$ denotes the $j$th compositional structure (filter) and $h_{i,j}$ denotes the activation of the $j$th compositional structure on the $i$th tensor ($X_i$).
This problem is biconvex and, generally, ill-defined, as the solution might not be unique (multiple combinations of filters and activations can reconstruct the collection of tensors).
To address the biconvexity of the problem, we find an approximate solution through an alternating optimization of the filters and their activations, in convex subproblems (addressed further ahead).
To address the fact that this optimization problem can be ill-defined, we can introduce priors on the activations (\emph{e.g.} sparse activations, activations smooth or piecewise smooth activations in specific dimensions, ...) or in the filters (\emph{e.g.} filters with fixed components in a number of dimensions, smooth or piecewise smooth activations in specific dimensions, ...).
Our problem becomes then:
\begin{align}
f, \left[ \begin{matrix}h_1 \\ \vdots \\ h_D \end{matrix} \right] = \arg \min_{f,h} \underbrace{\left\Vert \left[ \begin{matrix}\mathbf{X}_1 \\ \vdots \\ \mathbf{X}_D \end{matrix} \right] - \sum_k f_k \ast  \left[ \begin{matrix}h_{1,k} \\ \vdots \\ h_{D,k} \end{matrix} \right] \right\Vert }_{\textrm{data fit}} + \underbrace{\mathcal{P}_h(h) + \mathcal{P}_f(f) }_\textrm{prior information},
\end{align}
where $\mathcal{P}_h(h)$ denotes a prior on the activations and $\mathcal{P}_f(f)$ denotes a prior on the filters.

The solution of the data fit component of the optimization problem is computationally challenging as the data and the filters increase in number of dimensions and size.
We note however, that we can efficiently solve the data fit problem in the spectral domain as
\begin{align}
  \arg \min_{f,h} & \left\Vert \left[ \begin{matrix}\mathbf{X}_1 \\ \vdots \\ \mathbf{X}_D \end{matrix} \right] - \sum_k f_k \ast  \left[ \begin{matrix}h_{1,k} \\ \vdots \\ h_{D,k} \end{matrix} \right] \right\Vert  &=   \nonumber \\
  \arg \min_{f,h} & \left\Vert \mathcal{F}\left(\left[ \begin{matrix}\mathbf{X}_1 \\ \vdots \\ \mathbf{X}_D \end{matrix} \right] - \sum_k f_k \ast  \left[ \begin{matrix}h_{1,k} \\ \vdots \\ h_{D,k} \end{matrix} \right] \right) \right\Vert & = \nonumber \\
  \arg \min_{f,h} & \left\Vert \mathcal{F}\left(\left[ \begin{matrix}\mathbf{X}_1 \\ \vdots \\ \mathbf{X}_D \end{matrix} \right] \right) - \mathcal{F}\left(\sum_k f_k \ast  \left[ \begin{matrix}h_{1,k} \\ \vdots \\ h_{D,k} \end{matrix} \right] \right) \right\Vert & = \nonumber \\
  \arg \min_{f,h} & \left\Vert \mathcal{F}\left(\left[ \begin{matrix}\mathbf{X}_1 \\ \vdots \\ \mathbf{X}_D \end{matrix} \right] \right) - \sum_k \mathcal{F}(f_k)   \mathcal{F}\left(\left[ \begin{matrix}h_{1,k} \\ \vdots \\ h_{D,k} \end{matrix} \right] \right) \right\Vert &
\end{align}
where $\mathcal{F}$ denotes the Fourier transform.

\subsection{Prediction Formulation}
Let $\mathbf{X} = [\mathbf{X}_1, \hdots, \mathbf{X}_D ]$ be a collection of $D$ tensors representing $D$ different social signals, and
$\mathbf{Y} = [\mathbf{Y}_1, \hdots, \mathbf{Y}_M]$ a collection of $M$ tensors representing $M$ target social signals (social signals we want to predict).
Our goal is the estimation of $\mathbf{Y}$ from the knowledge of the \emph{directed} patterns of interaction $\mathbf{X} \rightarrow \mathbf{Y}$.

The  estimation of the directed patterns of interaction allows not only to understand patterns of interaction from a causality point of view (not approached in this document) but also to estimate $\mathbf{Y}$ given $\mathbf{X}$.
This is of utmost importance in the prediction of time-series and tensor evolution.

We approach the problem from the same approach as the analysis part.
\begin{align}
f_1,h_1 = \arg\min_{f,h} \| \mathbf{X} - \sum_{k=1}^{k_1} f_k \ast h_k \| + \mathcal{P}_h(h) + \mathcal{P}_f(f),
\end{align}
where $\mathcal{P}_h$ and  $\mathcal{P}_f$ denote priors on the activations and filters, respectively.
This problem can be solved in the spectral domain as follows,
\begin{align}
f_1,h_1 = \arg\min_{f,h} \| \mathbf{X} -  \sum_{k=1}^{k_1}  \mathcal{F}(f_k)  \mathcal{F}(  h_k) \| + \mathcal{P}_h(h) + \mathcal{P}_f (f),
\end{align}
where $\mathcal{F}(.)$ denotes the Fourier transform operator.
Assuming convexity of the priors on the activations and filters, this is a biconvex problem, which can be solved approximately as,
\begin{align}
  f^{t+1} = \arg\min_{f} \|  \mathcal{F}(\mathbf{X}) - \sum_{k=1}^{k_1}  \mathcal{F}(f_k)  \mathcal{F}(  h^{t}_k) \| + \mathcal{P}_f (f), \\
 h^{t+1} = \arg\min_{h} \|  \mathcal{F}(\mathbf{X}) - \sum_{k=1}^{k_1}  \mathcal{F}(f^{t+1}_k) \mathcal{F}(  h_k) \| +  \mathcal{P}_h (h).
\end{align}
This corresponds to finding simultaneously the filters and the activations for $\mathbf{X}$.

We then find the filters $g$ such that,
\begin{align}
g = \arg\min_{g} \|  \mathcal{F}(\mathbf{Y}) - \sum_{k=1}^{k_1}  \mathcal{F}(g_k) \mathcal{F}(  h_k) \| + \mathcal{P}_h(h) + \mathcal{P}_g(g).
\end{align}
This means that we find filters $g$ such that we can reconstruct $\mathbf{Y}$ from the filters $g$ and the activations derived from $\mathbf{X}$.

The pair of filters $f$ and $g$ provide a path between the known and accessible $\mathbf{X}$ and the unknown and $\mathbf{Y}$,
\begin{align}
  \textrm{train} & \nonumber\\
&  (\mathbf{X}_\textrm{train}) \rightarrow (\underline{f},h) \\
  & (\mathbf{Y}_\textrm{train}, h) \rightarrow(\underline{g}) \\
  \nonumber \\
  \textrm{predict} & \nonumber \\
  & (\mathbf{X}_\textrm{test}, \underline{f}) \rightarrow (h_\textrm{test})\\
&  (h_\textrm{test}, \underline{g}) \rightarrow (\mathbf{Y}_\textrm{predicted})
\end{align}

\end{document}


\section{Distance}
\label{sec:Distance}
In order to measure the similarity of different samples, the Dynamic Time Warping (DTW) algorithm was used. DTW compares sequences of feature vectors to one another, even if the sequences are of different lengths. This is important, since the samples in our sample base have different lengths. DTW stretches and compresses sample feature vector sequences for maximum similarity. It can thus find similarities in samples that are time-stretched versions of one another or that contain each other.\\
DTW first compares every feature vector of one sample to every feature vector of another sample by calculating the Euclidean distance between each feature vector. This creates a cost matrix $C$ of distances between every block $f_n$ of the first sample to every block $f'_m$ of the second sample. This matrix is of dimension $N \times M$, where $N$ and $M$ are the lengths of the feature sequences of both samples, respectively.\\
\[
    C_{n,m} = ||f_n - f'_m||
\]
Then, DTW searches for the cheapest path through $C$. In order to not having to search every possible path through this matrix, DTW only calculates the cheapest path $P_{n,m}$ from $C_{0,0}$ to $C_{n,m}$ for every positive time step from $P_{n-1,m}, P_{n,m-1}, P_{n-1,m-1}$ to $P_{n,m}$. This can be done recursively for all $n,m$ by starting from low $n,m$ and working line- or column-wise towards high $n,m$.\\
\[
    P_{n,m} = \min \left(
    \begin{array}{rll}
      P_{n-1,m}   &+ &C_{n,m} \cdot c_\text{horz}\\
      P_{n,m-1}   &+ &C_{n,m} \cdot c_\text{vert}\\
      P_{n-1,m-1} &+ &C_{n,m} \cdot c_\text{diag}
    \end{array}
    \right)
\]
where $c_\text{horz} = \frac{1}{N}$, $c_\text{vert} = \frac{1}{M}$, and $c_\text{diag} = ||\frac{1}{N},\frac{1}{M}||$ are the stepping distances through the matrix.\\
The final distance between the samples is then calculated by adding all the steps $C_{n,m} \cdot c_\text{x}$ on the cheapest paths from $C_{0,0}$ to $C_{N,M}$. This sum is equivalent to $P_{N,M}$. The stepping distances ensure that sample comparisions for long samples are not automatically bigger than sample comparisons of short samples.\\
This algorithm was implemented in Python, but it was too slow for practical comparisons of big sample sets. Thus, we further implemented it in C and called that version from Python, which provided two orders of magnitude of speedup. Still, calculating all the distances between all the samples in the sample database takes about two hours on a modern computer\footnote{Since the multiprocessing module in Python seems to have problems with Windowss, calculation times are four times longer there}. This distance matrix was saved as Pandas DataFrame to a HD5 database of 60~Mb.


% \section*{Parametric Bootstrap for the 100-year Atlantic wave}

\subsection*{a)}
We will here find the inverse of the atlantic wave distribution, approximated as a \textit{Gumpel Distribution}. To begin finding an inverse we insert a new variable for the function and solve for x. This is done below

\[ F(x;\mu,\beta))=\exp\left(-\exp\left(-\frac{x-\mu}{\beta}\right)\right), x \in \mathbb{R} \]
Here we insert $u$ for the function
\begin{align*}
&u = \exp \left \{ -\exp\left(-\frac{x-\mu}{\beta}\right)\right\} \Rightarrow \ln(u)=-\exp\left(-\frac{x-\mu}{\beta}\right) \Rightarrow \\
& \ln\left\{\ln\left(\frac{1}{u}\right)\right\}=-\frac{x-\mu}{\beta} \Rightarrow \mu-\beta \ln \left \{\ln\left ( \frac{1}{u}\right ) \right \}=x.
\end{align*}


\section{ODE Model}
To begin with a simplistic Ordinary Differential Equation (ODE) model was used. This would help us with forming expectations of the system and parameterization for the upcoming Agent Based Model (ABM). The ODE model is less computationally costly at the tradeoff of not being able to capture complex behaviour when compared to ABM.

The model we developed is based on a simple Logistic Model \cite{Logistic} but with an addition environmental dependence on carrying capacity. The ``environment" consists of the resources, oxygen and testosterone which have their own equations for production and consumption. Although a real cell would depend on a lot more resources, for simplicity they're all assumed to be in excess. The equations are given below:
\begin{equation}
  \frac{dy_i}{dt} = r_i y_i (1 - \frac{y_i}{K_{min} + \rho_i f_i(O_2) f_i(test)} )- \delta_i y_i
  \label{celleq}
\end{equation}
\begin{equation}
  \frac{dO_2}{dt} = p_{O_2} - \sum_i \mu_{O_2,i} y_i - \lambda_{O_2} O_2
  \label{o2eq}
\end{equation}
\begin{equation}
  \frac{dtest}{dt} = p_{test} y_{T^p} - \sum_i \mu_{test,i} y_i - \lambda_{test} test
  \label{testeq}
\end{equation}
\begin{equation}
  f_i(res) = \begin{cases}
    1 &\text{if } ul_{res,i} \leq res\\
    \frac{res-ll_{res,i}}{ul_{res,i}-ll_{res,i}} &\text{if } ll_{res,i} < res < ul_{res,i}\\
    0 &\text{if } res \leq ll_{res,i}\\
  \end{cases}
  \label{freseq}
\end{equation}

Where,
\begin{itemize}
  \item $i \in \{T^+,T^p,T-\}$ and $res \in \{O_2,test\}$
  \item $y_i$ is the no. of cells of cell type $i$
  \item $r_i$ is the population growth rate of cell type $i$
  \item $\delta_i$ is the population death rate of cell type $i$
  \item $K_{min}$ is the carrying capacity in the absence of resources.
  \item $\rho_i$ is the carrying capacity coming up through the environment/resources
  \item $f_{i,res}$ is the functional dependence of cell type $i$ on resource $res$, normalised to 1. Note that $f_{T^-,test}=1$
  \item $p_{res}$ is the production rate of resource, either as bulk or by cells
  \item $\mu_{res,i}$ is the uptake of resource $res$ by cell type $i$. Note that $\mu_{test,T^-} = 0$
  \item $\lambda_{res}$ is the decay rate of resource $res$
  \item $ll_{res,i}$ is the lower limit/threshold level of resource $res$ for carrying capacity of cell type $i$.
  \item $ul_{res,i}$ is the upper limit/saturation level of resource $res$ for carrying capacity of cell type $i$.
\end{itemize}

\subsection{Parameters}
\subsubsection{Resource Normalization}
For ease of use we took the typical tissue levels (TTL) of a resource and normalised the resource values with it. The corresponding parameters for the resource would also be normalised by the TTL. These values were taken from \cite{Steward} for oxygen and \cite{Titus} for testosterone.
\begin{table}[h]
\centering
\begin{tabular}{|l|l|}
  \hline
  Resource & Value             \\
  \hline
  $O_2$    & 2.5 mmHg          \\
  $test$   & 3.74 pmol/g tissue\\
  \hline
\end{tabular}
\end{table}
\subsubsection{Tissue Volume}
Some of the parameters differed from the other in terms of volume/mass of tumour. According to \cite{Stamey}, the volume of prostate cancer tumour vaired between 0.4 to 12 ml. A tumour volume of 1 ml was assumed for the model, which, translates to 1.03 g.
\subsubsection{Growth Parameters}
The following values were taken from \cite{ATCC-lncap,ATCC-22rv1} for doubling time ($\tau_d$) and \cite{Jain} for death rate. Considering the exponential phase of Equation \ref{celleq} with low cell numbers and excess resources,
\begin{equation}
r_i = \frac{ln(2)}{\tau_d} + \delta_i
\end{equation}
\begin{table}[h]
\centering
\begin{tabular}{|l|l|l|l|l|}
  \hline
  Cell Type & Cell Line & $\tau_d$ (hr) & $\delta_i$ (min$^{-1}$) & $r_i$ (min$^{-1}$)    \\
  \hline
  $T^+$     & LNCap     & 34            & $2.5 \times 10^{-3}$    & $2.84 \times 10^{-3}$ \\
  $T^p$     & 22Rv1     & 40            & $2.5 \times 10^{-3}$    & $2.79 \times 10^{-3}$ \\
  $T^-$     & PC3       & 25            & $1.6 \times 10^{-4}$    & $6.23 \times 10^{-4}$ \\
  \hline
\end{tabular}
\end{table}
\subsubsection{Carrying Capacity}
Due to constitutive dependence on the resources and to avoid divided by zero, $K_{min}$ is set to 1. \\
Assuming the equilibrium values of the cells ($y^*$) to be 10000 in the excess of resources, the values of $\rho$ were obtained on setting Equation \ref{celleq} to 0,
\begin{equation}
\rho_i=\frac{r_i}{r_i-\delta_i} y^*
\end{equation}
\begin{table}[h]
\centering
\begin{tabular}{|l|l|}
  \hline
  Cell Type & $\rho$             \\
  \hline
  $T^+$     & $8.35 \times 10^4$ \\
  $T^p$     & $9.62 \times 10^4$ \\
  $T^-$     & $1.34 \times 10^4$ \\
  \hline
\end{tabular}
\end{table}
\subsubsection{Oxygen uptake and production}
The following values were taken from \cite{HailJr}.
\begin{table}[h]
\centering
\begin{tabular}{|l|l|l|}
  \hline
  Cell Type & Raw value (nmol/min/$10^6$ cells) & Normalised (min$^{-1}$cell$^{-1}$) \\
  \hline
  $T^+$     & 5.5                               & $1.63 \times 10^{-6}$              \\
  $T^p$     & 5.5                               & $1.63 \times 10^{-6}$              \\
  $T^-$     & 3.5                               & $1.04 \times 10^{-6}$              \\
  \hline
\end{tabular}
\end{table}
\\Using the uptake values for $T^-$ and assuming the equilibrium value for $O_2$ to be TTL (normalised to 1) in Equation \ref{o2eq} set to 0,
\begin{equation}
  p_{O_2} = \lambda_{O_2} O_2^* + y_i^* \mu_i
\end{equation}
\subsubsection{Testosterone uptake and production}
Assuming the equilibrium value for $test$ to be TTL (normalised to 1) in Equation \ref{testeq} set to 0, the following constraint is obtained.
\begin{equation}
  p_{test} - \mu_{test,T^p} = \frac{test^* \lambda_{test}}{y_{T^p}^*} = 4 \times 10^{-4}
\end{equation}
\\$\mu_{test,T^+}$ has no constraint and has to be explored.
\subsubsection{Limits}
Since the resources are normalised to 1, the limits would $\in [0,1]$. This needs to be explored.


\section{Syntax to define a mathematical function\label{parser}}
In a mathematical function,
used for example in field definition, it's possible to use the predifined function (an object parser is used to
evaluate the functions) :

ABS\ \ \ \ : absolute value function

COS \ \ \ \ : cosine function

SIN\ \ \ \ : sine function

TAN\ \ \ \ : tangent function

ATAN\ \ : arctangent function

EXP\ \ \ \ : exponential function

LN\ \ \ \ : natural logarithm function

SQRT \ \ : square root function

INT\ \ \ \ : integer function

ERF\ \ \ \ : error function

RND(x)\ \ : random function (values between 0 and x)

COSH\ \ \ \ : hyperbolic cosine function

SINH\ \ \ \ : hyperbolic sine function

TANH\ \ \ \ : hyperbolic tangent function

ACOS\ \ \ \ : inverse cosine function

ASIN\ \ \ \ : inverse sine function

ATANH\ \ : inverse hyperbolic tangent function

NOT(x)\ \ : NOT x (returns 1 if x is false, 0 otherwise)

SGN(x)\ \ : SGN x (returns 1 if x is positive, -1 if negative, 0 if zero)

x\_AND\_y \ \ : boolean logical operation AND (returns 1 if both x and y are true, else 0)

x\_OR\_y\ \ : boolean logical operation OR (returns 1 if x or y is true, else 0)

x\_GT\_y\ \ : greater than (returns 1 if x{\textgreater}y, else 0)

x\_GE\_y\ \ : greater than or equal to (returns 1 if x{\textgreater}=y, else 0)

x\_LT\_y\ \ : less than (returns 1 if x{\textless}y, else 0)

x\_LE\_y\ \ : less than or equal to (returns 1 if x{\textless}=y, else 0)

x\_MIN\_y \ \ \ \ \ : returns the smallest of x and y

x\_MAX\_y \ \ \ \ : returns the largest of x and y

x\_MOD\_y \ \ \ \ : modular division of x per y

x\_EQ\_y \ \ \ \ \ \ \ \ : equal to (returns 1 if x==y, else 0)

x\_NEQ\_y \ \ \ \ \ : not equal to (returns 1 if x!=y, else 0)

\bigskip

You can also use the following operations:

+\ \ : addition

{}- \ \ : subtraction

/ \ \ : division

*\ \ : multiplication

\%\ \ : modulo

\$\ \ : max

\^{} \ \ : power

{\textless}\ \ : less than

{\textgreater}\ \ : greater than

[\ \ : less than or equal to

]\ \ : greater than or equal to

\bigskip

You can also use the following constants:

Pi \ \ : pi value (3,1415{\dots})

\bigskip

The variables which can be used are:

x,y,z \ \ : coordinates

t \ \ : time

\bigskip

{\bfseries
Examples:}

Champ\_front\_fonc\_txyz\index{Champ\_front\_fonc\_txyz} \ 2 \ cos(y+x\^{}2) \ t+ln(y)

Champ\_fonc\_xyz\index{xyz} dom 2 tanh(4*y)*(0.95+0.1*rnd(1)) 0.

\bigskip

{\bfseries
Possible errors:}

Error 1:

Champ\_fonc\_txyz 1 \ cos(10*t)*(1{\textless}x{\textless}2)*(1{\textless}y{\textless}2)

Previous line is wrong. It should be written as:

Champ\_fonc\_txyz 1 \ cos(10*t)*(1{\textless}x)*(x{\textless}2)*(1{\textless}y)*(y{\textless}2)

\bigskip
Error 2:

Champ\_front\_fonc\_xyz 1 \ 20*(x{\textless}-2)+10*(y]-5)+3*(z{\textgreater}0)

Previous line is wrong because negative values are not written between parentheses. It should be written as:

Champ\_front\_fonc\_xyz 1 \ 20*(x{\textless}(-2))+10*(y](-5))+3*(z{\textgreater}0)


\subsection{Random forests}

These use bagging techniques with random trees.

At each node, rather than sample the whole data we sample a random selection.

Get \(d\) dimensions, and sample \(m\) of them at each node.

Choose \(m\le \sqrt d\)


%!TEX root = paper.tex
\subsection{Depth-averaged \nheswe}
The linearized system of equations \eqref{eq:nh_conti}--\eqref{eq:nh_closure} reduces to
\begin{align*}
 0 &= \partial_t\xi+\partial_x(du), \\
 \partial_tu + g\partial_x\xi &= \frac{1}{2\fnh}d^2\partial_{txx} u,
%  \label{eq:nh_1D_Bouss_Disp}
\end{align*}
where the scalar $\fnh$ determines the vertical pressure profile. The resulting dispersion relation is
\begin{equation}
 \omega^2_{\text{nh,}\fnh}=\frac{\csw^2\kappa^2}{1+\frac{(\kappa d)^2}{2\fnh}},
 \label{eq:disp_nh_fnh}
\end{equation}
where $\csw=\sqrt{gd}$ is the linear shallow water gravity wave speed.
The dispersion relation resulting from the quadratic vertical pressure profile ($\fnh=\frac{3}{2}$) is the same as for the Serre equations, so
\begin{equation}
 \omega^2_{\text{nh,quad}}=\frac{\csw\kappa^2}{1+\frac{(\kappa d)^2}{3}}=\csw^2\kappa^2\left(1-\frac{1}{3}(kd)^2+\frac{1}{9}(kd)^4+\mathcal{O}(kd)^6 \right),
 \label{eq:disp_nh_quadr}
\end{equation}
whereas the linear vertical pressure profile ($\fnh=2$) leads to the dispersion relation
\begin{equation}
 \omega^2_{\text{nh,lin}}=\frac{\csw^2\kappa^2}{1+\frac{(\kappa d)^2}{4}}=\csw^2\kappa^2\left(1-\frac{1}{4}(kd)^2+\frac{1}{16}(kd)^4+\mathcal{O}(kd)^6 \right) \label{eq:disp_nh_lin}
\end{equation}
% The same result is given in \cite{Cui.2014} for the linear vertical profile.
For comparison, the dispersion relation from the full linear inviscid equations
is (see, for instance, \cite{Whitham.1974})
\begin{equation}
 \omega^2_{\text{full}}=g\kappa\tanh\left(\kappa d  \right)=\csw^2\kappa^2\left(1-\frac{1}{3}(kd)^2+\frac{2}{15}(kd)^4+\mathcal{O}(kd)^6 \right) .
 \label{eq:disp_full}
\end{equation}
Therefore, the quadratic vertical profile of the \nhp\ is exact in comparison with the dispersion relation from the full linear inviscid equations up to the first order, whereas the linear profile leads to a match only in the zeroth order.


\subsection{Upper Confidence bound 1 applied to Trees (UCT)}

\subsubsection{Introduction}

Way to choose paths

\(\dfrac{w}{n}+c\sqrt {\dfrac{\ln N}{n}}\)

\(w\) is number of wins from node chosen

\(n\) is number of simulations from node chosen

\(N\) is number of simulations that have happened this many layers in


% Optimal truncation
\section{Optimal Truncation}
\label{sec:truncation}

In any decomposition-based approximation the question emerges of how many terms to keep. The optimal value depends both on the true rank of the underlying signal but also the signal to noise ratio: truncating the decomposition of a rank $r$ data matrix at $r$ does not necessarily give the result that minimizes error. Almost always, the suggestion is to use a heuristic method, but it turns out that one can do better. The discussion below uses the context of singular value decomposition, but the problem of optimal truncation is more general and is an example tuning the hyperparameter that determines the capacity of an estimator to mold itself to a dataset.

\subsection{Scree Plots, Heuristic Methods}
\label{sec:scree}
The canonical heuristic approaches focus on \textit{scree plots}, which simply show the singular values in decreasing order, or cumulative plots, which show the fraction of the total sum of singular values that the first $n$ singular values add up to. In case of PCA, the cumulative plot is directly interpretable as the fraction of the explained variance by the first $n$ principal components, where the total variance is given by $\sigma_2 = \sum_i \sigma_i^2$. \\

A scree plot is sometimes referred to as an \textit{elbow plot}, and a common approach is to truncate at the elbow, which is known as \textit{elbow truncation}. Scree refers to the rock fragments that gather at the foot of a steep hill due to erosion, but I wouldn't really know, because I haven't gone outside in a while.

Another common way to go about this would be to, for example, keep only the singular components that account to 90\% of the decomposition.

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{svd_scree.png}
    \caption{Left: Original image of Princess Yvonne und Prince Alexander zu Sayn-Wittgenstein, photographed by their mother, Princess Marianne, in 1955 (1280 by 1277 pixels). Middle: Scree plot, which shows that the first few components are dominant in the image. Right: Cumulative plot that shows that the first singular component accounts for more than 20\% of the decomposition.}
    \label{fig:svd_scree}
\end{figure}

Either method is not limited to SVD or PCA, but also show up whenever, for example, someone wants to decide on the number of clusters in cluster analysis, the number of regressors in mutlivariate regression (only sometimes), or independent component analysis (ICA).

\subsection{Gavish \& Donoho's Optimal Threshold for SVD}

I came across this through \citeasnoun{gavish2013optimal} and Steve Brunton's youtube channel \cite{stevebrunton}. The idea is the following. Assume that some data matrix $\mathbf{X}\in\mathbb{R}^{N \times p}$ consists of a signal $\mathbf{X}_{r}$ with a rank $r$ substructure, and normally distributed noise with mean zero $\mathbf{X}_{\epsilon}\sim \mathscr{N}(0,\sigma_{\epsilon}^2)$. The method assumes that the rank of the signal $r$ is small compared to the rank of $\mathbf{X}$.

\begin{equation}
\mathbf{X} = \mathbf{X}_r + \mathbf{X}_{\epsilon}
\end{equation}

The distribution of singular values of a matrix $\mathbf{X}_{\epsilon}\sim \mathscr{N}(0,\sigma_{\epsilon}^2)$ is known up to the variance. \possessivecite{gavish2013optimal} method is to truncate the decomposition of $\mathbf{X}$ at the singular value that fall below the largest singular value of $\mathbf{X}_{\epsilon}$. It turns out that, with respect to average mean square error, this truncation is asymptotically always better than any other truncation and even always better than the true thresholding at rank $r$ when the rank of the underlying signal happens to be known. When the variance of the noise $\sigma^2_{\epsilon}$ is not known, then \citeasnoun{gavish2013optimal} instead use the median singular value of $\mathbf{X}$.

Let $y_{\mathrm{med}} = \mathrm{med}\{\sigma_i: 1\leq0\leq n\}$ be the median singular value. Then the optimal hard threshold for a matrix $\mathbf{X}\in\mathbb{R}^{m \times n}$ is given by:

\begin{equation}
\tau = \omega(\beta) y_{\mathrm{med}}
\end{equation}

Where $\beta= \frac{m}{n}$ is the aspect ratio of the matrix $\mathbf{X}$ so that $0\leq \beta \leq 1$,  and $\omega(\beta)$ is a constant that needs to be calculated numerically:

\begin{equation}
\omega(\beta) = \frac{\lambda_{*}(\beta)}{\sqrt{\mu_{\beta}}}
\end{equation}

Where $\lambda_{*}(\beta)$:

\begin{equation}
\lambda_{*}(\beta) = \sqrt{2(\beta+1) + \frac{8\beta}{(\beta+1)+\sqrt{\beta^2 + 14\beta+1}}}
\end{equation}

And $\mu_{\beta}$ is unfortunately the median of the Mar\v{c}enko-Pastur distribution, which is the unique solution to:

\begin{equation}
\int_{\beta_{-}^x}\frac{\sqrt{(\beta_{+}-t)(t-\beta_{-})}}{2\pi t} \mathrm{d}t = \frac{1}{2}
\end{equation}

With $\beta_{\pm} = (1\pm\beta)^2$. Cautiously, \citeasnoun{gavish2013optimal} provide the approximation:

\begin{equation}
\omega(\beta) \approx 0.56\beta^3 - 0.95\beta^2 + 1.82\beta + 1.43
\end{equation}

Which has error bounded by $\leq0.02$ on the interval $0.001\leq\beta \leq 1$.

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{truncated_svd.png}
    \caption{Left: Princess and Prince zu Sayn-Wittgenstein with different amounts of Gaussian noise added. Center: The same image estimated using TSVD with Gavish \& Donoho's approximate rank threshold. Right: Singular value spectrum of original, noise and truncated images. TSVD reduces the average mean square error (AMSE) with respect to the original image by over 50\%. Visually, the difference is not too perceptible.}
    \label{fig:truncated_svd}
\end{figure}

\begin{figure}
\centering
    \includegraphics[width=\textwidth]{truncated_svd2.png}
    \caption{Average means square error relative to the original image for the photo with added noise (blue) and after TSVD denoising (organge). The percentage of components retained via Gavish \& Donoho's approximate rank threshold shown in green. $\sigma_{\epsilon}$ is the standard deviation of the added (Gaussian) noise and $\sigma_0$ is the standard deviation of the pixel values of the original image. When $\sigma_{\epsilon}$ is small, G \& D automatically truncates the TSVD at about 40\% of the singular values, which is why, at first, the "denoised" image has greater error than the noisy issues.}
    \label{fig:truncated_svd}
\end{figure}


\section{Extrema of a Function}\label{sec:ExtremaSection}
In calculus, there is much emphasis placed on analyzing the behaviour
of a function $f$ on an interval $I$. Does $f$ have a maximum value on $I$?
Does it have a minimum value? How does the interval $I$ impact our discussion
of extrema?

% % % % % % % % % % % %
% Subsections to include

\input{5-applications-of-derivatives/5-2-1-local-extrema}
\input{5-applications-of-derivatives/5-2-2-absolute-extrema}

% Exercises are at the end of each subsection above


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Conditional probability}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Exploring probabilities with a contingency table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Relapse}

Researchers randomly assigned 72 chronic users of cocaine into three groups: desipramine (antidepressant), lithium (standard treatment for cocaine) and placebo. Results of the study are summarized below.

{\small
\begin{center}
\begin{tabular}{l | c c | c}
			& 		& no 		&  \\
			& relapse	& relapse	& total \\
\hline
desipramine	& 10		& 14		& 24 \\
lithium		& 18		& 6		& 24 \\
placebo		& 20		& 4		& 24 \\
\hline
total			& 48		& 24		& 72
\end{tabular}
\end{center}
}

\ct{\webURL{http://www.oswego.edu/~srp/stats/2_way_tbl_1.htm}}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}

\dq{What is the probability that a patient did not relapse?}

{\small
\begin{center}
\begin{tabular}{l | c c | c}
			& 		& no 		&  \\
			& relapse	& relapse	& total \\
\hline
desipramine	& 10		& 14		& 24 \\
lithium		& 18		& 6		& 24 \\
placebo		& 20		& 4		& 24 \\
\hline
total			& 48		& 24		& 72
\end{tabular}
\end{center}
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Marginal and joint probabilities}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Marginal probability}

\dq{What is the probability that a patient relapsed?}

{\small
\begin{center}
\begin{tabular}{l | c c | c}
			& 		& no 		&  \\
			& relapse	& relapse	& total \\
\hline
desipramine	& 10		& 14		& 24 \\
lithium		& 18		& 6		& 24 \\
placebo		& 20		& 4		& 24 \\
\hline
total			& \only<1>{48}\only<2->{\red{48}}		& 24		&  \only<1>{72}\only<2->{\red{72}}
\end{tabular}
\end{center}
}

\onslide<2->{P(relapsed) = $\frac{48}{72} \approx 0.67$} \\

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Joint probability}

\dq{What is the probability that a patient received the antidepressant (desipramine) \underline{and} relapsed?}

{\small
\begin{center}
\begin{tabular}{l | c c | c}
			& 		& no 		&  \\
			& relapse	& relapse	& total \\
\hline
desipramine	& \only<1>{10} \only<2->{\red{10}}		& 14		& 24 \\
lithium		& 18		& 6		& 24 \\
placebo		& 20		& 4		& 24 \\
\hline
total			& 48	& 24		&  \only<1>{72} \only<2->{\red{72}}
\end{tabular}
\end{center}
}

\onslide<2->{P(relapsed and desipramine) = $\frac{10}{72} \approx 0.14$} \\

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Defining conditional probability}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Conditional probability}

\formula{Conditional probability}{
The conditional probability of the outcome of interest $A$ given condition $B$ is calculated as
\[ P(A|B) = \frac{P(A~and~B)}{P(B)} \]
}

\pause

\twocol{0.5}{0.5}
{
{\small
\begin{center}
\begin{tabular}{l | c c | c}
			& 		& no 		&  \\
			& relapse	& relapse	& total \\
\hline
desipramine	& 10		& 14		& 24 \\
lithium		& 18		& 6		& 24 \\
placebo		& 20		& 4		& 24  \\
\hline
total			& 48		& 24		&  72
\end{tabular}
\end{center}
}
}
{
\begin{eqnarray*}
&&P(relapse |  desipramine) \\
&&= \frac{P(relapse ~ and ~ desipramine)}{P(desipramine)} \\
\pause
&&= \frac{10 / 72}{24 / 72} \\
\pause
&&= \frac{10}{24} \\
\pause
&&= 0.42
\end{eqnarray*}
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Conditional probability (cont.)}

\dq{If we know that a patient received the antidepressant (desipramine), what is the probability that they relapsed?}

{\small
\begin{center}
\begin{tabular}{l | c c | c}
			& 		& no 		&  \\
			& relapse	& relapse	& total \\
\hline
\rowcolor[gray]{.7}
desipramine	& \only<1>{10} \only<2->{\red{10}}			& 14		& \only<1>{24} \only<2->{\red{24}} \\
lithium		& 18		& 6		& 24 \\
placebo		& 20 		& 4		& 24  \\
\hline
total			& 48	& 24		&  72
\end{tabular}
\end{center}
}

\onslide<2->{P(relapse $|$  desipramine) = $\frac{10}{24} \approx 0.42$} \\

\onslide<3->{
$\:$ \\
P(relapse $|$  lithium) = $\frac{18}{24} \approx 0.75$ \\
P(relapse $|$  placebo) = $\frac{20}{24} \approx 0.83$ \\
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Conditional probability (cont.)}

\dq{If we know that a patient relapsed, what is the probability that they received the antidepressant (desipramine)?}

{\small
\begin{center}
\begin{tabular}{l | >{\columncolor[gray]{0.7}[0pt]}c c | c}
			& 		& no 		&  \\
			& relapse	& relapse	& total \\
\hline
desipramine	& \only<1>{10} \only<2->{\red{10}}			& 14		& 24 \\
lithium		& 18		& 6		& 24 \\
placebo		& 20		& 4		& 24  \\
\hline
total			& \only<1>{48} \only<2->{\red{48}}	& 24		&  72
\end{tabular}
\end{center}
}

\onslide<2->{P(desipramine $|$  relapse) = $\frac{10}{48} \approx 0.21$} \\

\onslide<3->{
$\:$ \\
P(lithium $|$  relapse) = $\frac{18}{48} \approx 0.375$ \\
P(placebo $|$  relapse) = $\frac{20}{48} \approx 0.42$ \\
}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{General multiplication rule}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{General multiplication rule}

\begin{itemize}

\item Earlier we saw that if two events are independent, their joint probability is simply the product of their probabilities. If the events are not believed to be independent, the joint probability is calculated slightly differently.

\pause

\item If $A$ and $B$ represent two outcomes or events, then
\formula{\[ P(A~and~B) = P(A|B) \times P(B) \]}
Note that this formula is simply the conditional probability formula, rearranged.
\pause

\item It is useful to think of $A$ as the outcome of interest and $B$ as the condition.

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Independence considerations in conditional probability}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Independence and conditional probabilities}

Consider the following (hypothetical) distribution of gender and major of students in an introductory statistics class:

{\small
\begin{center}
\begin{tabular}{l | c c | c}
			& social	& non-social 		&  \\
			& science	& science	& total \\
\hline
female		& 30		& 20		& 50 \\
male			& 30		& 20		& 50 \\
\hline
total			& 60		& 40		& 100
\end{tabular}
\end{center}
}

\pause

\begin{itemize}

\item The probability that a randomly selected student is a social science major is \pause $\frac{60}{100} = 0.6$.

\pause

\item The probability that a randomly selected student is a social science major given that they are female is \pause $\frac{30}{50} = 0.6$.

\pause

\item Since $P(SS | M)$ also equals 0.6, major of students in this class does not depend on their gender: P(SS $|$ F) = P(SS).

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Independence and conditional probabilities (cont.)}

Generically, if $P(A|B) = P(A)$ then the events $A$ and $B$ are said to be independent.

\pause

\begin{itemize}

\item Conceptually: Giving $B$ doesn't tell us anything about $A$.

\pause

\item Mathematically: We know that if events $A$ and $B$ are independent, $P(A~and~B) = P(A) \times P(B)$. Then,
\[ P(A|B) = \frac{P(A~and~B)}{P(B)} = \frac{P(A) \times P(B)}{P(B)} = P(A) \]

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Tree diagrams}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Breast cancer screening}

\begin{itemize}

\item American Cancer Society estimates that about 1.7\% of women have breast cancer. \\
{\small\webURL{http://www.cancer.org/cancer/cancerbasics/cancer-prevalence}}

\item Susan G. Komen For The Cure Foundation states that mammography correctly identifies about 78\% of women who truly have breast cancer. \\
{\small\webURL{http://ww5.komen.org/BreastCancer/AccuracyofMammograms.html}}

\item An article published in 2003 suggests that up to 10\% of all mammograms result in false positives for patients who do not have cancer. \\{\small \webURL{http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1360940}}

\end{itemize}

\vfill

\Note{These percentages are approximate, and very difficult to estimate.}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Inverting probabilities}

\dq{When a patient goes through breast cancer screening there are two competing claims: patient had cancer and patient doesn't have cancer. If a mammogram yields a positive result, what is the probability that patient actually has cancer?}

\pause

\twocol{0.7}{0.3}{
\includegraphics[width=\textwidth]{3-2_conditional_probability/figures/cancer_tree/cancer_tree_first}
}
{
\pause
{\footnotesize
\begin{eqnarray*}
&&P(C | +) \\
\pause
&&= \frac{P(C~and~+)}{P(+)} \\
\pause
&&= \frac{0.0133}{0.0133 + 0.0983} \\
\pause
&&= 0.12
\end{eqnarray*}
}
}

\pause

\Note{Tree diagrams are useful for inverting probabilities: we are given $P(+|C)$ and asked for $P(C|+)$.}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Practice}

\pq{Suppose a woman who gets tested once and obtains a positive result wants to get tested again. In the second test, what should we assume to be the probability of this specific woman having cancer?}

\begin{enumerate}[(a)]
\item 0.017
\solnMult{0.12}
\item 0.0133
\item 0.88
\end{enumerate}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Practice}

\pq{What is the probability that this woman has cancer if this second mammogram also yielded a positive result?}

\twocol{0.2}{0.7}
{
\begin{enumerate}[(a)]
\item 0.0936
\item 0.088
\item 0.48
\solnMult{0.52}
\end{enumerate}
}
{
\solnGr{\only<2->{
\includegraphics[width=\textwidth]{3-2_conditional_probability/figures/cancer_tree/cancer_tree_second}
}}
}

\soln{\only<3->{
{\small \[P(C | +) = \frac{P(C~and~+)}{P(+)} = \frac{0.0936}{0.0936+0.088} = 0.52\]}
}}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Bayes' Theorem}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Bayes' Theorem}

\begin{itemize}

\item The conditional probability formula we have seen so far is a special case of the Bayes' Theorem, which is applicable even when events have more than just two outcomes.

\pause

\item \hl{Bayes' Theorem:}
\formula{
\[ P(outcome~A_1~of~variable~1~|~outcome~B~of~variable~2) \]
\[ = \frac{P(B|A_1)P(A_1)}{P(B|A_1)P(A_1) + P(B|A_2)P(A_2) + \cdots + P(B|A_k)P(A_k)} \]
}
where $A_2$, $\cdots$, $A_k$ represent all other possible outcomes of variable 1.

\end{itemize}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Application activity: Inverting probabilities}

\app{{\footnotesize A common epidemiological model for the spread of diseases is the SIR model, where the population is partitioned into three groups: Susceptible, Infected, and Recovered. This is a reasonable model for diseases like chickenpox where a single infection usually provides immunity to subsequent infections. Sometimes these diseases can also be difficult to detect. \vspace{2mm} \\
Imagine a population in the midst of an epidemic where 60\% of the population is considered susceptible, 10\% is infected, and 30\% is recovered. The only test for the disease is accurate 95\% of the time for susceptible individuals, 99\% for infected individuals, but 65\% for recovered individuals. (Note: In this case accurate means returning a negative result for susceptible and recovered individuals and a positive result for infected individuals). \vspace{2mm} \\
Draw a probability tree to reflect the information given above. If the individual has tested positive, what is the probability that they are actually infected?
}}

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{frame}
\frametitle{Application activity: Inverting probabilities (cont.)}

\vspace{-0.5cm}

\includegraphics[width=\textwidth]{3-2_conditional_probability/figures/sir_tree/sir_tree}

\pause

\[ P(inf | +) = \frac{P(inf~and~+)}{P(+)} = \frac{0.099}{0.03 + 0.099 + 0.105} \approx 0.423 \]

\end{frame}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%\end{frame}


\section{Experiments}
In this section, we briefly present our experiments on some of the
algorithms presented at the previous sections. Two sets of experiments are presented.

The first tests are on powers
of random gaussian matrices of different sizes. The goal of this set of numerics
is to check the performance of the algorithms and make sure they are working
as expected. We also analyze the sharpness of the bounds dictated by the theory.
In particular, the bound on the expectation of the error given
by \ref{thm:avg-frob-error-gauss} and
the error estimation procedure which was motivated by
Lemma \ref{thm:aposteriori}.
We have used an \textit{oversampling parameter} $p=5$ for all of the experiments.

The second set of experiments analyses the performance of the
\textit{Randomized Power Method} \ref{alg:randomized-power-iteration} on MNIST
and on a matrix appearing in image processing.

\subsection{Details of the implementation}
The experiments have been implemented in \verb|Matlab|. The reproducible
source code can be found at the following
\href{https://github.com/alexnowakvila/ProbAlgosProj}{Github repository}
together with the \LaTeX~  of the report. Please read \verb|README| documentation
to run the code on the \verb|Matlab| interactive command line.
\subsection{Gaussian Matrices} \label{sec:gaussian-matrices}
In this set of experiments, we test and analyze the performance of the
Algorithms \ref{alg:randomized-range-finder},
\ref{alg:adaptive-randomized-range-finder},
\ref{alg:randomized-power-iteration}
and \ref{alg:fast-randomized-range-finder}.

The experiments are performed on powers of gaussian random
matrices of the form:

\begin{equation}\label{eq:gaussian-matrices}
\mtx{A} = \frac{1}{\sqrt{m}+\sqrt{n}}\mtx{G}, \hspace{0.5cm}
 G_{ij}\sim N(0,1)
\end{equation}
The normalization in \ref{eq:gaussian-matrices} is to make sure that the norm
of $\mtx{A}$ is around 1 with high probability
\footnote{This is a direct consequence of Sudakov-Fernique's inequality that
compares the supremum of two gaussian processes when one is dominated
by the other. More precisely, you have the bound on the expectation
of the norm $\Expect\|\mtx{G}\|\leq\sqrt{m}+\sqrt{n}$
and also an accompanying tail bound
$\Prob{\|\mtx{G}\| \geq \sqrt{m} + \sqrt{n} + t}\leq 2\exp(-ct^2)$.
Using Gordon's inequality (generalization of Sudakov Fernique's), you can
also prove lower bounds on the smallest singular value
$\Expect\|\mtx{G}^\dagger\|\geq \sqrt{m} - \sqrt{n}$ and
$\Prob{\|\mtx{G}\|\leq \sqrt{m}-\sqrt{n}-t}\leq 2\exp(-ct^2)$.
These results on concentration of measure have been studied at the Theory
Reading Group following the book on High Dimensional Probability
from R.Vershynin which I highly recommend
\cite{vershynin2016high}.}.
As seen from Equation \ref{eq:sing-values-power}, the singular values decay faster
for higher powers of the matrix.
\subsubsection{Experiments on Randomized Range Finder
\ref{alg:randomized-range-finder}} See Figure \ref{fig:exp1-1}.
\begin{figure}[H]\label{fig:exp1-1}
\begin{center}
\includegraphics[width=\textwidth, trim=0cm 8cm 0cm 9cm, clip=true]{figures/1-4.pdf}
\end{center}
\caption{Comparison between theoretical mean bound \ref{thm:avg-frob-error-gauss}
and numerical error \ref{eq:range-error} produced by Algorithm \ref{alg:randomized-range-finder}
 for $(\mtx{A}\mtx{A}^\adj)^r\mtx{A}$ with $r=0,1,2$. }
\end{figure}

\subsubsection{Experiments on Randomized Power Iteration
\ref{alg:randomized-power-iteration}} See Figure \ref{fig:exp1-2}.
\begin{figure}[H] \label{fig:exp1-2}
\begin{center}
\includegraphics[width=\textwidth, trim=0cm 8cm 0cm 9cm, clip=true]{figures/1-5.pdf}
\end{center}
\caption{Comparison between theoretical mean bound from Corollary
\ref{cor:power-method-spec-gauss} and numerical
error \ref{eq:range-error} produced by Algorithm \ref{alg:adaptive-randomized-range-finder}
for $\mtx{A}$ and $q=1,2,3$}
\end{figure}

\subsubsection{Experiments on Adaptive Randomized Range Finder
\ref{alg:adaptive-randomized-range-finder}}
In this experiment, Algorithm \ref{alg:adaptive-randomized-range-finder} is applied
to $(\mtx{A}\mtx{A}^\adj)^2\mtx{A}$ with pre-specified tolerances ranging
from 0 to 1. We have fitted a line to the numerical
error produced vs the tolerance (the dependence was close to linear). The slope of the
resulting line
is $\textbf{0.045}$, i.e, for the matrices used in the experiment, the
bound \ref{eq:errorest} has a numerical suboptimality of 0.045.
\subsubsection{Experiments on Fast Randomized Range Finder
\ref{alg:fast-randomized-range-finder}}
Algorithm \ref{alg:fast-randomized-range-finder} has been applied to several
powers of $\mtx{A}$ (using SRFT test matrices) and the error curve for
different target ranks is essentially the same as the one found using
Algorithm \ref{alg:randomized-range-finder}.

\subsection{Real Datasets}
\subsubsection{MNIST}
In this experiment, we used 10,000 examples from
the MNIST dataset \cite{lecun1998mnist} of 28x28
black and white images of handwritten digits.

The goal is to compare Algorithm \ref{alg:randomized-range-finder} with the
power version \ref{alg:randomized-power-iteration} on the matrix
$\mtx{M}\in\Rspace{10000\times 784}$ where each row is a flattened image.
The singular values of $\mtx{M}$ decay fast, hence, the power scheme does not
significantly improve the randomized scheme for $q>1$.
See Figure \ref{fig:exp2}.

\subsubsection{A Large, Sparse, Noisy Matrix Arising in Image Processing}
One way to tackle standard tasks in image processing is to use the information
about the local geometry of the image through the \textit{graph Laplacian}.
The dominant eigenvectors of this operator provide coordinates that help to smooth
out noisy image patches \cite{szlam2008regularization}.

First, define $\widetilde{\mtx{W}}$ as
\begin{equation}\label{eq:similarities}
\widetilde{w}_{ij}=\exp\{-\|\mtx{x}^{(i)} - \mtx{x}^{(j)}\|^2/\sigma^2\}
\end{equation}
where $\mtx{x}^{(i)}$ is a vector gathering the intensities of a neighborhood
of a certain size of the pixel $i$. Here, $\sigma$ controls the level of
sensitivity. We then define the sparse matrix $\mtx{W}$ constructed by
zeroing out all the entries in $\widetilde{\mtx{W}}$ except the 7 largest
ones in each row.
Finally, the laplacian operator is defined as
\begin{equation}\label{eq:graph-laplacian}
\mtx{L}=\mtx{I}-\mtx{D}^{-1/2}\mtx{W}\mtx{D}^{-1/2}
\end{equation}
where $\mtx{D}$ is the diagonal matrix with entries $d_{ii}=\sum_{j}w_{ij}$.

In order to find the low-frequency eigenvectors of $\mtx{L}$,
we must find the eigenvectors with dominant eigenvalues of

\begin{equation}\label{eq:adj}
\mtx{A}=\mtx{D}^{-1/2}\mtx{W}\mtx{D}^{-1/2}
\end{equation}

In our experiment, we have taken a patch of size 57x57 from the classical
$\verb|Lenna.png|$ photo. The matrix $\widetilde{\mtx{W}}$ is constructed
with patches of size 5x5 centered at the corresponding pixel with
zero-padding at the boundary. The resulting
matrix $\mtx{A}$ has size 3249x3249 and has low decaying eigenvalues. Hence,
as can be seen in Figure \ref{fig:exp2}, the power scheme from
Algorithm \ref{alg:randomized-power-iteration} is very advantageous.

\begin{figure}[H] \label{fig:exp2}
\begin{center}
\includegraphics[width=\textwidth, trim=0cm 8cm 0cm 7cm, clip=true]{figures/2-2.pdf}
\end{center}
\caption{Approximation error \ref{eq:range-error} for the
power scheme \ref{alg:randomized-power-iteration} for MNIST
(Left) and $\mtx{A}$ \ref{eq:adj} (Right) for $k$ ranging from $1$ to $100$.}
\end{figure}


\subsection{The Link test}

\subsubsection{Introduction}

Alternative to RESET

We have \(\hat y\).

We regress \(y=\alpha + \beta \hat y + \gamma \hat y^2\).

We test that \(\gamma =0\).

If it is not, then this suggests the model is misspecified.