\section{Homology of CW-complexes}
\begin{lemma}
There are isomorphisms:
\begin{equation*}
 H_q(X_n,X_{n-1})\xrightarrow{\cong} H_q(X_n/X_{n-1},\ast)= H_q\left(\bigvee_{\alpha\in A_n}S^n_\alpha,\ast\right)=\begin{cases}0 & q\neq n \\ \Z[A_n] & q=n\end{cases}
\end{equation*}
\end{lemma}
Let's talk about ``characteristic maps''. This is a map $\left(\coprod D^n_\alpha,\coprod S^{n-1}_\alpha\right)\to (X_n,X_{n-1})$. This is like a ``relative homeomorphism'' (I was drinking water, so this isn't exactly accurate). We have a map $ H_q(X_n,X_{n-1})\to H_q(X_n/X_{n-1},\ast)$, to get a commutative diagram:
\begin{equation*}
\xymatrix{ H_q\left(\coprod D^n_\alpha,\coprod S^{n-1}_\alpha\right)\ar[r]\ar[d] & H_q\left(\bigvee S^n_\alpha,\ast\right)\ar[d]\\
 H_q(X_n,X_{n-1})\ar[r] & H_q(X_n/X_{n-1},\ast)}
\end{equation*}
The right arrow is an isomorphism. The top arrow is an isomorphism. The lemma says that the bottom map is an isomorphism, so that $ H_q\left(\coprod D^n_\alpha,\coprod S^{n-1}_\alpha\right)\to H_q(X_n,X_{n-1})$ is an isomorphism. This is called the ``cellular $n$-chains'' on $X$.

Now, fix $q$. For $q=0$, there is:
\begin{equation*}
\xymatrix{ H_1(X_1,X_0)\ar[d] & H_0(X_1,X_0)=0 & H_0(X_2,X_1)=0\\
 H_0(X_0)\ar[r]\ar[drrr] & H_0(X_1)\ar[r]\ar[drr]\ar[u] & H_0(X_2)\ar[u]\ar[r]\ar[dr] & \cdots\ar[d]\\
 & H_1(X_2,X_1)=0\ar[u] & & H_0(X)}
\end{equation*}
We know that $ H_0(X_1,X_0)=0$, but $ H_1(X_1,X_0)$ is not necessarily $0$. This means that $ H_0(X_0)\to H_0(X_1)$ is surjective, and $ H_0(X_1)\cong H_0(X_2)$, and so on for higher dimensions. This makes sense because adding higher dimensional cells does not change path components.

Let's try this for $q>0$. Then you have:
\begin{equation*}
\xymatrix{ & H_{q+1}(X_q,X_{q+1})=0\ar[d]& H_{q+1}(X_{q+1},X_q)\ar[d]\\
\ar[r]\cdots & H_q(X_{q-1})\ar[r]\ar[dr] & H_q(X_q)\ar[r]\ar[d] & H_q(X_{q+1})\ar[r]\ar[d] & H_q(X_{q+2})\ar[d]\ar[r] & \cdots\\
& & H_q(X_{q},X_{q-1}) & H_q(X_{q+1},X_q)=0 & H_q(X)}
\end{equation*}
So the first maps ($ H_q(X_0)\to H_q(X_1)\to\cdots$) are isomorphisms, the map $ H_q(X_{q+1})\to H_q(X_q)$ is an injection, and the map $ H_q(X_q)\to H_q(X_{q+1})$ is surjective. But also, $ H_q(X_{q+1})\cong H_q(X_{q+2})\cong \cdots$. But also, $ H_q(X_0)\cong 0$, and we have:
\begin{corollary}
$ H_q(X)=0$ for $q>\dim X=n$.
\end{corollary}
\begin{lemma}
$ H_q(X_n)\cong H_q(X)$ for $n>0$.
\end{lemma}
I want you to have the following picture in mind. We have a diagram coming from the lexseq in the homology of a pair:
\begin{equation*}
\xymatrix{C_{n+1}(X)= H_{n+1}(X_{n+1},X_n)\ar[d]^\partial\ar[dr]^d & & 0= H_{n-1}(X_{n-2})\ar[d]\\
 H_n(X_n)\ar[r]^j\ar[d] & C_n(X)= H_n(X_n,X_{n-1})\ar[r]^\partial\ar[dr]^d & H_{n-1}(X_{n-1})\ar[d]^j\\
 H_n(X_{n+1})\ar[d] & & C_{n-1}(X)= H_{n-1}(X_{n-1},X_{n-2})\\
0 = H_n(X_{n+1},X_n)}
\end{equation*}
Now, $\partial\circ j=0$. So the composite of the diagonals is zero, i.e., $d^2=0$, and we have a chain complex! More precisely, we get a chain complex, denoted $C_\ast(X)$. This is the ``cellular chain complex'' of $X$. We should compute the homology of this chain complex. Well, $ H_n(C_\ast(X))=\ker d/\img d$. Now, $\ker d=\ker (j\circ\partial)$. But $j$ is injective, so $\ker d=\ker\partial$. Also, $\img d=\img(j\circ\partial)=j(\img\partial)$ because $j$ is injective.

The kernel of $\partial$ is the image of $j$ by exactness, but $j$ is a monomorphism, so $\ker\partial\cong H_n(X)$. Now, $ H_n(C_\ast(X))\cong\frac{ H_n(X)}{\img(\partial)}$. This is equal to $ H_n(X_{n+1})$, again by exactness. But out lemma shows that $ H_n(X_{n+1})= H_n(X)$. In other words, we've proved:
\begin{theorem}
If $X$ is a CW-complex, then $ H_\ast(C_\ast(X))\cong H_\ast(X)$. I didn't use specific attaching maps at all, so this is natural in ``skeletal'' maps of CW-complexes.
\end{theorem}
What is the differential? You have a relative cycle in dimension $(n+1)$, you're taking its boundary, and then working relative the $(n-1)$-skeleton. You'll see this better in the example we're going to do now, namely projective space.
\begin{example}
We'll try $ H_\ast(\mathbf{RP}^n)$. We have: $\mathrm{sk}_k(\RP^n)=\RP^k$, which are just $1$-dimensional subspaces of $\mathbf{R}^{k+1}$. Think of the inclusion $\mathbf{R}^{k+1}\to\mathbf{R}^{n+1}$ as the inclusion of the first $(k+1)$ basis vectors. This is a CW-complex because the map $S^{k-1}\to \mathbf{RP}^{k-1}$ is a double cover, and you have a pushout:
\begin{equation*}
\xymatrix{S^{k-1}\ar[r]\ar@{^(->}[d] & \mathbf{RP}^{k-1}\ar@{^(->}[d]\\
D^k\ar[r] & \mathbf{RP}^k}
\end{equation*}
The attaching maps are the double cover maps.
\end{example}
The notation is as follows. $\mathbf{RP}^n=\mathbf{RP}^{n-1}\cup_f D^n=\mathbf{RP}^{n-1}\cup_f e^n$. The $e^n$ is the notation for an $n$-cell. In particular, $\mathbf{RP}^n=e_0\cup_f e_1\cup_f\cdots\cup_f e_n$. You have:
\begin{equation*}
\xymatrix{0 & C_0(\mathbf{RP}^n)\ar[d]\ar[l] & C_1(\mathbf{RP}^n)\ar[d]\ar[l] & \cdots\ar[l]\ar[d] & C_n(\mathbf{RP}^n)\ar[d]\ar[l] & 0\\
& \Z\langle e^0\rangle & \Z\langle e^1\rangle\ar[l]^{d=0} & \cdots\ar[l] & \Z\langle e^n\rangle\ar[l]}
\end{equation*}
The first differential is zero because we know what $ H_0(\mathbf{RP}^n)$ is (it's $\Z$!). I have $S^{n-1}\xrightarrow{f}\mathbf{RP}^{n-1}\to \mathbf{RP}^{n-1}/\mathbf{RP}^{n-2}=S^{n-1}$. Also recall the commutative diagram from before.
\begin{equation*}
\xymatrix{ H_n(D^n,S^{n-1})\ar[r]^\partial \ar[d]^\cong & H_{n-1}(S^{n-1})\ar[r]\ar[d]^\cong & H_{n-1}(S^{n-1},\ast)\ar[d]^\cong\\
C_n= H_n(\mathbf{RP}^n,\mathbf{RP}^{n-1})\ar[r]^\partial & H_{n-1}(\mathbf{RP}^{n-1}) \ar[r] & H_{n-1}(\mathbf{RP}^{n-1},\mathbf{RP}^{n-2})=C_{n-1}
}
\end{equation*}
The first map on the top is an isomorphism. The bottom composite is our differential. So the map $ H_{n-1}(S^{n-1})\to H_{n-1}(S^{n-1},\ast)$. Therefore, $S^{n-1}\xrightarrow{\text{double cover}}\mathbf{RP}^{n-1}\xrightarrow{\text{pinching}} S^{n-1}$.
\begin{equation*}
\xymatrix{S^{n-1}\ar[r]^{\text{double cover}}\ar[dr] & \mathbf{RP}^{n-1}\ar[r]^{\text{pinching}} & S^{n-1}\\
 & S^{n-1}/S^{n-2}=S^{n-1}\vee S^{n-1}\ar[ur]}
\end{equation*}
One of the maps $S^{n-1}\to S^{n-1}$ from the wedge is the identity, and the other map is the antipodal map, as can be seen by looking at a picture. If $\alpha$ is the antipodal map, then $S^{n-1}\vee S^{n-1}\to S^{n-1}$ is $[1,\alpha]$. If $\sigma$ is a generator of $ H_{n-1}(S^{n-1})$, we have $\sigma\mapsto (\sigma,\sigma)\mapsto \sigma+\alpha_\ast\sigma$. What is the degree of $\alpha_\ast: H_{n-1}(S^{n-1})\to H_{n-1}(S^{n-1})$, so $\deg\alpha=(-1)^n$. Thus the composite, and hence the attaching map, is $(1+(-1)^n)\sigma$. This means the cellular chain complex is:
\begin{equation*}
\xymatrix{0 & \Z\ar[l]^0 & \Z\ar[l]^2 & \cdots\ar[l]^0 & \Z\ar[l]^{2\text{ or }0} & 0\ar[l] & 0\ar[l] & \cdots\ar[l]}
\end{equation*}
We'll continue next time\footnote{Why don't we work in $\Z/2\Z$ coefficients? This is so much easier then. :P}.


\subsection{Momentum}

We define the momentum as:

\(p_j = \dfrac{\delta L}{\delta \dot q_j}\)

\(p_j = \dfrac{\delta }{\delta \dot q_j}\sqrt {(\mathbf {\dot q})^T\mathbf M\mathbf {\dot q}}\)


\section*{Exercises}

\begin{ex}
  Find the least squares approximation for the system of equations
  \begin{equation*}
    \begin{array}{r@{~}c@{~}r@{~}c@{~}r@{~}c@{~}r}
       x &+& 2y &+& 2z &=&   5, \\
       x &+&  y &-&  z &=&  11, \\
       x &+& 2y &-&  z &=& -18, \\
      2x &-&  y &+& 2z &=&  0. \\
    \end{array}
  \end{equation*}
  \begin{sol}
    $(x,y,z) = (-1,-1,2)$.
  \end{sol}
\end{ex}

\begin{ex}
  Find the least squares approximation for the system of equations
  \begin{equation*}
    \begin{mymatrix}{rrr}
      -1 & 2 & 1 \\
      -1 & 0 & -1 \\
      2 & 0 & 2 \\
      0 & 0 & 2 \\
      1 & 2 & 2 \\
    \end{mymatrix}
    \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
    =
    \begin{mymatrix}{r} 1 \\ 1 \\ 3 \\ -2 \\ 4 \end{mymatrix}.
  \end{equation*}
  \begin{sol}
    $(x,y,z) = (2,2,-1)$.
  \end{sol}
\end{ex}

\begin{ex}
  Consider the points $(x_1,y_1) = (-1,0)$, $(x_2,y_2) = (0,3)$,
  $(x_3,y_3) = (1,3)$, $(x_4,y_4) = (2,5)$, $(x_5,y_5) = (3,9)$.  Find
  the least squares line for these points.
  \begin{sol}
    $y = 2+2x$.
  \end{sol}
\end{ex}

\begin{ex}
  Consider the points $(x_1,y_1) = (-1,4)$, $(x_2,y_2) = (0,-2)$,
  $(x_3,y_3) = (1,4)$, $(x_4,y_4) = (2,2)$.  Find the least squares
  parabola for these points.
  \begin{sol}
    $y = 1-x+x^2$.
  \end{sol}
\end{ex}


\chapter{Complexity Classes}

A complexity class is a set of related problems.  Before we talk about
specific classes, it is important to understand the difference between
a decision problem and an optimization problem.

\section{Optimization Problems}

An optimization problem is a problem of the form ``find the optimal
solution for problem p.''  Each such problem may have a different
defintion of optimal, and there may be many optimal solutions.

\section{Decision Problems}

A decision problem is much more limited in scope.  These are problems
with strictly yes/no answers, often of the form ``is there a solution of
size (at least or at most) k to problem p?''.

Notice that optimization problems have related decision problems.  For
any optimization problem we can ask the decision problem ``is x the
optimal solution to problem p?''.

\section{P}

The complexity class $P$ is the set of all decision problems which are
solvable in polynomial time.  That is to say, all problems for which
there exists an algorithm to solve the problem bounded above by
$O(n^d)$ where $d \in \mathbb{Z}$.

\section{NP}

The complexity class $NP$ is the set of all decision problems for
which a certificate (possible solution) may be verified in polynomial
time.

\section{$P \subseteq NP$}

It should be obvious that any problem in $P$ is also in $NP$, since if
we can find the answer in polynomial time without the certificate, we
must certainly be able to do so with the certificate.

\section{P = NP?}

It is not obvious if there are problems in $NP$ but not in $P$ or if
all problems in $NP$ are also in $P$.  In fact,
\href{http://www.claymath.org/}{the Clay Mathematics Institute} has
\href{http://www.claymath.org/millennium/P_vs_NP/}{listed this as one
  of the Millenium Prizes} for which the award for solving is 1
million USD.


\section{Inverting \partialsum and \adjacentdifference}
\Label{sec:partialsuminv}
\Label{sec:adjacentdifferenceinv}

In this section we show that under appropriate preconditions
the algorithms \partialsum and\\
\adjacentdifference are inverse to each other.

\subsection{Inverting \partialsum}

Let \inl{a[0..n-1]} and \inl{b[0..n-1]} be the respective input and output
of \partialsum.
We have in other words
\begin{align*}
   \mathtt{b}[0] &= \mathtt{a}[0] \\
   \mathtt{b}[1] &= \mathtt{a}[0] + \mathtt{a}[1] \\
                 &\vdotswithin{=} \\
   \mathtt{b}[n-1]  &= \mathtt{a}[0] + \mathtt{a}[1] + \ldots + \mathtt{a}[n-1] \\
\end{align*}

If we apply now the algorithm \adjacentdifference to \inl{b[0..n-1]}, then
we find for its output \inl{a'[0..n-1]}
\begin{alignat*}{3}
   \mathtt{a'}[0] &= \mathtt{b}[0]                        &\quad &=\quad \mathtt{a}[0] \\
   \mathtt{a'}[1] &= \mathtt{b}[1] - \mathtt{b}[0]        &\quad &=\quad \mathtt{a}[1] \\
                 &\vdotswithin{=}  \\
   \mathtt{a'}[n-1] &= \mathtt{b}[n-1] - \mathtt{b}[n-2]  &\quad &=\quad \mathtt{a}[n-1]
\end{alignat*}

Before we start show the \acsl lemmas of our claim, we present
the predicate \logicref{DefaultBounds} in order to express that the values
in the input (and output!) array~\inl{a[0..n-1]} do not overflow.

\input{Listings/DefaultBounds.acsl.tex}

Lemma \PartialSumInverse from the following listing
expresses as \acsl lemmas
that the algorithms \partialsum and \adjacentdifference are inverse to each other.

\input{Listings/NumericInverse.acsl.tex}

%\clearpage

The following listing now shows \isoc function \partialsuminv
(both the contract and the implementation).
This function calls first \partialsum and then \adjacentdifference.

\input{Listings/partial_sum_inv.c.tex}

The contract of \partialsuminv formulates preconditions that shall guarantee
that during the computation neither arithmetic overflows (property~\inl{bounds})
nor unintended aliasing of arrays (property~\inl{sep}) occur.
Under these precondition, \framac shall verify
that the final call to \specref{adjacentdifference} just restores the original contents
of~\inl{a[0..n-1]} that we supplied for the initial call to \specref{partialsum}.

\clearpage

\subsection{Inverting \adjacentdifference}

After executing the function \specref{adjacentdifference} on
the input array \inl{a[0..n-1]} the output array \inl{b[0..n-1]}
holds the following values
\begin{align*}
   \mathtt{b}[0] &= \mathtt{a}[0] \\
   \mathtt{b}[1] &= \mathtt{a}[1] - \mathtt{a}[0] \\
                 &\vdotswithin{=} \\
   \mathtt{b}[n-1] &= \mathtt{a}[n-1] - \mathtt{a}[n-2] \\
\end{align*}

If we call now \partialsum with the array \inl{b[0..n-1]}
as input, then we obtain for its output \inl{a'[0..n-1]}
\begin{alignat*}{3}
   \mathtt{a'}[0] &= \mathtt{b}[0]
                  &\quad &=\quad \mathtt{a}[0] \\
   \mathtt{a'}[1] &= \mathtt{b}[0] + \mathtt{b}[1]
                  &\quad &=\quad \mathtt{a}[1] \\
                  &\vdotswithin{=} \\
   \mathtt{a'}[n-1]  &=\ \mathtt{b}[0] + \mathtt{b}[1] + \ldots + \mathtt{b}[n-1]
                     &\quad &=\quad \mathtt{a}[n-1]
\end{alignat*}
%
which means that applying \specref{partialsum} on the output of
\adjacentdifference produces the original input.
Lemma \logicref{AdjacentDifferenceInverse} expresses this property as a lemma.

The function \implref{adjacentdifferenceinv} first calls \adjacentdifference and then\\
\partialsum.
The contract of this function formulates preconditions that shall guarantee
that during the computation neither arithmetic overflows (property~\inl{bound})
nor unintended aliasing of arrays (property~\inl{sep}) occur.
In order to improve the automatic verification of
\adjacentdifferenceinv we also use lemma \logicref{UnchangedTransitive}.
Lemma \logicref{AdjacentDifferenceInverseBounds} simplifies
the verification of the precondition \inl{bounds} of \partialsum.

\input{Listings/adjacent_difference_inv.c.tex}


\lesson{3}{Nov 29 2021 Mon (21:06:20)}{Polynomial Transformations}{Unit 3}

\begin{definition}[Polynomial Function]
    A \bf{Polynomial Function} is a function composed of $1$ or more terms, at least $1$ of which contains a variable. A polynomial function whose degree is $3$ is called a \bf{Cubic Function}. A \bf{Polynomial Function} whose degree is $4$ is called a \bf{Quartic Function}.

    Simple polynomial functions may be graphed by hand by substituting x-coordinates and solving for the corresponding y-coordinates.

    The real number solutions of a polynomial equation can be found by identifying the x-intercepts from the graph of the function. Graph the function, and find the point(s) of intersection between the graph and the x-axis.
\end{definition}

\subsubsection*{Odd vs Even vs Neither Polynomial Functions}

\begin{center}
	\begin{table}[h]
		\begin{tabular}{ |r|c|l| }
			\hline
			Result & Function Type & Symmetry \\
			\hline
			\multirow{3}{4em}{} $f(-x) = f(x)$	                & Even function	       &  y-axis    \\
                                $f(-x) = -f(x)$                 & Odd function         &  Origin    \\
                                $f(-x) \neq f(x)$ or $f(-x)$    & Neither even and odd &            \\
								\hline
		\end{tabular}

		\caption{Odd vs Even vs Neither Polynomial Functions}
		\label{table:odd-vs-even-vs-neither-polynomial-functions}
	\end{table}
\end{center}

\newpage


\documentclass[11pt,twoside]{article}

\usepackage[headings]{fullpage}
\usepackage[utopia]{mathdesign}

\pagestyle{myheadings}
\markboth{Eigenworms}{Eigenworms}

\input{../../fncextra}

\begin{document}

\begin{center}
  \bf Early bird gets the eigenworm
\end{center}

The idea and data for this lab come from Stephens \textit{et al.}, ``Dimensionality and Dynamics in the Behavior of \textit{C.\ elegans},'' \textit{PLoS Comput Biol}, 2008. In this work the authors captured video of worms moving as they were subjected to stimuli (from ``standard'' to ``painful''). Using image processing, they found 100 points representing a path along the back of single worm within each frame of the video and computed a representation independent of rotation and translation using the tangents to the path. The result is a $100\times n$ matrix of tangent angles for $n$ frames of video. They used $n=56200$.

The authors then noted that dimension reduction by the SVD is extraordinarily successful for this data set; they showed that the data are well characterized by a small number of ``eigenworms''.\footnote{One often sees the ``eigen-'' prefix used in this context, but the SVD is a more fundamental description of the mathematics.} This makes some sense, as the motions are constrained a great deal by anatomy and kinematics.

Suppose $\m{T}$ is the matrix of angles; i.e., column $\bft_j$ is the vector of angles in the $j$th video frame. Suppose we have the full SVD $\m{T}=\m{U} \mS \mV^T$. This would make $\mV$ an $n\times n$ matrix, which would not fit in memory, so we have to use a thin SVD. In the text we only did this for an $m\times n$ matrix with $m>n$, which is not the case for $\m{T}$. However, $\m{T}^T=\m{V} \mS^T \mU$ does have a thin form in which $\m{T}^T=\widehat{\mV} \widehat{\mS}^T \mU$, where $\widehat{\mV}$ is only $n\times 100$ and $\widehat{\mS}$ is $100\times 100$. Finally, this gives us
\begin{equation}
  \label{eq:reducesvd}
  \m{T}=\mU \widehat{\mS} \widehat{\mV}^T,
\end{equation}
the thin SVD we need.

Observe that
\begin{equation}
  \m{T} = \sum_{k=1}^{100} \sigma_k \bfu_k \bfv_k^T.
\end{equation}
Because the singular values are always in decreasing order, we may approximate the original matrix by
\begin{equation}
  \m{T}_r = \sum_{k=1}^{r} \sigma_k \bfu_k \bfv_k^T,
\end{equation}
for some rank $r\ll 100$.  The range of this matrix is spanned by $\bfu_1,\ldots,\bfu_r$, which are the eigenworms. A good way to express the proportion of $\m{T}$ that is captured by $\m{T}_r$ is the ratio
\begin{equation}
  \tau_r = \frac{\|\mathbf{s}_r\|^2_2}{\|\mathbf{s}_{100}\|^2_2},
\end{equation}
where $\mathbf{s}_k$ is the vector $\bigl[ \sigma_1,\ldots,\sigma_k \bigr]$.

One use of the eigenworms is to create a compact representation of the data. A column $\bft_j$ of the original data can be expressed in terms of its closest approximation as a linear combination of the eigenworms:
\[
  \bft_j \approx c_1 \bfu_1  + \cdots + c_r \bfu_r = \mU_r \bfc,
\]
where $\mU_r$ is $100\times r$. This is a least squares problem, but by orthogonality its solution is $\bfc=\mU_r^T\bft_j$. The $r$ values in this vector give the components of $\bft_j$ in the principal eigenworm directions and could be used as a low-dimensional representation for further analysis.

\subsection*{Goals}

Given the data matrix, you will perform the SVD analysis, find a reasonable value for the cutoff rank $r$ using the coefficients $\tau_k$, and extract the eigenworms.

\subsection*{Preparation}

Read section 7.5. Read the online help for \texttt{svd}; it uses the term ``economy size'' for the thin SVD.

\subsection*{Procedure}

\begin{enumerate}
\item Load the \texttt{shapes.mat} file from the assignment site. It has the matrix \texttt{T}.

\item On one graph, plot the first three columns of $\m{T}$. (These are tangent angles of the worm's body as a function of arc length.)

\item Using \texttt{svd}, compute the three matrices in the thin SVD~\eqref{eq:reducesvd}.

\item Let \texttt{s} be the vector of singular values. Using it, plot $1-\tau_r$ versus $r$ on a semi-log scale, for $r=1,\dots,100$. From this plot it should be clear that $r=4$ is a compelling choice. Compute and print out the value of $\tau_4$.

\item In a 2-by-2 subplot grid, plot the first 4 eigenworms.

\item For the first three columns of $\mT$, plot the best approximation of each column by the leading 4 eigenworms. (The results will be much smoother curves than in step~1.)
\end{enumerate}

\subsection*{Extra}
The data has been presented as $\theta(s)$, i.e.\ tangent angle as a function of arclength. The tangent and normal vectors to the curve are given by
\begin{equation}
  \begin{split}
    \bft'(s) &= \frac{d\theta}{ds} \mathbf{n}(s),\\
    \mathbf{n}'(s) &= -\frac{d\theta}{ds} \mathbf{t}(s), \\
    \bft(0) &= \bigl[1,0\bigr], \quad \mathbf{n}(0) = \bigl[ 0,1 \bigr].
  \end{split}
\end{equation}
The position vector $\bfr(s)=\bigl[x(s),y(s)\bigr]$ is then given by $\bfr'(s)=\bft(s)$, $\bfr(0)=\bigl[ 0,0 \bigr]$. For each of the four eigenworms plotted in step~5, solve these equations and plot $\bfr(s)$. (Hint: You may either discretize the ODEs using Euler's method and then plug in the computed $\theta$, or you may interpolate the computed $\theta$ and plug that into an IVP solver. High accuracy is not a concern here.)

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\section{Logical Formulas}

\frame{\tableofcontents[currentsection,hideallsubsections, firstsection=2, sections={2-4}]}

% - Logical formulas
%    - Logical formulas can be expressed as truth tables.

\subsection{Predicate Cauculus and Truth Tables}
\begin{frame}
  \frametitle{Why Mathematical Language?}
  Human language can be imprecise, so we have mathematical language that can be more specific:\bigskip

  \begin{center}
    "Go to the supermarket to buy 1 milk pack. If they have eggs, buy 12."
  \end{center}
  \bigskip

  Which of the following is correct?
  \begin{itemize}
    \item If the supermarket has eggs, buy 1 milk pack and 12 eggs.
    \item If the supermarket has eggs, buy 12 milk packs.
  \end{itemize}
  \bigskip

  To avoid this imprecision, we prefer to use mathematical language when talking about logical relationships and proofs.
\end{frame}

% Truth table.

\begin{frame}{Predicate Calculus and Logical Operators}
  The mathematical language that we use in this lecture is called \emph{Predicate Calculus}. Predicate calculus connects {\bf Predicates} and {\bf Propositions} using logical operators.\bigskip

  Many of the logical operators you already know from boolean logic:
  \begin{itemize}
    \item {\bf AND}, {\bf OR}, {\bf XOR}, {\bf NOT}, etc...
  \end{itemize}\bigskip

  There are a few more unusual logical operators too:
  \begin{itemize}
    \item {\bf IMPLIES}, {\bf IFF}, {\bf FOR ALL}, {\bf EXISTS}
  \end{itemize}
\end{frame}

\begin{frame}{Predicate Calculus and Truth Tables}
  To evaluate a formula in predicate calculus, we can use \structure{Truth Tables}, which describe every possible truth value to each proposition.\bigskip

  {\bf Example:} P AND Q IMPLIES R\bigskip

  \begin{tabular}{|lll|l|l|}
    P & Q & R & P AND Q & P AND Q IMPLIES R\\
    \hline
    {\bf TRUE} & {\bf TRUE} & {\bf TRUE} & {\bf TRUE} & {\bf TRUE} \\
    {\bf TRUE} & {\bf TRUE} & FALSE & {\bf TRUE} & FALSE \\
    {\bf TRUE} & FALSE & {\bf TRUE} & FALSE & {\bf TRUE} \\
    {\bf TRUE} & FALSE & FALSE & FALSE & {\bf TRUE} \\
    FALSE & {\bf TRUE} & {\bf TRUE} & FALSE & {\bf TRUE} \\
    FALSE & {\bf TRUE} & FALSE & FALSE & {\bf TRUE} \\
    FALSE & FALSE & {\bf TRUE} & FALSE & {\bf TRUE} \\
    FALSE & FALSE & FALSE & FALSE & {\bf TRUE} \\
  \end{tabular}
\end{frame}

\subsection{Validity and Satisfiability}
% - In mathematics and proofs: Every/Exists and their relationship
%    with validity and satisfiability. (Not only boolean)

\begin{frame}{Logic Operators: "For All" and "Exists"}
  Two of the operators we mentioned are special, and deserve some special attention:

  \begin{block}{Operator: For all}
    For a predicate $P(x)$, FOR ALL $P(x)$ is True if $P(x)$ is true for {\bf every} value of x. It is equivalent to a chain of "AND"s:

    \[
      F(P(x)): \forall x_i \in X, P(x_0) \land P(x_1) \land \ldots \land P(x_n)
    \]

  \end{block}

  \begin{block}{Operator: Exists}
    For a predicate $P(x)$, EXISTS $P(x)$ is True if $P(x)$ is true for {\bf any} value of x. It is equivalent to a chain of "OR"s:

    \[
      E(P(x)): \exists x_i \in X, P(x_0) \lor P(x_1) \lor \ldots \lor P(x_n)
    \]
  \end{block}
\end{frame}

% - Validity: Always True
% - Satisfiability: True for some values
%    - Can test both with truth tables

\begin{frame}{Validity and Satisfiability}
  The logical operators "exists" and "for all" are closed linked to the concepts of "Validity" and "Satisfiability":\bigskip

  \begin{itemize}
    \item {\bf A logical formula is \structure{Valid} if}: The formula evaluates for true for every possible assignment of every variable.\medskip

    For example: $P \lor \text{ NOT } P \implies Q$ is valid.\bigskip

    \item {\bf A logical formula is \structure{Satisfiable} if}: The formula evaluates for true for at least one possible assignment of variables.\medskip

    For example: $P \lor (Q \land R)$ is satisfiable
  \end{itemize}\bigskip
\end{frame}

\begin{frame}{Validity and Satisfiability}{Proofs and Validity}
  There are many important implications and uses for the concepts of validity and satisfiability. For example, we can use these concepts when designing proofs.\bigskip

  If we define a proposition that we want to prove as a logical formula, we can say that the proposition is true if the logical formula is \structure{Valid}.\bigskip

  On the other hand, we can define a \structure{proof by contradiction} by showing that a logical formula that indicates the negative of the proposition is \structure{satisfiable}.
\end{frame}

% - Equivalence: A and B have exactly the same truth table (IFF)
%    - Rewriting code

\begin{frame}{Equivalence}{Comparison of Two Formulas}
  Another related concept is \structure{Equivalence}. We say that two logical formulas are equivalent, if their result is identical for every variable assignment.\vfill

  For example: $\text{NOT }(P \lor Q)$ is equivalent to $\text{NOT } P \land \text{ NOT } Q$\\
  \hfill (DeMorgan's Law)\vfill

  The equivalence of two formulas is useful when rewriting code, and showing that two different pieces of code have the same result.
\end{frame}

\begin{frame}{Validity, Satisfiability, Equivalence and Truth tables}
  We can show the Validity, Satisfiability, or Equivalence of logical formulas using Truth tables:\bigskip

  \begin{itemize}
    \item {\bf A formula is valid}: If all lines in the truth table evaluate to TRUE.
    \item {\bf A formula is satisfiable}: If at least one line in the truth table evaluates to TRUE.
    \item {\bf Two formulas are equivalent}: If all lines in the truth table of the two formulas evaluate to the same value.
  \end{itemize}\bigskip

  However, remember that the size of a truth table is $2^n$, where $n$ is the number of variables in a formula, so this approach might not be feasible for complex formulas.
\end{frame}

\subsection{The Satisfiability Problem (SAT problem)}

\begin{frame}{The Satisfiability Problem}

  Consider the problem of simplifying a computer program: Given a program defined as a logical formula $A$, we want to find a smaller formula $B$ that has the same functionality.\vfill

  We can test if a certain $B$ is equivalent to $A$ by testing if the expression $A \iff B$ is {\bf valid}. Alternatively, We can test that $B$ is {\bf not} equivalent to $A$ by testing if $\text{NOT }(A \iff B)$ is {\bf satisfiable}. If we can find only one variable assignment where $A$ and $B$ are not equal, then we can discard the program candidate $B$.\vfill

  This kind of analysis is useful for making programs run faster, or for creating simpler and cheaper hardware.
\end{frame}

\begin{frame}{The Satisfiability Problem}{Proving equivalences}
  The basic algorithm for proving equivalence in a SAT problem is to test each combination of variables (each line in the truth table). As we discussed before, the number of lines is $2^n$, so this can take a very long time.\vfill

  Interestingly, if we KNOW one set of variables that satisfy the formula, it is very quick to test it. Just evaluate the formula.\vfill

  This characteristic of SAT: "Very slow to find the answer, very fast to check the answer", is one of the key characteristics of NP-hardness. If you can find a quick solution to the SAT problem, you would become a very famous computer scientist!
\end{frame}


\chapter{Fixed Point Theorem}
Probbaly one of the most surprinsg phenomenon in the world of esoteric
programming is existence of quines, the programs that print themself.\marginurl{%
Probably the most impresive example of a quine is Quine Relay, a Ruby program
  that generates Rust program that generates Scala program that generates
  \dots (through 128 languages in total)\dots REXX program that generates the
  original Ruby code again.
}{github.com/mame/quine-relay}
This chapter proves existence of such programs in almost all programming
languages.

\begin{theorem}[Kleene's Fixed Point Theorem]
\label{theorem:kleene}
  Let $h : \N \to \N$ be a total computbale function, and let $U : \N^2 \to \N$
  be a G\"odel universal function. Then there is $n \in \N$ such that $U_n$ is
  equal to $U_{h(n)}$.
\end{theorem}
\begin{proof}
  Let $f : \N \to \N$ be a computable function such that no computable function
  $g : \N \to \N$ can differ from $f$ everywhere, such a function exists by
  \Cref{theorem:intersecting-function}. Note that there is a total computable
  function $g : \N \to \N$ such that $U_{f(n)} = U_{g(n)}$ provided that $f(n)$
  is defined. Indeed, let us consider $V(x, y) = U(f(x), y)$; since $U$ is a
  G\"odel universal function, there is a total function $g(n)$ such that
  $V(x, y) = U(g(x), y)$.

  Assume for the sake of contradiction that $U_n \neq U_{h(n)}$ for all $n \in
  \N$. Let $t : \N \to \N$ be a total computable function such that $t(n) =
  h(g(n))$. It is clear that if $f$ is different from $t$ evrywhere, which
  contradictis to the definition of $f$.
\end{proof}

\begin{corollary}
  Let $U : \N^2 \to \N$ be a G\"odel universal function. Then there is $n \in
  \N$ such that $U(n, x) = n$ for all $x \in \N$.
\end{corollary}
\begin{proof}
  Let $q : \N \to \N$ be a computable total function such that $U(q(n), x) = n$
  for all $x \in \N$ (such a function exists since $U$ is a G\"odel universal
  function). Note that there is $n$ such that $U_n$ is equal to $U_{q(n)}$ which
  implies that $U(n, x) = U(q(n), x) = n$ for all $x \in \N$.
\end{proof}

\begin{exercise}
  Prove that there is a program on the programing of your choice that prints its
  text backwards.
\end{exercise}

Potentially, the function $h$ in Kleene's fixed point theorem
(\Cref{theorem:kleene}) may depend on a parameter; however, even in this case
there is a fixed point theorem.
\begin{theorem}
\label{theorem:kleene-parameter}
  Let $h : \N^2 \to \N$ be a total computbale function, and let $U : \N^2 \to \N$
  be a G\"odel universal function. Then there is total computable function $m
  \in \N$ such that $U(h(p, m(p)), x) = U(m(p), x)$ for all $p, x \in \N$
\end{theorem}

\begin{chapterendexercises}
  \exercise Show that there are different $p, q \in \N$ such that $U(p, x) = q$
    and $U(q, x) = p$ for all $x \in \N$.
  \exercise Let $h : \N \to \N$ be a total computbale function, and let
    $U : \N^2 \to \N$ be a G\"odel universal function. Show that
    there are infinetely many $n \in \N$ such that $U_n = U_{h(n)}$.
  \exercise Prove \Cref{theorem:kleene-parameter}.

\end{chapterendexercises}


\section{Integral Domains, Maximal and Prime Ideals}
\subsection{Integral Domains}
\begin{definition}
    An integral domain is a ring $R$ with $0\neq 1$ and $ab=0$ implies $a=0$ or $b=0$.
\end{definition}
\begin{definition}
    In a ring $R$, an element $a\neq 0$ is called a zero divisor if $\exists b\in R,b\neq 0,ab=0$.
\end{definition}
So an integral domain is a ring without zero divisors.
\begin{example}
    1. All fields are integral domains.\\
    2. Any subring of an integral domain is an integral domain.
    Hence $\mathbb Z[i]\le\mathbb C$ is an integral domain.\\
    3. (non-example) $\mathbb Z\times\mathbb Z$ is not an integral domain since $(1,0)(0,1)=(0,0)$.
\end{example}
\begin{lemma}
    If $R$ is an integral domain, so is $R[X]$.
\end{lemma}
\begin{proof}
    Let $f,g\in R[X]$ be nonzero polynomials.
    Suffice to show that $\deg(fg)=\deg(f)+\deg(g)$.
    Indeed, if
    $$f(X)=\sum_{k=0}^na_kX^k,g(X)=\sum_{k=0}^mb_kX^k,a_n,b_m\neq 0$$
    Then $f(X)g(X)=a_nb_mX^{n+m}+\cdots$, but since $R$ is an integral domain, $a_nb_m\neq 0$, therefore $\deg(fg)=n+m=\deg(f)+\deg(g)$.
\end{proof}
\begin{lemma}
    Let $R$ be an integral domain and $0\neq f\in R[X]$.
    Then the number of roots of $f$ in $R$ is at most $n$.
\end{lemma}
\begin{proof}
    Exercise.
\end{proof}
\begin{theorem}
    Any finite subgroup of the multiplicative group of a field is cyclic.
\end{theorem}
\begin{example}
    $(\mathbb Z/p\mathbb Z)^\times$ is cyclic.\\
    Also, $U_m=\{x^m=1:x\in\mathbb C\}$ is cyclic.
\end{example}
\begin{proof}
    Let $F$ be a field and $A$ a finite subgroup of $F^\times$.
    So $A$ is a finite abelian group, and if it is not cyclic, then by Theorem \ref{fin_abe_struct}, it contains a subgroup isomorphic to $C_m\times C_m$ for some $m\ge 2$, but then $f(X)=X^m-1$ has at least $m^2$ roots, contradicting the preceding lemma.
\end{proof}
\begin{proposition}
    Any finite integral domain is a field.
\end{proposition}
\begin{proof}
    Consider a finite integral domain $R$ and $0\neq a\in\mathbb R$.
    The map $\phi:R\to R$ by $r\mapsto ra$.
    This map is injective since $R$ is an integral domain, but then it is automatically surjective since $R$ is finite.
    So there is some $r$ such that $ra=1$.
\end{proof}
Combining these two gives that every finite integral domain has cyclic multiplicative group.
\begin{theorem}
    Let $R$ be an integral domain, then there is a field $F$ with the following properties:\\
    1. $R\le F$.\\
    2. Every element of $F$ can be written as $ab^{-1}$ where $a,b\in R$.
\end{theorem}
Consequently, such an $F$ is the unique minimal field containing $R$.
$F$ is called the field of fractions.
\begin{example}
    The field of fractions of $\mathbb Z$ is $\mathbb Q$.
\end{example}
\begin{proof}
    Consider the set $F=(R\times R\setminus\{0\})/\sim$ where
    $$(a,b)\sim (c,d)\iff ad=bc$$
    We write the equivalence class containing $(a,b)$ as $a/b$.
    One can show that this is an equivalence relation since $R$ is an integral domain and that the following operations are well-defined:
    $$(a/b)+(c/d)=(ad+bc)/(bd),(a/b)(c/d)=(ac)/(bd)$$
    $F$ is obviously a field under these two operations.
    Also we can embed $R$ into $F$ by $r\mapsto r/1$ and we have $a/b=(a/1)(1/b)=(a/1)(b/1)^{-1}=ab^{-1}$, so this is the field we want.
\end{proof}
\begin{example}
    1. The field of fraction of the Gaussian integers $\mathbb Z[i]$ is the set $\{ab^{-1}:a,b\in\mathbb Z[i]\le\mathbb C\}$.
    In fact, $F$ is exactly numbers in the form $p+iq,p,q\in\mathbb Q$.\\
    2. The field of fraction of the polynomial ring $R[X]$ over a ring $R$ is called the field of fractions $R(X)$ of $R$.
\end{example}
\subsection{Prime and Maximal Ideals}
\begin{lemma}
    A ring $R$ is a field iff its only ideas are $\{0\},R$.
\end{lemma}
\begin{proof}
    Trivial.
\end{proof}
\begin{definition}
    Let $S$ be a collection of subsets of a set $X$.
    $A\in S$ is maximal if there does not exists $B\in S$ such that $A\subsetneq B$.\\
    An ideal $I\unlhd R$ is maximal if it is maximal in the set of all proper ideas $\mathcal I_R=\{J\unlhd R:\{0\}\subsetneq J\subsetneq R\}$.
\end{definition}
\begin{proposition}
    Let $I\unlhd R$, then $R/I$ is a field iff $I$ is maximal.
\end{proposition}
\begin{proof}
    $R/I$ is a field iff $I/I,R/I$ are the only ideals of $R/I$, which happens iff $I$ and $R$ are the only ideals of $R$ containing $I$ iff $I$ is maximal.
\end{proof}
\begin{definition}
    An ideal $I\unlhd R$ is prime if $I\neq R$ and $ab\in I$ implies that at least one of $a,b$ is in $I$.
\end{definition}
\begin{example}
    The prime ideals of $\mathbb Z$ are $p\mathbb Z$ with $p$ prime or $0$.
    Incidentally (or not), $p\mathbb Z$ are also all maximal ideals of $\mathbb Z$.
\end{example}
\begin{proposition}
    Let $I\unlhd R$, then $I$ is prime iff $R/I$ is an integral domain.
\end{proposition}
\begin{proof}
    $I$ is prime iff $ab\in I\implies a\in I\lor b\in I$ iff $ab+I=I\implies a+I=I\lor b+I=I$ iff $I$ is an integral domain.
\end{proof}
\begin{remark}
    Combining the results reveals that every maximal ideal is prime.
\end{remark}
\begin{remark}
    If $\operatorname{char}(R)=n\ge 2$, then $\mathbb Z/n\mathbb Z\le R$, hence $n$ is prime.
    In particular, the characteristic of a field $F$ is either $0$ or a prime number.
    When the field has characteristic $0$, then $\mathbb Z\le F$, hence $\mathbb Q\le F$ since $F$ is a field.
\end{remark}


\documentclass[12pt]{article}

\usepackage{mathtools}

\title{Bayes's theorem and logistic regression}
\author{Allen B. Downey}

\newcommand{\logit}{\mathrm{logit}}
\renewcommand{\P}{\mathrm{P}}
\renewcommand{\O}{\mathrm{O}}
\newcommand{\LR}{\mathrm{LR}}
\newcommand{\LO}{\mathrm{LO}}
\newcommand{\LLR}{\mathrm{LLR}}
\newcommand{\OR}{\mathrm{OR}}
\newcommand{\LOR}{\mathrm{LOR}}
\newcommand{\IF}{\mathrm{if}}
\newcommand{\notH}{\neg H}

\setlength{\headsep}{3ex}
\setlength{\parindent}{0.0in}
\setlength{\parskip}{1.7ex plus 0.5ex minus 0.5ex}

\begin{document}

\maketitle

\begin{abstract}
My two favorite topics in probability and statistics are
Bayes's theorem and logistic regression.  Because there are
similarities between them, I have always assumed that there is
a connection.  In this note, I demonstrate the
connection mathematically, and (I hope) shed light on the
motivation for logistic regression and the interpretation of
the results.
\end{abstract}

\section{Bayes's theorem}

I'll start by reviewing Bayes's theorem, using an example that came up
when I was in grad school.  I signed up for a class on Theory of
Computation.  On the first day of class, I was the first to arrive.  A
few minutes later, another student arrived.  Because I was expecting
most students in an advanced computer science class to be male, I was
mildly surprised that the other student was female.  Another female
student arrived a few minutes later, which was sufficiently
surprising that I started to think I was in the wrong room.  When
another female student arrived, I was confident I was in the wrong
place (and it turned out I was).

As each student arrived, I used the observed data to update my
belief that I was in the right place.  We can use Bayes's theorem to
quantify the calculation I was doing intuitively.

I'll us $H$ to represent the hypothesis that I was in the right
room, and $F$ to represent the observation that the first other
student was female.  Bayes's theorem provides an algorithm for
updating the probability of $H$:

\[ \P(H|F) = \P(H)~\frac{\P(F|H)}{P(F)}\]

Where

\begin{itemize}

\item $\P(H)$ is the prior probability of $H$ before the other
student arrived.

\item $\P(H|F)$ is the posterior probability of $H$, updated based
on the observation $F$.

\item $\P(F|H)$ is the likelihood of the data, $F$, assuming that
the hypothesis is true.

\item $P(F)$ is the likelihood of the data, independent of $H$.

\end{itemize}

Before I saw the other students, I was confident I was in the right
room, so I might assign $\P(H)$ something like 90\%.

When I was in grad school most advanced computer science classes were
90\% male, so if I was in the right room, the likelihood of the
first female student was only 10\%.  And the likelihood of three
female students was only 0.1\%.

If we don't assume I was in the right room, then the likelihood of
the first female student was more like 50\%, so the likelihood
of all three was 12.5\%.

Plugging those numbers into Bayes's theorem yields $\P(H|F) = 0.64$
after one female student, $\P(H|FF) = 0.26$ after the second,
and $\P(H|FFF) = 0.07$ after the third.

\section{Logistic regression}

Logistic regression is based on the following functional form:

\[ \logit(p) = \beta_0 + \beta_1 x_1 + ... + \beta_n x_n \]

where the dependent variable, $p$, is a probability,
the $x$s are explanatory variables, and the $\beta$s are
coefficients we want to estimate.  The $\logit$ function is the
log-odds, or

\[ \logit(p) = \ln \left( \frac{p}{1-p} \right) \]

When you present logistic regression like this, it raises
three questions:

\begin{itemize}

\item Why is $\logit(p)$ the right choice for the dependent
variable?

\item Why should we expect the relationship between $\logit(p)$
and the explanatory variables to be linear?

\item How should we interpret the estimated parameters?

\end{itemize}

The answer to all of these questions turns out to be Bayes's
theorem.  To demonstrate that, I'll use a simple example where
there is only one explanatory variable.  But the derivation
generalizes to multiple regression.

On notation: I'll use $\P(H)$ for the probability
that some hypothesis, $H$, is true.  $\O(H)$ is the odds of the same
hypothesis, defined as

\[ \O(H) = \frac{\P(H)}{1 - \P(H)} \]

I'll use $\LO(H)$ to represent the log-odds of $H$:

\[ \LO(H) = \ln \O(H) \]

I'll also use $\LR$ for a likelihood ratio, and $\OR$ for an odds
ratio.  Finally, I'll use $\LLR$ for a log-likelihood ratio, and
$\LOR$ for a log-odds ratio.

\section{Making the connection}

To demonstrate the connection between Bayes's theorem and
logistic regression, I'll start with the odds form
of Bayes's theorem.  Continuing the previous example,
I could write

\begin{equation} \label{A}
\O(H|F) = \O(H)~\LR(F|H)
\end{equation}

where

\begin{itemize}

\item $\O(H)$ is the prior odds that I was in the right room,

\item $\O(H|F)$ is the posterior odds after seeing one female student,

\item $\LR(F|H)$ is the likelihood ratio of the data, given
the hypothesis.

\end{itemize}

The likelihood ratio of the data is:

\[ \LR(F|H) = \frac{\P(F|H)}{\P(F|\notH)} \]

where $\notH$ means $H$ is false.

Noticing that logistic regression is expressed in terms of
log-odds, my next move is to write the log-odds form of
Bayes's theorem by taking the log of Eqn~\ref{A}:

\begin{equation} \label{B}
\LO(H|F) = \LO(H) + \LLR(F|H)
\end{equation}

If the first student to arrive had been male, we would write

\begin{equation} \label{C} \nonumber
\LO(H|M) = \LO(H) + \LLR(M|H)
\end{equation}

Or more generally if we use $X$ as a variable to represent
the sex of the observed student, we would write

\begin{equation} \label{D}
\LO(H|X) = \LO(H) + \LLR(X|H)
\end{equation}

I'll assign $X=0$ if the observed student is female and
$X=1$ if male.  Then I can write:

\begin{equation} \label{E} \nonumber
\LLR(X|H) = \left\{
  \begin{array}{lr}
    \LLR(F|H) & \IF ~X = 0\\
    \LLR(M|H) & \IF ~X = 1
  \end{array}
\right.
\end{equation}

Or we can collapse these two expressions into one by using
$X$ as a multiplier:

\begin{equation} \label{F}
\LLR(X|H) = \LLR(F|H) + X [\LLR(M|H) - \LLR(F|H)]
\end{equation}

\section{Odds ratios}

The next move is to recognize that
the part of Eqn~\ref{F} in brackets is the log-odds ratio
of $H$.  To see that, we need to look more closely at odds ratios.

Odds ratios are often used in medicine to describe the association
between a disease and a risk factor.  In the example scenario, we
can use an odds ratio to express the odds of the hypothesis
$H$ if we observe a male student, relative to the odds if we
observe a female student:

\[ \OR_X(H) = \frac{\O(H|M)}{\O(H|F)} \]

I'm using the notation $\OR_X$ to represent the odds ratio
associated with the variable $X$.

Applying Bayes's theorem to
the top and bottom of the previous expression yields

\[ \OR_X(H) = \frac{\O(H)~\LR(M|H)}{\O(H)~\LR(F|H)} =
\frac{\LR(M|H)}{\LR(F|H)}\]

Taking the log of both sides yields

\begin{equation} \label{G}
\LOR_X(H) = \LLR(M|H) - \LLR(F|H)
\end{equation}

This result should look familiar, since it appears in
Eqn~\ref{F}.

\section{Conclusion}

Now we have all the pieces we need; we just have to assemble them.
Combining Eqns~\ref{F} and \ref{G} yields

\begin{equation} \label{H}
\LLR(H|X) = \LLR(F) + X~\LOR(X|H)
\end{equation}

Combining Eqns~\ref{D} and \ref{H} yields

\begin{equation} \label{I}
\LO(H|X) = \LO(H) + \LLR(F|H) + X~\LOR(X|H)
\end{equation}

Finally, combining Eqns~\ref{B} and \ref{I} yields

\[ \LO(H|X) = \LO(H|F) + X~\LOR(X|H) \]

We can think of this equation as the log-odds form of Bayes's theorem,
with the update term expressed as a log-odds ratio.  Let's compare
that to the functional form of logistic regression:

\[ \logit(p) = \beta_0 + X \beta_1 \]

The correspondence between these equations suggests the following
interpretation:

\begin{itemize}

\item The predicted value, $\logit(p)$, is the posterior log
odds of the hypothesis, given the observed data.

\item The intercept, $\beta_0$, is the log-odds of the
hypothesis if $X=0$.

\item The coefficient of $X$, $\beta_1$, is a log-odds ratio
that represents odds of $H$ when $X=1$, relative to
when $X=0$.

\end{itemize}

This relationship between logistic regression and Bayes's theorem
tells us how to interpret the estimated coefficients.  It also
answers the question I posed at the beginning of this note:
the functional form of logistic regression makes sense because
it corresponds to the way Bayes's theorem uses data to update
probabilities.

\end{document}

DSC 530 - Data Exploration and Analysis/ThinkStats2/book/note.tex


% !TEX root = main.tex

%-------------------------------------------------
\section{Moments}\label{sec:moments}

%-----------------------------
\subsection{Expectation of transformed variables}\label{sec:expe_transformed}

Let $X$ be a random variable and let $g:\R\to\R$ be a transformation.

\ben
\it If $g$ is a \emph{non-negative} function, the expected value of $g(X)$ is
\[
\expe\big[g(X)\big] = \begin{cases}
	\displaystyle\sum_{i=1}^{\infty} g(x_i)f(x_i)		& \text{if $g(X)$ is discrete, and} \\[3ex]
	\displaystyle\int_{-\infty}^{\infty} g(x)f(x)\,dx	& \text{if $g(X)$ is continuous.}
\end{cases}
\]

\it If $g$ is a \emph{signed function}, the expected value of $g(X)$ is
\[
\expe\big[g(X)\big] = \expe\big[g^{+}(X)\big] - \expe\big[g^{-}(X)\big],
\]
provided $\expe\big[g^{+}(X)\big]$ and $\expe\big[g^{-}(X)\big]$ are not both infinite, where $g^{+}$ and $g^{-}$ are respectively the positive and negative parts of $g$,
\[
g^{+}(x)	= \begin{cases}  g(x) & \text{ if } g(x)\geq 0, \\ 0 & \text{otherwise,}\end{cases}
\qquad\text{and}\qquad
g^{-}(x)	= \begin{cases} -g(x) & \text{ if } g(x)<0, \\ 0 & \text{otherwise.}\end{cases}
\]
\een

\begin{example}
For $X\sim\text{Uniform}[-1,1]$ find $\expe(1/X^2)$ and $\expe(1/X)$.
\begin{solution}
The PDF of $X$ is $f(x)=1/2$ for $x\in[-1,1]$ and zero otherwise.
\ben
\it Let $g(x) = 1/x^2$. This is a non-negative function, so
\begin{align*}
\expe\left(\frac{1}{X^2}\right)
	= \int_{-\infty}^{\infty} g(x)f(x)\,dx
	= \frac{1}{2}\int_{-1}^{1} \frac{1}{x^2}\,dx
	= \int_{0}^{1}\frac{1}{x^2}\,dx
	= \infty,
\end{align*}
%so $\expe(1/X^2)$ is infinite.

\it Let $g(x) = 1/x$. This is a signed function, so we must consider its positive and negative parts separately:
\[
g^{+}(x)	= \begin{cases}  1/x & \text{ if}\quad x\geq 0, \\ 0 & \text{ if}\quad x < 0,\end{cases}
\qquad\text{and}\qquad
g^{-}(x)	= \begin{cases}  0 & \text{ if}\quad x\geq 0, \\ -1/x & \text{ if}\quad x < 0.\end{cases}
\]
Thus
\begin{align*}
\expe\left(\frac{1}{X}\right)
	& = \int_{0}^{\infty} g^{+}(x)f(x)\,dx - \int_{0}^{\infty} g^{-}(x) f(x)\,dx \\
	& = \frac{1}{2}\int_0^1 \frac{1}{x}\,dx - \frac{1}{2}\int_{-1}^0 \frac{-1}{x}\,dx \\
	& = \frac{1}{2}\int_0^1 \frac{1}{x}\,dx - \frac{1}{2}\int_{0}^{1} \frac{1}{x}\,dx \\
	& = \infty - \infty,
\end{align*}
so $\expe(1/X)$ is undefined.
\een
\end{solution}
\end{example}

%-----------------------------
\subsection{Moments}

We now consider transformations of the form $g(x)=x^k$ for various values of $k\in\Z$.
% definition: raw moments
\begin{definition}
The expectation $\expe(X^{k})$ is called the \emph{$k$th moment about the origin} of $X$.
\end{definition}

%In particular,
\bit
\it $\expe(X^0) = 1$,
\it $\expe(X)$ is the \emph{mean} of $X$, often denoted by $\mu$,
\it $\expe(X^2)$ is the \emph{mean-square} of $X$.
\eit

% definition: central moments
\begin{definition}
The expectation $\expe\big[(X-\mu)^{k}\big]$ is called the \emph{$k$th moment about the mean} of $X$.
\end{definition}

%In particular,
\bit
\it $\expe\big[(X-\mu)^{0}\big] = 1$,
\it $\expe\big[(X-\mu)\big] = 0$,
\it $\expe\big[(X-\mu)^2\big]$ is the \emph{variance} of $X$, often denoted by $\sigma^2$.
\it $\expe\big[(X-\mu)^2\big]^{1/2}$ is the \emph{standard deviation} of $X$, often denoted by $\sigma$.
%\it $\sqrt{\mu_2}$ is the \emph{standard deviation} of $X$, denoted by $\sigma$.
\eit

%-----------------------------
\subsection{Location, scale and shape}

When trying to describe a distribution, it is natural to look for its \emph{location}, \emph{scale} (size) and \emph{shape}.

%% mean and variance
%For any random variable $X:\Omega\to\R$,
%\ben
%\it the first moment of $X$ about the origin ($\mu$)  describes its location,
%\it the second moment of $X$ about the mean ($\sigma^2$) describes its scale, and
%\it the higher moments of $X$ describe the shape of its distribution.
%\een

\paragraph{Location}
To locate $X$, we look for a point $c\in\R$ such that the expected squared deviation $\expe\big[(X-c)^2\big]$ around this point is minimum. By the linearity of expectation,
\[
\expe\big[(X-c)^2\big] = \expe(X^2 - 2cX + c^2) = \expe(X^2) - 2c\expe(X) + c^2
\]
To find the value of $c$ that minimises the expected squared deviation, we differentiate the right-hand side with respect to $c$ and set the resulting expression to zero. This yields $c=\expe(X)$, so the location of $X$ is described by its \emph{expectation} (or first moment about the origin), $\mu$.

\paragraph{Scale}
The size of a distribution should not depend on its location, so we consider the \emph{centred} variable
\[
Y=X-\expe(X),
\]
which has the property $\expe(Y)=0$. The expected squared deviation of $X$ around its mean $\expe(X)$ is its \emph{variance} (or second moment about the mean), so the size of $X$ is described by its \emph{standard deviation}, $\sigma$.

\paragraph{Shape}
The shape of a distribution should not depend on its location nor its scale. Thus we consider the higher moments of the so-called \emph{standardised} variable,
\[
Z = \frac{X-\mu}{\sigma},
\]
which has the properties $\expe(Z)=0$ and $\var(Z)=1$.

% definition
\begin{definition}
The \emph{skewness} of a random variable $X$ is defined to be
\[
\gamma_1 = \expe\left[\left(\frac{X-\mu}{\sigma}\right)^3\right] %= \frac{\mu_{3}}{\sigma^3}
\]
%where $\mu_3$ is the third central moment of $X$, and $\sigma$ is its standard deviation.
\end{definition}

Skewness is a measure of \emph{asymmetry}:
\bit
\it Negative skew ($\gamma_1 < 0$): long tail on the left, mass concentrated on the right.
\it Positive skew ($\gamma_1 > 0$): long tail on the right, mass concentrated on the left.
\eit

% example: skewness of binomial
\begin{example}\label{ex:skewness_binomial}
If $X\sim\text{Binomial}(n,p)$, some tedious algebra shows that the skewness of $X$ is
\[
\gamma_1 = \frac{1-2p}{\sqrt{np(1-p)}} = \begin{cases}
	< 0 	& \text{if } p > \frac{1}{2} \\
	= 0 	& \text{if } p = \frac{1}{2} \\
	> 0 	& \text{if } p < \frac{1}{2} \\
\end{cases}
\]
\end{example}

% definition: kurtosis
\begin{definition}
The \emph{excess kurtosis} of a random variable $X$ is defined to be
\[
\gamma_2 = \expe\left[\left(\frac{X-\mu}{\sigma}\right)^4\right] -3.%= \frac{\mu_{4}}{\sigma^4},
\]
%and the \emph{excess kurtosis} is $\gamma_2 = \beta_2 - 3$.
\end{definition}

Kurtosis is a measure of \emph{peakiness}. The normal distribution has $\gamma_2=0$, so excess kurtosis provides a measure of peakiness relative to that of the normal distribution.
\bit
\it Negative excess kurtosis ($\gamma_2 < 0$): tall and peaky with heavy tails (\emph{leptokurtic}),
\it Positive excess kurtosis ($\gamma_2 > 0$): low and wide with light tails (\emph{platykurtic}).
\eit

\begin{example}\label{ex:kurtosis_binomial}
If $X\sim\text{Binomial}(n,p)$ some more tedious algebra shows that the excess kurtosis of $X$ is
\[
\gamma_2 = \frac{1-6p(1-p)}{np(1-p)} = \begin{cases}
	< 0 				& \text{if } \left|p -\frac{1}{2}\right| < \frac{1}{2\sqrt{3}} \\
	> 0 				& \text{if } \left|p -\frac{1}{2}\right| > \frac{1}{2\sqrt{3}} \\
\end{cases}
\]
\end{example}

% exercises: moments
\begin{exercise}
\begin{questions}

\question
Let $X$ be a continuous random variable with uniform density on the interval $[-1,1]$,
\[
f(x) = \begin{cases}
	\frac{1}{2}	& \text{ if } x\in[-1,+1] \\
	0			& \text{ otherwise.}
\end{cases}
\]
Compute the moments $\expe(X)$, $\expe(X^2)$, $\expe(X^3)$, $\expe(1/X)$ and $\expe(1/X^2)$.
\begin{answer}
Since $xf(x)$, $x^2f(x)$ and $x^3f(x)$ are continuous and bounded over $\supp(f)=[-1,1]$, the first three expectations can be computed using definite integrals. This is not the case for$\expe(1/X)$ and $\expe(1/X^2)$.

\ben
\it % << (i)
$\displaystyle\expe(X) = \frac{1}{2}\int_{-1}^1 x\,dx = 0.$
%Let $g(x) = x$. This is a signed function, so
%\begin{align*}
%\expe(X)
%	& = \int_{-\infty}^{\infty} g^{+}(x)f(x)\,dx - \int_{-\infty}^{\infty} g^{-}(x) f(x)\,dx \\
%	& = \frac{1}{2}\int_0^1 x\,dx - \frac{1}{2}\int_{-1}^0 (-x)\,dx \\
%	& = \frac{1}{2}\left[\frac{x^2}{2}\right]_0^1 - \frac{1}{2}\left[\frac{-x^2}{2}\right]_{-1}^0
%	= \left(\frac{1}{4}-0\right) - \left(0 + \frac{1}{4}\right)
%	= 0.
%\end{align*}
%
\it % << (ii)
$\displaystyle\expe(X^2) = \frac{1}{2}\int_{-1}^1 x^2\,dx = \frac{1}{3}.$
%Let $g(x) = x^2$. This is a non-negative function, so
%\begin{align*}
%\expe(X^2)
%	& = \int_{-\infty}^{\infty} g(x)f(x)\,dx \\
%	& = \frac{1}{2}\int_{-1}^{1} x^2\,dx
%	= \int_{0}^{1} x^2\,dx
%	= \left[\frac{x^3}{3}\right]_0^1
%	= \frac{1}{3}.
%\end{align*}
%
\it % << (iii)
$\displaystyle\expe(X^3) = \frac{1}{2}\int_{-1}^1 x^3\,dx = 0.$
%Let $g(x) = x^3$. This is a signed function, so
%\begin{align*}
%\expe(X^3)
%	& = \int_{-\infty}^{\infty} g^{+}(x)f(x)\,dx - \int_{-\infty}^{\infty} g^{-}(x) f(x)\,dx \\
%	& = \frac{1}{2}\int_0^1 x^3\,dx - \frac{1}{2}\int_{-1}^0 (-x^3)\,dx \\
%	& = \frac{1}{2}\left[\frac{x^4}{4}\right]_0^1 - \frac{1}{2}\left[\frac{-x^4}{4}\right]_{-1}^0
%	= \left(\frac{1}{8}-0\right) - \left(0 + \frac{1}{8}\right)
%	= 0.
%\end{align*}

\it % << (iv)
Let $g(x) = 1/x$. This is a signed function, so
\begin{align*}
\expe\left(\frac{1}{X}\right)
	& = \int_{-\infty}^{\infty} g^{+}(x)f(x)\,dx - \int_{-\infty}^{\infty} g^{-}(x) f(x)\,dx \\
	& = \frac{1}{2}\int_0^1 \frac{1}{x}\,dx - \frac{1}{2}\int_{-1}^0 \frac{-1}{x}\,dx \\
	& = \frac{1}{2}\int_0^1 \frac{1}{x}\,dx - \frac{1}{2}\int_{0}^{1} \frac{1}{x}\,dx
	= \infty - \infty
\end{align*}
so $\expe(1/X)$ is undefined.

\it % << (v)
Let $g(x) = 1/x^2$. This is a non-negative function, so
\begin{align*}
\expe\left(\frac{1}{X^2}\right)
	& = \int_{-\infty}^{\infty} g(x)f(x)\,dx \\
	& = \frac{1}{2}\int_{-1}^{1} \frac{1}{x^2}\,dx
	= \int_{0}^{1}\frac{1}{x^2}\,dx
	= \infty
\end{align*}
so $\expe(1/X^2)$ is infinite.
\een
\end{answer}

\question
Let $X$ be a continuous random variable with the following PDF:
\[
f(x) = \begin{cases}
	1-|x|	& \text{ if } x\in[-1,1] \\
	0		& \text{ otherwise.}
\end{cases}
\]
For what values of $k\in\Z$ do the moments $\expe(X^{k})$ exist?
\begin{answer}
For $k>0$,
\[
\expe(X^{k}) = \int_{-1}^{0}x^{k}(1+x)\,dx + \int_{0}^{1}x^{k}(1-x)\,dx < \infty
\]
Let $k < 0$. If $k$ is even then $X^k$ is non-negative, so
\begin{align*}
\expe(X^k)
	& = \expe\big((X^{+})^{k}\big) = +\infty\\
\intertext{If $k$ is odd,}
\expe(X^k)
	& = \expe\big((X^{+})^{k}\big) - \expe\big((X^{-})^{k}\big)
		= \infty - \infty
\end{align*}
so in this case the moment $\expe(X^k)$ does not exist.
\end{answer}

\question
Let $X$ be a discrete random variable with the following PMF:
\[
\prob(X=x) = \frac{45}{\pi^4x^4} \qquad\text{for $x=\pm 1,\pm 2,\pm 3\ldots$ (and zero otherwise).}
\]
\begin{parts}
\part For what values of $k\in\Z$ do the moments $\expe(X^{k})$ exist?
\begin{answer}
Let $c = 45/\pi^4$.
\bit
\it If $k\leq 2$ then $\expe(X^k)$ exists and is finite because
\[
\expe(X^k)
	= c\sum_{x\neq 0}\frac{x^k}{x^4}
	= 2c\sum_{x=1}^{\infty}\frac{1}{x^{4-k}}
	< \infty.
\]
\it If $k>2$ and $k$ is even, $\expe(X^k)$ is infinite because
\[
\expe(X^k)
	= c\sum_{x\neq 0}\frac{x^k}{x^4}
	= c\sum_{x=1}^{\infty}\frac{1}{x^{4-k}}
	= \infty.
\]
If $k>2$ and $k$ is odd, $\expe(X^k)$ does not exist because
\[
\expe(X^k)
	= c\sum_{x\neq 0}\frac{x^k}{x^4}
	= c\sum_{x=1}^{\infty}\frac{1}{x^{4-k}} - c\sum_{x=1}^{\infty}\frac{1}{x^{4-k}}
	= \infty - \infty.
\]
\eit
\end{answer}
\part Compute the variance of $X$ and its first negative moment $\expe(X^{-1})$.
\begin{answer}
The first two moments of $X$ are
\begin{align*}
\expe(X)
	& = c\sum_{x\neq 0}\frac{1}{x^3}
	  = c\sum_{x=1}^{\infty}\frac{1}{x^3} - c\sum_{x=1}^{\infty}\frac{1}{x^3} = 0 \\
\expe(X^2)
	& = c\sum_{x\neq 0}\frac{1}{x^2}
	  = 2c\sum_{x=1}^{\infty}\frac{1}{x^2}
	  = 2\left(\frac{45}{\pi^4}\right)\left(\frac{\pi^2}{6}\right) = \frac{15}{\pi^2},
\end{align*}
so $\var(X)=15/\pi^2$. Similarly,
\begin{align*}
\expe\left(\frac{1}{X}\right)
	& = c\sum_{x\neq 0}^{\infty}\frac{1}{x^5} \\
	& = c\sum_{n=1}^{\infty}\frac{1}{n^5} - c\sum_{x=1}^{\infty}\frac{1}{x^5} \\
	& = 0.
\end{align*}
In fact, all negative odd-integer moments are zero.
\end{answer}
\end{parts}

\end{questions}
\end{exercise}


\subsection{Analytic Maps}
\begin{definition}
    Let $R$ and $S$ be Riemann surfaces.
    A continuous map $f:R\to S$ is analytic or holomorphic if for any chart $(\phi,U)$ on $R$ and $(\psi,V)$ on $S$, the map $\psi\circ f\circ \phi^{-1}$ is analytic on $\phi(U\cap f^{-1}(V))$.
\end{definition}
\begin{lemma}\label{analytic_local}
    A continuous map $f:R\to S$ of Riemann surfaces is analytic iff for each $p\in R$, there is a chart $\phi_p,U_p$ on $R$ with $p\in U_p$ and a chart $(\psi_p,V_p)$ on $V$ with $f(p)\in V_p$ such that $\psi_p\circ f\circ \phi_p^{-1}$ is analytic on $\phi_p(U_p\cap f^{-1}(V_p))$
\end{lemma}
The key point of the proof is basically that the function we want to be analytic can be written as a composition of local analytic functions composed with transition maps.
\begin{proof}
    The ``only if'' direction is immediate.
    For the ``if'' direction, given charts $(\phi,U)$ on $R$ and $(\psi,V)$ on $S$, it suffices to show that $\psi\circ f\circ\phi^{-1}$ is analytic at $\phi(p)$ for any $p\in U\cap f^{-1}(V)$.
    We know from hypothesis that there is some charts $(\phi_p,U_p)$ on $R$ and $(\psi_p,V_p)$ on $S$ with $p\in U_p,f(p)\in V_p$ and $\psi_p\circ f\circ\phi_p^{-1}$ is analytic on $\phi_p(U\cap f^{-1}(V))$, hence in particular at $\phi_p(p)$.
    Hence
    $$\psi\circ f\circ\phi^{-1}=(\psi\circ\psi_p^{-1})\circ(\psi_p\circ f\circ\phi_p^{-1})\circ (\phi_p\circ\phi^{-1})$$
    is analytic at $\phi(p)$.
\end{proof}
\begin{lemma}\label{analytic_compose}
    If $f:R\to S$ and $g:S\to T$ are analytic, so is $g\circ f:R\to T$.
\end{lemma}
\begin{proof}
    Simple corollary of Lemma \ref{analytic_local}.
\end{proof}
\begin{definition}
    A conformal equivalence or biholomorphism is an analytic bijection of Riemann surfaces with analytic inverse.
\end{definition}
It is an equivalence relation by Lemma \ref{analytic_compose}.
\begin{example}
    The map $f:\mathbb C\to\bar{\mathbb C}$ with $z\mapsto \bar{z}$ is a conformal equivalence.
\end{example}


\subsection{Probability of null}

\(P(\Omega )=1\)

\(P(\Omega \lor \varnothing )=1\)

\(P(\Omega )+P(\varnothing )=1\)

\(P(\varnothing )=0\)

\subsection{Monotonicity}

Consider \(E_i\subseteq E_j\):

\(E_j=E_i\lor E_k\)

\(P(E_j)=P(E_i\lor E_k)\)

Disjoint so:

\(P(E_j)=P(E_i)+P(E_k)\)

We know that \(P(E_k)\ge 0\) from axiom \(1\) so:

\(P(E_j)\ge P(E_i)\)

\subsection{Bounds of probabilities}

As all events are subsets of the sample space:

\(P(\Omega )\ge P(E)\)

\(1\ge P(E)\)

From axiom \(1\) then know:

\(\forall E\in F [0\le P(E)\le 1]\)

\subsection{Union and intersection for null and universal}

\(P(E\land \varnothing )=P(\varnothing )=0\)

\(P(E\lor \Omega )=P(\Omega )=1\)

\(P(E\lor \varnothing)=P(E)\)

\(P(E\land \Omega )=P(E)\)

\subsection{Separation rule}

Firstly:

\(P(E_i)=P(E_i\land \Omega)\)

\(P(E_i)=P(E_i\land (E_j\lor E_j^C))\)

\(P(E_i)=P((E_i\land E_j)\lor (E_i\land E_j^C))\)

As the latter are disjoint:

\(P(E_i)=P((E_i\land E_j)+(E_i\land E_j^C))\)

\subsection{Addition rule}

We know that:

\(P(E_i\lor E_j)=P((E_i\lor E_j)\land (E_j\lor E_j^C))\)

By the distributive law of sets:

\(P(E_i\lor E_j)=P((E_i\land E_j^C)\lor E_j)\)

\(P(E_i\lor E_j)=P((E_i\land E_j^C)\lor (E_j\land (E_i\lor E_i^C))\)

By the distributive law of sets:

\(P(E_i\lor E_j)=P((E_i\land E_j^C)\lor (E_j\land E_i)\lor (E_j\land E_i^C))\)

As these are disjoint:

\(P(E_i\lor E_j)=P(E_i\land E_j^C)+ P(E_j\land E_i)+P(E_j\land E_i^C)\)

From the separation rule:

\(P(E_i\lor E_j)=P(E_i)-P(E_i\land E_j)+ P(E_j\land E_i)+P(E_j)-P(E_j\land E_i)\)

\(P(E_i\lor E_j)=P(E_i)+P(E_j)-P(E_i\land E_j)\)

\subsection{Probability of complements}

From the addition rule:

\(P(E_i\lor E_j)=P(E_i)+P(E_j)-P(E_i\land E_j)\)

Consider \(E\) and \(E^C\):

\(P(E\lor E^C)=P(E)+P(E^C)-P(E\land E^C)\)

We know that \(E\) and \(E^C\) are disjoint, that is:

\(E\land E^C=\varnothing\)

Similarly by construction:

\(E\lor E^C=\Omega \)

So:

\(P(\Omega )=P(E)+P(E^C)-P(\varnothing)\)

\(1=P(E)+P(E^C)\)


\subsection{Priors of trees}

\subsubsection{Priors for simple trees}

We can define a tree as a set of nodes: \(T\).

For each node we define a splitting variable \(k\) and a splitting threshold \(r\).

Our prior is \(P(T, k, r)\).

We split this up to:

\(P(T, k, r)=P(T)P(k, r|T)\)

\(P(T, k, r)=P(T)P(k|T)P(r|T, k)\)

So we want to estimate:

\begin{itemize}
\item \(P(T)\) - The number of nodes.
\item \(P(k|T)\) - Which variables we split by, given the tree size.
\item \(P(r|T, k)\) - The cutoff, given the tree size and the variables we are splitting by.
\end{itemize}

\subsubsection{Priors for mixed trees}

If at the leaf we have a parametric model, our prior is instead:

\(P(T, k, r, \theta )=P(T)P(k|T)P(r|T, k)P(\theta | T, k, r)\)

We then need to additionally estimate \(P(\theta | T, k, r)\).


\section{Adaptive Step Sizes}

\begin{enumerate}[label=\alph*), leftmargin=*]

%% a)
\item
%

Let a Moving Average process $x(n)$ of order $q=1$, MA(1), such that:

\begin{equation}
    x(n) = 0.9 \eta(n - 1) + \eta(n), \quad \eta \sim \mathcal{N}(0, 0.5)
\end{equation}

When LMS is used with a constant step-size $\mu$, a trade-off between convergence speed, smoothness and oscillations magnitude at steady-state is inevitable.
Gradient Adaptive Step-Size (GASS) algorithms attempt to combine the positive aspects of both
large and small values of $\mu$ by adapting its magnitude according to the gradient $\nabla_{\mu} \mathcal{J}$, where $\mathcal{J}$ the objective function
minimised by the adaptive filter.

Different GASS algorithms, namely Benvenist, Ang \& Farhan and Matthews \& Xi, are implemented and their performance is compared to simple constant step-size LMS algorithm.
Weight error and squared prediction error curves are provided at figure \ref{fig:3_2_a}.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-step-sizes/assets/a/weight_error_curves}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-step-sizes/assets/a/squared_prediction_error}
    \end{subfigure}
    \caption{GASS LMS: weight error and squared prediction error curves.}
    \label{fig:3_2_a}
\end{figure}

All GASS algorithms converge to the true process parameters faster than simple LMS (within less than 50 steps), while unsurprisingly,
the most computationally intensive algorithm, Benvenist ($\mathcal{O}(M^{2})$), adapts first of all and scores the smallest squared prediction error.
Moreover, their steady-state error is much smaller (below $-300dB$) than the fixed step-size LMS algorithm (at $-200dB,\ -120dB,\ -30dB$ for $\mu = 0.1,\ 0.05,\ 0.01$, respectively),
improving the EMSE from the previous part.

%% b)
\item
%

Starting from the update equation based on the \textit{a posteriori error} $e_{p}(n) = d(n) - \mathbf{x}(n)^{T} \vw(n + 1)$:

\begin{equation}
    \vw(n + 1) = \vw(n) + \mu e_{p}(n) \mathbf{x}(n)
\label{eq:nlms_a_error}
\end{equation}

Multiply both sides with $-\mathbf{x}(n)^{T}$ and add $d(n)$ to construct $e_{p}(n)$ on the LHS:

\begin{equation}
    d(n) - \mathbf{x}(n)^{T} \vw(n + 1) = d(n) - \mathbf{x}(n)^{T} \vw(n) - \mathbf{x}(n)^{T} \mu e_{p}(n) \mathbf{x}(n)
\end{equation}

Note that the LHS term is the a posteriori error, $e_{p}(n)$, while the first RHS term the a priori error, $e(n)$:

\begin{align}
    e_{p}(n)    &= e(n) - \mu e_{p}(n) \| \mathbf{x}(n) \|^{2} \\
    e_{p}(n)    &= e(n) \frac{1}{1 + \mu \| \mathbf{x}(n) \|^{2}}
                % &= e(n) \bigg[ \frac{1 + \mu \| \mathbf{x}(n) \|^{2} - \mu \| \mathbf{x}(n) \|^{2} }{1 + \mu \| \mathbf{x}(n) \|^{2}} \bigg] \\
                % &= e(n) \bigg[ 1 - \mu \frac{\| \mathbf{x}(n) \|^{2}}{1 + \mu \| \mathbf{x}(n) \|^{2}} \bigg]
\label{eq:aa_errors}
\end{align}

Substituting (\ref{eq:aa_errors}) in update equation (\ref{eq:nlms_a_error}):

\begin{align}
    \vw(n + 1)  &= \vw(n) + \mu \frac{1}{1 + \mu \| \mathbf{x}(n) \|^{2}} e(n) \mathbf{x}(n) \\
    \vw(n + 1)  &= \vw(n) + \frac{1}{\frac{1}{\mu} + \| \mathbf{x}(n) \|^{2}} e(n) \mathbf{x}(n) \\
    \vw(n + 1)  &= \vw(n) + \frac{\beta}{\epsilon + \| \mathbf{x}(n) \|^{2}} e(n) \mathbf{x}(n) \label{eq:nlms}
\end{align}

where for $\beta = 1$ and $\epsilon = \frac{1}{\mu}$ we showed that the NLMS update (\ref{eq:nlms}) is equivalent to the update equation based on the \textit{a posteriori error},
given by (\ref{eq:nlms_a_error}).

%% c)
\item
%

The implementation of Generalized Nnormalized Gradient Descent (GNGD) algorithm is compared with the the Benvenist GASS algorithm, and the weight error and squared prediction error curves
are provided at figure \ref{fig:3_2_c}. We notice that the GNGD algorithm converges to the true process parameters within 30 timesteps,
faster than the Benvenist GASS algorithm which reaches a steady-state in 50 timesteps. This is also reflected in the squared prediction error curves,
where the GNGD error is always smaller than the Benvenist GASS prediction error.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-step-sizes/assets/c/weight_error_curves}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/adaptive-signal-processing/adaptive-step-sizes/assets/c/squared_prediction_error}
    \end{subfigure}
    \caption{GNGD vs Benvenist GASS: weight error and squared prediction error curves.}
    \label{fig:3_2_c}
\end{figure}

In the previous part we empirically showed the superiority of the Benvenist GASS algorithm over the other GASS algorithms and the standard LMS, in terms of performance (speed \& prediction error).
Nonetheless, this improved performance comes with an increased computational complexity load.

Let $M$ the model order (number of lags in MA or AR model process), then the input vector $\mathbf{x}(n) \in \sR^{M}$.

Each update of the Benvenist GASS algorithm involves the calculation of
\begin{itemize}
    \item the outer product $\mathbf{x}(n-1) \mathbf{x}(n-1)^{T}$
    \item the matrix product $\big[I - \mu(n-1) \mathbf{x}(n-1) \mathbf{x}(n-1)^{T} \big] \boldsymbol{\psi}(n - 1)$
\end{itemize}

which both have quadratic complexity in $M$. Hence Benvenist GASS algorithm is $\mathbf{\mathcal{O}(M^{2})}$.

Each update of the GNGD algorithm involves only inner product calculations, additions of $M$-dimensional vectors and scalar operations, all bounded by linear complexity in $M$.
Therefore, GNGD algorithm is $\mathbf{\mathcal{O}(M)}$.

Surprisingly, the GNGD algorithm does not only perform better (convergence speed and prediction error) than the Benvenist GASS algorithm, but it is also computationally less expensive.

%
\end{enumerate}


%!TEX root = TTK4215-Summary.tex
\section{Extremum seeking}
The basic idea: Want to find the optimal plant input $\theta^*$ that maximises the output $y$. Add a slow periodic perturbation to our estimate $\hat{\theta}$. If increasing $\theta$ increases $y$, then $y$ will oscillate in phase with $\theta$. In the opposite case, $y$ with be out of phase with $\theta$. The DC component of $y$ is removed with a high-pass filter, and the result multiplied with the perturbation signal. The product will have a positive DC component if $y$ and $\theta$ are in phase, and negative DC if they are out of phase. This DC component is extracted with a low-pass filter, and is then a value indicating how far off $\theta$ is, and in what direction. This is integrated and multiplied with a gain $k$ to form the estimate $\hat{\theta}$. Pretty clever.

\begin{figure}[htbp]
\begin{center}
	\includegraphics[width = \textwidth]{extremum-seeking}
	\caption{Block diagram for extremum seeking}
	\label{fig:MRAC}
\end{center}
\end{figure}

TTK4215 System identification and adaptive control/sec-extremum.tex


One key issue posed from the consensus mechanism described in \cite{TWP} is the propagation and distribution of information across producers and eventually across the entire network. This is particularly an issue for the distribution of the list of producers   that a specific producer considers to have created the correct ledger state update. With larger producer pool which are necessary for security consideration, large lists of information are impractical for distribution. When it it is considered the every PID is 32Bytes long, for a group of 1000 producers lists would reach 32KBytes if the raw lists of producers are distributed. While compression can be used, what is gained in size of the elements is lost in time for compression and decompression of the lists. This is especially true when it is considered that every producer could potentially have to decompress 999 lists from a pool of 1000 producers. Furthermore using a User Datagram Protocol (UDP) as used in Catalyst, the potential for lost information on elements of that size is high. Therefore a more compact and efficient method is required. Bloom filters \\

Bloom filters are data structures that allow proof of membership of a set. They work is such a way that they can never provide a false negative i.e. showing that an element is in fact in the bloom filter as false. They can however show false positives meaning that an element that is not in the bloom filter can in fact come back true. Bloom filters are simple byte arrays of a specified length. Upon initialisation all bits within the byte array are set to 0. For each element to be added to the bloom filter it is hashed a defined number of times and the corresponding bit for the hash is flipped to a 1. For example for a small bit array \verb'0bx0000' and element \verb'Hello' is hashed to give the digest \verb'04' then the $4^{th}$ bit would be flipped so the bloom filter would become \verb'0bx1000'. This works on a modulo point, so if the digest was \verb'05' then the first bit would be flipped giving the bloom filter \verb'0bx0001'. For each element added to the bloom filter multiple hashes are created and added to the bloom filter, for example if 3 hashing functions are used 3 bits in total will be flipped in the bloom filter and these 3 flipped bits will represent the element. \\

Bloom filters are made up of four key variables, these are:

\begin{itemize}
\item $m$ - The size of the bloom filter in bits.
\item $n$ - The number of elements to be added to the bloom filter.
\item $k$ - The number of hashing functions that are performed on each element added to the bloom filter.
\item $p$ - Probability of a false positive occurring for an element within the bloom filter.
\end{itemize}

The size of a bloom filter can be defined by:

\begin{center}
$m = -(n \times log(p)) \div (log(2)^2)$
\end{center}

The optimum hash count of a bloom filter can be defined by:

\begin{center}
$k = (m \div n) \times log(2)$
\end{center}

The false positive rate for a bloom filter is:

\begin{center}
$p = (1 - e^{-kn \div m})^k$
\end{center}

The number of elements added to the bloom filter is uniform across all producers as this must be set to the total number of producers for one cycle. Each of these variables can be changed in order to form a bloom filter for whatever accuracy, size or amount of producers it is required.\\

The reason for choosing a bloom filter over directly transferring the lists is for size considerations. Distributing a list containing 100 or a 1000 elements will be extremely cumbersome due to the size. While a bloom filter for 1000 elements would be approximately 1.4kBytes in size. Furthermore, due to the goal of the list creation is to efficiently deduce the members of the worker pool that created the correct delta and that bloom filters are an efficient method for membership validation the use of bloom filters lends itself to the Catalyst consensus algorithm.

\subsection{Statistics}

Using the script in \cite{python} we can determine the probability of failure for any one particular node. Probability of failure is defined as the probability of any one node failing to hold a complete bloom filter of all producers who successfully produced the correct ledger state update. The process for a producer is such that, they produce a bloom filter of all producer they retrieved correct proposed ledger state updates at the beginning of the campaigning phase. This bloom filter is then passed of to  all other producers. However, due to natural inefficiencies it is unlikely all producers will receive all of the proposed updates therefore, the correct list must be constructed from all the received bloom filters. It is this created list for each producer that we will test. We analyse two variables, firstly is the message propagation ratio, this is the percentage of messages that we assume each producer recieves from the rest of the network, i.e. at a message propagation ratio of 0.75, for a producer pool of 100, each producer would receive randomly 75 other producers bloom filters. The second variable we will be testing is the size of the producer pool.


\lab{Algorithms}{Temporal Complexity and Sparse Matrices}{Complexity and Sparse Matrices}

\objective{Introduce the concept of temporal complexity and explore SciPy's special methods for working with sparse matrices.}

\section*{Temporal Complexity}
One of the most important questions in scientific computing is ``How long will this operation take?"
The concept of temporal complexity attempts to answer this question by determining how much time a function needs to operate on a given size of input.
For example, suppose a particular algorithm operating on a matrix of size $n$ (that is, an $n \times n$ matrix) requires the following number of calculations:

\begin{equation*}
f(n) = \frac{3n^3}{2} + 75n^2 + 250n + 30
\end{equation*}

As the input gets larger, the growth of $f(n)$ is dominated by the $n^3$ term.
For this reason we say that $f(n) \in O(n^3)$, or more commonly, that $f(n)$ is $O(n^3)$ (spoken ``Big O of n cubed'' or ``order of n cubed").
This notation is borrowed from analysis and captures the most important behavior of our temporal complexity. It describes the expected growth rate of the execution time of our algorithm. We also refer to this as \emph{asymptotic notation}.
We will discuss this concept later, but this is a simple introduction to the notion of complexity and Big O.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{complexitycurves.pdf}
\caption{Some common asymptotic curves.}
\end{figure}

\begin{definition}[Temporal Complexity]
Let $n$ represent the problem size.  Let $f(n)$ and $g(n)$ be functions mapping natural numbers to positive real numbers. We say that $f \in O(g(n))$ if there exists a real number $c > 0$ and a natural number $n_0$ such that for all natural numbers $n > n_0$, $f(n) \leq cg(n)$.
When this is satisfied, we say that $g(n)$ is an asymptotic upper bound for $f(n)$.
The notation $f = O(g(n))$ is also common.
\end{definition}

What does this have to do with computer programs?
Every algorithm executed on a computer corresponds to a function
that returns the number of steps taken (and therefore the time)
given an input of size $n$.  Certain problems can be solved with fewer steps,while others require many more.

For example, given two matrices of size $n$, matrix addition is $O(n^2)$. This is because it takes approximately $n^2$ steps to add $n^2$ elements for our $n \times n$ arrays.
Comparatively, calculating the inverse of a matrix using Gaussian
row reduction is $O(n^3)$. (There are in fact more efficient
algorithms for matrix inversion.)

How do we determine the temporal complexity of a piece of code?
Calculating the exact temporal complexity of a formula is fairly difficult.
However, to asymptotically analyze a function is fairly straightforward.

\begin{lstlisting}
s = 0
for i in xrange(100):
    s = s + i
\end{lstlisting}

The code above is $O(n)$ because it takes approximately $n$ steps to complete. For-loops are a good indicator of the complexity.  A double for-loop strongly suggests $O(n^2)$ or worse.  Asymptotic analysis reveals that
if we have an array of size $n \times n$ and it takes $x$ seconds to execute an $O(n^2)$ algorithm,
then by increasing the size of the array to $2n \times 2n$, we should expect the same algorithm
to terminate in about $(2)^2 x = 4x$ seconds.  Sometimes, when you don't have code to look at,
you can approximate the asymptotic growth of a function by timing it on various sized inputs.

\begin{problem}
Time the runtime of the following code for \li{n = 1000, 2000, 4000, 8000}.

\lstinputlisting[style=fromfile]{test.py}

Now write a function that takes no arguments and does the following:
\begin{enumerate}
\item Plots the four runtimes (use \li{[1000, 2000, 4000, 8000]} or an equivalent array object as your domain).
The plot should look like Figure \ref{prob1} if using the \li{plt.scatter} command.
\item Returns the average ratio between successive runtimes (one such ratio would be the runtime for $n = 2000$ divided by the runtime for $n = 1000$).
\end{enumerate}
\end{problem}

\begin{figure}
\centering
\includegraphics[width=\textwidth]{prob1.pdf}
\caption{The plot of problem 1}
\label{prob1}
\end{figure}

We can also classify an algorithm by the amount of memory it needs to execute.  We will use the term \emph{spatial complexity} to measure the amount of memory an algorithm uses.  The definition of spatial complexity is the same as the definition of temporal complexity, with the obvious difference that $n$ refers to space rather than time.
In practice, spatial and temporal complexity are not completely independent of each other. The amount of memory required by an algorithm can affect its speed in several ways. The most important consideration is when the memory usage exceeds the amount of available RAM. When this occurs, the machine must use the hard disk or some other slower storage method. Doing so makes the read and write operations become substantially slower.

\section*{Sparse Matrices}
A sparse matrix is a matrix that has relatively few nonzero elements.
SciPy has several different ways to store sparse matrices. Each way has its own benefits and the reader is encouraged to do additional research. It is important to note that SciPy's \li{sparse} methods refer to the way a matrix is stored and not the actual sparsity of the matrix in question; we can store a dense matrix using SciPy's \li{sparse} methods, but we will not see efficiency gains as we do when we use these methods on matrices that are actually sparse.

\begin{table}
\centering
\begin{tabular}{|l|l|}
\hline
Function & Description \\
\hline
\li{sparse.bsr_matrix()} & Compressed Block Sparse Row\\
\li{sparse.coo_matrix()} & Coordinate\\
\li{sparse.csc_matrix()} & Compressed Sparse Column\\
\li{sparse.csr_matrix()} & Compress Sparse Row\\
\li{sparse.dia_matrix()} & Sparse Diagonal\\
\li{sparse.dok_matrix()} & Dictionary of Keys\\
\li{sparse.lil_matrix()} & Linked List\\
\hline
\end{tabular}
\caption{Sparse matrix representations in SciPy}
\label{smr}
\end{table}

The following showcases some of SciPy's sparse matrix representation functions listed in Table \ref{smr}

\begin{lstlisting}
>>> import numpy as np
>>> from scipy import sparse

>>> A = np.diagflat([2, 3, 4])
>>> A
array([[2, 0, 0],
       [0, 3, 0],
       [0, 0, 4]])

>>> B = sparse.csc_matrix(A)
>>> B
<3x3 sparse matrix of type '<type 'numpy.int64'>'
	with 3 stored elements in Compressed Sparse Column format>

# Convert B to a storage in a full matrix format
>>> C = B.todense()
>>> C
matrix([[2, 0, 0],
        [0, 3, 0],
        [0, 0, 4]])

\end{lstlisting}

Notice that the matrix $A$ has only three non-zero entries so we consider it to be a sparse matrix.
In memory, an array stores a piece of data (be it an integer, float, or complex number) in each entry. This means that a $3 \times 3$ matrix requires a total 9 blocks of memory.
However, if we leverage the sparsity of $A$ we realize that we really only need to store 3 numbers.
The \li{sparse} methods do exactly this: they store only the non-zero entries and their locations in the matrix.
An entire array is no longer being stored, decreasing complexity.

SciPy has many methods for performing operations on sparse matrices. For example, to convert back to a dense matrix we use the \li{todense()} method (of the sparse matrix). We can also convert between the different types of sparse matrices.

Note that if you want to make a sparse diagonal matrix, the best way to do so isn't by creating a diagonal matrix using \li{diagflat()} and then making it sparse by using \li{sparse}. Instead it is much better to use the \li{sparse.spdiags()} method like so:

\begin{lstlisting}
# sparse.spdiags(data, diags, m, n)
sparse.spdiags([2, 3, 4], 0, 3, 3)
\end{lstlisting}

Note that \li{diags=0} sets our data along the main diagonal for an $m \times n$ matrix. If \li{diags} $> 0$ then our data is set along the indicated upper diagonal. Similarly, if \li{diags} $< 0$ then our data is set along the indicated lower diagonal.

Often, when we are using sparse matrices it is because we are dealing with matrices that are otherwise too large to be handled efficiently when represented in full form.

%insert problem here that lets them play around with large sparse matrices in diff forms

\section*{Banded Matrices}
A banded matrix is one whose only non-zero entries are diagonal
strips.  For example, the matrix
\begin{equation*}
A = \begin{pmatrix}
1 & 2 & 0 & 0 \\
3 & 4 & 5 & 0 \\
0 & 6 & 7 & 8 \\
0 & 0 & 9 & 10
\end{pmatrix}
\end{equation*}
is banded because the only non-zero entries are located in three nonzero diagonals.  This particular type of banded matrix is called a tri-diagonal matrix.

Banded matrices are easily created using the \li{diagflat()} method.
For example, the matrix $A$ from above can be created by doing:

\begin{lstlisting}
# np.diagflat(data, k=0)
>>>np.diagflat([3, 6, 9], -1) + np.diagflat([1, 4, 7, 10], 0) + np.diagflat([2, 5, 8], 1)
array([[ 1,  2,  0,  0],
       [ 3,  4,  5,  0],
       [ 0,  6,  7,  8],
       [ 0,  0,  9, 10]])

\end{lstlisting}
Note that \li{data} must be a flattened array and the keyword argument \li{k} is an integer that determines which diagonal is set with the given data.

However, creating this tri-diagonal required using \li{np.diagflat} three times. Often, a better way to create a tri-diagonal is it use the \li{sparse.spdiags()} method. %Note that
This is because many banded matrices are sparse.

For example, we can create the same matrix from above using the command:
\begin{lstlisting}
>>> Z = np.array([[3,6,9,0],[1,4,7,10],[0,2,5,8]])
>>> sparse.spdiags(Z, [-1, 0, 1], 4, 4)
<4x4 sparse matrix of type '<type 'numpy.int64'>'
	with 10 stored elements (3 diagonals) in DIAgonal format>

\end{lstlisting}

Note that the diagonals are all the same length, unlike when we used the \li{np.diagflat} to construct this matrix. For more information, check the documentation by typing \li{sparse.spdiags?}.

\begin{problem}
Write a function that takes an integer argument \li{n} and returns a full $n\times n$
tri-diagonal array with $2$'s along the diagonal and $-1$'s along
the two sub-diagonals above and below the diagonal.

\emph{Helpful Hint}: Use the \li{np.diagflat()} command.
\label{full_tridiag}
\end{problem}

We will now create a tri-diagonal array with uniformly distributed random entries. This example also demonstrates the efficiency of using sparse arrays.

\begin{lstlisting}
>>>B = np.random.rand(3, 10000)

>>> A = sparse.spdiags(B, range(-1, 2), 10000, 10000)

# Only do this if you have several gigabytes of memory
>>> denseA = A.todense()

>>> A.data.nbytes # About .24MB of memory.
240000

>>> denseA.nbytes # About 762.9MB of memory.
800000000

\end{lstlisting}

Even when a complete matrix is too large to represent in memory,
we can still visualize it using the \li{plt.spy()} command from Matplotlib,
which essentially shows the location of non-zero entries in a matrix.
The output of \li{plt.spy(A)} in this case is shown in Figure \ref{fig:mpl_spy}.

\begin{figure}
\centering
\includegraphics[width=\textwidth]{spy.png}
\caption{The output of the \li{spy()} command}
\label{fig:mpl_spy}
\end{figure}

\begin{problem}
Write a function that accepts an integer argument \li{n} and returns the same array as Problem \ref{full_tridiag}, but as a sparse array.
You must build this as a sparse matrix from the beginning.

\emph{Helpful Hint}: Use the \li{sparse.spdiags()} method. \li{format} should be set to \li{'csr'}.
\label{prob:sparse_tridiag}
\end{problem}

\section*{Using Sparse Matrices}
Consider the linear system $A x = b$, where $A$ is a $100000\times 100000$ tri-diagonal matrix.
Storing a full matrix of that size would normally require 10 billion double-precision floating-point numbers.  Since it takes 8 bytes to store a double, it would take roughly 80GB to store the full matrix.  For most desktop computers, that fact alone makes the system numerically prohibitive to solve.
The temporal complexity of this problem is even more problematic. Methods for directly solving an arbitrary linear system are usually $O(n^3)$. As a result, even if the computer could store an 80GB matrix in RAM, it would still take several weeks to solve the system. However, since we don't typically have computers with that much available RAM, most of the
matrix would have to be stored on the hard drive, so the computation would probably take between $6$ months to a year.

The point is, even as computers increase in processing speed and memory, we can still easily construct problems that they will struggle to solve in a reasonable amount of time. However, if we take advantage of the sparse structure of the tri-diagonal matrix, we can solve the linear system, even with a modest modern computer.  This is because all of those zeros don't need to be stored and we don't need to do as many operations to row reduce the tri-diagonal system. Note that you should never solve a sparse system by inverting the matrix, since the inverse of a sparse matrix will most likely be not sparse and therefore computationally unwieldy.

Let's first compute the spatial complexity of the above system when considered as a sparse matrix.  There are three diagonals that have roughly $100000$ non-zero entries.  That's $300000$
double-precision floating point numbers, which is about 2.4 MB (Less storage than your favorite song).  As a result, it will easily fit into the computer's RAM.  Furthermore, the temporal complexity for solving a tri-diagonal matrix is $O(n)$. Let's see how long it takes to solve the system for random data:

\begin{lstlisting}
# Note that we need to import sparse.linalg as its own module instead of simply using the command sparse.linalg.spdiags
from scipy.sparse import linalg as sl
D = np.random.rand(3, 100000)
b = np.random.rand(1, 100000)
A = sparse.spdiags(D,[-1,0,1],100000,100000, format='csr')
def solSys():
    return sl.spsolve(A, b)

%timeit solSys()

\end{lstlisting}

\begin{problem}
Write a function that accepts an integer argument \li{n} as well as a keyword argument \li{sparse} whose value is either \li{True} or \li{False} (default to \li{False}). Then do the following:
\begin{enumerate}
\item Inside of the function, use your previous solutions to generate an $n \times n$ tri-diagonal array $A$ -- either sparse or full depending on the value of the \li{sparse} argument.
\item Generate an $n \times 1$ random array $b$
\item Solve the system $Ax = b$, using either \li{scipy.sparse.linalg.spsolve} or \li{scipy.linalg.solve}
(again depending on the value of \li{sparse}) and return the solution.
\item Time the function for \li{n = 2000} using both the sparse and the full option.
\end{enumerate}
\end{problem}

\begin{problem}
Write a function that accepts an integer argument \li{n} and returns $\lambda n^2$, where
$\lambda$ is the smallest eigenvalue of the same $n \times n$ sparse tri-diagonal array as described above.

To calculate $\lambda$, use \li{scipy.sparse.linalg.eigs}. You will need to set the \li{which = 'SM'}
optional argument. If \li{A} is your tri-diagonal sparse matrix, you will need to use
\li{A.asfptype()} as the argument for the \li{eigs} function (so that the matrix has the right data type).
Check the documentation of the \li{eigs} function to see what it returns, so that you will know how to
get the smallest eigenvalue.

What value does $\lambda n^2$ approach as $n \rightarrow \infty$?

\li{Helpful Hint}: It's the square of an important number.
This is related to operator theory: the second derivative operator has this eigenvalue in certain cases.
\end{problem}

\section*{Other Sparse Commands}
One important method of sparse matrix objects is the \li{nonzero()} method, which is related to the number of nonzero entries in the matrix. This number is important because it is an indicator of the amount of time and space that is required to operate on the sparse matrix.
You should be aware that there is some overhead to using and storing the sparse matrix data structure. Sparsely represented matrices are very beneficial when the number of nonzero entries is relatively small compared to the total number of entries.
When the matrix has many nonzero entries, a sparse representation becomes disadvantageous.
To see this, observe:

\begin{lstlisting}
>>> A = np.random.rand(600, 600)

>>> B = sparse.csc_matrix(A)

>>> def square(A):
>>>     return np.power(A, 2)

>>> %timeit square(A)
100 loops, best of 3: 9.53 ms per loop

>>> %timeit square(B)
1 loops, best of 3: 941 ms per loop
\end{lstlisting}

Notice that it takes much longer to square the sparse matrix.
This is because the sparse matrix data structure is optimized for matrices that are actually sparse. The array $A$ is entirely nonzero. Thus, you incur the overhead of the sparse array representation without any benefits since there are no entries you are not required to store or compute.

To summarize, only use a sparse matrix when your matrix is in fact sparse. Using sparse matrices for mostly nonzero arrays will negatively impact performance and memory requirements.

Just as with dense arrays, we can pre-allocate sparse matrices.
Sometimes it is necessary to create sparse matrices that do not have a nice banded pattern. We initialize a sparse matrix just like any other array. The most efficient sparse matrix for pre-allocation is \li{LIL}. Once you are done constructing the sparse matrix and wish to perform calculations, you should convert to a more efficient sparse matrix (CSR or CSC).

\begin{lstlisting}
>>> Z = sparse.lil_matrix((400, 300))

>>> Z[1,34] = 23

>>> Z[23,32] = 56

>>> Z[2,:] = 13.2

>>> Z
<400x300 sparse matrix of type '<type 'numpy.float64'>'
	with 302 stored elements in LInked List format>

\end{lstlisting}

When the matrix \li{Z} from above is initialized, all entries of the $400 \times 300$ \li{LIL} sparse matrix are assumed to be zero.
We can then work with the sparse matrix as though it were a dense array.
Note that only 302 elements are being stored for a matrix with 120000 possible entries.

\begin{problem}
Create a $500\times 500$ matrix and vector of length 500, both full of random values. Use the \li{A.dot(b)} command to multiply your matrix and your vector, and time how long it takes to do so. Then convert your matrix to sparse format and again time how long it takes to multiply it by your vector using \li{A.dot(b)}.
\end{problem}


\section{Poisson Processes}
\label{sec:Poisson-Processes}

The Poisson process is the most widely used model for arrivals into a system because it is often a realistic representation of natural models and, since it exposes the Markovian property, it is analytically tractable.
It greatly models natural process where we observe aggregation of large number of independent entities \footnote{This is true thanks to the Limiting Theorem \cite{ross2014introduction}}.

The Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time and/or space if these events occur with a known average rate and independently of the time since the last event. If inter-arrival times are Exponentially distributed, then the number of arrivals within a given time is Poisson distributed.

\begin{definition}[Poisson Distribution]
\label{def:Poisson-Distribution}
	A random variable $N(t)$ is Poisson distributed with rate\footnote{if $N(t) \sim Poisson(\lambda)$, $\lambda$ is called \textit{rate} because $\expected{N(t)}=\lambda t$.} $\lambda$ ($N(t) \sim Poisson(\lambda)$) if $N(t)$ has the following probability mass function:

	\begin{equation}
	\label{eqn:Poisson-PMF}
	f(k,t) = \frac{(\lambda t)^{k} e^{-\lambda t}}{k!}
	\end{equation}
\end{definition}

The p.d.f of $X \sim Poisson(\lambda)$ is shown in \Cref{fig:poisson-pdf}.

\begin{figure}[tp]
\label{fig:Poisson-PMF}
	\centering
	\includegraphics{fig/poisson-pdf}
	\caption{Poisson p.m.f.}
\end{figure}

From the \Cref{def:Poisson-Distribution} follows that the cumulative distribution function of $N(t) \sim Poisson(\lambda)$ is

\begin{equation}
\label{eqn:Poisson-CDF}
F(k,t) = \sum_{x=0}^{t} f(k,x)
\end{equation}

The Poisson distribution has mean

\begin{equation}
\label{eqn:Poisson-Mean}
\expected{N(t)} = \lambda t
\end{equation}

and variance

\begin{equation}
\label{eqn:Poisson-Variance}
\variance{N(t)} = \lambda t
\end{equation}

\begin{definition}[Poisson Process]
\label{def:poisson-process-statistical}
	A Poisson process with rate $\lambda$ is a sequence of events such that
	(i) $N(0)=0$,
	(ii) it has independent increments,
	(iii) $\probability{N(s+t)-N(s) = n} \sim Poisson(\lambda t)$
\end{definition}

Notice that point (iii) in \Cref{def:poisson-process-statistical} implies stationary increments.
Furthermore, the assumption of independent and stationary increments implies that, at any point in time, the process statistically restarts itself; that is, the process from any point onward is independent of all previously occurred (independent increments) and has the same distribution of the original process (stationary increments).

\begin{definition}[Poisson Process]
\label{def:poisson-process-practical}
	A Poisson process with rate $\lambda$ is a sequence of events such that
	(i) the interarrival times are Exponential r.v. with rate $\lambda$, and
	(ii) $N(0)=0$.
\end{definition}

A Poisson process has stationary and independent increments.

Here are some useful properties of the Poisson distribution.

\begin{theorem}[Poisson Merging]
\label{thm:Poisson-Merging}
	Given two independent Poisson process with rate $\lambda_{1}$ and $\lambda_{2}$, the merged process is a Poisson process with rate $(\lambda_{1} + \lambda_{2})$.
\end{theorem}

\begin{theorem}[Poisson Splitting]
\label{thm:Poisson-Splitting}
	Given a Poisson process with rate $\lambda$, whose events are partitioned in class-A with probability $p$ and class-B with probability $(1-p)$, the class-A process is a Poisson process with rate $p \lambda$ and the class-B process is a Poisson process with rate $(1-p) \lambda$, and these processes are independent.
\end{theorem}

\begin{theorem}[Poisson Uniformity - Single Event]
\label{thm:Poisson-Uniformity-Single-Event}
	Given that one event of a Poisson process has occurred by time $t$, that event is equally likely to have occurred anywhere in $[0,t]$.
\end{theorem}

\begin{theorem}[Poisson Uniformity - Multiple Events]
\label{thm:Poisson-Uniformity-Multiple-Events}
	If $k$ events of a Poisson process occur by time $t$, then the $k$ events are distributed independently and uniformly in $[0,t]$.
\end{theorem}