\chapter{Exploratory Data Analysis}
\chaptermark{EDA}
\label{sec:exploratory}

In this chapter, we give a short review of methods for \emph{exploratory data analysis} (EDA), \aka \emph{descriptive statistics}.\marginnote{Descriptive Statistics}
Recall that our goal is an assumptions-free description of our data.
EDA thus consist of computing interpretable summaries of the data, called \emph{summary statistics}, and visualizations.

\section{Summary Statistics}
\label{sec:summary_statistics}

We now distinguish between summary statistics that apply to attributes, categorical by definition, and variables, continuous by definition.

\subsection{Summarizing Categorical Data}

\subsubsection{Univariate}
Summarizing a vector of categorical data can naturally be done by tabulating it, i.e., computing the frequency and relative frequency of each category.
Clearly averages, medians, and the likes are incomputable, since categorical data has no ordering, nor does it admit simple operations such as summation.

\begin{extra}
Variability of categorical data can clearly not be measured by its variance, since it does not admit a summation operation.
It is, however, possible to define different measures of variability that do apply.
The \emph{entropy} is such an example.\marginnote{Entropy}
\end{extra}

\subsubsection{Bivariate}
Generalizing the univariate case to bivariate, or multivariate, one can keep tabulating. I.e., compute the frequency, and relative frequency, of combinations of categories.

\subsection{Summarizing  Continuous Data}
Continuous variables admit many more mathematical manipulations than categorical attributes.

\subsubsection{Univariate}

We start by presenting the most natural summaries of the data. Without going into the formal definition, we refer to them as \emph{summary of location}.\marginnote{Location Summaries}
These include:

\begin{definition}[The Mean]
The \emph{mean}, or \emph{average}, is defined as
\begin{align}
	\bar{x}:= \frac{1}{n}\sum_{i=1}^{n} x_i
\end{align}
\end{definition}

\begin{definition}[The Median]
The median is the observation that is smaller than half of the sample and larger than half of the sample.
\end{definition}

\begin{definition}[$\alpha$-Trimmed Mean]
The $\alpha$-trimmed mean is the average of the observations left after ignoring the largest and the smallest $(100\alpha) \%$ of them.
\end{definition}
The \naive average is the $0$-trimmed mean, and the median is the $0.5$-trimmed mean.

From summaries of location, we move to summaries of \emph{scale}. \marginnote{Summary of Scale}

\begin{definition}[The Standard Deviation]
\begin{align}
	s(x):= \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i-\bar{x})^2}.
\end{align}
\end{definition}

For the following, we require the definition of the sample quantiles, themselves \textbf{not} a scale summary.

\begin{definition}[$\alpha$ Quantile]
The $\alpha$-quantile of a sample $x=(x_1,\dots,x_n)$, denoted $x_\alpha$, is the observation that is larger than $(100\alpha)\%$, and smaller then  $(100(1-\alpha))\%$ of the sample.
\end{definition}
The empirical maximum and minimum are then $x_{1.0}$ and $x_{0.0}$, respectively.

\begin{definition}[The Range]
\label{def:range}
\begin{align}
	\range{x}:= \max_i\set{x_i}-\min_i\set{x_i}= x_{1.0}-x_{0.0}.
\end{align}
\end{definition}

\begin{definition}[The Inter Quantile Range- IQR]
\label{def:iqr}
\begin{align}
	IQR(x):= x_{0.75}-x_{0.25}.
\end{align}
\end{definition}

\begin{definition}[The Median Absolute Deviation- MAD]
\label{def:mad}
\begin{align}
	MAD(x):= c \: \set{|x-x_{0.5}|}_{0.5}.
\end{align}
where $c$ is some constant.
In the \R function \rcode{mad()}, $c$ is set at $1.4826$ so that it estimates $\sigma$ in a Gaussian population.
\end{definition}

After summaries of scale, we move to summaries of \emph{skewness}, or \emph{asymmetry}.

\begin{definition}[Yule Skewness Measure]
\begin{align}
	YULE(x):= \frac{\frac{1}{2} (x_{0.75}+x_{0.25})-x_{0.5}}{\frac 12IQR(x)}
\end{align}
\end{definition}

\subsubsection{Bivariate}
From univariate data $x$, we move to bivariate $x,y$.
Clearly we can apply univariate summaries component-wise.
We want, however, to summarize the \emph{joint} behaviour of the data.
For this purpose, we assume that data comes in pairs, implying that $x$ and $y$ are of same length.

\begin{definition}[Covariance]
The sample covariance, or \emph{empirical} covariance is defined as
\begin{align}
	Cov(x,y):= \frac{\sum_{i=1}^{n} (x_i-\bar{x})(y_i-\bar{y})}{n-1}
\end{align}
\end{definition}

\begin{definition}[Pearson's Correlation Coefficient]
\emph{Pearson’s Correlation Coefficient}, or \emph{Pearson's Moment Product Correlation Coefficient}, is defined as
\begin{align}
	r(x,y):= \frac{(n-1) Cov(x,y)}{S(x) S(y)}= \frac{\sum_{i=1}^{n} (x_i-\bar{x})(y_i-\bar{y})}{S(x) S(y)}
\end{align}
\end{definition}
We can dwell into the meaning and intuition underlying Pearson's correlation coefficient, but we will not.
The curious reader is reffered to \cite{rodgers_thirteen_1988}.

The next measure of association captures a more general association.
\begin{definition}[Spearman's Correlation Coefficient]
Spearman's correlation coefficient is merely Pearson's correlation coefficient computed on the \emph{ranks} of $x$ and $y$.
\end{definition}

We conclude by noting that \emph{regression coefficients} are also a measure of association.

\subsubsection{Multivariate Data}
Multivariate data, both continuous (variables), and discrete (attributes), admits a vast realm of method for summary and visualization.
Clearly, associations between several variables can be very complicated so that the more we try to summarize, the more information we give up. On the other hand, and unlike the univariate and bivariate case, our minds will need some type of simplification since they cannot grasp the raw data (did you ever try to imagine how $\mathbb{R}^4$ looks like?).
As usual, we emphasize that our purpose is to summarize the joint association in the data.
For component-wise summaries, we can always apply the univariate summaries one variable at a time.

By far the most popular measures of joint association are the covariance matrix and correlation matrix.

\begin{definition}[Covariance Matrix]
\label{def:covariance}
For multivariate data consisting of $x_1,\dots,x_j,\dots,x_p$ vectors, each with $n$ entries: $x_{j,1},\dots,x_{j,n}$, we define the (sample) covariance matrix to be a $p\times p$ matrix whose elements are the (sample) covariances between corresponding vectors:
\begin{align}
	\hat{\Sigma}_{k,l}:= Cov(x_k, x_l).
\end{align}
\end{definition}

\begin{extra}[Sample Covariance Matrix]
The matrix $\hat{\Sigma}$ has many useful properties.
The curious reader is referred to \cite{petersen_matrix_2006}, and references therein, for more details.
\end{extra}

\begin{definition}[Correlation Matrix]
For multivariate data consisting of $x_1,\dots,x_j,\dots,x_p$ vectors, each with $n$ entries: $x_{j,1},\dots,x_{j,n}$, we define the (sample) correlation matrix to be a $p\times p$ matrix whose elements are the (Pearson) correlations between corresponding vectors:
\begin{align}
	\hat{R}_{k,l}:= r(x_k, x_l)
\end{align}
\end{definition}

\begin{extra}[Multivariate Data Analysis]
Multivariate analysis is an important, and very actively studied field in statistics and machine learning.
A non-comprehensive list of methods that belong to this realm include
Principal Component Analysis (PCA),\marginnote{PCA, SVD,ICA}
Factor Analysis (FA),
Independent Component Analysis (ICA),
Graphical Models,
Dimensionality Reduction,
Manifold Learning,
Self Organizing Maps,
etc.
Ask me for reference books or courses if this topic interests you.
\end{extra}

\afterpage{\clearpage}

\section{Visualization}
\label{sec:visualizations}

\subsection{Visualizing Categorical Data}

\subsubsection{Univariate}
Much like computing summaries, there is not much to be said about visualizing univariate categorical variables.
The most natural, and perhaps only visualization, is the \emph{bar plot}, illustrated in Figure~\ref{fig:barplot}.

\begin{remark}[Pie Chart]
About those pie charts.
There is really no reason to use them. Ever\footnote{\url{http://www.businessinsider.com/pie-charts-are-the-worst-2013-6}.}.
\end{remark}

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight]{art/categorical-data1x}
\caption[Bar Plot]{The Bar-Plot. \newline
\url{http://www.r-tutor.com/elementary-statistics/qualitative-data/bar-graph}}
\label{fig:barplot}
\end{figure}

\subsubsection{Bivariate}
Visualizing a two-way cross-table can be done using an extension of the bar-plot.
Several extensions exist. By far, the most informative and recommended figure, in this author's view, is the \emph{mosaic plot}, illustrated in Figure~\ref{fig:mosaic}. \marginnote{Mosaic Plot}

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight]{art/mosaic1}
\caption[Mosaic Plot]{Mosaic Plot. \newline
\url{http://www.statmethods.net/advgraphs/mosaic.html}}
\label{fig:mosaic}
\end{figure}

\afterpage{\clearpage}

\subsection{Visualizing Continuous Data}

\subsubsection{Univariate}
Visualization of univariate continuous vectors can present the raw data, or it distribution (i.e.- discarding the indexes).
The most basic visualizations are the \emph{dotchart}, \emph{histogram}, \emph{boxplot}, \emph{stem-and-leaf plot}.
These are illustrated in figures \ref{fig:dot_plot}, \ref{fig:histogram_eruptions}, \ref{fig:boxplot}, \ref{fig:stem_and_leaf} respectively.

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight]{art/lmCm0}
\caption[Dot Plot]{Dot Plot. \newline \url{http://stackoverflow.com/questions/15109822/r-creating-scatter-plot-from-data-frame}}
\label{fig:dot_plot}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight]{art/histogram_eruptions}
\caption[Histogram]{Histogram.
Notice the ticks on the x axis. These are the raw data points. Make sure you always add them, with the \rcode{rug()} \R command.
\newline \url{http://compbio.pbworks.com/w/page/16252882/Basic}}
\label{fig:histogram_eruptions}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight]{art/ex6aout}
\caption[BoxPlot]{Boxplot. \newline \url{http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm}}
\label{fig:boxplot}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight]{art/stem_and_leaf}
\caption[Stem and Leaf Pot]{Stem-and-leaf plot. \newline
\url{https://www.mathsisfun.com/data/stem-leaf-plots.html}}
\label{fig:stem_and_leaf}
\end{figure}

\subsubsection{Bivariate}
The simultaneous visualization of two continuous variables, can naturally be done with a \emph{scatter plot}.
More sophisticated visualization, which generalizes the histogram into two dimensions, is the \emph{hexbin plot}.
These are illustrated in figures \ref{fig:scatterplot}, and \ref{fig:hexbin}, respectively.

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight]{art/scatterplot}
\caption[Scatter Plot]{Scatter Plot. \newline
\url{http://texample.net/tikz/examples/scatterplot/}}
\label{fig:scatterplot}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight]{art/hexbin2}
\caption[HexBin Plot]{HexBin plot (a 2D histogram).
\newline \url{http://www.r-bloggers.com/5-ways-to-do-2d-histograms-in-r/}}
\label{fig:hexbin}
\end{figure}

\subsubsection{Multivariate Data}
Since we cannot possibly visualize data in more than $3$-dimensions, and we clearly prefer data in $1$ or $2$ dimensions, the visualization of multivariate data will typically consist of summarizing the data into $1D$ or $2D$, and then applying the above mentioned visualization techniques.

An important exception is due to the observation that a computer image, is essentially a matrix.
We can thus visualize matrices, with a simple image, and in particular, covariance and correlation matrices, as illustrated in Figure~\ref{fig:covariance_image}.

A second exception is when the data has both continous variables and discrete attributes.
Endlessly many combinations are then possible.
The author strongly recommends to visit Hans Rosling's \emph{Gap Minder} at \url{http://www.gapminder.org/world} for an excellent interactive visualization. \marginnote{Gap Minder}

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight]{art/covarianceSupervised}
\caption[Covariance Matrix]{Image of covariance matrix. \newline
\url{http://cs.brown.edu/courses/csci1950-g/results/final/sghosh/}}
\label{fig:covariance_image}
\end{figure}

\subsection{On-Line Visualization}
For the purpose of quality control, we may often want an \emph{on-line} visualization, and not \emph{off-line}, as the ones previously discussed.
This is the purpose of \emph{dashboards}, illustrated in Figure~\ref{fig:dashboard}.\marginnote{Dashboard}

\begin{figure}[h]
\centering
\includegraphics[height=0.3\textheight, width=0.9\linewidth]{art/dashboard}
\caption[Dashboard]{Dashboard. \newline
\url{http://www.iconics.com/Home/Products/AnalytiX/Quality-AnalytiX.aspx}}
\label{fig:dashboard}
\end{figure}

\afterpage{\clearpage}


\documentclass[paper-main.tex]{subfiles}

\begin{document}

\section{Open-source code}
\label{app:code}
This project is implemented in Python 3 scripts and jupyter notebooks and MATLAB.
We refer the reader to the Supplementary Material for software references.
The current build and sample data can be found at:
\url{https://github.com/daccordeon/gravexplain}

\section{Detecting a sinusoidal signal in Gaussian noise}
\label{app:sinusoid_likelihood}

In this appendix, we demonstrate that the modulus of the Fourier transform is an appropriate detection statistic when searching for a sinusoidal signal in Gaussian noise.
We describe the data as
\begin{equation}
x(t) = s(t) + n(t)\,,
\label{eqn:GNdata}
\end{equation}
where $s(t)$ and $n(t)$ are the signal and noise, respectively.
The signal takes the form
\begin{equation}
s(t) = A \cos\left[{\omega t + \phi}\right]\,,
\label{eqn:GNmodel}
\end{equation}
where $A$, $\omega$, and $\phi$ are the amplitude, angular frequency and phase of the signal, respectively.
The noise $n(t)$ is a fluctuating zero-mean time series with the following property: if we define an inner product between two arbitrary time series $u(t)$ and $v(t)$ as
\begin{equation}
\ip{u}{v} = \frac{1}{T} \int_0^T \mathrm{d} t \, u(t) v(t)\,,
\label{eqn:ipuv}
\end{equation}
where $T$ is the total time of the observation, then the likelihood $\mathcal{L}$ of measuring the noise-noise product $\ip{n}{n}$ is given by
\begin{equation}
\mathcal{L} = \exp\left( -\frac{1}{2}\ip{n}{n}\right)\,.
\label{eqn:ipnn}
\end{equation}
Equations~\ref{eqn:ipuv} and~\ref{eqn:ipnn} define what it means for noise to be Gaussian through the fundamental measurement of $\ip{n}{n}$.

The likelihood of measuring the signal $s(t)$ in the presence of noise follows from Eqns.~\ref{eqn:GNdata} and~\ref{eqn:ipnn} by replacing $n(t)$ in Eqn.~\ref{eqn:ipnn} with $x(t) - s(t) = n(t)$ from Eqn.~\ref{eqn:GNdata}.~\cite{JKS:1998,Jaynes:2003}
The result is
\begin{eqnarray}
\mathcal{L} &=& \exp\left( -\frac{1}{2}\ip{x-s}{x-s} \right) \label{eqn:likeOne}\,, \\
            &=& \exp\left( -\frac{1}{2}\ip{x}{x} - \frac{A^2}{2} \ip{\cos[\omega t + \phi]}{\cos[\omega t + \phi]} \right. \nonumber\\
                 &&\hspace{3em} \left. +  A \ip{x}{\cos[\omega t + \phi]} \vphantom{\frac12}\right) \,,\\
                 &=& \exp \left( -\frac{1}{2}\ip{x}{x} - \frac{A^2}{4} + A \ip{x}{\cos[\omega t + \phi]} \right)\,.\label{eqn:logLike}
\end{eqnarray}
We then maximise Eqn~.\ref{eqn:logLike} with respect to $A$, obtaining
\begin{eqnarray}
\label{eqn:maxLogLike}
\mathcal{L}_\mathrm{max} = \exp \left( - \frac{1}{2} \ip{x}{x} + \ip{x}{\cos[\omega t + \phi]}^2 \right)\,,
\end{eqnarray}
for $A = 2\ip{x}{\cos[\omega t + \phi]}$.
From the second term of Eqn.~\ref{eqn:maxLogLike}, we see that the maximum likelihood of a sinusoidal signal in Gaussian noise is the modulus of the cosine Fourier transform, plus the term $\ip{x}{x}$, which is independent of $\omega$ and $\phi$ and can therefore be ignored when searching over $\omega$.

Two important points must be made about the above procedure.
(i) Fundamentally the goal is to maximize $\mathcal{L}$ in Eqn.~\ref{eqn:likeOne} by varying $s(t)$ through $A$.
For the special case of the Gaussian likelihood (Eqn.~\ref{eqn:ipnn}), which peaks at $\ip{n}{n}= 0$, this is equivalent to minimizing the difference between $x(t)$ and $s(t)$ as evident in Eqn.~\ref{eqn:ipuv}.
In general, however, minimizing the difference between $x(t)$ and $s(t)$ is not equivalent always to the fundamental goal of maximising $\mathcal{L}$, for example if $\mathcal{L}$ peaks at $\ip{n}{n} \neq 0$, or if $\mathcal{L}$ has multiple maxima.
(ii) The maximum likelihood $\mathcal{L}_{\rm max}$ in Eqn.~\ref{eqn:maxLogLike} (or equivalently its logarithm) defines the detection statistic.
When its value exceeds a threshold (chosen freely by the analyst) at some value of $\omega$, a signal is deemed to have been detected at that value of $\omega$.
Therefore the specific functional form of Eqn.~\ref{eqn:maxLogLike} matters, which is a second reason why one must start from Eqn.~\ref{eqn:ipnn} rather than Eqn.~\ref{eqn:likeOne}, in addition to reason (i).

\section{Viterbi algorithm}
\label{app:viterbi}
This appendix contains some details regarding the implementation of the Viterbi algorithm described in Section~\ref{sec:viterbi}.
The Viterbi algorithm~\cite{Viterbi:1967} is a classic method in signal processing, whose theoretical underpinnings and implementation are accessible to undergraduate students.
See the Supplementary Material for further resources and other pseudocode examples available online.

Here, we present some pseudocode (below) of the implementation used in Section~\ref{sec:viterbi}. We use Fourier amplitudes, normalized between $(0, 1)$ by dividing by the maximum value in the grid, as multiplicative weights.
To avoid numerical underflow we take the logarithm of the Fourier amplitudes, which we can equivalently use as additive weights.

Let $X$ be a grid of $j=0,\ldots,N_f$ rows and $i=0,\ldots,N_t$ columns of additive weights for each node. Let Y and Z be grids of the same shape to store the weight of the best path to each node and the row index of the previous node on that path, respectively. Let W be a length $N_t+1$ array to store the final sequence of row indices for the optimal path. We restrict paths to only move one cell up or down at a time (or stay constant). For the boundary cases of $j=0,N_f$, we only search over $k \in \{0,1\}$ and $k \in \{-1,0\}$, respectively, to stay inside the grid (this is not shown in the pseudocode below).

\begin{algorithmic}
\Function{Viterbi}{$X$}
    % \State $a \gets a+1$
    % \State \Return $a$

    \For{each row $j=0,\ldots,N_f$}
		\State $Y_{0,j} \gets X_{0,j}$
    \EndFor

    \For{each column $i=1,\ldots,N_t$}
	    \For{each row $j=0,\ldots,N_f$}

	    	\State $Y_{i,j} \gets X_{i,j} + \underset{k \in \{-1,0,1\}}{\max} (Y_{i-1,j+k})$
	    	\State $Z_{i,j} \gets j + \underset{k \in \{-1,0,1\}}{\arg\max} (Y_{i-1,j+k})$

	    \EndFor
    \EndFor

    \State $W_{N_t} \gets \underset{j=0,\ldots,N_f}{\arg\max} (Y_{N_t,j})$

    \For{each col $i=N_t-1,\ldots,0$}

		\State $W_i \gets Z_{i+1, W_{i+1}}$

    \EndFor

    \State \Return $W$
\EndFunction
\end{algorithmic}

\end{document}


\chapter{Sets}

Sets are one of the most basic mathematical tools. The purpose of this
chapter is to define sets, and do basic reasoning about them. We're
also going to give a loose definition of common sets we deal
with. Spoiler: it will later turn out that much of what we do in this
chapter is actually wrong, and can lead to some
contradictions.\footnote{If you're really impatient, and can't wait
  for another chapter, you can read the \link{Wikipedia page on
    Russell's
    paradox}{https://en.wikipedia.org/wiki/Russell\%27s_paradox}}

Sets are collections of objects. The easiest way to denote sets is to
list their elements between curly braces: $\braces{}$.

\begin{displaymath}
  \mset{1,2,3,4}
\end{displaymath}

There is no notion of duplication or order, so all of the following
sets are the same

\begin{tabu}{rp{5.5cm}}
  $\mset{1,2,3,4}$         & \\
  $\mset{1,1,2,2,3,3,4,4}$ & (Each element is duplicated.) \\
  $\mset{2,1,4,3,4,4,1}$   & (The order is different, and some of the elements are duplicated.)
\end{tabu}

There's a set with no elements, called the ``null set'', or the
``empty set'', or a variety of other names. It's usually denoted
$\emptyset$, $\phi$, or some variant thereof. I'm going to use $\nil$.

First of all, if we have a set $A$ and an object $x$, and $x$ is an
element of $A$, we'll write $x \in A$. Note: I've have an old textbook
that uses the Greek letter epsilon ($\epsilon$ or $\varepsilon$) in
place of $\in$. The $\in$ symbol kind of looks like an `E', which you
can mentally associate with ``element''. The symbol $\in$ can be read
as ``is an element of''. If some potential element $y$ is \xtb{not} in
$A$, then we'll write $y \notin A$.\footnote{As a general rule, if you
  have some symbol that indicates that something is true, crossing out
  the symbol indicates that thing is false.}

Continuing with the above example, we have the set
$\mset{1,2,3,4}$. We have

\begin{displaymath}
  1 \in \mset{1,2,3,4}
\end{displaymath}

but

\begin{displaymath}
  5 \notin \mset{1,2,3,4}
\end{displaymath}

We can have sets that have infinitely many elements. For instance, the
``natural numbers'', usually denoted with $\N$, are
infinite.\footnote{Conventions differ on whether or not $\N$ includes
  $0$. It doesn't matter all that much, especially as far as this book
  is concerned.} These are positive whole numbers (i.e. not fractions
or decimals).

\begin{displaymath}
  \N = \mset{1,2,3,4,\dots}
\end{displaymath}

The ``integers'' are any whole numbers, which include $0$ and negative
numbers. The integers are usually denoted with $\Z$. The $\Z$ stands
for ``Zahlen'', which is German for ``numbers''.\footnote{I heard this
  from a professor once. If it's wrong, blame him.}

\begin{displaymath}
  \Z = \mset{\dots, -3, -2, -1, 0, 1, 2, 3, \dots}
\end{displaymath}

\section{Set comprehensions}

Before we go much further, I want to introduce the notion of a ``set
comprehension'', or ``set builder notation''. The basic format is this:

\begin{displaymath}
  \scomp{\text{\xtb{What each element looks like}}}{\text{\xtb{Conditions}}}
\end{displaymath}

The ``conditions'' are just things that have to hold true about the
element.

Here are some examples:

\begin{displaymath}
  \scomp{x \in \N}{x < 5} = \mset{1,2,3,4}
\end{displaymath}
\begin{displaymath}
  \scomp{2x \in \N}{x < 5} = \mset{2,4,6,8}
\end{displaymath}
\begin{displaymath}
  \scomp{x \in \Z}{x < 5} = \mset{\dots,-3,-2,-1,0,1,2,3,4}
\end{displaymath}

Each set comprehension should be read in two parts: the part before
the colon, and the part after the colon.

\begin{itemize}
\item $x \in \N$ means that we are choosing\footnote{Yes, we'll get to
    the axiom of choice later. Calm down.} elements of $\N$, and we're
  calling the element $x$.
\item $x < 5$ means that we are choosing all elements of $\N$ which
  are less than $5$. In the part before the colon, we are choosing one
  such element as an example, and calling it $x$.
\end{itemize}

\subsection{Trying these on your computer}

Many programming languages have ``list comprehensions'', which are
conceptually similar to set comprehensions. For instance, in Haskell,
we can construct the equivalent of our sets from above like this:

\begin{lstlisting}[language=Haskell,caption={The sets from above, in Haskell}]
ghci> [ x | x <- [1..4] ]
[1,2,3,4]
ghci> [ 2*x | x <- [1..4] ]
[2,4,6,8]
\end{lstlisting}

In Python, the syntax is a bit different, but:

\begin{lstlisting}[language=Python,caption={The same thing in Python}]
>>> [ x for x in range(1,5) ]
[1, 2, 3, 4]
>>> [ 2*x for x in range(1,5) ]
[2, 4, 6, 8]
\end{lstlisting}

You probably already have Python installed on your system. Open a
terminal, and run the command \monospace{python} (hit Enter after
typing ``python''). If not, try following the instructions here:
\barelink{https://wiki.python.org/moin/BeginnersGuide/Download}.

You probably do \xtb{not} have Haskell installed on your system,
unless you installed it on purpose. You can find documentation on how
to install Haskell here: \barelink{http://haskellstack.org/}.

\subsubsection{Lists v. Sets}

Lists are different from sets, because, in a list, order and
duplication matter. You can test this out in Haskell like this:

\begin{lstlisting}[language=Haskell,caption={List equality in Haskell}]
ghci> [1,2,3,4] == [1,2,3,4]
True
ghci> [1,2,3,4] == [4,3,2,1]
False
ghci> [1,2,3,4] == [1,2,3,1,3,3,1,4,2]
False
\end{lstlisting}

Haskell actually has a type for sets, called \monospace{Set}. To turn
a list into a set, you use the function \monospace{fromList}

\begin{lstlisting}[language=Haskell,caption={Set equality in Haskell}]
ghci> import Data.Set
ghci> fromList [1,2,3,4] == fromList [1,2,3,4]
True
ghci> fromList [1,2,3,4] == fromList [4,3,2,1]
True
ghci> fromList [1,2,3,4] == fromList [1,4,3,1,2]
True
\end{lstlisting}

\section{Rational and Real Numbers}

The reason I introduced the set comprehension notation is because (1)
it's useful and important, and (2) it's convenient to define the
rational numbers (numbers that can be expressed as a ratio of two
integers) like this:

\begin{displaymath}
  \Q = \scomp{\frac{x}{y} \in \R}{x, y \in \Z; y \ne 0}
\end{displaymath}

Remember, kids: \xtb{you can't divide by zero}, hence the $y \ne 0$
part. The notation $x, y \in \Z$ is just shorthand for $x \in \Z$ and
$y \in \Z$. Mathematicians are incredibly lazy, so you'll often see
confusing notational shorthand used as a replacement for actually
explaining an idea. The symbol $\R$ refers to the ``real
numbers''. The symbol $\Q$ stands for ``quotient''.

Rational numbers are numbers that can be expressed as a ratio of two
integers. Real numbers turn out to be really hard to define. You can
for now think of a real number as any number that can be written down,
supposing one had infinite paper, ink, and time. For instance, $\pi$
cannot be expressed as the ratio of two integers, but is a real
number.

\subsection{Numerical representations of Real numbers}

As it turns out, this definition of real numbers doesn't work! The
reason is, there are cases where you write down two different numbers,
and they refer to the same number.

(I stole this example from a professor of mine, Peter Alfeld, who
presumably stole it from someone else.\cite{pa-unique})

\begin{example}[Issues with numerical representations of real numbers]
  Consider the number $x = 0.9999999\dots$. We can multiply by
  $10$. To do so, we just move everything one digit to the left.

  \begin{rclmath}
      x & = & 0.99999\dots \\
    10x & = & 9.99999\dots \\
  \end{rclmath}

  Well, we can take $9x = 10x - x$, and get this

  \begin{rclmath}
    10x & = & 9.99999\dots \\
      x & = & 0.99999\dots \\
     9x & = & 9 \\
  \end{rclmath}

  We get this because every single digit to the right of the decimal
  point cancels. We can take $x = \frac{9x}{9}$ and get

  \begin{rclmath}
     9x & = & 9 \\
      x & = & 1 \\
  \end{rclmath}

  This is a silly example, and doesn't constitute a ``proof''. However,
  it should give you a glimpse at why numerical representations of real
  numbers are problematic.
\end{example}

Here's another example, which I stole from LeRoy Eide (he showed me
this in person).

\begin{example}[Issues with numerical representations of integers]
  Take a number that begins with an endless string of $9$s to the
  left.

  \begin{displaymath}
    x = \dots99994
  \end{displaymath}

  Let's take $x + 6$.

  \begin{displaymath}
    \begin{tabu}{rr}
        & \dots99994 \\
      + & 6 \\
      \tabucline \\
      = & 0 \\
    \end{tabu}
  \end{displaymath}

  Try it! You add the $6$ to the $4$, and get $10$. Fair enough, put
  $0$ and carry the $1$

  \tabulinesep=0ex
  \begin{displaymath}
    \begin{tabu}{rr}
        & \text{1\hspace{2.1mm}}  \\
        & \dots99994 \\
      + & 6 \\
      \tabucline \\
      = & 00 \\
    \end{tabu}
  \end{displaymath}

  Let's try it again

  \begin{displaymath}
    \begin{tabu}{rr}
        & \text{1\hspace{0.9mm}1\hspace{2.1mm}}  \\
        & \dots99994 \\
      + & 6 \\
      \tabucline \\
      = & 000 \\
    \end{tabu}
  \end{displaymath}

  Continue this process...

  \begin{displaymath}
    \begin{tabu}{rr}
        & \text{\dots 1\hspace{0.9mm}1\hspace{0.9mm}1\hspace{0.9mm}1\hspace{2.1mm}}  \\
        & \dots99994 \\
      + & 6 \\
      \tabucline \\
      = & \dots 00000 \\
    \end{tabu}
  \end{displaymath}

  Therefore, we conclude that $\dots999994 + 6 = 0$, and therefore
  that $\dots99994 = -6$.
\end{example}

Note, we do use infinite carry all the time. For instance, it's why

\begin{displaymath}
  \frac{2}{3} + \frac{1}{3} = 0.666\dots + 0.333\dots = 0.999\dots = 1
\end{displaymath}

Infinite carry actually follows nicely from something called
``mathematical induction''. I'll explain induction in a later chapter.

\begin{example}(More insanity with numbers)
  We can take this a step further, and combine the two cases.

  Let's look at $x = \dots99994$, and examine what happens when we
  take $\frac{x}{100}$.

  \begin{displaymath}
    \frac{x}{100} = \dots999.94
  \end{displaymath}

  Let's look at $\frac{x}{10}$

  \begin{displaymath}
    \frac{x}{10} = \dots999.4
  \end{displaymath}

  Well, remember $\frac{x}{10} = \frac{10x}{100}$. Let's let
  $y = \frac{x}{100}$. Therefore, $10y = \frac{x}{10}$. Let's do what
  we did before, and take $10y - y$

  \begin{displaymath}
    \begin{tabu}{rr}
        & \dots999.4\text{\hspace{2mm}} \\
      - & \dots999.94 \\
      \tabucline \\
      = & -0.54 \\
      = & -\frac{54}{100} \\
    \end{tabu}
  \end{displaymath}

  Now we have that $9y = -\frac{54}{100}$. Therefore, $y =
  -\frac{6}{100} = -\frac{3}{50}$. However, we had that $y =
  \frac{x}{10}$, and therefore $10y = x$. Thus, we have

  \begin{displaymath}
    x = -\frac{3}{5} = -0.6
  \end{displaymath}

  Remember, before, we proved $x = -6$.
\end{example}

The problem here is that \xtb{decimal (base $10$) representations of
  real numbers are not unique}. That sounds like a contradiction, but
it really isn't. It turns out, and we might prove this later, that you
can tweak the rules of writing numbers down to make decimal
representations unique. For decimal numbers, we assert that the number
can't begin or end with an infinite string of $9$s.

\section{Unions and Intersections}
\section{Differences}
\section{De Morgan's Laws}


% -*- root: Main.tex -*-
\section{SVD}
$\mathbf{A} = \mathbf{U} \mathbf{D} \mathbf{V}^\top = \sum_{k=1}^{\operatorname{rank}(\mathbf{A})} d_{k,k} u_k (v_k)^\top$\\
$\mathbf{A} \in \mathbb{R}^{N \times P}, \mathbf{U} \in \mathbb{R}^{N \times N}, \mathbf{D} \in \mathbb{R}^{N \times P}, \mathbf{V} \in \mathbb{R}^{P \times P}$\\
$\mathbf{U}^\top \mathbf{U} = I = \mathbf{V}^\top \mathbf{V}$ ($\mathbf{U}, \mathbf{V}$orthonormal)\\
$\mathbf{U}$ columns are eigenvectors of $\mathbf{A} \mathbf{A}^\top$, $\mathbf{V}$ columns are eigenvectors of $\mathbf{A}^\top \mathbf{A}$, $\mathbf{D}$ diagonal elements are singular values.\\
$(\mathbf{D}^{-1})_{i,i} = \frac{1}{\mathbf{D}_{i, i}}$ (don't forget to transpose)

1. calculate $\mathbf{A}^\top \mathbf{A}$.\\
2. calculate eigenvalues of $\mathbf{A}^\top \mathbf{A}$, the square root of them, in descending order, are the diagonal elements of $\mathbf{D}$.\\
3. calculate eigenvectors of $\mathbf{A}^\top \mathbf{A}$ using the eigenvalues resulting in the columns of $\mathbf{V}$.\\
4. calculate the missing matrix: $\mathbf{U} = \mathbf{A} \mathbf{V} \mathbf{D}^{-1}$.\\
5. normalize each column of $\mathbf{U}$ and $\mathbf{V}$.
computational cost = $O(mn^2 + m^2n), m\geq n$
\subsection*{Low-Rank approximation}
Using only $K$ largest eigenvalues and corresponding eigenvectors. $\tilde{\mathbf{A}}_{i, j} = \sum_{k=1}^K \mathbf{U}_{i, k} \mathbf{D}_{k,k} \mathbf{V}_{j, k} = \sum_{k=1}^K \mathbf{U}_{i, k} \mathbf{D}_{k,k} (\mathbf{V}^\top)_{k, j}$.

\subsection*{Eckart-Young Theorem}
$\min_{rank(B)=K} ||A-B||_F^2 = ||A-A_k||_F^2 = \sum_{r=k+1}^{rank(A)} \sigma_r^2$
$\min_{rank(B)=K} ||A-B||_2 = ||A-A_k||_2 = \sigma_{k+1}$

\subsection*{Linear Auto-encoder}
By EY, the optimal $C,D$ matrices s.t. $\hat X = DCX$ are $C=AU_k^\top, D=U_kA^{-1}$ with $A$ a rotation matrix $\rightarrow$low interpretability of the latent space and allow weight sharing $C=D^\top$ w/o loss of generality.

\subsection*{SVD Thresholding}
$\textbf{B}^* = shrink_\tau (\textbf{A}) := \text{argmin}_\textbf{B}\left[\frac{1}{2}\|A-B\|_F^2+\tau \|B\|_*\right]$ \\
$D_\tau = diag(max(0, \sigma_i - \tau))$ \\
proj. op: $\Pi(\textbf{X}) = x_{ij}$ if $(i,j) \in I$ else $0$ \\
$ \textbf{B}_{t+1} = \textbf{B}_t + \eta_t \Pi(A - schrink_\tau (\textbf{B}_t)) $


\chapter{Appendix - Math Prerequisites}
\label{appendix:gmm_log_prob}
For a two-dimensional \ac{GMM} $g \sim GMM(\mu, \Sigma, \pi, \rho)$ with mean vector $\mu$, variance matrix $\Sigma$, mode importance vector $\pi$ and Pearson's correlation coefficients $\rho$ the probability at $(x, y)$ can be computed as:\footnote{\href{https://de.wikipedia.org/wiki/Mehrdimensionale_Normalverteilung}{Wikipedia - Multivariate Normal Distribution}}.

\begin{align}
f_m(x, y | \mu, \sigma, \rho)
&= \frac{1}{2 \pi \sqrt{\det \Sigma}} \exp \left(- \frac{1}{2} (\boldsymbol{x} - \boldsymbol{\mu})^T \Sigma (\boldsymbol{x} - \boldsymbol{\mu}) \right) \\
&= {\frac{\sqrt{1- \rho^2}}{2 \pi \sigma_x \sigma_y}
\exp - \frac{1}{2 (1 - \rho^2)}} \left( \frac{( x - \mu _x)^2}{\sigma_x^2} +
\frac {(y - \mu_y)^2}{\sigma_y^2}-{\frac {2\rho (x - \mu_x)(y - \mu_y)}
{\sigma_x \sigma_y}} \right)
\end{align}

The Pearson coefficient is the covariance for two normalized random variables $X$ and $Y$, in their normalized form $\tilde{X} = (X - \mu_x)/\sigma_x$ and $\tilde{Y} = (Y - \mu_y) / \sigma_y$. Using the law of linear transformations of covariances we get:\footnote{\href{https://en.wikipedia.org/wiki/Pearson_correlation_coefficient}{Wikipedia - Pearson correlation coefficient}}

\begin{equation}
Cov(\tilde{X}, \tilde{Y}) = \frac{1}{\sigma_x \sigma_y} Cov(X, Y) = \rho_{X, Y}
\end{equation}

The expected value over all modes $m \in [0, M]$ super-imposes the \ac{PDF} for the bi-variate Gaussian distributions, weighting each one by its importance $\pi_m$:

\begin{equation}
\mathbb{E}_{GMM}[x, y] = \sum_{m=0}^M \pi_m \cdot f_m(x, y) =  \sum_{m=0}^M \exp \left( \log \pi_m + \log f_m(x, y) \right)
\end{equation}

\chapter{Appendix - Detailed Planning Approach Comparison}
There are several categories from which the properties of planning approaches can be evaluated:

\begin{itemize}
    \item Optimality: trajectory cost vs minimal possible (optimal) cost
    \begin{itemize}
        \item globally: $J(x(t), u(t)) = J^*(x(t), u(t)) \forall t$
        \item locally: $J(x(t), u(t)) \approx J^*(x(t), u(t)) \forall x(t) + \epsilon, u(t) + \epsilon $
        \item not at all
    \end{itemize}

    \item Risk-Awareness: Point-wise constraints are easier to fulfill since each state can be regarded individually, however it assumes independence between the states, which is clearly not given due to the dynamical constraints of the ego. As shown in \cite{JansonSP15} both the additive and multiplicative formulation do not scale with increasing planning horizon, since the accumulated risk converges to infinity. Also the using point-wise constraint often is a very conservative choice, as it does not allow to take more risk at some point to be more efficient or to save risk somewhere else.
    \begin{itemize}
        \item trajectory-wise: $\sum_{k = 0}^N r(x_k, u_k) \leq R_{max}$
        \item point-wise: $r(x_k, u_k) \leq R_{max} \forall k$
    \end{itemize}

    \item Computational Feasibility:
    \begin{itemize}
        \item real-time planning
        \item online policy/trajectory correction (e.g. using perturbation)
        \item not real-time applicable
    \end{itemize}

    \item Explainability \& Guarantees: When the result of the planning algorithm is not interpretable it is very hard (or even impossible) to give theoretical guarantees with respect to safety constraints. Therefore for these approaches an additional step, determining the empirical risk of collision using Monte Carlo simulations, is necessary, at the cost of additional run-time.
\end{itemize}

\begin{center}
\begin{tabular}{c||p{2cm}|p{1cm}|p{2cm}|p{1cm}|p{1cm}|p{1cm}|p{1cm}|p{1cm}}
    Method &
    \rotatebox[origin=c]{90}{Space ?} &  \rotatebox[origin=c]{90}{Risk-Awareness ?} &
    \rotatebox[origin=c]{90}{PPDF Model ?} &
    \rotatebox[origin=c]{90}{Risk as cost ?} &
    \rotatebox[origin=c]{90}{Optimality ?} &
    \rotatebox[origin=c]{90}{Parametrization ?} &
    \rotatebox[origin=c]{90}{Interactive ?} &
    \rotatebox[origin=c]{90}{Interpretability ?} \\
    \hline\hline

    CCMPC & continuous & point-wise & approximate (uni-Gaussians) & no & locally & sparse & no & yes \\
    \hline
    SIPP & discrete & point-wise & accurate (sampled GMM) & no & globally & sparse & no & yes \\
    \hline
    DRL & continuous & none & none & no & none & much & yes & no \\
    \hline
    PORCA & continuous & none & none & none & none & much & yes & no \\
    \hline
    SACBP & continuous & yes & accurate (sampled traject.) & yes & locally & sparse & no & yes \\
    \hline
    POMDP & discrete & yes & accurate & yes & globally & much (state definition) & no & yes \\
    \hline
    IRL & discrete & yes & accurate & none & none & much (optimal trajectory) & medium & medium

\end{tabular}
\end{center}
%


\lesson{4}{Sep 22 2021 Wen (12:43:32)}{Complex Numbers}{Unit 1}

\subsubsection*{Imaginary Numbers}

What pairs of identical factors will produce $-1$ when multiplied? \\ $1 \times
1$ doesn't work since it equals positive $1$. The same is true of $-1 \times 1$
which also equals positive $1$. \\ $1 \times -1$ equals $-1$, but these are not
identical factors.

\begin{definition}[$i$]
    To solve this problem, the concept of the imaginary number \textit{i} was
    invented. The imaginary number was defined to be:

    \[ i = \sqrt{-1} \]

    With this definition, the square root of negative radicands, in addition to
    positive radicands, may be simplified.
\end{definition}

\begin{example}[$i$]
    Simplify: $\sqrt{-8}$

    \begin{align}
        \sqrt{-1} \times \sqrt{8} &= i \times \sqrt{8} \\
                                  &= i \sqrt{8} \\
                                  &= i \sqrt{4} \times \sqrt{2} \\
                                  &= 2i \sqrt{2}
    \end{align}
\end{example}

\subsubsection*{Squaring Imaginary Numbers}

You know that $i = \sqrt{-1}$ \\
Consequently, $i^2 = (\sqrt{-1})^2$ \\
Since squaring a square root will eliminate the square root sign:

\begin{align}
    \sqrt{3^2} &= \sqrt{3} \times \sqrt{3} \\
               &= \sqrt{9} \\
               &= 3
\end{align}

Then:

\begin{align}
    i^2 &= (\sqrt{-1})^2 \\
        &= i^2 \\
        &= -1
\end{align}

\subsubsection*{Cubing Imaginary Numbers}

What about cubing the imaginary number:

\begin{align}
    i^3 &= i \times i \times i \\
        &= i^3 \\
        &= (i \times i) \times i \\
        &= i^3 = i^2 \times i \\
        &= i^3 = -1 \times i \\
        &= i^3 \\
        &= -i
\end{align}

\subsubsection{Biquadrate Imaginary Numbers}

\textbf{i} to the fourth power can be found in a similar way:

\begin{align}
    i^4 &= i \times i \times i \times i \\
        &= i^4 \\
        &= (i \times i) \times (i \times i) \\
        &= i^4 \\
        &= -1 \times -1 \\
        &= i = 1
\end{align}

\subsubsection*{Simplifying $i$}

\begin{example}[Simplifying $i$]
    Simplify: $i^{27}$

    \begin{align}
        \frac{27}{4} &= 6 R \longrightarrow 3 \\
                     &= i^{R \longrightarrow 3} \\
                     &= -i
    \end{align}
\end{example}

\newpage


\section{Methodology}

\subsection{\textbf{Echo State Networks}}
An ``echo state network'' (also
called a ``liquid state machine'' \cite{lukosevicius_practical_2012}) is a type
of recurrent neural network that uses a single layer of many neurons called a
``reservoir.'' The reservoir has an adjacency matrix $\bm{A}$ that
\begin{enumerate}
	\item is sparsely populated
	\item is connected by uniformly random weights centered at zero
	\item has a large number of neurons
\end{enumerate}
A reservoir computer also satisfies the \textit{echo state property}
\cite{pathak_model-free_2018, lukosevicius_reservoir_2009}. This
property ensures that a system's state has a decaying influence on future states
(like an echo of sound or ripples on water). This property is satisfied in most
cases when the the absolute value of the greatest eigenvalue of
$\bm{A}$ (the spectral radius, $\rho$) \cite{lukosevicius_reservoir_2009} is,
\begin{align}
	\rho(\bm{A}) < 1.
\end{align}
However, the echo state property can still be satisfied for a spectral radius
greater than unity \cite{lukosevicius_practical_2012}.

\begin{figure}[H]
	\includegraphics[width=\columnwidth]{reservoir_network.jpg}
	\caption{A basic reservoir computer or echo state network. The connections in
	the reservoir are given by $\bm{A}$}
	\label{fig:RCmodel}
\end{figure}

Figure \ref{fig:RCmodel} gives a visual representation of a basic
\acrshort{ESN}. An
input vector of length \textit{K} is mapped to the reservoir layer by an input
weight matrix $\bm{W_{in}}$. The state of the reservoir is mapped to an output
layer of length \textit{N} with an output weight matrix $\bm{W_{out}}$.
The output weight matrix is trained through
backpropagation using a loss function like cross entropy
\cite{pathak_model-free_2018, vlachas_backpropagation_2020}.

In this
work, the input vector is a function of time, $\bm{u}(t)$, and the output
vector is the next state of the system, $\bm{u}_p(t+\Delta t)$. Ideally, the
difference between the prediction, $\bm{u}_p$, and the actual, $\bm{u}_a$, is
minimized. \glspl{ESN} are capable of mapping an input vector of any size to
an output vector of any size. For example, $\bm{u}(t)$ could be the total
demand
at $t$ and $\bm{u}_p(t+\Delta t)$ could be the total demand at $t+\Delta t$. In
that scenario the reservoir is a one-to-one map. Alternatively, $\bm{u}(t)$
could be several data points, temperature, air pressure, irradiance, wind
speed, and total demand, at time $t$ and $\bm{u}_p(t+\Delta t)$ could be the
net demand at $t+\Delta t$. In this case the reservoir is a many-to-one map.
Determining the combination of input data that leads to the most accurate
predictions is a key area of research and an important part of this study
\cite{kobylinski_high-resolution_2020,bianchi_reservoir_2020}. For the results
shown below, $\bm{u}(t)$ is just the hourly demand.

\subsection{\textbf{Hyperparameter Search}}
Due to the architecture of \glspl{ESN}, the weights and connections
inside the reservoir do not need to be trained and, in our choice of
implementation, cannot be. This dramatically reduces the training time because
only the linear output layer needs to be trained. One drawback of this approach
is its sensitivity to hyperparameters, which must be carefully chosen before
running the network \cite{ pathak_model-free_2018,
lukosevicius_practical_2012, lukosevicius_reservoir_2009,gallicchio_deep_2019}.
Here, we perform grid searches
to establish which combination of hyperparameters minimizes the mean squared
error of the model,
\begin{align}
	MSE &= \frac{1}{N}\sum_i^N (\hat{y} - y_i)^2\\
	\intertext{where}
	\hat{y} &= \mbox{the average value of the ouput.}\nonumber
\end{align}

\subsection{\textbf{Constructing the \gls{ESN}}}

We constructed the \gls{ESN} in our initial demonstration with the open source
Python package \texttt{pyESN} \cite{korndorfer_pyesn_2015}. The code
is freely available on GitHub. Two models were trained with
historical demand data from \gls{uiuc} in 2017.  Each model had a
reservoir of 500 neurons. One was trained on 1000 hours of
data and predicted the following 100 hours of demand in two hour
increments. The other was trained on 3500 hours of data and, once again,
predicted the following 100 hours.

A grid search informed our choice of hyperparameters for these models, shown in
Figure \ref{fig:gridsearch}. This search determined the optimal combination of
spectral radius ($\rho$) and noise injection (for regularization of reservoir
neurons), shown in Figure \ref{fig:gridsearch}, following the recommendations
from \cite{lukosevicius_practical_2012}. Figure
\ref{fig:gridsearch} shows that there may be multiple combinations of
hyperparameters that give a low mean squared error. The hyperparameters that
will be used in future work reflect a global minimum in the parameter space
where $\rho = 1.3$ and \texttt{noise} $= 0.01$.

\begin{figure}[h]
  \centering
  \includegraphics[width=0.8\columnwidth]{noise_spectral_radius.png}
  \caption{A grid search over a range of spectral radii and noise levels. The
  optimal set minimizes the mean squared error.}
  \label{fig:gridsearch}
\end{figure}


\subsection{The multinomial data generating process}

\subsubsection{Introduction}

In the binomial case we had:

\(z_i=\alpha + \beta x_i +\epsilon_i \)

And set \(y_i\) to \(1\) if \(z_i>0\)

In the multinomial case we have \(m\) alternatives

\(z_{ij}=\alpha + \beta x_{ij} +\epsilon_{ij} \)

And set \(y_{ij}=1\) if \(z_{ij}>z_{ik}\forall k\ne j\)

\subsubsection{Generalised version}

We can rewrite this as:

\(z_{ij}=v_{ij} +\epsilon_{ij} \)

Where:

\(v_{ij}=\alpha+\beta x_{ij}\)

In this case \(v\) does not depend on \(j\), but in other formulations it could.

\subsubsection{Probabilities}

\(P_{ij}=P(y_{ij}=1|x_{ij})\)

\(P_{ij}=P(z_{ij}>z_{ik}\forall k\ne j)\)

\(P_{ij}=P(\epsilon_{ik} <v_{ij} -v_{ik} +\epsilon_{ij}\forall k\ne j)\)

\subsubsection{The form of the multinomial model: Intercepts}

Previously we described the multinomial model

\(z_{ij}=v_{ij} +\epsilon_{ij} \)

Where:

\(v_{ij}=\alpha+\beta x_{ij}\)

The probability of \(j\) being chosen is.

\(P_{ij}=P(\epsilon_{ik} <v_{ij} -v_{ik} +\epsilon_{ij}\forall k\ne j)\)

Interceps in \(v\) cancel out. Therefore in the basic model there is no need to use

\(v_{ij}=\alpha+\beta x_{ij}\)

We can instead use:

\(v_{ij}=\beta x_{ij}\)

\subsubsection{The form of the multinomial model: Conditional model}

We have :

\(v_{ij}=\beta x_{ij}\)

What do we include in \(x_{ij}\)?

We can include observable characteristics for each product:

\(v_{ij}=\alpha_j + \beta x_j\)

One of the \(\alpha_j\) must be normalised to \(0\), as only differences matter. We cannot tell the difference if all \(\alpha \) are raised by the same amount.

For consistency with other models we can write this as:

\(v_{ij}=\beta x_{ij}\)

Even though this does not vary from individual to individual.

Here \(\beta \) represents average preferences for each product characteristic.

\subsubsection{The form of the multinomial model: The multinomial model}

We have differing characteristics for each individual:

\(v_{ij}=\beta x_i\)

However this adds a constant for each product. For this to discriminate we need varying coefficients.

\(v_{ij}=\beta_j x_i\)

As we only observe differences, one of the \(\beta_j\) must be normalised to \(0\).

We can rewrite this.

\(v_{ij}=\sum_k \beta_k\delta_{kj} x_i\)

\(v_{ij}=\beta z_{ij}\)

The original \(x_i\) is dense and contains data about the individual.

\(z_{ij}\) is sparce and only has entries in the \{j\} section.

Here \(\beta \) represents how the

\subsubsection{The form of the multinomial model: Combined multinomial and conditional model}

If we have observations of the characteristics of both individuals and alternatives we can write:

\(v_{ij}=\beta_m m_{ij}+\beta_cc_{ij}\)

\(v_{ij}=\beta x_{ij}\)

Here \(\beta \) represents both:

\begin{itemize}
\item Average preferences for customer characteristics (conditional)
\item How preferences change as individual characteristics change (multinomial)
\end{itemize}


\chapter{Set Theory}
Set Theory is one of the most fundamental topics in Abstract/Discrete
Mathematics. If you have ever heard about Relations, Equivalence Classes,
Groups, Rings, and Fields, they are all based off Set Theory! This chapter
covers the following and it is essential that you know all of it:

\section{Definitions}
\begin{definition}
    Set A is said to be a \underline{\textbf{subset}} of the set B if ($\forall
    x \in A$)[$x \in B$].
\end{definition}

\begin{definition}
    The \underline{\textbf{union}} of set A and set B is the set: \{$x \mid x
    \in A \vee x \in B$\}.
\end{definition}

\begin{definition}
    The \underline{\textbf{intersection}} of A and B is the set: \{$x \mid x
    \in A \wedge x \in B$\}.
\end{definition}

\begin{definition}
    The \underline{\textbf{difference}} of A and B is the set: \{$x \mid x \in
    A \wedge x \not\in B$\}.
\end{definition}

\begin{definition}
    Sets A and B are said to be \underline{\textbf{equal}} if $(A \subset B)
    \wedge (B \subset A)$.
\end{definition}

\begin{definition}
    The \underline{\textbf{complement}} of A is the set: \{$x \mid x \in D
    \wedge x \not\in A$\}, where D is the domain or universal set.
\end{definition}

\begin{definition}
    The \underline{\textbf{cardinality}} of A is the number of elements in A.
\end{definition}

\section{Proofs}
This class in particular will not test too many hard proofs in Set Theory. In
fact, most proofs in this chapter will require you to construct simple direct
proofs. So, let us show you an example of how proofs in set theory might look
like:

\begin{question}
    If A and B are two sets, show that $\overline{A \cap B} = \overline{A} \cup
    \overline{B}$.
\end{question}

\begin{proof}
    We need to show that $\overline{A \cap B} \subset \overline{A} \cup
    \overline{B}$ and $\overline{A} \cup \overline{B} \subset \overline{A \cap
    B}$

    Let $x$ be an arbitrary element $\in \overline{A \cap B}$
    \[
        \begin{split}
            \implies &x \not\in A \cap B \\
            \implies &\neg(x \in A \wedge x \in B) \\
            \implies &x \not\in A \vee x \not\in B \\
            \implies &x \in \overline{A} \cup \overline{B} \\
            \implies &\overline{A \cap B} \subset \overline{A} \cup \overline{B}
        \end{split}
    \]

    Let $x \in \overline{A} \cup \overline{B}$

    \[
        \begin{split}
            \implies &x \in \overline{A} \vee x \in \overline{B} \\
            \implies &\neg x \in A \vee \neg x \in B \\
            \implies &\neg (x \in A \wedge x \in B) \\
            \implies &\neg (x \in A \cap B) \\
            \implies &x \in \overline{A \cap B} \\
            \implies &\overline{A} \cup \overline{B} \subset \overline{A \cap B}
        \end{split}
    \]

    Thus, $\overline{A \cap B} = \overline{A} \cup \overline{B}$.
\end{proof}

You also need to know set identities but they are the same as the basic
identities of propositional logic, so I haven't included them in this section.

I have also skipped the section of Generalized Unions and Intersections as they
are based on the basic definition of union and intersection of sets. Otherwise,
that is pretty much it with regards to set theory!


\section{Question 3}
\begin{equation}
	G(s) = \dfrac{K\exp(-\tau s)}{s(s^2 + s + a)}, \quad \tau>0, \quad a > 0, \quad K > 0
\end{equation}
The transfer function in which s is replaced by $j\omega$ where $\omega$ is frequency.

% The phase angle:
% $$
% \angle G(jw) = \left. \angle K \right. - (\angle j\omega + \left.\angle \left((j\omega)^2 + j\omega + a \right))\right.
% $$
% $$
% \angle G(jw) = \left. \angle K \right. - (\angle j\omega + \left.\angle \left(j\omega -\omega^2 + a \right))\right.
% $$
% $$
% \angle G(jw) = \tan^{-1}\left(\dfrac{0}{K}\right) - \tan^{-1}\left(\dfrac{\omega}{0}\right) - \tan^{-1}\left(\dfrac{\omega}{a - \omega^2}\right)
% $$
% $$
% \angle G(jw) = 0^{\circ} - 90^{\circ} - \tan^{-1}\left(\dfrac{\omega}{a - \omega^2}\right) = 0 - \dfrac{\pi}{2} - \tan^{-1}\left(\dfrac{\omega}{a - \omega^2}\right)
% $$
% The amplitude ratio:
% $$
% \left\vert G(j\omega) \right\vert = \left\vert \dfrac{K}{j\omega\left((j\omega)^2 + j\omega + a \right)} \right\vert
% =  \left\vert \dfrac{K}{j\omega\left(j\omega -\omega^2 + a\right)} \right\vert
% $$
% $$
% \left\vert G(j\omega) \right\vert = \dfrac{\left\vert K \right\vert}{\left\vert j\omega \right\vert
% \left\vert j\omega -\omega^2 + a \right\vert
% } = \dfrac{K}{\omega\sqrt{\omega^2 + (\omega^2-a)^2}}
% $$
Cause $a>0$ we have just one pole on imaginary axis.
\begin{figure}[H]
	\caption{Contour near the origin of the s plane and closed contour in the s plane}
	\centering
	\includegraphics[width=12cm]{../Figure/Q2/nyquist_s_plane.png}
\end{figure}
On the semicircular path with radius $\epsilon$ (where $\epsilon \ll 1 $), the complex variable s can be written
$$
s = \epsilon \exp(j\theta)
$$
where $\theta$ varies from $-\dfrac{\pi}{2}$ to $\dfrac{\pi}{2}$.
\begin{figure}[H]
	\caption{s-Plane}
	\centering
	\includegraphics[width=12cm]{../Figure/Q2/nyquist_s_plane_name.png}
\end{figure}
Now we calculate $j\omega$ in A, C, D and F.

A:

$s = j\omega,\quad \omega = -\epsilon$
$$
 \lim_{\omega\to -\epsilon}G(j\epsilon)  = \dfrac{K\exp(\tau j\epsilon)}{-j\epsilon((j\epsilon)^2 - j\epsilon + a)} = \dfrac{K\exp(\tau j\omega)}{-j\epsilon(-\epsilon^2 - j\epsilon + a)} = \dfrac{K\exp(\tau j\epsilon)}{j\epsilon^3 - \epsilon^2 - aj\epsilon}
$$
$$
\lim_{\epsilon \to 0}\dfrac{K\exp(\tau j\epsilon)}{j\epsilon^3 - \epsilon^2 - aj\epsilon} = \lim_{\epsilon \to 0} \dfrac{K\exp(\tau j\epsilon)}{-aj\epsilon} = \lim_{\epsilon \to 0} \dfrac{Kj\exp(\tau j\epsilon)}{a\epsilon} =  \infty \angle 90^{\circ}
$$

C:

$s = j\omega,\quad \omega = \epsilon$
$$
 \lim_{\omega\to \epsilon}G(j\omega)  = \dfrac{K\exp(-\tau j\epsilon)}{j\epsilon((j\epsilon)^2 + j\epsilon + a)} = \dfrac{K\exp(-\tau j\epsilon)}{j\epsilon(-\epsilon^2 + j\epsilon + a)} = \dfrac{K\exp(-\tau j\epsilon)}{j\epsilon^3 - \epsilon^2 - aj\epsilon}
$$
$$
\lim_{\epsilon \to 0}\dfrac{K\exp(-\tau j\epsilon)}{j\epsilon^3 - \epsilon^2 + aj\epsilon} = \lim_{\epsilon \to 0} \dfrac{K\exp(-\tau j\epsilon)}{-aj\epsilon} = \lim_{\epsilon \to 0} \dfrac{-Kj\exp(-\tau j\epsilon)}{a\epsilon} =  \infty \angle -90^{\circ}
$$

A to C:

$s = j\omega,\quad \omega = \epsilon\exp(j\theta), \quad \theta: -\dfrac{\pi}{2}\to 0 \to \dfrac{\pi}{2}(CCW)$
$$
 \lim_{\epsilon\to 0}G(\epsilon\exp(j\theta))  =\dfrac{K\exp(-\tau j\epsilon)}{\epsilon\exp(j\theta)((\epsilon\exp(j\theta))^2 + \epsilon\exp(j\theta) + a)} =  \dfrac{K\exp(-\tau j\epsilon)}{\epsilon^3\exp(3j\theta) + \epsilon^2\exp(2j\theta) + a\epsilon\exp(j\theta)}
$$
$$
\lim_{\epsilon \to 0}\dfrac{K\exp(-\tau j\epsilon)}{a\epsilon\exp(j\theta)} = \lim_{\epsilon \to 0} \dfrac{K\exp(-j\theta-\tau j \epsilon)}{a\epsilon} =  \infty \angle -\theta
$$

D:

$s = j\omega,\quad \omega = \infty$
$$
\lim_{\omega\to \infty}G(j\omega)  =  \lim_{\omega\to \infty}\dfrac{K\exp(-\tau j\omega)}{j\omega((j\omega)^2 + j\omega + a)} = \lim_{\omega\to \infty} \dfrac{K\exp(-\tau j\omega)}{j\omega(-\omega^2 + j\omega + a)} = \lim_{\omega\to \infty} \dfrac{K\exp(-\tau j\omega)}{-j\omega^3 - \omega^2 + aj\omega}
$$
$$
\lim_{\omega\to \infty} \dfrac{K\exp(-\tau j\omega)}{-j\omega^3} = \lim_{\omega\to \infty} \dfrac{jK\exp(-\tau j\omega)}{\omega^3}
= 0 \angle 90^{\circ}
$$

F:

$s = j\omega,\quad \omega = -\infty$
$$
\lim_{\omega\to -\infty}G(j\omega)  =  \lim_{\omega\to -\infty}\dfrac{K\exp(-\tau j\omega)}{j\omega((j\omega)^2 + j\omega + a)} = \lim_{\omega\to -\infty} \dfrac{K\exp(-\tau j\omega)}{j\omega(-\omega^2 + j\omega + a)} = \lim_{\omega\to -\infty} \dfrac{K\exp(-\tau j\omega)}{-j\omega^3 - \omega^2 + aj\omega}
$$
$$
\lim_{\omega\to -\infty} \dfrac{K\exp(-\tau j\omega)}{-j\omega^3} = \lim_{\omega\to -\infty} \dfrac{jK\exp(-\tau j\omega)}{\omega^3}
= 0 \angle -90^{\circ}
$$

\begin{figure}[H]
	\caption{Nyquist plot using MATLAB ($a = 1, K = 1, \tau = 0.01$)}
	\centering
	\includegraphics[width=12cm]{../Figure/Q3/MATLAB-Nyquist.png}
\end{figure}

Now we know shape of nyquist plot. we find out where it's equal to $-1$.

$$
G(j\omega) = -1 + 0j = \dfrac{K\exp(-\tau j\omega)}{j\omega((j\omega))^2 + j\omega + a)} = \dfrac{K\exp(-\tau j\omega)}{-j\omega^3 - \omega^2 + aj\omega} = -1
$$

$$
\to j\omega^3 + \omega^2 -aj\omega = K\exp(-\tau j\omega) \to j\omega^3 + \omega^2 -aj\omega - K\exp(-\tau j\omega) = 0
$$
if we assume $\omega\tau \ll 1$ the problem is like previous question.
Two equation and two unknowns.
\begin{align*}
	j\omega^3 &=aj\omega-Kj\omega\tau \to \omega^2 = a-K\tau\\
	\omega^2 &=K \to \omega^2 = K
\end{align*}
When $a = K$ the nyquist plot cross from $-1$ point. When $a-\tau\omega>K$ the nyquist plot is before $-1$ and when $a-\tau\omega<K$ the nyquist plot cross $-1$ and system is unstable.
\begin{figure}[H]
	\caption{Stable Nyquist plot using MATLAB ($a = 2, K = 1, \tau = 0.01$)}
	\centering
	\includegraphics[width=12cm]{../Figure/Q3/MATLAB-Nyquist_stable.png}
\end{figure}
\begin{figure}[H]
	\caption{Unstable Nyquist plot using MATLAB ($a = 1, K = 2, \tau = 0.01$)}
	\centering
	\includegraphics[width=12cm]{../Figure/Q3/MATLAB-Nyquist_unstable.png}
\end{figure}

When $\tau \ll 1$ it's not true and we have nonlinear equation that is hard to slove.
\begin{align*}
	j\omega^3 &=aj\omega - Kj\sin(\tau\omega)\\
	\omega^2 &=K\cos(\tau\omega)
\end{align*}

when $\tau$ increase we must increase $a$ or decrease $K$ for stable system.

\begin{figure}[H]
	\caption{Unstable Nyquist plot using MATLAB ($a = 1, K = 1, \tau = 2$)}
	\centering
	\includegraphics[width=12cm]{../Figure/Q3/MATLAB-Nyquist_increase_tau.png}
\end{figure}

\begin{figure}[H]
	\caption{Stable Nyquist plot using MATLAB ($a = 2, K = 1, \tau = 2$)}
	\centering
	\includegraphics[width=12cm]{../Figure/Q3/MATLAB-Nyquist_increase_tau_and_a.png}
\end{figure}
When we increase $\tau$ it's make our system unstable.

\begin{figure}[H]
	\caption{Unstable Nyquist plot  with large $\tau$ using MATLAB ($a = 2, K = 1, \tau = 100$)}
	\centering
	\includegraphics[width=12cm]{../Figure/Q3/MATLAB-Nyquist_increase_tau_very_lage.png}
\end{figure}

\begin{figure}[H]
	\caption{Stable Nyquist plot with large $\tau$ and small $K$ using MATLAB ($a = 2, K = 0.00001, \tau = 100$)}
	\centering
	\includegraphics[width=12cm]{../Figure/Q3/MATLAB-Nyquist_increase_tau_very_lage_K_very_low.png}
\end{figure}


\documentstyle[a4wide,12pt]{article}

\begin{document}

\section*{Building a model for the solar system using ordinary differential equations}

\subsection*{Introduction.}

The aim of this project is to develop a code for simulating the solar
system using a widely popular algorithm for solving coupled ordinary
differential equations, the so-called velocity Verlet algorithm. An
important aspect of this project is to be able to object orient your
code. There are several coupled ordinary differenatial equations where
the basic equations, except for various physical constants and
variables, are rather similar. Thus, write once and run many times,
one of the central points of object orientation, is something which
makes our program easier to extend and build upon when we add more
planets or moons or other astronomical objects. The basic equations
which govern the system are rather simple, a set of coupled equations
that codify Newton's law of motion due the gravitational force.

In the first part however, we will limit ourselves (in order to test the algorithm)
to a hypothetical solar system
with the Earth only orbiting around the sun.
The only force in the problem is gravity. Newton's law of gravitation  is given by a force $F_G$
\[
F_G=\frac{GM_{\odot}M_{\mathrm{Earth}}}{r^2},
\]
where $M_{\odot}$ is the mass of the Sun and $M_{\mathrm{Earth}}$ is the mass of the Earth. The gravitational constant is $G$ and $r$ is the distance between the Earth and the Sun.
We assume that the Sun has a mass which is much larger
than that of the Earth. We can therefore safely neglect the
motion of the Sun in this problem.  In the first part of this project, your aim is to compute the motion
of the the Earth using different methods for solving ordinary differential equations.

We assume that the orbit of the Earth around the Sun
is co-planar, and we take this to be the $xy$-plane.
Using Newton's second law of motion we get the following equations
\[
\frac{d^2x}{dt^2}=\frac{F_{G,x}}{M_{\mathrm{Earth}}},
\]
and
\[
\frac{d^2y}{dt^2}=\frac{F_{G,y}}{M_{\mathrm{Earth}}},
\]
where $F_{G,x}$ and $F_{G,y}$ are the $x$ and $y$ components of the gravitational force.

We will use so-called astronomical units when rewriting our equations.
Using astronomical units (AU as abbreviation)it means that
one astronomical unit of length, known as 1 AU, is the average distance between the Sun and Earth, that is
$1$ AU = $1.5\times 10^{11}$ m.  It can also be convenient to use years instead of seconds since years match
better the time evolution of the solar system. The mass of the Sun is $M_{\mathrm{sun}}=M_{\odot}=2\times 10^{30}$ kg. The masses of all relevant planets and their distances from the sun are listed in the table here in kg and AU.

\begin{quote}
\begin{tabular}{ccc}
\hline
\multicolumn{1}{c}{ Planet } & \multicolumn{1}{c}{ Mass in kg } & \multicolumn{1}{c}{ Distance to  sun in AU } \\
\hline
Earth   & $M_{\mathrm{Earth}}=6\times 10^{24}$ kg     & 1AU                    \\
Jupiter & $M_{\mathrm{Jupiter}}=1.9\times 10^{27}$ kg & 5.20 AU                \\
Mars    & $M_{\mathrm{Mars}}=6.6\times 10^{23}$ kg    & 1.52 AU                \\
Venus   & $M_{\mathrm{Venus}}=4.9\times 10^{24}$ kg   & 0.72 AU                \\
Saturn  & $M_{\mathrm{Saturn}}=5.5\times 10^{26}$ kg  & 9.54 AU                \\
Mercury & $M_{\mathrm{Mercury}}=3.3\times 10^{23}$ kg & 0.39 AU                \\
Uranus  & $M_{\mathrm{Uranus}}=8.8\times 10^{25}$ kg  & 19.19 AU               \\
Neptun  & $M_{\mathrm{Neptun}}=1.03\times 10^{26}$ kg & 30.06 AU               \\
Pluto   & $M_{\mathrm{Pluto}}=1.31\times 10^{22}$ kg  & 39.53 AU               \\
\hline
\end{tabular}
\end{quote}

\noindent
Pluto is no longer considered  a planet, but we add it here for historical reasons. It is optional in this project to include Pluto and eventual moons.

In setting up the equations we can limit ourselves to a co-planar motion and use only the $x$ and $y$ coordinates. But you should feel free to extend your equations to three dimensions, it is not very difficult and the data from NASA are all in three dimensions.

\href{{http://www.nasa.gov/index.html}}{NASA} has an excellent site at \href{{http://ssd.jpl.nasa.gov/horizons.cgi#top}}{\nolinkurl{http://ssd.jpl.nasa.gov/horizons.cgi\#top}}.
From there you can extract initial conditions in order to start your differential equation solver.
At the above website you need to change from \textbf{OBSERVER} to \textbf{VECTOR} and then write in the planet you are interested in.
The generated data contain the $x$, $y$ and $z$ values as well as their corresponding velocities. The velocities are in units of AU per day.
Alternatively they can be obtained in terms of km and km/s.

For the first system below involving only the Earth and the Sun, you could just initialize the position with say $x=1$ AU
and $y=0$ AU.

\paragraph{a: The Earth-Sun system.}
We assume that mass units can be obtained by using the fact that Earth's orbit is almost circular around the Sun.
For circular motion we know that the force must obey the following relation
\[
F_G= \frac{M_{\mathrm{Earth}}v^2}{r}=\frac{GM_{\odot}M_{\mathrm{Earth}}}{r^2},
\]
where $v$ is the velocity of Earth.
The latter equation can be used to show that
\[
v^2r=GM_{\odot}=4\pi^2\mathrm{AU}^3/\mathrm{yr}^2.
\]

Discretize the above differential equations and set up an algorithm
for solving these equations using Euler's forward algorithm and the
so-called velocity Verlet method discussed in the lecture notes and
lecture slides.

\paragraph{b: Writing an object oriented code for the Earth-Sun system.}

Write then a program which solves the above differential equations for
the Earth-Sun system using Euler's method and the velocity Verlet
method.  Write these codes without object orientation first in order
to make sure everything is running correctly. Thereafter you should
start planning to object orient your code.  Try to figure out which
parts and operations could be written as classes and generalized.
Your task here is to think of the program flow and figure out which
parts can be abstracted and reused for many types of operations.

For those of you who will focus on the Molecular Dynamics version of
Project 5, much of the structures developed here as well as the
implementation of the Verlet algorithm, can be used in that project as
well.

\paragraph{c: Test of the algorithms.}

Find out which initial value for the velocity that gives a circular
orbit and test the stability of your algorithm as function of
different time steps $\Delta t$.  Make a plot of the results you
obtain for the position of the Earth (plot the $x$ and $y$ values
and/or if you prefer to use three dimensions the $z$-value as well)
orbiting the Sun.

Check also for the case of a circular orbit that both the kinetic and the potential energies are conserved.
Check also if the  angular momentum is conserved. Explain why these quantities
should be conserved.

Discuss eventual differences between the Verlet algorithm and the
Euler algorithm. Consider also the number of FLOPs involved and
perform a timing of the two algorithms for equal final times.

We will use the velocity Verlet algorithm in the remaining part of the project.

\paragraph{d: Escape velocity.}
Consider then a planet which begins at a distance of 1 AU from the sun. Find out by trial and error
what the initial velocity must be in order for the planet to escape from the sun.  Can you find an exact answer?  How does that match your numerical results?

Try also to change the gravitional force, by replacing
\[
F_G=\frac{GM_{\odot}M_{\mathrm{Earth}}}{r^2},
\]
with
\[
F_G=\frac{GM_{\odot}M_{\mathrm{Earth}}}{r^{\beta}},
\]
where you let $\beta\in [2,3]$. What happens to the earth-sun system when $\beta$ creeps towards $3$? Comment your results.

\paragraph{e: The three-body problem.}
We will now study the three-body problem, still with the Sun kept fixed as the center of mass of the system  but
including Jupiter (the most massive planet in the solar system, having a mass that is approximately 1000 times
smaller than that of the Sun) together with the Earth. This leads to a three-body problem. Without Jupiter, the Earth's motion is stable and unchanging with time. The aim here is to find out how much Jupiter alters the Earth's motion.

The program you have developed can easily be modified by simply adding the magnitude of the force betweem the Earth and Jupiter.

This force is given again by
\[
F_{\mathrm{Earth-Jupiter}}=\frac{GM_{\mathrm{Jupiter}}M_{\mathrm{Earth}}}{r_{\mathrm{Earth-Jupiter}}^2},
\]
where $M_{\mathrm{Jupiter}}$ is the mass of the sun and $M_{\mathrm{Earth}}$ is the mass of Earth.
The gravitational constant is $G$ and $r_{\mathrm{Earth-Jupiter}}$ is the distance between Earth and Jupiter.

We assume again that the orbits of the two planets are co-planar, and we take this to be the $xy$-plane (you can easily extend the equations to three dimensions).
Modify your first-order differential equations in order to accomodate both the
motion of the Earth and Jupiter by taking into account the distance in $x$ and
$y$ between the Earth and Jupiter. Set up the algorithm and plot the positions of the Earth and Jupiter using the velocity Verlet algorithm.
Discuss the stability of the solutions using your Verlet solver.

Repeat
the calculations by increasing the mass of Jupiter by a factor of 10 and 1000
 and plot the position of the Earth.  Study again the stability of the Verlet solver.

\paragraph{f: Final model for all planets of the solar system.}
Finally, using our Verlet solver, we carry out a real three-body calculation where all three systems,
the Earth, Jupiter and the Sun are in motion. To do this, choose the center-of-mass position of the three-body system as
the origin rather than the position of the sun. Give the Sun an initial velocity which makes the total momentum of the system exactly zero (the center-of-mass will remain fixed). Compare these results with those from the previous exercise and comment your results. Extend your program to include all planets in the solar system (if you have time, you can also include the various moons, but it is not required) and discuss your results. Use the above NASA link  to set up the initial positions and velocities for all planets.

\paragraph{g: The perihelion precession of Mercury.}

An important test of the general theory of relativity was to compare its prediction for the
perihelion precession of Mercury to the observed value. The observed value of the perihelion precession, when
all classical effects (such as the perturbation of the orbit due to gravitational attraction from the other planets) are
subtracted, is $43''$ ($43$ arc seconds) per century.

Closed elliptical orbits are a special feature of the Newtonian $1/r^2$ force. In general, any correction to the
pure $1/r^2$ behaviour will lead to an orbit which is not closed, i.e.~after one complete orbit around the Sun, the
planet will not be at exactly the same position as it started. If the correction is small, then each orbit around
the Sun will be almost the same as the classical ellipse, and the orbit can be thought of as an ellipse whose
orientation in space slowly rotates. In other words, the perihelion of the ellipse slowly precesses around the Sun.

You will now study the orbit of Mercury around the Sun, adding a general relativistic correction to the Newtonian
gravitational force, so that the force becomes
\[
F_G = \frac{GM_\mathrm{Sun}M_\mathrm{Mercury}}{r^2}\left[1 + \frac{3l^2}{r^2c^2}\right]
\]
where $M_\mathrm{Mercury}$ is the mass of Mercury, $r$ is the distance between Mercury and the Sun, $l=|\vec{r}\times\vec{v}|$ is the magnitude of Mercury's orbital angular momentum per unit mass,
and $c$ is the speed of light in vacuum. Run a simulation
over one century of Mercury's orbit around the Sun with no other planets present, starting with Mercury at perihelion on the $x$ axis.
Check then the value of the perihelion angle $\theta_\mathrm{p}$, using
\[
\tan \theta_\mathrm{p} = \frac{y_\mathrm{p}}{x_\mathrm{p}}
\]
where $x_\mathrm{p}$ ($y_\mathrm{p}$) is the $x$ ($y$) position of Mercury at perihelion, i.e.~at the point
where Mercury is at its closest to the Sun. You may use that the speed of Mercury at perihelion is $12.44\,\mathrm{AU}/\mathrm{yr}$, and that the distance to the Sun
at perihelion is $0.3075\,\mathrm{AU}$.
You need to make sure that the time resolution used in your simulation
is sufficient, for example by checking that the perihelion precession you get with a pure Newtonian force is at least
a few orders of magnitude smaller than the observed perihelion precession of Mercury. Can the observed perihelion
precession of Mercury be explained by the general theory of relativity?

\end{document}


\chapter{Interpolating with Polynomials}

Let's say someone on a distant planet records video of a hammer being
throw up into the air.  They send you three random frames of the
hammer in flight. Each frame has a timestamp and you can clearly see
how high the hammer is in each one. Can you create a 2nd degree
polynomial that explains the entire flight of the hammer?

That is, you have three points $(t_0, h_0), (t_1, h_1), (t_2, h_2)$.
Can you find $a,b,c$ such that the graph of $at^2 + bt + c = t$ passes
through all three points?

The answer is yes.  In fact, given any $n$ points, there is exactly
one $n-1$ degree polynomial that passes through all the points.

There are a lot of variables floating around. Let's make it concrete:
The photos are taken at $t = 2$ seconds, $t = 3$ seconds, and $t = 4$
seconds. In those photos, the height of the hammer is $5m$, $7m$, and
$6m$. So, we want our polynomial to pass through these points: (2, 5),
(3, 7), (4,6).

\includegraphics[width=0.7\textwidth]{interpolation.png}

How can you find that polynomial? Let's do it in small steps. Can you
create a 2nd degree polynomial that is not zero at $t = 2$, but is zero
at $t = 3$ and $t = 4$? Yes, you can: $(x - 3)(x - 4)$ has
exactly two roots at $t = 3$ and $t = 4$.  The value of this polynomial at
$t = 2$ is $(2 - 3)(2 - 4) = 2$. We really want it to be $5m$, so
we can divide the whole polynomial by 2 and multiply it by 5.

Now we have the polynomial:
\begin{equation*}
f_0(x) = \frac{5}{(2 - 3)(2 - 4)}(x - 3)(x - 4) = \frac{5}{2}x^2 - \frac{35}{2}x + 30
\end{equation*}
This is a second degree polynomial that is 5 at $t=2$ and 0 at $t=3$ and $t=4$.

Now we create a polynomial that is 7 at $t=3$ and 0 at $t= 2$ and $t=4$:
\begin{equation*}
f_1(x) = \frac{7}{(3 - 2)(3 - 4)}(x - 2)(x - 4) = -7x^2 +42x - 56
\end{equation*}

Finally, we create a polynomial that is 6 at $t=4$ and zero at $t=2$ and $t=3$:
\begin{equation*}
f_2(x) = \frac{6}{(4 - 2)(4 - 3)}(x - 2)(x - 3) = 3x^2 - 15x + 18
\end{equation*}

Adding these three polynomials together gives you a new polynomial that touches all three points:
\begin{equation*}
  f(x) = \frac{5}{2}x^2 - \frac{35}{2}x + 30  - 7x^2 + 42x - 56 + 3x^2 - 15x + 18  = -\frac{3}{2}x^2 + \frac{19}{2}x -8
\end{equation*}

You can test this with your \pytype{Polynomial} class. Create a file called \filename{test\_interpolation.py}. Add this code:
\begin{Verbatim}
from Polynomial import Polynomial
import matplotlib.pyplot as plt

in_x = [2,3,4]
in_y = [5,7,6]

pn = Polynomial([-8, 19/2, -3/2])
print(pn)

# These lists will hold our x and y values
x_list = []
y_list = []

# Starting x
current_x = 1.5

while current_x <= 4.5:
    # Evaluate pn at current_x
    current_y = pn(current_x)

    # Add x and y to respective lists
    x_list.append(current_x)
    y_list.append(current_y)

    # Move x forward
    current_x += 0.05

# Plot the curve
plt.plot(x_list, y_list)

# Plot black circles on the given points
plt.plot(in_x, in_y, "ko")
plt.grid(True)
plt.show()
\end{Verbatim}

You should get a nice plot that shows the graph of the polynomial
passing through those three points.

In general, then, if you give me any three points $(t_0, h_0), (t_1, h_1), (t_2, h_2)$, here is a second degree polynomial that pass through all three points:
\begin{equation*}
\frac{h_0}{(t_0 - t_1)(t_0 - t_2)}(x - t_1)(x - t_2) + \frac{h_1}{(t_1 - t_0)(t_1 - t_2)}(x - t_0)(x - t_2) + \frac{h_2}{(t_2 - t_0)(t_2 - t_1)}(x - t_0)(x - t_1)
\end{equation*}

What if you are given 9 points ($(t_0, h_0), (t_1, h_1), \ldots, (t_8,
h_8)$) and want to find a 8th degree polynomial that passes through
all of them? Just what you would expect:
\begin{equation*}
\frac{h_0}{(t_0 - t_1)(t_0 - t_2)\ldots(t_0 - t_8)}(x - t_1)(x - t_2)\ldots(x - t_8) + \ldots + \frac{h_8}{(t_8 - t_0)\ldots(t_8-t_7)}(x - t_0)\dots(x - t_7)
\end{equation*}

\textit{FIXME: Do I need to define summation and prod here?}

The general solution is, given $n$ points, the $n-1$ degree polynomial that goes through them is
\begin{equation*}
  y =\sum_{i=0}^{n}\left ( \prod_{\stackrel{\!0\leq j\leq n}{j\neq i}}\frac{x-t_j}{t_i-t_j}\right ) h_i
\end{equation*}

That would be tedious for a person to compute, but computers love this
stuff. Let's create a method that creates instances of Polynomial
using interpolation.

\section{Interpolating polynomials in python}

Your method will take two lists of numbers, one contains x-values and
the other contains y-values. So comment out the line that creates the
polynomial in \filename{test\_interpolation.py} and create it from two lists:
\begin{Verbatim}
in_x = [2,3,4]
in_y = [5,7,6]
# pn = Polynomial([-8, 19/2, -3/2])
pn = Polynomial.from_points(in_x, in_y)
print(pn)
\end{Verbatim}

Add the following method to your Polynomial class in \filename{Polynomial.py}
\begin{Verbatim}
    @classmethod
    def from_points(cls, x_values, y_values):
        coef_count = len(x_values)

        # Sums start with a zero polynomial
        sum_pn = Polynomial([0.0] * coef_count)
        for i in range(coef_count):

            # Products start with the constant 1 polynomial
            product_pn = Polynomial([1.0])
            for j in range(coef_count):

                # Must skip j=i
                if j != i:
                    # (1x - x_values[j]) has a root at x_values[j]
                    factor_pn = Polynomial([-1 * x_values[j], 1])
                    product_pn = product_pn * factor_pn

            # Scale so product_pn(x_values[i]) = y_values[i]
            scale_factor  = y_values[i] / product_pn(x_values[i])
            scaled_pn = scale_factor * product_pn

            # Add it to the sum
            sum_pn = sum_pn + scaled_pn

        return sum_pn
\end{Verbatim}

It should work exactly the same as before.  You should get the same
polynomial printed out as before. You shoud get the same plot of the
curve passing through the three points.

How about five points? Change \pyvar{in\_x} and \pyvar{in\_y} at the
start of \filename{test\_interpolation.py}:
\begin{Verbatim}
in_x = [1.7, 2, 2.7, 3.5, 4, 4.4]
in_y = [8, 12, 1, 4, -1, 6]
\end{Verbatim}

You should get a polynomial that passes through all five points:
\begin{equation*}
11.21x^5 - 171.05x^4 + 1019.44x^3 - 2957.53x^2 + 4161.78x - 2258.75
\end{equation*}
It should look like this:
\includegraphics[width=0.7\textwidth]{fiveinterp.png}


The goal of transient analysis is to compute the temperature profile \mq, which
is defined in \eref{temperature-profile}, that corresponds to a given power
profile \mp, which is defined in \eref{power-profile}, by solving the system
given in \eref{temperature-model-original}. Traditionally, this operation is
undertaken numerically \cite{skadron2003}; however, we are interested in
obtaining and working with an analytical solution to the system, since such a
solution has many advantages, as we shall see later on.

\subsection{\pasttitle}

Let us discuss an analytical approach to solving the system of differential
equations given in \eref{temperature-differential-original}. To begin with, the
system is rewritten as follows:
\[
  \frac{\d \tvs(t)}{\d t} = \tm{A} \tvs(t) + \m{C}^{-1} \tm{B} \vp(t)
\]
where
\[
  \tm{A} = -\m{C}^{-1} \m{G}.
\]
Next, we apply a technique that is taken from the family of exponential
integrators, which have good stability properties; the interested reader is
referred to \cite{hochbruck2010} for an overview. Multiplying both sides of the
above system of differential equations by $e^{-\tm{A} t}$ and noting that
\[
  e^{-\tm{A} t} \frac{\d \tvs(t)}{\d t} = \frac{\d e^{-\tm{A} t} \tvs(t)}{\d t} + e^{-\tm{A} t} \tm{A} \tvs(t),
\]
we obtain
\[
  \tvs(t) = e^{\tm{A} t} \tvs(0) + e^{\tm{A} t} \int_0^t e^{-\tm{A} \tau} \m{C}^{-1} \tm{B} \vp(\tau) \d \tau,
\]
which is the solution at time $t$ starting from the initial condition $\tvs(0)$
at time~0.

Imagine now that the power consumption of the processing elements does not
change over time: $\vp(t) = \vp$. In this case, the system is a system of linear
differential equations that has the following analytical solution:
\begin{equation} \elab{transient-solution-original}
  \tvs(t) = e^{\tm{A} t} \tvs(0) + \tm{A}^{-1} (e^{\tm{A} t} - \m{I}) \m{C}^{-1} \tm{B} \vp
\end{equation}
where $\m{I}$ is the identity matrix.

Suppose now that the sampling interval \dt of \mp is small enough so that the
power consumption in each interval $[t_i, t_{i + 1})$ can be reasonably
approximated by a constant equal to $\vp_i = \vp(t_i)$. The corresponding \mq
can then be found by applying the following recurrence derived from
\eref{transient-solution-original}:
\begin{equation} \elab{transient-recurrence-original}
  \tvs_{i} = \tm{E} \tvs_{i - 1} + \tm{F} \vp_i
\end{equation}
for $i = \range{1}{\ns}$ where
\begin{align*}
  & \tvs_0 = \v{0} = (\range{0}{0}), \\
  & \tm{E} = e^{\tm{A} \dt}, \text{ and} \\
  & \tm{F} = \tm{A}^{-1}(e^{\tm{A} \dt} - \m{I})\m{C}^{-1} \tm{B}.
\end{align*}
Note that, in order to obtain the actual \mq, the recurrence should be followed
by \eref{temperature-algebraic-original}, which involves two trivial algebraic
operations.

Similarly to the observations made in \cite{thiele2011, pagani2015}, our
experience shows that the approach to transient analysis described above
provides a significant performance improvement compared to iterative solutions
to systems of ordinary differential equations, such as the fourth-order
Runge--Kutta method \cite{press2007}. However, there is still room for
improvement, as we discuss next.

\subsection{\solutiontitle}
\slab{transient-solution}

Even though the matrices $\tm{E}$ and $\tm{F}$ have to be computed only once,
they necessitate two computationally problematic operations: the matrix
exponential and matrix inverse involving $\tm{A} \in \real^{\nn \times \nn}$,
which is a generic matrix. It is preferable to have a symmetric matrix $\m{A}
\in \real^{\nn \times \nn}$ when these operations are concerned, since such a
matrix admits the eigendecomposition, which can be seen in
\eref{eigendecomposition}. Having computed such a decomposition, the calculation
of the matrix exponential and matrix inverse becomes trivial as follows:
\begin{align}
  & e^{\m{A} \dt}
  = \m{U} e^{\m{\Lambda} \dt} \transpose{\m{U}}
  = \m{U} \, \diagonal{e^{\lambda_1 \dt}}{e^{\lambda_\nn \dt}} \transpose{\m{U}} \text{ and} \elab{matrix-exponential} \\
  & \m{A}^{-1}
  = \m{U} \m{\Lambda}^{-1} \transpose{\m{U}}
  = \m{U} \, \diagonal{\lambda_1^{-1}}{\lambda_\nn^{-1}} \transpose{\m{U}}. \elab{matrix-inverse}
\end{align}

In order to obtain such an $\m{A}$, we propose to perform an auxiliary
transformation. Recall first that the conductance matrix $\m{G}$ is a symmetric
matrix, which, intuitively, is due to the fact that, if node~$i$ is connected to
node~$j$ with a certain conductance, node~$j$ is also connected to node~$i$ with
the same conductance; see \fref{thermal-circuit}. However, $\tm{A} = -\m{C}^{-1}
\m{G}$ does not have this property. The desired symmetry can be kept intact
using the following substitution:
\[
  \begin{split}
    & \vs(t) = \m{C}^{\frac{1}{2}} \tvs(t) \text{ and} \\
    & \m{A} = -\m{C}^{-\frac{1}{2}} \m{G} \m{C}^{-\frac{1}{2}}
  \end{split}
\]
where $\m{A}$ is symmetric, since
\[
  \transpose{\m{A}}
  = -\transpose{(\m{C}^{-\frac{1}{2}} \m{G} \m{C}^{-\frac{1}{2}})}
  = -\transpose{(\m{C}^{-\frac{1}{2}})} \transpose{\m{G}} \transpose{(\m{C}^{-\frac{1}{2}})}
  = -\m{C}^{-\frac{1}{2}} \m{G} \m{C}^{-\frac{1}{2}}
  = \m{A}.
\]
Consequently, \eref{temperature-model-original} is rewritten as follows:
\begin{subnumcases}{\elab{temperature-model}}
  \frac{\d \vs(t)}{\d t} = \m{A} \vs(t) + \m{B} \vp(t) \elab{temperature-differential} \\
  \vq(t) = \transpose{\m{B}} \vs(t) + \vq_\ambient \elab{temperature-algebraic}
\end{subnumcases}
where
\[
  \m{B} = \m{C}^{-\frac{1}{2}} \tm{B}.
\]
Similarly, the solution in \eref{transient-solution-original} becomes
\[
  \vs(t) = e^{\m{A} t} \vs_0 + \m{A}^{-1} (e^{\m{A} t} - \m{I}) \m{B} \v{p},
\]
and the recurrence in \eref{transient-recurrence-original} becomes
\begin{equation} \elab{transient-recurrence}
  \vs_i = \m{E} \vs_{i - 1} + \m{F} \vp_i
\end{equation}
for $i = \range{1}{\ns}$ where
\begin{align*}
  & \vs_0 = \v{0}, \\
  & \m{E} = e^{\m{A} \dt}, \text{ and} \\
  & \m{F} = \m{A}^{-1} \left(e^{\m{A} \dt} - \m{I}\right) \m{B}.
\end{align*}
Using the eigendecomposition in \eref{eigendecomposition}, the last equation can
be efficiently computed in the following way:
\[
  \m{F} = \m{U} \, \diagonal{\frac{e^{\lambda_1 \dt} - 1}{\lambda_1}}{\frac{e^{\lambda_\nn \dt} - 1}{\lambda_\nn}} \transpose{\m{U}} \m{B}.
\]
As before, the recurrence in \eref{transient-recurrence} should be followed by
\eref{temperature-algebraic} in order to obtain \mq. The above auxiliary
transformation is helpful not only for transient analysis but also in other
contexts, as we shall see later on.

Let us note at this point that there have been attempts to simplify the
temperature model by making additional assumptions in order to reduce the size
of the circuit, thereby speeding up the solution process. For instance, the
techniques proposed in \cite{bao2010, rai2011} are targeting single-core
platforms, and the approach described in \cite{rao2009} is aimed at homogeneous
platforms and applications where the execution times of tasks are comparable
with the thermal time constant of the thermal package, which is in the order of
100~s. Such techniques can be combined with what we present in this thesis as
long as $\m{C}$ remains a diagonal matrix, and $\m{G}$ remains a symmetric
positive definite matrix.


% this is a comment!!!
% these don't appear in your final rendered document

\documentclass{article} % Starts an article
\usepackage{amsmath} % Imports amsmath

% enables text wrapping
\usepackage{url}

\usepackage{graphicx} % takes care of graphic including machinery

\usepackage{subfig}

\usepackage[margin=1in,letterpaper]{geometry} % decreases margins

\title{\LaTeX Introduction} % Title
% \LaTeX prints LaTeX all fancy like
\author{Jeffery Russell}

\begin{document} % Begins a document
  \maketitle % places document at start of file
  \LaTeX{} is a document preparation system for
  the \TeX{} typesetting program.

\section{Math}

	In \LaTeX, there are tons of options for writing math.
	Over time you will memorize a lot of them, however, TexMaker has a side panel with all the math symbols. Math can be done in-line with single dollar signs like this: $y = mx +b$. Isn't this cool! However, math can also be done in bigger blocks that you can then reference.

  % The following shows some math examples
  \begin{align}
  	\label{math1}
    E_0 &= mc^2 \\
    E &= \frac{mc^2}{\sqrt{1-\frac{v^2}{c^2}}}
  \end{align}

 Look at equation \ref{math1}. Isn't that a cool function! Math blocks can also be constructed using the double dollar sign. But, these don't get a reference number associated with them
\footnote{This is a footnote!}.

$$
\begin{bmatrix}
1 & 3\\
7 & 5
\end{bmatrix} \odot
\begin{bmatrix}
6 & 8\\
4 & 2
\end{bmatrix}\\*
$$

\section{Tables}

% use this website to generate tables: https://www.tablesgenerator.com/

\begin{table}[h!]
\centering
\begin{tabular}{l|l|l|l}
 &  Short & Medium & Long \\ \hline
 Topical & 1.0  &  0.6 & 0.2  \\ \hline
 User & 0.0  & 0.2  & 0.2
\end{tabular}
\caption{\label{tab:googlePerformance}Google Precision Results}
\end{table}

If I am talking about something in a table, I can refer to table \ref{tab:googlePerformance} like this\footnote{Table generated with help of website tablesgenerator.com}.
That is it!

\section{Figures} \label{sec:figures}

What about including cool images in your document that looks all professional like?

\begin{figure*}[h!]
    \centering  % this centers the graphic on the page
    \includegraphics[width=.8\textwidth]{data_visuzlization.png} % graphic has a width of 80% of the page
    \caption{Plot Shoing Age, Height, and Snowfolk Class} % text to display with figure
    \label{fig:dataViz} % label for you to cite the figure with
\end{figure*}

% What about the [h!] at the begining of figure?
% This parameter determines where the figure will be placed, there are several options:
%
% h -- place float here
% t -- place at top of page
% b -- place at bottom of page
% p -- put on special page for floats
% ! -- do what I specify and don't optimize page layout

After you have your figure, you can refer to it like math, and tables like this: figure \ref{fig:dataViz} is cool.
Why is this cool I hear you asking? Well, if you update the graphic, you just have to place it in this folder and overwrite the old one.
You don't have to search for it in a word document.
Also if you insert a graphic above figure \ref{fig:dataViz}, you don't need to re-label anything.

\subsection{Side By Side Figures} \label{sec:sidebyside}

The command subsection is used to create a subsection. There are also commands for subsubsections.
You can also reference sections like this: section \ref{sec:sidebyside}.
We can see that section \ref{sec:sidebyside} is inside section \ref{sec:figures}.
If you want to reference a section, you have to add a label to it that is on the same line as the section command.

\begin{figure*}[h!]
    \centering
    \subfloat{{\includegraphics[width=0.45\textwidth]{height_roc_graph.png}}}%
    \qquad
    \subfloat{{\includegraphics[width=0.45\textwidth]{age_roc_graph.png}}}%
    \caption{ROC Curves for Age and Height Classifications}%
    \label{fig:rocCurves}
\end{figure*}

Creating a side by side figure is very similar to how you would create a normal figure.
However, note that for this you need the package "subfig". Referencing this is the same:
figure \ref{fig:rocCurves}.

\subsection{Citations}

Wow, let's cite my favorite book \cite{DUMMY:1}.
It is typical to put all your bibtex citations in a file ending with the extension ".bib".
At the end of the document, you can create the bibliography using the "\\bibliography" command.
What is amazing about this is that you can then specify what type of bibilography it generates. IE: you can cite using ieee, MLA, APA, etc. This package takes care of all the formatting and ordering of your citations. Another cool thing is that, you can have loads of citations in your bib file, but, the references list will get generated using only the ones that you ended up using in your paper.

\newpage

\bibliography{example}
\bibliographystyle{ieeetr}

\end{document}


\section*{Exercise 1.1}
The number of all the conditions that selecting 200 people from 2000 people is
\[
    N_1=\binom{2000}{120}.
\]
The number of the conditions that my friend and I are selected is equal to the number of conditions that selecting 118 people from 1998 people. That is
\[
    N_2=\binom{1998}{118}.
\]
Hence, the probability that my friend and I are selected is
\[
    P=\frac{\binom{1998}{118}}{\binom{2000}{120}}=\frac{1998!}{118!1880!}\div\frac{2000!}{120!1880!}\approx0.00357.
\]

\section*{Exercise 1.2}
\enum{
\item
We denote that $C=B\backslash A$, which means $A\cap C=\varnothing$.

Since $P[\bigcup_k A_k]=\sum_k P[A_k]$, then $P[B]=P[A\cup C]=P[A]+P[C]$.

Since $P[C]\geq 0$, then $P[B]\geq P[A]$.

Hence $P[A]\leq P[B]$.

\item
Since $A$ and $B$ are independent, then $P[A\cap B]=P[A]P[B]>0$.

Hence $A\cap B \neq \varnothing$.

Thus $A$ and $B$ are not mutually exclusive.

\item
We denote that $C=A\cap B$, so that
\[
    (A\backslash C)\cap(C)\cap(B\backslash C)=\varnothing.
\]

Since $P[\bigcup_k A_k]=\sum_k P[A_k]$ for $\bigcap_k A_k=\varnothing$, then

\spl{
    P[A\cup B]&=P[(A\backslash C)\cup C\cup(B\backslash C)]\\
    &=P[A\backslash C]+P[C]+P[B\backslash C]\\
    &=P[A\backslash C]+P[C]+P[B\backslash C]\textbf{+P[C]-P[C]}\\
    &=(P[A\backslash C]+P[C])+(P[B\backslash C]+P[C])-P[C]\\
    &=P[A]+P[B]-P[C]\\
    &=P[A]+P[B]-P[A\cap B].
}
}

\section*{Exercise 1.3}
\enum{
\item
No.

Assume that the probability of "heads" for one toss is $p$, then
\spl{
    &P[\text{two heads}]=p^2=\frac{1}{3}\\
    &P[\text{no head}]=(1-p)^2=\frac{1}{3}\\
    &P[\text{one head}]=1-(1-p)^2-p^2=\frac{1}{3}\\
}
There's no such $p$ satisfying the equations.

\item
No.

Assume that the probabilities of "heads" are respectively $p_1$ and $p_2$, then
\spl{
    &P[\text{two heads}]=p_1p_2=\frac{1}{3}\\
    &P[\text{no head}]=(1-p1)(1-p_2)=\frac{1}{3}\\
    &P[\text{one head}]=p_1(1-p_2)+p_2(1-p_1)=\frac{1}{3}\\
}
We obtain $
\left\{
\begin{aligned}
    p_1&=\frac{3+\sqrt{3}i}{6}\\
    p_2&=\frac{3-\sqrt{3}i}{6}\\
\end{aligned}
\right.
$

Hence there's no real root for $p_1$ and $p_2$.
}

\section*{Exercise 1.4}
\enum{
\item
The probability that the selected participant has been asked the first question is $\frac{1}{2}$; the probability that the selected participant has \textbf{not} been asked the first question is $\frac{1}{2}$.

We denote "claim to see the barn" as "B", "has been asked the first question" as "A".

\[
    P=P[B|A]P[A]+P[B|\urcorner A]P[\urcorner A]=0.17\times0.5+0.03\times0.5=0.1
\]

\item
Since
\spl{
    P[A]&=0.5\neq0\\
    P[B|A]&=0.17,\\
}
then
\[
    P[B]=0.1\neq P[B|A].
\]
Hence seeing the barn is not independent of being asked the first question.
}

\section*{Exercise 1.5}
We suppose that the chips we can get are only from the market. Also, we suppose that the stolen chips are all sold on the market.

Denote that "being defective" as "D" and "being stolen" as "S", so that
\[
    P[S|D]=\frac{P[D|S]P[S]}{P[D]}.
\]
Since the stolen chips are stolen before inspection, then
\begin{equation}\label{1}
    P[D|S]=0.5.
\end{equation}

For the probability that the chip we get was stolen,
\begin{equation}\label{2}
    P[S]=0.01.
\end{equation}

Finally,
\begin{equation}\label{3}
    P[D]=P[D|\urcorner S]P[\urcorner S]+P[D|S]P[S]=0.05\times0.99+0.5\times0.01=0.0545.
\end{equation}

According to Results \ref{1}, \ref{2} and \ref{3}, we obtain
\[
    P[S|D]=\frac{0.5\times0.01}{0.0545}=\frac{10}{109}\approx0.0917.
\]

\section*{Exercise 1.6}
The prisoner is right.

Denote that "A", "B", "C" means A, B, C is going to die, respectively; "B*" means B is told not to die by the warden.

For example, B is told not to die. Then we obtain
\spl{
    P[A|B*]&=\frac{P[B*|A]P[A]}{P[B*|A]P[A]+P[B*|B]P[B]+P[B*|C]P[C]}\\
    &=\frac{\frac{1}{2}\times\frac{1}{3}}{\frac{1}{2}\times\frac{1}{3}+0\times\frac{1}{3}+1\times\frac{1}{3}}=\frac{1}{3}.
}
That is $P[C|B*]=1-P[A|B*]-P[B|B*]=1-\frac{1}{3}-0=\frac{2}{3}$.

In this case, the chance that A is going to die does not change, but the chance that C is going to die doubles.

\section*{Exercise 1.7}
\enum{
\item
\[
m_X(t)=E[e^{tX}]=\sum_{k=1}^n e^{tx_k}\frac{1}{n}=\frac{1}{n}\sum_{k=1}{n}e^{tx_k},\quad t\in\mathbb{R}.
\]
\item
\spl{
    E[X]=\frac{\dd m_X(t)}{\dd t}\bigg|_{t=0}=\frac{1}{n}\sum_{k=1}^n x_ke^{tx_k}\bigg|_{t=0}=\frac{1}{n}\sum_{k=1}^n x_k.
}
\spl{
    Var[X]&=E[X^2]-E[X]^2=\frac{\dd^2 m_X(t)}{\dd t^2}\bigg|_{t=0}-(\frac{\dd m_X(t)}{\dd t}\bigg|_{t=0})^2\\
    &=\frac{1}{n}\sum_{k=1}^n x_k^2 e^{tx_k}\bigg|_{t=0}-(\frac{1}{n}\sum_{k=1}^n x_k)^2\\
    &=\frac{1}{n}\sum_{k=1}^nx_k^2-(\frac{1}{n}\sum_{k=1}^n x_k)^2.
}
}

\section*{Exercise 1.8}
Since there exists some $\varepsilon$ such that $m_X(t)=m_Y(t)$ for all $t\in (-\varepsilon,\varepsilon)$, then their high order of derivatives exist and are the same. In other words,
\[
    \left\{
    \begin{aligned}
        &E[X]=\frac{\dd m_X(t)}{\dd t}\bigg|_{t=0}=\frac{\dd m_Y(t)}{\dd t}\bigg|_{t=0}=E[Y]\\
        &E[X^2]=\frac{\dd^2 m_X(t)}{\dd t^2}\bigg|_{t=0}=\frac{\dd^2 m_Y(t)}{\dd t^2}\bigg|_{t=0}=E[Y^2]\\
        &\vdots\\
        &E[X^{n}]=\frac{\dd^n m_X(t)}{\dd t^n}\bigg|_{t=0}=\frac{\dd^n m_Y(t)}{\dd t^n}\bigg|_{t=0}=E[Y^{n}]\\
    \end{aligned}
    \right.
\]

We also know that
\begin{equation}\label{eq1}
    \left\{
    \begin{aligned}
        &E[X]=\sum_{x=0}^nxf_X(x)\\
        &E[X^2]=\sum_{x=0}^nx^2f_X(x)\\
        &\vdots\\
        &E[X^n]=\sum_{x=0}^nx^nf_X(x)\\
    \end{aligned}
    \right.
\end{equation}

Similarly, for $(Y,f_Y)$,
\begin{equation}\label{eq2}
    \left\{
    \begin{aligned}
        &E[Y]=\sum_{x=0}^nxf_Y(x)\\
        &E[Y^2]=\sum_{x=0}^nx^2f_Y(x)\\
        &\vdots\\
        &E[Y^n]=\sum_{x=0}^nx^nf_Y(x)\\
    \end{aligned}
    \right.
\end{equation}

We also know that
\begin{equation}\label{eq3}
\begin{split}
    &f_X(0)+f_X(1)+\hdots+f_X(n)=1,\\
    &f_Y(0)+f_Y(1)+\hdots+f_Y(n)=1.
\end{split}
\end{equation}

To simplify the calculations, we denote that
\[
    \textbf{E}_{(n+1)\times1}=\left(
    \begin{aligned}
        &E[Y]\\
        &E[Y^2]\\
        &\vdots\\
        &E[Y^n]\\
        &1\\
    \end{aligned}
    \right)
    =
    \left(
    \begin{aligned}
        &E[X]\\
        &E[X^2]\\
        &\vdots\\
        &E[X^n]\\
        &1\\
    \end{aligned}
    \right),
\]
\[
    \textbf{A}_{(n+1)\times(n+1)}=\left(
    \begin{tabular}{cccc}
        0 & 1 & $\hdots$ & n\\
        $0^2$ & $1^2$ & $\hdots$ & $n^2$\\
        &$\vdots$&&\\
        $0^n$ & $1^n$ & $\hdots$ & $n^n$\\
        1 & 1 & $\hdots$ & 1\\
    \end{tabular}
    \right),
\]

where $\textbf{det(A)}\neq0$.

\[
    \textbf{X}_{(n+1)\times1}=\left(
    \begin{aligned}
        &f_X(0)\\
        &f_X(1)\\
        &\vdots\\
        &f_X(n-1)\\
        &f_X(n)\\
    \end{aligned}
    \right),\quad
    \textbf{Y}_{(n+1)\times1}=
    \left(
    \begin{aligned}
        &f_Y(0)\\
        &f_Y(1)\\
        &\vdots\\
        &f_Y(n-1)\\
        &f_Y(n)\\
    \end{aligned}
    \right)
\]

Hence, equations \ref{eq1}, \ref{eq2}, \ref{eq3} can be presented as
\begin{equation}\label{eq4}
\begin{split}
    &\textbf{A}\textbf{X}=\textbf{E}\\
    &\textbf{A}\textbf{Y}=\textbf{E}
\end{split}
\end{equation}

We can tell that Eqs. \ref{eq4} are the same equation system. Since $\textbf{det(A)}\neq0$, we find that the equation system obtains only one solution.

Thus $\textbf{X}=\textbf{Y}$; that is
\begin{equation*}
    \left\{
    \begin{aligned}
        &f_X(0)=f_Y(0)\\
        &f_X(1)=f_Y(1)\\
        &\vdots\\
        &f_X(n)=f_Y(n)\\
    \end{aligned}
    \right.
\end{equation*}

In other words, $f_X(x)=f_Y(x)$ for $x=0,1,\hdots,n$.

\section*{Exercise 1.9}
\enum{
\item
Applying the total probability formula, we obtain
\[
    P[Z=z]=\sum_{y=1}^zP[Z=z|Y=y]P[Y=y].
\]
Since X and Y are independent, then
\[
    P[Z=z|Y=y]=P[X=z-y|Y=y]=P[X=z-y].
\]

Thus
\[
    P[Z=z]=\sum_{y=1}^{z}P[X=z-y]P[Y=y]=\sum_{x+y=z}P[X=x]P[Y=y].
\]

\item
For a geometric random variable, $P_g[X=x]=(1-p)^{x-1}p$.

For a pascal distribution with $r=2$, $$P_p[X=x]=\binom{x-1}{1}p^2(1-p)^{x-2}=(x-1)p^2(1-p)^{x-2}$$.

The sum of two independent geometric random variables is
\spl{
    P[Z=X+Y]&=P[Z=z]\\
    &=\sum_{x+y=z}P_g[X=x]P_g[Y=y]\\
    &=\sum_{x+y=z}(1-p)^{x-1}p(1-p)^{y-1}p\\
    &=\sum_{x+y=z}(1-p)^{x+y-2}p^2\\
    &=\sum_{x+y=z}(1-p)^{z-2}p^2
}
Since $x=1,\hdots,z-1$ and $y=1,\hdots,z-1$, then there are $z-1$ terms of $(1-p)^{z-2}p^2$.
\[
    P[Z=z]=(z-1)(1-p)^{z-2}p^2=P_p[X=x].
\]

Hence, the sum of two independent geometric random variables follows a Pascal distribution with $r = 2$.
}