\lab{Gibbs Sampling and LDA}{Gibbs Sampling and LDA}
\objective{Understand the basic principles of implementing a Gibbs sampler. Apply this to Latent Dirichlet Allocation.}

\section*{Gibbs Sampling}
Gibbs sampling is an MCMC sampling method in which we construct a Markov chain which is used to sample from a desired joint (conditional) distribution
\begin{equation*}
\mathbb{P}(x_{1},\cdots,x_{n} | \mathbf{y}).
\end{equation*}
Often it is difficult to sample from this high-dimensional joint distribution, while it may be easy to sample from the one-dimensional
conditional distributions
\begin{equation*}
\mathbb{P}(x_{i} | \mathbf{x}_{-i}, \mathbf{y})
\end{equation*}
where $\mathbf{x}_{-i} = x_{1},\cdots,x_{i-1},x_{i+1},\cdots,x_{n}.$

\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{Gibbs Sampler}{}
    \State \textrm{Randomly initialize } $x_1,x_2,\ldots,x_n$.
    \For{$k = 1, 2, 3, \ldots$}
        \For{$i = 1, 2, \ldots,n$}
            \State \textrm{Draw } $x \sim \mathbb{P}(x_{i} | \mathbf{x}_{-i}, \mathbf{y})$
            \State \textrm{Fix } $x_i = x$
        \EndFor
        \State $\mathbf{x}^{(k)}= (x_1,x_2,\ldots,x_n)$
    \EndFor
\EndProcedure
\end{algorithmic}
\caption{Basic Gibbs Sampling Process.}
\label{alg:gibbs}
\end{algorithm}
A Gibbs sampler proceeds according to Algorithm \ref{alg:gibbs}.
Each iteration of the outer for loop is a \emph{sweep} of the Gibbs sampler, and the value of $\mathbf{x}^{(k)}$ after a sweep is a \emph{sample}.
This creates an irreducible, non-null recurrent, aperiodic Markov chain over the state space consisting of all possible $\mathbf{x}$.
The unique invariant distribution for the chain is the desired joint distribution
\begin{equation*}
\mathbb{P}(x_{1},\cdots,x_{n} | \mathbf{y}).
\end{equation*}
Thus, after a burn-in period, our samples $\mathbf{x}^{(k)}$ are effectively samples from the desired distribution.

Consider the dataset of $N$ scores from a calculus exam in the file \texttt{examscores.npy}.
We believe that the spread of these exam scores can be modeled with a normal distribution of mean $\mu$ and variance $\sigma^{2}$.
Because we are unsure of the true value of $\mu$ and $\sigma^2$, we take a Bayesian approach and place priors on each parameter to quantify this uncertainty:
\begin{align*}
\mu & \sim N(\nu, \tau^{2})\quad &&\text{(a normal distribution)} \\
\sigma^{2} & \sim IG(\alpha, \beta) &&\text{(an inverse gamma distribution)}
\end{align*}
Letting $\mathbf{y} = (y_1,\ldots,y_N)$ be the set of exam scores, we would like to update our beliefs of $\mu$ and $\sigma^2$ by sampling from the posterior
distribution
\begin{equation*}
\mathbb{P}(\mu, \sigma^{2} | \mathbf{y}, \nu, \tau^{2}, \alpha, \beta).
\end{equation*}
Sampling directly can be difficult. However, we \emph{can} easily sample from the following conditional distributions:
\begin{align*}
\mathbb{P}(\mu | \sigma^{2}, \mathbf{y}, \nu, \tau^{2}, \alpha, \beta) & = \mathbb{P}(\mu | \sigma^{2}, \mathbf{y}, \nu, \tau^{2})\\
\mathbb{P}(\sigma^{2} | \mu, \mathbf{y}, \nu, \tau^{2}, \alpha, \beta) & = \mathbb{P}(\sigma^{2} | \mu, \mathbf{y}, \alpha, \beta)
\end{align*}
The reason for this is that these conditional distributions are \emph{conjugate} to the prior distributions, and hence are part of the same distributional
families as the priors. In particular, we have
\begin{align*}
\mathbb{P}(\mu | \sigma^{2}, \mathbf{y}, \nu, \tau^{2}) &= N(\mu^*, (\sigma^*)^2)\\
\mathbb{P}(\sigma^{2} | \mu, \mathbf{y}, \alpha, \beta) &= IG(\alpha^*, \beta^*),
\end{align*}
where
\begin{align*}
(\sigma^*)^2 &= \left(\frac{1}{\tau^2}+\frac{N}{\sigma^2}\right)^{-1}\\
\mu^* &= (\sigma^*)^2\left(\frac{\nu}{\tau^2} + \frac{1}{\sigma^2}\sum_{i=1}^N y_i \right)\\
\alpha^* &= \alpha + \frac{N}{2}\\
\beta^* &= \beta + \frac{1}{2}\sum_{i=1}^N (y_i-\mu)^2
\end{align*}
We have thus set this up as a Gibbs sampling problem, where we have only to alternate between sampling $\mu$ and sampling $\sigma^{2}$.
We can sample from a normal distribution and an inverse gamma distribution as follows:
\begin{lstlisting}
>>> from math import sqrt
>>> from scipy.stats import norm
>>> from scipy.stats import invgamma
>>> mu = 0. # the mean
>>> sigma2 = 9. # the variance
>>> normal_sample = norm.rvs(mu, scale=sqrt(sigma))
>>> alpha = 2.
>>> beta = 15.
>>> invgamma_sample = invgamma.rvs(alpha, scale=beta)
\end{lstlisting}
Note that when sampling from the normal distribution, we need to set the \li{scale} parameter to the standard deviation, \emph{not} the variance.

\begin{problem}
Write a function that accepts data $\y$, prior parameters $\nu$, $\tau^2$, $\alpha$, and $\beta$, and an integer $n$.
Use Gibbs sampling to generate $n$ samples of $\mu$ and $\sigma^2$ for the exam scores problem.

Test your sampler with priors $\nu=80$, $\tau^{2} = 16$, $\alpha = 3$, and $\beta = 50$, collecting $1000$ samples.
Plot your samples of $\mu$ and your samples of $\sigma^{2}$.
They should each to converge quickly.
\end{problem}

We'd like to look at the posterior marginal distributions for $\mu$ and $\sigma^2$.
To plot these from the samples, use a kernel density estimator from \li{scipy.stats}.
If our samples of $\mu$ are called \li{mu_samples}, then we can do this with the following code.
\begin{lstlisting}
>>> import numpy as np
>>> from matplotlib import pyplot as plt
>>> from scipy.stats import gaussian_kde

>>> mu_kernel = gaussian_kde(mu_samples)
>>> x = np.linspace(min(mu_samples) - 1, max(mu_samples) + 1, 200)
>>> plt.plot(x, mu_kernel(x))
>>> plt.show()
\end{lstlisting}

\begin{figure}[H]
    \begin{subfigure}[b]{.49\textwidth}
        \includegraphics[width=\textwidth]{figures/mu_posterior.pdf}
        \caption{Posterior distribution of $\mu$.}
    \end{subfigure}
    \begin{subfigure}[b]{.49\textwidth}
        \includegraphics[width=\textwidth]{figures/sigma2_posterior.pdf}
        \caption{Posterior distribution of $\sigma^2$.}
    \end{subfigure}
\caption{Posterior marginal probability densities for $\mu$ and $\sigma^2$.}
\label{fig:post}
\end{figure}

Keep in mind that the plots above are of the posterior distributions of the \emph{parameters}, not of the scores. If we would like to compute the posterior distribution of a new exam score $\tilde{y}$ given our data $\mathbf{y}$ and prior parameters, we compute what is known as the \emph{posterior predictive distribution}:
\begin{equation*}
\mathbb{P}(\tilde{y} | \mathbf{y}, \lambda) = \int_{\Theta} \mathbb{P}(\tilde{y} | \Theta)\mathbb{P}(\Theta | \mathbf{y}, \lambda) d\Theta
\end{equation*}
where $\Theta$ denotes our parameters (in our case $\mu$ and $\sigma^{2}$) and $\lambda$ denotes our prior parameters (in our case $\nu, \tau^{2}, \alpha,$ and $\beta$).

Rather than actually computing this integral for each possible $\tilde{y}$, we can do this by sampling scores from our parameter samples. In other words, sample
\begin{equation*}
\tilde{y}_{(t)} \sim N(\mu_{(t)}, \sigma_{(t)}^{2})
\end{equation*}
for each sample pair $\mu_{(t)}, \sigma_{(t)}^{2}$. Now we have essentially drawn samples from our posterior predictive distribution, and we can use a kernel density estimator to plot this distribution from the samples.

\begin{figure}[H]
    \includegraphics[width=.7\textwidth]{figures/predictiveposterior.pdf}
    \caption{Predictive posterior distribution of exam scores.}
    \label{fig:predictive}
\end{figure}

\begin{problem} % Visualize Gibbs sampler results.
Plot the kernel density estimators for the posterior distributions of $\mu$ and $\sigma^{2}$.
You should get plots similar to those in Figure \ref{fig:post}.

Next, use your samples of $\mu$ and $\sigma^{2}$ to draw samples from the posterior predictive distribution.
Plot the kernel density estimator of your sampled scores.
Compare your plot to Figure \ref{fig:predictive}.
\end{problem}

\section*{Latent Dirichlet Allocation}

Gibbs sampling can be applied to an interesting problem in natural language processing (NLP): determining which topics are prevalent in a document.
\emph{Latent Dirichlet Allocation} (LDA) is a generative model for a collection of text documents.
It supposes that there is some fixed vocabulary (composed of $V$ distinct terms) and $K$ different topics, each represented as a probability distribution $\phi_{k}$ over the vocabulary, each with a Dirichlet prior $\beta$.
This means $\phi_{k,v}$ is the probability that topic $k$ is represented by vocabulary term $v$.

With the vocabulary and topics chosen, the LDA model assumes that we have a set of $M$ documents (each ``document'' may be a paragraph or other section of the text, rather than a ``full'' document).
The $m$-th document consists of $N_m$ words, and a probability distribution $\theta_{m}$ over the topics is drawn from a Dirichlet distribution with parameter $\alpha$.
Thus $\theta_{m,k}$ is the probability that document $m$ is assigned the label $k$.
If $\phi_{k,v}$ and $\theta_{m,k}$ are viewed as matrices, their rows sum to one.

We will now iterate through each document in the same manner.
Assume we are working on document $m$, which you will recall contains $N_{m}$ words.
For word $n$, we first draw a topic assignment $z_{m,n}$ from the categorical distribution $\theta_{m}$, and then we draw a word $w_{m,n}$ from the categorical distribution $\phi_{z_{m,n}}$. Throughout this implementation, we assume $\alpha$ and $\beta$ are scalars. In summary, we have
\begin{enumerate}
    \item Draw $\phi_{k} \sim \text{Dir}(\beta)$ for $1 \leq k \leq K$.
    \item For $1 \leq m \leq M$:
    \begin{enumerate}
        \item Draw $\theta_{m} \sim \text{Dir}(\alpha)$.
        \item Draw $z_{m,n} \sim \text{Cat}(\theta_{m})$ for $1 \leq n \leq N_{m}$.
        \item Draw $w_{m,n} \sim \text{Cat}(\phi_{z_{m,n}})$ for $1 \leq n \leq N_{m}$.
    \end{enumerate}
\end{enumerate}

We end up with $n$ words which represent document $m$.
Note that these words are \emph{not} necessarily distinct from one another; indeed, we are most interested in the words that have been repeated the most.

This is typically depicted with graphical plate notation as in Figure \ref{fig:ldaplates}.
\begin{figure}[h]
\centering
\begin{tikzpicture}[>=stealth', dot/.style=
    {circle,fill=black,minimum size=3pt,inner sep=0pt, outer sep=-1pt} ]

\node[draw,minimum height=3.2cm, minimum width=2.1cm](r1)[]{};
\node[draw,minimum height=5cm, minimum width=2.4cm, node distance=
    .4cm](r2)[above of=r1]{};

\node[node distance=1.3cm](Nm)[below of=r1]{$1 \le n \le N_m$};
\node[node distance=2.28cm](M)[below of=r2]{$1 \le m \le M$};

\node[node distance = 2.7cm](dummy3)[left of=r1]{};
\node[draw,minimum height=2cm, minimum width=2.1cm, node distance=
    .55cm](r3)[ below of =dummy3]{};
\node[node distance=.7cm](K)[below of=r3]{$1 \le k \le K$};

\node[node distance=.6cm](dummy)[above right of =Nm]{};
\node[circle, draw,  inner sep=1pt, fill=black!25!,node distance=.4cm]
    (w)[above of=dummy]{$w_{m,n}$};
\node[circle, draw,  inner sep=1pt, node distance=1.3cm](z)[above
    of=w]{$z_{m,n}$};
\node[circle, draw,  inner sep=1pt, node distance=1.3cm]
    (theta)[above of=z]{$\vec{\theta}_{m}$};

\node[node distance=1.2cm, inner sep=0pt](alpha)[above of=
    theta]{$\vec{\alpha}$};

\node[node distance=.6cm](dummy2)[above right of=K]{};
\node[node distance=.35cm, circle, inner sep=1pt, draw](phi)[above
    of=dummy2]{$\vec{\phi_k}$};
\node[node distance=1.4cm, inner sep=0pt](beta)[above of = phi]{$\vec{\beta}$};

\foreach \x/\y in {alpha/theta, theta/z, z/w, beta/phi, phi/w} \draw[->](\x)--(\y);

\end{tikzpicture}
\caption{Graphical plate notation for LDA text generation.}
\label{fig:ldaplates}
\end{figure}

In the plate model, only the variables $w_{m,n}$ are shaded, signifying that these are the only observations visible to us; the rest are latent variables. Our goal is to estimate each $\phi_{k}$ and each $\theta_{m}$. This will allow us to understand what each topic is, as well as understand how each document is distributed over the $K$ topics. In other words, we want to predict the topic of each document, and also which words best represent this topic.
We can estimate these well if we know $z_{m,n}$ for each $m, n$, collectively referred to as $\mathbf{z}$. Thus, we need to sample
$\mathbf{z}$ from the posterior distribution $\mathbb{P}(\mathbf{z} | \mathbf{w}, \alpha, \beta),$ where $\mathbf{w}$ is the collection words in the text corpus. Unsurprisingly, it is intractable to sample directly from the joint posterior distribution. However, letting $\mathbf{z}_{-(m,n)} = \mathbf{z}\setminus \{z_{m,n}\}$, the conditional posterior distributions
\[\mathbb{P}(z_{m,n} = k | \mathbf{z}_{-(m,n)}, \mathbf{w}, \alpha, \beta)\]
have nice, closed form solutions, making them easy to sample from.

These conditional distributions have the following form:
\begin{equation*}
\mathbb{P}(z_{m,n} = k | \mathbf{z}_{-(m,n)}, \mathbf{w}, \alpha, \beta) \propto \frac{(n_{(k,m,\cdot)}^{-(m,n)} + \alpha)(n_{(k, \cdot, w_{m,n})}^{-(m,n)} + \beta)}{n_{(k,\cdot,\cdot)}^{-(m,n)} + V \beta}
\end{equation*}
where
\begin{align*}
n_{(k,m,\cdot)} & = \mbox{ the number of words in document $m$ assigned to topic $k$} \\
n_{(k,\cdot,v)} & = \mbox{ the number of times term $v = w_{m,n}$ is assigned to topic $k$} \\
n_{(k,\cdot,\cdot)} & = \mbox{ the number of times topic $k$ is assigned in the corpus} \\
n_{(k,m,\cdot)}^{-(m,n)} & = n_{(k,m,\cdot)} - \indicator_{z_{m,n} = k} \\
n_{(k,\cdot,v)}^{-(m,n)} & = n_{(k,\cdot,v)} - \indicator_{z_{m,n} = k} \\
n_{(k,\cdot,\cdot)}^{-(m,n)} & = n_{(k,\cdot,\cdot)} - \indicator_{z_{m,n} = k}
\end{align*}

Thus, if we simply keep track of these count matrices, then we can easily create a Gibbs sampler over the topic assignments. This is actually a particular class of samplers known as \emph{collapsed Gibbs samplers}, because we have collapsed the sampler by integrating out $\theta$ and $\phi$.

We have provided for you the structure of a Python object \li{LDACGS} with several methods, listed at the end of the lab.
The object is already defined to have attributes \li{n\_topics}, \li{documents}, \li{vocab}, \li{alpha}, and \li{beta}, where \li{vocab} is a list of strings (terms), and documents is a list of dictionaries (a dictionary for each document). Each entry in dictionary $m$ is of the form $n : w$, where $w$ is the index in \li{vocab} of the $n^{th}$ word in document $m$.

Throughout this lab we will guide you through writing several more methods in order to implement the Gibbs sampler. The first step is to initialize our assignments, and create the count matrices $n_{(k,m,\cdot)}, n_{(k,\cdot,v)}$ and vector $n_{(k,\cdot,\cdot)}$.

\begin{problem}
Complete the method \li{initialize()}.
By randomly assigning initial topics, fill in the count matrices and topic assignment dictionary. In this method, you will initialize the count matrices (among other things). Note that the notation
provided in the code is slightly different than that used above. Be sure to understand how the formulae above
connect with the code.

To be explicit, you will need to initialize $nmz$, $nzw$, and $nz$ to be zero arrays of the correct size.
Then, in the second for loop, you will assign z to be a random integer in the correct range of topics.
In the increment step, you need to figure out the correct indices to increment by one for each of the three arrays.
Finally, assign $topics$ as given.
\end{problem}

The next method we need to write fully outlines a sweep of the Gibbs sampler.

\begin{problem}
Complete the method \li{_sweep()}, which needs to iterate through each word of each document. It should call on the method \li{_conditional()} to get the conditional distribution at each iteration.

Note that the first part of this method will undo what \li{initialize()} did.
Then we will use the conditional distribution (instead of the uniform distribution we used previously) to pick a more accurate topic assignment.
Finally, the latter part repeats what we did in \li{initialize()}, but does so using this more accurate topic assignment.
\end{problem}

\begin{comment}
Take out this problem to make the lab easier.
We need to write the method to create the appropriate conditional distribution.

\begin{problem}
Complete the method \li{_conditional()}.
It accepts arguments $m,w$ where $m$ is the document and $w$ is an index of \li{vocab}.
Don't forget to normalize to ensure you are actually returning a distribution!
\end{problem}
\end{comment}
We are now prepared to write the full Gibbs sampler.

\begin{problem}
Complete the method \li{sample()}.
The argument \emph{filename} is the name and location of a .txt file, where each line is considered a document.
The corpus is built by method \li{buildCorpus}, and stopwords are removed (if argument \emph{stopwords} is provided).
Burn in the Gibbs sampler, computing and saving the log-likelihood with the method \li{\_loglikelihood}.
After the burn in, iterate further, accumulating your count matrices, by adding \li{nzw} and \li{nmz} to \li{total\_nzw} and \li{total\_nmz} respectively, where you only add every \emph{sample\_rate}$^{th}$ iteration.
Also save each log-likelihood.
\end{problem}

You should now have a working Gibbs sampler to perform LDA inference on a corpus.
Let's test it out on one of Ronald Reagan's State of the Union addresses, found in \texttt{reagan.txt}.

\begin{problem}

Create an \li{LDACGS} object with $20$ topics, letting $\alpha$ and $\beta$ be the default values.
Run the Gibbs sampler, with a burn in of $100$ iterations, accumulating $10$ samples, only keeping the results of every $10$th sweep.
Use \texttt{stopwords.txt} as the stopwords file.
\end{problem}

Plot the log-likelihoods. How long did it take to burn in?

We can estimate the values of each $\phi_{k}$ and each $\theta_{m}$ as follows:

\begin{align*}
\widehat{\theta}_{m,k} & = \frac{n_{(k,m,\cdot)} + \alpha}{K \cdot \alpha + \sum_{k=1}^{K} n_{(k,m,\cdot)}} \\
\widehat{\phi}_{k,v} & = \frac{n_{(k,\cdot,v)} + \beta}{V \cdot \beta + \sum_{v=1}^{V} n_{(k,\cdot,v)}}
\end{align*}

We have provided methods \li{phi} and \li{theta} that do this for you. We often examine the topic-term distributions $\phi_{k}$ by looking at the $n$ terms with the highest probability, where $n$ is small (say $10$ or $20$).
We have provided a method \li{topterms} which does this for you.

\begin{problem}
Using the methods described above, examine the topics for Reagan's addresses. As best as you can, come up with labels for each topic.
If $ntopics=20$ and $n=10$, we will get the top $10$ words that represent each of the $20$ topics; for each topic, decide what these ten words jointly represent.
\end{problem}

We can use $\widehat{\theta}$ to find the paragraphs in Reagan's addresses that focus the most on each topic. The documents with the highest values of $\widehat{\theta}_{k}$ are those most heavily focused on topic $k$.
For example, if you chose the topic label for topic $p$ to be \emph{the Cold War}, you can find the five highest values in $\widehat{\theta_{p}}$, which will tell you which five paragraphs are most centered on the Cold War.
\begin{comment}
This problem was difficult for the first cohort, and since it does not add much to the lab it has been removed.

\begin{problem}
In your above topic analysis, you should have found a topic about the Cold War and one about education. Find the five paragraphs in Reagan's addresses that most closely focus on each of these topics, according to the above method.
\end{problem}
\end{comment}

Let's take a moment to see what our Gibbs sampler has accomplished.
By simply feeding in a group of documents, and with no human input, we have found the most common topics discussed, which are represented by the words most frequently used in relation to that particular topic.
The only work that the user has done is to assign topic labels, saying what the words in each group have in common.
As you may have noticed, however, these topics may or may not be \emph{relevant} topics.
You might have noticed that some of the most common topics were simply English particles (words such as \emph{a}, \emph{the}, \emph{an}) and conjunctions (\emph{and}, \emph{so}, \emph{but}).
Industrial grade packages can effectively remove such topics so that they are not included in the results.

\newpage

\section*{Additional Material} % ==============================================

\subsection*{LDACGS Source Code} % --------------------------------------------

\begin{lstlisting}
class LDACGS:
    """Do LDA with Gibbs Sampling."""

    def __init__(self, n_topics, alpha=0.1, beta=0.1):
        """Initialize system parameters."""
        self.n_topics = n_topics
        self.alpha = alpha
        self.beta = beta

    def buildCorpus(self, filename, stopwords_file=None):
        """Read the given filename and build the vocabulary."""
        with open(filename, 'r') as infile:
            doclines = [line.rstrip().lower().split(' ') for line in infile]
        n_docs = len(doclines)
        self.vocab = list({v for doc in doclines for v in doc})
        if stopwords_file:
            with open(stopwords_file, 'r') as stopfile:
                stops = stopfile.read().split()
            self.vocab = [x for x in self.vocab if x not in stops]
            self.vocab.sort()
        self.documents = []
        for i in range(n_docs):
            self.documents.append({})
            for j in range(len(doclines[i])):
                if doclines[i][j] in self.vocab:
                    self.documents[i][j] = self.vocab.index(doclines[i][j])

    def initialize(self):
        """Initialize the three count matrices."""
        self.n_words = len(self.vocab)
        self.n_docs = len(self.documents)

        # Initialize the three count matrices.
        # The (i,j) entry of self.nmz is the number of words in document i assigned to topic j.
        self.nmz = np.zeros((self.n_docs, self.n_topics))
        # The (i,j) entry of self.nzw is the number of times term j is assigned to topic i.
        self.nzw = np.zeros((self.n_topics, self.n_words))
        # The (i)-th entry is the number of times topic i is assigned in the corpus.
        self.nz = np.zeros(self.n_topics)

        # Initialize the topic assignment dictionary.
        self.topics = {} # key-value pairs of form (m,i):z

        for m in range(self.n_docs):
            for i in self.documents[m]:
                # Get random topic assignment, i.e. z = ...
                # Increment count matrices
                # Store topic assignment, i.e. self.topics[(m,i)]=z
                raise NotImplementedError("Problem 3 Incomplete")

    def sample(self,filename, burnin=100, sample_rate=10, n_samples=10, stopwords=None):
        self.buildCorpus(filename, stopwords)
        self.initialize()
        self.total_nzw = np.zeros((self.n_topics, self.n_words))
        self.total_nmz = np.zeros((self.n_docs, self.n_topics))
        self.logprobs = np.zeros(burnin + sample_rate*n_samples)
        for i in range(burnin):
            # Sweep and store log likelihood.
            raise NotImplementedError("Problem 5 Incomplete")
        for i in range(n_samples*sample_rate):
            # Sweep and store log likelihood
            raise NotImplementedError("Problem 5 Incomplete")
            if not i % sample_rate:
                # accumulate counts
                raise NotImplementedError("Problem 5 Incomplete")

    def phi(self):
        phi = self.total_nzw + self.beta
        self._phi = phi / np.sum(phi, axis=1)[:,np.newaxis]

    def theta(self):
        theta = self.total_nmz + self.alpha
        self._theta = theta / np.sum(theta, axis=1)[:,np.newaxis]

    def topterms(self,n_terms=10):
        self.phi()
        self.theta()
        vec = np.atleast_2d(np.arange(0,self.n_words))
        topics = []
        for k in range(self.n_topics):
            probs = np.atleast_2d(self._phi[k,:])
            mat = np.append(probs,vec,0)
            sind = np.array([mat[:,i] for i in np.argsort(mat[0])]).T
            topics.append([self.vocab[int(sind[1,self.n_words - 1 - i])] for i in range(n_terms)])
        return topics

    def toplines(self,n_lines=5):
        lines = np.zeros((self.n_topics,n_lines))
        for i in range(self.n_topics):
            args = np.argsort(self._theta[:,i]).tolist()
            args.reverse()
            lines[i,:] = np.array(args)[0:n_lines] + 1
        return lines

    def _removeStopwords(self, stopwords):
        return [x for x in self.vocab if x not in stopwords]

    def _conditional(self, m, w):
        dist = (self.nmz[m,:] + self.alpha) * (self.nzw[:,w] + self.beta) / (self.nz + self.beta*self.n_words)
        return dist / np.sum(dist)

    def _sweep(self):
        for m in range(self.n_docs):
            for i in self.documents[m]:
                # Retrieve vocab index for i-th word in document m.
                # Retrieve topic assignment for i-th word in document m.
                # Decrement count matrices.
                # Get conditional distribution.
                # Sample new topic assignment.
                # Increment count matrices.
                # Store new topic assignment.
                raise NotImplementedError("Problem 4 Incomplete")

    def _loglikelihood(self):
        lik = 0

        for z in range(self.n_topics):
            lik += np.sum(gammaln(self.nzw[z,:] + self.beta)) - gammaln(np.sum(self.nzw[z,:] + self.beta))
            lik -= self.n_words * gammaln(self.beta) - gammaln(self.n_words*self.beta)

        for m in range(self.n_docs):
            lik += np.sum(gammaln(self.nmz[m,:] + self.alpha)) - gammaln(np.sum(self.nmz[m,:] + self.alpha))
            lik -= self.n_topics * gammaln(self.alpha) - gammaln(self.n_topics*self.alpha)

        return lik
\end{lstlisting}


\section*{Exercises}

\begin{ex}
  What is an element of $\R^1$?
\end{ex}

\begin{ex} Given the points $P=(2,0,-4)$ and $Q=(5,-2,1)$, find $\longvect{PQ}$ and $\longvect{QP}$.

  \begin{sol}

    \begin{equation*}
      \longvect{PQ}  = \longvect{0Q} - \longvect{0P}= \begin{mymatrix}{c}
        5-2 \\
        -2-0 \\
        1-(-4)
      \end{mymatrix} = \begin{mymatrix}{c}
        3 \\
        -2 \\
        5
      \end{mymatrix}
    \end{equation*}

    \begin{equation*}
      \longvect{QP}  = \longvect{0P} - \longvect{0Q}= \begin{mymatrix}{c}
        2-5 \\
        0-(-2) \\
        (-4)-1
      \end{mymatrix} = \begin{mymatrix}{c}
        -3 \\
        2 \\
        -5
      \end{mymatrix}
    \end{equation*}

  \end{sol}
\end{ex}

\begin{ex} Find $x$ and $y$ so that $\vect{u}= \mat{5x-3y, 4}^T$ and
  $\vect{v}=\mat{2x-2y, 2y}^T$ are equal in $\R^2$.

  \begin{sol}
    We need $5x-3y=2x-2y$ and $4=2y$. The unique solution is
    $x=\frac{2}{3}$ and $y=2$.
  \end{sol}
\end{ex}


I am only considering subdivision surfaces based on triangles
\cite{hoppe-et-al-94,hoppe-thesis-94}.

\subsection{Approximating meshes}
\label{sec:Approximating-meshes}

A common approach to the use of subdivision surfaces is
to approximate the limit surface by the {\it subdivided mesh,} $\M^s$,
a $k$ times subdivided version of the {\it control mesh,} $\M^c$
(a typical value for $k$ is 2).
The positions of the $n^c$ vertices of the control mesh,
$\p^c = (\p^c_0 \ldots \p^c_{n-1}) \\in \Reals^{3n^c},$
and the $n^s$ vertices of the subdivided mesh,
$\p^s = (\p^s_0 \ldots \p^s_{n-1}) \in \Reals^{3n^s},$
are related by the {\it subdivision transform}
$\S : \Reals^{n^c} \mapsto \Reals^{n^s}$.
If
$\x^c = (x^c_0 \ldots x^c_{n-1}) \in \Reals^{n^c}$,
$\y^c = (y^c_0 \ldots y^c_{n-1}) \in \Reals^{n^c}$,
and
$\z^c = (z^c_0 \ldots z^c_{n-1}) \in \Reals^{n^c}$,
are the $x, y,$ and $z,$ coordinates of $\p^c$,
and $\x^s, \y^s, \z^s$ are the same coordinates
of $\p^s$, then
\begin{eqnarray}
\x^s & = & \S \x^c
\\
\y^s & = & \S \y^c
\nonumber
\\
\z^s & = & \S \z^c
\nonumber
\end{eqnarray}
We can use the above to define $\S_3 : \Reals^{3n^c} \mapsto \Reals^{3n^s}$,
so that
\begin{equation}
\p^s = \S_3 \p^c.
\end{equation}

If $f(\p^s) = f(\S_3 \p^c)$ is a penalty function applied to the subdivided mesh,
then the gradient with respect to the positions of
the vertices of the control mesh is simply:
\begin{equation}
\Gc{\p^c}{f(\S_3 \p^c)}{\q^c} = \S_3^{\dagger} \Gc{\p^s}{f(\p^s)}{\q^s = \S_3 \q^c}
\end{equation}


\section{Optimization}
\label{sec:RNN-optim}

As described previously, we make use of a reference offline labelling of the training data, marking the segments where an SWR is present. This reference labelling can be used to create a training signal $y_t$, that we want the RNN output $n_t$ to approach. As $n_t \in (0, 1)$ (see \cref{eq:out}), we can define for example $y_t = 1$ during SWR events, and $y_t = 0$ outside them. Alternatively, we can construct $y_t$ to be $1$ only around the start of each reference SWR segment.

Given a target signal $y_t$, and the actual RNN output $n_t$, we can compare them to quantify how close the RNN output matches the target signal. We choose cross-entropy for this comparison. The so called \emph{loss function} $\ell_t$ is then defined as:
%
\begin{equation}
\label{eq:loss}
\ell(n_t, y_t) = - y_t \log(n_t) - (1 - y_t) \log(1 - n_t)
\end{equation}
%
The loss $\ell_t$ is lower whenever the RNN output is more similar to the target signal. The mean loss $\expval{\ell_t}$ over some dataset can thus be used as a proxy for how useful the RNN output $n_t$ is for SWR detection. Just as in \cref{ch:GEVec}, we want to find the weights $w_i$ (the elements of the matrices $\W_\_$ and the vectors $\bias_\_$ from \cref{eq:out,eq:hnew}) that minimize this expected loss $\expval{\ell_t}$. Unlike in \cref{ch:GEVec}, there is no known way to find the global minimum of this function.

Instead, a form of gradient descent is used to iteratively and stochastically approach a local minimum of $\expval{\ell_t}$. We divided the training data into 300 ms long chunks. For each such chunk, the total loss $L = \sum_{300 \text{ms}} \ell_t$ is calculated. Then, the partial derivatives $\pdv*{L}{w_{i,t}}$ of this loss are calculated, for each parameter $w_i$ in the RNN, and for each timestep $t$ in the chunk. This calculation of the gradient of $L$ is done through the so called \emph{backpropagation} algorithm \cite{Rumelhart1986}, which is an application of the chain rule from calculus. In this case we apply so called ``backpropagation through time'' (BPTT) -- a common way to train RNN's \cite{Goodfellow2016}. Such derivative calculations are often done through an automatic differentiation program. In our case, we used the \texttt{PyTorch} library \cite{Paszke2017}.

For each parameter $w_i$ in the RNN, the BPTT algorithm yields a set of partial derivatives $\{ \pdv*{L}{w_{i,t_1}}, \pdv*{L}{w_{i,t_2}}, \tdots \}$ that each tell how much and in which direction the chunk loss $L$ changes when the parameter $w_i$ is increased at timestep $t$. Taking the mean of this set of partial derivatives over all timesteps in the chunk yields a value $\pdv*{L}{w_i}$ that can be used to tell how the parameter $w_i$ should be adapted to decrease the loss $L$:
%
\begin{equation}
\label{eq:SGD}
w_i \leftarrow w_i - \eta \pdv{L}{w_i}
\end{equation}
%
(i.e. if $\pdv*{L}{w_i} > 0$ then the loss would increase by increasing $w_i$; so decrease $w_i$). Doing this for all parameters $w_i$ of the RNN results in a complete so called gradient step. $\eta$ determines the step size, and is called the \emph{learning rate}. Having a separate and adaptive learning rate $\eta_{i,t}$ for each parameter has been shown to greatly increase convergence speed of stochastic gradient descent. We used the AdaMax optimization algorithm \cite{Kingma2014} to update our RNN parameters with such adaptive learning rates, using the default hyperparameters as suggested in the paper.

\section{Regularization}
\label{sec:RNN-regularization}

Repeating this procedure for all chunks, and for multiple passes over the training data, yields a trained RNN. The danger then exists that the network is \emph{overfit} to the training data; i.e. it achieves a low loss on the training data, but it performs badly on unseen data. We avoided this with so called \emph{early stopping} on a validation set.

As described in \cref{sec:recording}, the first 20 minutes of the 34 minutes-long recording were used as training data for data-driven online algorithms. For the RNN, these 20 minutes were further split into a part for training proper (the first 15.6 minutes), and a part for validation (the final 4.4 minutes). The RNN was trained for 50 passes over the proper training part. After each training pass (or \emph{epoch}), the loss on the validation set was calculated, as an estimate of how the RNN would perform on unseen data. The RNN where the validation loss was lowest was then chosen for the final evaluation of online SWR detection performance on the held-out test data. See \cref{fig:validloss} for an example validation loss curve over training time.

\begin{figure}
\img[0.62]{validloss_fullrect}
\captionn{Regularization by early stopping}{Evolution of total loss on the validation data during training of the RNN of \cref{fig:RNN-envelopes}.}
\label{fig:validloss}
\end{figure}

% todo: dropout


\subsection{Homomorphisms form a vector space}

If we can can show that scalars can act on morphisms, then we can shwn that morphisms on a vector space are themselves a vector space.

Scalars can act on morphisms, and so morphisms of vector spaces are themselves vector spaces.

\subsubsection{Dimensions of homomorphisms}

We can identify the dimensionality of this new vector space from the dimensions of the original vector spaces.

\(\dim (\hom(V, W))=\dim V \dim W\)


%---------------------------Shape and Size-----------------------------
\section{Shape and Size\label{s:tet-shape-and-size}}

Let $S$ be the shape as defined in \S\ref{s:tet-shape}
and $R$ be the relative size squared as defined in \S\ref{s:tet-rel-size-squared}.
Then the shape and size metric is
\begin{displaymath}
q = S R
\end{displaymath}

\tetmetrictable{shape and size}%
{$1$}%                                        Dimension
{$[0.2,1]$}%                                  Acceptable range
{$[0,1]$}%                                    Normal range
{$[0,1]$}%                                    Full range
{Dependent on $\overline{V}$}%                Equilateral tet
{\cite{knu:03}}%                              Citation
{v\_tet\_shape\_and\_size}%                            Verdict function name


\section{Pinhole camera}
\label{sec:pinhole_camera}
Pinhole camera is the simplest camera model, where light passes through a tiny hole (from which the name \textit{pinhole camera}) of a box: an inverted image of the scene is projected on the opposite side of the box itself. This effect, known as \textit{camera obscura effect}, was studied since 500 BC (the first writings are back to the chinese Mozi) and it is the underlying principle of the $19^{th}$ century cameras. An example of the geometry of this camera is shown in Figure \ref{fig::pinhole}: as it can be seen, for each 3D point there is only one ray of light that passes through the pinhole. This is an ideal condition which allows to neglect distortions, such as blurring. Furthermore it is free of lenses, a condition that accords you to neglect distortions such as vignetting or radial and tangential distortions.
\begin{figure}[t!]
  \centering
  \begin{minipage}[c]{.48\textwidth}
  	\centering
    \includegraphics[width=\textwidth]{./images/tech/pinhole.png}
    \caption{The geometry of a \\ pinhole camera}
    \label{fig::pinhole}
  \end{minipage}
  \hfill
  \begin{minipage}[c]{.48\textwidth}
  	\centering
    \includegraphics[width=\textwidth]{./images/tech/image_plane.png}
    \caption{Mathematical model}
    \label{fig:math_model}
  \end{minipage}
  %\caption{Examples of different types of occlusions}
  %\label{fig:occlusions}
\end{figure}

Thanks to these simplifications, the mathematical model that describes the relations between the 3D world points and their projections in the image, is very simple. Let's look at the Figure \ref{fig:math_model}.
% Let us define the \textit{camera coordinate system} $\left(e_x', e_y', e_z' \right)$: the origin $c = \left(0,0,0 \right)$ will represent the so called camera center (i.e. the pinhole).
% We form the line between $X = \left( X_1^w, X_2^w, X_3^w \right)$ and $c$ and intersect it with the plane $z = k$ called \textit{image plane}, to generate a projection $x = (x_1, x_2, k)$ of a scene point $X$. We will refer to $e_Z$ as the \textit{viewing direction}.
% Note that if $k > 0$, the image plane is placed in front of the camera center and the image will not appear upside down. In this case we talk about \textit{virtual image}, but in real models we put $k < 0$. \\
% Since $Xc$ is a direction vector of the viewing ray we can parametrize it by the expression
%  \begin{equation*}
%    c + s(X - c) = sX \qquad s \in \mathds{R}
%  \end{equation*}
% where $sX_3^w = k$ (intersection with the plane $z = k$). Note that this model does not take into account the scene projection from the image plane to the sensor plane. \\
Let $\left(e_x', e_y', e_z' \right)$ be the \textit{camera coordinate system}, centered in $C = \left(0,0,0 \right)$, and let $C$ be the camera center (i.e. the pinhole). Then, let us define the \textit{image plane} as the 2D plane in the world in which the sensor lies. The point 2D $x = (x_1, x_2)$ in the image plane, is related to the 3D world point $X' = (X'_1, X'_2, X'_3)$ through a linear pathway for point $C$. We can parametrize this transformation with the expression:
  \begin{equation}
    \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = - \frac{f}{X'_3} \begin{pmatrix} X'_1 \\ X'_2 \end{pmatrix}
    \label{eq:image-plane}
  \end{equation}
where $f$ is the \textit{focal length} (the distance from the pinhole to which the rays are focused) of the ideal camera. This is a very simple model, but a point in the image plane does not correspond to a unique point in the world as there are three unknowns on the right hand side. Thanks to the collinearity condition from the points $X'$, $x$ and $C$, the \acs{DLT} (Direct Linear Transformation, an algorithm used to determine a set of variables from a set of similarity relations) is used: it is simple to solve this intersection and to determine this last projection. \\

However, this case is unrealistically simple, because the object and image plane are parallel. In real applications, the point $X$ can lie on a plane that have an arbitrary position and rotation, with respect to the image plane. Let us put this new plane into a \textit{global coordinate reference system} $\left( e_x, e_y, e_z \right)$; all points in the 3D world and all the camera movements will be related to this system.
% In Figure \ref{fig:math_model} we can see how point $X$ is projected in the image plane, but a questions arise: how we can locate $X$ in the world and where image plane is located respecting to the world? To answer to these questions we have to introduce a new \textit{global coordinate reference system} $\left( e_x, e_y, e_z \right)$; all points in the 3D world and all the camera movements will be related to this system.
In this way the projection of the point $X$ in the camera reference system $\left( e_x', e_y', e_z' \right)$ is a simple rotation and translation, that in \textit{homogeneous coordinates} is:
  \begin{equation}
    \label{eq:extrinsic}
    \begin{pmatrix}
      X'_1 \\ X'_2 \\ X'_3
    \end{pmatrix}
    =
    \begin{bmatrix}
      R & t
    \end{bmatrix}
    \begin{pmatrix}
      X_1 \\ X_2 \\ X_3 \\ 1
    \end{pmatrix}
    =
    H
    \begin{pmatrix}
      X_1 \\ X_2 \\ X_3 \\ 1
    \end{pmatrix}
  \end{equation}
where $R$ is a $3 \times 3$ rotation matrix and $t$ a $3 \times 1$ translation vector, and they are referred to as \textit{extrinsic parameters}, while the matrix $H$ is called \textit{homography matrix}. This is the first projection shown in Figure \ref{fig:perspective_projection}.
\begin{figure}[t!]
  \centering
  \includegraphics[width=\textwidth]{./images/tech/perspective_projection.PNG}
  \caption{Projection chain from the point $X$ in the world reference system, to $x'$ in the sensor reference system.}
  \label{fig:perspective_projection}
\end{figure} \\

The Equation \ref{eq:image-plane} shown us that the image plane is embedded in $\mathds{R}^3$, so we need to project the point $x'$ in the $\mathds{N}^2$ sensor coordinate system (in pixel unit). This is possible using a $3 \times 3$ matrix $K$ of the \textit{intrinsic parameters}
  \begin{equation*}
    \label{eq:intrinsic_matrix}
    K =
    \begin{pmatrix}
      \gamma_1	& s			& c_x \\
      0			& \gamma_2	& c_y \\
      0			& 0				& 1
    \end{pmatrix}
  \end{equation*}
where $\left( c_x, c_y \right)$ are the coordinates of the point $C$ in the sensor reference system. The pair $\left( \gamma_1, \gamma_2 \right)$ are scale factors that translate the image plane unit ($mm$) into sensor unit ($pixel$). Finally, $s$ is called skew factors, and it forces sensor rows and columns to be perpendicular. These are properties of the used camera and are related to non-ideality of camera construction. The projection (the second in Figure \ref{fig:perspective_projection}) is performed as follow:
  \begin{equation}
    \label{eq:intrinsic}
    \begin{pmatrix}
      x'_1 \\ x'_2 \\ 1
    \end{pmatrix}
    = K
    \begin{pmatrix}
      x_1 \\ x_2 \\ 1
    \end{pmatrix}
  \end{equation} \\

The chain of all these projections, shown in Figure \ref{fig:perspective_projection}, is called \textit{perspective projection}. A common way to indicate Equations \ref{eq:extrinsic} and \ref{eq:intrinsic} in a single formula through the \textit{homogeneous coordinates} is
  \begin{equation}
    \label{eq:perspective_projection}
    \lambda
    \begin{pmatrix}
      x_1 \\ x_2 \\ 1
    \end{pmatrix}
    = KH
    \begin{pmatrix}
      X_1 \\ X_2 \\ X_3 \\ 1
    \end{pmatrix}
  \end{equation}
where the parameter $\lambda$ takes into account the projection in Equation \ref{eq:image-plane}.

%--------------------------------------------------%
\subsection{Lenses}
\label{subsec:lenses}
As mentioned above, Equation \ref{eq:perspective_projection} does not consider many non-ideality that affect the quality of images acquisitions.

In Section \ref{sec:pinhole_camera} we introduced the fact that, ideally, only one ray per point passes through the pinhole. If on the one hand this guarantees focus, on the other the impression of the scene in the sensor requires too much time. The increasing of the size of the pinhole allows the passage of light, reducing sensor exposure time. However, in this case the projection of the point on the sensor is the result of the mixing of many light rays, condition that reduces the image sharpness until becomes a continuous smear. Lenses are used to solve this problem.

The role of the lenses is the same as the pinhole: in fact, it allows the passage of light. Their advantage compared to the pinhole is the ability to converge many light rays on a specific point, allowing much more light, and reducing film exposure times. Lens models can be quite complex, so a common practice is considered: the \textit{thin lens approximation}. A thin lens is a lens with a negligible thickness compared to the radii of curvature of its surface. In this way Equation \ref{eq:perspective_projection} remains the same. Despite this, the use of lenses introduces some other issues. \\

The amount of light that impresses the sensors is proportional to the lens diameter. The bigger the diameter is, the more light enters in the camera, but we have to consider also the \textit{magnification}. Magnification is the process of enlarging (factor greater than one) or decreasing (factor less than one, this situation is also called ``minification'') appearance of something. In this case magnification refers to the ability to see more details of the world in a single image. That said, the brightness of the image depends inversely on magnification. A simple way to indicate \textit{aperture} of a lens (the opening through which light travels), is using the \textit{f-number} $K$, defined as
  \begin{equation}
    K = \frac{f}{d}
    \label{eq:fnumber}
  \end{equation}
where $f$ is the lens focal length and $d$ the aperture diameter. As we can see in Equation \ref{eq:fnumber}, $K$ decreasing at increasing of $d$. Thanks to $K$, it is possible to compare lenses, considering image luminosity, focal length and magnification. Note that this is a simple rule with no effects on the Equation \ref{eq:perspective_projection}. \\

Another problem is the focus of the lens. While a pinhole camera is permanently on-focus (ideal condition), this is not always valid for a lens. In reality a lens is on focus only at a specific distance: this means that only the point at that distance will be perfectly sharp. All the other points that are out-of-focus are projected on film as circles, called \textit{circles of confusion}. The blur spot shape is due by the aperture shape (that typical is a circle from which the name) and its size increases with the distance from the focus plane. Differently from chemical film, digital sensor are made as a matrix of photosensitive elements (pixels) that convert light in electrical signals; this means that sensors resolution is not infinite. If the circle of confusion is smaller than pixels sizes, we can consider that point on focus. The space range, around the focus plane in which the scene looks reasonably sharp, is called \textit{depth of field} (\acs{DOF}). An example of these effects, common in literature, is shown in Figure \ref{fig:dof}.
%  \begin{wrapfigure}{L}{0.5\textwidth}
%    \centering
%    \includegraphics[width=0.5\textwidth]{./images/dof_txt.jpg}
%    \caption{Example of a very shallow \acs{DOF}.}
%    \label{fig:dof}
%  \end{wrapfigure} \\
  \begin{figure}[t!]
    \centering
    \begin{minipage}[c]{.48\textwidth}
      \centering
 \includegraphics[width=\textwidth]{./images/tech/dof_txt.jpg}
      \caption{Example of a very shallow \acs{DOF}.}
      \label{fig:dof}
    \end{minipage}
    \hfill
    \begin{minipage}[c]{.48\textwidth}
      \centering
 \includegraphics[width=0.74\textwidth]{./images/tech/airydisk.jpg}
      \caption{Example of an \textit{airy disk}.\\ ~}
      \label{fig:airy-disk}
    \end{minipage}
  \end{figure} \\

The second big advantage of lenses against the pinhole, is the reduction of the diffraction. Diffraction is an effect generated by the interferences of light waves, when the light finds an obstacle or a hole, similar to the size of its wavelength. As the divergent rays now travel to different distances, some of them move out of phase and begin to interfere with each other, adding in some places and partially or completely canceling out in others. This effect, well known in many fields of interest, from electromagnetic to sound, in photography is known as \textit{airy disk} (by his discoverer, George Airy) and it is shown in Figure \ref{fig:airy-disk}. As for \acs{DOF}, even diffraction is negligible if it is smaller than the size of pixels. Diffraction could be present also in the \acs{DOF} of the camera, and depends only by the $f$-number, not by focal length. Lenses reduce this noise on the image, setting their aperture up appropriately. \\

Regardless of the lens model chosen (i.e. thin or thick lenses), their use introduces distortions due to the nature of the lens itself. In geometric optics, a distortion is a deviation from rectilinear projection, caused by small blemishes in the lenses and their alignment. The main optical aberration are two: \textit{radial distortion} and \textit{tangential distortion}. \\
Almost all of the deformation is radial in nature, this is the reason why distortion are more apparent from the center toward the edges of the image. This particular type of distortion is able to change the direction of straight lines, and it is evident during 3D calibration phases, when straight reference objects are seen from the camera as curve. Furthermore, it is often due to the need to expand the field of vision using a camera with short focal distance. Some example of its effects can be seen in Figure \ref{fig:teo-distorsions}. \\
Vice versa, tangential distortion is typically negligible than radial one, then it is ignored in most mathematical models.
  \begin{figure}[h!]
    \centering
 \includegraphics[width=0.9\textwidth]{./images/tech/distorsions.png}
    \caption{Example of radial distortions: on the left distortions free image is shown; in center a \textit{pincushion} aberration; on the right a \textit{barrel} aberration.}
    \label{fig:teo-distorsions}
  \end{figure}

%--------------------------------------------------%
\subsection{Scheimpflug principle}
In the previous subsection we talked about \acs{DOF} and the problem to maintain focus on the whole scene. Furthermore, we dealt with the Figure \ref{fig:dof} to show the result. Anyway, in the figure we can see that the subject lies on a plane not parallel with the sensor plane. Typical cameras and lenses are designed so that sensor plane, lens plane and subject plane are parallel to each other. This makes the focus of the camera very simple but, as said in Subsection \ref{subsec:lenses} talking about the rectilinear projection, the effective focal length depends by the position of the target with respect to the sensor itself. \\

Since the early $20^{th}$ century, the study of rotating the lens with respect to the sensor, and its effects on image acquisition, has been a widespread practice. In $1901$, Carpenter (one of the fathers of cinema) patented the first prototype of the so called ``view camera'' \cite{pat:carpentier}. From these studies, the Captain T. Scheimpflug patented \cite{pat:scheimpflug} in 1904. With his patent, Scheimpflug was the first to formulate the mathematical problem. The \textit{Scheimpflug principle} is a geometric rule that describes the relations between the sensor plane, then lens plane and the plane of focus when the lens plane is not parallel to the image plane. To achieve this situation, some cameras, such as view cameras, allow to tilt either the lens and the film, relative to the other. \\
  \begin{figure}[b!]
    \centering
    \begin{minipage}[c]{0.49\textwidth}
      \centering
      \includegraphics[width=\textwidth]{./images/tech/sch_par.png}
    \end{minipage}
    \hfill\
    \begin{minipage}[c]{0.49\textwidth}
      \centering
      \includegraphics[width=\textwidth]{./images/tech/sch_tilt.png}
    \end{minipage}
    \caption{(On the left) For standard cameras, the sensor, lens and focus planes are parallel to one another. (On the right) For a view camera, tilting lens causes the plane of sharp focus to tilt as well.}
    \label{fig:scheimpflug}
  \end{figure}

This principle asserts that when the lens is tilted, lens plane, image plane and the plane of focus all intersects in a line, called \textit{Scheimpflug line}, as illustrated in Figure \ref{fig:scheimpflug}. In this way, a subject that is not parallel to the sensor can be completely in focus. Nevertheless, this first relation does not give any information about how to tilt the lens to achieve the intended position for the plane of focus. This information arises from the laws of optics, thanks to the \textit{hinge rule}, similar to Scheimpflug one. The required amount of lens tilt is given by the expression:
  \begin{equation*}
    \alpha = \arcsin \left( \frac{f}{J} \right)
  \end{equation*}
where $f$ is the focal length of the lens, and $J$ is the distance from the lens and the \textit{hinge line}. The hinge line is the intersection between a plane parallel with the sensor and passing through the lens one, and the plane of focus. \\
From this principle we can also determine the \acs{DOF} of the camera. It can be demonstrated that the limits of the \acs{DOF} are also planes, that passes through the hinge line, and symmetrical compared to the plane of focus. To be precise, these planes lies at a distance $J$ from the plane of focus, distance measured at the \textit{hyperfocal distance} $H$ from the hinge line \cite{book:ftvc}. The scenario is illustrated in Figure \ref{fig:sch_dof}.
  \begin{figure}[h!]
    \centering
    \includegraphics[width=0.8\textwidth]{./images/tech/sch_dof.png}
    \caption{Depth of field in view cameras, with tilted lenses.}
    \label{fig:sch_dof}
  \end{figure}

%--------------------------------------------------%
\subsection{Overview on digital cameras}
\label{subsec:overview-cameras}
To complete the introduction on digital camera we think that it could be useful to linger over some practical aspects, in particular about what concerns manufacturing of the sensor and image acquisitions. \\

In Subsection \ref{subsec:lenses} we introduced the quantization effect due by pixels. If it relaxes the problem of the lens focus, on the other hand it introduces noise in light to image conversion. We briefly analyse the two most popular sensor types on the market: \acs{CMOS} and \acs{CCD} (Charge-Coupled Device).

In \acs{CMOS} sensors, each element has its own signal amplifier: this allows to improve the camera frame rate and to isolate regions of interests via hardware. On the contrary, pixels haven't the same sizes neither the same doping, which reduces the quality of the sensor itself. The latest CMOS sensors on the market are good enough to be used in computer vision, achieving excellent results.

\acs{CCD} are high-scale integration sensors that use only one signal amplifier, ensuring the same amplification constant for each element. However, this requires that sensor rows are converted one by one, by lowering the camera frame rate. Furthermore these sensors give a very small, but non-zero response to a zero input and they saturate for very bright stimuli. In spite of that, they are much less noisy than the CMOS.

These differences are very delicate, specially considering the fields of application of the camera. For example, in laser triangulation systems, the acquired images are dark to highlight the laser light. In situation like this, when the brightness of the image is very low\footnote{In this case we consider source of light with brightness near to the base noise level of the sensor.}, signals amplification offsets are meaningful. The mono-pixel amplifiers in \acs{CMOS} are more noisy and generate less uniform values than \acs{CCD}, resulting in a more homogeneous output. This source of noise is known as \textit{thermal noise}. High quality systems provide different solutions to reduce this effect as much as possible. In Figure \ref{fig:thermal-noise} an example of the noise is shown.
  \begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\textwidth]{./images/tech/dark-example.jpg}
    \caption{Example of uncontrolled dark noise in astronomical photography.}
    \label{fig:thermal-noise}
  \end{figure} \\

Another problem due to sensor quantization, is colours acquisition. Each pixel is able to acquire only light signals, resulting in grey scale acquisitions. For this reason sensors needed filters to encode colours. The most widespread is \textit{Bayer's color filter array} (Bayer's \acs{CFA}). A \acs{CFA} is an array in which passband filters are placed, according to a known pattern. In this way each pixel is able to encode only a specific colour. Many algorithms are used to reconstruct the scene; they interpolate signals collected by near pixels and extract the correct colour for each pixel. To perform measures, this is a waste of pixels: the presence of the filter reduces the sensor surface useful to collect world details. For these reasons, grey scale cameras are often used.


\chapter{Kalman Filtering}
\label{ch:kalman_filter}
%
The Kalman filter is over 50 years old but is still one of the most important and common data fusion algorithms in use today.
Named after Rudolf E.Kálmán, the great success of the Kalman filter is due to its small computational requirement, elegant recursive properties, and its status as the optimal estimator for one-dimensional linear systems with Gaussian error statistics.
Typical uses of the Kalman filter include smoothing noisy data and providing estimates of parameters of interest.
Kalman filtering is used in a wide range of applications include global positioning system receivers, in control systems, through to the smoothing the output from laptop trackpads, and many more.

\section{Overview of Kalman Filtering}

Kalman filter are typically used to estimate parameters which change with time.
Parameters with no process noise are called deterministic.
A Kalman filter has measurements $y_t$, with noise $y_t$, and a state vector $\hat x_t$ (or a parameter list) which have specified statistical properties.

The observation equation at time t:
\begin{equation}
    y_t = H_t x_t + \epsilon_t	 \label{eq:kfObs}
\end{equation}

The state transition equation:
\begin{equation}
    x_{t+} = F_t x_t + w_t
\end{equation}

The kalman filter processing is broken up into three main steps.

\textit{Prediction} {uses a process noise model} to 'predict' the parameters at the next data epoch, subscript is time quantity refers to, where as the superscript refers to the time of the data:
\begin{equation}
    \hat{x}_{t+1}^t = F_t \hat{x}_t^t
\end{equation}
where, $F_t$ is the state transition matrix
\begin{equation}
    P_{t+1}^t = F_t P_t^t F_t^\intercal + Q_t
\end{equation}
where, $Q_t$ is the process noise covariance matrix.
The state transition matrix $F$ projects the state vector (parameters) forward to the next epoch.
\begin{itemize}
    \item For random walk $F$ = 1
    \item For rate terms: $F$ is matrix
    $\begin{bmatrix}
    1 & \delta t\\
    0 & 1
  \end{bmatrix}$
    \item for FOGM: $F$ = $e^{-\delta t \beta}$
    \item For white noise $F$ = 0
\end{itemize}
The second equation projects the covariance matrix of the state vector, $P$, forward in time. Contributions from the state transition and process noise ($Q$ matrix).
$Q$ elements are 0 for deterministic parameters.
%
\textit{The Kalman gain} {is the matrix} that allocates the differences between the observation at time t+1 and their predicted value at this time based on the current values of the state vector according to the noise in the measurements and the state vector noise.

\section{Comparison between Weighted Least Squares and Kalman Filtering}

\begin{itemize}
    \item In kalman filtering apriori constraints must be give for all parameters. This is not needed in weighted least squares, but can also be done.
    \item Kalman filters can allow for 0 variance parameters, this cannot be done in WLS, as this requires the inversion of the constraint matrix.
    \item Kalman filter can allow for a method of applying absolute constraints, WLS can only tightly constrain parameters.
    \item Kalman filters are more prone to numerical stability problems, and take longer to run (they have more parameters).
    \item Process noise models can be implemented in WLS, but they are computationally slow.
\end{itemize}

\section{Implementation in the PEA}

\subsection{Robust Kalman Filter Philosophy}

It is well known that the Kalman filter is the optimal technique for estimating parameters of interest from sets of noisy data - provided the model is appropriate.

In addition, statistical techniques may be used to detect defects in models or the parameters used to characterise the data, providing opportunities to intervene and make corrections to the model according to the nature of the anomaly.

By incorporating these features into a single generic module, the robustness that was previously available only under certain circumstances may now be automatically applied to all systems to which it is applied. These benefits extend automatically to all related modules (such as RTS), and often perform better than modules designed specifically to address isolated issues.

\subsection{Initialisation}

When parameters' initial values are not known a-priori, it is often possible to determine them using a least-squares approach.

To minimise processing times, the minimal subset of existing states, measurements, and covariances are used in least-squares estimation whenever the initial value and variance of a parameter is unspecified.

For rate parameters, multiple epoch’s worth of data are required for an ab-initio initialisation. This logic is incorporated into the filter and is applied automatically as required.

\subsection{Outlier detection, Iteration, and Hypothesis Testing}

As a statistical machine, the Kalman filter is capable of detecting measurements that do not fit within the system as modelled.

In these cases, the model may be adjusted on-the-fly, to allow all measurements to be continued to be used without contaminating the results in the filter.

A typical example of a modelling error in GNSS processing is a cycle-slip, in which the ambiguity term (which usually modelled with no change over time) has a discontinuity. Other examples may include clock-jumps or satellite burns.

Hypotheses are to be generated for any measurements that are statistical outliers, and the model iterated as required.

\subsection{Performance Optimisation}

The inversion of large matrices as required by the Kalman filter easily dominates the processing time required during operation. Techniques are available to reduce, and distribute this processing burden across multiple processors.

The Eigen library is used for algebraic manipulation which allows for automatic parallelisation of vector algebra, and improves code robustness by checking matrix dimensions while in use.

\subsubsection{Chunking}

By dividing measurements into multiple smaller sub-matrices, the long inversion times may be reduced, as the inversion order is of $O(n^3)$

\subsubsection{Blocking}
By separating the filter covariance matrix into a block-diagonal form, individual blocks of the filter may be processed individually, without degredation in accuracy. This may improve performance, and may also enable blocks that are relatively independent to be processed separately, albeit with some degredation in accuracy.

\section{Configuration} \label{KFConfig}

All elements within the kalman filter are configured using the yaml configuration file, and use a consistant format.

\subsection{default\_filter\_parameters}

\begin{lstlisting}[language=yaml,caption=Filter Parameters:]

default_filter_parameters:

    stations:

        error_model:        elevation_dependent         #uniform elevation_dependent
        code_sigmas:        [0.15]
        phase_sigmas:       [0.0015]

        pos:
            estimated:          true
            sigma:              [0.1]
            proc_noise:         [0] #0.57 mm/sqrt(s), Gipsy default value from slow-moving
            proc_noise_dt:      second
            #apriori:                                   # taken from other source, rinex file etc.
            #frame:              xyz #ned
            #proc_noise_model:   Gaussian

        clk:
            estimated:          true
            sigma:              [0]
            proc_noise:         [10]
            proc_noise_dt:      second
            #proc_noise_model:   Gaussian

        clk_rate:
            estimated:          false
            sigma:              [500]
            proc_noise:         [1e-4]
            proc_noise_dt:      second
            clamp_max:          [+500]
            clamp_min:          [-500]

    satellites:

        clk:
            estimated:          true
            sigma:              [1000]
            proc_noise:         [1]
            #proc_noise_dt:      min
            #proc_noise_model:   RandomWalk

        # clk_rate:
        #     estimated:          true
        #     sigma:              [10]
        #     proc_noise:         [1e-5]
        #     # clamp_max:          [+500]
        #     # clamp_min:          [-500]

        orb:
            estimated:          false

    eop:
        estimated:  true
        sigma:      [40]

override_filter_parameters:

    stations:
        #ALIC:
            pos:
                sigma:              [0.001]
                proc_noise:         [0]
\end{lstlisting}

The majority of estimated states are configured in this section. These configurations are applied to all estimates unless another configuration overrides these parameters in the override\_filter\_parameter section.

The parameters that are available for estimation include:
\begin{itemize}
\item stations:
\begin{itemize}
\item pos
\item pos\_rate
\item clk
\item clk\_rate
\item amb
\item trop
\item trop\_grads
\end{itemize}
\item satellites:
\begin{itemize}
\item pos (coming soon)
\item pos\_rate (coming soon)
\item clk
\item clk\_rate
\item orb
\end{itemize}
\end{itemize}

\subsection*{estimated:}

Boolean to add the state(s) to the kalman filter for estimation.

\subsection*{sigma:}

List of a-priori sigma values for each of the components of the state.

If the sigma value is left as zero (or not initialised), then the initial variance and value of the state will be estimated by using a least-squares approach.
In this case, the user must ensure that the solution is likely rank-sufficient, else the least-squares initialisation will fail.

For states with multiple elements (eg, X,Y,Z positions), multiple sigma values may be added to the list. However, if insufficient values are added to the list, the intialiser will use the last value in the list for any extra elements.
ie. Setting \lstinline{sigma: [10]} is sufficient to set all x,y,z components of the apriori standard deviation to 10.

\subsection*{proc\_noise:}

List of process noises to be added to the state during state transitions. These are typically in m/sqrt(s), but different times may be assigned separately.
As for the sigma list, the last value will be used for any elements exceeding the list length.

\subsection*{proc\_noise\_dt:}

Unit of measure for process noise.
May be left undefined for seconds, or using sqrt\_second, sqrt\_seconds, sqrt\_minutes, sqrt\_hours, sqrt\_days, sqrt\_weeks, sqrt\_years.

\subsection{override\_filter\_parameters:}

In the case that a specific station or satellite requires an alternate configuration, or to exclude estimates entirely, the override\_filter\_parameters section may be used to overwrite selected components of the configuration.

\subsection{user\_filter\_parameters, network\_filter\_parameters:}

The internal operation of the kalman filter is specified in this section. It has a large impact on the robustness, and associated processing time that the filter will achieve.

\begin{lstlisting}[language=yaml,caption=Filter Operating Parameters:]

user_filter_parameters:

    max_filter_iterations:      5 #5
    max_prefit_removals:        3 #5

    rts_lag:                    -1      #-ve for full reverse, +ve for limited epochs
    rts_directory:              ./
    rts_filename:               PPP-<CONFIG>-<STATION>.rts

    inverter:                   LLT         #LLT LDLT INV

\end{lstlisting}

\subsection*{max\_prefit\_removals:}

Maximum number of pre-fit residuals to reject from the filter.

After the vector of residuals has been generated and before the filter update stage is computed, the residuals are compared with the expected values given the existing states and design matrix.
If the values are deemed to be unreasonable - because the variances of the transformed states and measurements do not overlap to with a 4-sigma level of confidence - then these measurements are deweighted by deweight\_factor, to prevent the bad values from contaminating the filter.

These measurements are recorded as being rejected, and may have additional consequences according to other configurations such as phase\_reject\_limit.

\subsection*{max\_filter\_iterations:}

Maximum number of times to compute the full update stage due to rejections.

This is similar to the max\_filter\_rejections parameter, but the 4-sigma check is performed with post-fit residuals, which are much more precise.

Rejections that occur in this stage require the entire filter inversion to be repeated, and has an associated performance hit when used excessively.

\subsection*{inverter:}

There are multiple inverters that may be used within the kalman filter update stage, which may provide different performance outcomes in terms of processing time and accuracty and stability.

The inverter may be selected from:
\begin{itemize}
\item llt
\item ldlt
\item inv
\end {itemize}

\subsection{outage\_reset\_limit:}
Maximum number of epochs with missed phase measurements before the ambiguity associated with the measurement is reset.

\subsection{phase\_reject\_limit:}
Maximum number of phase measurements to reject before the ambiguity associated with the measurement is reset.

\subsection*{rts\_X:}

For details about rts configuration, see section \ref{ch:RTS}

\subsection{Process Noise Guidelines}

Currently in the PEA we have random walk process noise models implemented.

The units are typically in meters, and they are given as $\sigma$ = $\sqrt{variance}$

For a random walk process noise, the process noise is incremented at each epoch as $\sigma^2\times dt$ where dt is the time step between filter updates.

If you want to allow kinematic processing, then you can increase the process noise e.g.\\
proc\_noise [0.003]\\
proc\_noise\_dt: second\\

equates to $0.003\frac{1}{\sqrt{s}}$
\\
Or if you wanted highway sppeds 100km/hr = 28 m/s\\
proc\_noise [28]\\
proc\_noise\_dt: second

A nice value for using VMF as an apriori value is 0.1mm /sqrt(s)
%
\begin{lstlisting}
trop:
    estimated:          true
    sigma:              [0.1]
    proc_noise:         [0.01]
    proc_noise_dt:      hour
\end{lstlisting}

\section{Recommended Reading}

\begin{enumerate}
    \item https://ocw.mit.edu/courses/earth-atmospheric-and-planetary-sciences/12-540-principles-of-the-global-positioning-system-spring-2012/lecture-notes/MIT12\_540S12\_lec13.pdf
\end{enumerate}


\section{Background}
\label{background}

\subsection{Resilience analysis using Graph Theory}
\label{resilience}

The seminal works in \cite{2000Natur.406..378A,DaqingAl14} address contingency analysis for the power grid using simulation of complex networks. Typically, the power grid is modeled using generator, transmission, and distribution nodes. In such scale-free networks, a large fraction of the nodes have low degrees and a substantially smaller fraction exhibit high degrees. Those networks are established to be resilient against random failures upon nodes, and the overall loss to be contingent upon the targeting of the high-degree hubs \cite{AlbertAl00, CohenAl00, CohenAl01, CallawayAl00}.
\edited{
In addition to topological analysis of the power grid, other metrics can be used to assess its vulnerability. These are based on the physical properties of the network such as line resistance and sensitivity, which encapsulate the impedance matrix and consequently, the  ``electrical centrality'',  the equivalent of topological centrality, is computed and node removal strategies can be tested accordingly~\cite{Hines:2007bt,WangST10}.

Equivalently, vertex vulnerability or network robustness can be evaluated through percolation measures such as the average inverse geodesic and the giant component, which emerges subsequent to attacks and exhibits a dependence on the node removal strategy and on the topology of the network under scrutiny~\cite{Bianconi:2016ka,Callaway:vd,Karrer:2014ep,Radicchi:2015gp}.
We will thus focus our effort on a graph-based approach to vulnerability since the physical properties of the power grid were not made available to us.}

In our Spark implementation we aim to distribute the random and cascading failure scenarios explored by \cite{2000Natur.406..378A}, which in turn models the North American power grid using its transmission lines, and examines its connectivity in relation to a small set of high impact nodes.

The notion of connectivity is based on the notion of betweenness centrality. A node's betweenness centrality is a focal measure of connectedness in a graph, built around the notion of shortest paths. Given any two vertices in a graph, there exists at least one shortest path between the vertices. When the length of the path is infinity, it is understood to say that there is no path to connect the given two vertices. If the graph is weighted, the shortest path is obtained by minimising the number of edges that the path passes through. Else, the path is obtained by minimising the sum of the weights of the edges that the path passes through. Given an arbitrary vertex in a graph, its betweenness centrality is defined to be the number of shortest paths that pass through the given vertex. Exploiting betweenness centrality has been widespread in a number of works addressing power grid vulnerability analysis. The work in \cite{2000Natur.406..378A} employs this notion using four scenarios each of which simulates a unique temporal mode of removal of vertices. For example, the overall connectivity of the graph is re-examined upon removal of transmission nodes according to the following orders: (1) totally random order (2) decreasing order of node degrees (load) (3) decreasing order of node betweennees centrality and (4) decreasing order of node betweenness centrality in cascading order resulting from the nodes' dynamical removal. \edited{
The cascading scenario is based on recalculated information or more precisely after the identification of the most central node of the initial graph and subsequent to its removal a new graph ensues. The central node of the latter should then be computed and then the process is iterated over.}

The approach taken in \cite{JinAl10} considers the intensity of the power flowing on a transmission branch, as opposed to the degree of nodes. The resulting graph is weighted (but still undirected), and the contingency analysis method there is based on applying edge betweenness centrality \cite{GirvanAl02} to the power grid topology. High-impact components in the power grid are defined using the most traversed edges. We believe this approach is less exhaustive than the one adopted by \cite{2000Natur.406..378A}, and we do not pursue it here.

Our spatial understanding of the propagation of faults in the Lebanese power grid makes classical use of the concept of spatial correlation \cite{CavagnaAl10, MakseAl95}. As a leading example that we follow, this measure is used in \cite{DaqingAl14} to measure the relation between failures separated at some distance $r$. More details follow in Sec. \ref{methods}.

\subsection{Distributed Computation frameworks}
\label{distcomp}
In the following, we discuss two distributed computation frameworks: (1) MapReduce and Pregel; (2) Spark and GraphX.

\subsubsection{MapReduce and Pregel}
The last few years have witnessed an uptake in distributed data processing research. Among the leading frameworks to exploit distributed computation on commodity hardware is the MapReduce paradigm \cite{mapreduce}. A typical MapReduce program consists of the ``Map'' operator that parcels out work to various nodes within the cluster or map, and the ``Reduce'' phase that applies a reduction operator on the results from each node into a global query. The key contributions of the MapReduce framework are the scalability and fault-tolerance achieved for a variety of applications by optimizing the execution engine, for example, by reassigning tasks when a given execution fails. As with all other parallel and distributed paradigms, the performance of an efficient MapReduce algorithm is contingent upon a reduced communication cost. Of particular challenge is how to efficiently process large graphs. Graph algorithms often exhibit poor locality of reference, and a low compute-to-memory access ratio, which affects the scalability of their parallel adaptations. It is also difficult to maintain a steady degree of parallelism over the course of execution of graph algorithms. Additionally, expressing a graph algorithm in MapReduce requires passing the entire state of the graph from one stage to the next, thus imposing significant communication as well as serialisation in the parallel code.

The first serious development for supporting graph algorithms using the MapReduce framework is found in Google's Pregel \cite{Pregel}. Instead of coordinating the steps of a chained MapReduce program, Pregel is able to process iteration over supersteps under the Bulk Synchronous Parallel model \cite{Biss04, McColl2, Valiant}. In a BSP algorithm, a computation proceeds in a series of global supersteps. Each superstep consists of three phases:
\begin{enumerate}
\item{A concurrent computation superstep: each processor performs local computations using values stored in the local, fast memory of the processor.}
\item{A communication superstep: the processes exchange data between themselves if needed for the aggregation of the results computed in (1) above.}
\item{A barrier synchronisation superstep: each processor halts until all other processes have reached the same barrier.}
\end{enumerate}
According to this model, a graph algorithm in Pregel is organised as a sequence of iterations, and can be described from the point of view of a vertex, that manages its state and sends messages only to its neighbours. Pregel keeps vertices and edges on the machine that performs computation, and uses network transfers only for messages.

\subsubsection{Spark and GraphX}

Spark, a distributed computation framework built around the MapReduce paradigm, is a recent Apache foundation software project supported by an execution engine for big data processing. Spark provides for in-memory computation, which refers to the storage of information in the main random access memory (RAM) of dedicated servers rather than in relational databases running on relatively slower disk drives. Using over $80$ high-level operators, Spark makes it possible to write code more succinctly, and till this point in time, is considered one of the fastest frameworks for big data processing. Spark's most notable properties are also thanks to its core, which, in addition for serving as the base engine for large-scale parallel and distributed data processing, is able to handle memory management and fault recovery, scheduling, distributing and monitoring jobs on a cluster, as well as interacting with storage systems.

Spark hinges on parallel abstract data collections called RDDs (resilient distributed datasets), which can be distributed across a cluster. These RDDs are immutable, partitioned data structures that can be manipulated through multiple operators like Map, Reduce, and Join. For example, RDDs are created through parallel transformations (e.g., map, group by, filter, join, create from file systems). RDDs can be cached (in-memory) by allowing to keep data sets of interest locally across operations, thus contributing to a substantial speedup. At the same time, Spark uses lineage to support fault tolerance, i.e., record all the operations/transformations that yield RDDs from a source data. In case of failure, an RDD can be reconstructed given the transformation functions contributing to that RDD. Additionally, after creating RDDs, it is possible to analyse them using actions such as count, reduce, collect and save. Note that all operations/transformations are lazy until one runs an action. At that point, the Spark execution engine pipelines operations and determines an execution plan.

Borrowing from Pregel, GraphX~\cite{graphx} is a platform built on top of Spark that provides APIs for parallel and distributed processing on large graphs. In GraphX, each graph is mapped into different RDDs, where in each RDD one applies the computation on the graph using the ``think like a vertex'' model.


\section{Periodic Orbits as Solutions of System of Equations, the Galerkin Operator}

%stationary? ordinary?
Given a system of $n \in \N$ possibly non-linear, autonomous, ordinary, first-order differential equations $\mathbf{x}$
	\begin{equation} \label{eq:SDE}
		\frac{d\mathbf{x}}{dt} = \mathbf{f}(t, \mathbf{x}) \text,
	\end{equation}
we are interested in numerically computed, periodic solutions.
That is, solutions $\mathbf{y}: \R \to \C^n$ which obey $\mathbf{y}(t) = \mathbf{y}(t+T)$ for all $t \in \R$ and some period $T \in \R$.
This is the general case, as differential equations of any degree can be converted to a system of first-order differential equations.

\paragraph{Model} Solution candidates need to be modeled in a certain way.
The periodicity constraint suggests using a multidimensional trigonometric polynomial of degree $m \in \N$
	\[
		\mathbf{y} \coloneqq \sum_{k = -m}^m \mathbf{y}_k \exp\left(i \omega k t\right) \text{,}
	\]
where $\mathbf{y}_k \in \C^n$, $\omega = \frac{2\pi}T$. %$\mathbf{y}_{-k} = \Re(\mathbf{y}_k) - i \Im(\mathbf{y}_k)$ for $k \in \N$, $-m \le k \le m$,
Solution candidates of this form satisfy $\mathbf{y}(t) = \mathbf{y}(t+T)$ by definition.
%A function of this form is defined solely by its $m+1$ unique coefficients $\mathbf{y}_k$ for $0 \le k \le m$.

\paragraph{Optimality Criterion} Finding good solutions, that is, functions $\mathbf{y}$ which at least approximate $\mathbf{y}^\prime = \mathbf{f}(t, \mathbf{y})$, requires a way of judging whether a solution candidate is indeed a correct solution.
In this case, Galerkin's method takes this role.
A useful property of the trigonometric polynomial is, that it can be trivially differentiated
	\begin{equation}
			\frac{d\mathbf{y}}{dt} = \sum_{k = -m}^m i \omega k \mathbf{y}_k \exp\left(i \omega k t\right) \text.
	\label{eq:trigondiff}
	\end{equation}
Employing this property in the definition of the differential equation system yields
	\begin{align*}
			& \frac{d\mathbf{y}}{dt} = \mathbf{f}(t,\mathbf{y})\\
		\Leftrightarrow\ & \mathbf{f}(t,\mathbf{y}) - \frac{d\mathbf{y}}{dt} = 0\\
		\Leftrightarrow\ & \mathbf{f}(t,\mathbf{y}) - \sum_{k = -m}^m i \omega k \mathbf{y}_k \exp\left(i \omega k t\right) = 0 \text.
	\end{align*}
The difference between these two functions is called the \emph{residual} $\mathbf{r}(t) \coloneqq \mathbf{f}(t,\mathbf{y}) - \frac{d\mathbf{y}}{dt}$.
A candidate $\mathbf y$ is a solution if and only if $\mathbf r = 0$.
%Checking for $\mathbf{r}(t) = 0$ would require comparing the two functions at infinitely many points. %TODO fix... not changed by galerkin
The solution of the system of differential equations will generally not be representable by a trigonometric polynomial.
The residual can thus not genuinely equal zero.

\paragraph{Galerkin's Method} Galerkin's method relaxes the equality requirement such that only projections onto a set of so called \emph{trial vectors}, need to vanish.
This is equivalent to requiring a projection of $\mathbf{r}$ onto the subspace spanned by the trial vectors to be zero.
Choosing the complex oscillations as a basis for this subspace as well is a solid choice.
Basically the residual is approximated by a trigonometric polynomial and a solution is required to only minimize this representation.
There are other factors supporting this choice: The residual is periodic as well, because of orthogonality many terms can cancel each other out, it allows us to employ the FFT for many operations, and the resulting system of equations is almost balanced.

\paragraph{System of Equations} This yields $N = 2m+1$ equations, one for each trial vector $\mathbf{v}_k = \exp\left( i \omega k t \right)$ for $-m \le k \le m$
	\begin{align}
		0 &= \langle \mathbf{r} , \mathbf{v}_k \rangle \nonumber \\
		&= \langle \mathbf{f}(t,\mathbf{y}), \mathbf{v}_k \rangle - \left\langle \frac{d\mathbf{y}}{dt}, \mathbf{v}_k \right\rangle \nonumber \\
		&= \langle \mathbf{f}(t,\mathbf{y}), \mathbf{v}_k \rangle - i \omega k \mathbf{y}_k\label{eq:innerprod} \text.
	\end{align}
The last step exploits that the complex exponential functions are orthonormal.
%The last step exploits that there is always exactly one component in the candidate function' derivative which is not orthogonal to the trial vectors.
For these $N$ equations, there are $N+1$ variables: $N$ unique coefficients and $\omega$.
This represents the situation, that at this point there is still one degree of freedom: Each phase shifted version of a solution is still a solution.
We thus introduce another generic equation called the \emph{anchor} equation, which basically chooses one of these solutions.
In this case $\langle \mathbf{y}(0), (\delta_{1i})_{i \in \N_n} \rangle = 0$ is used:
For $t = 0$, the solution needs to intersect the hyperplane defined by being zero in the first component.
This can be formulated by requiring the corresponding coefficients to sum up to zero.
The anchor equation needs to be adapted to the system considered: If there are no intersections with this plane, another equation needs to be chosen.

%motivation tedious, error prone
\paragraph{Discretization} When transferring these theoretical constructs to a practical setting, the main change is that solutions $\mathbf{y}$ are not considered to be functions of continuous time, but vectors representing discrete time.
Consequently, solution candidates are represented by linear combinations of discrete vectors as well
	\[
		\mathbf{\mathbf{y}} = \left( \sum_{k=0}^{N-1} \mathbf{y}_k \exp\left(i \frac{2\pi}{N} j k\right) \right)_{0 \le j \le N-1} \text.
	\]
This is no real limitation at that point, as solution candidates are band limited by construction.
Define the \emph{DFT matrix} $\mathbf F \in \R^{N \times N}$, the coefficient vector $\mathbf Y \in \left(\C^n\right)^N$ and a diagonal matrix $\mathbf K \in \R^{N\times N}$
	\begin{align*}
		\mathbf{F} &\coloneqq N^{-1} \left( \exp\left(-i \frac{2\pi}{N} j k \right) \right)_{0 \le j,k \le N-1} \\
		\mathbf{Y} &\coloneqq \left(\mathbf{y}_k\right)_{0 \le k \le N-1} \\
		\mathbf K &\coloneqq \diag\left( [0:m]\, ||\, [-m:-1] \right) \text.
	\end{align*}
$\mathbf Y$ is a vector of vectors, while somewhat unusual, this allows for more concise notation later on, than notation as a matrix.
Using these constructs allows us to reformulate
	\[
			\mathbf{y} = \mathbf{F^{-1}} \mathbf{Y} \text.
	\]
Considering only the equations from projections onto trial vectors, and considering $\omega$ to be known the system of equations becomes %TODO define
	\[
		\mathbf{F} \mathbf{f}\{\mathbf F^{-1} \mathbf{Y}\} - i \omega \mathbf K \mathbf Y = 0\text,
	\]
%where $\mathbf{\tilde f}(\mathbf Y) = \mathbf f\{\mathbf Y\}$ simply row wise applies $\mathbf{f}$, the function defining the system of differential equations (see \autoref{eq:SDE}).
where $\mathbf{f}$ defines the system of differential equations (see \autoref{eq:SDE}).

Compare this to \autoref{eq:innerprod}.
Essentially, $\mathbf{f}\{\mathbf F^{-1} \mathbf{Y}\}$ is a sampled version of $\mathbf f(t,\mathbf y)$.
Multiplication by $\mathbf F$ corresponds to calculating the inner products with the trial vectors, which are the rows of $\mathbf F$.
The term $i \omega \mathbf K \mathbf Y$ corresponds to \autoref{eq:trigondiff}, focusing on the coefficients only.

Introducing these new constructs might at first glance appear unnecessarily complicated, because the system of equations was already known, and could be used in the state it was in.
However, an implementation in the basic form would have required different treatment for each dynamic system, while in this form, it is only trivially dependent of $\mathbf f$ and the degree of the trigonometric polynomial.
%That is, the auxiliary vectors and matrices are trivially constructed, as is $\mathbf{\tilde f}$ from $\mathbf f$.
While knowing the system of equations is important, the core task of finding a good solution requires the Jacobian of the system.
The great advantage is, that from this form the system can be derived in a general way.

\paragraph{Deriving the System} %Let $D$ denote a differential operator.
The only variable in the system is $\mathbf Y$, thus
	\begin{align}
		& \frac{d}{d\mathbf Y} \left( \mathbf{F} \mathbf{f}\{\mathbf{F}^{-1} \mathbf{Y}\} - i \omega \mathbf K \mathbf Y \right) \nonumber \\
		=\ & \mathbf{F} \frac{d \mathbf{f}\{\mathbf Y\}}{d\mathbf Y} \left( \mathbf{F}^{-1} \mathbf{Y} \right) \mathbf{F}^{-1} - i \omega \mathbf K (\delta_{ij} \mathbf I_n)_{i,j \in \N_N} \label{eq:vecsys} \text,
	\end{align}
%TODO careful: dY/dY != I
% where $\mathbf J_{\mathbf{\tilde f}} \coloneqq \frac{d \mathbf{f}\{\mathbf Y\}}{d\mathbf Y}$ is the Jacobian of $\mathbf{\tilde f}$, a vector-by-vector derivative, which yields a matrix.
where $\frac{d \mathbf{f} \{ \mathbf Y \}}{d \mathbf Y}$ is a vector-by-vector derivative, which yields a matrix.
However, because both vector's elements are vectors themselves, the entries of the matrix are again vector-by-vector derivatives.
This results in an $N \times N$ matrix of matrices with
	\[
			\left( \frac{d \mathbf{f}\{\mathbf Y\}}{d\mathbf Y} \right)_i^j = \frac{d}{d\mathbf Y_j} \mathbf f (\mathbf Y_i) = \delta_{ij} \frac{d \mathbf f(\mathbf x)}{d \mathbf x}(\mathbf Y_i) \text, %TODO fix x
	\]
where $\frac{d \mathbf f(\mathbf x)}{d\mathbf x}$ is the Jacobian of $\mathbf f$.
% The matrix $\mathbf J_{\mathbf{\tilde f}}$ is diagonal, thus
% 	\[
% 		\mathbf J _ \mathbf{\tilde f} (\mathbf Y) = \diag\left( \left(\mathbf J _ \mathbf f \left(\mathbf Y_i\right) \right)_{i \in \N_N} \right)
% 	\]
At no point different cells of $\frac{d \mathbf f(\mathbf x)}{d\mathbf x}$ interact.
Hence, the expression for the Jacobian of the whole system can be considered separately for each partial derivative in the Jacobian of $\mathbf f$.
For $k,l \in \N_n$ define the $N \times N$ diagonal matrix which extracts a single partial derivative
	\[
		\mathbf A_{kl}(\mathbf Y) = \diag\left( \frac{\partial f_k}{\partial x_l} \left\{\mathbf Y_i\right\} \right) \text.
	\]
For each $k$ and $l$ \autoref{eq:vecsys} then becomes
	\begin{equation}
		\mathbf{F} \mathbf A_{kl}(\mathbf{F}^{-1} \mathbf{Y}) \mathbf{F}^{-1} - i \delta_{kl} \omega \mathbf K
	\label{eq:syssingle}
	\end{equation}
This expression is again free of nested vectors.

%Because of the diagonal structure of the matrix, the term $\frac{d\mathbf{\tilde f}}{d\mathbf x}(\mathbf{F}^{-1} \mathbf{Y}) \mathbf{F}^{-1}$ is essentially a multiplication of each row of $\mathbf F^{-1}$ with a constant.
Because $\mathbf A_{kl}(\mathbf{F}^{-1} \mathbf{Y})$ is diagonal, $\mathbf A_{kl}(\mathbf{F}^{-1} \mathbf{Y}) \mathbf{F}^{-1}$ is the rows of $\mathbf F^{-1}$ each multiplied by a scalar.
This can also be seen as an element wise multiplication of the columns $\mathbf c_k$ of $\mathbf F^{-1}$ by a column vector $\mathbf d$ defined by the diagonal of $\mathbf A_{kl}(\mathbf{F}^{-1} \mathbf{Y})$
	\begin{align*}
		&\mathbf c_k \coloneqq \left(\exp\left(i\frac{2\pi}{N} j k\right)\right)_{0 \le j \le N-1}\\
		&\mathbf d \coloneqq \frac{\partial f_k}{\partial x_l} \{\mathbf Y\} \text.
	\end{align*}
A column of $\mathbf{F} \mathbf A_{kl}(\mathbf{F}^{-1} \mathbf{Y}) \mathbf{F}^{-1}$ is then the DFT of the product of two signals $\mathcal F(\mathbf d \cdot \mathbf c_k)$.
Because of the structure of $\mathbf c_k$, this corresponds to a simple periodic shift of the Fourier coefficients of the discrete signal $\mathbf d$ by $k$ steps.
Using this for every column of the complete term yields
	\[
			\mathbf{F} \mathbf A_{kl}(\mathbf{F}^{-1} \mathbf{Y}) \mathbf{F}^{-1} = \left( \mathbf \F(\mathbf d)_{((i-j))_N} \right)_{0 \le i,j \le N} \text,
	\]
that is, a simple circulant matrix.

Rebuilding the complete derivative from these matrices is a matter of subtracting the remaining diagonal term and merging the results.
% The general form of \autoref{eq:syssingle} is $\mathbf G = \mathbf M \cdot D \mathbf Y$.
%The differential operator $D$ has been left undefined and \autoref{eq:syssingle} most of the term is independent of the choice of $D$.
%This is important, because in an optimization setting real and imaginary parts of the Fourier coefficients need to be treated separately.
% That is, derivatives of $\frac{d \Re \mathbf G}{d\Re \mathbf Y}$, $\frac{d \Im \mathbf G}{d\Re \mathbf Y}$, $\frac{d \Re \mathbf G}{d\Im \mathbf Y}$, $\frac{d \Im \mathbf G}{d\Im \mathbf Y}$ will be required.
Since a central part of this project requires numerical optimization of the coefficients, their real and imaginary parts need to be treated separately.
This especially means separately deriving the real and imaginary parts of system of equations by their real and imaginary parts.
However, all these cases can be reduced to $\frac{d}{d \mathbf Y}$
\begin{align*}
	\frac{d \Re}{d\Re \mathbf Y} = \Re \frac{d}{d \mathbf Y} &&& \frac{d \Re}{d\Im \mathbf Y} = -\Im \frac{d}{d \mathbf Y}\\
	\frac{d \Im}{d\Re \mathbf Y} = \Im \frac{d}{d \mathbf Y} &&& \frac{d \Im}{d\Im \mathbf Y} = \Re \frac{d}{d \mathbf Y} \text.
\end{align*}