\chapter{Background}
This chapter reviews the theories necessary for the main part of the thesis. As we are creating topic models of pretrained language models, we need to know what those are and how they work. In the following sections, we explore their relevant origin  on top of providing an overview on current evaluation and comparison techniques.

% =============================
\section{Language Modeling}
% =============================
\subsection{Definition and Application}
Language models are created to distill valuable insights from raw text without the help of a human. To name a few tasks of language models:
\begin{itemize}
    \item \textbf{Natural-language generation (NLG)} extracts and transforms information in the form of raw data or text into readable and human understandable text.
    \item \textbf{Natural-language understanding (NLU)} takes text as input and translates it into a more formal representation, so that computer applications can understand it. For example simple commands given to robots or more complex ones, like comprehension of newspaper articles.
    \item \textbf{Automatic summarization of texts} produces a readable summary of a fragment of text and can be extractive or abstractive. Extractive summarization concatenates strings with different length without knowing what they actually mean. On the other hand, abstractive summarization creates a context sensitive summary by using a combination of statistical and linguistic properties of the analyzed text. \cite{gupta2010survey}
    \item \textbf{Machine translation} translates text or speech from one language to another.
    \item \textbf{Question answering}, with or without context, answers text questions from humans. Answers are created with knowledge from large databases. This is usually a knowledge database or an unstructured collection of natural language documents. The questions themselves can be closed or open-ended.
    \item \textbf{Speech recognition} creates text equivalent to any input in the form of audio speech.
    \item \textbf{Optical character recognition (OCR)} scans visual input and transforms it into machine-encoded text. Such input can be to images of typed, handwritten or printed text in the from a scanned document, a photo of a document or a scene-photo.
    \item \textbf{Paraphrasing} is a subcategory of text-to-text generation and is the art of paraphrasing words, sentences and full texts.
\end{itemize}

The standard definition of a language model is a model of the probability distribution over strings $p(\mathbf{y})$ in a natural language. This model should provide an estimate of the (unconditional) probability of observing some string $\mathbf{y} \in\mathcal{Y}$, where $\mathcal{Y}$ is the set of all valid strings in a language. Different attempts over time resulted in different model architectures, e.g. the n-gram (unigram, bigram, trigram, etc.) model, the log-linear model and the neural model. For large pretrained language models, neural language models deliver state of the art results on various NLP tasks. Compared to traditional language models, neural models allow conditioning on increasingly larger contexts with only linear increases in the number of parameters. They parameterize words as vectors (word embeddings) and use these vectors as inputs. These parameters are learned as part of the training process of neural models. Each neural model has a specifically designed architecture for the data it was designed to learn, which puts restrictions on the learning algorithm. Furthermore, the ability to generalize to unseen data introduces an inductive bias to our model, which in itself is not a bad thing. In fact, we need our model to generalize to unseen data. This way, the model will be able to apply the learned knowledge to different subtasks it was not trained for, e.g. prediction of news articles or movie reviews.

A first attempt to neural language modeling by Bengio et al., 2003 \cite{bengio2003neural} uses feed-forward neural network models. However, with increasing lexicon size, the computations regarding the output layer become computationally intractable. The reason being that the predicted output on a particular input needs to be a probability distribution over the entire lexicon (softmax over the vocabulary). With recurrent neural networks (RNN), this limitation is avoided as they can learn arbitrary long influences without being dependant on the size of the lexicon (infinite context). In an RNN, each node is aware of its direct predecessor and the information from earlier nodes is only indirectly available through the encoding of the encodings. This leads to a loss of information as the lexicon gets bigger.

% =======================================
\subsection{Attention Mechanism}
The attention mechanism poses a solution to the loss of information in large RNN networks. It overcomes the limitation of the fixed-size vector representation of the input sequence. Attention comes in the form of a hidden layer and computes a categorical distribution (or hierarchy of categorical distributions) over the input. The input can be the direct words or intermediate input from other layers. \cite{kim2017structured}

The concept of the attention mechanism was introduced by Bahadanau et al., 2014 \cite{bahdanau2014neural} for machine translation (see fig. \ref{fig:attentionMT}) and has demonstrated success on various NLP tasks.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{figures/attention}
    \caption{Attention Mechanism for machine translation with encoder (blue) and decoder (red). \cite{attentionmechanism}}
    \label{fig:attentionMT}
\end{figure}
Later, Yang et al., 2016 \cite{yang2016hierarchical} introduced the intra or self-attention mechanism for document classification. Because not all words contribute equally to the representation of the meaning of a sentence, the attention mechanism was adapted to a word level, such that it relates different positions of a single sequence to compute a representation of the sequence.

Vaswani et al., 2017 \cite{vaswani2017attention} exploited this technique in their proposed architecture, the Transformer, and introduced an adaptation of self-attention and the new multi-head attention.

% =======================================
\subsubsection{Self-Attention}
The self-attention mechanism \cite{vaswani2017attention} makes inputs aware of each other and tells them which are the more important ones. The outputs of this mechanism are aggregates of these interactions and the attention scores. Every input is represented by three vectors, i.e. a key $k$, a query $q$ and a value $v$. They are a product of the input vector and three global weight matrices, i.e. $W^Q,\;W^K,\;W^V$ (see fig. \ref{fig:selfattention}), which are learned during the training process. Every input uses those matrices to calculate their three representations.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{figures/selfattention}
    \caption{Self-Attention Mechanism: Illustrating input/embeddings (green), query (pink), keys (orange) and values (blue). \cite{illustratedtransformer}}
    \label{fig:selfattention}
\end{figure}
Depending on where to put attention, different scoring functions are used. In the Transformer architecture, the scaled-dot-product (see fig. \ref{fig:scaled-dot-product-attention}) is used.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.2\textwidth]{figures/scaled-dot-product-attention}
    \caption{Scaled Dot-Product. \cite{vaswani2017attention}}
    \label{fig:scaled-dot-product-attention}
\end{figure}
The output of the scaled-dot-product attention are the weighted values $V$ according to eq. \ref{eq:sdp}.
\begin{equation}\label{eq:sdp}
    \text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}\right)\mathbf{V}
\end{equation}

% =======================================
\subsubsection{(Masked) Multi-Head Attention}\label{sec:multheadat}
the goal is to make the different inputs aware of each other at different positions. Apart from the collectively learned weight matrices ($W^Q,\;W^K,\;W^V$), we calculate the output for input $x_1$ by using the query vector $q_1$ in all self-attentions instead of using each their own query vector ($q_2, q_3, ...)$. The resulting self-attention vectors are then summed up giving us the multi-head attention vector for input $x_1$. This process is repeated for every input, but in practice, this is done very efficiently with matrix multiplications (see fig. \ref{fig:multi-head-attention}).
\begin{figure}[H]
    \centering
    \includegraphics[width=0.3\textwidth]{figures/multi-head-attention}
    \caption{Multi-Head Attention. \cite{vaswani2017attention}}
    \label{fig:multi-head-attention}
\end{figure}

The masked multi-head attention is a slight variation of the normal multi-head attention. Before the softmax step (see fig. \ref{fig:scaled-dot-product-attention}), the multi-head-attention layer uses only earlier positions in the output sequence. This is done by setting all future positions to minus infinity.

% =======================================
\subsection{The Transformer}
Vaswani et al., 2017 \cite{vaswani2017attention} posed a solution, to the limitations of the RNN architectures by combining original attempts to neural language modeling, which were based on feed-forward networks, with the attention mechanism. The so called Transformer architecture relies entirely on the attention mechanism to draw out the global dependencies between the input and output \cite{illuaatte}. It predicts $y_t$ by only using the $k$ most recent inputs, $x_{t-k+1},...,x_t$ (fig. \ref{fig:feedforward0}). To be able to depend on the most $k$ recent inputs, the model must assume a strong conditional independance.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/feedforward0}
    \caption{Illustration of an autoregressive feed-forward language model. \cite{feedforward}}
    \label{fig:feedforward0}
\end{figure}
Please note that in the next step the output is generated conditioned on the $k$ most recent inputs and becomes part of the input. The window of $k$ inputs then shifts one step to the right (fig. \ref{fig:feedforward1}).
\begin{figure}[H]
    \centering
    \includegraphics[width=1.0\textwidth]{figures/feedforward1}
    \caption{Illustration of the next step of an autoregressive feed-forward language model. \cite{feedforward}}
    \label{fig:feedforward1}
\end{figure}
The originally proposed Transformer architecture (see fig. \ref{fig:transformer-vis}) is an alteration of the sequence-to-sequence or encoder-decoder model \cite{attentionmechanism}.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.7\textwidth]{figures/transformer}
    \caption{Illustration of the original Transformer Architecture. \cite{vaswani2017attention}}
    \label{fig:transformer-vis}
\end{figure}
Fig. \ref{fig:transformer-vis} just visualizes the incorporation of one encoder and one decoder. In practice, multiple encoders and decoders are put in series (see fig. \ref{fig:transformer-mult}).
\begin{figure}[H]
    \centering
    \includegraphics[width=0.6\textwidth]{figures/mult_trafo}
    \caption{Illustration of the encoder-decoder stack. \cite{howdotrafo}}
    \label{fig:transformer-mult}
\end{figure}
We can see that first the learned embeddings get encoded with a positional encoding. This is done by adding a one-hot vector to the embedding, which indicates the position of each word.

The encoder encodes the full sequence into a context vector with fixed length. It is doing that by stepping through the input time steps. In the Transformer architecture this context vector comes in the form of a set of attention vectors $K$ and $V$.

The decoder steps through the output time steps. While doing that, it reads from the two attention vectors, $K$ and $V$, mentioned before. These are used by each decoder in its \textit{Multi-Head Attention} (see sec. \ref{sec:multheadat}) block and help the decoder focus on appropriate places in the input sequence. After multiple decoding steps, the output after a linear and softmax block is a probability distribution where a word is sampled from (see sec. \ref{sec:gentext}). The sampled word of each iteration is then appended to the original input and again fed into the decoder to calculate the next output, and so on. Just like with the encoder inputs, the positional encodings are added to the word embeddings at every iteration.

After basic training or pretraining, the language model is usually fine-tuned for specific NLP tasks. Fine-tuning means training on a task-specific dataset. Depending on performance and task, additional linear layers are added at the end of the language model and some parameters of the model are frozen, i.e. they do not change while fine-tuning. \cite{trafoarch}

% =======================================
\subsection{Transformer Based Architectures}
The realization that the encoder and decoder part can be used separately for different NLP tasks led to different language models based on the Transformer architecture.

% =======================================
\subsubsection{GPT-1}
Focusing research on the decoder part (see fig. \ref{fig:gpt}) resulted in the well-known GPT-1 model \cite{radford2018improving}. GPT-1 computes the probability $p(\mathbf{y})$ of a complete sentence given nothing or a beginning of a sentence, where $\mathbf{y}\in\mathcal{Y}$ and $\mathcal{Y} = \{\textsc{bos}+\mathbf{v}+\textsc{eos}\mid \mathbf{v} \in \mathcal{V}^* \}$. $\mathbf{y}= \langle y_1, y_2, \dots\rangle$ is a sequence consisting of natural language tokens  $y\in\mathcal{V}$ from a set vocabulary $\mathcal{V}$, padded with special \textsc{bos} and \textsc{eos} tokens to demarcate the beginning and end of a string, respectively.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.3\textwidth]{figures/gptarch}
    \caption{The GPT-1 Architecture. \cite{radford2018improving}}
    \label{fig:gpt}
\end{figure}
GPT-1 is a unidirectional model, meaning it is only aware of what came before. It learns using unlabeled data and is fine-tuned by providing examples of specific downstream tasks like classification, sentiment analysis, textual entailment, etc. Its major advantage is the massive volume of data it was pretrained on. This makes adapting the \textit{raw}-trained model possible for different NLP tasks with very little data.

% =======================================
\subsubsection{GPT-2}
The GPT-2 \cite{gpt-2} model pretty much follows the architecture of the original GPT-1 model (see fig. \ref{fig:gpt2}). New additions are that each sub-block now has the layer normalization from the end in the beginning and to the layer normalization block in the middle an additional layer is added (see fig. \ref{fig:gpt2}). The weights of the residual layers (used at initialization) are newly scaled by a factor of $1/\sqrt{N}$ where $N$ is the number of residual layers. This accounts for the accumulation on the residual path with model depth. The vocabulary is expanded to 50,257 tokens, the context size to 1024 tokens and the batchsize to 512 documents \cite{gpt-2}. This resulted in 1.5B parameters for the largest model in contrast to the 117M parameters for GPT-1.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.4\textwidth]{figures/gpt2}
    \caption{The GPT-2 Architecture. \cite{heilbron2019tracking}}
    \label{fig:gpt2}
\end{figure}
The model was trained on the WebText dataset \cite{radford2019language}. This dataset was created by scraping web pages which have been curated, respectively filtered, by humans and then filtered with a combination of content extractors. Important to note is that Wikipedia was not included as it is a common data source for other datasets and could complicate analysis due to overlapping training data with test evaluation tasks. \cite{gpt-2}

% =======================================
\subsubsection{Transformer-XL}
Transformer-XL \cite{transformer-xl} builds upon the decoder part of the Transformer architecture. It extends the memory of the fixed-length context by conditioning on the sequences that came before. Without disrupting temporal coherence, Transformer-XL additionally solves the issue of context fragmentation. Context fragmentation is one of the main limitations of the original Transformer model and happens when a model tries to fit a document, the document gets segmented because it is too big, and long-term dependencies are then lost.

The decoder from the original Transformer architecture is modified as such, that it functions like a cell from the RNN architecture where the memory transfer happens in the attention mechanism (see fig. \ref{fig:transformer-xl-arch}).
\begin{figure}[H]
    \centering
    \includegraphics[width=0.5\textwidth]{figures/transformer-xl-arch}
    \caption{The Transformer-XL model. \cite{dowdell2020language}}
    \label{fig:transformer-xl-arch}
\end{figure}
Transformer-XL was trained separately on multiple datasets including Wikitext-103 and experiments have shown that it learns much longer dependencies.

% =======================================
\subsubsection{Probing Language Models}
A language model tries to learn the hidden structures and connections between sentences and words in the training corpus. To shed light on these hidden properties, researchers try to isolate linguistic phenomena through specifically designed multi-class classification problems, so called probing tasks. We assume that if a system has a good performance on a probing task, it has encoded the phenomena we were probing for. Those probes can have a low computational cost but there is a hidden complexity, i.e. we do not know the ideal setup. This means, we do not know how complex our probe has to be to find a property, e.g. whether a linear model suffices or deep learning is required. Thus, what part of the training data we use for training the probe, is rather unclear. We know in fact that with enough training data, a complex enough model is able to learn any linguistic phenomena we want it to find, and we therefore have to be aware of false positives. Recent studies have shown that it is definitely possible to memorize a good amount of specific labeling decisions without being dependant on the representation's linguistic properties. \cite{desintpro, gul2019linspector, hewitt2019structural, white2021non, lingwis}

% =======================================
\section{Generating Text From Pretrained Models}\label{sec:gentext}
% =======================================
\subsection{Ancestral Sampling}\label{sec:sampling}
Based on the standard definition of a language model and therefore on an autoregressive language model $p_\theta$, a string $\mathbf{y}\sim p_\theta$ can be generated from the conditional probability distributions. Given the relationship in eq. \ref{eq:joint}, we can set $y_1 = \textsc{bos}$ or the empty string, and sample $y_t\sim p(\cdot\mid\mathbf{y}_{<t})$ until we sample the \textsc{eos} token.
\begin{equation}\label{eq:joint}
    p(\mathbf{y})=p(y_1,...,y_t)=p(y_1)\cdot p(y_2\mid y_1)\cdot...\cdot p(y_t\mid y_{t-1})=\prod_{t=1}^{|\mathbf{y}|} p(y_t\mid\mathbf{y}_{<t})
\end{equation}
This means, using the chain rule of probability, we can model the probability of an entire string as the product of the conditional probability of each word given its prior context. Here we define $\mathbf{y}_{<t} = \langle y_1, \dots, y_{t-1}\rangle$. We call this ancestral sampling, as we start by sampling the ancestor $p(y_1)$, then $p(y_2\mid y_1)$ and so on.

% =======================================
\subsection{Top-P Sampling}
In Top-P or \textit{nucleus} sampling \cite{holtzman2019curious} we choose from the smallest possible set of words whose cumulative probability exceeds the probability $p$. Among this set of words, the probability mass is redistributed. Thus, the number of words in the set can dynamically increase and decrease according to the probability distribution of the next word.

Formally, Top-P sampling reduces the set of words $\mathcal{V}$ from the distribution $p(y\mid y_{1:i-1})$ to the smallest subset $\mathcal{V}^{(p)}\subset \mathcal{V}$, s.t.
\begin{equation}
    \sum_{y\in \mathcal{V}^{(p)}}{p(y\mid y_{1:i-2})}\geq p.
\end{equation}
Illustrated by an example in fig. \ref{fig:topp}, we set $p=0.93$. Top-P sampling chooses the minimum number of words, such that their summed up probability mass exceeds $p=93\%$. Here, this set is defined as $\mathcal{V}_{top-p}$ (per definition, equal to $\mathcal{V}^{(0.93)}$). On the left side the 9 most likely words have to be included, where on the right side only the top 3 words are needed to exceed $93\%$. \cite{trafogen}
\begin{figure}[H]
    \centering
    \includegraphics[width=1.0\textwidth]{figures/topp}
    \caption{Top-P sampling example. \cite{trafogen}}
    \label{fig:topp}
\end{figure}

% =======================================
\subsection{Typical Sampling}
Typical sampling \cite{meister2022typical} tries to imitate human communication. Instead of choosing words that have a certain probability mass, like Top-P sampling, it samples according to the information content of the words. The information content of the set of words is expected to be close to the information content of the model. Which is equal to its conditional entropy, i.e. we assume that for human-like text eq. \ref{eq:toppdiff} is very small.
\begin{equation}
    \varepsilon=\left|H(p(\cdot\mid \mathbf{y}_{<t}))+log\, p(y_t\mid \mathbf{y}_{<t})\right|
\label{eq:toppdiff}
\end{equation}
The sampled distribution is then limited to the word subset as $\mathcal{V}^{(\tau)}\subseteq\mathcal{V}$. This subset has to minimize eq. \ref{eq:toppdiff2} at each time step. Therefore, the subset changes constantly.
\begin{equation}
    \sum_{y\in\mathcal{V}^{(\tau)}}{\varepsilon},\text{ s.t. }\sum_{y\in\mathcal{V}^{(\tau)}}{p(y\mid \mathbf{y}_{<t})}\geq\tau.
\label{eq:toppdiff2}
\end{equation}
After reducing the word count in the subset, it has to be renormalized according to eq. \ref{eq:norm}.
\begin{equation}
    \begin{aligned}
        \pi(y\mid\mathbf{y}_{<t})=\left\{
        \begin{matrix*}[l]
            p(y\mid\mathbf{y}_{<t})/\sum_{y\in\mathcal{V}^{(\tau)}}{p(y\mid\mathbf{y}_{<t})}, & \text{if }y\in \mathcal{V}^{(k)} \\
            0, & \text{else}
        \end{matrix*}\right.
    \end{aligned}
\label{eq:norm}
\end{equation}

% =======================================
\subsection{Evaluation and Metrics}
In general, it is a good idea not only to evaluate our trained model by the loss or the accuracy, but also by testing them on an actual task. We call that extrinsic evaluation. By embedding our trained model directly in an application and measuring how much the application improves, we can immediately see how different models perform and behave. For example, if we want to measure the difference in performance of two models trained for speech recognition. To achieve that, we run the speech recognizer two times and can then directly see which one of the results is a more accurate transcription. This process requires us to train a full system, which makes it computationally expensive and, depending on your model and processing power, very slow. Therefore, it is sometimes more practical to have a metric (better than accuracy or loss) that can cost-efficiently evaluate potential improvements in a language model and its quality, independent of any application. We call this intrinsic evaluation. This evaluation needs, apart from the model, a test set. The models we want to compare are then fitted onto this test set and whichever model assigns a higher probability, or rather more accurately predicts the test set, is a better model. The metric itself is based on the probability of the test set. Important is that any part of the test set should not be used for training. Otherwise, it would introduce a bias to the model which makes the probabilities for the test set higher than they actually are. An example for a widely used probability-based metric is Perplexity, or short PP. \cite{slp3}

% =======================================
\subsubsection{Perplexity}\label{sec:perplexity}
Perplexity \cite{perplexity} is an intrinsic evaluation method and is the inverse probability of the test set, normalized by the number of words. For a test set$_w=w_1,w_2,...,w_N$ the Perplexity is:
\begin{equation}
    PP(W)=\sqrt[n]{\frac{1}{P(w_1,...,w_N)}}.
    \label{eq:perplexity}
\end{equation}
To a human-like and syntactically correct text, a model assigns a high probability. Low probabilities should be the result of fake, incorrect, or highly infrequent sentences. Therefore, the best model is going to be the one that assigns the highest probability to a real and correct test set. By assigning a high probability, the model is not perplexed (surprised) by it (see fig. \ref{fig:perplexity}). This implicates that a language model with low perplexity has a good understanding of how the language works. A simpler interpretation of the perplexity score is interpreting it as the weighted branching factor. This means, if we have a perplexity of 42, the model is confused because it has to pick from 42 words when trying to guess the next word. \cite{perplexity}
\begin{figure}[H]
    \centering
    \includegraphics[width=1.0\textwidth]{figures/perplexity_example}
    \caption{Example of the Perplexity score. \cite{perplexity}}
    \label{fig:perplexity}
\end{figure}
Perplexity can also be defined as the exponential of the cross-entropy (see eq. \ref{eq:crosspp}), which is equivalent to the previous definition (eq. \ref{eq:perplexity}).
\begin{equation}
    PP(W)=2^{H(W)}=2^{-\frac{1}{N}log_2 P(w_1,...,w_N)}
\label{eq:crosspp}
\end{equation}

% =======================================
\section{Topic Modeling}
% =======================================
\subsection{Definition and Application}
A \textit{topic} is normally a phrase or a set of words that describes the content of a text. Topic modeling tries to find such topics. It is an unsupervised machine learning technique which has the ability to analyze a set of documents. Within those documents it detects patterns in multiple sentences down to the word level. It then clusters groups of words and similar expressions to best characterize this set of documents. This technique is useful for a wide range of data analysis from social media posts, emails, chats, customer support conversations, up to cancers' genomic samples. The process of dividing a corpus can in general be divided into two parts:
\begin{enumerate}
    \item A set of topics, where the topic is described in some specific way
    \item Multiple sets of grouped documents according to the topics they cover
\end{enumerate}
We assume that every document is represented by a distribution over topics. This distribution is the result of the probability mass for every topic covering this document. Those topics are defined and learned in the process, like latent variables. It is an unsupervised technique and does not require a training dataset. Hidden patterns and insights are found in the process. This does not guarantee accurate results. \cite{tmint}

An well-known topic modeling technique is called latent Dirichlet allocation (LDA) \cite{blei2003latent}.

% =======================================
\subsection{Latent Dirichlet Allocation (LDA)}
LDA \cite{blei2003latent} is a statistical model of a collection of documents. The model assumes that a document consists of multiple topics and a topic of multiple words. This variability of random multinomial distributions is modeled with the help of the Dirichlet distribution. The Dirichlet distribution basically models a probability distribution of probability distributions. The figure below illustrates the intuition:
\begin{figure}[H]
    \centering
    \includegraphics[width=1\textwidth]{figures/topicmodel}
    \caption{The intuition behind Latent Dirichlet Allocation. \cite{kim2019insider}}
\end{figure}
We assume that \textit{every document is a probability distribution over topics} (far right) and \textit{every topic is a probability distribution over words} (far left). The number of topics is a parameter defined by the user at initialization.
We also assume that each document is generated by the following steps:
\begin{enumerate}
    \item Randomly choose a distribution over topics (the histogram on the right)
    \item Randomly choose the number of words for the document.
    \item For each word:
    \begin{enumerate}
        \item Randomly choose a topic (the colored coins)
        \item Randomly choose a word from the corresponding topic distribution (color from the coin matching the topic color on the left)
    \end{enumerate}
\end{enumerate}
To understand this process mathematically (see fig. \ref{fig:lda}), we reverse engineer a corpus. First, we assume a corpus has $M$ documents and a document has $N$ words $w_n$. The number of words is sampled from a poisson distribution $N\sim Poisson(\xi)$. Every word $w_n$ in a document is sampled from a multinomial distribution $\beta$ conditioned on the topic $z_n$. The topic $z_n$ is again sampled from a multinomial distribution $z_n\sim Multinomial(\theta)$. And at last, the multinomial distribution $Multinomial(\theta)$, that defines the topic distribution for a document, is sampled from a Dirichlet distribution: $\theta\sim Dir(\alpha)$.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/lda}
    \caption{Graphical model representation of the LDA model. \cite{blei2003latent}}
    \label{fig:lda}
\end{figure}

Simply speaking, if an LDA model is trained on a corpus, what we get are two sets of probability distributions. One set is the topic-word distribution where we get the multinomial distribution of words for every topic. The other is the multinomial distribution of topics for every document.

With those two sets of distributions, we can make statements about what topics occur within our documents and what words those topics define. We can go even further, with LDA we can generate documents and therefore make assumptions what topics future documents might be about.

However, a new document may contain words or topics that did not appear in any of the documents in the training corpus. These topics and words would have zero probability in the topic-word distribution. To smooth the multinomial parameters, the document-topic distribution is sampled from a Dirichlet distribution, but not the topic-word distribution. For this reason, we have to modify our LDA model. The modification as seen in fig. \ref{fig:lda_smooth} samples the multinomial topics-word distribution, $\beta$, also from a Dirichlet distribution $\beta\sim Dir(\eta)$.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.8\textwidth]{figures/lda_smooth}
    \caption{Graphical model representation of the smoothed LDA model. \cite{blei2003latent}}
    \label{fig:lda_smooth}
\end{figure}
There are different strategies to train an LDA model. The original paper trains the LDA model through inference with variational Bayes \cite{blei2003latent} but since then there have been various other approaches, e.g. streaming Gibbs sampling \cite{gao2016streaming}, Markov random fields \cite{he2019end}, variational message passing \cite{taylor2021variational}, and many more.

In this work we are using LDA in combination with online variational Bayes \cite{hoffman2010online} which relies on online stochastic optimization. This technique manages to effortlessly analyze a massive amount of documents and can adapt to new documents at a later point in time without being trained from scratch.

% =======================================
\subsection{Neural LDA}
The parameters of the original LDA model have to be learned by variational inference techniques (e.g. EM algorithm for empirical Bayes, Gibbs sampling). This limits the model, especially for new data. Neural LDA proposes a new inference technique (AVITM) \cite{neuralLDA} that utilizes the inference network and does not require running slow variational optimization. To model a Dirichlet distribution with a neural network, they use a Laplace approximation \cite{mackay1998choice} so that the model prior could be assumed Gaussian. With that, the inference network can be defined as a neural network by using the reparametrization trick \cite{kingma2013auto} and the neural network is then able to successfully mimic the probabilistic inference process.

% =======================================
\subsection{Evaluation and Metrics}
% =======================================
\subsubsection{Metrics of Quality (Topic Coherence)}\label{sec:coherence}
According to João Pedro \cite{topiccoherencemeasures}, topic modeling algorithms are a theoretical technique based on mathematics and statistics. However, a mathematically sound and perfect model does not necessarily reflect an ideal model from a human perspective. For example, a topic modeling algorithm containing the following two topics shows this problem.
\begin{enumerate}
    \item Topic: fish, shark, tuna, sea. \textit{(In our sense a good topic)}
    \item Topic: nurse, brick, cheese, dark. \textit{(In our sense rather a bad topic)}
\end{enumerate}
As a human, one might think the first topic contains more correlating words. On the other hand, for the algorithm, they are probably equal concerning correlating words. Considering our goal to understand data, we want topics that are human-friendly. Misleading and meaningless topics are a consequence of blindly trusting the algorithm. Topic coherence measures aim to put the quality of human perception about the created topics in a number to make the evaluation more feasible. Topic Coherence answers the question of how well a topic fits into a text considering the word combinations (the context) in it. This also means that the measure is not only dependent on the topic itself but also on the original text used for training the topic model. In general, we not only want to judge one topic but all generated topics. Therefore, we have to somehow combine all individual topic scores at the end.

The topic coherence measure is defined in a pipeline-like structure (segmentation, probability calculation, confirmation measure, aggregation) as shown in fig. \ref{fig:tcm}. Each combination of those components uniquely defines different types of coherence scores. The reference corpus and the topics $t$ are received as inputs and a real valued confirmation measure $c$ is returned.
\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/topiccoherence}
    \caption{General topic coherence measures structure. \cite{topiccoherencemeasures}}
    \label{fig:tcm}
\end{figure}
\textbf{The segmentation part} $s$ creates pairs of word subsets. For example, let $W={w_1,...,w_n}$ be the top-n most probable words of a topic $t$. Then, let $S$ be the set of subset pairs from $W$: $S={(W^{'},W^{*}),W^{'},W^{*}\subseteq W}$. $W^{*}$ is used to confirm $W^{'}$ in the probability calculation part. To get a better grasp of this segmentation let us look at some concrete examples for $W={"seal","egg","fibsh"}$:
\begin{equation}
    \begin{split}
        S^{one}_{one}(W)=\{&(seal,egg),(seal,fibsh),(egg,seal),\\
        & (egg,fibsh),(fibsh,seal),(fibsh,egg)\}
    \end{split}
\end{equation}
\begin{equation}
    \begin{split}
        S^{one}_{all}(W)=\{&(\{seal\},\{egg,fibsh\}),(\{egg\},\{seal,fibsh\}),\\
        & (\{fibsh\},\{seal,egg\})\}
    \end{split}
\end{equation}

\textbf{The probability calculation part} $p$ defines the function used to calculate the probabilities drawn from the corpus. Examples of such functions are the following:
\begin{itemize}
    \item \textbf{P}\textit{bd} (boolean document) calculates the probability as the number of documents the word or words $w$ occur in, divided by the total number of documents.
    \item \textbf{P}\textit{bs} (boolean sentence) calculates the probability as the number of sentences the word or words $w$ occur in, divided by the total number of sentences.
    \item \textbf{P}\textit{bsw} (boolean sliding window) calculates the probability as the number of sliding windows the word or words $w$ occur in, divided by the total number of sliding windows.
\end{itemize}
\textbf{The confirmation measure part} $m$ calculates a score by evaluating the probability function over the segmentations $S$ from the previous part. A low confirmation measure implicates that the words in $W^{'}$ are disconnected with the words in $W^{*}$ and vice versa (see fig. \ref{fig:tcp}).
\begin{figure}[H]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/coherenceproba}
    \caption{Example of application of a confirmation measure $m$. \cite{topiccoherencemeasures}}
    \label{fig:tcp}
\end{figure}
There are two different types of confirmation measures:

\textit{Direct confirmation measures} use the subsets $W^{'}$ and $W^{*}$ to calculate the resulting score. E.g.:
\begin{equation}
    m_r(S_i)=\frac{P(W^{'},W^{*})}{P(W^{'})P(W^{*})}
\end{equation}
\begin{equation}
    m_{lr}(S_i)=log\frac{P(W^{'},W^{*})+\epsilon}{P(W^{'})P(W^{*})}
\end{equation}
\begin{equation}
    m_c(S_i)=\frac{P(W^{'},W^{*})}{P(W^{*})}
\end{equation}

\textit{Indirect confirmation measures} use the subset $W^{'}$ and $W^{*}$ separately to build a vector for all the words $W$ (see eq. \ref{eq:confmeas1}, \ref{eq:confmeas2}).
\begin{equation}
    \vec{v}_m(W^{'})=\left\{\sum_{w_i\in W^{'}}{m(w_i,w_j)}\right\}_{j=1,2,...,|W|}
    \label{eq:confmeas1}
\end{equation}
\begin{equation}
    \vec{v}_m(W^{'})=\left\{\sum_{w_i\in W^{'}}{m(w_i,w_j)}\right\}_{j=1,2,...,|W|}
    \label{eq:confmeas2}
\end{equation}
Those two vectors are combined with, for example, the cosine similarity function (see eq. \ref{eq:cos}).
\begin{equation}
    \tilde{m}_{cos}(W^{'},W^{*})=s_{cos}(\vec{v}_m(W^{'}),\vec{v}_m(W^{*}))
    \label{eq:cos}
\end{equation}
The idea behind the indirect confirmation measure is to find relations that the direct method could not.

\textbf{The aggregation part} $\sigma$ combines all previously calculated values into one. This value is the final topic coherence score and can be, for example, the arithmetic mean.

Currently, the score that has the highest correlation with all available human topic-ranking data is the coherence score $C_v$ \cite{syed2017full}. The type of the coherence score, as mentioned before, is uniquely defined by the segmentation, the probability calculation, the confirmation measure and the aggregation:
\begin{itemize}
    \item $S_{set} = \{(W', W^{*})\mid W' = \{w_i\}; w_{i} \in W; W^{*} = W\}$
    \item \textbf{P}\textit{bsw} - Boolean Sliding Window
    \item $\phi_{S_i}(\vec{u},\vec{w})=\frac{\sum_{i=1}^{|W|}{\vec{u}_i}\cdot\vec{w}_i}{\|\vec{u}\|_2\cdot\|\vec{w}\|_2},\;$ with $\;
           \vec{v}(W^{'})=\left\{\sum_{w_i\in W^{'}}{\textbf{NPMI}(w_i,w_j)^{\gamma}}\right\}_{j=1,...,|W|}\;$ and
            $\;\textbf{NPMI}(w_i,w_j)^{\gamma}=\left(\frac{\text{log}\frac{P(w_i,w_j)+\epsilon}{P(w_i)\cdot P(w_j)}}{-\text{log}(P(w_i,w_j)+\epsilon}\right)$
    \item $\sigma=$ The arithmetic mean
\end{itemize}

A full insight in topic coherence measures with more examples of each part can be found in \textit{Exploring the Space of Topic Coherence Measures} by Röder et al. \cite{roder2015exploring}

% =======================================
\subsubsection{Metrics of Differences Between Models (Distance)}\label{sec:diffmet}
Topic models generate probability distributions and one way to compare those models with each other is to measure the similarity between them. We can do this by using the Jensen-Shannon distance. The Jensen-Shannon distance is the square root of the Jensen-Shannon divergence, also known as the information radius or total divergence to the average. The Jensen-Shannon divergence \cite{fuglede2004jensen} is based on the Kullback-Leibler divergence ($D_{KL}$) with the difference that it is symmetric and it always has a finite value.
\begin{equation}
    JSD(P||Q)=\frac{1}{2}D_{KL}\left(P\left|\left|\frac{1}{2}(P+Q)\right.\right.\right)+\frac{1}{2}D_{KL}\left(Q\left|\left|\frac{1}{2}(P+Q)\right.\right.\right),
\end{equation}
with $D_{KL}(P||Q)$ denoting the relative entropy of $P$ with respect to $Q$.

The more similar the two distributions $P$ and $Q$ are, the closer to zero the resulting value is. The evaluation between two topic models requires both to have been trained with the same vocabulary. The vocabularies have to be exactly the same, including the positions of the words. Otherwise, additional and very tedious coding is required.

Furthermore, the topics of two topic models are not guaranteed to match the same topic IDs (1-1, 2-2, etc.). Therefore, the result is going to be a matrix ($M_{i,j}$ with $i,j\in[0,...,t]$, where $t=\textit{num topics}-1$), where all topics are compared to each other. The matching topics are then found by the lowest score for every line (or row).


\documentclass{anstrans}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\title{Mixed Hybrid Finite Element Method Eddington Acceleration of Discrete Ordinates Source Iteration}
\author{Samuel S. Olivier*, Jim E. Morel}

\institute{Department of Nuclear Engineering, Texas A\&M University, College Station, TX 77843}

\email{smsolivier@tamu.edu}

%%%% packages and definitions (optional)
\usepackage{graphicx} % allows inclusion of graphics
\usepackage{booktabs} % nice rules (thick lines) for tables
\usepackage{microtype} % improves typography for PDF

\usepackage{xspace}
\usepackage{siunitx}

\newcommand{\SN}{S$_N$\xspace}
\renewcommand{\vec}[1]{\bm{#1}} %vector is bold italic
\newcommand{\vd}{\bm{\cdot}} % slightly bold vector dot
\newcommand{\grad}{\vec{\nabla}} % gradient
\newcommand{\ud}{\mathop{}\!\mathrm{d}} % upright derivative symbol
\newcommand{\pderiv}[2]{\frac{\partial #1}{\partial #2}}
\newcommand{\dderiv}[2]{\frac{\ud #1}{\ud #2}}
\newcommand{\edd}{\langle \mu^2 \rangle}

% add section on MHFEM system
% order of accuracy of accelerated system

\begin{document}
\section{Introduction}
	% Two of the most challenging computational tasks are radiation transport and hydrodynamics.
	One of the most challenging computational tasks is simulating the interaction of radiation with matter.
	A full description of a particle in flight includes three spatial variables ($x$,$y$ and $z$), two angular or direction of flight variables ($\mu =$ the cosine of the polar angle and $\gamma =$ the azimuthal angle), one energy variable ($E$) and one time variable ($t$). Numerical solutions require discretizing all seven variables leading to immense systems of algebraic equations. In addition, material properties can lead to vastly different solution behaviors making generalized numerical methods for radiation transport difficult to attain \cite{adams}.

	% The conservation of mass, momentum and energy in hydrodynamics simulations leads to a hyperbolic system of partial differential equations dependent on the time derivatives of velocity and two state variables. This results in five variables but only three equations leading to the requirement of additional equations to reach problem closure \cite{hydro}.

	% Radiation transport and hydrodynamics can be combined using operator splitting, where the radiation transport and hydrodynamics

	A national lab with whom we work has plans to develop a high-order radiation-hydrodynamics code. The hydrodynamics portion is discretized using the Mixed Hybrid Finite Element Method (MHFEM) where values are taken to be constant within a cell with discontinuous jumps at both cell edges \cite{mhfem}. MHFEM is particularly suited for hydrodynamics but not for radiation transport. This work seeks to develop an acceleration scheme capable of robustly reducing the number of iterations in Discrete Ordinates Source Iteration calculations while being compatible with MHFEM multiphysics.

\section{Background}
	The steady-state, mono-energetic, isotropically-scattering, fixed-source Linear Boltzmann Equation in slab geometry is:
		\begin{equation} \label{eq:bte}
			\mu \pderiv{\psi}{x}(x, \mu) + \Sigma_t(x) \psi(x,\mu) =
			\frac{\Sigma_s(x)}{2} \int_{-1}^{1} \psi(x, \mu') d\mu' + \frac{Q(x)}{2} \,,
		\end{equation}
	where $\mu = \cos\theta$ is the cosine of the angle of flight $\theta$ relative to the $x$--axis, $\Sigma_t(x)$ and $\Sigma_s(x)$ the total and scattering macroscopic cross sections, $Q(x)$ the isotropic fixed-source and $\psi(x, \mu)$ the angular flux \cite{adams}. The factors of 1/2 are consistent with the following definition of the scalar flux:
		\begin{equation} \label{eq:phiDef}
			\phi(x) = \int_{-1}^1 \psi(x, \mu) \ud \mu \,.
		\end{equation}
	Equation \ref{eq:bte} is an integro-differential equation due to the placement of the unknown, $\psi(x,\mu)$, under both a derivative and an integral.

	The Discrete Ordinates (\SN) angular discretization sets $\mu$ to discrete values stipulated by an $N$--point Gauss quadrature rule. The scalar flux is then
		\begin{equation} \label{eq:quad}
			\phi(x) = \int_{-1}^1 \psi(x, \mu) \ud\mu
				\xrightarrow{\text{S}_N} \sum_{n=1}^N w_n \psi_n(x) \,,
		\end{equation}
	where $\psi_n(x) = \psi(x,\mu_n)$ and the $w_n$ are the quadrature weights corresponding to each $\mu_n$ \cite{llnl}. To remain consistent with Eq. \ref{eq:phiDef}, the quadrature weights sum to 2. The \SN equations are then
		\begin{equation} \label{eq:sn}
			\mu_n \dderiv{\psi_n}{x}(x) + \Sigma_t(x) \psi_n(x) =
			\frac{\Sigma_s(x)}{2} \phi(x) + \frac{Q(x)}{2} \,,
		\end{equation}
	where $n = 1, 2, \dots, N$ and $\phi(x)$ is defined by Eq. \ref{eq:quad}. This is now a system of $N$ coupled, ordinary differential equations.

	The Source Iteration (SI) solution method decouples the \SN equations by lagging the right side of Eq. \ref{eq:si}. In other words,
		\begin{equation} \label{eq:si}
			\mu_n \dderiv{\psi_n^{\ell+1}}{x}(x) + \Sigma_t(x) \psi_n^{\ell+1}(x) =
			\frac{\Sigma_s(x)}{2} \phi^{\ell}(x) + \frac{Q(x)}{2} \,, 1 \leq n \leq N \,,
		\end{equation}
	where the superscripts indicate the iteration index.
	Equation \ref{eq:si} represents $N$ independent, first-order, ordinary differential equations each of which are easily solved by the well-known sweeping process.

	The iteration process begins with an initial guess for the scalar flux, $\phi^0(x)$. Equation \ref{eq:si} is solved, using $\phi^0(x)$ on the right side, for $\psi_n^1(x)$. $\phi^1(x)$ is then computed using Eq. \ref{eq:quad} and is used to update the right side of Eq. \ref{eq:si}.
	This process is repeated until
		\begin{equation} \label{eq:converg}
			\frac{\|\phi^{\ell+1}(x) - \phi^{\ell}(x)\|}{\|\phi^{\ell+1}(x)\|} < \epsilon \,,
		\end{equation}
	where $\epsilon$ is a sufficiently small tolerance.

	If $\phi^0(x) = 0$, then $\phi^\ell(x)$ is the scalar flux of particles that have undergone at most $\ell - 1$ collisions \cite{adams}. Thus, the number of iterations until convergence is directly linked to the number of collisions in a particle's lifetime. Typically, SI becomes increasingly slow to converge as the ratio of $\Sigma_s$ to $\Sigma_t$ approaches unity and the amount of particle leakage from the system goes to zero. SI is slowest in large, optically thick systems with small losses to absorption. In full radiation transport simulations each iteration could involve solving for hundreds of millions of unknowns. To minimize computational expense, acceleration schemes must be developed to rapidly increase the rate of convergence of SI.

	Fortunately, the regime where SI is slow to converge is also the regime where Diffusion Theory is most accurate. A popular method for accelerating SI is Diffusion Synthetic Acceleration (DSA) where each source iteration involves both a transport sweep and a diffusion solve. DSA requires carefully differencing the \SN and diffusion steps in a consistent manner to prevent instability in highly scattering media with coarse spatial grids \cite{alcouffe,morel}. DSA is not applicable in the setting of this presentation due to the incompatibility of MHFEM and \SN and the increased computational expense of solving consistently differenced diffusion. A new acceleration method is needed that avoids the consistency pitfall of DSA.

\section{Eddington Acceleration}
	The zeroth and first angular moments of Eq. \ref{eq:bte} are
		\begin{subequations}
		\begin{equation} \label{eq:zero}
			\dderiv{}{x} J(x) + \Sigma_a(x) \phi(x) = Q(x) \,,
		\end{equation}
		\begin{equation} \label{eq:first}
			\frac{\ud}{\ud x} \edd(x) \phi(x) + \Sigma_t(x) J(x) = 0 \,,
		\end{equation}
		\end{subequations}
	where $J(x) = \int_{-1}^{1} \mu \ \psi(x, \mu) \ud \mu$ is the current and
		\begin{equation} \label{eq:eddington}
			\edd(x) = \frac{\int_{-1}^1 \mu^2 \psi(x, \mu) \ud \mu}{\int_{-1}^1 \psi(x, \mu) \ud \mu}
			% \xrightarrow{\text{S}_N} \frac{\sum_{n=1}^N \mu_n^2 w_n\psi_n(x)}{\sum_{n=1}^N w_n \psi_n(x)}
		\end{equation}
	the Eddington factor. In \SN, the Eddington factor is
		\begin{equation} \label{eq:edd_sn}
			\edd(x) = \frac{\sum_{n=1}^N \mu_n^2 w_n\psi_n(x)}{\sum_{n=1}^N w_n \psi_n(x)} \,.
		\end{equation}
	Note that no approximations have been made to arrive at Eqs. \ref{eq:zero} and \ref{eq:first}. The Eddington factor is the true angular flux weighted average of $\mu^2$ and therefore Eqs. \ref{eq:zero} and \ref{eq:first} are just as accurate as Eq. \ref{eq:bte}.

	This formulation is beneficial because Eq. \ref{eq:zero} is a conservative balance equation and---if $\edd(x)$ is known---the moment equations' system of two first-order, ordinary differential equations can be solved directly with well-established methods. However, computing $\edd(x)$ requires knowledge of the angular flux.

	In Eddington Acceleration, \SN is used to compute the Eddington factor needed to solve the moment equations. Source iteration is then:
		\begin{enumerate}
			\item Given the previous estimate for the scalar flux, $\phi^{\ell}(x)$, solve Eq. \ref{eq:si} for $\psi_n^{\ell+1/2}(x)$.
			\item Compute $\edd^{\ell+1/2}(x)$ with Eq. \ref{eq:edd_sn}.
			% \item Interpolate $\edd(x)$ onto the MHFEM grid
			\item Solve the moment equations for $\phi^{\ell+1}(x)$ using $\edd^{\ell+1/2}(x)$.
			% \item Use the moment equations' $\phi(x)$ on the right hand side of Eq. \ref{eq:si}.
			\item Update the scalar flux estimate on the right side of Eq. \ref{eq:si} with $\phi^{\ell+1}(x)$ and repeat the iteration process until the scalar flux converges.
		\end{enumerate}
	% This process is one source iteration consisting of an \SN transport step to compute the Eddington factor and an MHFEM acceleration step to compute $\phi(x)$. The scalar flux from the acceleration step is used in the right hand side of Eq. \ref{eq:si} and steps 1--4 are repeated until the acceleration step's $\phi(x)$ converges according to Eq. \ref{eq:converg}.
W
	Acceleration occurs because the angular shape of the angular flux, and thus the Eddington factor, converges much faster than the scalar flux. In addition, the moment equations model the contributions of all scattering events at once, reducing the dependence on source iterations to introduce scattering information. The solution from the moment equations is then an approximation for the full flux and not the $\ell - 1$ collided flux as it was without acceleration.

	In addition to acceleration, this scheme allows the \SN equations and moment equations to be solved with arbitrarily different spatial discretization methods. \SN can be spatially discretized using normal methods, such as Linear Discontinuous Galerkin or Diamond Difference, while the moment equations can be solved with MHFEM.

\section{Results}
	\begin{figure}
		\centering
		\includegraphics[width=8.5cm]{figs/accel.pdf}
		\caption{A comparison of the number of iterations until convergence for unaccelerated, Eddington accelerated, and DSA S$_8$ SI. }
		\label{fig:comparison}
	\end{figure}

	As a proof of concept for Eddington Acceleration, a Diamond Differenced \SN code was created along with an MHFEM solver for Eqs. \ref{eq:zero} and \ref{eq:first}. The test problem of steady-state, one-group, isotropically-scattering, fixed-source radiation transport in slab geometry with a reflecting left boundary and vacuum right boundary was used to compare unaccelerated, Eddington accelerated, and DSA S$_8$ SI. The slab had a thickness of \SI{20}{cm} and was discretized into 100 spatial cells. The total macroscopic cross section was set to \SI{1}{cm^{-1}} leading to a total optical thickness of 20 and an optical thickness per cell of 0.2.

	Figure \ref{fig:comparison} shows the number of iterations until the convergence criterion in  Eq. \ref{eq:converg} was met with $\epsilon = \num{1e-6}$ for varying ratios of $\Sigma_s$ to $\Sigma_t$. Aside from $\Sigma_s/\Sigma_t = 0$ where acceleration is not possible, the ratio of unaccelerated to Eddington accelerated iterations ranged between 2.5 and 750. This suggests that acceleration is occurring and that Eddington Acceleration does not just do twice the amount of work in each iteration.

	\begin{figure}
		\centering
		\includegraphics[width=8.5cm]{figs/eddCon_si.pdf}
		\caption{The convergence rate of $\phi(x)$ compared to $\edd(x)$ for unaccelerated S$_8$ with $\Sigma_s/\Sigma_t = 0.75$. }
		\label{fig:conv_si}
	\end{figure}

	Figure \ref{fig:conv_si} shows the convergence criterion
		\begin{equation}
			\frac{\|f^{\ell+1} - f^{\ell}\|}{\|f^{\ell+1}\|}
		\end{equation}
	as a function of unaccelerated iteration number for $f = \phi(x)$ and $f = \edd(x)$. The large drop in the convergence criterion between the first and second iterations supports the claim that the angular shape of the angular flux, and thus the Eddington factor, converges much more rapidly than the scalar flux. When compared to Fig. \ref{fig:conv_edd}, a plot of the convergence criterion versus number of iterations for Eddington accelerated S$_8$, it is clear that Eddington Acceleration transfers the fast rate of convergence of $\edd(x)$ to $\phi(x)$.

	\begin{figure}
		\centering
		\includegraphics[width=8.5cm]{figs/eddCon_mu.pdf}
		\caption{The convergence rate of $\phi(x)$ compared to $\edd(x)$ for Eddington accelerated S$_8$ with $\Sigma_s/\Sigma_t = 0.75$. }
		\label{fig:conv_edd}
	\end{figure}

	% add convergence rate of phi v edd
	% include discussion on acceleration v just doing 2 times as much work. ie actual acceleration is happening

\section{Conclusions}
	The proposed acceleration scheme successfully accelerated S$_8$ source iteration calculations in slab geometry for a wide range of $\Sigma_s/\Sigma_t$. In the pure scattering regime ($\Sigma_s = \Sigma_t$), source iteration was accelerated by a factor of 750. This scheme is especially suited for multiphysics applications because the transport and acceleration steps do not need to be consistently differenced. In addition, the acceleration step produces a conservative solution that is computationally inexpensive compared to a transport sweep. Future work that will also be presented is the application of Eddington Acceleration to Linear Discontinuous Galerkin discretized \SN.

% \section{Acknowledgments}

\bibliographystyle{ans}
\bibliography{../bibliography.bib}
\end{document}


\chapter{Conclusion}\label{chap:conclusion}
In order to combat the issue of contamination of publicly accessible water supplies, namely fecal contamination, the \cp{} Biological Sciences Department teamed up with the \cp{} Computer Science Department to build a library-based \mst{} method, called \cplop{}.
Using the \ecoli{} isolated from fecal samples as \fiblong{}, \cp{} students pyrosequence the \pcr{}-amplified \itslong{} regions of the \ecoli{} and store the resulting vector, called a \pyro{}, in the \cplop{} database for later retrieval, analysis, and comparison.
This thesis investigates two \mst{} methodologies: a \dbased{} clustering method built for \cplop{} that clusters for \bslongs{} in order to classify an unknown \isol{} and the \kraplong{} (\krap{}), a set of four \knnlong{} list resolution strategies for data with multiple \compfuncs{}.

\section{Clustering for \BSlongs{}}

In this paper, we study the accuracy of a clustering-based MST approach which scales significantly better: the bacterial isolate information stored in \cplop{} is clustered using an efficient density-based clustering technique.
It clusters data by taking two parameters --- the minimum number of neighbors and an \eps{} range that those neighbors must be within to form a cluster --- and performs range queries in a spatial index that performs \bigo{\log{n}} look-up on neighbors using a comparison metric.
Compared to previous work \cite{DBLP:conf/bibm/McGovernDKBVG15, montana2013ontological}, it requires fewer comparisons to other \isols{} and computational resources, by being able to perform reasonably fast clusterings on a consumer laptop in minutes.

To ascertain how well this technique classifies, we build a notion of cluster purity.
By calculating the proportion of the entire cluster that the most-plural \spec{} makes up, we hope to understand how a density-based clustering algorithm clusters the \cplop{} data.
Furthermore, we inspect coverage and overall accuracy.

Results are that for \minneigh{} between 1 and 5, respectively, we are able to cluster between 72.9\% and 52.1\% of the \isols{} in \cplop{}, with between 51.0\% and 41.2\% falling into pure clusters and another 34.4\% to 43.8\% falling into clusters where their \spec{} is the most dominant.
Most clusters have high purity and low number of unique species, which is promising for using this for \mst{}.
Transient strains are also visible in the clustering technique, which will further aid the biologists working on \cplop{} in researching transient strains and improving \mst{} techniques.
Future work will leverage other \mst{} techniques designed for \cplop{} against this to make up for the lack of coverage and transient strains.
\section{\krap{} Effectiveness}
Generally, when using \kNN{}, it is preferred to use single digit $k$ values. Through our investigation of these various \kNN{} classification algorithms, we find that that general advice holds true. For our dataset, using $k\geq5$ does not produce significantly different results. Choosing $k<5$ is a dangerous notion, since it is likely that an outlier may make its way into the \knnlong{} list, confounding the results. Staying with $5\leq k\leq9$ appears to be a safe and reasonable option, providing a good balance between accuracy and filtering of isolates.

%\subsection{Choosing $\alpha$}
Outside of this study, we choose to differentiate between strains of \ecoli{} using $\alpha=0.99$. It appears that using this value is advantageous. There were, however, some exceptions to those results, motivating us to consider non-thresholded \knnlong{} lists when classifying an unknown isolate.

%\subsection{Choosing a Resolution Algorithm}
The four resolution algorithms --- meanwise, winner, union, and intersection --- each have their own quirks and behaviors as we alter $k$ and $\alpha$.

Meanwise, which currently uses the Euclidean norm to resolve different metrics, did not respond to the $\alpha$ threshold and completely stopped classifying anything for $\alpha$ near 1. This is very likely due to Euclidean norm mapping $([0,1],\dots,[0,1])\rightarrow [0,\sqrt{1+\dots+1}]$. To get around this, we multiplied the resulting norm by a factor of $\sqrt2$, which may have unexpected results. We may investigate this further, or choose a more natural norming method, like arithmetic or geometric mean. With no $\alpha$ filtering, it performed third best with an overall 73.2\% classification accuracy.

Winner performs worst, classifying accurately between 65\% and 68\% of the time. Some alterations to this algorithm may make it more reliable, such as only counting the species that appear in all lists.

Unionwise performs very well. Without filtering the \knnlong{} lists by $\alpha$, we find that the unionwise method classifies best, with an overall accuracy of 76.4\%. However, once we add in $\alpha$ filtering, the unionwise does not improve, staying relatively close to 76\%.

Intersection performs best when we use $\alpha$. This is likely due to the ``list'' actually being a set of common isolates. Overall, the accuracy was 74.7\%, 78\%, and 85.9\% for $\alpha = $0.00 (no filtering), 0.98, and 0.99 respectively.

Overall, we find that the intersection algorithm performs the best and recommend moving forward with it. While unionwise did perform well, it did not respond well to thresholding and still did not perform as well as the intersection algorithm overall. Meanwise and winner may be more useful with previously mentioned modifications and we may investigate these in the future.

%The high performance of the intersection method is likely due to the fundamental structure and strain differentiation methods we use in CPLOP. Give the pyroprint of two \ecoli{} cultures, a Pearson Correlation within 0.99 defines a strain. Restricting $\alpha$ to values near this limit gave us better overall accuracy, since we get are allowed to search a wide distance around the \isol{} to find \isol{}s that mathch both regions, and we filter out \isol{} that are a different strain. While we find that resolution by intersection is a very good method, we will likely still use variations of the other methods to aid in classification.

%\subsection{Species Representation}
Poorly distributed representation of species and environmental incomparabilities are issues endemic to library-based MST. CPLOP has an overabundance of Cow and Human \isol{}s, and an underrepresentation of many of the species in the database. This dilutes the \knnlong{} list considerably for species like the Chicken and Cat.

Library population issues aside, environmental limitations are another concern for accuracy. Nearly every sample in the library comes from a 30 mile radius around Cal Poly, making the collected \pyro{}s potentially incomparable to \pyro{}s collected from a different region.

%Interestingly, environment factors in in another way, which our data shows about Bats . Classifying Bats is highly accurate and may be due to their limited involvement in other species' environment and small size. Cows have a wide variety of \ecoli{} strains, which allows them to show up in many other species' \knnlong{} lists. Bats on the other hand only have 37 \isol{}s in CPLOP all from a single \host{}, yet manage to be above 95\%, regardless of the method. We hypothesize that this is due to their limited interaction and we may investigate further.

\section{Future Work}
Future work should incorporate \krap{} into \cplop{}, study the \dbscan{} clustering method using the overall clustering entropy, and investigate whether combining the strain-based and \isol{}-based approach improves \mst{} and is more efficient on the \cplop{} database.
Incorporating \krap{} into \cplop{} will provide researchers the ability to make transparent, repeatable assertions as to the \spec{} of an unknown \isol{}.
Investigating clustering entropy can give us more insight into the makeup of clusters and extending this concept to \krap{} classifications, to measure how ``close the competition is'' between \spec{} in the \knnlong{} lists.
Merging strains into the \krap{} methods will reduce the number of comparisons, since querying against the database will involve querying against clusters of \isols{}, as opposed to \isols{} themselves.
Whether this benefits the accuracy of \mst{} with \cplop{} needs to be investigated.

\subsection{\cplop{} Incorporation of \krap{}}
Future work needs to incorporate \krap{} into \cplop{} directly, so \cp{} researchers have direct access to the \mst{} methodologies.
\cplop{} researchers showed interest in viewing the \knnlong{} lists when using \krap{} on an \isol{}.
Building an interactive \mst{} workflow can give researchers a better insight into \spec{} determinations and help them avoid having to painstakingly perform \mst{} by hand.
Future work needs to at least make \krap{} available to researchers in \cplop{} and should consider building an visually interactive way of using it.

\subsection{Entropy}
Entropy \index{clustering entropy} is another validity measure that for clusterings, represents the ``degree to which each cluster consists of objects of a single class'' \cite{tan2006introduction} and for \krap{} can tell us how contentious the \spec{} determination was.
Similar to the cluster and clustering purity, cluster and clustering entropy can give us an idea of the nature of \bslongs{} created by a clustering method.
Ideally, a good clustering method minimizes the clustering entropy.

Consider a cluster $C$, consisting of datapoints with class labels from $\mathcal{L}$.
Given a class label $L\in\mathcal{L}$, the value $P_C(L)$ calculates the proportion of datapoints in $C$ that have class label $L$.
The individual \textit{cluster entropy} is:
\index{cluster entropy}
\begin{equation}
e(C) = \sum_{L\in\mathcal{L}}^{} P_C(L)\log_2{P_C(L)}
\end{equation}

In addition to computing the entropy of individual clusters we want to have an understanding of the overall entropy on the entire dataset for a given clustering.
As before, given a \textit{clustering} $\mathcal{C} = \{C_1,\dots,C_n\}$ on a dataset, we define the size $\mathcal{M}$ of the set of clusters:
\index{clustering}
\begin{equation}
\mathcal{M} = \sum_{i = 1}^{n} |C_i|
\end{equation}
The overall \textit{clustering entropy} is:
\index{clustering entropy}
\begin{equation}\label{eq:clustering:entropy}
\sum_{i=1}^{n} \frac{|C_i|}{\mathcal{M}}\cdot\nu(C_i)
\end{equation}
One can think of \eqref{eq:clustering:entropy} as a form of weighted arithmetic mean of the individual entropies.
Larger clusters, as a result, affect this overall value more.
Future work should use this metric to compare clusterings with different parameters or methodologies, seeking to minimize the entropy, and extend the analysis to \krap{} classifications.

\subsection{\kNClong{}}
Combining the strain-based and \isol{}-based method of classification is a natural next step.
Using \krap{} as a simple fallback method when the strain-based method fails to classify an unknown \spec{} \isol{} is one approach.
Incorporating strains directly into \krap{} is yet another, wherein the neighbors in the \knnlong{} list may be either \isols{} or strains --- which we can represent as clusters.
As a result, we would instead have a \kNClong{} classification algorithm, through which, we can still apply the resolution strategies from \krap{}.
Future work should consider different cluster weighting methods for the \spec{} in clusters that appear in the \knnlong{} list and determine whether any of them are useful for \mst{}.

\subsection{Efficiency Study}
The efficiency of the classification methodologies in this work and any combinations thereof needs to be investigated as well as their performance on the hardware that supports the \cplop{} database.
Offline determination of strains, be it through clustering or otherwise, can occur offline, speeding up the determination of strain membership during \mst{}.
Combining the two approaches in this work into a \kNClong{} methodology may also speed up \mst{}.
\kNN{}, and \krap{} as a result, compares the unknown \isol{} to every datapoint in the database, but if instead the datapoints might be clusters, fewer comparisons may need to be made.