\documentclass{article}

\usepackage[final]{nips_2016} % produce camera-ready copy
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{natbib}
\usepackage{amsmath}   % vj: best to load this for math
% use \mathrm{...} for text in math mode.

\usepackage{amsthm} % vj: provides \begin{proof}...\end{proof}!

\usepackage{listings}
\usepackage{color}
\newcommand{\Beta}{B}
\newcommand{\alt}{\,|\,}
\newcommand{\defeq}{\stackrel{{\scriptscriptstyle def}}{=}}

\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]

\newtheorem{theorem}{Theorem}[section]
\newtheorem{convention}{Convention}[section]
\newtheorem{corollary}{Corollary}[theorem]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{digression}[theorem]{Digression}

\newtheorem*{remark}{Remark}
\newtheorem*{example}{Example}
\newtheorem*{CHECK}{{\sc Check}}
\newtheorem*{todo}{{\sc Todo}}
\newcommand{\mymathhyphen}{{\hbox{-}}}

\title{Working Notes on Probability and Statistics}
\author{
  Vijay Saraswat \\
  IBM TJ Watson Research Center \\
  Yorktown Heights, NY 10598\\
  \texttt{vijay@saraswat.org}
}
\begin{document}

\lstset{ %
  language=Prolog,
  basicstyle=\footnotesize,
%  numbers=left,
  numberstyle=\footnotesize,
  stepnumber=1,
  numbersep=5pt,
  backgroundcolor=\color{white},
  showspaces=false,
  showstringspaces=false,
  showtabs=false,
  frame=single,
  tabsize=2,
  captionpos=b,
  breakatwhitespace=false,
  escapeinside={\%*}{*)}
}
  % framexleftmargin=15pt}

\maketitle
\begin{abstract}
  Notes for myself around crucial elements of probability and statistics necessary for logic / knowledge representation researchers to understand.
\end{abstract}

\section{PCC notes}

There is no original material in this note. Most of the content is drawn from Wikipedia articles, \cite{lawler} and \cite{pgm-book}. We focus on gathering in one place the material on mathematical foundations of the modern approach to probabilities. Note the complexity of the theory -- the need to delve into measurable spaces and metric spaces --  arises from the desire to work with continuous random variables; the theory of discrete random variables is straightforward.

\subsection{Probability}

We start with the axiomatic approach.

\subsection{\(\sigma\)-algebras}
\begin{definition}[\(\sigma\)-algebra] Given a set \(X\), a \(\sigma\)-algebra on \(X\) is a collection \(\Sigma\) of subsets of \(X\) that includes the empty set and is closed under complements (wrt \(X\)), and countable unions. \((X,\Sigma)\) is called a {\em measurable space}.
\end{definition}
The conditions imply that \(\Sigma\) is closed under countable intersection, using De Morgan's laws. \(\sigma\)-algebras specialize {\em set-algebras} in demanding closure under countable, rather than finite, unions.

There are several motivations for \(\sigma\)-algebras. One would like a notion of ``size'' for sets. Clearly, the size of the empty set should be zero and the size of the countable union of disjoint sets should be the sum of their sizes. A key problem is apparently that the Axion of Choice causes problems with an attempt to specify a size for {\em every} set of reals (under the usual notion of length for subsets of the real line), see Vitali sets.

\begin{CHECK}
  Delve into this. Similar to the need to reduce the function space when looking to find spaces \(D\) s.t. \(D \sim (D \rightarrow D)\)? Check {\em non-measurable sets}.

  A Vitali set is a subset of \([0,1]\) such that for each real number \(r\) there is exactly one number \(v\in V\) such that \(v-r\) is a rational number. Their existence needs the Axiom of Choice.
\end{CHECK}

So we look to identify a smaller collection of sets for which the measure should be definable. Clearly \(\emptyset\) should be in it, and it should be closed under countable unions of disjoint sets. The only other requirement is that the collection of sets should be closed under complementation (with the universal set, \(X\)).

\begin{CHECK}
Why is this natural? Because you want to consider the entire set \(X\) measurable? Then if a subset \(Y\) is measurable, its complement should be measurable too, which will give the nice property that the size of the complement is the size of \(X\) less the size of \(Y\).
\end{CHECK}

A second motivation is the requirement that we should be able to take limits of sequences of sets, and this forces us to consider closure under countable unions and intersections. For instance for a sequence \(A_1, A_2, \ldots, \) of subsets of \(X\) we would like to define

\[ \lim_{n \to \infty} \sup{} A_n \defeq \bigcap_{n=1}^{\infty} \bigcup_{m=n}^{\infty} A_m
\]

\noindent and correspondingly  limit infimums.

Third, in probability theory, we are often concerned with taking conditional probabilities. These require working with subsets of the \(\sigma\)-algebra that are themselves \(\sigma\)-algebras. Having countable unions provides additional expressive power as this example shows. Consider the sample space of infinite number of coin tosses, \(\Omega=\{(x_1,x_2,\ldots)\alt x_i \in \{0,1\}, i \geq 1\}\). Now we want to consider situations in which \(n\) tosses have already been made. This can be characterized by the \(\sigma\)-algebra \({\cal F}_n = \{A \times \Omega \alt A \subseteq \{0,1\}^n\}\). Clearly \({\cal F}_1 \subset {\cal F}_2 \subset \ldots\), and we would like their limit to be a \(\sigma\)-algebra as well.

A crucial tool for reasoning about \(\sigma\)-algebras is the Dynkin \(\pi\mymathhyphen{}\lambda\) theorem. Given a set \(X\), a {\em \(\pi\)-system} is a collection \(P\) of subsets of \(X\) closed under finite intersections. A {\em Dynkin system} (\(\lambda\)-system) \(D\) is a collection of subsets of \(X\) containing \(X\) and closed under complements and countable unions of disjoint subsets.

The theorem says that if \(D\) contains \(P\) then it contains \(\sigma(P)\), the \(\sigma\)-algebra generated by \(P\).

\begin{example}
  Given a set \(X\), \(\{\emptyset, X\}\) is a \(\sigma\)-algebra. So is \(2^X\), the set of all subsets of \(X\).
\end{example}

\begin{definition}[Lebesgue measure]
  Let the length \(l(I)\) of an interval \(I\) of the reals with lower-bound \(a\) and upper-bound \(b\) be \(b-a\). The interval can be closed or open on either end. The {\em Lebesgue outer measure}, \(\lambda^{\star}(E)\) is defined for \(E\subseteq \Re\) by:
  \[ \inf{} \{\Sigma_{k=1}^{\infty} l(I_k) : E \subseteq \cup_{k=1}^{\infty} I_k, (I_k)_{k \in \mathbb{N}}\ \mbox{sequence of open intervals}\}\]
\end{definition}

Suppose \(S\) is a countable subset of the reals, e.g.{} the set of rational numbers. Then \(\lambda^{\star}(S)=0\), since the sequence of open intervals \(I_1=(x_1,x_1), I_2=(x_2,x_2), \ldots\) (for \(x_1,x_2,\ldots\) an enumeration of \(S\)) covers \(S\), but \(l(I_j)=0\), for all \(j\).

\begin{definition}[Lebesgue \(\sigma\)-algebra]
  The {\em Lebesgue \(\sigma\)-algebra} is the collection of all sets \(E \subseteq \Re\) satisfying the condition that for every \(A\subseteq \Re\):
  \[ \lambda^{\star}(A) = \lambda^{\star}(A\cap E) + \lambda^{\star}(A \cap E^c)\]

  On every such \(E\) the {\em Lebesgue measure} \(\lambda(E)\) is just \(\lambda^{\star}(E)\).
\end{definition}
\begin{CHECK}
  Work out that this is closed under countable unions. This probably involves Carath\'{e}odory's theorem and needs the Axiom of Choice.
\end{CHECK}

\begin{proposition}
Vitali sets are not measurable by the Lebesgue measure.
\end{proposition}

The Cantor set (see below) is an example of an uncountable set of reals which has a measure, and its measure is \(0\).

\subsection{Measure spaces}
\begin{definition}[Measure Space] For a set \(X\), a measurable space \((X,\Sigma)\) over \(X\), let \(\mu\) be a function from \(\Sigma\) to the extended reals satisfying:
  \begin{enumerate}
  \item \(\mu(E) \geq 0\) for all \(E \in \Sigma\).
  \item \(\mu(\emptyset)=0\).
  \item \(\mu\) is \(\sigma\)-additive: for every countable collection \(E_1, E_2, \ldots\) of disjoint sets in \(\Sigma\), \(\mu(\bigcup_i E_i)=\Sigma_i \mu(E_i)\)
  \end{enumerate}
Then \((X,\Sigma,\mu)\) is a measure space.
\end{definition}

\begin{proposition}
  The reals form a measure space with the Lebesgue \(\sigma\)-algebra and the Lebesgue measure.
\end{proposition}

A measure \(\mu\) over a measurable space \((X,\Sigma)\)is {\em finite} if \(\mu(x)\) is finite for all \(E\in \Sigma\). It is {\em \(\sigma\)-finite} if \(X\) is the countable union of measurable sets with finite measure.

The Lebesgue measure on the reals is \(\sigma\)-finite, but not finite. (The reals can be covered by the measurable sets \([k,k+1)\), for integers \(k\), with measure \(1\).)

  A measure space is {\em complete} if every subset of a null set (measurable set with measure \(0\)) is measurable. (It will have measure \(0\).)
  \begin{digression}
    Why is completeness important? Because it shows up even in simple situations, such as the tensor product of measure spaces. Consider the Lebesgue measure space \((\Re, L, \lambda)\). To construct the two-dimensional space on \(\Re^2\), it might make sense to consider the \(\sigma\)-algebra to be \(L\otimes L\). But this is not complete. Evey singleton set has measure \(0\), hence \(\lambda^2(\{0\}\times A)=\lambda(\{0\}) \lambda(A)=0\), for any \(A \in L\), hence for \(A=\Re\). Now choose \(A\) to be the Vitali set, so that \(\lambda(A)\not\in L\), hence \(\{0\}\times A \not\in L\), and the constructed measure space is not complete.
  \end{digression}

The Lebesgue measure space over the reals is complete.

%vj: need to get this right. One can define many measure spaces over the reals. For instance, let \(\Sigma\) be the set of all subsets of the reals which are empty or contain their infimum and supremum, and let \(\mu(E)\) return the difference between the supremum and infimum if \(E\) is non-empty, and \(0\) otherwise.

\begin{definition}[Measurable function]
  A function between two measurable spaces is measurable if the preimage of every measurable set is measurable. That is, \(f:(X,\Sigma) \rightarrow (X',\Sigma')\) is measurable if \(f^{-1}(E)\defeq \{x\alt f(x)\in E\} \in \Sigma\) for every \(E\in\Sigma'\).
\end{definition}

The collection of measurable spaces forms a category with measurable functions as morphisms.

\begin{proposition}
The sum and product of two real-valued measurable functions is measurable. The pointwise supremum, infimum, limit superior and limit inferior of a sequence of real-valued measurable functions are measurable.
\end{proposition}

Any non-measurable set \(Y \subset X\) can be used to generate a non-measurable function \(f:(X,\Sigma)\rightarrow (\Re, {\cal B})\), for \({\cal B}\) the Borel algebra on \(\cal R\) (through its indicator function). (See below for the definition of Borel algebra.)

The Lebesgue integral plays an important role in probability theory, by serving to generalize the intuitive idea of integration as ``finding the area under the curve'' to much more complex classes of functions. Let \(f:\Re \rightarrow \Re^+\) be a non-negative real-valued function. The {\em Reimann integral} generalizes the idea of finding the area by fitting columns of small width under the function \(f\), but fails to be well-defined for certain functions (e.g. \(\lambda x.\, x^{-2}\) since it has unbounded range). The Lebesgue integral instead uses the idea of fitting thin rows (strips) under the curve, that is, partitioning the range of \(f\). The area is \(f^{\star}(t)=\mu(\{x \alt f(x) > t\}) {\mathrm dt}\).  The Lebesgue integral is then defined as \(\int_0^{\infty} f^{\star}(t){\mathrm dt}\) which is a well-defined improper Riemann integral since \(f^{\star}\) is a non-negative decreasing function.

\subsection{Metric spaces}

\begin{definition}[Metric Space] A {\em metric space} \((M,d)\) is a set \(M\) with a metric \(d:M \times M \rightarrow \Re\) satisfying (a)~\(d(x,y)=0\) iff \(x=y\), (b)~ \(d\) is symmetric, (c)~\(d\) satisfies the triangle inequality: \(d(x,z) \leq d(x,y)+d(y,z)\), for any \(x,y,z\in M\).

  The metric space is {\em complete} if every Cauchy sequence of points in \(M\) has a limit that is also in \(M\). (A Cauchy sequence in \(M\) is a sequence of elements of \(M\) which is such that for any given real number \(\epsilon > 0\) there are only finitely many elements \(x,y\) in the sequence such that \(d(x,y) \geq \epsilon\).)
\end{definition}

\((\Re, \lambda x,y.\, |x-y|)\) is a complete metric space. So is Euclidean \(n\)-space with Euclidean distance as metric.

Every metric space \(M\) is a topological space, with base given by {\em open balls} of radius \(r > 0\) (for \(r\) a real number), i.e. the set \(\{m \in M \alt d(m,x) < r\}\) for every \(x\in M\).

A {\em neighborhood} of a point \(p\) in a topological space is any superset of an open set containing \(p\).

In a topological space \(S\), a {\em cover} for \(X \subseteq S\) is a collection \(U\) of subsets of \(S\) whose union is a superset of \(X\). A sub-cover of \(U\) is a subset of \(U\) that still covers \(X\). The cover is {\em open} if it consists of open sets.  \(S\) is said to be {\em compact} if each of its covers has a finite sub-cover. It is said to be {\em locally compact} if every point has a compact neighborhood.

A topological space is said to be {\em Hausdorff} if distinct points have disjoint neighborhoods. This condition implies uniqueness of limits of sequences.

The reals, with the topology given above, are locally compact and Hausdorff.

In a topological space \(X\), a {\em Borel set} is any set that can be formed from open sets through countable unions, intersections or relative complementation. The set of all Borel sets forms a \(\sigma\)-algebra, the {\em Borel algebra}, \({\cal B}\). This is the smallest algebra containing all open sets. Borel sets are important in measure theory since any measure defined on open sets of a space must be defined on Borel sets of that space.

A {\em Borel measure} on a topological space is any measure defined on the Borel algebra of the space. Over the reals, the measure \(\mu\) which assigns to every half-open interval \((a,b]\) the measure \(b-a\) is called {\em the} Borel measure.

\begin{proposition}
The Borel measure space on the reals is not complete.
\end{proposition}

\begin{proof}
To see this, consider the Cantor set (e.g.{} obtained by deleting the middle third of intervals, recursively, countably many times, starting with the unit interval \([0,1]\)). The Cantor set is a Borel set. It has measure zero.\footnote{To see this, consider the sequence of sequences of open intervals defined as follows. The first sequence is \((0,1/3),(2/3,1)\). This covers the Cantor set, and its measure is \(2/3\). Now consider the sequence at the second step of the iteration \((0,1/9),(2/9,1/9), (6/9,7/9),(8/9,9/9)\). It has measure \(4/9\). The \(n\)th such sequence has measure \((2/3)^n\). The inf of this set is \(0\).} It is uncountable. Its power set, therefore, has cardinality strictly greater than the reals. Therefore it has some subset which is not a Borel set.
\end{proof}

A (possibly incomplete) measure space \((X,\Sigma,\mu)\) can be converted uniformly into a complete measure space \((X,\Sigma_0,\mu_0)\); the smallest such \(\sigma\)-space is said to be its {\em completion}.

\begin{proposition}
  The completion of the Borel measure space is the Lebesgue measure space.
\end{proposition}

Let \((X,T)\) be a Hausdorff space, and \(\Sigma\) be a \(\sigma\)-algebra on \(X\) that contains the topology \(T\), so every open set is measurable. A measure \(\mu\) on \(\Sigma\) is {\em locally finite} iff for every point \(p\) of the space \(X\), there is an open neighborhood \(N_p\) of \(p\) s.t. \(\mu(N_p)\) is finite. It is said to be {\em inner regular} if every \(E\in \Sigma\) can be approximated from within by compact sets. That is:
\[ \mu(E) = {\mathrm sup}\{\mu(K)\alt K \subseteq E, K {\mathrm compact}\}\]

A measure that is locally finite and inner regular is said to be a {\em Radon measure}.

\begin{proposition}
  The Lebesgue measure is a Radon measure.
\end{proposition}

\subsection{Probability spaces}

\begin{definition}[Probability Space]
  A {\em probability space} is a measure space \((\Omega, S, P)\) with total measure \(1\) (i.e.{} \(p(\Omega)=1\)). It is said to be {\em discrete} if \(\Omega\) is finite or countably infinite, otherwise it is {\em continuous}.
\end{definition}
\(\Omega\) is thought of as a space of {\em outcomes}, and \(S\) as a set of (measurable) {\em events}, representing the disjunction of the contained outcomes. \(P\) is said to be the {\em probability measure} of the space, and we say ``probability of \(E\)'' for \(P(E)\).

Any probability space is locally finite because the whole space is assigned a finite measure (\(1\)). Often we require probability spaces to be complete in the sense discussed above.

\begin{convention}\label{convention:discrete-ps}
For a discrete probability space, \(S\) is usually chosen as the powerset \(2^{\Omega}\), and the probability measure \(p\) can be uniquely defined by extending a function \(p:\Omega\rightarrow [0,1]\) satisfying \(1=\Sigma_{\omega \in \Omega} f(\omega)\) to \(S\), via \(p(E)=\Sigma_{\omega \in E} p(\omega)\).  Hence we will present a discrete probability space by specifying just the pair \((\Omega, p:\Omega \rightarrow [0,1])\).
\end{convention}

\begin{example}
  The probability space associated with an infinite sequence of coin tosses is: (\(0\) is ``heads'', and \(1\) ``tails'')
  \[ \Omega = \{(\omega_1, \omega_2, \ldots) \alt \omega_i \in \{0,1\}\}\]

  For each positive integer \(n\), let \(\Omega_n\) be the finite set with \(2^n\) elements given by:
  \[\Omega_n=\{(\omega_1, \ldots, \omega_n)\alt \omega_i \in \{0,1\}\}\]

  Clearly, \((\Omega_n, 2^{\Omega_n},p_n=\lambda x:\Omega_n.2^{-n})\) is a probability space (where we utilize the convention above and define the probability measure only on \(\Omega_n\)).

  Going back to an earlier example, let consider situations in which \(n\) tosses have already been made. Define the  \(\sigma\)-algebra \({\cal F}_n= \{A\times \Omega \alt A \subseteq \{0,1\}^n\}\).  Clearly, \({\cal F}_n\) is a finite \(\sigma\)-algebra with cardinality \(2^{2^n}\) and  \({\cal F}_j \subset {\cal F}_{j+1}\), for all \(j\). Define \({\cal F}^0=\bigcup_j {\cal F}_j\) and \(p(A\times \Omega)= p_n(A)\).

  Interestingly, \({\cal F}^0\) is an algebra (closed under finite unions) but not a \(\sigma\)-algebra. (The set \(\{(1,1,\ldots)\}\) is not in \({\cal F}^0\) but can be witten as a countable intersection of the sets \({\bf 1}_j \times \Omega\in {\cal F}^0\), where \({\bf 1}_j = \{(\omega_1,\ldots, \omega_j)\alt \omega_1=\omega_2\ldots=\omega_j=1\}\).) The Carath\'{e}odory Extension theorem can be applied here to produce a complete measure space containing  ${\cal F}^0$, as laid out in \cite{lawler}.
\end{example}

The {\em frequentist} interpretation relies on the notion of an {\em experiment} with a defined outcome, that can be repeated independently and as aften as required. Under these conditions one can define th probability of an outcome as the ratio of the number of times the desired outcome occurs, with the total number \(N\) of trials, as \(N\) tends to infinity.

An alternate intepretation of probability is {\em degree of belief}, e.g.{} about whether it is going to rain tomorrow.  A way to make this precise is to relate the degree of belief to bets that a person would be willing to make about the outcome. Assuming that the person is rational, hence willing to place only such bets as are not going to yield negative results for the person, it can be shown that the person's beliefs must satisfy the conditions for probability distributions laid out above.

\begin{CHECK}
  Check this.
\end{CHECK}

Next, consider the problem of revising one's belief about the probability of an event \(\beta\), given the information that an event \(\alpha\) has occurred. One can model this situation by focusing only on those states of the world in which \(\alpha\) occurs, and, assuming it is non-empty, examining the proportion of those such states in which \(\beta\) occurs. Thus the conditional probability, \(P(\beta\alt \alpha)\) can be defined by:
\[ P(\beta \alt \alpha) \defeq \frac{P(\beta \cap \alpha)}{P(\alpha)}
\]
\noindent (provided that \(P(\alpha)\not= 0\)).

One can look at this definition as asserting the {\em chain rule}:
\[
P(\alpha \cap \beta)=P(\beta \alt \alpha) P(\alpha)
\]

And it also immediately yields Bayes' Rule:

\[ P(\alpha \alt \beta) = \frac{P(\beta \alt \alpha) P(\alpha)}{P(\beta)}
\]

\begin{definition}[Random Variable]
  Given a probability space \((\Omega, S, p)\), a random variable is a measurable map \(X:(\Omega,S) \rightarrow (\Omega', S')\), for \((\Omega', S')\) a measurable space. It induces a probability measure on \(S'\) by \(p_X(E)=p(X^{-1}(E))\), yielding the probability space \((\Omega',S', p_X)\). \(p_X\) is said to be the {\em probability distribution} of the random variable.
\end{definition}
Note that the probability distribution of the random variable no longer directly references the source probability space \((\Omega, S, p)\).

Observing that the identity map \(\lambda x : \Omega.\,x\) in \((\Omega,S)\rightarrow (\Omega,S)\) is a
measurable map, we get:

\begin{proposition}
  For any probability space \((\Omega, S, p)\), \(p\) is the probability distribution of a random variable.
\end{proposition}

Often the target measurable space is taken to be the space over the reals with the Borel algebra, \((\Re,{\cal B})\).

If \(p_X\) maps a countable set \(E \in S'\) to \(1\) then \(X\) is said to be a {\em discrete random variable}. If \(p_X\) maps every singleton set (and, hence, every countable set) in \(S'\) to \(0\) then \(X\) is said to be a {\em continuous random variable}.

Clearly, all random variables defined on a discrete probability space are discrete.

\begin{proposition}
Every random variable can be represented as the sum of a discrete random variable and a continuous random variable .
\end{proposition}

When the range of the random variable is the reals, the probability distribution is sometimes given in terms of the {\em cumulative distribution function}:
   \[ F_X(x) = p_X((-\infty, x])\]

This function \(F_X\) is non-decreasing, and right continuous. \(F_X(x)\) takes on the value \(0\) in the limit as \(x\) tends to \(-\infty\), and \(1\) as \(x\) tends to \(\infty\).

\begin{proposition}
  Any function satisfying the four properties above can be the distribution function of a random variable.
\end{proposition}

The probability distribution for the random variable, \(p_X\), can be recovered from the cumulative distribution function \(F_X\) by setting \(p_X(y)\) to \(F_X(y)\) for \(y\) the  open sets \((-\infty,x]\), and then extending uniquely to the Borel sets.

\subsubsection{Probability density functions}

For some continuous random variables \(X\) over the reals, there exists a function \(f_X:\Re \rightarrow [0,\infty)\) s.t.
    \[p_X([a,b]) = \int_a^b f_X(x) \mathit{dx}\]

    Such a function (the {\em probability density function}, pdf) is the {\em Radon-Nikodym derivative} wrt the Lebesge measure. If it exists, then
    \[F_X(x) = \int_{\infty}^x f_X(t) \mathit{dt}\]

    If \(f_X\) is continuous at \(t\) then \(f_X(t)=F_X'(t)\). It also satisfies \(1=\int_{\infty}^{\infty} f(x)\mathrm{dx}\). Conversely, any nonnegative function integrating to \(1\) is the density of a random variable.

%Note that every probability measure \(\mu\) on \((\Re,{\cal B})\) is the probability distribution of the random variable \(X\) defined on the probability space \((\Re,{\cal B},\mu)\) by \(X=x\). That is, \(\mu_X=\mu\).

    %% NEed to talk about Radon-Nikodym derivative.

\subsubsection{Example probability distributions}
Below we shall follow Convention~\ref{convention:discrete-ps} for describing discrete probability distributions. By \({\bf k}\) we shall mean the set \(\{0,1,\ldots, k-1\}\).

The {\em categorical distribution} has \(k\) outcomes. It takes as parameter a unit \(k\)-vector \(p\), and is defined by \(({\bf k}, \lambda x:{\bf k}.p_x)\), where we use the notation \(p_x\) to select the \(x\)'th component of the vector \(p\). We can think of it as corresponding to the probability of a specific outcome when throwing a die with \(k\) faces. The categorical distribution over {\bf 2} is called the {\em Bernoulli} distribution.

The {\em multinomial distribution} \(f\) is obtained by throwing the same \(k\)-sided die \(n\) times. Recall that the number of ways of choosing \(x=(x_0, \ldots, x_{k-1})\) values (totalling \(n\)) out of \(n\) is \(\binom{n}{x_0,\ldots,x_{k-1}}=\frac{n!}{\Pi_i x_i!}\). The probability of one such selection is \(\Pi_i p_i^{x_i}\). Thus the distribution is
\[ (\Omega=\{x \alt x_i \geq 0, \Sigma_i x_i = n\},
   \lambda x:\Omega.\, \frac{n!}{\Pi_i x_i!}\Pi_i p_i^{x_i})\]

Recall that the gamma function is defined for real argument values by \(\Gamma(x)=(x-1)!\). Then the probability measure can also be written as:
\[ f(x; n, p) = \frac{\Gamma(1+\Sigma_i x_i)}{\Pi_i \Gamma(1+x_i)}\Pi_i p_i^{x_i}
\]

The {\em beta distribution} takes two parameters \(\alpha\), \(\beta\), and is defined over the closed unit interval \([0,1]\). It is of use in situations where the random variable can only take values in a bounded interval, e.g.{} the variable represents the proportion of some physical quantity in a sample. It has pdf:
\[ f_X(x; \alpha, \beta) = \frac{1}{k} x^{\alpha-1}(1-x)^{\beta-1}\]

The constant \(k\) is determined by the requirements of pdfs that the integral over the interval of definition should total 1:
\[ \int_0^1 u^{\alpha-1}(1-u)^{\beta -1} du  = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)} = \Beta(\alpha,\beta)\]
\noindent where \(\Beta\) is the beta-function.
It is the conjugate prior to the Bernoulli, and binomial distributions.

The {\em Dirichlet distribution} is the multivariate version of the beta distribution, parametrized by a vector \(\alpha\) of positive real numbers, with pdf:
\[f(x_1, \ldots, x_k; \alpha_1, \ldots, \alpha_k)
  = \frac{\Gamma(\Sigma_{i=1}^k \alpha_i)}{\Pi_{i=1}^k \Gamma(\alpha_i)} \Pi_{i=1}^k x_i^{\alpha_i -1}\]

for \(x_i\) non-negative and summing to \(1\), and \(0\) elsewhere. Thus the {\em support} of the Dirichlet\footnote{The points in the domain mapped to non-zero values} is the set of discrete probability distributions over \(k\) outcomes (aka the {\em \(k-1\) simplex}).

For the {\em symmetric} Dirichlet, all the \(\alpha_i\) have the same value; the pdf can be written as:
\[f(x_1, \ldots, x_k; \alpha) = \frac{\Gamma(k\alpha)}{\Gamma(\alpha)^k} \Pi_{i=1}^k x_i^{\alpha -1}\]

\paragraph{Conjugate distributions}

The Dirichlet distribution is conjugate to the multinomial.

\subsection{Expectations and moments}

\section{Probabilistic Programming}
\paragraph{Notes on LDA}

We can describe the generative process for Hierarchical Dirichlet Allocation with the following probabilistic program. Given is the dictionary of words, a colletion of Docs, the number of topics (\(KK\)), and parameter vectors \(Alpha\), \(Beta\), and \(Chi\).

\begin{lstlisting}
/* Generative LDA model for 1 to M documents.*/
lda(Words, Docs, K, Alpha, Beta, Chi) {
  M = size(Docs),
  V = size(Words),
  for _K in 1..K
     TopicWords(_K) | Alpha ~ Dirichlet(Alpha),

  for D in 1..M {
     DocTopics(D) | Beta ~ Dirichlet(Beta),
     N(D) | Chi ~ Poisson(Chi),
     for I in 1..N(D) {
       Z(I) | Beta ~ DocTopics(D),
       Doc(I) | Alpha, Beta, Chi ~ TopicWords(Z(I))
     }
  }
}

\end{lstlisting}

\bibliographystyle{abbrvnat}
\setcitestyle{authoryear,open={((},close={))}}
\bibliography{bib}

\appendix

\section{Some definitions}

Discrete exponential family distributions are of the form
\[f_X(x \alt \theta) = h(x) exp(\eta(\theta) \circ T(x) - A(\theta))\]

\noindent for given functions \(T(\_), h(\_), \eta(_), A(\_)\). Here \(\theta\) is the parameter of the family.

The {\em likelihood} of a parameter value \(\theta\), given outcomes \(x\) for r.v. \(X\) is the probability of those outcomes given the parameter values, viewed as a function of \(\theta\):
\({\cal L}(\theta \alt x) \defeq p_{\theta}(X=x)\).

{\em Independent and identicall distributed} (iid) sequence of random variables.

A weaker assumption is {\em exchangeability}. A finite or infinite sequence \(X_1, X_2, \ldots\) of random variables is said to be exchangeable if its joint probability distribution is invariant under finite permutations.

Frequentist and Bayesian interpretation

de Finetti's theorem.

\(\chi^2\) test

Radon-Nikodym theorem

%(Wikipedia entry)
sufficient statistic

correlation between random variables

Pearson product-moment correlation coefficient

\end{document}

Bayesian, Random Forest, CART modeling, Contextual Bandit etc


%Dokumenttyp + Encoding
\documentclass[final, paper=a4, paper=portrait, pagesize=auto, fontsize=10pt,english]{scrartcl}
\usepackage[utf8]{inputenc}

\usepackage[a4paper, total={6.5in, 9in}]{geometry}
\usepackage{graphicx}
\usepackage[]{hyperref}
\usepackage{amsmath}

\usepackage{lmodern} %Schriftarten
\usepackage[english]{babel}
\usepackage{color}
\usepackage{placeins}
\definecolor{mygreen}{rgb}{0.33,0.63,0.33}
\definecolor{mygray}{rgb}{0.95,0.95,0.95}
\definecolor{myblue}{rgb}{0.23,0.23,0.52}
\renewcommand{\arraystretch}{1.5}

\usepackage{listings} %for code
\lstset{
	 backgroundcolor=\color{mygray},
	 commentstyle=\color{mygreen},
	 numbers=left,
	 keywordstyle=\textbf,
	 language=C++,
	 tabsize=3,
	 basicstyle=\footnotesize,
}

\begin{document}

\begin{titlepage}
	\centering
	\includegraphics[width=0.15\textwidth]{brockLogo}\par\vspace{1cm}
	{\scshape\LARGE Brock University\par}
	\vspace{0.5cm}
	{\Large Department of Computer Science\par}
	\vspace{1cm}
	{\Large Report on the final project of COSC 3P98\par}
	\vspace{1.5cm}
	{\huge\bfseries Mandelbrot and Julia Sets\par}
	\vspace{2cm}
	{\Large\itshape Torben Krüger, Maike Rees\par}
	\vspace{2cm}
	{\large January 2016\par}
\end{titlepage}

\newpage

\section{Introduction}
During the process of the course, fractals catched our interest so we desided to make an application about fractals for our final project of the course. The Mandelbrot Set is one of the most popular fractals and it is closely related to the Julia Set so we chose it as our main focus. There are a lot of android apps, online applications and pretty pictures about and of Mandelbrot Sets that inspired us further more.

We developed the Application in c/c++ with openGl, freeglut and freeimage. We created it on Windows 7 in Eclipse Mars and migrated it later into a visual studio(2013) project.

\section{Functionality}
The Application demonstrates the relation between the Mandelbrot set and Julia sets. Furthermore you get interesting observations through the coloring of the sets. Additionally, images can be saved.
Here is a screenshot of our program.\\\\
\includegraphics[width=1.0\textwidth]{screenie1}

\section{User Manual}
The Application comes with two windows. One where the Mandelbrot set is shown and in the other one you can see a Julia set.

You have multiple options here. With left-click of your mouse you can zoom into the chosen set. The point of your mouse click is going to be the center of the zoomed image.

With a right click a context menu opens. You can chose to zoom in and out, reset the set, change the color or save an image of the set.

If you open this menu in the Mandelbrot set you additionally have the option to show the Julia set of the chosen Point. It will then show the according Julia Set in the second Window.

\section{Architecture and Implementation}
The architecture of the application is not too complicated. It is object-oriented and structured into header and cpp - files.

\subsection{Complex Numbers}
Since we're calculating with complex numbers most of the time in those sets there is a class called Complex.h that describes those complex numbers as a struct. Every complex number comes with a real and imaginary part which are represented by two integers.

This struct also overloads methods for the + and * operator. It also contains a method that squares the number and one that calculates the absolute value of the complex number. Those methods are all needed to calculate the sets.

\begin{lstlisting}
struct complex {
	double r, i;
	double getAbsolute() {...}
	complex pow2() {...}
	inline complex operator+(complex a) {...}
	inline complex operator*(complex a) {...}
};
\end{lstlisting}

\subsection{Mandelbrot Set}
For a Mandelbrot set we set up a header and the cpp-file that implements the header methods.
The Mandelbrot set has public methods to calculate the set, zoom in and out of it, reset it as well as some Getters and Setters.

\begin{lstlisting}
class MandelbrotSet {
public:
	MandelbrotSet(unsigned int, unsigned int);
	virtual ~MandelbrotSet();
	void calculate();
	void setWidth(unsigned int);
	void setHeight(unsigned int);
	void setIterations(unsigned int);
	void reset();
	void zoom(unsigned int, unsigned int);
	void zoomOut(unsigned int, unsigned int);
	void setColorMode(int);
};
\end{lstlisting}

\subsection{Julia Set}
Looking at the formular of both sets, one can see that they are very close. That is why we decided to let the Julia set be a child of the Mandelbrot set. Since there are only a few changes that we had to make in the Julia classes, we save a lot of code and our implementation gets a lot easier with this design decision.

The Julia set only overrides the cnstructor and the calculate(), draw() and reset() methods. Additionally it has a complex number k, that is needed for the calculation.

\begin{lstlisting}
class JuliaSet: public MandelbrotSet {
public:
	JuliaSet(unsigned int, unsigned int);
	void draw(unsigned int, unsigned int, complex);
	void calculate();
	void reset();
	complex k;
};
\end{lstlisting}

\subsection{The Main Class}
The Main.cpp puts it all together. It sets up the OpenGl / freeglut environment, handles mouse event and the menus and starts the drawing of the sets. \\\\
When clicking on an option of the menu the program often needs to know where the mouse is right now to calculate with the right coordinates. For this purpose the application is using the passive motion function that glut offers. It always saves the current mouse coordinates in the global struct.

\section{Coloring}
The Mandelbrot set itself only tells us if the pixel is in the set or not. So basically only two colors are needed to display the Mandelbrot set. Like in this screenshot (chose colormode 'pure' in the application). \\\\
\includegraphics[width=0.7\textwidth]{screeniePure}

A lot of fun comes with different coloring techniques so we want to take a closer look to it.

The Mandelbrot calculates its pixel with a preset maximum number of iterations (because it can calculate infinitely, but thats obviously not helping us). The bigger the maximum number of iterations, the more precise is the image. But this also means that with a big number of max. iterations the calculating time is way longer that with a small number. So the idea behind the iterations is that every iterations looks a little bit closer into the Mandelbrot set. So at some point, the algorithm will determine that a pixel is not included in the mandelbrot set and will then break the for-loop. If the pixel is inlcuded in the Mandelbrot set the for-loop will terminate regularly after all iterations have been run through.\\ \\
To make this a little more clear, here is the code snippet:\\

\begin{lstlisting}
void MandelbrotSet::draw(unsigned int x, unsigned int y, complex c) {
	complex z = c;
	unsigned int n = 0;

	for (n = 0; n < iterations; n++) {

		if (z.getAbsolute() > 4) {
			break; // the pixel is not in the Madelbrot set
		}
		z = z.pow2() + c;
	}
	data[x + y * width] = getColor(n); // get the color based on the iteration
}
\end{lstlisting}

This means that every pixel has a number n that is the iteration at wich the for-loop terminated.
For the pixels that are in the Mandelbrot set
\begin{displaymath}
n = number \: of \: max \: iterations
\end{displaymath}
and for all the others
\begin{displaymath}
0 \leq n  \: \textless \: number \: of \: max \: iterations
\end{displaymath}

According to this number n the different coloring methods calculate the color.

\subsection{Linear Coloring}
E.g. the greyscale colormode uses a linear coloring method.
First of all the step between each color is calculated. That means if the number of iterations is big the steps are small and the other way round. Then the red value is calculated. The pixels within the Mandelbrot set get the color black.

\begin{lstlisting}
float step = 1.0f / iterations;
...
// greyscale
red = 2 * n * step * 255;
if (n > iterations - 2) {
	red = 0;
}
blue = red;
green = red;
\end{lstlisting}

In the red and white colormode, the application uses a gradient from black to red for the first half of the iterations and a gradient from red to white for the second half of the iterations.
\begin{lstlisting}
red = 2 * n * step * 255;
if (n < iterations / 2 - 1) { // black to red
	green = 0;
	blue = 0;
} else if (n < iterations - 2) { // red to white
	green = 255;
	blue = 255;
} else { // pixel in the set are black
	red = 0;
	green = 0;
	blue = 0;
}
\end{lstlisting}

\subsection{Logarithmic Coloring}

The Logarithmic coloring method provides more detailed coloring for the edges of the Mandelbrot set. The linear coloring uses one color gradient wheras the logarithmic coloring uses 7 different color gradients. Furthermore it seperates the iterations in unequal parts to apply the gradient.

\begin{lstlisting}
if (n == iterations) {
} else if (n < 64) {
	red = n * 2.0f;
} else if (n < 128) {
	red = ((((n - 64.0f) * 128.0f) / 126.0f) + 128.0f);
} else if (n < 265) {
	red = ((((n - 128.0f) * 62.0f) / 127.0f) + 193.0f);
} else if (n < 512) {
	red = 1.0f;
	green = ((((n - 256.0f) * 62.0f) / 255.0f) + 1.0f);
} else if (n < 1024) {
	red = 1.0f;
	green = ((((n - 512.0f) * 63.0f) / 511.0f) + 64.0f);
} else if (n < 2048) {
	red = 1.0f;
	green = ((((n - 1024.0f) * 63.0f) / 1023.0f) + 128.0f);
} else /*if (n < 4096)*/ {
	red = 255;
	green = ((((n - 2048.0f) * 63.0f) / 2047.0f) + 192.0f);
}
\end{lstlisting}

The result is that the edges are more nicely shaded in comparison to the linear gradient. As you see in the two images below, the logarithmic gradient has a smooth gradient from red to yellow and the linear gradient 'jumps' between red and white. \\\\

Difference between logarithmic (left) and linear (right) coloring. \\
\includegraphics[width=0.5\textwidth]{screenie2}
\includegraphics[width=0.5\textwidth]{screenie4}
\includegraphics[width=0.5\textwidth]{screenie7}
\includegraphics[width=0.5\textwidth]{screenie8}

\section{Mathematic Background}
\section{Discussion}
Including the coloring gives a nice extra observation. Let's discuss this with the help of an example. \\
\includegraphics[width=0.7\textwidth]{screenie5}

The Mandelbrot set has the property that the pixels that are closer to the boarder have higher iteration numbers. For this coloring method this means that white pixels are very close to the boarder of the set and red pixel are closer than black pixels but not as close as the white pixels to the boarder.

This means that the red pixels that look like roots of a tree tell us that there is a boarder some where close to them. So if we zoom in deep enough we will find a part of the Mandelbrot set there as well. \\

\includegraphics[width=0.4\textwidth]{screenie6}
\includegraphics[width=0.4\textwidth]{screenie7}

Looking at these zoomed pictures we also see that the small part of the Mandelbrot set has the same shape as the big one where we started. This is another interesting property of the Mandelbrot set.

\section{References}

\end{document}


\section{Expressiveness of \EUTXO{}}
\label{sec:expressiveness}

In this section, we introduce a class of state machines that can admit
a straightforward modelling of smart contracts running on an \EUTXO{}
ledger. The class we choose corresponds closely to Mealy
machines~\cite{mealy} (deterministic state transducers). The transition
function in a Mealy machine produces a value as well as a new
state. We use this value to model the emission of constraints which
apply to the current transaction in the ledger. We do not claim that
this class captures the full power of the ledger: instead we choose it
for its simplicity, which is sufficient to capture a wide variety of
use cases.

We demonstrate how one can represent a smart contracts using Mealy
machines and formalise a \textit{weak bisimulation} between the
machine model and the ledger model.  Furthermore, we have mechanised
our results in Agda\site{
https://github.com/\GitUser/formal-utxo/tree/\AgdaCommit/Bisimulation.agda
}, based on an executable
specification of the model described in
Section~\ref{sec:formal-model}.

\subsection{Constraint Emitting Machines}
We introduce Constraint Emitting Machines (\CEM{}) which are based on
Mealy machines. A \CEM{} consists of its type of states \s{S} and inputs
\s{I}, a predicate function $\s{final} : \s{S} \rightarrow \s{Bool}$
indicating which states are final and a valid set of transitions,
given as a function $\s{step} : \s{S} \rightarrow \s{I} \rightarrow
\s{Maybe}\ (\s{S} \times \s{TxConstraints})$\footnote{
The result may be \s{Nothing}, in case no valid transitions exist from a given state/input.
}
from source state and input symbol to target state and constraints and
denoted $\CStep{s}{i}{\txeq}$.

The class of state machines we are concerned with here diverge from
the typical textbook description of Mealy Machines in the following aspects:
\begin{itemize}
\item The set of states can be infinite.

\item There is no notion of \textbf{initial state}, since we would not
  be able to enforce it on the blockchain level. Therefore, each
  contract should first establish some initial trust to bootstrap the
  process.
  One possible avenue for overcoming this limitation is to built a notion of \textit{trace simulation}
  on top of the current relation between single states,
  thus guaranteeing that only valid sequences starting from initial states appear on the ledger.
  For instance, this could be used to establish inductive properties of a state machine and
  carry them over to the ledger; we plan to investigate such concerns in future work.

\item While \textbf{final states} traditionally indicate that the
  machine \textit{may} halt at a given point, allowing this
  possibility would cause potentially stale states to clutter the \UTXO{}
  set in the ledger. Thus, a \CEM{} final state indicates that the
  machine \textit{must} halt. It will have no continuing transitions
  from this point onward and the final state will not appear in the
  \UTXO{} set. This corresponds to the notion of a \emph{stopped}
  process~\cite{sangiorgi} which cannot make any transitions.

\item The set of output values is fixed to \emph{constraints} which
  impose a certain structure on the transaction that will implement
  the transition.  Our current formalisation considers a limited set
  of first-order constraints, but these can easily be extended without
  too many changes in the accompanying proofs.
\end{itemize}

\subsection{Transitions-as-transactions}

We want to compile a smart contract $\mathcal{C}$ defined as a \CEM{} into
a smart contract that runs on the chain. The idea is to derive a
validator script from the step function, using the data value to hold the
state of the machine, and the redeemer to provide the transition signal.
A valid transition in a \CEM{}
will correspond to a single valid transaction on the chain. The
validator is used to determine whether a transition is valid and the
state machine can advance to the next state. More specifically, this
validator should ensure that we are transitioning to a valid target
state, the corresponding transaction satisfies the emitted constraints
and that there are no outputs in case the target state is final:

\[
\mkValidator{\mathcal{C}}(s, i, \mi{txInfo}) = \left\{
  \begin{array}{lll}
  \true  & \mi{if} \ \CStep{s}{i}{\txeq} \\
         & \mi{and} \ \satisfies(\mi{txInfo}, \txeq) \\
         & \mi{and} \ \checkOutputs(s', \mi{txInfo}) \\
  \false & \mi{otherwise}
  \end{array}
\right.
\]

\noindent
Note that unlike the step function which returns the new state, the
validator only returns a boolean. On the chain the next state is
provided with the transaction output that ``continues'' the state
machine (if it continues), and the validator simply validates that
the correct next state was provided.\footnote{
A user can run the step function locally to determine the correct next state off-chain.
}

\subsection{Behavioural Equivalence}
We have explained how to compile state machines to smart contracts but
how do we convince ourselves that these smart contracts will behave as
intended? We would like to show
\begin{inparaenum}[(1)]
\item that any valid transition in a \CEM{} corresponds to a valid transaction
  on the chain, and
\item that any valid transaction on the chain corresponds to a valid transition.
\end{inparaenum}
We refer to these two properties as soundness and completeness below.

While state machines correspond to automata, the automata theoretic
notion of equivalence --- trace equivalance --- is too coarse when we
consider state machines as running processes. Instead we use
bisimulation which was developed in concurrency theory for exactly
this purpose, to capture when processes behave the
same~\cite{sangiorgi}. We consider both the state machine and the
ledger itself to be running processes.

If the state machine was the only user of the ledger then we could
consider so-called strong bisimulation where we expect transitions in
one process to correspond to transitions in the other and
vice-versa. But, as we expect there to be other unrelated transactions
occurring on the ledger we instead consider weak bisimulation where
the ledger is allowed to make additional so-called \emph{internal}
transitions that are unrelated to the behaviour we are interested in
observing.

The bisimulation proof relates steps of the \CEM{} to new transaction
submissions on the blockchain.  Note that we have a \textit{weak}
bisimulation, since it may be the case that a ledger step does not
correspond to a \CEM{} step.

\begin{definition}[Process relation]
A \CEM{} state $s$ corresponds to a ledger $l$ whenever $s$ appears
in the current \UTXO{} set,
locked by the validator derived from this \CEM{}:
\[
\Sim{l}{s}
\]
\end{definition}

\begin{definition}[Ledger step]
Concerning the blockchain transactions, we only consider valid
ledgers.\footnote{ In our formal development, we enforce validity
  statically at compile time.  }  Therefore, a valid step in the
ledger consists of submitting a new transaction $tx$,
valid w.r.t. to the current ledger $l$,
resulting in an extended ledger $l'$:
\[
\LStep{l}{tx}
\]
\end{definition}

\begin{proposition}[Soundness]
Given a valid \CEM{} transition $\CStep{s}{i}{\txeq}$ and a valid ledger
$l$ corresponding to source state $s$, we can construct a valid
transaction submission to get a new, extended ledger $l'$ that
corresponds to target state $s'$:
\[
\infer[\textsc{sound}]
  {\exists tx\ l'\ .\ \LStep{l}{tx}\ \wedge \Sim{l'}{s'}}
  {%
    \CStep{s}{i}{\txeq}
  & \Sim{l}{s}
  }
\]
\end{proposition}

\paragraph{Note.}
We also require that the omitted constraints are satisfiable in the
current ledger and the target state is not a final one, since there
would be no corresponding output in the ledger to witness
$\Sim{l'}{s'}$.  We could instead augment the definition of
correspondence to account for final states, but we refrained from
doing so for the sake of simplicity.

\begin{proposition}[Completeness]
Given a valid ledger transition $\LStep{l}{tx}$ and a \CEM{} state $s$
that corresponds to $l$, either $tx$ is irrelevant to the current \CEM{}
and we show that the extended ledger $l'$ still corresponds to source
state $s$, or $tx$ is relevant and we exhibit the corresponding \CEM{}
transition $\CStep{s}{i}{\txeq}$\footnote{ We cannot provide a
  correspondence proof in case the target state is final, as explained
  in the previous note.  }:
\[
\infer[\textsc{complete}]
  { \Sim{l'}{s}\ \vee\ \exists i\ s'\ \txeq\ .\ \CStep{s}{i}{\txeq} }
  { \LStep{l}{tx}
  & \Sim{l}{s}
  }
\]
\end{proposition}

Together, soundness and completeness finally give us weak bisimulation.
Note, however, that our notion of bisimulation differs from the textbook one (e.g. in Sangiorgi~\cite{sangiorgi}),
due to the additional hypotheses that concern our special treatment of constraints and final states.


\section{Methodology}\label{sec:method-ML}
This section presents the experiments done with machine learning techniques to predict GPU kernels. Two different approaches with machine learning were proposed. First approach was done with the same features than analytical model, trying to be fair with the analytical model; and second approach was done using a phase of features extraction.

The methodology of the first approach is shown in Section~\ref{ssec:methodFair} and the methodology of the second approach is presented in Section~\ref{ssec:methodExrtact}.
In the second approach was used correlation analysis and hierarchical clustering to select the features to use a input in the machine learning algorithms. Profile information was collected of each Application over each GPU.
We used three different machine learning methods: Linear Regression, Support Vector Machines and Random Forest.
There exists other machine learning techniques with sophisticated learning process. However, in this work, we wanted to use simple models to prove that they achieve reasonable predictions.

The error of the predictions is presented using the Mean Absolute Percentage Error (MAPE), see equation~\ref{ec:MAPE}. In this equation, $t_m$ is the measured time and $t_k$ is the predicted time. With this error we have analysed the reliability of our approaches.

\begin{equation}
    mape = \frac{100}{n}\sum _{t=1}^{n}\left|{\frac {{t_m}-{t_k}}{{t_m}}}\right| \label{ec:MAPE}
\end{equation}

We used R to automate the statistical analyses, in conjunction with the \texttt{e1071} and \texttt{randomForest} packages to use the \texttt{svm} and \texttt{randomForest} functions respectively. Collected data, experimental results and source codes are publicly available\footnote{Hosted at GitHub: \texttt{\scriptsize https://github.com/marcosamaris/gpuperfpredict} [Accessed on 6 march 2018]} under Creative Commons Public License.

To perform our experiments with machine learning techniques, we first collected the performance profiles (metrics and events) for each kernel and GPU.
All data was collected using the CUDA profiling tool \textbf{nvprof}. This profiling tool enables us to collect data from the command-line reducing the overhead of this process. This process is done without modifications on application source codes. nvprof has four modes to collect information from command-line: summary mode, GPU-trace/API-trace mode, event/metric summary mode and event/metric trace mode. We have used GPU-trace mode and event/metric trace mode.

GPU-trace mode permits collecting data only from CUDA API functions, such as kernel executions, memory transfer throughput between CPU and GPU, among others, without the overhead of profiling the main instructions. Execution times were collected in this mode, along with other features, such as CUDA grid and block dimensions, number of registers per thread, static and dynamic shared memory, device information, streams and the name of executed kernels. A total of 16 features were collected in this mode.

In event/metric trace mode, all events and metrics are collected for each kernel execution. Although this causes a large overhead in the execution of the kernels, it gives detailed information about the behavior and performance of the executed CUDA kernel functions. All applications were iterated over their selected parameters. The number of events and metrics varied according to the compute capability of the GPUs. All collected information resulted in an approximated size of 12.5GB and the process spent up to 14 days in each GPU.

For each sample, the metrics, events and traces information were collected in different phases, therefore avoiding the overhead over the measured execution time of the application. The execution time was taken from GPU-trace mode, thus each GPU-trace was executed ten times, and for the experiments was taken the mean of these samples and it was with a confidence interval of 95\%.

\subsection{Machine Learning without extraction features}\label{ssec:methodFair}
These experiments were done with all the kernels presented in Section~\ref{ssec:useCases}, we used the two-dimensional applications (matrix multiplication and matrix addition) using three different sizes for the CUDA thread blocks, $8^2$, $16^2$ and $32^2$, and input sizes or matrix sizes we vary from $2^{8}$ to $2^{13}$ increasing the matrix size in a step of $2^8$. We took 32 samples per block size, resulting in 96 samples per GPU and a total of 768 samples. For the uni-dimensional problems (dot product and vector addition), the number of thread blocks also was varied in three configurations, $8^2$, $16^2$ and $32^2$, and we used input sizes or vector sizes from $2^{17}$ to $2^{28}$. From $2^{17}$ to $2^{22}$ 6 samples were taken, and from $2^{23}$ to $2^{28}$ 63 samples were taken. For these two applications 69 samples for each configuration were taken, resulting in 207 samples per GPU and a total of 1656 samples. For sub-array maximum problem, 69 samples with the original configuration were taken, for a total of 552 samples.

All Rodinia applications were iterated over their selected parameters shown in Table~\ref{tab:GPUs}. For most kernels of Rodinia applications, we generated many samples, but we could generate only 57 samples on each GPU from the Back-Propagation (BCK) application. To limit the bias on the learning algorithms, we selected 100 random samples from the other applications in both approaches. It was, this sampled data, that was used as dataset for the ML algorithms in both approaches.

To be fair with the analytical model, we then choose similar communication and computation parameters to use as feature inputs for the machine learning algorithms. We performed the evaluation using cross-validation, that is, for each target GPU, we performed the training using the other 8 GPUs, testing the model in the target GPU. This process was done for each application separately.

To compare fairly our analytical model with each ML techniques, the features that we used to feed the Linear Regression, Support Vector Machines and Random Forest algorithms are presented in Table \ref{tab:predictors}.

\begin{table}[htpb]
    \centering
    \scalebox{.825}{
        \begin{tabular}{c  c}
        \toprule
        \textbf{Feature}&\textbf{Description} \\\midrule
        \texttt{num\_of\_cores} & \specialcell{Number of cores per GPU} \\ \midrule
        \texttt{max\_clock\_rate} & \specialcell{GPU Max Clock rate} \\\midrule
        \texttt{Bandwidth} & \specialcell{Theoretical Bandwidth} \\\midrule
        \texttt{Input Size} & \specialcell{Size of the problem} \\\midrule
        \texttt{totalLoadGM} & \specialcell{Load transaction in Global Memory} \\\midrule
        \texttt{totalStoreGM} & \specialcell{Store transaction in Global Memory} \\\midrule
        \texttt{TotalLoadSM} & \specialcell{Load transaction in Shared Memory} \\\midrule
        \texttt{TotalStoreSM} & \specialcell{Store transaction in Global Memory} \\\midrule
        \texttt{FLOPS SP} & \specialcell{Floating operation in Single Precision} \\\midrule
        \texttt{BlockSize} & \specialcell{Number of threads per blocks} \\\midrule
        \texttt{GridSize} & \specialcell{Number of blocks in the kernel} \\\midrule
        \texttt{No. threads} & \specialcell{Number of threads in the applications } \\\midrule
        % \texttt{Executed IPC} & \specialcell{Instructions executed per cycle} \\\midrule
        \texttt{Achieved Occupancy } & \specialcell{Ratio of the average active warps per active cycle to the maximum number of warps ed on a multiprocessor.} \\\midrule
        \end{tabular}}
        \caption{Features used as input in the machine learning techniques}
    \label{tab:predictors}
\end{table}

To generate the flags \texttt{totalLoadGM}, \texttt{totalStoreGM}, \texttt{TotalLoadSM} and \texttt{TotalStoreSM}, the number of requests was divided by the number of transactions per request.

We first transformed the data to a $log_2$ scale and, after performing the learning and predictions, we returned to the original scale using a $2^{pred}$ transformation~\citep{Barnes:2008:RAS}, reducing the non-linearity effects. Figure~\ref{fig:QQplot} shows the difference between the trained model without (left-hand side graph) and with (right-hand side graph) logarithmic scale. The linear regression resulted in poor fitting in the tails, resulting in poor predictions. This problem was solved with the $log_2$ transformation.

\begin{figure}[htpb]
 \centering
 \includegraphics[scale=.3]{./images/QQplot.eps}
 \caption{Quantile-Quantile Analysis of the generated models}
 \label{fig:QQplot}
\end{figure}

\subsection{ML with extraction features}\label{ssec:methodExrtact}

In a machine learning process, the accuracy of the predictions depends on the quality of the pre-processing phase~\citep{Dasu:2003:EDM:861869}. In this section, we present a extraction features process to determine a set of parameters that permits predicting the running time of GPU applications in different contexts or scenarios. As it was mentioned above, we collected separately the set of events and metrics and the execution times of each application of Section~\ref{ssec:useCases} over all GPUs presented in Section~\ref{ssec:GPUTestbed}.

To do these experiments, kernels of matrix addition, dot product, vector addition and sub-array maximum problem were used. The same number of samples than last section was used. All Rodinia applications were used. As before was mentioned, to limit the bias on the learning algorithms, we selected 100 random samples from the other applications. In total 16 kernels were used for these experiments. It was, this sampled data, that was used as dataset for the ML algorithms.

We cleaned the collected information, dropping features with no variation among executions and without concordance among GPUs. Each cleaned sample of the dataset resulted in 85 kernel features plus 11 GPU architecture features, for a total of 96 features. We then performed a correlation analysis, followed by a clustering analysis to reduce the number of features (dimensionality reduction).

We first transformed the data into a $log_2$ scale and, followed by a normalization, reducing the non-linearity effects~\citep{Barnes:2008:RAS} (line 2 and 3 of Algorithm~\ref{fig:methods}). To select a small number of application execution features to use with the ML algorithms, we performed correlation and clustering analyses over the data. We performed the correlation analysis using the Spearman Correlation Coefficient (SCC), since it captures relations and variations among features over different scales by using the rank values of the features. We evaluated the SCC for all features against the kernel execution times and applied a threshold of 0.75, keeping only the 21 features that had a SCC above it (line 4). These features are shown below:

\begin{algorithm}[htpb]
\SetAlgoLined
\KwResult{Prediction of the ML techniques}
\textbf{Input: Data sets}\;
ScaledData = scaleLog(Data sets)\;
NormalizedData = Normalize(ScaledData)\;
SCC = corr(NormalizedData, 0.75)\;
    \ForAll{No. Param in [5, 10]}{
    dendroG = hclust(corr(SCC))\;
    cutDendro = cutTree(dendroG, No. Param)\;
    features = variance(cutDendro)\;
        \ForAll{context in [GPUs, Kernels]}{
        \ForAll{ML in [LM, SVM, RF]}{
        trainingSet != features[context]\;
        testSet == features[context]\;

        Model = ML( trainingSet)\;
        output = predict(testSet)\;
        }
     }
 }
\caption{Algorithm of the methodological process done in this second approach}
\label{fig:methods}
\end{algorithm}

\begin{lstlisting}[basicstyle=\small, numbers=none]
 [1] "elapsed_cycles_sm"
 [2] "gld_inst_32bit"
 [3] "gst_inst_32bit"
 [4] "inst_executed"
 [5] "inst_issued1"
 [6] "gld_request"
 [7] "gst_request"
 [8] "active_cycles"
 [9] "global_load_transactions"
[10] "global_store_transactions"
[11] "device_memory_read_transactions"
[12] "l2_read_transactions"
[13] "l2_write_transactions"
[14] "issued_control.flow_instructions"
[15] "executed_control.flow_instructions"
[16] "issued_load.store_instructions"
[17] "executed_load.store_instructions"
[18] "issue_slots"
[19] "control.flow_instructions"
[20] "load.store_instructions"
[21] "misc_instructions"
\end{lstlisting}

To further reduce the features, we used a hierarchical clustering algorithm. We first create a similarity matrix over all features, using the Spearman correlation coefficient. This similarity matrix is the input for the clustering algorithm. This algorithm them performs the hierarchical clustering by first assigning each feature to its own cluster, and then proceeding iteratively, at each stage joining the two most similar clusters, continuing until there is just a single cluster, as shown in Figures~\ref{fig:Cuthclust:5} and \ref{fig:Cuthclust:10}, which present dendrograms built from 20 features.

\begin{figure}[htbp]
    \centering
    \includegraphics[scale=.75]{./images/cluster-5.pdf}
    \caption{Dendrogram with 5 clusters to select 1 features from each one}
    \label{fig:Cuthclust:5}
\end{figure}

We can then select a number of clusters by cutting the tree at a certain height. For instance, in Figure~\ref{fig:Cuthclust:5}, the horizontal blue dashed line cuts the tree in a height that keeps 5 clusters and in Figure~\ref{fig:Cuthclust:5}, the horizontal blue dashed line cuts the tree in a height that keeps 10 clusters. Finally, we performed a variance analysis to select which feature correlates better with the kernel execution time.

\begin{figure}[htbp]
    \centering
    \includegraphics[scale=.75]{./images/cluster-10.pdf}
    \caption{Dendrogram with 10 clusters to select 1 features from each one}
    \label{fig:Cuthclust:10}
\end{figure}

Regarding the GPU parameters, we performed a correlation analysis and experimented with different number of parameters. As, it was not possible to get a high Spearman  Correlation Coefficient with the GPU parameters, the parameters were taken considering the order of the SCC. In this way, we compared the GPU parameters against the kernel execution times and tested with different number of GPU parameters. GPU parameters were 11 and during the experiments we concluded that few GPU parameters captured the relations between hardware and applications.

\begin{lstlisting}[basicstyle=\small,numbers=none]
[1]  compute_capability
[2]  gpu_clock_rate
[3]  num_of_cores
[4]  num_of_sm
[5]  num_sp_per_sm
[6]  L2_size
[7]  bus
[8]  memory_clock
[9]  theoretical_flops
[10] bandwidth
[11] global_memory_size
\end{lstlisting}

In these experiments, we used three different machine learning methods: Linear Regression, Support Vector Machines and Random Forest. These machine learning techniques achieved a high precision in the predictions with the extracted features. Ensemble methods were also tested, but the additional computational complexity did not justify the small improvement in accuracy of the predictions. We evaluate the ML algorithms using sets of 5 and 10 features, and considered two experimental contexts:

\begin{enumerate}
\item GPUs: we evaluated if the ML algorithm could predict the execution time over a previously unseen GPU. We used a leave-one-out cross validation procedure, by selecting the samples from one GPU as a test set and using the samples from the other 8 GPUs as training set.

\item Kernels: in this case we predict the execution time of a previously unseen CUDA kernel. We used the same leave-one-out cross validation procedure, but now selecting the samples from a single kernel as a test set and using the samples from the other 15 kernels as training set.
\end{enumerate}


%Suggested order of slides
% slides-mc-losses
% slides-mc-softmax-regression
% slides-mc-binary-reduction
% slides-mc-codebooks

\subsection{Multiclass Classification and Losses}
\includepdf[pages=-]{../slides-pdf/slides-mc-losses.pdf}

\subsection{Softmax Regression}
\includepdf[pages=-]{../slides-pdf/slides-mc-softmax-regression.pdf}

\subsection{One-vs-Rest and One-vs-One}
\includepdf[pages=-]{../slides-pdf/slides-mc-binary-reduction.pdf}

\subsection{Designing Codebooks and ECOC}
\includepdf[pages=-]{../slides-pdf/slides-mc-codebooks.pdf}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% UMB-CS110-2015S: Introduction to Computing
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/UMB-CS110-2015S
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\def \topDirectory {.}
\def \texDirectory {\topDirectory/src/main/tex}

\documentclass[12pt,letterpaper,twoside]{article}
\usepackage{\texDirectory/template/style/directives}
\usepackage{\texDirectory/template/style/assignment}
\input{\texDirectory/template/config}

\begin{document}

\doc{title}{Lab Session Problems}
\doc{points}{0}

\prepare{header}

\subsection*{Week 5}
\hfill \textbf{March 5, 2015}

Consider a two-dimensional array matrix with $11$ rows and $11$ columns, corresponding to a Cartesian plot with x axis $[-10, 10]$ and y-axis $[-10, 10]$. Initialize this matrix such that each element in matrix is zero. Imagine that the matrix corresponds to a $11$ meters in $11$ meters map that represents footsteps of a subject in real-world. It means all elements of the matrix in which the subject is or has been is shown with one and all square-meters that the person has not visited yet is shown with zero. Imagine that someone is initially at position \texttt{matrix[6][6]}.
\begin{enumerate}[itemsep=0pt]
\item At every instance of time, a random number between $1$ and $4$ is generated and based on generated number the person decides to go up, right, bottom or left. Generate such number and imagine the subject has moved in corresponding direction. Show current footsteps of the person in a program \texttt{MagicWorld1.java}.
\item Write a program \texttt{MagicWorld2.java} that computes how many moves it would take until the person reaches an element on the edge of the world.
\item Modify the previous program to \texttt{MagicWorld3.java} such that when the person moves in one direction, doesn't go back in that direction in his next step. How many moves would it take in this scenario that the person reaches the edge of the world.
\item Try to modify your program to \texttt{MagicWorld4.java} that presents an algorithm that can maximize the number of movements it takes that the person reaches the edge of the world.
\end{enumerate}

\end{document}


\documentclass{anstrans}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\title{Implementation of the $SP_3$ equations in a MOOSE-based application}
\author{Roberto E. Fairhurst Agosta, Kathryn D. Huff}

\institute{
University of Illinois at Urbana-Champaign, Dept. of Nuclear, Plasma, and Radiological Engineering\\
ref3@illinois.edu
}

%%%% packages and definitions (optional)
\usepackage{graphicx} % allows inclusion of graphics
\usepackage{booktabs} % nice rules (thick lines) for tables
\usepackage{microtype} % improves typography for PDF
\usepackage{xspace}
\usepackage{tabularx}
\usepackage{caption}
\usepackage{floatrow}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{placeins}
\usepackage{amsmath}
\usepackage[acronym,toc]{glossaries}
\include{acros}
\makeglossaries

\usepackage[printwatermark]{xwatermark}
\usepackage{xcolor}
\usepackage{graphicx}
\usepackage{lipsum}

\newcommand{\SN}{S$_N$}
\renewcommand{\vec}[1]{\bm{#1}} %vector is bold italic
\newcommand{\vd}{\bm{\cdot}} % slightly bold vector dot
\newcommand{\grad}{\vec{\nabla}} % gradient
\newcommand{\ud}{\mathop{}\!\mathrm{d}} % upright derivative symbol

\newcolumntype{c}{>{\hsize=.56\hsize}X}
\newcolumntype{b}{>{\hsize=.7\hsize}X}
\newcolumntype{s}{>{\hsize=.74\hsize}X}
\newcolumntype{f}{>{\hsize=.1\hsize}X}
\newcolumntype{a}{>{\hsize=.45\hsize}X}
%\usepackage[pagestyles]{titlesec}
%\titleformat*{\subsection}{\normalfont}
%\titleformat{\section}{\bfseries}{Item \thesection.\ }{0pt}{}

%\newwatermark[allpages,color=gray!50,angle=45,scale=3,xpos=0,ypos=0]{DRAFT}

\begin{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% \section{Abstract}

% \textit{
%     The Multi-physics Object-Oriented Simulation Environment is a framework that supports the development of applications for solving nonlinear systems of differential equations.
%     This work presents the implementation of an application in that framework for solving the Simplified $P_3$ equations.
%     We conducted two exercises with results as low as 12 pcm and as high as 145 pcm from the reference results, successfully validating the proposed application.
% }

\section{Introduction}

% Motivation
Multi-physics simulations are necessary for assessing the safety characteristics of nuclear reactors \cite{fairhurst-agosta_multi-physics_2020}.
Solving the neutronics equations provides information on the power distribution, which plays a crucial role in the thermal-fluids behavior of the reactor.
Therefore, the right modeling of the neutron flux is necessary.
% what I am doing
This work presents the implementation and verification of the Simplified $P_3$ ($SP_3$) equations \cite{gelbard_spherical_1960} in a \gls{MOOSE}-based application as an alternative for solving the neutronics equations in a nuclear reactor.

% MOOSE
MOOSE \cite{gaston_moose_2009} is a computational framework that supports engineering analysis applications.
In a nuclear reactor, several partial differential equations describe its physical behavior.
These equations are typically nonlinear, and they are often coupled to each other.
MOOSE is an open-source \gls{FEM} framework that supports the development of applications for solving such systems.
% MOOSE supports the development of applications for solving such systems.
% MOOSE is an open-source \gls{FEM} framework.
% The framework itself relies on LibMesh \cite{kirk_libmesh_2006} and PetSc \cite{balay_petsc_2016} for solving nonlinear equations.
% MOOSE-based applications define weak forms of the governing equations and modularize the physics expressions into "kernels."
% Kernels are C++ classes containing methods for computing the residual and Jacobian contributions of individual pieces of the governing equations.
% MOOSE and LibMesh translate them into residual and Jacobian functions.
% These functions become inputs into PetSc solution routines.
All the software built on the MOOSE framework shares the same \gls{API}, facilitating relatively easy coupling between different phenomena.
While the $SP_3$ equations solve the neutronics equations in a nuclear reactor, other applications may solve the thermal-fluids, and given they share the same API; their integration is straightforward.

% lit - review: the SPN approximation
The $P_N$ method \cite{davidson_neutron_1957} discretizes the transport equation by expanding the angular dependence of the neutron flux in spherical harmonics, considering up to order $N$ polynomials.
If $N \rightarrow \infty$, the solution of the $P_N$ equations tends to the exact transport solution.
In three-dimensional geometries, the number of $P_N$ equations is proportional to $(N+1)^2$, whereas, in one-dimensional planar geometries, the number of $P_N$ equations is $(N+1)$.
Gelbard \cite{gelbard_spherical_1960} proposed the $SP_N$ approximation by replacing the second derivatives in the one-dimensional planar $P_N$ equations with three-dimensional Laplacian operators.
This approximation considerably reduces the number of equations conserving a reasonable accuracy \cite{capilla_applications_2009}.

The $SP_N$ approximation has the disadvantage that the solution does not usually converge to the true transport solution as $N \rightarrow \infty$.
Additionally, the theoretical basis of Gelbard's formulation of $SP_N$ approximation was weak.
For these reasons, the method did not gain widespread use until the 2000s, when thanks to Pomraning \cite{pomraning_asymptotic_1993}, Brantley, and Larsen's \cite{brantley_simplifiedP3_2000} contribution, the method gained a stronger theoretical basis.
% Although Gelbard proposed the $SP_N$ approximation in the 60s, the method did not have a strong theoretical basis until the 90s with Pomraning \cite{pomraning_asymptotic_1993}, Brantley, and Larsen's \cite{brantley_simplifiedP3_2000} contribution.
% Additionally, the $SP_N$ approximation has the disadvantage that the solution does not usually converge to the true transport solution as $N \rightarrow \infty$.

In practice, the $SP_N$ equations are most accurate for diffusive problems or for problems in which the solution behaves nearly one-dimensionally and has weak tangential derivatives at material interfaces.
For problems with strong, multidimensional transport effects, such as voids, streaming regions, or geometrically complex regions, the $SP_N$ solutions are less accurate \cite{downar_parcs_2004}.
However, several results show that the $SP_N$ approximation yields more accurate solutions than the diffusion approximation \cite{beckert_development_2007} \cite{fliscounakis_potential_2012} \cite{ryu_finite_2013} \cite{khosravi_mirzaee_reactor_2019} with considerably less computational expense than the discrete ordinates ($S_N$) method \cite{brantley_simplifiedP3_2000}.
% why is SP_3 preferred over diffusion
For example, the $SP_3$ approximation is preferable over the diffusion approximation for modeling reactors using MOX/UO$_2$ fuel assemblies.
MOX fuel assemblies have higher thermal absorption and fission cross-sections than UO$_2$ fuel assemblies, and consequently, their thermal flux is lower while their power production higher.
Modeling these characteristics using the diffusion approximation may be challenging \cite{brantley_simplifiedP3_2000} \cite{capilla_applications_2009}.

% lit - review: software using the SPN approximation
The $SP_3$ approximation gained usage throughout the last couple of decades and currently, different software uses it to solve the neutron transport equation.
% Some of those software are SCOPE2 \cite{tatsumi_object-oriented_2002}, PARCS \cite{downar_parcs_2004}, DYN3D \cite{beckert_development_2007}, SIMULATE-5 \cite{bahadir_studsviks_2009}, and COCAGNE \cite{fliscounakis_potential_2012}.
Some of those software are PARCS \cite{downar_parcs_2004}, DYN3D \cite{beckert_development_2007}, and COCAGNE \cite{fliscounakis_potential_2012}.

\section{Methodology}

This section describes the methodology followed for solving the equations.
% Equations P3
Davidson \cite{davidson_neutron_1957} defined the one-dimensional multi-group P$_N$ equations.
For $N=3$ and steady-state, the equations become

\begin{align}
    & \frac{d}{dx} \phi_{1,g} + \Sigma_{t,g} \phi_{0,g} = \sum_{g'=1}^G \Sigma_{s0,g' \rightarrow g} \phi_{0,g'} + \frac{\chi_g}{k_{eff}} \sum_{g'=1}^G \nu\Sigma_{f,g'} \phi_{0,g'} \notag \\ & \quad \quad \quad \quad \quad \quad \quad + Q_{0,g}  \label{eq:P3-0} \\
    & \frac{1}{3} \frac{d}{dx} \phi_{0,g} + \frac{2}{3}\frac{d}{dx}\phi_{2,g} + \Sigma_{t,g} \phi_{1,g} = \sum_{g'=1}^G \Sigma_{s1,g' \rightarrow g} \phi_{1,g'} + Q_{1,g} \label{eq:P3-1} \\
    & \frac{2}{5} \frac{d}{dx}\phi_{1,g} + \frac{3}{5}\frac{d}{dx}\phi_{3,g} + \Sigma_{t,g} \phi_{2,g} = \sum_{g'=1}^G \Sigma_{s2,g' \rightarrow g} \phi_{2,g'} + Q_{2,g} \label{eq:P3-2} \\
    & \frac{3}{7}\frac{d}{dx}\phi_{2,g} + \Sigma_{t,g} \phi_{3,g} = \sum_{g'=1}^G \Sigma_{s3,g' \rightarrow g} \phi_{3,g'} + Q_{3,g} \label{eq:P3-3} \\
    \intertext{where}
    & \phi_{n,g} = \mbox{$n^{th}$ moment of group $g$ neutron flux}  \notag \\ %  } [n \cdot cm^{-2} \cdot s^{-1}]
    & \Sigma_{t,g} = \mbox{group $g$ macroscopic total cross-section}  \notag \\ %  } [cm^{-1}]
	& \Sigma_{sn,g' \rightarrow g} = \mbox{$n^{th}$ moment of the group $g'$ to group $g$} \notag \\
	& \mbox{macroscopic scattering cross-section}  \notag \\ %  } [cm^{-1}]
	& \nu\Sigma_{f,g} = \mbox{group $g$ macroscopic production cross-section}  \notag \\ %  } [cm^{-1}]
	& \chi_{g} = \mbox{group $g$ fission spectrum}  \notag \\ %  } [cm^{-1}]
	& k_{eff} = \mbox{eigenvalue}  \notag \\ %  } [-]
	& Q_{n,g} = \mbox{$n^{th}$ moment of group $g$ external neutron source}  \notag \\  % } [n \cdot cm^{-3} \cdot s^{-1}]
	& G = \mbox{number of energy groups}.  \notag % } [-]
\end{align}

Assuming an isotropic external source and negligible anisotropic group-to-group scattering \cite{brantley_simplifiedP3_2000}
\begin{align}
	& Q_{n,g} = 0, \quad n > 0 \notag \\
	& \Sigma_{sn,g' \rightarrow g} = 0, \quad g' \ne g, \quad n > 0 \notag
\end{align}

\noindent
simplifies equations \ref{eq:P3-1} and \ref{eq:P3-3}, allowing to express the odd moments of the flux $\phi_{1,g}$ and $\phi_{3,g}$ as functions of the even moments $\phi_{0,g}$ and $\phi_{2,g}$.
Introducing $\phi_{1,g}$ and $\phi_{3,g}$ into equations \ref{eq:P3-0} and \ref{eq:P3-2} reduces the system from four to two equations.
Introducing the variables $\Phi_{0,g}$ and $\Phi_{2,g}$, reorganizing the equations, and replacing the second derivatives by Laplacian operators \cite{gelbard_spherical_1960} yields the SP$_3$ equations \cite{beckert_development_2007}

% Equations SP_3
\begin{align}
    & - D_{0,g} \Delta \Phi_{0,g} + \Sigma_{0,g} \Phi_{0,g} - 2 \Sigma_{0,g} \Phi_{2,g} = S_{0,g} \label{eq:SP3-0e} \\
    & - D_{2,g} \Delta \Phi_{2,g} + \left( \Sigma_{2,g} + \frac{4}{5} \Sigma_{0,g} \right) \Phi_{2,g} - \frac{2}{5} \Sigma_{0,g} \Phi_{0,g} = -\frac{2}{5} S_{0,g} \label{eq:SP3-2e}
    \intertext{where}
	& \Sigma_{n,g} = \Sigma_{t,g} - \Sigma_{sn,g' \rightarrow g} \notag \\
    & \Phi_{0,g} = \phi_{0,g} + 2 \phi_{2,g} \notag \\
    & \Phi_{2,g} = \phi_{2,g} \notag \\
    & D_{0,g} = \frac{1}{3 \Sigma_{1,g}} \label{eq:diff-rem} \\
    & D_{2,g} = \frac{9}{35 \Sigma_{3,g}} \notag \\
    & S_{0,g} = \sum_{g'\ne g}^G \Sigma_{s0,g' \rightarrow g} \left( \Phi_{0,g'} - 2 \Phi_{2,g'} \right) \notag \\
    & \quad \quad + \frac{\chi_g}{k_{eff}} \sum_{g'=1}^G \nu\Sigma_{f,g'} \left( \Phi_{0,g'} - 2 \Phi_{2,g'} \right) + Q_{0,g}. \notag
\end{align}

% BCs
The Marshak-like vacuum \glspl{BC} complete the system of equations \cite{beckert_development_2007}

\begin{align}
    & \frac{1}{4} \Phi_{0,g} \pm \frac{1}{2} \hat{n} \cdot J_{0,g} - \frac{3}{16} \Phi_{2,g} = 0 \label{eq:SP3-BC1a} \\
    - & \frac{3}{80} \Phi_{0,g} \pm \frac{1}{2} \hat{n} \cdot J_{2,g} + \frac{21}{80} \Phi_{2,g} = 0 \label{eq:SP3-BC2a}
    \intertext{where}
    & J_{n,g} = -D_{n,g} \nabla \Phi_{n,g}. \notag
\end{align}

Finally, multiplying equations \ref{eq:SP3-0e} and \ref{eq:SP3-2e} by a test function and integrating over the domain yields the weak form of the equations modularized into kernels in the MOOSE-based application.
For brevity, we will not display the derivation of the kernels here.
Such procedure is standard in weighted residual methods and can be found in \cite{ryu_finite_2013} and any finite elements book \cite{quarteroni_numerical_1994}.

\section{Results}

This section presents several numerical results that validate the calculation scheme.
The following sections discuss the results of a one group exercise presented in \cite{brantley_simplifiedP3_2000} and describe the results of the C5 MOX Benchmark \cite{cavarec_benchmark_1994}.

\subsection{One group two-dimensional problem}

This section describes a one group, isotropic-scattering eigenvalue problem introduced by Brantley and Larsen \cite{brantley_simplifiedP3_2000}.
This section also presents the eigenvalue obtained with the $SP_3$ solver and compares it against the reference value.
Figure \ref{fig:2D} shows the problem's geometry, and Table \ref{tab:cross-sections} specifies its cross-sections.
Due to the problem's symmetry, the model included only a quarter of the core.
We created the mesh using the \textit{Gmsh} \cite{geuzaine_gmsh_2009} meshing tool.
The mesh had $6 \times 10^{3}$ elements.
The simulation convergence criterion was $10^{-8}$ for the neutron flux.

\begin{figure}[H] %or H
    \centering
    \includegraphics[width=0.75\linewidth]{figures/mesh2.png}
    \hfill
    \caption{Geometry of the one group eigenvalue problem. Fuel in red. Moderator in gray.}
    \label{fig:2D}
\end{figure}

% \usepackage{booktabs}
% \begin{table}[htbp!]
\begin{table}[h]
	\centering
	\caption{Cross-sections of the one group eigenvalue problem \cite{brantley_simplifiedP3_2000}. Values expressed in $cm^{-1}$.}
	\label{tab:cross-sections}
	\begin{tabular}{llll}
	\toprule
	Material	& $\Sigma_t$ & $\Sigma_{s0}$ & $\nu\Sigma_f$ \\
	\midrule
	Moderator	& 1.00		& 0.93			& 0.00			\\
	Fuel		& 1.50		& 1.35			& 0.24			\\
	\bottomrule
	\end{tabular}
\end{table}

Table \ref{tab:keff-1st} compares the eigenvalue obtained with the $SP_3$ solver and the reference value \cite{brantley_simplifiedP3_2000} using the equation
\begin{align}
  % & \Delta_\rho = \left| \rho_{SP_3} - \rho_{Ref} \right| = \left| \frac{k_{SP_3}-1}{k_{SP_3}} - \frac{k_{Ref}-1}{k_{Ref}} \right| = \left| \frac{k_{SP_3}-k_{Ref}}{k_{SP_3} k_{Ref}} \right| \label{eq:delta-rho} \\
  & \Delta_\rho = \left| \frac{k_{SP_3}-k_{Ref}}{k_{SP_3} k_{Ref}} \right| \label{eq:delta-rho} \\
  \intertext{where}
  & \Delta_\rho = \mbox{reactivity difference } [pcm] \notag \\
  & k_{SP_3} = \mbox{eigenvalue obtained with $SP_3$ solver} \notag \\
  & k_{Ref} = \mbox{reference eigenvalue.} \notag
\end{align}

\begin{table}[htbp!]
	\centering
	\caption{Comparison between the result obtained with the $SP_3$ solver and the reference result for the one group eigenvalue problem.}
	\label{tab:keff-1st}
	\begin{tabular}{lll}
	\toprule
		$k_{Ref}$	& $k_{SP_3}$ 	& $\Delta_{\rho}$	\\
	\midrule
	 	0.79862		& 0.79854		& 12				\\
	\bottomrule
	\end{tabular}
\end{table}

\subsection{C5 MOX Benchmark}

This section introduces the C5 MOX Benchmark \cite{cavarec_benchmark_1994} and presents the $SP_3$ solver results.
% intro to the benchmark
The \gls{OECD}/\gls{NEA} developed this benchmark to validate methods and identify their strengths, limitations, and accuracy, and suggest needs for method development.
% benchmark specification
Two types of fuel assembly (MOX and UO$_2$) and a reflector comprise the core, shown in Figure \ref{fig:bench1}.
Each fuel assembly consists of a 17 $\times$ 17 array of squared pin cells, as displayed in Figures \ref{fig:bench2} and \ref{fig:bench3}.
The dimensions of each pin cell are 1.26 $\times$ 1.26 cm, being 21.42 $\times$ 21.42 cm the dimensions of each assembly, and 128.52 $\times$ 128.52 cm of the whole core.
The benchmark \cite{cavarec_benchmark_1994} specifies the cross-sections, which have a two-energy group structure.

\begin{figure}[htbp!] %or H
    \centering
    \includegraphics[width=0.85\linewidth]{figures/bench-config.png}
    \hfill
    \caption{C5 MOX benchmark configuration. \textit{R} correponds to the reflector region. Image reproduced from \cite{capilla_applications_2009}.}
    \label{fig:bench1}
\end{figure}

\begin{figure}[htbp!] %or H
    \centering
    \includegraphics[trim=0 0 0 2.1cm, clip=true, width=0.75\linewidth]{figures/bench-config2.png}
    \hfill
    \caption{Structure of the UO$_2$ assembly. Image reproduced from \cite{capilla_applications_2009}.}
    \label{fig:bench2}
\end{figure}

When no anisotropic component of the scattering cross-section is available, the benchmark recommends applying the diagonal transport correction
\begin{align}
  & D_{0,g} = \frac{1}{3 \Sigma_{tr,g}} \label{eq:diff-tr} \\
  & \Sigma_{tr} = \Sigma_{t,g} - \bar{\mu}_{g} \Sigma_{s0,g} \notag
  \intertext{where}
  & \Sigma_{tr, g} = \mbox{group $g$ transport cross-section} \notag \\
  & \bar{\mu}_{g} = \mbox{group $g$ average cosine deviation angle.} \notag
\end{align}

% \begin{figure}[htbp!] %or H
\begin{figure}[h] %or H
    \centering
    % \includegraphics[width=0.95\linewidth]{figures/bench-config3.png}
    \includegraphics[trim=0 0 0 2.1cm, clip=true, width=0.75\linewidth]{figures/bench-config3.png}
    \hfill
    \caption{Structure of the MOX assembly. Image reproduced from \cite{capilla_applications_2009}.}
    \label{fig:bench3}
\end{figure}

For the sake of comparison, we conducted the exercise with and without the transport correction for calculating $D_{0,g}$ with equations \ref{eq:diff-tr} and \ref{eq:diff-rem}, respectively.
Due to the problem's symmetry, the model included only a quarter of the core.
The mesh had $2.4 \times 10^{4}$ elements.
The simulation convergence criterion was $10^{-8}$ for the neutron flux.

Table \ref{tab:keff-2nd} compares the eigenvalues obtained with the $SP_3$ solver and the references.
For the method without correction, we used an eigenvalue reference value from Capilla et al. \cite{capilla_applications_2009} as they conducted the exercise without correction.
For the method with transport correction, we used the reference value from the benchmark \cite{cavarec_benchmark_1994}.
% \usepackage{booktabs}
\begin{table}[htbp!]
	\centering
	\caption{Comparison between the results obtained with the $SP_3$ solver using no correction (equation \ref{eq:diff-rem}), the transport correction (equation \ref{eq:diff-tr}), and the reference results for the C5 MOX Benchmark.}
	\label{tab:keff-2nd}
	\begin{tabular}{lrrr}
	\toprule
							& $k_{Ref}$ & $k_{SP_3}$	& $\Delta_{\rho}$	\\
	\midrule
	No correction			& 0.96969	& 0.97106		& 145				\\
	Transport correction	& 0.93755	& 0.93792		& 43				\\
	\bottomrule
	\end{tabular}
\end{table}

% discussion about the transport correction
The eigenvalues obtained with the $SP_3$ solver are within 145 pcm of the reference values.
However, the difference between the reference values of the different schemes is 3535 pcm, suggesting that the use of the transport correction is necessary.
% examples of codes using the diagonal transport correction PARCS \cite{downar_parcs_2004}, DYN3D \cite{beckert_development_2007}

% discuss results
Our calculations included the pin power distribution, whose reference values the benchmark specifies \cite{cavarec_benchmark_1994}.
For conciseness, we only calculated the power distribution using the transport correction.
To simplify displaying the results, Figure \ref{fig:power-distrib} presents the assembly power distribution with the largest pin power relative difference in each assembly.
The MOX assembly hosts the largest pin power relative difference, its value being 1.8\%.

\begin{figure}[htbp!] %or H
    \centering
    \includegraphics[width=0.95\linewidth]{figures/distrib.png}
    \hfill
    \caption{Assembly power distribution in the C5 MOX Benchmark. Top: Assembly power distribution. Bottom: pin power relative difference expressed in \%.}
    \label{fig:power-distrib}
\end{figure}

\section{Conclusions}

MOOSE is a computational framework that solves systems of nonlinear differential equations.
As part of this work, we implemented the kernels to solve the steady-state $SP_3$ equations in a MOOSE-based application.
Additionally, we carried out two exercises whose reference results were known.
The first exercise solved a one-group eigenvalue problem with a simple geometry, with a result within 12 pcm.
The second exercise studied the C5 MOX Benchmark, solving it using different approaches and obtaining results within 145 pcm.
The calculated pin power values were within 2\% error from the reference.

While the $SP_3$ equations solve the neutronics equations in a nuclear reactor, future work may develop other applications to solve the thermal-fluids or integrate this application to existing applications.
MOOSE-based applications share the same API making their integration straightforward.

\section{Acknowledgements}

This research is being performed using funding received from the DOE Office of Nuclear Energy's University Program (Project 20-19693, DE-NE0008972) 'Evaluation of micro-reactor requirements and performance in an existing well-characterized micro-grid'.
Prof. Huff is also supported by the Nuclear Regulatory Commission Faculty Development Program (award NRC-HQ-84-14-G-0054 Program B), the Blue Waters sustained-petascale computing project supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993) and the state of Illinois, the DOE ARPA-E MEITNER Program (award DE-AR0000983), and the DOE H2@Scale Program (Award Number: DE-EE0008832).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{ans}
\bibliography{bibliography}
\end{document}

% \begin{figure}[htbp!] %or H
%     \centering
%     \includegraphics[width=0.95\linewidth]{figures/radial-layout.png}
%     \hfill
%     \caption{Core radial layout. Image reproduced from \cite{oecd_nea_benchmark_2017}.}
%     \label{fig:radial}
% \end{figure}

% \begin{align}
%     \frac{1}{v_g}\frac{\partial}{\partial t} \phi_g &= \nabla \cdot D_g
%     \nabla \phi_g - \Sigma_g^r \phi_g \sum_{g \ne g'}^G
%     \Sigma_{g'\rightarrow g}^s \phi_{g'} \label{eq:diffusion}
%     \intertext{where}
%     C_i &= \mbox{concentration of delayed neutron precursors} \notag \\
%     &\phantom{{}=1} \mbox{in precursor group $i$}.
% \end{align}


\section{Complexity Analysis}\label{sec:analysis}

In this section, we assume a synchronous communication model with
point-to-point message passing. In this model, all nodes start
executing the algorithm simultaneously and time is divided into
synchronous rounds, i.e., when a message is sent from node $v$ to
its neighbor $u$ at time-slot $t$, it must arrive at $u$ before
time-slot $t+1$.

We first analyze the message and time complexity of the IPv6 address
allocation phase of Matrix. Then, we look into the message
complexity of the control plane of Matrix after the network
initialization phase.

Note that Matrix requires that an underlying acyclic topology (Ctree) has
been constructed by the network before the address allocation starts, i.e.,
every node knows who its parent in the Ctree is.
Moreover, one of the building blocks of Matrix is the IPv6 multihop host
configuration, performed by MHCL \cite{2016techreport}.
%Therefore, we have that
%$Time(Matrix^{IP})$ = $Time(MHCL)$ and $Msg(Matrix^{IP})$ = $Msg(MHCL)$.
%The address allocation phase consists in (1) number of descendants aggregation,
%using a convergecast routine, in which leaf nodes start sending messages to
% their parents and parents aggregate the received information and forward it upwards
%until the root is reached; (2) the root broadcasts address allocation
%information downward along the tree $T$, each message being acknowledged. The
% time complexity of each phase is $depth(T)$ and the message complexity is $2(n-1)$.

\begin{theorem} \cite{2016techreport} For any network of size $n$ with a
spanning collection tree $Ctree$ rooted at node $root$, the message and time
complexity of Matrix protocol in the address allocation phase is
$Msg(Matrix^{IP}(Ctree, root))$ = $O(n)$ and $Time(Matrix^{IP}$ $(T, root))$
= $O(depth(Ctree))$, respectively. This message and time complexity is
asymptotically optimal.
\end{theorem}
\begin{proof}
The address allocation phase is comprised of a tree broadcast and a tree
convergecast.
In the broadcast operation, a message (with address allocation information) must be
sent to every node by the respective parent, which needs $\Omega(n)$ messages.
Moreover the message sent by the root must reach every node at distance
$depth(Ctree)$ hops away, which needs $\Omega(depth(Ctree))$ time-slots.
Similarly, in the convergecast operation, every node must send a message to its
parent after having received a message from its children, which needs
$\Omega(n)$ messages. Also, a message sent by every leaf node must reach the
root, at distance $\leq depth(Ctree)$, which needs $\Omega(depth(Ctree))$
time-slots.
\end{proof}

Next, we examine the communication cost of the routines involved in
the alternative routing, performed in the presence of persistent
node and link failures.

%\begin{theorem} For any failure with a duration time $\Delta$, that originates
%$\mathcal{L_{CTP}}$ topology changes and an interval of $\delta$ between
%beacons, $Message(Matrix(\Delta,\delta,\mathcal{L_{CTP}} ))=
%O(\mathcal{L_{CTP}})$.\end{theorem}
%\begin{proof}
%When a failure occurs, a node that has its parent changed to an adoptive parent
% must send beacons every $\delta$ time. If this failure occurs by $\Delta$ time, that is, the node stays for $\Delta$ time with its current parent as its adoptive parent, then it sends $\frac{\Delta}{\delta}$ beacons. Since one failure can originate $\mathcal{L_{CTP}}$ changes, then the number of beacons sent is $\mathcal{L_{CTP}}\times\frac{\Delta}{\delta}$.
%\end{proof}

\begin{theorem} Consider a network with $n$ nodes and a failure event that
causes $\mathcal{L_{CT}}$ links to change in the collection tree Ctree for at
most $\Delta$ ms.
Moreover, consider a beacon interval of $\delta$ ms.
The control message complexity of Matrix to perform
alternative routing is $Msg(Matrix^{RC})$ = $O(n)$.
\end{theorem}
\begin{proof}
Consider the $\mathcal{L_{CT}}$ link changes in the collection tree
Ctree. Note that $\mathcal{L_{CT}} = O(n)$ since Ctree is acyclic
and, therefore, has at most $n-1$ links. Every link that was changed
must be inserted in the RCtree table of the respective (new) parent
and kept during the interval $\Delta$ using regularly sent beacons
from the child to the parent. Given a beacon interval of $\delta$,
the total number of control messages is bounded by
${\Delta}/{\delta} \times \mathcal{L_{CT}} = O(n)$.
\end{proof}

Note that, in reality, the assumptions of synchrony and
point-to-point message delivery do not hold in a 6LoWPAN. The moment
in which each node joins the tree varies from node to node, such
that nodes closer to the root tend to start executing the address
allocation protocol earlier than nodes farther away from the root.
Moreover, collisions, node and link failures can cause delays and
prevent messages from being delivered. We analyze the performance of
Matrix in an asynchronous model with collisions and transient node
and link failures of variable duration through simulations in
Section~\ref{sec:results}.