\subsection{Probabilistic modelling} \label{section:probabilistic_modelling}

A scientific model is a simple theoretical representation of a complex natural phenomenon to allow the systematic study of its behaviour. The general idea is that if a model is able to explain some observations, it might be capturing its true underlying laws and can therefore be used to make future predictions.
In particular, statistical models are a powerful abstraction of nature. They consist of a set of observed variables and a set of (hidden) parameters. The procedure of fitting the parameters using a set of observations is called inference or learning.

One of the major challenges of inference when dealing with real datasets is the distinction between signal and noise. An ideal model should learn only the information relevant to gain explanatory power while disregarding the noise. However, this is a non-trivial task in most practical situations. Very complex models will tend to overfit the training data, capturing large amounts of noise and consequently leading to a bad generalisation performance to independent datasets. On the other hand, simplistic models will fit the data poorly, resulting in low explanatory power.

The ideas above can be formalised using the framework of probability and statistics.

\subsection{Maximum likelihood inference} \label{section:maximum_likelihood}

A common approach is to define a statistical model of the data $\bfY$ with a set of parameters $\btheta$ that define a probability distribution $p(\bfY|\btheta)$, called the likelihood function. A simple approach to fit a model is to estimate the parameters $\hat{\btheta}$ that maximise the likelihood:
\[
	\hat{\btheta} = \argmax_{\btheta} p(\bfY|\btheta)
\]
This process is called maximum likelihood learning \cite{Stigler2008,Bishop2006,Murphy}. However, in this setting there is no penalisation for model complexity, making maximum likelihood solutions can overfit when the data is relatively sparse. Generalisations that account for model complexity have been proposed that include regularising terms that shrink parameters to small values. However, these are often particular cases of the more general framework of Bayesian statistics \cite{Hastie,Bishop2006,Murphy}.

\subsection{Bayesian inference}  \label{section:bayesian_inference}
In the Bayesian framework, the parameters themselves are treated as random unobserved variables and we aim to obtain probability distributions for $\btheta$, rather than a single point estimate. To do so, prior beliefs are introduced into the model by specifying a prior probability distribution $p(\btheta)$. Then, using Bayes' theorem \cite{Bayes1763}, the prior hypothesis is updated based on the observed data $\bfY$ by means of the likelihood $p(\bfY|\btheta)$ function, which yields a posterior distribution over the parameters:
\[
	p(\btheta|\bfY) = \frac{p(\bfY|\btheta) p(\btheta)}{p(\bfY)}
\]
where $p(\bfY)$ is a constant term called the marginal likelihood, or model evidence \cite{Bishop2006,Murphy}.\\
The choice of the prior distribution is a key part of Bayesian inference and captures beliefs about the distribution of a variable before the data is taken into account. With asymptotically large sample sizes, the choice of prior has negligible effects on the posterior estimates, but it becomes critical with sparse data \cite{Bishop2006,Murphy,Gelman2013}.

There are two common considerations when defining the prior distributions. The first relates to the incorporation of subjective information, or predefined assumptions, into the model. For example, one could adapt the prior distribution to match the results from previous experiments (i.e. an informative prior). Alternatively, if no prior information is available one could set set uninformative priors by following maximum entropy principles \cite{Jaynes1968}.

The second strategy is based on convenient mathematical properties to make inference tractable. If the likelihood and the prior distributions do not belong to the same family of probability distributions (they are not conjugate) then inference becomes more problematic \cite{Raiffa1961,Bishop2006,Murphy,Gelman2013}. The existence of conjugate priors is one of the major reasons that justify the widespread use of exponential family distributions in Bayesian models \cite{Gelman2013}.

Again, the essential point of Bayesian inference is that an entire posterior probability distribution is obtained for each unobserved variable. This has the clear advantage of naturally handling uncertainity in the estimation of parameters. For instance, when making predictions, a fully Bayesian approach attempts to integrate over all possible values of all unobserved varaibles, effectively propagating uncertainity across multiple layers of the model. Nevertheless, this calculation is sometimes intractable and one has to resort to point estimates \cite{Bishop2006,Murphy,Gelman2013}. The simplest approximation to the posterior distribution is to use its mode, which leads to the maximum a posteriori (MAP) estimate:
\[
	\hat{\btheta} = \argmax_{\btheta} p(\btheta) p(\bfY|\btheta)
\]
This is similar to the maximum likelihood objective function, but with the addition of a term $p(\btheta)$. When the prior distribution is well chosen, this term penalises for model complexity. Therefore, in contrast to standard (non-penalised) maximum likelihood inference, Bayesian approaches naturally handle the problem of model complexity and overfitting \cite{Bishop2006,Murphy,Gelman2013}. At the limit of infinite observations, the influence of the prior to the posterior is negligible and the MAP estimate converges towards the Maximum likelihood estimate, hence providing a rational link between the two inference frameworks.

\subsubsection{Deterministic approaches for Bayesian inference} \label{section:deterministic_bayesian_inference}
The central task in Bayesian inference is the direct evaluation of the posterior distributions and/or the computation of expectations with respect to the posterior distributions. In sufficiently complex models, closed-form solutions are not available and one has to resort to approximation schemes, which broadly fall into two classes: stochastic or deterministic \cite{Gelman2013,Blei2016}.

Stochastic approaches hinge on the generation of samples from the posterior distribution via a Markov Chain Monte Carlo (MCMC) framework. Such techniques have the appealing property of generating exact results at the asymptotic limit of infinite computational resources. However, in practice, sampling approaches are computationally demanding and suffer from limited scalability to large datasets \cite{Blei2016}. \\
In contrast, deterministic approaches are based on analytical approximations to the posterior distribution, which often lead to biased results. Yet, given the appropriate settings, these approaches are potentially much faster and scalable to large applications \cite{Bishop2006,Murphy,Blei2016}.

% \paragraph{Laplace approximation}% \label{section:laplace_approximation}
% The Laplace approximation is probably the simplest of the deterministic tecniques, where the aim is to construct a Gaussian approximation around the mode of the true posterior distribution using a second-order Taylor expansion \cite{Bishop2006,Murphy}.\\
% Suppose $\bfX$ contains all unobserved variables. The true posterior distribution can be written as:
% \[
% p(\bfX) = \frac{f(\bfX)}{Z}
% \]
% where $f(\bfX)$ is a function that depends on the unobserved variables and $Z$ is an unknown normalisation constant to ensure that $\int p(\bfX) d\bfX = 1$.

% The second-order Taylor expansion of $\log f(\bfX)$ centered around its (known) mode $\hat{\bfX}$ is:
% \[
% 	\log f(\bfX) \approx \log f(\hat{\bfX}) - \frac{1}{2} (\bfX-\hat{\bfX})^T \bfA (\bfX-\hat{\bfX})
% \]
% where $\bfA = \nabla^2 \log f(\hat{\bfX})$ is the Hessian matrix of $\log f(\bfX)$ evaluated at $\hat{\bfX}$.\\
% Notice three things. First, the first-order term of the Taylor expansion is zero because $\hat{\bfX}$ is a stationary point. Second, the $\log$ function is monotonically increasing and therefore a maximum of $\log f(\bfX)$ is also a maximum of $f(\bfX)$. Third, the mode of the posterior $p(\bfX)$ must be known, which requires the use of (complex) optimisation algorithms.\\
% Taking the exponential in both sides:
% \[
% 	f(\bfX) \approx f(\hat{\bfX}) \exp\{ -\frac{1}{2} (\bfX-\hat{\bfX})^T \bfA (\bfX-\hat{\bfX}) \}
% \]
% which leads to the following multivariate Gaussian distribution approximation $q(\bfX) = \Ndist{\bfX}{\hat{\bfX},\bfA}$:
% \[
% 	q(\bfX) = \frac{\mid A \mid^{1/2}}{(2\pi^{d/2})} \exp\{ -\frac{1}{2} (\bfX-\hat{\bfX})^T \bfA (\bfX-\hat{\bfX}) \}
% \]
% where $d$ is the number of unobserved variables. \\
% Despite its simplicity, the Laplace approximation is a useful strategy that has been successfully applied in practice. Nonetheless, this approximation has notable caveats: first, is limited by its own local definition, ignoring all the density beyond the mode of the posterior. Second, it does not apply to discrete variables. Third, the inversion of the Hessian is very expensive in high-dimensional settings.
% 	% More sophisticated generalisations of the Laplace approximation have also been proposed \cite{Rue2009}
% 	% To-do: check where does the inverse appear

\subsection{Variational inference}  \label{section:variational_inference}

Variational inference is a deterministic family of methods that have been receiving widespread attention due to a positive balance between accuracy, speed, and ease of use \cite{Blei2016, Zhang2017}. The core framework is derived below.

In variational inference the true (but intractable) posterior distribution $p(\bfX|\bfY)$ is approximated by a simpler (variational) distribution $q(\bfX|\bTheta)$ where $\bTheta$ are the corresponding parameters. The parameters, which we will omit from the notation, need to be tuned to obtain the closest approximation to the true posterior.\\
The distance between the true distribution and the variational distribution is calculated using the KL divergence:
\[
\KL(q(\bfX)||p(\bfX|\bfY)) = - \int q(\bfX) \log \frac{p(\bfX|\bfY)}{q(\bfX)} d\bfX
\]
Note that the KL divergence is not a proper distance metric, as it is not symmetric. In fact, using the reverse KL divergence $\KL(q(\bfX)||p(\bfY|\bfX))$ defines a different inference framework called expectation propagation \cite{Minka2001}.

If we allow any possible choice of $q(\bfX)$, then the minimum of this function occurs when $q(\bfX)$ equals the true posterior distribution $p(\bfX|\bfY)$. Nevertheless, since the true posterior is intractable to compute, this does not lead to any simplification of the problem. Instead, it is necessary to consider a restricted family of distributions $q(\bfX)$ that are tractable to compute and subsequently seek the member of this family for which the KL divergence is minimised.

Doing some calculus it can be shown (see \cite{Bishop2006,Murphy}) that the KL divergence $\KL(q(\bfX)||p(\bfX|\bfY))$ is the difference between the log of the marginal probability of the observations $\log(\bfY)$ and a term $\Lagr(\bfX)$ that is typically called the Evidence Lower Bound (ELBO):
\[
	\KL(q(\bfX)||p(\bfX|\bfY)) = \log p(\bfY) - \Lagr(\bfX)
\]
Hence, minimising the KL divergence is equivalent to maximising $\Lagr(\bfX)$ (\Cref{fig:ELBO}):
\begin{align} \label{eq_elbo1} \begin{split}
	\Lagr(\bfX) &= \int q(\bfX) \Big( \log \frac{p(\bfX|\bfY)}{q(\bfX)} + \log p(\bfY) \Big) d\bfX \\
	%&= \int \Big( q(\bfX) \log \frac{p(\bfX|\bfY)}{q(\bfX)} + q(\bfX)\log p(\bfY) \Big) d\bfX\\
	%&= \E_q [\log p(\bfX|\bfY)] - \E_q [\log q(\bfX)] + \E_q [\log p(\bfY)] \\
	&= \E_q [\log p(\bfX,\bfY)] - \E_q [\log q(\bfX)]
\end{split} \end{align}
The first term is the expectation of the log joint probability distribution with respect to the variational distribution. The second term is the entropy of the variational distribution.
Importantly, given a simple parametric form of $q(\bfX)$, each of the terms in \Cref{eq_elbo1} can be computed in closed form. In some occasions, we will use the following form for the ELBO:
\begin{equation} \label{eq_elbo2}
	\Lagr(\bfX) = \E_q [\log p(\bfY|\bfX)] + (\E_q [\log p(\bfX)] - \E_q [\log q(\bfX)])
\end{equation}
where the first term is the expectation of the log likelihood and the second term is the difference in the expectations of the $p$ and $q$ distributions of each hidden variable.

In conclusion, variational learning involves minimising the KL divergence between $q(\bfX)$ and $p(\bfX|\bfY)$ by instead maximising $\Lagr(\bfX)$ with respect to the distribution $q(\bfX)$.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.35\linewidth]{lower_bound}
	\caption{The quantity $\Lagr(\bfX)$ provides a lower bound on the true log marginal likelihood $\log p(\bfY)$, with the difference being given by the Kullback-Leibler divergence $\KL(q||p)$ between the variational distribution $q(\bfX)$ and the true posterior $p(\bfX|\bfY)$}
	\label{fig:ELBO}
\end{figure}

There are several approaches to define $q(\bfX)$, the two most commonly used are called (unparametric) mean-field and (parametric) fixed-form \cite{Zhang2017,Blei2016}.

\subsubsection{Mean-field variational inference}  \label{section:mean_field}

The most common type of variational Bayes, known as the mean-field approach, assumes that the variational distribution factorises over M disjoint groups of unobserved variables \cite{Saul1996}:
\begin{equation} \label{eq:mean_field}
	q(\bfX) = \prod_{i=1}^{M} q(\bfx_i)
\end{equation}
where typically all unobserved variables are assumed to be independent. Importantly, notice that no parametric assumptions were placed regarding the nature of $q(\bfx_i)$.

Evidently, in sufficiently complex models where the unobserved variables have dependencies this family of distributions do not contain the true posterior (\Cref{fig:mean_field}). Yet, this is a key assumption to obtain an analytical inference scheme that yields surprisingly accurate results \cite{Blei2006,Faes2011,Braun2007}.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.7\linewidth]{mean_field}
	\caption{Illustrative example of sampling from a true posterior distribution (blue) versus a fitted mean-field varaitional distribution (red) in a model with two (correlated) unobserved variables. The mean-field approximation wrongly assumes that the unobserved variables are independent.}
	\label{fig:mean_field}
\end{figure}

Using calculus of variations (derivations can be found in \cite{Bishop2006,Murphy}), it follows that the optimal distribution $q(\bfX)$ that maximises the lower bound $\Lagr(\bfX)$ is
\begin{equation} \label{eq:optimal}
	\log \hat{q}_i(\bfx_i) = \E_{-i} [\log p(\bfY,\bfX)] + \mathrm{const}
\end{equation}
where $\E_{-i}$ denotes an expectation with respect to the $q$ distributions over all variables $\bfx_j$ except for $\bfx_i$.\\
The additive constant is set by normalising the distribution $\hat{q}_i(\bfz_i)$:
\[
	\hat{q}(\bfx_i) = \frac{\exp(\E_{-i}[\log p(\bfY,\bfX)])}{\int \exp(\E_{-i}[\log p(\bfY,\bfX)]) d\bfX}
\]
While the form of $\hat{q}(\bfx_i)$ is not restricted to a specific parametric form, it can be shown that when using conjugate priors, the distributions $\hat{q}_i(\bfx_i)$ have the same functional form as the priors $\hat{p}(\bfx_i)$.
%An example is shown in Appendix X, but a detailed mathematical treatment with derivations of multiple examples can be found in \cite{Bishop2006,Murphy,Zhao2009}.

\subsubsection{Fixed-form variational inference}  \label{section:fixed_form}

An alternative straightforward strategy is to directly define a parametric form for the distribution $q(\bfX)$ with some parameters $\bTheta$. Once the choice of $q(\bfX)$ is made, the parameters $\bTheta$ are optimised to minimise $\KL(q(\bfX)||p(\bfX|\bfY))$ (the variational problem):
\begin{align}
	\hat{\bTheta} &= \argmin_{\bTheta} \KL(q(\bfX)||p(\bfX|\bfY)) \\
	&= \E[\log(q(\bfX)) - \log(p(\bfX,\bfY))]
\end{align}
Numerically optimising this function requires the evaluation of expectations with respect to $q(\bfX)$. In closed form, this is only feasable for a limited group of variational distributions. Alternatively, one can attempt Monte Carlo approximations, but in practice this turns to be slow and leads to high-variance estimates \cite{Braun2007,Ranganath2014,Braun2007}.

Typically, one would choose this distribution to factorise over parameters and to be of the same (exponential) family as the prior $p(\bfX)$. In such case there is a closed form coordinate-ascent scheme available, and it turns out that the fixed-form formulation is equivalent to the (non-parametric) mean-field derivation when using conjugate priors.\\
Unfortunately, for generic models with arbitrary families of distributions, no closed-form variational distributions exist \cite{Zhang2017,Blei2016}.

However, while the parametric assumption certainly limits the flexibility of variational distributions, the advantage of this formulation is that it opens the possibility to use fast gradient-based methods for the inference procedure \cite{Hoffman2012,Ranganath2014}.

%\paragraph{Black box variational inference}

\subsection{Expectation Propagation}  \label{section:expectation_propagation}

Expectation Propagation (EP) is another deterministic strategy with a similar philosophy as the Variational approach. It is also based on minimising the KL divergence between a variational distribution $q(\bfX)$ and the true posterior $p(\bfX|\bfY)$, but while variational inference minimises $KL(p||q)$, EP maximises the reverse KL-divergence $KL(q||p)$.

Interestingly, this simple difference leads to an inference scheme with strikingly different properties. This can be understood by inspecting the differences between the two KL divergence formulas:

Variational inference:
\begin{equation} \label{eq:kl_vb}
	\KL(q(\bfX)||p(\bfX|\bfY)) = - \int q(\bfX) \log \frac{p(\bfX|\bfY)}{q(\bfX)}
\end{equation}
Expectation propagation:
\begin{equation} \label{eq:kl_ep}
	\KL(p(\bfX|\bfY)||q(\bfX)) = - \int p(\bfX|\bfY) \log \frac{q(\bfX)}{p(\bfX|\bfY)}
\end{equation}
In regions of $\bfX$ where the true posterior density $p(\bfX|\bfY)$ is small, setting a large density for $q(\bfX)$ has a much stronger penalisation in \Cref{eq:kl_ep} than in \Cref{eq:kl_vb}. Hence, EP tends to avoid areas where the density $p(\bfX|\bfY)$ is very low, even if it does not correspond to areas of very high-density in $p(\bfX|\bfY)$. In contrast, in \Cref{eq:kl_vb} there is a strong penalty for having low-density $q(\bfX)$ values.\\
As discussed in \cite{Bishop2006}, the practical consequences of this duality can be observed when the posterior is multi-modal. In VI, $q(\bfX)$ converges towards areas of high-density in $p(\bfX|\bfY)$, namely local optima. In contrast, EP tends to capture as much non-zero density regions from $p(\bfX|\bfY)$ as possible, thereby averaging across all optima. In the context of doing predictions, the VI solution is much more desirable than the EP solution, as the average of two good parameter values is not necessarily a good value itself.\\
A detailed mathematical treatment of EP, including derivations for specific examples, can be found in \cite{Bishop2006,Murphy,Minka2001}

\begin{figure}[H]
	\centering
	\includegraphics[width=1.0\linewidth]{VB_vs_EP}
	\caption{Illustrative comparison of Variational inference and Expectation Propagation. Shown is the (a) Density and (b) Variance of the true posterior distribution $p(\bfX|\bfY)$ (grey), the variational distribution (orange) and the expectation propagation distribution (green).}
	\label{}
\end{figure}

\subsection{Conclusions}

In this section we have introduced Bayesian modelling and variational inference methods, which will be used later in this chapter.\\
More generally, variational inference is growing in popularity for the analysis of big datasets and it has been applied to a myriad of different problems, including genome-wide association studies \cite{Carbonetto2012}, population genetics, \cite{Raj2014}, network analysis \cite{Sanguinetti2006} and natural language processing \cite{Blei2003}.

Yet, despite its increasing success, there is significant room for improvement. First and foremost, the theoretical guarantees of variational inference are not as developed as in sampling-based MCMC schemes \cite{Blei2016,Zhang2017,Nakajima2007}. As an example, the mean-field setting makes strong independence assumptions about the parameters.  Although it tends to be surprisingly effective, it is not clear in which applications the dependencies between the parameters are important enough that the mean-field approximation could potentially break.\\
More generally, an open research problem is understanding what are the statistical properties of the variational posterior with respect to the exact posterior \cite{Blei2016,Zhang2017}.

As we shall demonstrate later, alternative strategies have been considered to allow some dependencies between the variables, resulting in \textit{structured} mean-field approximations \cite{Hoffman2014,Titsias2011}. However, they often lead to very complex (if not intractable) inference frameworks.

Finally, another area of extensive research is how to extend the applicability of VI to non-conjugate models. As discussed in \Cref{section:deterministic_bayesian_inference}, the ELBO of non-conjugate models contains intractable integrals, and setting up an inference scheme requires the use of either stochastic Monte Carlo approximations or deterministic approximations that introduce additional lower bounds \cite{Zhang2017,Seeger2012,Khan2017}. In this thesis we follow this rationale to derive an inference framework for a model with non-Gaussian likelihoods.


\subsection{Fourier transform of polynomials}
We begin by noting that an activation
function $\sigma$, which satisfies a polynomial growth condition
$|\sigma(x)| \leq C(1 + |x|)^n$ for some constants $C$ and $n$, is a
tempered distribution. As a result, we make this assumption on our
activation functions in the following theorems. We briefly note that
this condition is sufficient, but not necessary (for instance an
integrable function need not satisfy a pointwise polynomial growth
bound) for $\sigma$ to be represent a tempered distribution.

 We begin by studying the convolution of $\sigma$ with a Gaussian mollifier. Let $\eta$ be a Gaussian mollifier
 \begin{equation}
  \eta(x) = \frac{1}{\sqrt{\pi}}e^{-x^2}.
 \end{equation}
Set $\eta_\epsilon=\frac{1}{\epsilon}\eta(\frac{x}{\epsilon})$. Then consider
\begin{equation}
\label{sigma-epsilon}
\sigma_{\epsilon}(x):=\sigma\ast{\eta_\epsilon}(x)=\int_{\mathbb{R}}\sigma(x-y){\eta_\epsilon}(y)dy
\end{equation}
for a given activation function $\sigma$.
It is clear that $\sigma_{\epsilon}\in C^\infty(\mathbb{R})$. Moreover, by considering the Fourier transform (as a tempered
distribution) we see that
\begin{equation}\label{eq_278}
 \hat{\sigma}_{\epsilon} = \hat{\sigma}\hat{\eta}_{\epsilon} = \hat{\sigma}\eta_{\epsilon^{-1}}.
\end{equation}

We begin by stating a lemma which characterizes the set of polynomials in terms of their
 Fourier transform.
\begin{lemma}\label{polynomial_lemma} Given a tempered distribution
  $\sigma$,  the following statements are equivalent:
\begin{enumerate}
\item $\sigma$ is a polynomial
\item $\sigma_\epsilon$ given by \eqref{sigma-epsilon} is a polynomial for any
  $\epsilon>0$.
\item $\text{\normalfont supp}(\hat{\sigma})\subset \{0\}$.
\end{enumerate}
\end{lemma}
\begin{proof}
  We begin by proving that (3) and (1) are equivalent.  This follows
  from a characterization of distributions supported at a single point
  (see \cite{strichartz2003guide}, section 6.3). In particular, a
  distribution supported at $0$ must be a finite linear combination of
  Dirac masses and their derivatives.  In particular, if
  $\hat{\sigma}$ is supported at $0$, then
  \begin{equation}
   \hat{\sigma} = \displaystyle\sum_{i=1}^n a_i\delta^{(i)}.
  \end{equation}
  Taking the inverse Fourier transform and noting that the inverse
  Fourier transform of $\delta^{(i)}$ is $c_ix^i$, we see that
  $\sigma$ is a polynomial. This shows that (3) implies (1), for the
  converse we simply take the Fourier transform of a polynomial and
  note that it is a finite linear combination of Dirac masses and
  their derivatives.

  Finally, we prove the equivalence of (2) and (3). For this it
  suffices to show that $\hat{\sigma}$ is supported at $0$ iff
  $\hat{\sigma}_\epsilon$ is supported at $0$. This follows from
  equation \ref{eq_278} and the fact that $\eta_{\epsilon^{-1}}$ is
  nowhere vanishing.
\end{proof}

As an application of Lemma \ref{polynomial_lemma}, we give a
simple proof of the result in the next section.

497-6DL/7 Deep Neural network Functions/7.4-DNN_Qualitative.tex


\subsection{Subgroups}

A subgroup of a group is a subset of a group, which also forms a group with the same element.

For example all even numbers are a subgroup of the addition group of integers.


\subsection{The Naive Bayes classifier}

\subsubsection{Calculating the Naive Bayes estimator}

With the Naive Bayes assumption we have:

\(P(y|x_1,x_2,...,x_n)\propto P(x_1|y)P(x_2|y)...P(x_n|y)P(y)\)

We now choose \(y\) which maximises this.

This is easier to calculate, as there is less of a sample restriction.

This is used when evidence is also in classes, as the chance of any individual outcome on a continuous probability is \(0\).

\subsubsection{Estimating \(P(y)\)}

We can easily calculate \(P(y)\), by looking at the frequency across the sample.

\subsubsection{Estimating \(P(x_1|y)\)}

Normally, \(P(x_1|y)=\dfrac{n_c}{n_y}\), where:

\begin{itemize}
\item \(n_c\) is the number of instances where the evidence is \(c\) and the label is \(y\).
\item \(n_y\) is the number of instances where the label is \(y\).
\end{itemize}

\subsubsection{Regularising the Naive Bayes estimator}

To reduce the risk of specific probabilities being zero, we can adjust them, so that:

\(P(x_1|y)=\dfrac{n_c+mp}{n_y+m}\), where:

\begin{itemize}
\item \(p\) is the prior probability. If this is unknown, use \(\dfrac{1}{k}\), where \(k\) is the number of classes.
\item \(m\) is a parameter called the equivilant sample size.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 8}

Prove that any two nodes $x$ and $y$ in a rooted tree have a unique \textbf{least common ancestor} $z$.
This simply means that there is a path $P_1$ from $z$ to $x$, and a path $P_2$ from $z$ to $y$, and that the only vertex those two paths have in common is $z$.
(And further that there is exactly one node $z$ with this property.)

\subsection*{Solution}

Proof is given in two steps.
First we show that any two distinct non-root nodes $x$ and $y$ in a tree have a least common ancestor $z$.
Then, we will show that this least common ancestor is unique.
\begin{enumerate}
\item[] If $x$ and $y$ are two distinct non-root nodes, there exists paths of the form $r \rightarrow \cdots \rightarrow x$ and $r \rightarrow \cdots \rightarrow y$ where $r$ is the root of the tree.
Even if there are no node in common between the two paths, node $r$ would be a common ancestor to $x$ and $y$.
On the other hand if the two paths have a node $z$ in common, paths $P_1$ and $P_2$ would exists from $z$ to $x$ and $y$ respectively where $P_1$ and $P_2$ have no nodes in common except $z$.
In this case $z$ would be the least common ancestor of $x$ and $y$.
\item[] Now we prove by contradiction that there is exactly one least common ancestor for nodes $x$ and $y$.
Assume there are moer than one least common ancestors.
Then there are at least two distinct nodes $z_1$ and $z_2$ with descendants $x$ and $y$.
This immediately leads to a simple loop $z_1 \rightarrow \cdots \rightarrow x \rightarrow \cdots \rightarrow z_2 \rightarrow \cdots \rightarrow y \rightarrow \cdots \rightarrow z_1$ which contradicts the definition of the tree.
Therefore the assumption is false and least common ancestor is unique.
\end{enumerate}


\chapter{Matrix-Chain Multiplication Algorithm}
The objective of this challenge is to calculate the minimum amount of scalar
multiplications a group of matrices needs to return the complete and correct
matrix product. Consider three matrices $\mathbf{A}[10][100]$, $\mathbf{B}[100]
[5]$, and $\mathbf{C}[5][50]$. There are two ways to multiply these matrices:
\begin{enumerate}
\item $(\mathbf{A}\times\mathbf{B})\times\mathbf{C}$: $\mathbf{A}\times
    \mathbf{B}$ requires $10\times100\times5=5000$ scalar multiplications.
    $\mathbf{AB}\times\mathbf{C}$ requires $10\times5\times50=2500$. Therefore
    the total amount of scalar multiplications is $5000+2500=7500$.
\item $\mathbf{A}\times(\mathbf{B}\times\mathbf{C})$: $\mathbf{B}\times
    \mathbf{C}$ requires $100\times5\times50=25000$ scalar multiplications.
    $\mathbf{A}\times\mathbf{BC}$ requires $10\times100\times50=50000$.
    Therefore the total amount of scalar multiplications is $25000+50000=75000$.
\end{enumerate}

Given a chain of $n$ matrices and an integer array of $n+1$ elements where each
matrix $\mathbf{A_i}$ shows the dimesion of $\left(p_{i-1},~p_i\right)$, the
following solutions are algorithms that minimise scalar multiplication amounts.

\section{Optimal Solution}
\imb[4][26]{\alg/matrixchain.cpp}

Matrix chain multiplication has $T(n)=\Theta\left(n^2\right)$ subproblems, each
featuring no more than $n-1$ choices. Therefore the time complexity of this
algorithm is: \[ T(n)=\Theta\left(n^3\right) \]


\chapter{The Basics of Maximum Likelihood Estimation \label{chapter:mlebasics}}

Beneath our discussions of classification, regression, and probability distributions in Chapters~\ref{chapter:classification}, \ref{chapter:regression}, and \ref{chapter:probabilitydistributions} lies the tricky problem of \textbf{model fitting}. We've seen what classification and regression models look like, but we still haven't addressed how to fit these models using training data.

Linear and logistic regression models are fit using a technique called \textbf{maximum likelihood (ML) estimation}, in which the model parameters are adjusted to maximize the joint probability of the observed data, or likelihood, given the model.

For example, consider the five different datasets from Question~\ref{question:likexamples}. In each case, you have some data and an assumption about which probability distribution the data are drawn from. The job of maximum likelihood estimation is to use the data to identify the correct distributional parameters, such as $\mu$ and $\sigma$ (in the case of the normal distribution) or $\lambda$ (in the case of the Poisson distribution). This process is a type of \textbf{statistical inference}.

\section{The Likelihood and Log-Likelihood}

Let $p(x|\theta)$ be the probability distribution that governs our data. Here, $\theta$ stands in for all of the parameters we want to fit.

If we draw independent\footnote{Independent sampling just means that the values of different samples do not depend on each other. When the samples are drawn independently from the same distribution, their joint probability density is just the product of the individual probability densities (which are all the same).} samples from $p(x|\theta)$, the {\bf joint probability density function} for all $n$ observations is:
$$ p(x^{(1)}, x^{(2)}, \dots, x^{(n)}|\theta) = \prod_{i=1}^n p(x^{(i)}|\theta). $$
Since the data are known but the parameter(s) $\theta$ are unknown, we will view this quantity as a function of $\theta$. This is just a change in notation:
$$ \mathcal{L}(\theta) = \prod_{i=1}^n p(x^{(i)}|\theta). $$

The higher the joint probability of the data (the more ``likely'' the data are) given $\theta$, the higher the value of this function. We call $\mathcal{L}(\theta)$ the \textbf{likelihood}\footnote{The distributions we have discussed so far are from a broad family of probability distributions called the {\bf exponential family}. One of the properties of this family is that the log-likelihood is concave. Practically speaking, this means that if we maximize the log-likelihood by setting derivatives equal to zero, we are guaranteed to (a) get only one solution, and (b) find a maximum (not a minimum or an inflection point).}. Frequently we will want to work with the logarithm of the likelihood, which we call the \textbf{log-likelihood}, because it has some nice properties, including allowing us to manipulate sums instead of products\footnote{Note that if the function $f(z)$ has a maximum at $z'$, the function $\log f(z)$ will also have a maximum at $z'$, because the logarithmic function is monotonically increasing. So we will get the same parameter estimate(s) either way.}:
$$ \log \mathcal{L}(\theta) = \sum_{i=1}^n \log p(x^{(i)}|\theta). $$

In maximum likelihood estimation, we seek to find the $\theta$ for which the likelihood (or log-likelihood) is maximized. We do this by taking derivatives of the log-likelihood with respect to the various parameters and setting them equal to zero. The best-fit parameter estimates obtained in this way are called the \textbf{maximum likelihood estimates (MLEs)}.
\vspace{5mm}

\begin{question}{}
What are some reasons why we might want to fit data to a probability distribution?
% Data summarization (don't need to keep entire dataset around)
% Sampling
% Many common models and hypothesis tests rely on parametric assumptions (T-tests, etc.)
% We are less likely to overfit to particularities of our own datasets, such as outliers (KNN, etc.)
% Detecting outliers or faulty assumptions
\end{question}

\section{Example: Fitting Data to a Normal Distribution}

Imagine you have some data from a lab test that measures the concentration of a particular biomarker. You have data from $100$ different subjects. A histogram of the raw data looks like this:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/normal-likelihood-example-data.png}
\end{center}

You want to find the normal distribution that best describes these data so you can create a reference distribution for this lab test. To do this, think about trying out several distributions with different values of $\mu$ and $\sigma$ and choosing the one that maximizes the log-likelihood. For example, here are three different normal distributions with different values of $\mu$ and $\sigma=0.7$:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/normal-likelihood-example-data-normal.png}
\end{center}
Here is what happens to the log-likelihood as you vary $\mu$. The log-likelihoods of the three distributions shown in the plot above are shown as dots with their corresponding colors, and the maximum likelihood estimate is shown as a vertical dotted line.
\begin{center}
\includegraphics[width=0.7\textwidth]{img/normal-likelihood-example-vary-mu.png}
\end{center}
Now, here's what happens to the log-likelihood when we vary $\sigma$, keeping $\mu$ fixed at its maximum likelihood estimate from the graph above. Again, the maximum likelihood estimate is shown as a vertical dotted line.
\begin{center}
\includegraphics[width=0.7\textwidth]{img/normal-likelihood-example-vary-sigma.png}
\end{center}
For the record, I simulated these data from a normal distribution with $\mu=2.5$ and $\sigma=0.75$. The maximum likelihood estimates obtained from this dataset are $\hat{\mu} = 2.45$ and $\hat{\sigma} = 0.74$.

\section{Analytical Calculations of MLEs}

In some simple cases, the MLEs can be calculated analytically. We will now go through a bunch of examples of how to find the MLEs of the probability distributions we saw in Chapter~\ref{chapter:probabilitydistributions}.

\subsection{Bernoulli Distribution}

The Bernoulli distribution is described in Section~\ref{sect:bernoulli}. Our goal is to find the parameter, $\mu$, of this distribution, given some observed data, $x^{(1)}, \dots, x^{(n)}$. The data will consist of a list of 1s and 0s, since Bernoulli random variables can only take the values 0 or 1.

To find $\hat{\mu}$, our MLE for $\mu$, we first write down the log-likelihood:
\begin{align*}
\log \mathcal{L}(\mu) &= \sum_{i=1}^n \log p(x^{(i)}|\mu) \\
&= \sum_{i=1}^n \log \left( \mu^{x^{(i)}}(1-\mu)^{1-x^{(i)}} \right) \\
&= \sum_{i=1}^n \left[ x^{(i)} \log(\mu) + (1-x^{(i)}) \log(1-\mu) \right] \end{align*}
Then we take the derivative of the log-likelihood with respect to $\mu$:
\begin{align*}
\frac{d}{d \mu} \log \mathcal{L}(\mu) = \sum_{i=1}^n \left[ \frac{x^{(i)}}{\mu} - \frac{1-x^{(i)}}{1-\mu} \right]
\end{align*}
The MLE of $\mu$ will occur when the likelihood is maximized, which happens when the first derivative equals zero. So to solve for $\hat{\mu}$, we set the derivative equal to zero and rearrange:
\begin{align*} \sum_{i=1}^n \left[ \frac{x^{(i)}}{\hat{\mu}} - \frac{1-x^{(i)}}{1-\hat{\mu}} \right] = 0 & \implies (1 - \hat{\mu}) \sum_{i=1}^n x^{(i)} = \hat{\mu} \sum_{i=1}^n (1 - x^{(i)}) \\
& \implies \boxed{\hat{\mu} = \frac{1}{n} \sum_{i=1}^n x^{(i)}} \end{align*}
We see that the MLE, $\hat{\mu}$, is simply the sum of our data -- i.e. the number of data points where the outcome is 1 -- divided by the total number of observations.

This makes sense: if you want to know the probability that a coin will come up heads, a good way to estimate it is to flip the coin a bunch of times and calculate the fraction of observations in which the coin comes up heads.

\subsection{Binomial Distribution}

The binomial distribution is described in Section~\ref{sect:binomial}. We will make one notational change from that section, which is to call the number of Bernoulli trials $m$ instead of $n$, since we are using $n$ to refer to the number of data samples. To keep things simple, we will assume that $m$ is a known quantity. As before, we first write down the log-likelihood:
\begin{align*}
\log \mathcal{L}(\mu) &= \sum_{i=1}^n \log p(x^{(i)}|m,\mu) \\
&= \sum_{i=1}^n \log \left[ {m\choose x} \mu^x (1 - \mu) ^ {m-x} \right] \\
&= \sum_{i=1}^n \left[\log(m!) - \log(x!) - \log((m-x)!) + x^{(i)} \log(\mu) + (m-x^{(i)}) \log(1-\mu) \right] \end{align*}
Then we take the derivative of the log-likelihood with respect to $\mu$:
\begin{align*}
\frac{d}{d \mu} \log \mathcal{L}(\mu) = \sum_{i=1}^n \left[ \frac{x^{(i)}}{\mu} - \frac{m-x^{(i)}}{1-\mu} \right]
\end{align*}
We set this equal to zero and solve for $\hat{\mu}$ (the maximum likelihood estimate of $\mu$):
\begin{align*} \sum_{i=1}^n \left[ \frac{x^{(i)}}{\hat{\mu}} - \frac{m-x^{(i)}}{1-\hat{\mu}} \right] = 0 & \implies (1 - \hat{\mu}) \sum_{i=1}^n x^{(i)} = \hat{\mu} \sum_{i=1}^n (m - x^{(i)}) \\
& \implies \boxed{\hat{\mu} = \frac{1}{nm} \sum_{i=1}^n x^{(i)}} \end{align*}

\begin{question}{}
Interpret the MLE for the parameter, $\mu$, of a binomial distribution, assuming fixed $m$ (number of trials). Does the MLE for $\mu$ make intuitive sense to you? Think through a few of your examples from Question~\ref{question:binomialex}.
\end{question}

\subsection{Normal Distribution \label{sect:mlenormal}}

The normal distribution is described in Section~\ref{sect:normal}. We will follow the same procedure as in the previous two sections, except that now we have two parameters to solve for, $\mu$ and $\sigma$, instead of one. First, we write down the log-likelihood:
\begin{align*}
\log \mathcal{L}(\mu, \sigma) &= \sum_{i=1}^n \log p(x^{(i)}|\mu, \sigma) \\
&= \sum_{i=1}^n \log \left( \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{(x^{(i)}-\mu)^2}{2 \sigma^2}} \right) \\
&= -\frac{n}{2} \log (2 \pi) - \frac{n}{2} \log \sigma^2 - \frac{1}{2 \sigma^2} \sum_{i=1}^n \left( x^{(i)} - \mu \right)^2 \\
\end{align*}
To find the MLE for $\mu$, we take the derivative of the log-likelihood with respect to $\mu$:
\begin{align*}
\frac{\partial}{\partial \mu} \log \mathcal{L}(\mu, \sigma) = \frac{1}{\sigma^2} \sum_{i=1}^n \left( x^{(i)} - \mu \right)
\end{align*}
We set this equal to zero and solve for $\hat{\mu}$ (the maximum likelihood estimate of $\mu$):
\begin{align*} \frac{1}{\sigma^2} \sum_{i=1}^n \left( x^{(i)} - \mu \right) = 0 & \implies \boxed{\hat{\mu} = \frac{1}{n} \sum_{i=1}^n x^{(i)}} \end{align*}
To find the MLE for $\sigma$, we then take the derivative of the log-likelihood with respect to $\sigma$:
\begin{align*} \frac{\partial}{\partial \sigma} \log \mathcal{L}(\mu, \sigma) &= -\frac{n}{\sigma} + \frac{1}{\sigma^3} \sum_{i=1}^n \left( x^{(i)} - \mu \right)^2
\end{align*}
We set this equal to zero and solve for $\hat{\sigma}$ (the maximum likelihood estimate of $\sigma$)\footnote{One detail: it turns out this estimate is biased because it depends on the MLE for $\mu$. An unbiased version has $n-1$ in the denominator instead of $n$. The effect of this is minimal unless $n$ is small.}. Note that the answer depends on our previously calculated MLE for $\mu$:
\begin{align*}
-\frac{n}{\hat{\sigma}} + \frac{1}{\hat{\sigma}^3} \sum_{i=1}^n \left( x^{(i)} - \mu \right)^2 = 0  & \implies \boxed{\hat{\sigma} = \sqrt{\frac{1}{n} \sum_{i=1}^n \left( x^{(i)} - \hat{\mu} \right)^2}}
\end{align*}

\begin{question}{}
Interpret the MLEs for the parameters, $\mu$ and $\sigma$, of a normal distribution. Do these results make intuitive sense to you? Think through a few of your examples from Question~\ref{question:normalex}.
\end{question}

\subsection{Poisson Distribution}

The Poisson distribution is described in Section~\ref{sect:poisson}. To find the MLE for $\lambda$, its mean, we first (as usual) write down the log-likelihood:
\begin{align*}
\log \mathcal{L}(\lambda) &= \sum_{i=1}^n \log p(x^{(i)}|\lambda) \\
&= \sum_{i=1}^n \log \left( \frac{e^{-\lambda} \lambda^{x^{(i)}}}{x^{(i)}!} \right) \\
&= \sum_{i=1}^n \left[ -\lambda + x^{(i)} \log(\lambda) - \log(x^{(i)}!) \right] \end{align*}
Now we take the derivative of the log-likelihood with respect to $\lambda$:
\begin{align*}
\frac{d}{d \lambda} \log \mathcal{L}(\lambda) = \sum_{i=1}^n \left[ -1 + \frac{x^{(i)}}{\lambda} \right]
\end{align*}
We set this equal to zero and solve for $\hat{\lambda}$ (the maximum likelihood estimate of $\lambda$):
\begin{align*} \sum_{i=1}^n \left[ -1 + \frac{x^{(i)}}{\hat{\lambda}} \right] = 0 & \implies \boxed{\hat{\lambda} = \frac{1}{n} \sum_{i=1}^n x^{(i)}} \end{align*}

\begin{question}{}
Interpret the MLE for the parameter, $\lambda$, of a Poisson distribution. Does this result make intuitive sense to you? Think through a few of your examples from Question~\ref{question:poissonex}.
\end{question}

\subsection{Geometric Distribution}

The geometric distribution is described in Section~\ref{sect:geometric}. To find the MLE for $\mu$, we first write down the log-likelihood:
\begin{align*}
\log \mathcal{L}(\mu) &= \sum_{i=1}^n \log p(x^{(i)}|\mu) \\
&= \sum_{i=1}^n \log \left( (1-\mu)^{x^{(i)}} \mu \right) \\
&= \sum_{i=1}^n \left[ x^{(i)} \log(1-\mu) + \log(\mu) \right] \end{align*}
Now we take the derivative of the log-likelihood with respect to $\mu$:
\begin{align*}
\frac{d}{d \mu} \log \mathcal{L}(\mu) = \sum_{i=1}^n \left[ -\frac{x^{(i)}}{1-\mu} + \frac{1}{\mu} \right]
\end{align*}
We set this equal to zero and solve for $\hat{\mu}$ (the maximum likelihood estimate of $\mu$):
\begin{align*} \sum_{i=1}^n \left[ -\frac{x^{(i)}}{1-\hat{\mu}} + \frac{1}{\hat{\mu}} \right] = 0 & \implies \frac{n}{\hat{\mu}} = \frac{1}{1 - \hat{\mu}} \sum_{i=1}^n x^{(i)} \\
& \implies \boxed{\hat{\mu} = \frac{n}{\sum_{i=1}^n (x^{(i)}+1)} }\end{align*}

\begin{question}{}
Interpret the MLE for the parameter, $\mu$, of a geometric distribution. Does this result make intuitive sense to you? Think through a few of your examples from Question~\ref{question:geometricex}.
\end{question}

\subsection{Exponential Distribution}

The exponential distribution is described in Section~\ref{sect:exponential}. To find the MLE for $\lambda$, we first write down the log-likelihood:
\begin{align*}
\log \mathcal{L}(\lambda) &= \sum_{i=1}^n \log p(x^{(i)}|\lambda) \\
&= \sum_{i=1}^n \log \left( \lambda e^{-\lambda x^{(i)}} \right) \\
&= \sum_{i=1}^n \left[\log(\lambda) - \lambda x^{(i)} \right] \end{align*}
Now we take the derivative of the log-likelihood with respect to $\lambda$:
\begin{align*}
\frac{d}{d \lambda} \log \mathcal{L}(\lambda) = \sum_{i=1}^n \left[ \frac{1}{\lambda} - x^{(i)} \right]
\end{align*}
We set this equal to zero and solve for $\hat{\lambda}$ (the maximum likelihood estimate of $\lambda$):
\begin{align*} \sum_{i=1}^n \left[ \frac{1}{\hat{\lambda}} - x^{(i)} \right] = 0 & \implies \boxed{\hat{\lambda} = \frac{n}{\sum_{i=1}^n x^{(i)}}} \end{align*}

\begin{question}{}
Interpret the MLE for the parameter, $\lambda$, of an exponential distribution. Does this result make intuitive sense to you? Think through a few of your examples from Question~\ref{question:exponentialex}.
\end{question}

\section{Summary of MLEs for Common Distributions}

The table below contains a summary of the MLEs of various parameters from some common probability distributions.

\begin{center} {\small
\begin{tabular}{lccc}
\toprule
Distribution & Parameter & ML Estimate & Domain of $x^{(i)}$ \\
\midrule
Univariate Normal & $\mu$ & $\displaystyle \cfrac{1}{n} \sum_{i=1}^n x^{(i)}$  & $\mathbb{R}$ \\
& $\sigma$ & $\displaystyle \frac{1}{n} \sum_{i=1}^n \left(x^{(i)} - \hat{\mu}\right)^2 $ & $\mathbb{R}$ \\
Multivariate Normal & $\mu$ & $\displaystyle \frac{1}{n} \sum_{i=1}^n x^{(i)}$ & $\mathbb{R}^m$ \\
& $\boldsymbol\Sigma$ & $\displaystyle \frac{1}{n} \sum_{i=1}^n (x^{(i)}-\hat{\mu})(x^{(i)}-\hat{\mu})^T$ & $\mathbb{R}^m$ \\
Bernoulli & $\mu$ & $\displaystyle \frac{1}{n} \sum_{i=1}^n x^{(i)}$ & $\{0, 1\}$ \\
Binomial (fixed $m$) & $\mu$ & $\displaystyle \frac{1}{nm} \sum_{i=1}^n x^{(i)}$ & $\left\{ 0, 1, \dots, m \right\}$ \\
Poisson & $\lambda$ & $\displaystyle \frac{1}{n} \sum_{i=1}^n x^{(i)}$ & $\left\{ 0, 1, \dots \right\}$\\
Geometric & $\mu$ & $\displaystyle \cfrac{n}{\sum_{i=1}^n (x^{(i)} + 1)} $ & $\left\{ 0, 1, \dots \right\}$ \\
Exponential & $\lambda$ & $\displaystyle \cfrac{n}{\sum_{i=1}^n x^{(i)}} $ & $\mathbb{R}^+$ \\
\bottomrule
\end{tabular}}
\end{center}

\begin{question}{}
In Question~\ref{question:likexamples}, we examined several examples of experimental conditions and datasets and discussed which probability distribution best modeled each one. Using the formulas above and the actual datasets from Question~\ref{question:likexamples}, calculate the MLEs for the parameter(s) of your chosen probability distributions.
\end{question}


\chapter{Generalised Spherical Harmonics Formalism}\label{app_gsh}

\section{Formalism}
A general formalism to describe complex tensor fields on the surface of a 2-sphere is that of Generalised Spherical Harmonics. A generalisation of the spherical harmonic given by $Y_{lm}^N$, with an added index $N$, is used to decompose tensor fields of arbitrary rank \cite{phinney_gsh}. It reduces to the spherical harmonic when $N=0$.
\begin{equation}
Y_{lm}^0(\theta,\phi) = Y_l^m(\theta,\phi)
\end{equation}
These functions couple with the GSH unit vectors given by
\begin{equation}
\begin{array}{ccc} \ev{-} = \frac{1}{\sqrt{2}}(\ev{\theta} - i \ev{\phi}), & \ev0 = \ev{r}, & \ev{+} = -\frac{1}{\sqrt{2}}(\ev{\theta} + i \ev{\phi}) \\ \end{array}
\end{equation}
to form tensor spherical harmonics
\begin{equation}
\Yv_{lm}^{N} \equiv Y_{lm}^{N} \ev{\alpha_1}\ev{\alpha_2} \ldots \ev{\alpha_q}
\end{equation}
where $\alpha_1 + \alpha_2 +\ldots + \alpha_q = N$. These tensor functions by construction are eigenfunctions of the total angular momentum operator $\hat{\mathbf{J}} = \hat{\mathbf{L}} + \hat{\mathbf{s}}$, where the first part $\hat{\mathbf{L}}$ is the rotational generator for the scalar part of a field and the latter part $\hat{\mathbf{s}}$ generates rotation of the vector basis. Explicit expressions for $Y_{lm}^{N}$ can be found in \cite{DT98}. Eigenvalue equations are as below
\begin{equation}
\hat{J}_{z} \mathbf{Y}_{lm}^N = m\mathbf{Y}_{lm}^N
\end{equation}
\begin{equation}
\hat{J}^2 \mathbf{Y}_{lm}^N = l(l+1) \mathbf{Y}_{lm}^N
\end{equation}
The set of all tensor GSHs of a particular rank form a complete basis for tensor fields of the same rank in 3D space. Hence, a tensor field $F(\mathbf{r})$ of rank $M$ can be decomposed as

\begin{equation}
\mathbf{F}(\mathbf{r}) = \sum_{s=0}^{\infty} \sum_{t=-s}^{s}\sum_{\mu=-M}^{M} F_{st}^{\mu}(r) \mathbf{Y}_{st}^{\mu}
\end{equation}
where $F_{st}^{\mu}(r)$ are its purely radially dependent GSH components.
\section{Conventions}\label{app_conventions}
\begin{equation}
\gam{l} \equiv \sqrt{\frac{2l+1}{4\pi}}
\end{equation}

\begin{equation}
\om{l}{N} \equiv \sqrt{\frac{(l+N)(l-N+1)}{2}}
\end{equation}
Note that $\om{l}{-N} = \om{l}{N+1}$.

\begin{equation}
g_{\mu\nu} \equiv \ev{\mu}\cdot \ev{\nu} = \left( \begin{matrix}
0 & 0 & -1 \\
0 & 1& 0  \\
-1 & 0 & 0
\end{matrix} \right)
\end{equation}
where indices in the matrix run as $(-,0,+)$.
\begin{equation}
\ev{\mu}^* \cdot \ev{\nu} = \delta_{\mu\nu}
\end{equation}

\section{Spherical triple integral}
\begin{equation}
\int_0^{2\pi} d\phi \int_{0}^{\pi} d\theta\sin\theta Y_{l_1m_1}^{N_1}Y_{l_2m_2}^{N_2}Y_{l_3m_3}^{N_3} = 4\pi \gam{l_1}\gam{l_2}\gam{l_3} \wigfull{l_1}{l_2}{l_3}{N_1}{N_2}{N_3} \wigfull{l_1}{l_2}{l_3}{m_1}{m_2}{m_3}
\end{equation}
Explicit expressions for $\wigfull{l_1}{l_2}{l_3}{m_1}{m_2}{m_3}$ can be found in \cite{DT98}. The property $\enc{Y_{lm}^{N}}^* = (-1)^{m+N}Y_{l\bar{m}}^{\bar{N}}$ makes it useful while taking inner products of tensor fields in our analysis.
Note that works like \cite{lavely92}, \cite{hanasoge17} etc. use a slighlty different convention defining $Y_{lm}^{N}$ as $Y_{lm}^{N}/\gam{l}$ as in our convention. In this work, we have followed the convention of \cite{DT98}.
\section{Rotation}

Tensor GSH functions $\mathbf{Y}_{lm}^N$ obey the same rotation laws as spherical harmonics $Y_{l}^{m}$. Consider two coordinates systems on a sphere $(\theta,\phi)$ and $(\theta',\phi')$ such that $\phi=0$ and $\phi'=0$ planes coincide. Also, for all points on this plane, let $\theta' = \theta-\beta$. This transformation law corresponds to the primed axis being tilted with respect to the unprimed axis by an angle $\beta$. Tensor GSH functions in the two coordinates are related as follows
\begin{equation}
\mathbf{Y}_{lm}^{N}(\theta',\phi') = \sum_{m'=-l}^{l} d_{mm'}^{(l)}(\beta) \mathbf{Y}_{lm'}^{N}(\theta,\phi)
\end{equation}
where $d_{mm'}^{(l)}$ is a $(2l+1)\times (2l+1)$ real matrix. Explicit form of $d^{(l)}_{mm'}$ can be found in \cite{DT98} and a python subroutine for calculating this can be  found in \textit{functions.py} in the Github repository \cite{main_repo}. Above relation holds for all points on the entire sphere.


\section{Math I - Algebra}
  \subsection{Generating Function Manager}
    \code{math/algebra/generating_function_manager.cpp}
  \subsection{Fast Fourier Transform}
    Compute the Discrete Fourier Transform (DFT) of a polynomial in $O(n \log n)$ time.
    \code{math/algebra/fft.cpp}
  \subsection{FFT Polynomial Multiplication}
    Multiply integer polynomials $a, b$ of size $an, bn$ using FFT in $O(n \log n)$. Stores answer in an array $c$, rounded to the nearest integer (or double).
    \code{math/algebra/fft-poly-mul.cpp}
  \subsection{Number Theoretic Transform}
    Other possible moduli: $2 113 929 217 (2^{25}), 2 013 265 920 268 435 457 (2^{28}, with g = 5)$
    \code{math/algebra/ntt.cpp}
  \subsection{Polynomial Long Division}
    Divide two polynomials $A$ and $B$ to get $Q$ and $R$, where $\frac{A}{B} = Q + \frac{R}{B}$.
    \code{math/algebra/poly-long-div.cpp}
  \subsection{Matrix Multiplication}
    Multiplies matrices $A_{p\times q}$ and $B_{q\times r}$ in $O(n^3)$ time, modulo \texttt{MOD}.
    \code{math/algebra/matmul.java}
  \subsection{Matrix Power}
    Computes for $B^e$ in $O(n^3 \log e)$ time. Refer to Matrix Multiplication.
    \code{math/algebra/matpow.java}
  \subsection{Fibonacci Matrix}
    Fast computation for $n$th Fibonacci $\left\{F_1,F_2,\ldots,F_n \right\}$ in $O(\log n)$:
    \[
    \begin{bmatrix}
        F_n \\
        F_{n-1}
    \end{bmatrix}
    =
    \begin{bmatrix}
        1 & 1 \\
        1 & 0
    \end{bmatrix}^n
    \times
    \begin{bmatrix}
        F_2 \\
        F_1
    \end{bmatrix}
    \]
  \subsection{Gauss-Jordan/Matrix Determinant}
    Row reduce matrix $A$ in $O(n^3)$ time. Returns \texttt{true} if a solution exists.
    \code{math/algebra/gauss-jordan.java}


\section{Maximum likelihood estimation of the free energies of states}

We start with the likelihood function, which defines the likelihood of
seeing our measured results given a set of free energy values for the
states in our network:

\begin{equation}
  L(\{\bar{\Delta}_{ij}\} | \{g_i\}) = \prod_{ij} \exp \left(
  \frac{-1}{2 \sigma^2_{ij}} ((g_j - g_i) - \bar{\Delta}_{ij})^2 \right)
  \label{likelihood}
\end{equation}

We want to use this function to find the most likely $\{g_i\}$ that
fits our data. We can do this by taking the gradient in g-space,
$\boldsymbol{\nabla}_g$, and setting it equal to the null
vector. Since the likelihood function is monotonic, we can maximize
the natural logarithm of (\ref{likelihood}), as this is will greatly
simplify the mathematics. The log-likelihood is given by:

\begin{equation}
  \ln{L} = \sum_{ij} \frac{-1}{2\sigma^2_{ij}} ((g_j - g_i) -
  \bar{\Delta}_{ij})^2
  \label{logl}
\end{equation}

We define the gradient operator as:

\begin{equation}
  \boldsymbol \nabla_g \equiv \sum_k \boldsymbol{\hat{g}_k}
  \frac{\partial}{\partial g_k}
\end{equation}

Taking the gradient of (\ref{logl}) and setting equal to the null
vector, we find:

\begin{align}
  \boldsymbol{0} = \boldsymbol \nabla_g \ln L &= \sum_{ij}
  \frac{-1}{2\sigma^2_{ij}} \boldsymbol \nabla_g ((g_j - g_i) -
  \bar{\Delta}_{ij})^2 \\
  &= \sum_{ij} \frac{-1}{\sigma^2_{ij}}
  ((g_j - g_i) - \bar{\Delta}_{ij})\left[ \boldsymbol{\hat g_j}  -
    \boldsymbol{\hat g_i} \right]  \\
   &= \sum_{ij} \frac{1}{\sigma^2_{ij}}
  ((g_j - g_i) - \bar{\Delta}_{ij})\left[ \boldsymbol{\hat g_i}  -
    \boldsymbol{\hat g_j} \right]
\end{align}

From here, we can compute $\{g_i\}$ numerically using
\verb|scipy.optimize.root|, which uses the hybrid Powell method to
find the roots of $\boldsymbol \nabla_g \ln L$. We provide the root
function the jacobian, whose elements are:

\begin{align}
  J_{nm} = \frac{\partial}{\partial g_n} \left( \boldsymbol \nabla _{g}
  \ln L \right)_m &= \frac{\partial}{\partial g_n} \sum_{ij}
  \frac{1}{\sigma^2_{ij}} \left[ (g_j - g_i ) -
    \boldsymbol{\bar{\Delta}_{ij}}\right](\delta_{mi} - \delta_{mj})
  \\
  &= \sum_{ij} \frac{1}{\sigma^2_{ij}} ( \delta_{nj} - \delta_{ni}
  )(\delta_{mi} - \delta_{mj}) \\
  &= \sum_{ij} \frac{1}{\sigma^2_{ij}} \left[ \delta_{nj} \delta_{mi}
    - \delta_{nj}\delta_{mj} - \delta_{ni}\delta_{mi} + \delta_{ni}
    \delta_{mj} \right] \\
\end{align}


\subsection{Trig Identities}

As we could see in Figure \ref{unitCircle}, $\sin$ and $\cos$ form a right triangle with a hypotenuse of length 1.
So, using the Pythagorean Theorem,
\begin{equation*}
	\sin^2{\theta} + \cos^2{\theta} = 1.
\end{equation*}
By dividing by $\sin^2$ or $\cos^2$, we can also get
\begin{equation*}
	1 + \cot^2{\theta} = \csc^2{\theta} \text{ and } \tan^2{\theta} + 1 = \sec^2{\theta}.
\end{equation*}
Together, these 3 identities are called the Pythagorean Identities.\\

We can also relate functions and co-functions.
\begin{equation*}
	\text{xxx}(\theta) = \text{cxx}\left(\frac{\pi}{2} - \theta\right)
\end{equation*}

Some of the most useful and used identities are the sum and difference.
\begin{equation*}
	\sin{\left(\alpha \pm \beta\right)} = \sin{\alpha}\cos{\beta} \pm \cos{\alpha}\sin{\beta}
\end{equation*} \begin{equation*}
	\cos{\left(\alpha \pm \beta\right)} = \cos{\alpha}\cos{\beta} \mp \sin{\alpha}\sin{\beta}
\end{equation*} \begin{equation*}
	\tan{\left(\alpha \pm \beta\right)} = \frac{\tan{\alpha} \pm \tan{\beta}}{1 \mp \tan{\alpha}\tan{\beta}}
\end{equation*} \begin{equation*}
	\sin{\alpha} \pm \sin{\beta} = 2\sin{\left(\frac{\alpha \pm \beta}{2}\right)}\cos{\left(\frac{\alpha \mp \beta}{2}\right)}
\end{equation*} \begin{equation*}
	\cos{\alpha} + \cos{\beta} = 2\cos{\left(\frac{\alpha + \beta}{2}\right)}\cos{\left(\frac{\alpha - \beta}{2}\right)}
\end{equation*} \begin{equation*}
	\cos{\alpha} - \cos{\beta} = -2\sin{\left(\frac{\alpha + \beta}{2}\right)}\sin{\left(\frac{\alpha - \beta}{2}\right)}
\end{equation*}