\documentclass[12pt]{article}
\usepackage{amsmath,amssymb,amsthm}
\usepackage{fullpage}
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage{listings}

\theoremstyle{definition}
\newtheorem{thm}{Theorem}[section]
\newtheorem{lem}[thm]{Lemma}
\newtheorem{defn}{Definition}[section]
\newtheorem{conj}{Conjecture}[section]
\newcommand{\floor}[1]{\left\lfloor #1 \right\rfloor}
\newcommand{\ceil}[1]{\left\lceil #1 \right\rceil}
\newcommand{\bigC}[0]{\mathcal{C}}
\begin{document}
\emergencystretch 3em
\title{How hard is it to detect {\em some} cliques?}

\author{Josh Burdick}

\maketitle

\begin{abstract}
Shannon's function-counting argument
\cite{shannon_synthesis_1949} showed that most Boolean functions have
exponential circuit complexity, but doesn't provide a specific example
of such a hard-to-compute function. A simple modification of that argument
shows that detecting a randomly-chosen subset of the $k$-vertex cliques in an
$n$-vertex graph requires, on average, $\Omega(n^{k/2})$ NAND gates.
Unfortunately,
this doesn't directly bound the complexity of detecting {\em all} of the cliques;
however, it seems like a related problem.
Here, we attempt to combine a counting argument with
random restrictions, to estimate the number
of NAND gates needed to detect some cliques (as a function
of the number of cliques).
\end{abstract}

\newpage

\tableofcontents

This is an attempt to obtain a lower bound on the the number of NAND gates
needed to detect {\em some} of the cliques in a graph (as opposed to
all of them).
Although it seems unlikely to work, hopefully
it will add to the long list of strange things which would happen
if P = NP \cite{fenner1996complexity}.

\section{A counting bound}
\label{countingBound}

The first component we use is a slight modification
of Shannon's function-counting argument
\cite{shannon_synthesis_1949}.

\subsection{Background: lower bounds from function counting}

It has long been known that computing {\em some} function of a bit-string
requires exponentially large circuits \cite{shannon_synthesis_1949}.
Let $f: \{0,1\}^m \rightarrow \{0,1\}$ be a function from bitvectors to bits.
If there are $m$ inputs to a circuit,
then there are $2^{2^m}$ possible functions from the $m$-input bitstring to
a one-bit output. Each of these functions, being different, must have a
different circuit.

Given a count of functions, we can then see, for instance,
how many unbounded fan-in NAND gates are needed to compute
those functions (see Theorem \ref{boundFromCounting}
in Appendix \ref{gateMath}).

\subsection{Counting CLIQUE-like functions}

Suppose we are given an $n$-vertex graph.
Let $k$-CLIQUE($n$) be the boolean function which
detects $k$-cliques: it outputs 1 if any $k$-clique
is present, and 0 otherwise. This is a classic
NP-complete problem.

We now consider a ``buggy'' variant of the $k$-CLIQUE function,
which only detects a subset of cliques. (Nomenclature note:
technically, since
we're only trying to output a 1 iff there's a clique, I should say
``detects''; ``finding'' a clique would mean outputting its
vertices as well. I'll probably write ``finds'' here for now,
even though it isn't exactly the right nomenclature.)

As a concrete example,
consider the set of ``buggy'' 6-clique detectors.
Maybe the circuit correctly
finds all the cliques. Or maybe it finds all of the cliques except $K_{1..6}$,
or it misses half the cliques, or finds none (and always outputs 0), or maybe
it only successfully finds $K_{1,3,4,5,7,8}$, et cetera.

We define a variant of $k$-CLIQUE which only
finds a subset of cliques.
Let $K$ denote the set of all possible
$k$-vertex cliques.

\begin{defn}
\label{BUGGYCLIQUE}
Let $A \subseteq K$.
Let $m = {n \choose 2}$ be the number of edges in the input graph.
$BUGGYCLIQUE(A): \{0,1\}^m \rightarrow \{0,1\}$ is the function which
is 1 iff any of the $K_k$s in $A$ is present.
That is, for each set $A$ of $K_k$s, $BUGGYCLIQUE(A)$
contains a function which is 1 if the input contains any $K_k \in A$,
and 0 otherwise. (Using this nomenclature,
$k$-CLIQUE(n) = $BUGGYCLIQUE(K)$).
\end{defn}

Of course, many of these functions are quite similar (e.g. all but one of them
output a 1 when you feed in all 1's). However, they're all slightly different.

\begin{thm}
\label{buggyDistinct}
There are  $2^{|K|}$ distinct $BUGGYCLIQUE$ functions.
\end{thm}
\begin{proof}

Let $A,B \subseteq K$, with $A \neq B$, and w.l.o.g.
let $x \in A-B$. Then $BUGGYCLIQUE(A)$ outputs 1 on the input
with just the edges in $x$ set to 1 (and 0 everywhere else),
while $BUGGYCLIQUE(B)$ outputs a 0.

There are $2^{|K|}$ many subsets of $K$,
and by the above, each corresponds to a diffferent function.
\end{proof}

For a given $A$, many of the functions in $BUGGYCLIQUE(A)$
are similar (for instance, most
of them output a 1 when all the edges are present);
but they are all distinct.
Although $2^{n \choose k}$ is a fairly large number,
it's still comfortably less than $2^{2^{n \choose 2}}$, the number of boolean
functions on the ${n \choose 2}$ input wires (one per edge).

\subsubsection{But {\em which} function requires many gates?}

Thus, there are $2^{n \choose k}$ different functions.
How many NAND gates do these take?
(We consider NAND gate circuits (with any fan-in) which find $k$-cliques in $n$-vertex
graphs, as a circuit with $n \choose 2$ inputs.)

Applying Theorem
\ref{boundFromCounting}, we know that at least one of the circuits requires
${\sqrt {2 {n \choose {k/2}} + b^2}} - b = \Omega(n^{k/2})$
NAND gates (where $b = {k \choose 2} - 0.5$).

Why doesn't this bound $k$-CLIQUE?
Because we don't know that the circuit which finds {\em all} of the
$K_k$s is one of these larger circuits. As far as what I've
shown thus far goes, it could be harder to find some weird subset of the $K_k$s.

Indeed, as far as what we've formally shown goes, the problem which needs
the most NAND gates could be finding a single clique! That's easily ruled out
(because that only needs one NAND gate, plus the output gate).

\subsection{Which sets of cliques are hard to find?}
\label{sec:whichCliques}

The hardness of these functions depends
on how the cliques they find are laid out.

Cliques are arguably difficult to draw in a two-dimensional space.
As an approximate diagram reflecting what we know,
we sketch a Hasse diagram of possible subsets of cliques. Although
we only draw a few subsets of three-vertex cliques
on six vertices, hopefully this provides some
intuition.

\begin{figure}
\centering
\includegraphics[width=1\textwidth]{R/Hasse.pdf}
\caption{Hasse diagram of BUGGYCLIQUE functions.
(a-d)
Detecting all the possible cliques in larger graphs will be
increasingly difficult (although {\em how much} harder isn't clear).
(e)
Detecting this set of cliques is definitely harder than (b),
since we can convert from (e) to (b) by feeding in 0's to
some set of edges.
(f) Detecting a set of cliques which doesn't overlap much will be
harder than detecting the same number of cliques, when they overlap
maximally (as in (b)).}
\label{fig:Hasse}
\end{figure}

\begin{thm}
\label{edgeZonking}
Let $A \subsetneq B \subseteq K$, such that $A$ is what remains
of $B$ after removing all cliques overlapping some edge $e$.
Then $|\bigC(BUGGYCLIQUE(B))| > |\bigC(BUGGYCLIQUE(A))|$.
\end{thm}
\begin{proof}
Feed in a 0 to $e$, which is in $B$; the remaining cliques are $A$.
The resulting
circuit computes $BUGGYCLIQUE(A)$, and so has size
at least $|\bigC(BUGGYCLIQUE(A))|$. But at least one
NAND gate has been removed by feeding in the 0.
\end{proof}

This shows that sometimes, finding a larger set of cliques is
harder. However, the above theorem doesn't help if the two
sets of cliques cover the same set of edges.

\begin{figure}
\label{overlappingTris}
\centering
\includegraphics[width=1\textwidth]{R/tri1.pdf}
\caption{A given number of cliques can overlap maximally (left),
or not overlap much (right).}
\label{fig:overlappingTriangles}
\end{figure}

Triangles can be detected using matrix multiplication \cite{itai_finding_1977},
and there are fast algorithms known for matrix multiplication
\cite{strassen_gaussian_1969}
\cite{williams_multiplying_2012}, so the set of all possible
cliques on some set of vertices (Figure \ref{overlappingTris}, left)
 can be detected
using fewer than one NAND gate per triangle (for large enough input graphs).

On the other hand, if the triangles overlap less (as in
Figure \ref{overlappingTris}, right),
then to detect all of the triangles, we will definitely need at least one
gate per triangle. We can see this by feeding in a 0 to some input
unique to a triangle, applying \ref{edgeZonking}, and repeating.

\subsection{Defining levels of CLIQUE}

To me, it seems a reasonably intuitive guess that the hardness of
computing $BUGGYCLIQUE(A)$ should be somehow related to
$|A|$, which is simply the number of cliques it ``sees''.

\begin{defn}
\label{CLIQUE-level}
Assume $n, k$ are fixed. For any $l$ such that
$0 \le l \le |K|$, let $K_l$ be the set of all sets
with exactly $l$ cliques.
\end{defn}

We abuse notation slightly, and write

\[
BUGGYCLIQUE(K_l) = \{ BUGGYCLIQUE(k) : k \in K_l \}
\]

What can we say about $E[|\bigC(BUGGYCLIQUE(K_l))|]$, for
some fixed $l$? Let $N = {n \choose k}$.
At the ``bottom'', there's only one $BUGGYCLIQUE(\emptyset)$
function, so the counting argument is useless.
In the ``middle'',
there are ${N \choose {N/2}}$ functions, and so the counting
argument gives a nontrivial lower bound for
$E[|\bigC(BUGGYCLIQUE(K_{N \choose {N/2}}))|]$ (although
without actually constructing even {\em one} difficult function!).
At the ``top'', again, there's only one $BUGGYCLIQUE(K) = k-CLIQUE$
function, so the counting argument is, once again, useless.

It would be nice if we could show that, as $l$ increases,
$E[|\bigC(BUGGYCLIQUE(K_l))|]$ also increases.
If we could prove something in general, for all levels $l$, then at
the top of the diagram, we'd be bounding just the function
$BUGGYCLIQUE(K_l) = k-CLIQUE$. (This suggests that doing so would
be difficult...)

\section{Using random restrictions}

Random restrictions have been used in lower bounds of formula
\cite{subbotovskaya1963comparison} and circuit \cite{hastad1987lower}
complexity (see also the slides at \cite{rossmanRestrictions}).
Here, we apply random restrictions to a set of functions (and circuits),
rather than just one function and circuit.

??? Is this just a ``gate-elimination argument''?

\subsection{Counting functions by their ``rank'' in a list}

This argument relies on measuring the size of a circuit,
for a given basis. (Here, we assume unbounded fan-in
NAND gates, but this doesn't seem crucial.)

\begin{defn}
\label{Rank}
For all $C \subseteq K$,
arrange all of the sets of cliques in nondecreasing order
of $|\bigC(BUGGYCLIQUE(C))|$ (using unbounded fan-in NAND gates,
breaking ties by some lexicographic order of circuits).

Let $A \subset K$. The {\em rank} of $A$, $rank(A)$, is the zero-based
index of $A$ in this list.
\end{defn}

If we can lower-bound $rank(A)$, then we should be able to translate
that into a
lower bound on $|\bigC(BUGGYCLIQUE(A))|$. (Possibly, if we
can only lower-bound $E[rank(A)]$, we may be able to get
a bound in terms of number of gates using Jensen's inequality?
Not clear.)

We can imagine a miles-long linear warehouse of chips with the minimal
circuit for each problem, sorted in terms of number of gates.
Note that the above list only includes functions in $BUGGYCLIQUE$.
There are a {\em ton} of other functions (parity, primality, etc.), but ignoring
those should still leave a valid lower bound for the functions in $BUGGYCLIQUE$.

\subsection{How much smaller are ``restricted'' circuits, on average?}

\begin{thm}
\label{vaguelyUpward}
Let $C \subseteq K$ be a set of cliques chosen uniformly at random
from $K$.

Let $a = {n-2 \choose k-2}$ (this is the number of $k$-cliques
intersecting a given edge).

Then there is a function $f: 2^K \rightarrow 2^K$ such that
\begin{itemize}

\item $E[|C| - |f(C)|] = a/2$

\item $E[rank(C) - rank(f(C))] = a/2$

\end{itemize}

\end{thm}
\begin{proof}

Pick a distinguished input edge $e$, and let $A \subseteq C$ be
the set of cliques in $C$ which include $e$, and $B \subseteq C$ be
the set of cliques in $C$ which don't include $e$.

We define $f(C) = B$.

Now, take the circuit computing $BUGGYCLIQUE(C)$, and
feed in a 0 to $e$. The resulting circuit computes
$BUGGYCLIQUE(B)$. Note that $A$ could contain anywhere
from 0 to $a$
cliques, so since $C$ is chosen uniformly at random,
$E[|A|] = a/2$.

Furthermore, $|\bigC(B)| \le |\bigC(C)|$, because feeding in
a zero to the circuit for $BUGGYCLIQUE(C)$ constructed a
(possibly non-optimal) circuit for $BUGGYCLIQUE(B)$.
Thus, $rank(B) \le rank(C)$.
This means that
$C$ is one of $a$ functions chosen uniformly at random,
and so, on average, there are $a/2$ circuits between $C$
and $B$, implying at least that large a difference in rank.

(Note that if $e$ ``misses'' all of the cliques in $C$, then $B = C$
and $rank(B) = rank(C)$. However, provided $e$ is in some clique in $C$,
then since we're feeding in a 0 to a NAND-gate circuit,
the inequality $rank(B) < rank(C)$ is strict.)

\end{proof}

In terms of the previous warehouse analogy, we can imagine that,
when an edge is disabled, the chip now has strictly fewer gates.
Perhaps the chip-maker spent their entire budget on designing the
optimal circuits, but then skimped on the wires connecting the chips?
In that case, assuming the missing wire is treated as a 0, the chip still
finds some cliques, and so might be sellable.
By construction of the warehouse, that means the faulty chip could be shifted,
on a giant conveyor belt,
toward the ``smaller'' (in number of gates) end of the warehouse.
That means that the corresponding set of cliques is also
in the ``smaller'' end of the warehouse.

Although both these numbers (the reduction in number of cliques, and
the reduction in rank) have an expected value of $a/2$,
they aren't necessarily the same, or even correlated.
(Indeed, when $n$ is large, the number of cliques is sharply peaked around $a/2$,
but the lower bound in reduction in rank is uniformly distributed across $0..a$.)

\subsubsection{Why this doesn't bound $k-CLIQUE$?}

\ref{vaguelyUpward} seems to show that ``finding more cliques is
at least a tiny bit harder on average.'' However, it only applies
on average.

What happens if we try to use restrictions to bound $k-CLIQUE$?
If we start ``at the top'',
feeding in a 0 to $k-CLIQUE$, which is one function,
we get only one other function.

However, it seems vaguely plausible to proceed ``bottom-up'',
estimating the difficulty of finding increasingly large
sets of cliques.

\subsection{Estimating the rank of functions, at each level}

It seems intuitive that finding a larger fraction of the cliques
should be harder.
So, another idea is to try to show this using induction on $l$.
As usual, the base case ($l=0$) is easy: finding zero cliques is easiest,
so its rank is 0.

The step case is not so straightforward. Suppose we've computed bounds for
levels 0 through $l-1$ (thus, we're using some sort of strong induction.)
In other words, we've lower-bounded $E[rank(K_i)]$, for $i \in [0,l-1]$.
Can we lower-bound $E[rank(K_l)]$?

Suppose we pick a set of cliques at level $l$, uniformly at random.
If we then pick an edge $e$ uniformly at random, and remove the cliques
which contain $e$, then the number of cliques removed will vary
depending on the level. When $l=1$, we'll remove 0 or 1 cliques; when $l=N$,
we'll remove all ${n-2 \choose k-2}$ cliques which contain $e$.

We can {\em estimate} the number of cliques removed, by supposing that each
clique is ``hit'' by $e$, with probability $p = {k \choose 2} / {n \choose 2}$.
If we assume that $e$ hitting each clique is an independent event, then
restricting a random set of $l$ cliques will hit $pl$ cliques, and
leave $(1-p)l$ cliques.
In general, when $n>>k$, $p$ will be pretty small. Thus, the step case at
level $l$ will depend on estimates for levels fairly ``high up'' near $l$.
Also, when $n$ is large, $N$ is way larger; so the distribution of
the fraction of cliques removed will be sharply peaked near $p$.

However, this induction isn't quite correct.
We may have a bound on $E[rank(K_i)]$, for $i$ in $[0,l-1]$.
But when we pick a random set of cliques $K_l$, and feed in a 0, we definitely
aren't sampling uniformly from levels 0 through $l-1$, because the graphs we're
sampling definitely lack one edge.

Thus, this is {\em only an estimate} of the rank of functions at level $l$.
To me, this raises (at least) the following questions:

\begin{itemize}

\item If we compute the estimate, what do we get?

\item Is there a way to modify this, to get an actual bound?

\end{itemize}

I only address the first question, and punt on the second.

\subsection{An estimated bound}

Thus, even though this is an estimate, we ``simulate'' this process, starting
at level 0, and going up to level $N = {n \choose k}$.
First, define $p$, the estimated probability that a 0 ``hits'' a clique:

\[ p = \frac{{k \choose 2}}{{n \choose 2}} \]

Assume that the number of cliques after feeding in a 0 follows a binomial
distribution, with $l$ ``trials'' and ``probability of success'' $1-p$.
$B(l)$ will then be a weighted average of $B(0)$ through $B(l-1)$. However,
if the 0 ``misses'' all the cliques, then we can't say anything about the
rank, and just use 0.
Furthermore, we know that at level $l$, we're adding ${N \choose l}$ new
functions.

Let $R(l)$ be the lower bound on the expected rank of functions in $K_l$.

\[
R(0) = 0
\]

For $0 < l \le N$, define
\[
	R(l) = [ \sum_{1\le i < N} binom(i; l, 1-p) R(i) ] + {N \choose l}/2
\]

The file {\tt py/approxRank.py} computes this bound. Here are the bounds
for two small cases.

\includegraphics[width=0.45\textwidth]{py/rank_n=6_k=3.pdf}
\includegraphics[width=0.45\textwidth]{py/rank_n=11_k=4.pdf}

\section{Related work}

This strategy relies heavily on a modification of Shannon's original
function-counting argument \cite{shannon_synthesis_1949},
combined with random restrictions
\cite{subbotovskaya1963comparison} \cite{hastad1987lower}.

A related question is whether problems
(such as $k$-SAT) are
hard on average \cite{bogdanov2006average}.
These efforts seem to focus more on whether
random
instances of a given problem are hard, rather
than using random problems to show that
a specific problem is hard.

??? What about:
- co-NP?
- formula size?

This lower-bound strategy also seems potentially
relevant to quantum computing,
as the argument makes few restrictions on the sort of gates used.
However, we did use the property of NAND gates that ``feeding in
a 0 disables a gate''; it's not clear whether that's needed,
or holds for quantum gates.

\section{Conclusion}

We give a lower bound on finding {\em some} set of cliques.
It is a modified form of Shannon's counting argument
\cite{shannon_synthesis_1949}, combined with random restrictions
\cite{subbotovskaya1963comparison} \cite{hastad1987lower}.
This suggests {\em approximate} bounds on functions {\em similar} to $k$-CLIQUE.
Unfortunately, however, this is an approximate bound,
and so doesn't bound the complexity of $k$-CLIQUE.

If this were true, then by the contrapositive,
\cite{fenner1996complexity} would give a concise list of some of the consequences...

\section{Acknowledgements}

The author would like to thank William Gasarch for introducing him
to lower bound strategies, and probabilistic proofs.
He would also like to thank the maintainers of
several entertaining and relevant blogs, including but
not limited to: the Computational Complexity blog
(Lance Fortnow and William Gasarch),
G\"odel's Lost Letter (Richard Lipton and Ken Regan),
and Shtetl-Optimized (Scott Aaronson).

\appendix
\section{Calculations for unbounded fan-in NAND gates}
\label{gateMath}

We used unbounded fan-in NAND gates largely because they are
convenient to convert from ``number of functions'' to
``number of gates''.

Let $\bigC(f)$ be the circuit
with the fewest unbounded fan-in NAND
gates computing $f$, and let $|\bigC(f)| = g$ be the number of gates in
that circuit. The
circuit could have at most $gm$ wires from inputs to gates, and ${g \choose 2}$
wires from gates to gates. We can view the possible circuits as a bitmask,
containing a 1 everywhere a gate is connected to an input (or another gate),
and 0 everywhere else.

\begin{thm}
\label{boundFromCounting}
Consider functions from $m$ bits to one bit of output.
This means that, with $g$ gates, we can represent at most
$2^{gm + {g \choose 2}}$ different boolean functions (with $m$ bits of input,
and one bit of output).
\end{thm}
\begin{proof}

The number of possible wires which are there, or not, is $gm + {g \choose 2}$,
which bounds how many possible circuits there are.
Some of these circuits compute the same function.
However, there can't be any more than this many circuits with this many wires.
\end{proof}

This means that if we have a large set of functions, and we know the size of
the set of functions, then we know that at least {\em one} of them requires
a large number of gates. (Knowing {\em which} function requires a lot, or many,
gates is still an issue).

We used unbounded-fan-in NAND gates because, given a number of circuits,
it's easier (I think!) to solve for the number of wires than if we had
used two-input gates.
Consider functions from $m$ bits to one bit of output.
Let $g$ be the number of gates, and $w$ be the number of wires.
Solving for the number of gates:

\begin{eqnarray*}
w & = & mg + {g \choose 2} \\
  & = & mg + g(g-1)/2 \\
  & = & mg + (g^2 - g) / 2 \\
  & = & 0.5g^2 + (m-0.5)g \\
0 & = & 0.5g^2 + (m-0.5)g - w \\
\end{eqnarray*}

We solve the quadratic formula (writing $b = m-0.5$ for simplicity), keeping
only the non-imaginary root.

\begin{eqnarray*}
g & = & -b \pm \sqrt{ b^2 + 2w} \\
  & = & {\sqrt {2w + b^2}} - b \\
\end{eqnarray*}

Thus, given a set of functions, we know that at least one of them requires
this many gates.

This expression is a bit inconvenient, and as $n$ increases, we
expect the number of gates to be much larger than the number of
inputs $b$. To avoid such issues, we can count the number of gates
beyond the number of inputs. This gives the bound

\begin{eqnarray*}
g & \ge & {\sqrt 2w} - m \\
\end{eqnarray*}

\bibliography{references}
\bibliographystyle{unsrt}

\end{document}


%*******************************************************************************
%*********************************** Second Chapter ***************************
%*******************************************************************************
%!TEX root = 0.main.tex

\section{Non uniform sampling schemes and the FEM Laplacian}

In Chapter 3, we used the fact that when the sampling scheme of the sphere is regular enough, the graph $G$ is such that the corresponding graph Laplacian $\mathbf {L=V\Lambda V}^\intercal$ has eigenvectors that are close enough to the ones of $\Delta_{\mathbb S^2}$ to design a graph with a low mean equivariance error. We showed a way to construct a graph $G'$ such that its graph Laplacian well approximates $\Delta_{\mathbb S^2}$ in the case of an equiarea sampling scheme of the sphere and we tested it on the HEALPix sampling scheme. In this Chapter we focus on sampling schemes that are less uniform than HEALPix. The sampling scheme $V=\{v_i\in \mathbb S^2\}$ that we will use for our study, very used in applications, is the so called \textit{equiangular sampling scheme} \cite{Driscoll:1994:CFT:184069.184073}.
This chapter is organized as follows: In section \ref{sec:Chapter3: Heat Kernel Graph Laplacian on the Equiangular Sampling} we first introduce the equiangular sampling, and then we present the results that we obtained with two different Graph Laplacians: the HKGL, and a graph proposed by Khasanova et al. \cite{Frossard2017GraphBasedCO}, specifically designed for this sampling scheme. In section \ref{sec:Chapter3: Using the Finite Element Method to approximate the Laplace-Beltrami operator on a manifold} we deepen how to use the Finite Element Method (FEM) to construct a discrete approximation of $\Delta _{\mathbb S^2}$ and how to derive a graph-like Laplacian from it, that shows a low equivariance error.
\subsection{Graph Laplacian on the Equiangular Sampling}
\label{sec:Chapter3: Heat Kernel Graph Laplacian on the Equiangular Sampling}

\subsubsection{The Equiangular Sampling}

Given the usual parametrization $x = x(\theta, \phi)$ of the sphere
\begin{align*}
	\mathbb{S}^{2}&=\left\{x=\left(x_{1}, x_{2}, x_{3}\right) \in \mathbb{R}^{3} :\|x\|_{\mathbb{R}^{3}}=\left(x_{1}^{2}+x_{2}^{2}+x_{3}^{2}\right)^{1 / 2}=1\right\}\\
	x_{1}&=\cos (\phi) \sin (\theta), \quad x_{2}=\sin (\phi) \sin (\theta), \quad x_{3}=\cos (\theta)
\end{align*}

Let $m\in\mathbb N$, the \textit{equiangular sampling} of bandwidth $b=2^m$ is given by
$
x_{j k}^{(b)}=x\left(\theta_{j}^{(b)}, \phi_{k}^{(b)}\right)
$
where
\begin{align*}
	\theta_{j}^{(b)} &:=\pi \frac{j}{2 b}, \quad \phi_{k}^{(b)} :=2 \pi \frac{k}{2 b}\\
	j&=0, ..., 2b-1 \text{ and }k=0, ..., 2b-1
\end{align*}

\begin{figure}[h]
	\centering
	\includegraphics[width=0.5\textwidth]{figs/Chapter3/equiangular.png}
	\caption{\label{fig:equiangular sampling}Equiangular sampling with bandwidth $b=8$}
\end{figure}
One has thus $n=4b^2$ points on the sphere, where all the points $x_{0 k}^{(b)}$ correspond to the north pole for every $k=0, ..., 2b-1$. Notice also that the south pole is never sampled. In figure \ref{fig:equiangular sampling} it can also be appreciated how the area close to the poles is much more sampled that the equator. One reason for which this sampling scheme is very used in application is the existence of the following result from \cite{Driscoll:1994:CFT:184069.184073}, that states that any band limited function can be exactly recovered from its sampled values $f\left(x_{j k}^{(b)}\right)$:
\vspace{0.5cm}
\begin{theorem}\label{theo:equiangular sampling theorem}
	Let \(l_{0} \in \mathbb{N}\) and \(m_{0} \in \mathbb{Z},\left|m_{0}\right| \leq l_{0} .\) If \(f=\sum_{l=0}^{b-1} \sum_{m=-l}^{l} \widehat{f}(l, m) Y_{l}^{m}\)
	then

	$$
	\begin{aligned} \widehat{f}\left(l_{0}, m_{0}\right)=& \frac{1}{4 b^{2}} \sum_{j=0}^{2 b-1} \sum_{k=0}^{2 b-1} f\left(x_{j k}^{(b)}\right) \overline{Y_{l_{0}}^{m_{0}}\left(x_{j k}^{(b)}\right)} \sin \left(\theta_{j}^{(b)}\right) \times \\ & \times \frac{4}{\pi} \sum_{l=0}^{b-1} \frac{1}{2 l+1} \sin \left((2 l+1) \theta_{j}^{(b)}\right) \end{aligned}
	$$
\end{theorem}
\vspace{0.5cm}
Theorem \ref{theo:equiangular sampling theorem} is the equivalent on the sphere of the well known Shannon's sampling theorem, that states the minimum sampling frequency at which a band limited signal $f:\mathbb R \to \mathbb R$ can be perfectly reconstructed, and is a precious tool when doing signal processing on the sphere.

\subsubsection{Heat Kernel Graph Laplacian on the equiangular sampling scheme}
Thanks to theorem \ref{theo:equiangular sampling theorem} not only we have a characterization of the space \\$F=\{f=\sum_{l=0}^{b-1} \sum_{m=-l}^{l} \widehat{f}(l, m) Y_{l}^{m}\}\subset L^2(\mathbb  S^2)$ of band limited functions on which the sampling operator $T: F\to \mathbb R^n$ is invertible, but we also an analytic expression for $T^{-1}$. Thanks to this we can calculate up to machine precision the equivariance error for any function $f\in F$ and any rotation $g\in SO(3)$. In figure \ref{fig:Equivariance error of the full HKGL} we calculated the mean equivariance error of the full HKGL averaging on mono-frequency signals for different bandwidths. We can observe a much cleaner behavior of the mean equivariance error than the one we observed for he HEALPix sampling, most probably due to the fact that this time we can exactly calculate $T^{-1}$. The error increases linearly with the frequency of the signal for all bandwidths, and it seems to converge to zero as $b$ increases. However, the spectrum of the graph Laplacian does not show the eigenvalues grouped in the expected pattern.
\begin{table}[h!]
	\centering
	\begin{tabular}{ c|c }
$b$ & $t$ \\
	\hline
4 & 0.5 \\
8 & 0.3 \\
16 & 0.1 \\
	\end{tabular}
	\caption{\label{table:equiangular kernel width}Kernel width $t$ used to construct the HKGL for each bandwidth $b$}
\end{table}

\begin{figure}[h!]
	\centering
	\includegraphics[width=\textwidth]{../codes/06.Equivariance_error/FullHKGLonequiangularsampling.png}
	\includegraphics[width=\textwidth]{../codes/02.HeatKernelGraphLaplacian/equiangular/equi_full_eigenvalues_16.png}
	\caption{\label{fig:Equivariance error of the full HKGL}Equivariance error of the full HKGL on the equiangular sampling by spherical harmonic degree $\ell$, and its spectrum for the bandwidth $b=8$ .}
\end{figure}

\subsubsection{A graph alternative to the HKGL for the equiangular sampling}\label{sec:Chapter3:Frossard}

Khasanova et al. \cite{Frossard2017GraphBasedCO} designed a discrete Laplacian that is explicitly intended to work on the sphere with the equiangular sampling. They studied a way to build a graph to analyze images produced by omnidirectional cameras. In their work they assume that the image is sampled on the sphere on the equiangular sampling. They consider the set $\mathcal G$ of all the possible graphs where each node is connected only to four of its nearest neighbours (North, Sud, West, East) and propose a weighting scheme $w_{ij}$ that minimizes the difference in the response to the polynomial spectral filter $\mathcal F = \mathbf L$ evaluated on images of the same object seen at different latitudes. In other words, they solve the minimization problem

\begin{equation}\label{eq:minimization frossard}
	\min_{W\in\mathcal G} \left|\mathcal{F}\left(\mathbf{y}\left(v_{ e}\right)\right)-\mathcal{F}\left(\mathbf{y}\left(v_{ i}\right)\right)\right|
\end{equation}
\begin{figure}
	\begin{center}
		\includegraphics[width=0.5\textwidth]{figs/Chapter3/frossard2.png}
	\end{center}
	\caption{\label{fig:frossard2}Khasanova et al. setting.}
\end{figure}
for the adjacency matrix $W$, where $\mathbf y(v_i)$ is the image of the object on the sphere centered on the vertex $v_i$, and $\mathcal F (\mathbf y(v_e))$ is the response of the filter at the vertex $v_e$ that lies at the same longitude of the vertex $v_i$ but on the equator (figure \ref{fig:frossard2}). In their work they prove that the optimal weights solving the minimization problem (\ref{eq:minimization frossard}) are given by weights $w_{ij}$ inversely proportional to the Euclidean distance between vertices:
\begin{equation}\label{eq:frossard weights}
	w_{ij} = \frac{1}{\norm{x_i-x_j}}
\end{equation}
This construction is interesting since it is adapted to the equiangular sampling, and leads to a very sparse graph with only 4 neighbors per vertex. Furthermore, to obtain the weights (\ref{eq:frossard weights}) every calculation was done in the \textit{spatial domain}, without any consideration about the spectral interpretation of the filter. In order to compare it to the HKGL we show the equivariance error by spherical harmonic degree in figure \ref{fig:Equivariance error of the Frossard-Khasanove graph}. It can be appreciated how this construction performs a little worse that the full HKGL for low bandwidth samplings, but converges faster than the \textit{full} HKGL, and its spectrum looks much more similar to the one of $\Delta_{\mathbb S^2}$.

\begin{figure}[h!]
	\centering
	\includegraphics[width=\textwidth]{../codes/06.Equivariance_error/KhasanovaFrossardgraphonequiangularsampling.png}
	\includegraphics[width=\textwidth]{../codes/02.HeatKernelGraphLaplacian/equiangular/equi_full_Khasanova_Frossard_eigenvalues_16.png}
	\caption{\label{fig:Equivariance error of the Frossard-Khasanove graph}Equivariance error of the Khasanova-Frossard graph on the equiangular sampling with bandwidth $b=8$ by spherical harmonic degree $\ell$, and its spectrum.}
\end{figure}

The Khasanova-Frossard graph showed us that it is possible to do better than the HKGL on the equiangular sampling. We asked ourself if there's a general way of building a more equivariant graph than the HKGL that is straightforward to implement on any sampling scheme of the sphere. To answer this question in the next section we study a more complex way to approximate the Laplace-Beltrami operator $\Delta_{\mathbb S^2}$: the linear FEM Laplacian. We will see that the comparison between the FEM Laplacian and the graph Laplacian will give us precious insights to better understand the limitations of graph Laplacians when it comes to deal with non uniform samplings of the sphere.
\subsection{The Finite Element Method approximation of the Laplace-Beltrami operator on the sphere}\label{sec:Chapter3: Using the Finite Element Method to approximate the Laplace-Beltrami operator on a manifold}

The Finite Element Method (FEM) is a numerical algorithm that allows to calculate a discrete approximation of the solution of the Laplace-Beltrami eigenvalue problem through a functional discretization of the differential operator $\Delta_{\mathbb S^2}$.  We put the necessary definitions and mathematical concepts necessary to properly introduce the weak formulation of a differential problem, the Galerkin method and finally the Finite Element Method in Appendix. For a detailed introduction to the FEM, we refer the reader to \cite{Quarteroni:1639539}\\
Let's transform the strong form of the eigenvalue problem on the Sphere on its \textit{weak} formulation. Let's multiply equation (\ref{eq:continous eigenvalue problem}) by a sufficiently regular function $v$ and integrate on $\mathbb S^2$. Since the sphere is a closed manifold and has no border, integrating by parts yields
\begin{equation}\label{eq:weak eigenvalue problem}
\begin{split}
&\text{Find } f\in H^1(\mathbb S^2), \lambda\in\mathbb R\text{ such that }\\
&\int_{\mathbb S^2} \nabla f(\mathbf x)\cdot\nabla v(\mathbf x) d\mathbf x = \lambda \int_{\mathbb S^2} f(\mathbf x)\cdot v(\mathbf x)d\mathbf x\quad \forall v\in H^1(\mathbb S^2)
\end{split}
\end{equation}
where $v$ has been chosen to belong to $H^1\subset L^2(\mathbb S^2)$, the Sobolev space of all the functions with derivative $\nabla f\in L^2(\mathbb S^2)$. The Finite Element Method first approximates the domain $\Omega$ with a triangulation $\mathcal T_h=\{ \tau_i \}$, and then \textit{projects} the weak problem (\ref{eq:weak eigenvalue problem}) into a finite dimensional subspace $V_h\subset H^1(\mathcal T_h)$ made by all the piecewise linear polynomials on the triangulation $\mathcal T_h$. Being the sphere convex, the triangulation $\mathcal T_h$ has been obtained by calculating the triangulation of the convex hull of the vertices of the chosen sampling scheme through the Qhull algorithm \cite{Barber96thequickhull}. Now, define $X_h^1$ to be the space of all the continuous, piecewise linear functions on $\Omega$
$$X_h^{1}=\{v_h: v_h\in C(\Omega): \left.v_h\right|_{\tau}\in\mathbb P^1\  \forall \tau\in \mathcal T_h\}$$
and set $V_h=X_h^1$. Since for the functions in $X_h^1$ the number of degrees of freedom is the same of the number of vertices of the mesh $n$, we need  $n$ basis functions $\phi_i, i=0,...,n-1$ to fully describe $X_h^{1}$. $\phi_i$ is defined as the continuous piecewise linear function such that
$$
\phi_i(x_j) = \delta_{ij}\quad i=0,...,n-1
$$
where $x_i$ are the points of the sampling scheme that have been used as vertices of the triangles $\tau$ of the triangulation $\mathcal T_h$. The support of $\phi_i$ i.e., the subset of $\mathcal T_h$ where $\phi_i$ is not zero, is made by all the triangles sharing the $i$th vertex. An example is shown in figure \ref{fig:basis function}.

\vspace{0.5cm}
\begin{remark}
	For a function $v_h \in X^1_h,\ v_h = v_0 \phi_0 +...  v_n \phi_n$ the coefficient $v_i$ is equal to the function $v_h$ evaluated in the $i$th vertex
	\begin{equation}\label{eq:dof and values}
	v_i = v_h( x_i)
	\end{equation}
\end{remark}\vspace{0.5cm}
\begin{figure}[h]
	\begin{center}
		\includegraphics[width=0.6\textwidth]{figs/Chapter3/sphere_mesh.png}
	\end{center}
	\caption{\label{fig:sphere mesh}A triangulation $\mathcal T_h$ of the sphere made with the vertices of the HEALPix sampling scheme with $N_{side}=8$.}
\end{figure}
\begin{figure}
	\begin{center}
		\includegraphics[width=0.4\textwidth]{figs/Chapter3/basisfunction.png}
	\end{center}
	\caption{\label{fig:basis function}Basis function of the space $X^{1}_h$}
\end{figure}

By writing $n$ times the equation (\ref{eq:weak eigenvalue problem}), setting each time the test function $v_h$ equal to the $i$th basis function $\phi_i$ of the space $X_h^1$, we obtain the generalized algebraic eigenvalue problem
\begin{equation}\label{eq:algebraic generalized eigenvalue problem}
\begin{aligned}
&\text{Find }(f,\lambda)\text{ such that }\mathbf A\mathbf f = \lambda\mathbf  B \mathbf f\\
&\begin{cases}
(\mathbf A)_{ij} &= \int_{\mathbb S^2}\nabla \phi_i(\mathbf{x})\cdot \nabla \phi_j(\mathbf{x})d\mathbf{x}\\
(\mathbf B)_{ij} &= \int_{\mathbb S^2} \phi_i(\mathbf{x}) \phi_j(\mathbf{x})d\mathbf{x}\\
(\mathbf f)_i &= f_i:\quad f(\mathbf x) = f_0\phi_0(\mathbf x)+ ... + f_{n-1}\phi_{n-1}(\mathbf x)
\end{cases}
\end{aligned}
\end{equation}
$\mathbf A$ is called the \textit{stiffness} matrix, and $\mathbf B$ is called the \textit{mass} matrix. Observe that being the Laplace-Beltrami operator self-adjoint, we have that $\mathbf A=\mathbf A^\intercal,$ $\mathbf B=\mathbf B^\intercal$. Being $\mathbf B$ non singular, the system (\ref{eq:algebraic generalized eigenvalue problem}) is equivalent to the eigenvalue problem
\begin{equation}\label{eq:algebraic  eigenvalue problem}
\mathbf B^{-1}\mathbf A\mathbf f = \lambda \mathbf f
\end{equation}

It can be shown \cite{Strang} that even though the matrix $\mathbf B^{-1}\mathbf A$ is not symmetric, its eigenvalues are still real and its eigenvectors are such that $\mathbf V\mathbf B\mathbf V^\intercal=\mathbf I$, where $\mathbf I$ is the identity matrix. The solution $\mathbf f$ that is the vector of the coefficients of the function $f_h$ in the basis $\phi_i$ corresponds exactly to the values of the function $f_h$ in the vertices.

\subsection{How to filter a signal with the linear FEM}
Calculating the discrete Fourier transform with the linear FEM means projecting the Fourier transform into the subspace $V_h$:

\begin{equation}\label{eq:FEM fourier}
	\hat f_{FEM} (\ell, m) = \int_{\eta \in \mathcal T_h} f_h(\eta) v_{i(\ell, m)}(\eta) d\eta = \mathbf v_{i(\ell, m)}^\intercal \mathbf B \mathbf f
\end{equation}

where $ v_{i(\ell, m)}$ is the solution to the eigenvalue problem (\ref{eq:algebraic generalized eigenvalue problem}), $f_h$ is the projection of $f$ on $V_h$, and $\mathbf B$ is the mass matrix. It follows that the filtering of a discretized signal $\mathbf f$ in the spectral domain is done by the following matrix  $\mathbf \Omega_K^{FEM}$:

\begin{equation}\label{eq:FEM filtering}
	 \mathbf \Omega_K^{FEM} := (\mathbf B\mathbf V^\intercal)^{-1}\mathbf K\mathbf V^\intercal\mathbf B
\end{equation}

where $\mathbf V^\intercal\mathbf B$ is the FEM Fourier transform matrix, $\mathbf K$ is the diagonal matrix that represent the chosen kernel for this filter, and $(\mathbf V^\intercal\mathbf B)^{-1}$ is the matrix representing the inverse FEM Fourier transform. We can already notice a fundamental difference between this way of filtering a signal and the way of filtering a signal with a graph. The FEM filtering uses not only the eigenvectors of the FEM Laplacian, but also the mass matrix $\mathbf B$. We computed the usual mean equivariance error per spherical harmonic degree for the diffusion filter $\mathbf K=\exp(-\mathbf \Lambda)$ on the equiangular sampling, and we show it in figure \ref{fig:FEM diffusion}.
\begin{figure}[h]
	\centering
	\includegraphics[width=\textwidth]{../codes/06.Equivariance_error/FEMdiffusiononequiangularsampling.png}
	\caption{\label{fig:FEM diffusion}Equivariance error of the linear FEM diffusion on the equiangular sampling by spherical harmonic degree.}
\end{figure}
We can see that the FEM filtering works better than the state-of-the-art graph of Khasanova-Frossard, but the matrix $\mathbf \Omega_K^{FEM}$ is full, and its computation requires the expensive inversion of the Fourier matrix $\mathbf V^\intercal\mathbf B$, making it not efficient if used in a CNN.
\subsubsection{A confront between FEM filtering and graph filtering}\label{sec:FEM filtering as a graph filtering}
Due to the fact that
$
\mathbf V^\intercal\mathbf B\mathbf V = \mathbf I
$,
we have that
$$
\begin{aligned}
	(\mathbf V^\intercal\mathbf B)^{-1} &= \mathbf V,\\
	\mathbf V^\intercal\mathbf B &= \mathbf V^{-1}.
\end{aligned}
$$
So, the FEM filter matrix $\mathbf \Omega_K^{FEM}$ can be rewritten as
\begin{equation}\label{eq:FEM filter simple}
	 \mathbf \Omega_K^{FEM}  = \mathbf V \mathbf K \mathbf V^{-1}
\end{equation}
Thus a polynomial filter
$$
k(\lambda)=P_\kappa(\lambda) = \sum_{k=0}^{\kappa-1} \theta_k \lambda^k
$$
would be implemented with a matrix
$$
\mathbf\Omega_K^{FEM} = \mathbf V \left(\sum_{k=0}^{\kappa-1} \theta_k \mathbf \Lambda^k \right)\mathbf V^{-1} = \sum_{k=0}^{\kappa-1} \theta_k (\mathbf B^{-1}\mathbf A)^k = P_\kappa(\mathbf B^{-1}\mathbf A).
$$

It is interesting to notice that FEM filtering (\ref{eq:FEM filter simple}) looks very much like graph filtering (\ref{eq:graph convolution}) but, given that $V$ is not orthogonal anymore, it has to replace the $\mathbf V^\intercal$ with $\mathbf V^{-1}$. Equation \ref{eq:FEM filter simple} implies that the FEM filtering could be implemented exactly as the graph filtering already implemented in DeepSphere, using the FEM matrix $\mathbf B^{-1}\mathbf A$ instead of the usual symmetric HKGL $\mathbf L_n^t$. Unfortunately, in order to make the evaluation of the polynomial $P_\kappa(\mathbf B^{-1}\mathbf A)$ efficient, we need the matrix $\mathbf B^{-1}\mathbf A$ to be sparse. The only way to do so is to have a \textit{sparse stiffness matrix} $\mathbf A$ and a \textit{diagonal mass matrix} $\mathbf B$. Unfortunately there is no likely way to have both these conditions satisfied at the same time, since this is one of the most well known trade offs of the FEM \cite{Strang}. To make $\mathbf B$ diagonal we need to choose $\phi_i$ to be an orthogonal basis of the finite dimensional functional space $V_h$. To do so, we will need to choose more complicated basis functions than the usual ones in figure \ref{fig:basis function}, \textit{most likely with a support that extends to the whole sphere}, making the stiffness matrix $\mathbf A$ not sparse anymore. A common workaround \cite{Strang} is the so called \textit{lumping} of the mass matrix, that consists in replacing the matrix $\mathbf B$ with the \textit{lumped} diagonal matrix $\mathbf D$ obtained by placing in each diagonal entry $(\mathbf D)_{ii}$ the sum of the elements of the $i$th row of the mass matrix $\mathbf B$:

\begin{equation}\label{eq:lumping}
\mathbf D = \text{diag}\{d_i\},\quad d_i = \sum_j (\mathbf B)_{ij}
\end{equation}
This approximation is well studied in the literature and it is proved to work well in many practical cases \cite{Quarteroni:1639539}. In this way the FEM matrix $\mathbf B^{-1}\mathbf A$ would be approximated by the matrix
$$
\mathbf D^{-1}\mathbf A.
$$
This matrix has the big advantage of having the same sparsity pattern of the stiffness matrix $\mathbf A$, making it very efficient for polynomial filtering. We can take one step further by using the symmetric matrix
$$
\mathbf D^{-1/2}\mathbf A\mathbf D^{-1/2}.
$$
To confront FEM and HKGL filtering we filter a unit mass signal \\$\mathbf{f}=(0,0,...0,1,0,...,0)^\intercal$ with the \textit{diffusion} filter $\exp{(-\tau \mathbf L)}$. It was chosen a very irregular sampling scheme, shown in figure \ref{fig:FEM lumped symmetric diffusion on irregular sampling}. It can be seen how the full HKGL compresses the signal around the equator where the sampling scheme is more sparse; on the other hand the FEM filtering manages to keep the diffusion homogeneous no matter the asymmetry in the sampling scheme.
The mean equivariance error using by spherical harmonic degree can be found in figure \ref{fig:Equivariance error of the lumped FEM Laplacian}. It can be seen how $\mathbf D^{-1}\mathbf A$ ha almost the same equivariance error of the full FEM Laplacian $\mathbf B^{-1}\mathbf A$, while using the symmetric matrix $\mathbf D^{-1/2}\mathbf A \mathbf D^{-1/2}$ makes the equivariance error much worse, especially at lower frequencies.
\clearpage
\begin{figure}[h!]
	\centering
	\includegraphics[width=\textwidth]{figs/Chapter3/diffusion.png}
	\caption{\label{fig:FEM lumped symmetric diffusion on irregular sampling}Symmetric lumped linear FEM diffusion and HKGL diffusion on an irregular sampling scheme of the sphere. The position of the filtered source signal is indicated with a black star.}
\end{figure}
\begin{figure}[h!]
	\centering
	\includegraphics[width=\textwidth]{../codes/06.Equivariance_error/LumpedFEMLaplacianonequiangularsampling.png}
	\includegraphics[width=\textwidth]{../codes/06.Equivariance_error/SymmetriclumpedFEMLaplacianonequiangularsampling.png}
	\caption{\label{fig:Equivariance error of the lumped FEM Laplacian}Equivariance error of the lumped FEM Laplacian and of the symmetric lumped FEM Laplacian on the equiangular sampling by spherical harmonic degree $\ell$.}
\end{figure}
\clearpage
\paragraph{Accuracy of the linear FEM spherical harmonics.} Thanks to the sampling theorem \ref{theo:equiangular sampling theorem}, we are able to compute the exact SHT (under the hypothesis of band limited signals) of the solutions to the eigenvalue problem (\ref{eq:algebraic generalized eigenvalue problem}) and we show in figures \ref{fig:FEMHealpix}, \ref{fig:FEMequiangular} the power spectrum of the eigenmodes for both HEALPix sampling and the equiangular sampling, as a measure of the goodness of the linear FEM approach to approximate the spherical harmonics.

\begin{figure}[h!]
	\centering
	\includegraphics[width=\linewidth]{../codes/03.FEM_laplacian/HEALPix/img/linearFEM.png}\\
	\includegraphics[width=\linewidth]{../codes/03.FEM_laplacian/HEALPix/img/linearFEM_diagonal.png}	\\
	\includegraphics[width=\linewidth]{../codes/03.FEM_laplacian/HEALPix/img/FEM_eigenvalues_16.png}	\\
	\caption{	\label{fig:FEMHealpix}Alignment of eigenspaces of the linear FEM Laplacian on HEALPix, and its spectrum for $N_{side}=16$}
\end{figure}
\begin{figure}[h!]
	\centering
	\includegraphics[width=\linewidth]{../codes/03.FEM_laplacian/equiangular/normal/img/linearFEM.png}
	\includegraphics[width=\linewidth]{../codes/03.FEM_laplacian/equiangular/normal/img/linearFEM_diagonal.png}
	\includegraphics[width=\linewidth]{../codes/03.FEM_laplacian/equiangular/normal/img/FEM_eigenvalues_16.png}
	\caption{\label{fig:FEMequiangular}Alignment of eigenspaces of the linear FEM Laplacian on the equiangular sampling, and its spectrum for $bw=32$}
\end{figure}

\clearpage


\subsubsection{Short Range Cusp Form}
\label{sec:onebodyjastrowsrcusp}

The idea behind this functor is to encode nuclear cusps and other details at very
short range around a nucleus in the region that the Gaussian orbitals of quantum
chemistry are not capable of describing correctly.
The functor is kept short ranged, because outside this small region, quantum chemistry
orbital expansions are already capable of taking on the correct shapes.
Unlike a pre-computed cusp correction, this optimizable functor can respond to
changes in the wave function during VMC optimization.
The functor's form is
\begin{equation}
\label{srcuspform}
u(r) = -\exp{\left(-r/R_0\right)} \left( A R_0 + \sum_{k=0}^{N-1} B_k \frac{ (r/R_0)^{k+2} }{ 1 + (r/R_0)^{k+2} } \right)
\end{equation}
in which $R_0$ acts as a soft cutoff radius ($u(r)$ decays to zero quickly beyond roughly this distance)
and $A$ determines the cusp condition.
\begin{equation}
\label{srcusplimit}
\lim_{r \to 0} \frac{\partial u}{\partial r} = A
\end{equation}
The simple exponential decay is modified by the $N$ coefficients $B_k$ that define
an expansion in sigmoidal functions, thus adding detailed structure in a short-ranged
region around a nucleus while maintaining the correct cusp condition at the nucleus.
Note that sigmoidal functions are used instead of, say, a bare polynomial expansion, as they
trend to unity past the soft cutoff radius and so interfere less with the exponential decay
that keeps the functor short ranged.
Although $A$, $R_0$, and the $B_k$ coefficients can all be optimized as variational
parameters, $A$ will typically be fixed as the desired cusp condition is known.

To specify this one-body Jastrow factor, use an input section like the following.

\begin{lstlisting}[style=QMCPXML]
<jastrow name="J1Cusps" type="One-Body" function="shortrangecusp" source="ion0" print="yes">
  <correlation rcut="6" cusp="3" elementType="Li">
    <var id="LiCuspR0" name="R0" optimize="yes"> 0.06 </var>
    <coefficients id="LiCuspB" type="Array" optimize="yes">
      0 0 0 0 0 0 0 0 0 0
    </coefficients>
  </correlation>
  <correlation rcut="6" cusp="1" elementType="H">
    <var id="HCuspR0" name="R0" optimize="yes"> 0.2 </var>
    <coefficients id="HCuspB" type="Array" optimize="yes">
      0 0 0 0 0 0 0 0 0 0
    </coefficients>
  </correlation>
</jastrow>
\end{lstlisting}

Here ``rcut'' is specified as the range beyond which the functor is assumed to be zero.
The value of $A$ can either be specified via the ``cusp'' option as shown above, in
which case its optimization is disabled, or through its own ``var'' line as
for $R_0$, in which case it can be specified as either optimizable (``yes'')
or not (``no'').
The coefficients $B_k$ are specified via the ``coefficients'' section,
with the length $N$ of the expansion determined automatically based on the length
of the array.

Note that this one-body Jastrow form can (and probably should) be used in conjunction
with a longer ranged one-body Jastrow, such as a spline form.
Be sure to set the longer-ranged Jastrow to be cusp-free!


% !TEX root = main.tex

%-------------------------------------------------
\section{Random points in the plane}\label{sec:joint}

Let $X$ and $Y$ be random variables on the same probability space.

% defn: joint distributions
\begin{definition}
\ben
\it The \emph{joint distribution} of $X$ and $Y$ is the function $\prob_{X,Y}$ defined on pairs of subsets of $\R$ by
\[
\prob_{X,Y}(C,D) = \prob(X\in C, Y\in D).
\]
\it The distributions $\prob_X(C)=\prob(X\in C)$ and $\prob_Y(D)=\prob(Y\in D)$ are called the \emph{marginal distributions} of $X$ and $Y$ respectively.
\een
\end{definition}

%The joint distribution of $X$ and $Y$ is completely determined by their joint CDF.
% defn: joint distributions
\begin{definition}
\ben
\it The \emph{joint CDF} of $X$ and $Y$ is the function $F_{X,Y}:\R^2\to[0,1]$ given by
\[
F_{X,Y}(x,y) = \prob(X\leq x,Y\leq y).
\]
\it The CDFs $F_X(x)=\prob(X\leq x)$ and $F_Y(y)=\prob(Y\leq y)$ are called the \emph{marginal CDFs} of $X$ and $Y$ respectively.
\een
\end{definition}

% defn: jointly discrete
\begin{definition}
\ben
\it $X$ and $Y$ are called \emph{jointly discrete} if the random vector $(X,Y)$ takes only countably many values in $\R^2$, in which case they are described by their \emph{joint PMF},
\[
\begin{array}{cccl}
f_{X,Y}:	& \R^2	& \to		& [0,1] \\
			& (x,y)	& \mapsto	& \prob(X=x,Y=y).
\end{array}
\]
\it The PMFs $f_X(x)=\prob(X=x)$ and $f_Y(y)=\prob(Y=y)$ are called the \emph{marginal PMFs} of $X$ and $Y$ respectively.
\een
\end{definition}

% defn: jointly continuous
\begin{definition}
\ben
\it $X$ and $Y$ are called \emph{jointly continuous} if their joint CDF can be written as
\[
F_{X,Y}(x,y) = \int_{-\infty}^x\int_{-\infty}^y f_{X,Y}(u,v)\,du\,dv \text{\qquad for all $x,y\in\R$,}
\]
for some integrable function $f_{X,Y}:\R^2\to[0,\infty)$ called the \emph{joint PDF} of $X$ and $Y$.
\it The PDFs $f_X(x)=F'_X(x)$ and $f_Y(y)=F'_Y(y)$ are called the \emph{marginal PDFs} of $X$ and $Y$ respectively, where $F_X(x)$ and $F_Y(y)$ are their marginal CDFs. Note that
\[
f_X(x) = \int_{-\infty}^{\infty}f_{X,Y}(x,y)\,dy
\text{\qquad and\qquad}
f_Y(y) = \int_{-\infty}^{\infty}f_{X,Y}(x,y)\,dx.
\]
\een

\end{definition}

%-----------------------------
\subsection{Independence}

Recall that two events $A$ and $B$ are \emph{independent} if $\prob(A\cap B)=\prob(A)\prob(B)$.

% defn: independence
\begin{definition}
Two random variables $X$ and $Y$ are said to be \emph{independent} if the events $\{X\leq x\}$ and $\{Y\leq y\}$ are independent for all $x,y\in\R$.
\end{definition}

\begin{lemma}\label{lem:product_marginal_cdfs}
\ben
\it $X$ and $Y$ are independent if and only if $F_{X,Y}(x,y) = F_X(x)F_Y(y)$ for all $x,y\in\R$.
\it If $X$ and $Y$ are jointly discrete, they are independent if and only if $f_{X,Y}(x,y) =f_X(x) f_Y(y)$ for all $x,y\in\R$.
\it If $X$ and $Y$ are jointly continuous, they are independent if and only if $f_{X,Y}(x,y) = f_X(x)f_Y(y)$ for all $x,y\in\R$ and $\supp(f_{X,Y})$ is a rectangular region in $\R^2$.
\een
%\ben
%\it $X$ and $Y$ are independent if and only if
%    \bit
%    \it $F_{X,Y}(x,y) = F_X(x)F_Y(y)$ for all $x,y\in\R$.
%    \eit
%\it If $X$ and $Y$ are jointly discrete, they are independent if and only if
%    \bit
%    \it $f_{X,Y}(x,y) =f_X(x) f_Y(y)\quad\text{for all}\quad x,y\in\R$.
%    \eit
%\it If $X$ and $Y$ are jointly continuous, they are independent if and only if
%    \bit
%    \it $f_{X,Y}(x,y) = f_X(x)f_Y(y)$ for all $x,y\in\R$, and
%    \it the support of $f_{X,Y}$ is a rectangular region in $\R^2$.
%    \eit
%\een
\end{lemma}
\begin{proof}
The first two parts follow directly from the definitions. For the jointly continuous case,
\bit
\it because $X$ and $Y$ are independent, $F_{X,Y}(x,y)=F_X(x)F_Y(y)$, and differentiating both sides with respect to $x$ and $y$ we get $f_{X,Y}(x,y)=f_X(x)f_Y(y)$.
\it If the value taken by $X$ affects the range of values taken by $Y$, then $Y$ clearly dependes on $X$. Thus for $X$ and $Y$ to be independent we need that
\[
\supp(f_{X,Y}) = \supp(f_X)\times \supp(f_Y),
\]
which is a rectangular region in $\R^2$.
\eit
\end{proof}

% example: discrete
\begin{exercise}\label{exc:joint_disc}
A fair die is rolled once. Let $\omega$ denote the outcome, and consider the random variables
\[
X(\omega) = \left\{\begin{array}{cl}
	1 & \text{ if $\omega$ is odd}, \\
	2 & \text{ if $\omega$ is even},
\end{array}\right.
\text{\quad and\quad}
Y(\omega) = \left\{\begin{array}{cl}
	1 & \text{ if $\omega\leq 3$}, \\
	2 & \text{ if $\omega\geq 4$}.
\end{array}\right.
\]
Find the joint PMF of $X$ and $Y$. Are $X$ and $Y$ independent?
\begin{solution}
\[
\begin{array}{c|cc|c}
		& Y=1 	& Y=2	& f_X	\\ \hline
X=1		& 1/3	& 1/6	& 1/2	\\
X=2		& 1/6 	& 1/3	& 1/2 	\\ \hline
f_Y		& 1/2	& 1/2	& 		\\
\end{array}
\]
$X$ and $Y$ are \emph{not} independent, because (for example)
\[
\prob(X=1,Y=1)\neq\prob(X=1)\prob(Y=1).
\]
\end{solution}
\end{exercise}

% example: continuous
\begin{example}
Let $X$ and $Y$ be jointly continuous random variables with the following joint PDF:
\[
f_{X,Y}(x,y) =
\left\{\begin{array}{ll}
	c(1-x)y	& \text{for } 0\leq y \leq x \leq 1, \\
	0		& \text{otherwise.}
\end{array}\right.
\]
\ben
\it Sketch the support of $f_{X,Y}$.
\it Are $X$ and $Y$ independent?
\it Find the marginal PDFs of $X$ and $Y$, and show that $c=24$.
%\it Show that $c=24$.
\een
\end{example}

\begin{solution}
\ben
\it % << (a)
$\supp(f_{X,Y})$ is the triangular region bounded by the lines $y=x$, $y=0$ and $x=1$.
\it % << (d)
The support of $f_{X,Y}$ is not a rectangular region in $\R^2$, so $X$ and $Y$ cannot be independent.
\it % << (c)
The marginal PDFs are:
\[\begin{array}{lll}
f_X(x)
	& = 24\int_0^x (1-x)y\,dy
	= 24(1-x)\left[\frac{y^2}{2}\right]_0^x
	& = \begin{cases}
		12x^2(1-x) & 0\leq x\leq 1, \\
		0			& \text{otherwise.}
		\end{cases} \\[3ex]
f_Y(y)
	& = 24\int_y^1 (1-x)y\,dx
	= 24y\left[x - \frac{x^2}{2}\right]_y^1
	& = \begin{cases}
		12y(1-y)^2 & 0\leq y\leq 1, \\
		0			& \text{otherwise.}
		\end{cases}
\end{array}\]
\it % << (b)
\bit
\it For fixed $x\in[0,1]$, we must have $0\leq y\leq x$.
\it For fixed $y\in[0,1]$, we must have $y\leq x\leq 1$.
\eit
\[
\int_{-\infty}^{\infty}\left(\int_{-\infty}^{\infty}f_{X,Y}(x,y)\,dy\right)\,dx
	= \int_0^1\int_0^x c(1-x)y\,dy\,dx
	= \frac{c}{2}\int_0^1 x^2-x^3\,dx
%	= \frac{c}{2}\left(\frac{1}{3}-\frac{1}{4}\right)
	= \frac{c}{24}.
\]
By the law of total probability this must equal $1$, so $c=24$.
\een
\end{solution}


\documentclass[paper.tex]{subfiles}

\usepackage{tikz}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{tabularx}
\usepackage{multicol}
\usepackage{algpseudocode}
\usepackage{algorithm}

% Add vertical spacing to tables
\renewcommand{\arraystretch}{1.4}

% Begin Document
\begin{document}

\section{$LU$ Factorization}

The first method to solve this system of equations will be $LU$ Factorization.
This involves decomposing the coefficient matrix $A$ into two factors, $L$ and $U$, resulting in an equation of the form $A = LU$.
Doing so gives:
\begin{equation*}
    \begin{bmatrix}
        1 & 0 & 0 & 0 & 0 \\
        \frac{1}{2} & 1 & 0 & 0 & 0 \\
        \frac{1}{2} & -1 & 1 & 0 & 0 \\
        0 & -2 & 1 & 1 & 0 \\
        0 & 2 & -\frac{3}{2} & -1 & 1
    \end{bmatrix}
    \begin{bmatrix}
        2 & 1 & 0 & 0 & 0 \\
        0 & \frac{1}{2} & -1 & 0 & -1 \\
        0 & 0 & -2 & 0 & -2 \\
        0 & 0 & 0 & 1 & 1 \\
        0 & 0 & 0 & 0 & 1
    \end{bmatrix}
\end{equation*}

Using the $LU$ decomposition it's possible to decompose the original matrix equation into two separate equations:
\begin{equation}
    Ux = y
\end{equation}
\begin{equation}
    Ly = b
\end{equation}

We must first solve for $y$ in equation (3).
\begin{equation}
    y = L^{-1}b
\end{equation}

This requires the inverse of $L$, which is:
\begin{equation*}
    L^{-1}=\begin{bmatrix}
        1 & 0 & 0 & 0 & 0 \\
        -\frac{1}{2} & 1 & 0 & 0 & 0 \\
        -1 & 1 & 1 & 0 & 0 \\
        0 & 1 & -1 & 1 & 0 \\
        -\frac{1}{2} & \frac{1}{2} & -\frac{1}{2} & 1 & 1
    \end{bmatrix}
\end{equation*}

Plugging in values for equation (4) gives:
\begin{equation*}
    y = \begin{bmatrix}
        1 & 0 & 0 & 0 & 0 \\
        -\frac{1}{2} & 1 & 0 & 0 & 0 \\
        -1 & 1 & 1 & 0 & 0 \\
        0 & 1 & -1 & 1 & 0 \\
        -\frac{1}{2} & \frac{1}{2} & -\frac{1}{2} & 1 & 1
    \end{bmatrix}
    \begin{bmatrix}
        100 \\
        0 \\
        -50 \\
        120 \\
        0
    \end{bmatrix}
    =\begin{bmatrix}
        100 \\
        -50 \\
        -150 \\
        170 \\
        45
    \end{bmatrix}
\end{equation*}

In order to compute for $x$, we rearrange equation (2) into:
\begin{equation}
    x = U^{-1}y
\end{equation}

Calculating the inverse of $U$ gives:
\begin{equation*}
    U^{-1}=\begin{bmatrix}
        \frac{1}{2} & -1 & \frac{1}{2} & 0 & 0 \\
        0 & 2 & -1 & 0 & 0 \\
        0 & 0 & -\frac{1}{2} & 0 & -1 \\
        0 & 0 & 0 & 1 & -1 \\
        0 & 0 & 0 & 0 & 1
    \end{bmatrix}
\end{equation*}

As given in equation (5), the product of the values of $U^{-1}$ and $y$ gives the solution $x$:
\begin{equation*}
    x = \begin{bmatrix}
        25 \\
        50 \\
        30 \\
        125 \\
        45
    \end{bmatrix}
\end{equation*}

\end{document}


% A simple template for LaTeX documents
%
% To produce pdf run:
%   $ pdflatex paper.tex
%

\documentclass[10pt, twocolumn]{article}
%\documentclass[12pt]{article}

% Begin paragraphs with new line
\usepackage{parskip}

% Change margin size
\usepackage[margin=0.5in]{geometry}

% Graphics Example:  (PDF's make for good plots)
\usepackage{graphicx}
% \centerline{\includegraphics{figure.pdf}}

% Allows hyperlinks
\usepackage{hyperref}

% Blocks of code
\usepackage{listings}
\lstset{basicstyle=\ttfamily, title=\lstname}
% Insert code like this. replace `plot.R` with file name.
% \lstinputlisting{plot.R}

% Monospaced fonts
%\usepackage{inconsolata}
% GNU \texttt{make} is a nice tool.

% Supports proof environment
\usepackage{amsthm}

% Allows writing \implies and align*
\usepackage{amsmath}

% Allows mathbb{R}
\usepackage{amsfonts}

% Numbers in scientific notation
\usepackage{siunitx}

% Use tables generated by pandas
\usepackage{booktabs}

% norm and infinity norm
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\inorm}[1]{\left\lVert#1\right\rVert_\infty}

% Statistics essentials
\newcommand{\iid}{\text{ iid }}
\newcommand{\Expect}{\operatorname{E}}
\newcommand{\Var}{\operatorname{Var}}
\newcommand{\Cov}{\operatorname{Cov}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\textbf{Binomial}
$X \sim B(n, p)$
\[
    p(k) = \binom{n}{k} p^k (1-p)^{n-k}
    \qquad k = 0, 1, \dots, n
\]
$\Expect X = np, \quad \Var X = np(1-p)$

mgf: $M_X (t) = (pe^t + 1 - p)^n$

Beta is conjugate prior, Fisher info $I(p) = \frac{1}{p(1 - p)}$

\textbf{Poisson}
$X \sim P(\lambda)$
\[
    p(k) = \frac{e^{-\lambda} \lambda^k}{k!}
    \qquad k = 0, 1, \dots
\]
$\Expect X = \lambda, \quad \Var X = \lambda$

mgf: $M_X (t) = e^{\lambda (e^t -1)}$

Gamma is conjugate prior, Fisher info $I(\lambda) = \frac{1}{\lambda}$

\textbf{Normal}
$X \sim N(\mu, \Sigma)$, $\Sigma$ positive definite
\[
    f(x) = \frac{\exp\{ - \frac{1}{2}(x - \mu)^T \Sigma^{-1} (x - \mu) \}}
        {(2\pi)^{\frac{k}{2}} \sqrt{\det(\Sigma)}}
        = \frac{1}{\sqrt{2 \pi} \sigma} e^{-\frac{(x - \mu)^2}{2
        \sigma^2}}
\]
mgf: $M_X (t) = \exp (\mu' t + \frac{1}{2} t' \Sigma t)$

Normal is conj. prior, Fisher info $I(\mu, \sigma^2) =
[\begin{smallmatrix}
        1 / \sigma^2 & 0 \\
        0 & 1 / 2\sigma^4 \\
\end{smallmatrix}]$

\textbf{Beta}
$ X \sim \text{Beta}(\alpha, \beta)$
\[
    f(x) = \frac{x^{\alpha-1}(1 - x)^{\beta-1}}{B(\alpha, \beta)}
    \qquad 0 \leq x \leq 1
\]
$\Expect X = \frac{\alpha}{\alpha + \beta},
\quad \Var X = \frac{\alpha \beta}{(\alpha + \beta)^2 (\alpha + \beta + 1)}$

using the beta function:
\[
    B(\alpha, \beta) =
    \frac{\Gamma(\alpha) \Gamma(\beta)}{\Gamma(\alpha+ \beta)} =
    \int_0^1 t^{\alpha -1} (1-t)^{\beta - 1}dt
\]

\textbf{Gamma}
$X \sim \text{Gamma}(\alpha, \beta)$
\[
    f(x) = \frac{\beta^\alpha x^{\alpha-1} e^{-\beta x}}{\Gamma(\alpha)}
    \qquad x > 0
\]
$\Expect X = \frac{\alpha}{\beta},
\quad \Var X = \frac{\alpha}{\beta^2}$

mgf: $M_X (t) = (1 - \frac{t}{\beta})^{-\alpha}, t < \beta$

$X \sim \text{Gamma}(\alpha, \beta) \iff \beta X \sim \text{Gamma}(\alpha, 1)$

$X_i \iid \text{Gamma}(\alpha_i, \beta)$, then
\[
    \sum X_i \sim \text{Gamma}(\sum \alpha_i, \beta)
\]
Gamma function: $\Gamma(\alpha) = \int_0^\infty t^{\alpha-1} e^{-t} dt$.

$\Gamma(\frac{1}{2}) = \sqrt{\pi}$.

$\Gamma(\alpha + 1) = \alpha \Gamma(\alpha)$

$\Gamma(k) = (k-1)!$ for $k$ positive integer.

\textbf{Exponential}
Special case: $\text{Exp}(\lambda) \equiv \text{Gamma}(1, \lambda)$

\textbf{Chi square}
Special case: $\chi^2_n \equiv \text{Gamma}(\frac{n}{2}, \frac{1}{2})$

Let $Z_i$ be iid $N(0, 1)$.

$\sum_{i=1}^n Z_i^2 \sim \chi^2_n$

Noncentral $\chi^2$. Let $Y \sim N(\mu, I)$ be an $n$ vector. Then
\[
    \norm{Y}^2 \sim \chi^2_n(\norm{\mu}^2)
\]

\textbf{F}
\[
    F(m, n) \equiv \frac{\frac{\chi^2_m}{m}}
        {\frac{\chi^2_n}{n}}
\]
Where numerator and denominator are independent $\chi^2$.

\textbf{T}
\[
    t(n) = \frac{N(0, 1)}
    {\sqrt{\frac{\chi^2_n}{n}}}
\]
Where numerator and denominator are independent.

\vspace{0.2in}
\hrule

Moment generating functions determine distribution
\[
    M_X(t) \equiv \Expect (e^{tX}),
    \quad M_X'(0) = \Expect(X)
\]
$X_i$ independently distributed $\iff$
\[
    M_{\sum X_i} (t) = \prod M_{X_i} (t)
\]
\textbf{Characteristic function}
\[
    \phi(t) = \Expect (e^{i t^T X)}
    = \Expect (\cos (t^T X)) + i \Expect(\sin(t^T X))
\]
Order statistics for sorted sample $X_{(1)}, \dots, X_{(n)}$ has pdf:
\[
    n! \prod_{i=1}^n f(X_{(i)}) \quad I(X_{(1)} < \dots < X_{(n)})
\]

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\textbf{Jensen's Inequality} if $S \subset R^k$ convex and closed, $g$ convex on $S$, $P[X
\in S] = 1$, and $\Expect X$ is finite, then $\Expect X \in S$, $\Expect
g(X)$ exists, and
\[
    \Expect g(X) \geq g(\Expect X)
\]
\textbf{Holder's Inequality} if $r, s > 1$ and $\frac{1}{r} + \frac{1}{s} =
1$ then
\[
    \Expect |XY| \leq (\Expect |X|^r)^{\frac{1}{r}}(\Expect |X|^s)^{\frac{1}{s}}
\]

$T(X)$ Sufficient means the distribution of $X | T(X)$ does not depend on
$\theta$.

Factorization theorem: $T(x)$ is sufficient $\iff$
\[
    f_{\theta}(x) = h(x) g(\theta, T(x))
\]
$L_x (\theta) = p_\theta (x) = p(x, \theta)$ likelihood is function of
$\theta$, density is function of $x$.

The likelihood ratio
\[
    \lambda_x (\theta) = \frac{L_x (\theta)}{L_x (\theta_0)}
\]
is minimal sufficient. To show $T(x)$ is minimal sufficient show that it is
sufficient and a function of the likelihood $\lambda_x (\theta)$.

\textbf{Fisher information}
\[
    I(\theta) = \Expect_\theta \left[ \frac{\partial}{\partial \theta}
    \log L_X (\theta) \right]^2
    = \Expect_\theta \left[ - \frac{\partial^2}{\partial \theta^2}
    \log L_X (\theta) \right]
\]
$\text{bias } \hat{v} \equiv \Expect (\hat{v}) - v$
\[
    MSE(\hat{v}) \equiv \Expect (\hat{v} - v)^2
    = \Var (\hat{v}) + (\text{bias } \hat{v})^2
\]
\textbf{Rao-Blackwell} Let $S(X)$ be an unbiased point estimator for
$g(\theta)$. Conditioning on a sufficient statistic $T(X)$ reduces
variance.
\[
    \Var_{\theta} (S(X)) \geq \Var_{\theta} (\Expect (S(X) | T(X)))
\]
Also holds for more general convex loss function $L$:
\[
    R(\theta, S) \equiv \Expect_{\theta} L(\theta, S(X)) \geq
    \Expect_{\theta} L(\theta, \Expect( S(X) | T(X)))
\]
\textbf{Completeness} $T(X)$ is complete if $\Expect g(T(X)) = 0$
implies $g = 0$ almost surely for all $\theta$.

\textbf{Cramer Rao Inequality} Let $g: \Theta \rightarrow R$. Suppose there
exists an unbiased estimator $U(X)$, $\Expect U(X) = g(\theta)$. Then
\[
    \Var_\theta U(X) \geq
    \left( \frac{\partial g(\theta)}{\partial \theta} \right)^T
    I(\theta)^{-1}
    \left( \frac{\partial g(\theta)}{\partial \theta} \right)
\]
Basu's Theorem - If $T(X)$ complete sufficient statistic and $A(X)$ is
ancillary then $A(X)$ and $T(X)$ are independent.

\textbf{Lehmann - Scheffe} Suppose $T(X)$ is complete sufficient. Then
there exists unique unbiased estimator $\Expect h(T(x))$ of $g(\theta) \in
R$ with smallest variance (MVUE).

\textbf{Exponential Families} $T(x)$ is natural sufficient statistic and is
complete sufficient if the $k$ parameter exponential family is full rank.
\[
    p(x, \theta) = h(x) \exp \{ \eta(\theta)^T T(x) - B(\theta) \}
\]
Canonical form model indexed by $\eta$.
\[
    q(x, \eta) = h(x) \exp \{ \eta^T T(x) - A(\eta) \}
\]
\[
    \dot{A} (\eta) = \Expect_\eta (T(X)) \quad
    \ddot{A} (\eta) = I(\eta) = \Var_\eta (T(X))
\]
Then moment generating function for $T(X)$ is
\[
    M_{T(X)}(t) = \exp \{A(t + \eta) - A(\eta) \}
\]
Equivalent statements useful for GLM's such as $Y \sim N(X \beta,
\sigma_0^2 I)$, where $Z$ is $n \times p$:

1. $I(\beta) = \frac{1}{\sigma_0^2} X^T X$ positive definite
2. rank$(X) = p$
3. model is identifiable. More generally another equivalent statement is
$\Var (T(X)) = \ddot{A} (\eta)$ is positive definite.

\subsection*{Decision Theory}

\textbf{Decision rule} $\delta: \mathcal{X} \rightarrow \mathcal{A}$, where
$\delta \in \mathcal{D}$, the space of possible decision rules and
$\mathcal{A}$ is the action space.

\textbf{Loss function} $l: \Theta \times \mathcal{A} \rightarrow \mathbb{R}^+$
Posterior mean minimizes square error loss; median minimizes absolute loss.

\textbf{Risk function} $R: \Theta \times \mathcal{D} \rightarrow
\mathbb{R}^+$ expected loss for a particular value of $\theta$
\[
    R(\theta, \delta) = \Expect_\theta l(\theta, \delta(X)) = \int
    l(\theta, \delta(x)) \cdot p_\theta (x) dx
\]
Bayes setup:
\[
    \pi(\theta | x) = \frac{p_\theta (x) \pi(\theta)}{m(x)}
\]
\textbf{Bayes decision rule} If there exists $\delta_\pi \in \mathcal{D}$
w.r.t prior $\pi$ such that
\[
    r(\pi, \delta_\pi) = \inf_{\delta \in \mathcal{D}} r(\pi, \delta)
\]
To find Bayes rule minimize the posterior risk:
\[
    \delta_\pi (x) = \min_{a \in \mathcal{A}} r_\pi (a | x)
\]

\textbf{Bayes risk} $r_\pi : \mathcal{D} \rightarrow
\mathbb{R}^+$ expected loss for fixed prior $\pi$
\[
    r_\pi (\delta) = \Expect_\pi R(\theta, \delta)
    = \int_\Theta R(\theta, \delta) \pi (d \theta)
    = \int_\mathcal{X} r_\pi(\delta(x) | x) m(x) dx
\]
To find Bayes risk: 1) find the Bayes rule 2) compute the risk function 3)
take the expectation of the risk wrt prior $\pi$.

\textbf{Minimax} decision rule $\delta^*$ minimizes the worst case
scenario, satisfies
\[
    \sup_{\theta} R(\theta, \delta^*) = \inf_{\delta} \sup_{\theta}
    R(\theta, \delta)
\]
To show $\delta^*$ is minimax, first check for constant risk $R(\theta,
\delta^* = c$ for all $\theta$, then find a
prior $\pi$ such that $\delta^*$ is the Bayes rule. This $\pi$ is least
favorable. More generally can find
a sequence of priors $(\pi_k)$ such that the Bayes risk $r_{\pi_k}
(\delta_{\pi_k}) \rightarrow c$.

\newpage

\subsection*{Multivariate Normal}

Stein's formula: $X \sim N(\mu, \sigma)$
\[
    \Expect (g(X) (X - \mu)) = \sigma^2 \Expect(g'(X))
\]
assuming these expectations are finite.

$X \sim N(\mu, \Sigma)$, $A$ an $m \times n$ matrix,
then
\[
    AX \sim N(A \mu, A \Sigma A^t)
\]
For $\Sigma$ full rank it's possible to transform between $Z \sim
N(0, I)$ and $X$:
\[
    X = \Sigma^{1/2} Z + \mu \qquad Z = \Sigma^{-1/2} (X - \mu)
\]
In block matrix form:
\[
    X =
    \begin{bmatrix}
        X_1 \\
        X_2 \\
    \end{bmatrix}
    \sim N \left(
    \begin{bmatrix}
        \mu_1 \\
        \mu_2 \\
    \end{bmatrix}
    ,
    \begin{bmatrix}
        \Sigma_{11} & \Sigma_{12} \\
        \Sigma_{21} & \Sigma_{22} \\
    \end{bmatrix}
\right)
\]
Assuming $\Sigma_{11}$ is positive definite then the conditional
distribution
\[
    X_2 | X_1 \sim N(\mu_2 + \Sigma_{21} \Sigma_{11}^{-1} (X_1 - \mu_1),
    \Sigma_{22} - \Sigma_{21} \Sigma_{11}^{-1} \Sigma_{12})
\]

\subsection*{Conditional Distributions}

Conditional pdf:
\[
    f_{X|Y}(x, y) \equiv \frac{f_{X, Y}(x, y)}{f_Y(y)}
\]
Iterated expectation:
\[
    E(Y) = E(E(Y | X))
\]
Conditional variance formula:
\[
    \Var(Y) = \Var(E(Y | X)) + E(\Var(Y | X))
\]

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection*{General Techniques}

Singular Value Decompostion (SVD) Any matrix $X$ can be written
\[
    X = UDV^T
\]
with $U, V$ orthogonal, and $D$ diagonal.

Moore Penrose Psuedoinverse $A^+$ exists uniquely for every matrix $A$.

Projection matrix $P$ are symmetric and idempotent. They have eigenvalues
either 0 or 1.
\[
    P = P^T \qquad P^2 = P
\]

Covariance of linear transformations
\[
    Cov(Ay, Bx) = A Cov(y, x) B^T
\]
$
    A =
    [\begin{smallmatrix}
        a & b \\
        c & d \\
    \end{smallmatrix}]
$
\[
    A^{-1} =
    \frac{1}{\det (A)}
    \begin{bmatrix}
        d & -b \\
        -c & a \\
    \end{bmatrix}
\]

Integration by parts:
\[
    \int uv' = uv - \int u'v
\]

Transformation theorem: Given some regularity conditions, $Y = g(X)$ has
pdf
\[
    p_Y(y) = p_X(g^{-1}(y)) |J_{g^{-1}(y)}|
\]

Matrix / Vector differentiation

$\frac{\partial A^T \beta}{\partial \beta} = A$,
$\frac{\partial \beta^T A \beta}{\partial \beta} = (A + A^t) \beta =
2A\beta$ for $A$ symmetric.

\end{document}


\title{Latent Space Models for Neural Data}

\subsection{Latent Space Models for Neural Data}

Many scientific fields involve the study of network data, including
social networks, networks in statistical physics, biological
networks, and information networks
\citep{newman2010networks,goldenberg2010survey}.

What we can learn about nodes in a network from their connectivity patterns?
We can begin to study this using a latent space model \citep{hoff2002latent}.
Latent space models embed nodes in the network in a latent space,
where the likelihood of forming an edge between two nodes depends on
their distance in the latent space.

We will analyze network data from neuroscience.
An interactive version with Jupyter notebook is available
\href{http://nbviewer.jupyter.org/github/blei-lab/edward/blob/master/notebooks/latent_space_models.ipynb}{here}.

\subsubsection{Data}

The data comes from \href{http://www-personal.umich.edu/~mejn/netdata/}{Mark Newman's repository}.
It is a weighted, directed network representing the neural network of
the nematode
\href{https://en.wikipedia.org/wiki/Caenorhabditis_elegans}{C.~Elegans}
compiled by \citet{watts1998collective} using experimental data
by \citet{white1986structure}.

The neural network consists of around $300$ neurons. Each connection
between neurons
is associated with a weight (positive integer) capturing the strength
of the connection.

First, we load the data.
\begin{lstlisting}[language=Python]
from observations import celegans

x_train = celegans("~/data")
\end{lstlisting}

\subsubsection{Model}

What can we learn about the neurons from their connectivity patterns? Using
a latent space model \citep{hoff2002latent}, we will learn a latent
embedding for each neuron to capture the similarities between them.

Each neuron $n$ is a node in the network and is associated with a latent
position $z_n\in\mathbb{R}^K$.
We place a Gaussian prior on each of the latent positions.

The log-odds of an edge between node $i$ and
$j$ is proportional to the Euclidean distance between the latent
representations of the nodes $|z_i- z_j|$. Here, we
model the weights ($Y_{ij}$) of the edges with a Poisson likelihood.
The rate is the reciprocal of the distance in latent space. The
generative process is as follows:

\begin{enumerate}
\item
For each node $n=1,\ldots,N$,
\begin{align}
z_n \sim N(0,I).
\end{align}
\item
For each edge $(i,j)\in\{1,\ldots,N\}\times\{1,\ldots,N\}$,
\begin{align}
Y_{ij} \sim \text{Poisson}\Bigg(\frac{1}{|z_i - z_j|}\Bigg).
\end{align}
\end{enumerate}

In Edward, we write the model as follows.
\begin{lstlisting}[language=Python]
from edward.models import Normal, Poisson

N = x_train.shape[0]  # number of data points
K = 3  # latent dimensionality

z = Normal(loc=tf.zeros([N, K]), scale=tf.ones([N, K]))

# Calculate N x N distance matrix.
# 1. Create a vector, [||z_1||^2, ||z_2||^2, ..., ||z_N||^2], and tile
# it to create N identical rows.
xp = tf.tile(tf.reduce_sum(tf.pow(z, 2), 1, keep_dims=True), [1, N])
# 2. Create a N x N matrix where entry (i, j) is ||z_i||^2 + ||z_j||^2
# - 2 z_i^T z_j.
xp = xp + tf.transpose(xp) - 2 * tf.matmul(z, z, transpose_b=True)
# 3. Invert the pairwise distances and make rate along diagonals to
# be close to zero.
xp = 1.0 / tf.sqrt(xp + tf.diag(tf.zeros(N) + 1e3))

x = Poisson(rate=xp)
\end{lstlisting}

\subsubsection{Inference}

Maximum a posteriori (MAP) estimation is simple in Edward. Two lines are
required: Instantiating inference and running it.
\begin{lstlisting}[language=Python]
inference = ed.MAP([z], data={x: x_train})
\end{lstlisting}

See this extended tutorial about
\href{/tutorials/map}{MAP estimation in Edward}.

One could instead run variational inference. This requires specifying
a variational model and instantiating \texttt{KLqp}.
\begin{lstlisting}[language=Python]
qz = Normal(loc=tf.Variable(tf.random_normal([N * K])),
            scale=tf.nn.softplus(tf.Variable(tf.random_normal([N * K]))))
inference = ed.KLqp({z: qz}, data={x: x_train})
\end{lstlisting}
See this extended tutorial about
\href{/tutorials/variational-inference}{variational inference in Edward}.

Finally, the following line runs the inference procedure for 2500
iterations.
\begin{lstlisting}[language=Python]
inference.run(n_iter=2500)
\end{lstlisting}

\subsubsection{Acknowledgments}

We thank Maja Rudolph for writing the initial version of this
tutorial.

\subsubsection{References}


\chapter{The Two- and Three-Dimensional Navier-Stokes Equations}

%%%%%%%
%Section
%%%%%%%
\section{Background}
The Navier-Stokes equations describe the motion of a fluid. In order to derive the Navier-Stokes equations we assume that a fluid is a continuum (not made of individual particles, but rather a continuous substance) and that mass and momentum are conserved. After making some assumptions and using Newton's second law on an incompressible fluid particle, the Navier-Stokes equations can be derived in their entirety. All details are omitted since there are many sources of this information, two sources that are particularly clear are Tritton~\cite{Tri88} and Doering and Gibbon~\cite{DoeGib95}; Gallavotti~\cite{Gal02} should also be noted for introducing both mathematical and physical aspects of these equations, and Uecker~\cite{Uec09} includes a quick derivation and some example Fourier Spectral Matlab codes. For a more detailed introduction to spectral methods for the Navier-Stokes equations see Canuto et al.~\cite{CHQZ07}. The incompressible Navier-Stokes equations  are
\begin{eqnarray}
\rho \left (\frac{\partial \mathbf{u}}{\partial t} + \mathbf{u} \cdot \nabla \mathbf{u} \right)&=&-\nabla p + \mu \Delta \mathbf{u} + \mathbf{f} \label{eq:NseConMom}\\
\nabla \cdot \mathbf{u} &=& 0. \label{eq:NseConMas}
\end{eqnarray}
In these equations, $\rho$ is density, $\mathbf{u}(x,y,z)=(u,v,w)$ is the velocity with components in the $x$, $y$ and $z$ directions, $p$ is pressure field, $\mu$ is dynamic viscosity (constant in incompressible case) and $\mathbf{f}$ is a body force (force that acts through out the volume). Equation \eqref{eq:NseConMom} represents conservation of momentum and eq.\ \eqref{eq:NseConMas} is the continuity equation which represents conservation of mass for an incompressible fluid.

%%%%%%%
%Section
%%%%%%%
\section{The Two-Dimensional Case}

We will first consider the two-dimensional case. A difficulty in simulating the incompressible Navier-Stokes equations is the numerical satisfaction of the incompressibility constraint in eq.\ \eqref{eq:NseConMas}, this is sometimes referred to as a divergence free condition or a solenoidal constraint. To automatically satisfy this incompressibility constraint in two dimensions, where
$$\mathbf u(x,y)=\left(u(x,y),v(x,y)\right)$$
it is possible to re-write the equations using a different formulation, the stream-function vorticity formulation. In this case, we let
$$u=\frac{\partial \psi}{\partial y}  \quad v=-\frac{\partial \psi}{\partial x},$$
where $\psi(x,y)$ is the streamfunction.  Level curves of the streamfunction represent streamlines\footnote{A streamline is a continuous curve along which the instantaneous velocity is tangent, see Tritton~\cite{Tri88} for more on this.} of the fluid field. Note that
$$\nabla\cdot\mathbf u =\frac{\partial u}{\partial x}+\frac{\partial v}{\partial y}= \frac{\partial^2\psi}{\partial x \partial y} - \frac{\partial^2\psi}{\partial y \partial x}=0,$$
so eq.\ \eqref{eq:NseConMas} is automatically satisfied. Making this change of variables, we obtain a single scalar partial differential equation by taking the curl of the momentum equation, eq.\ \eqref{eq:NseConMom}. We define the vorticity $\omega$, so that
$$\omega=\nabla\times\mathbf u= \frac{\partial v}{\partial x}-\frac{\partial u}{\partial y}=-\Delta \psi$$
and eq.\ \eqref{eq:NseConMom} becomes
\begin{align*}
&{}\hspace{2em}
\frac{\partial }{\partial x} \left[\rho\left(\frac{\partial v}{\partial t} +u \frac{\partial v}{\partial x} +v\frac{\partial v}{\partial y}\right) \right]
- \frac{\partial }{\partial y} \left[\rho\left(\frac{\partial u}{\partial t} +u \frac{\partial u}{\partial x} +v\frac{\partial u}{\partial y}\right) \right]
\\&{}=\frac{\partial }{\partial x} \left[\mu\left(\frac{\partial^2 v}{\partial x^2} + \frac{\partial^2 v}{\partial y^2}\right) + fy \right]
-\frac{\partial }{\partial y} \left[\mu\left(\frac{\partial^2 u}{\partial x^2} + \frac{\partial^2 u}{\partial y^2}\right) + fx \right]
\end{align*}
where $fx$ and $fy$ represent the $x$ and $y$ components of the force $\mathbf f$. Since the flow is divergence free,
$$\frac{\partial u}{\partial x}=-\frac{\partial v}{\partial x}$$
and so can simplify the nonlinear term to get
\begin{align*}
&{}\hspace{2em}\frac{\partial }{\partial x} \left[\left( u\frac{\partial v}{\partial x} +v\frac{\partial v}{\partial y}\right) \right]
- \frac{\partial }{\partial y} \left[\left(u \frac{\partial u}{\partial x} +v\frac{\partial u}{\partial y}\right) \right]
\\&{}=\frac{\partial u}{\partial x}\frac{\partial v}{\partial x}+u\frac{\partial^2 v}{\partial x^2 } +\frac{\partial v}{\partial x}\frac{\partial v}{\partial y} + v\frac{\partial^2v }{\partial x \partial y}
-\frac{\partial u}{\partial y}\frac{\partial u}{\partial x} - u\frac{\partial^2u }{\partial x\partial y} -\frac{\partial v}{\partial y}\frac{\partial u}{\partial y} -v \frac{\partial^2u }{\partial y^2 }
\\&{}= u\left( \frac{\partial^2 v}{\partial x^2 } - \frac{\partial^2u }{\partial x\partial y} \right) + v\left( \frac{\partial^2 v}{\partial x \partial y } - \frac{\partial^2u }{\partial^2 y} \right).
\end{align*}
We finally obtain
\begin{align} \label{eq:Ns2dStrVorA}
&{}\hspace{1em}\rho\left(\frac{\partial \omega}{\partial t} + u\frac{\partial \omega}{\partial x} + v\frac{\partial \omega }{\partial y}\right)
 =\mu\Delta\omega+\frac{\partial fy}{\partial x}- \frac{\partial fx}{\partial y}
 \end{align}
and
\begin{align} \label{eq:Ns2dStrVorB}
&{}\Delta \psi = -\omega.
\end{align}
Note that in this formulation, the Navier-Stokes equation is like a forced heat equation for the vorticity with a nonlocal and nonlinear term. We can take advantage of this structure in finding numerical solutions by modifying our numerical programs which give approximate solutions to the heat equation.

A simple time discretization for this equation is the Crank-Nicolson method, where the nonlinear terms are solved for using fixed point iteration. A tutorial on convergence of time discretization schemes for the Navier-Stokes equations can be found in Temam~\cite{Tem01}. The time discretized equations become
\begin{align} \label{eq:Ns2dStrVorACn}
&{}\hspace{2em}\rho\left[\frac{\omega^{n+1,k+1}-\omega^n}{\delta t} \right.
\\&{}\left.+\frac{1}{2} \left( u^{n+1,k}\frac{\partial \omega^{n+1,k}}{\partial x} + v^{n+1,k}\frac{\partial \omega^{n+1,k} }{\partial y}
+ u^{n}\frac{\partial \omega^{n}}{\partial x} + v^{n}\frac{\partial \omega^{n} }{\partial y} \right)\right] \notag
\\&{}=\frac{\mu}{2}\Delta\left(\omega^{n+1,k+1}+\omega^n\right) + \left.\left(\frac{\partial fx}{\partial y}-\frac{\partial fy}{\partial x} \right)\right\rvert_{t=(n+0.5)\delta t},\notag
\end{align}
and
\begin{align} \label{eq:Ns2dStrVorBCn}
&{}\Delta \psi^{n+1,k+1} = -\omega^{n+1,k+1}, \quad u^{n+1,k+1}=\frac{\partial \psi^{n+1,k+1}}{\partial y}, \quad v^{n+1,k+1}=-\frac{\partial \psi^{n+1,k+1}}{\partial x}.
\end{align}
In these equations, the superscript $n$ denotes the timestep and the superscript $k$ denotes the iterate. Another choice of time discretization is the implicit midpoint rule which gives,
\begin{align} \label{eq:Ns2dStrVorAimr}
&{}\hspace{2em}\rho\left[\frac{\omega^{n+1,k+1}-\omega^n}{\delta t} \right.
\\&{}\left.+ \left(\frac{u^{n+1,k}+u^n}{2}\right)\frac{\partial}{\partial x}\left(\frac{\omega^{n+1,k}+\omega^n}{2}\right) + \left(\frac{v^{n+1,k}+v^n}{2}\right)\frac{\partial}{\partial y}\left(\frac{ \omega^{n+1,k} +\omega^n}{2}\right)\right] \notag
\\&{}=\frac{\mu}{2}\Delta\left(\omega^{n+1,k+1}+\omega^n\right) + \left.\left(\frac{\partial fx}{\partial y}-\frac{\partial fy}{\partial x} \right)\right\rvert_{t=(n+0.5)\delta t},\notag
\end{align}
and
\begin{align} \label{eq:Ns2dStrVorBimr}
&{}\Delta \psi^{n+1,k+1} = -\omega^{n+1,k+1}, \quad u^{n+1,k+1}=\frac{\partial \psi^{n+1,k+1}}{\partial y}, \quad v^{n+1,k+1}=-\frac{\partial \psi^{n+1,k+1}}{\partial x}.
\end{align}

%%%%%%%
%Section
%%%%%%%
\section{The Three-Dimensional Case}

Here $\mathbf u = (u(x,y,z,t),v(x,y,z,t),w(x,y,z,t))$ -- unfortunately, it is not clear if this equation has a unique solution for reasonable boundary conditions and initial data. Numerical methods so far seem to indicate that the solution is unique, but in the absence of a proof, we caution the reader that we are {\it fearless engineers writing gigantic codes that are supposed to produce solutions to the Navier-Stokes equations when what we are really studying is the output of the algorithm} which we hope will tell us something about these equations\footnote{This is paraphrased from Gallavoti\cite[p.~VIII]{Gal02}} -- in practice, although the mathematical foundations for this are uncertain, these codes do seem to give information about the motion of nearly incompressible fluids in many, although not all situations of practical interest. Further information on this aspect of these equations can be found in Doering and Gibbon~\cite{DoeGib95}.

We will again consider simulations with periodic boundary conditions to make it easy to apply the Fourier transform. This also makes it easier to enforce the incompressibility constraint by using an idea due to Orszag and Patterson~\cite{OrsPat72} and also explained in Canuto et al.~\cite[p.~99]{CHQZ07}. If we take the divergence of the Navier-Stokes equations, we get
\begin{align}
\nabla\cdot \left(\mathbf u\cdot\nabla\mathbf u\right)=-\Delta p
\end{align}
because $\nabla\cdot\mathbf u=0$. Hence
\begin{align}
&{}p=-\Delta^{-1}\left[\nabla\cdot \left(\mathbf u\cdot\nabla\mathbf u\right)\right]
\end{align}
where $\Delta^{-1}$ is defined using the Fourier transform, thus if $f(x,y,z)$ is a mean zero, periodic scalar field and $\hat{f}$ is its Fourier transform, then
$$\widehat{\Delta^{-1}f}=\frac{\hat{f}}{k_x^2+k_y^2+k_z^2}$$
where $k_x$, $k_y$ and $k_z$ are the wavenumbers. The Navier-Stokes equations then become
\begin{align}
&{}\frac{\partial \mathbf u}{\partial t}=\frac{1}{\textup{Re}}\Delta \mathbf u - \mathbf u\cdot\nabla\mathbf u +\nabla \Delta^{-1}\left[\nabla\cdot \left(\mathbf u\cdot\nabla\mathbf u\right)\right], \label{eq:NS3dOrsPat}
\end{align}
for which the incompressibility constraint is satisfied, provided the initial data satisfy the incompressibility constraint.

To discretize \eqref{eq:NS3dOrsPat} in time, we will use the implicit midpoint rule. This gives,
\begin{align}
\hspace{1em}\frac{\mathbf u^{n+1}-\mathbf u^{n}}{\delta t} =&{}\frac{0.5}{\textup{Re}}\Delta\left( \frac{\mathbf u^{n+1}+\mathbf u^{n}}{2}\right)  - 0.25\left(\mathbf u^{n+1} + \mathbf u^{n}\right)\cdot\nabla\left(\mathbf u^{n+1} + \mathbf u^{n} \right)  \notag
\\&{}\hspace{1em} + 0.25\nabla\left[ \Delta^{-1}\left(\nabla\cdot \left[\left(\mathbf u^{n+1} + \mathbf u^{n}\right) \cdot\nabla\left(\mathbf u^{n+1} + \mathbf u^{n} \right)\right]\right)\right]. \label{eq:NS3dOrsPatIMR}
\end{align}

It is helpful to test the correctness of the programs by comparing them to an exact solution. Shapiro~\cite{Sha93} has found the following exact solution which is a good test for meteorological hurricane simulation programs, as well as for Navier-Stokes solvers with periodic boundary conditions
\begin{align*}
&{} u= -\frac{A}{k^2+l^2}\left[\lambda l\cos(kx)\sin(ly)\sin(mz)+mk\sin(kx)\cos(ly)\cos(mz) \right]\exp\left(-\frac{\lambda^2t}{\textup{Re}}\right)
\\&{} v= \frac{A}{k^2+l^2}\left[\lambda k\sin(kx)\cos(ly)\sin(mz)-ml\cos(kx)\sin(ly)\cos(mz) \right]\exp\left(-\frac{\lambda^2t}{\textup{Re}}\right)
\\&{} w= A\cos(kx)\cos(ly)\sin(mz)\exp\left(-\frac{\lambda^2t}{\textup{Re}}\right)
\end{align*}
where the constant $\lambda=\sqrt{k^2+l^2+m^2}$ and $l$, $k$ and $m$ are constants choosen with the restriction that the solutions are periodic in space. Further examples of such solutions can be found in Majda and Bertozzi~\cite[sec.~2.3]{MajBer02}.

%%%%%%%
%Section
%%%%%%%
\section{Serial Programs}
We first write Matlab programs to demonstrate how to solve these equations on a single processor. The first program uses Crank-Nicolson timestepping to solve the two-dimensional Navier-Stokes equations and is in listing \ref{lst:Ns2DMatlab}. To test the program, following Laizet and Lamballais\cite{LaiLam09} we use the exact Taylor-Green vortex solution on $(x,y)\in[0,1]\times[0,1]$ with periodic boundary conditions given by
\begin{align}
&{}u(x,y,t)=\sin(2\pi x)\cos(2\pi y)\exp(-8\pi^2\mu t)
\\&{}v(x,y,t)=-\cos(2\pi x)\sin(2\pi y)\exp(-8\pi^2\mu t).
\end{align}

\lstinputlisting[style=matlab_style,label=lst:Ns2DMatlab,caption={A Matlab program which finds a numerical solution to the 2D Navier Stokes equation.}]{./NavierStokes/Programs/NsMatlabCn/NavierStokes2DFFTCn.m}

The second program uses the implicit midpoint rule to do timestepping for the three-dimensional Navier-Stokes equations and it is in listing \ref{lst:Ns3DMatlab}. It also takes the Taylor-Green vortex as its initial condition since this has been extensively studied, and so provides a baseline case to compare results against.

\lstinputlisting[style=matlab_style,label=lst:Ns3DMatlab,caption={A Matlab program which finds a numerical solution to the 3D Navier Stokes equation.}]{./NavierStokes/Programs/NsMatlab3Dimr/NavierStokes3DfftIMR.m}

% subsection
\subsection{Exercises}
\begin{enumerate}
\item[1)] Show that for the Taylor-Green vortex solution, the nonlinear terms in the two-dimensional Navier-Stokes equations cancel out exactly.
\item[2)] Write a Matlab program that uses the implicit midpoint rule instead of the Crank-Nicolson method to obtain a solution to the 2D Navier-Stokes equations. Compare your numerical solution with the Taylor-Green vortex solution.
\item[3)] Write a Fortran program that uses the implicit midpoint rule instead of the Crank-Nicolson method to obtain a solution to the 2D Navier-Stokes equations. Compare your numerical solution with the Taylor-Green vortex solution.
\item[4)] Write a Matlab program that uses the Crank-Nicolson method instead of the implicit midpoint rule  to obtain a solution to the 3D Navier-Stokes equations.
\item[5)] Write a Fortran  program that uses the Crank-Nicolson method instead of the implicit midpoint rule  to obtain a solution to the 3D Navier-Stokes equations.
\item[6)] The Navier-Stokes equations as written in eqs.\ \eqref{eq:Ns2dStrVorA} and \eqref{eq:Ns2dStrVorB} also satisfy further integral properties. In particular show that
\begin{enumerate}
\item[a)] $$\frac{\rho}{2}\frac{\mathrm{d}}{\mathrm{d}t}\lVert \omega \rVert_{l^2}^2= -\mu\lVert \nabla \omega \rVert_{l^2}^2,$$
where
$$\lVert \omega \rVert_{l^2}^2=\int\int(\omega)^2\mathrm{d}x\mathrm{d}y$$
and
$$\lVert \nabla \omega \rVert_{l^2}^2=\int\int(\nabla\omega)\cdot(\nabla\omega)\mathrm{d}x\mathrm{d}y.$$
HINT: multiply the Eq.\ \eqref{eq:Ns2dStrVorA} by $\omega$ then integrate by parts.
\item[b)] Show that part (a) implies that
$$\lVert \omega(t=T) \rVert_{l^2}^2-\lVert \omega(t=0) \rVert_{l^2}^2=-\mu\int_0^T\lVert \nabla \omega \rVert_{l^2}^2\mathrm{d}t$$
\item[c)] Part (b) gives a property one can check when integrating the 2D Navier-Stokes equations. We now show that the implicit midpoint rule satisfies an analogous property. Multiply eq.\ \eqref{eq:Ns2dStrVorAimr} by $0.5(\omega^{n+1}+\omega^n)$, integrate by parts in space, then sum over time to deduce that
$$\lVert \omega^N \rVert_{l^2}^2-\lVert \omega^0 \rVert_{l^2}^2=-\frac{\mu}{4}\sum_{n=0}^{N-1}\left\lVert \nabla\left( \omega^n+\omega^{n+1}\right) \right\rVert_{l^2}^2\delta t.$$
\item[d)] Deduce that this implies that the implicit midpoint rule time stepping method is unconditionally stable, provided the nonlinear terms can be solved for\footnote{We have not demonstrated convergence of the spatial discretization, so this result assumes that the spatial discretization has not been done.}.
\end{enumerate}
\end{enumerate}

%%%%%%%
%Section
%%%%%%%
\section{Parallel Programs: OpenMP}

Rather than give fully parallelized example programs, we instead give a simple implementation in Fortran of the Crank-Nicolson  and implicit midpoint rule algorithms for the two-dimensional and three dimensional Navier-Stokes equations that were presented in Matlab. The program for the two-dimensional equations is presented in listing \ref{lst:For2dNs} and an example Matlab script to plot the resulting vorticity fields is in listing \ref{lst:MatlabPlot2dNS}. This program is presented in listing \ref{lst:For3dNs} and an example Matlab script to plot the resulting vorticity fields is in listing \ref{lst:MatlabPlot3dNS}.

\lstinputlisting[style=fortran_style,language=Fortran,label=lst:For2dNs,caption={A Fortran program to solve the 2D Navier-Stokes equations.}]{./NavierStokes/Programs/NavierStokes2dFortran/navierstokes.f90}

\lstinputlisting[style=matlab_style,label=lst:MatlabPlot2dNS,caption={A Matlab program to plot the vorticity fields and error produced by listing \ref{lst:For2dNs}.}]{./NavierStokes/Programs/NavierStokes2dFortran/plotdata.m}

\lstinputlisting[style=fortran_style,language=Fortran,label=lst:For3dNs,caption={A Fortran program to solve the 3D Navier-Stokes equations.}]{./NavierStokes/Programs/NavierStokes3dFortran/NavierStokes3DfftIMR.f90}

\lstinputlisting[style=matlab_style,label=lst:MatlabPlot3dNS,caption={A Matlab program to plot the vorticity fields produced by listing \ref{lst:For3dNs}.}]{./NavierStokes/Programs/NavierStokes3dFortran/plotdata.m}

\subsection{Exercises}
\begin{enumerate}
\item[1)] Verify that the program in listing \ref{lst:For2dNs} is second order accurate in time.
\item[2)] Use OpenMP directives to parallelize the example Fortran code for the two-dimensional Navier Stokes equations. Try and make it as efficient as possible.
\item[3)] Write another code which uses threaded FFTW to do the Fast Fourier transforms. This code should have a similar structure to the program in listing \ref{lst:For2dNlsOmp2}.
\item[4)] Use OpenMP directives to parallelize the example Fortran code for the three-dimensional Navier-Stokes equations in listing \ref{lst:For3dNs}. Try and make it as efficient as possible.
\item[5)] Write another code which uses threaded FFTW to do the Fast Fourier transforms for the three-dimensional Navier-Stokes equations. This code should have a similar structure to the program in listing \ref{lst:For2dNlsOmp2}.
\end{enumerate}
%%%%%%%
%Section
%%%%%%%
\section{Parallel Programs: MPI}

The code for this is very similar to the serial code in listing \ref{lst:For2dNs}. For completeness and to allow one to see how to parallelize other programs, we include it. The program uses the library 2DECOMP\&FFT. One difference between this program and the serial program is that a subroutine is included to write out data. Since this portion of the calculation is repeated several times, the program becomes more readable when the repeated code is placed in a subroutine. The subroutine is also generic enough that it can be reused in other programs, saving program developers time.

\lstinputlisting[style=fortran_style,language=Fortran,label=lst:For3dNsMPI,caption={A parallel MPI Fortran program to solve the 3D Navier-Stokes equations.}]{./NavierStokes/Programs/NavierStokes3dFortranMPI/NavierStokes3DfftIMR.f90}

\lstinputlisting[style=fortran_style,language=Fortran,label=lst:For3dNsMPIdata,caption={A subroutine to save real array data for the parallel MPI Fortran program to solve the 3D Navier-Stokes equations in listing \ref{lst:For3dNsMPI}.}]{./NavierStokes/Programs/NavierStokes3dFortranMPI/savedata.f90}

\lstinputlisting[style=make_style,language=make,label=lst:For3dNsMPImake,caption={A makefile to compile the parallel MPI Fortran program to solve the 3D Navier-Stokes equations.}]{./NavierStokes/Programs/NavierStokes3dFortranMPI/makefile}

\subsection{Exercises}
\begin{enumerate}
\item[1)] Use 2DECOMP{\&}FFT to write a two dimensional Navier-Stokes solver. The library is built to do three dimensional FFTs, however by choosing one of the arrays to have only one entry, the library can then do two dimensional FFTs on a distributed memory machine.
\item[2)] Uecker~\cite{Uec09} describes the expected power law scaling for the power spectrum of the enstrophy\footnote{The enstrophy is the square of the vorticity.} in two dimensional isotropic turbulence. Look up Uecker~\cite{Uec09} and then try to produce numerical data which verifies the power scaling law over as many decades of wavenumber space as are feasible on the computational resources you have access to. A recent overview of research work in this area can be found in Boffetta and Ecke~\cite{BofEck12}. Fornberg~\cite{For77} discusses how to calculate power spectra.
\item[3)] If we set $\mu=0$ the Navier Stokes equations become the Euler equations. Try to use the implicit midpoint rule and/or the Crank-Nicolson methods to simulate the Euler equations in either two or three dimensions. See if you can find good iterative schemes to do this, you may need to use Newton iteration. An introduction to the Euler equations is in Majda and Bertozzi~\cite{MajBer02}.
\item[4)] The Taylor-Green vortex flow initial conditions have been studied as a possible flow that could have a blow up in the maximum value of the absolute value of the gradient of the velocity at a point for the Euler and Navier-Stokes equations. In many of these simulations, symmetries have been used to get higher effective resolutions, see for example Cichowlas and Brachet~\cite{CicBra05}. Consider using the Kida-Pelz and Taylor-Green vortex as initial conditions for the Euler equations and adding non-symmetric perturbations. If you are unable to get an implicit time-stepping scheme to work, consider using an explicit scheme such as a Runge-Kutta method. How does the flow evolve in comparison to previous studies in the literature? An introduction to the blow up for the Euler equations is in Majda and Bertozzi~\cite{MajBer02}.
\item[5)] The three dimensional program we have written is not the most efficient since one can use a real to complex transform to halve the work done. Implement a real to complex transform in one of the Navier-Stokes programs.
\item[6)] The programs we have written can also introduce some aliasing errors.  By reading a book on spectral methods, such as Canuto et al.~\cite{CHQZ07}, find out what aliasing errors are. Explain why the strategy explained in Johnstone~\cite{Joh12} can reduce aliasing errors.
\end{enumerate}