\RequirePackage[l2tabu, orthodox]{nag} % Complaints if syntax isn't right.
\documentclass[10pt]{extarticle}

\usepackage[utf8]{inputenc}
\usepackage{extsizes} % Allow for more sizes such as 14pt or 17pt on document class.
\usepackage{mathtools} % Allows conditional math expressions, etc.
\usepackage{amsfonts} % Allows \mathbb{R}, which is the real numbers symbol.
\usepackage[export]{adjustbox} % Wraps \includegraphics with more keys (options).

% For writing pseudocode snippets: http://ctan.mackichan.com/macros/latex/contrib/algorithmicx/algorithmicx.pdf
\usepackage{algorithm}
\usepackage{algpseudocode}

\usepackage[top=3.5cm, bottom=3.5cm, left=3.5cm, right=3.5cm]{geometry}

\usepackage{microtype} % Makes document more readable by optimizing space between letters.

\usepackage{units} % Adds \nicefrac{1}{2}: (½), \unit[5]{m}{s}: 5 m/s.

\usepackage[colorlinks=false, pdfborder={0 0 0}]{hyperref} % Allow clicking references and the table of contents in pdfs.
\usepackage{cleveref} % Adds fig/formula etc. before references (use \cref).

%%% New commands.

% Don't output references in case they're empty - http://tex.stackexchange.com/questions/74476/how-to-avoid-empty-thebibliography-environment-bibtex-if-there-are-no-refere

\let\myBib\thebibliography
\let\endmyBib\endthebibliography

\renewcommand\thebibliography[1]{\ifx\relax#1\relax\else\myBib{#1}\fi}

% Norm -> ||something||
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

% Stdfig -> Used as \stdfig{width}{label_name}{caption}
% Requires: image called 'caption' in img folder.
% Output: A figure with the given width, labeled as 'fig:label_name'

\newcommand{\stdfig}[3]{
    \begin{figure}
    \centering
    \includegraphics[width = #1]{img/#2.eps}
    \caption{#3}
    \label{fig:#2}
    \end{figure}
}

\begin{document}

%% Front page.
\title{10.01 - Large Scale Machine Learning}

    \date{}

    \maketitle

\newpage
%% Abstract page.

%% Table of contents page.

%% Body start.
\section{Learning With Large
Datasets}\label{learning-with-large-datasets}

One of the reasons why algorithms are getting better is to simply get
more data.

\begin{quote}
``It's not who has the best algorithm that wins. It's who has the most
data.''
\end{quote}

Learning with large datasets comes with a price: computational time
(they run very slow). A solution for that is to simply take a randomized
subset of the whole data. We can plot the learning curves to see if that
works. For example, in \cref{fig:learning_curves_high_variance}, we can
see that getting more data is likely to help when we have high variance.
On the other hand, in \cref{fig:learning_curves_high_bias} we can see
that, when we have high bias, the amount of data doesn't matter after a
certain point. In this last case, we can think about adding more
features/more hidden layer units.

\stdfig{7cm}{learning_curves_high_variance}{Learning curves with high variance}

\stdfig{7cm}{learning_curves_high_bias}{Learning curves with high bias}

\section{Stochastic Gradient Descent}\label{stochastic-gradient-descent}

This algorithm is a modification to the standard gradient descent
(called ``Batch gradient descent'', as it uses all the data on every
iteration) that allows us to train better on large scale datasets.
\smallskip

In this new algorithm, we modify $J_{train}$ to look like this (which is
the same as the batch one but written in a different way):

\begin{equation}
cost(\theta, (x^{(i)}, y^{(i)})) = \frac{1}{2} (h_\theta(x^{(i)}) - y^{(i)}) ^2
\end{equation}

\begin{equation}
J_{train}(\theta) = \frac{1}{m} \sum_{i=1}^m cost(\theta,(x^{(i)}, y^{(i)}))
\end{equation}

We can see a description of the algorithm in
\cref{alg:stochastic_gradient_descent}.

It basically goes example by example, each time fitting the $\theta$
values a little bit better. What batch did was to have a sumation for
every example in the data, here we do the descent example by example.

In conclusion, stochastic can run much faster in huge datasets at the
cost of giving an approximation of the local minimum instead of the
exact local minimum.

\begin{algorithm}
    \caption{Stochastic gradient descent}
    \label{alg:stochastic_gradient_descent}
\begin{algorithmic}[1]
\State Randomly shuffle dataset.
\State \Repeat
    \For{$i\gets 1, m$}  \Comment{$m$ is the amount of examples in the dataset}
        \For{$j\gets 1, n$}  \Comment{$n$ is the amount of $\theta$ to fit}
            \State $\theta_j := \theta_j - \alpha (h_\theta(x^{(i)}) - y^{(i)}) x_j^{(i)}$
        \EndFor
    \EndFor
\Until{amount of times}  \Comment{Hyperparameter: if dataset is large 1 may be enough}
\end{algorithmic}
\end{algorithm}

\section{Mini-Batch Gradient Descent}\label{mini-batch-gradient-descent}

\begin{itemize}
\itemsep1pt\parskip0pt\parsep0pt
\item
  Batch gradient descent = use all $m$ examples in each iteration.
\item
  Stochastic gradient descent = use 1 example in each iteration.
\item
  Mini-Batch gradient descent = use $b$ examples in each iteration.
\end{itemize}

$b$ is the mini-batch size, with a typical size of 2-100. It might be
faster if the implementation is well vectorized.

\section{Stochastic Gradient Descent
Convergence}\label{stochastic-gradient-descent-convergence}

Back when we were using batch, we could plot $J_{train}(\theta)$ as a
function of the number of iterations of gradient descent and check
whether it was always decreasing or not.

In stochastic, we can't do that, though, as in order to plot
$J_{train}(\theta)$ we need to iterate over all $m$. What we can do is,
during learning, compute the $cost$ of the actual example before
updating $\theta$. Then, every 1000 iterations (say), we can plot the
$cost$ averaged over the last 1000 examples. This graphic can be noisy
and not decrease in some points, as we can see in
\cref{fig:stochastic_cost_plot_ideal}. If we see in the graph that the
line isn't decreasing, we could try to increase the amount of iteration
we average (\cref{fig:stochastic_cost_plot_noisy}).

\stdfig{7cm}{stochastic_cost_plot_ideal}{Ideal plot of the stochastic gradient descent}

\stdfig{7cm}{stochastic_cost_plot_noisy}{Noisy plot (blue) vs plot with a larger number of iterations (red) of the stochastic gradient descent}

\subsection{Learning rate tunning}\label{learning-rate-tunning}

Stochastic gradient descent gets near the local minimum and then wanders
around, never converging. If we want to try to make it converge, we can
slowly decrease $\alpha$ over time, for example with
\cref{eq:learning_rate_decrease}. This comes at the cost of tunning
extra hyperparameters.

\begin{equation} \label{eq:learning_rate_decrease}
\alpha = \frac{const1}{iterationNumber + const2}
\end{equation}

\section{Online learning}\label{online-learning}

This style of learning happens when we have a constant stream of data to
learn from, ie a website.

An example of that would be a shipping service website where user comes,
specifies origin and destination, you offer to ship their package for
some asking price, and users sometimes choose to use your shipping
service ($y$ = 1), sometimes not ($y$ = 0).

Features $x$ capture properties of user, of origin/destination and
asking price. We want to learn $p(y=1|x;\theta)$ to optimize price.
\smallskip

We will use \cref{alg:online_learning} for that. Basically, it's very
similar to stochastic gradient descent but we discard the user data
after using it. A nice property of this algorithm is that it adapts if
your users change over time.

\begin{algorithm}
    \caption{Online Learning}
    \label{alg:online_learning}
\begin{algorithmic}[1]
\While{forever}
    \State Wait until a new example comes in from the stream.
    \State Get the example $x$ and $y$.
    \For{$j\gets 1, n$}  \Comment{$n$ is the amount of $\theta$ to fit}
        \State $\theta_j := \theta_j - \alpha (h_\theta(x) - y) x_j$
    \EndFor
\EndWhile
\end{algorithmic}
\end{algorithm}

\section{Map Reduce and Data
Parallelism}\label{map-reduce-and-data-parallelism}

Sometimes we just have so much data that we can't train using a single
computer. We can use Map Reduce in order to solve that problem.
\smallskip

The idea is: imagine that we have 4 machines and are using batch
gradient descent. We can have each of the machines compute
$\nicefrac{1}{4}$ of the sum needed for a gradient descent step and then
collaborate with the result of the partial sums.

We can look at that visually in \cref{fig:map_reduce_scheme}. \smallskip

\stdfig{10cm}{map_reduce_scheme}{Scheme of a map reduce solution}

Many learning algorithms can be expressed as computing sums of cuntions
over the training set. This are the types of algorithms that can be run
with map-reduce. A good place to look for is the sums over all the
dataset ($\sum_{i=1}^m$), which can be parallelized. \smallskip

Map reduce can even be useful in a single computer. We can simply send
each job to a core instead of another computer.

%% Body end.

%% Bibliography.

\clearpage

    \nocite{*}

\bibliographystyle{plain}
\bibliography{references}

\end{document}

theory/10.01 - Large Scale Machine Learning/10.01 - Large Scale Machine Learning.tex


\newcommand{\RequiredPeers}{\ensuremath{N_\mathit{rs}}}

\chapter{Shardagnostic Genesis}
\label{genesis}

\section{Introduction}

\subsection{Background: understanding the Longest Chain rule}
\label{genesis:background:longest-chain}

Recall the Optimum chain selection rule:

\begin{definition}[Longest Chain Rule]
\label{longest-chain-rule}
A candidate chain is preferred over our current chain if
%
\begin{enumerate}
\item it is longer than our chain, and
\item the intersection point is no more than $k$ blocks away from our tip.
\end{enumerate}
\end{definition}

The purpose of chain selection is to resolve temporary forks that arise from the
normal operation of the protocol (such as when there are multiple leaders in a
single slot), and---importantly---to distinguish honest chains from chains
forged by malicious nodes. It is not a priori clear why choosing longer chains
over shorter chains would help distinguish malicious chains from honest chains:
why would an honest chain be longer?

Recall that the leadership schedule is based on stake: a node's probability of
being elected a leader in a given slot is proportional to their stake. By
assumption, the malicious nodes in the system together have less stake than the
honest nodes; security of the system as a whole critically depends on the
presence of this honest majority. This means that when a malicious node extends
the chain they can only produce a chain with relatively few filled slots: the
honest chain will be \emph{denser}. At least, this will be true near the
intersection point: as we get further away from that intersection point, the
malicious node can attempt to influence the leadership schedule for future slots
to their advantage.

The Optimum security analysis \cite{cryptoeprint:2017:573} tells us that provided
all (honest) nodes are online all the time, they will all share the same chain,
except for blocks near the tips of those chains. Moreover, blocks with a slot
number ahead of the wall clock are considered invalid. This means that the only
way\footnote{The chain sync client does actually allow for some clock skew.
Headers that exceed the clock skew are however not included in chain selection.}
for one chain to be longer than another is by having more filled slots between
the tip of the shared prefix and ``now'': in other words, they must be
\emph{denser}.\footnote{A slightly subtle point arises from era transitions on
the chain that may change parameters such as the active slot coefficient,
allowing for denser chains after the transition. Although an adversary can
introduce such a transition on their own fork, such a transition would not take
effect until one stability window later, and so this won't affect the density
near the intersection point. Era transitions that are agreed on on the main
chain  benefit the honest parties and the adversary equally.}
%
\begin{center}
\begin{tikzpicture}
\draw (0,0) -- (5,0) coordinate(branch) node{$\bullet$} node[pos=0.5,below]{$\underbrace{\hspace{5cm}}_\text{shared prefix}$};
\draw (branch) -- ++(1,  0.9) -- ++(2,0);
\draw (branch) -- ++(1,  0.3) -- ++(2,0);
\draw (branch) -- ++(1, -0.3) -- ++(2,0);
\draw (branch) -- ++(1, -0.9) -- ++(2,0);
\draw [ultra thick] (8,-1.5) -- (8,1.5) node[above]{now};
\end{tikzpicture}
\end{center}
%
This motivates the first part of the Longest Chain rule: chain length is a
useful proxy for chain density, which in turn reflects honesty. The second part
of the rule---the intersection point is no more than $k$ blocks away from our
tip---is important because we can only meaningfully compare density \emph{near
  the intersection point}. As we get further away from the intersection point,
an adversary can start to influence the leadership schedule. This means that if
the adversary's chain forks off from the honest chain far back enough, they can
construct a chain that is longer than the honest chain. The Longest Chain rule
therefore restricts rollback, so that we will simply not even consider chains
that fork off that far back. We can still resolve minor forks that happen in the
honest chain during the normal operation of the protocol, because---so the
analysis guarantees---those will not be deeper than $k$ blocks.

\subsection{Nodes joining late}
\label{genesis:background:joining-late}

When new nodes join the network (or rejoin after having been offline for a
while), they don't have the advantage of having been online since the start of
the system, and have no sufficiently long prefix of the honest chain available.
As we saw at the end of \cref{genesis:background:longest-chain}, simply looking
at chain length is insufficient to distinguish the honest chain from malicious
chains: given enough time, an adversary can produce a chain that is longer than
the honest chain:
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw (-2,0) node{$\bullet$} -- (0,0);
\draw (0,0) node{$\bullet$} -- (6,0) node{$\bullet$} node[right] {honest chain};
\draw (0,0) -- (0,-1) -- (8,-1) node{$\bullet$} node[right]{adversary's chain};
\path (0,0) -- (6,0) node[pos=0.5, above]{$\overbrace{\hspace{6cm}}^{\text{$\gg k$ blocks}}$};
\end{tikzpicture}
\end{center}
%
When a node's current chain is somewhere along that common prefix and uses the
longest chain rule, they will choose the adversary's chain rather than the
honest chain. Moreover, they will now be unable to switch to the honest chain,
because the intersection point with that chain is more than $k$ blocks ago. If a
node would get a ``leg up'' in the form of a reliable message telling which
chain to adopt when joining the network (such a message is known as a
``checkpoint'' in the consensus literature), the Optimum rule from that point
forward would prevent them from (permanently) adopting the wrong chain, but
Optimum cannot be used to help nodes ``bootstrap'' when they are behind.

So far we have just been discussing Optimum as it is described in theory. The
situation in practice is worse. In the abstract models of the consensus
algorithm, it is assumed entire chains are being broadcast and validated. In
reality, chains are downloaded and validated one block at a time.
We therefore don't see a candidate chain's \emph{true} length; instead, the
length of a candidate we see depends on how much of that candidate's chain we
have downloaded\footnote{Even if nodes did report their ``true length'' we would
have no way of verifying this information until we have seen the entire chain,
so we can make no use of this information for the purpose of chain selection.}.
Defining chain selection in terms of chain length, where our \emph{perceived}
chain length depends on what we decide to download, is obviously rather
circular. In terms of the above discussion, it means that the adversary's chain
doesn't even need to be longer than the honest chain:
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw (-2,0) node{$\bullet$} -- (0,0);
\draw (0,0) node{$\bullet$} -- (10,0) node{$\bullet$} node[right] {honest chain};
\draw
   (0,0)
  -- (0,-1)
  -- (4,-1)
    node{$\bullet$}
    node[right]{adversary's chain}
    node[pos=0.5, below]{$\underbrace{\hspace{4cm}}_{\text{$> k$ blocks}}$};
\end{tikzpicture}
\end{center}
%
If the adversary's chain contains more than $k$ blocks after the intersection
point, and we \emph{happen} to download that chain first, we would adopt it and
subsequently be unable to switch to the honest chain; after all, that would
involve a rollback of more than $k$ blocks, which the Optimum rule forbids.

\subsection{The Density rule}
\label{genesis:background:density-rule}

It is therefore clear that we need a different chain selection rule so that
nodes can (re)join, and the Shardagnostic Genesis paper \cite{cryptoeprint:2018:378}
proposes one, shown in \cref{genesis:maxvalid-bg}. In this chapter we will work
with a slightly simplified (though equivalent) form of this rule, which we will
term the Density Rule:
%
\begin{definition}[Density Rule]
A candidate chain is preferred over our current chain if it is denser
(contains more blocks) in a window of $s$ slots anchored at the intersection
between the two chains.
\end{definition}
%
(We will briefly discuss the differences between the rule in the paper and this
one in \cref{genesis:original}.) Technically speaking, $s$ is a
parameter of the rule, but the following default is a suitable choice both from
a chain security perspective and from an implementation perspective:\footnote{If
we change the epoch size, this value might have to be reconsidered, along with
the ledger's stability window.}

\begin{definition}[Genesis window size]
The genesis window size $s$ will be set to $s = 3k/f$.
\end{definition}

Unlike the Longest Chain rule, the Density rule does not impose a maximum
rollback. It does not need to, as it always considers density \emph{at the
intersection point}. This means that in a situation such as
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw (-2,0) node{$\bullet$} -- (0,0);
\draw (0,0) node{$\bullet$} -- (10,0) node{$\bullet$} node[right] {honest chain};
\draw
     (0,0)
  -- (0,-1)
  -- (4,-1)
    node{$\bullet$} node[right]{adversary's chain}
    node[pos=0.5, below]{$\underbrace{\hspace{4cm}}_{\text{$> k$ blocks}}$};
\end{tikzpicture}
\end{center}
%
if we happen to see and adopt the adversary's chain first, we can still adopt
the honest chain (which will be denser at the intersection point, because of the
honest majority).This would however involve a rollback of more than $k$ blocks;
we will discuss how we can avoid such long rollbacks in
\cref{genesis:avoiding-long-rollbacks}.

\section{Properties of the Density rule}

In this section we will study the Density rule and prove some of its
properties. The improved understanding of the rule will be beneficial in the
remainder of this chapter.

\subsection{Equivalence to the Longest Chain rule}

For nodes that are up to date, the Density rule rule does not change how chain
selection works.

\begin{lemma}
\label{lemma:tip-density-is-chain-length}
When comparing two chains with an intersection that is at most $s$ slots away
from the two tips, the Density rule just prefers the longer chain.
\end{lemma}

\begin{proof}
The two chains share a common prefix, and differ only in the blocks
within the last $s$ slots:
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw (0,0) -- (6,0) coordinate (I);
\draw (I) -- ++(1,1) -- ++(1.5,0);
\draw (I) -- ++(1,-1) -- ++ (2,0);
\draw [dashed]
    (I)
  -- ++(0, 1.5)
  -- ++(4, 0) node[pos=0.5,above]{$s$ slots}
  -- ++(0, -3)
  -- ++(-4, 0)
  -- cycle;
\end{tikzpicture}
\end{center}
%
Since the chain length in this case is simply the length of the length of the
common prefix plus the number of blocks in the window (i.e., their density),
the longer chain will also be denser in the window.

Just to be very explicit, \cref{lemma:tip-density-is-chain-length} does
\emph{not} hold when the intersection is more than $s$ slots away:
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw (0,0) -- (3,0) coordinate (I);
\draw (I) -- ++(1,1) -- ++(5.5,0);
\draw (I) -- ++(1,-1) -- ++ (6,0);
\draw [dashed]
    (I)
  -- ++(0, 1.5)
  -- ++(4, 0) node[pos=0.5,above]{$s$ slots}
  -- ++(0, -3)
  -- ++(-4, 0)
  -- cycle;
\end{tikzpicture}
\end{center}
%
In this case of course the longer chain may well not be denser in the window.
\end{proof}

\begin{lemma}[Rule Equivalence]
\label{lemma:rule-equivalence}
When comparing two chains with an intersection that is at most $k$ blocks away
from the two tips, the Density rule and the Longest Chain rule are equivalent.
\end{lemma}

\begin{proof}
First, observe that since the intersection is at most $k$ blocks away, the
maximum rollback condition of the Longest Chain rule is trivially satisfied.
Remains to show that the intersection is at most $s$ slots away, so that we can
apply \cref{lemma:tip-density-is-chain-length}. This is easiest to see by
contradiction: suppose the intersection is \emph{more} than $s$ slots away. Then
we would have a section on the chain which is more than $s$ slots long but
contains fewer than $k$ blocks; the Chain Growth analysis in
\cite{cryptoeprint:2017:573,cryptoeprint:2018:378} tells us that the probability
of this happening is negligibly small (provided $s$ is at least $3k/f$).
\end{proof}

\Cref{lemma:rule-equivalence} has a corollary that is of practical importance
for the consensus layer, as it re-establishes an invariant that we rely on
(\cref{never-shrink}):

\begin{lemma}
Alert nodes (that is, an honest node that has consistently been online)
will never have to switch to a shorter chain.
\end{lemma}

\begin{proof}
The Shardagnostic Genesis analysis \cite{cryptoeprint:2018:378} tells us that
alert nodes will never have to roll back by more than $k$ blocks. In other
words, the intersection between their current chain and any chain they might
have to switch to will be at most $k$ blocks ago. The lemma now follows
from \cref{lemma:rule-equivalence}.
\end{proof}

\subsection{Honest versus adversarial blocks}

In this section we will consider what kinds of chains an adversary might try to
construct.

\begin{lemma}
\label{lemma:adversarial-before-k}
An adversary cannot forge a chain that forks off more than $k$ blocks from an
alert node's tip and is denser than the alert node's chain at the intersection
between the two chains.
\end{lemma}

\begin{proof}
This is an easy corollary of the Shardagnostic Genesis analysis. If the adversary
would be able to construct such a chain, then by the Density rule the node
should switch to it. As mentioned above, however, the analysis tells us that
alert nodes never have to roll back more than $k$ blocks.
\end{proof}

\Cref{lemma:adversarial-before-k} has a useful specialisation for chains that an
adversary might try to construct near the wallclock:

\begin{lemma}
\label{lemma:adversarial-within-s}
An adversary cannot forge a chain that satisfies all of the below:
%
\begin{itemize}
\item It forks off no more than $s$ slots from the wallclock.
\item If forks off more than $k$ blocks before the tip of an alert node.
\item It is longer than the alert node's current chain.
\end{itemize}
\end{lemma}

\begin{proof}
This example satisfies the first two criteria but not the third:
%
\begin{center}
\begin{tikzpicture}[yscale=0.75]
\draw
     (0,0)
  -- (6,0) coordinate (s-back) node{$\bullet$}
  -- (7,0) coordinate (k-back) node{$\bullet$};
\draw
     (k-back)
  -- ++(2, 1)
  -- ++(0, -2) node[pos=0.5,right=0.5]{honest chains}
  -- cycle;
\draw
     (s-back)
  -- ++(1, -2)
  -- ++(1, 0) node[right=1.5cm]{adversary's chain};
\path
     (k-back)
  -- ++(0, 1)
  -- ++(2, 0) node[pos=0.5,above]{$\overbrace{\hspace{2cm}}^{\text{$\le k$ blocks}}$};
\path
     (s-back)
  -- ++(0, 1.75)
  -- ++(3, 0) node[pos=0.5,above]{$\overbrace{\hspace{3cm}}^{\text{$\le s$ slots}}$};
\draw [very thick] (9, -2.5) -- (9, 2.5) node[above]{now};
\end{tikzpicture}
\end{center}
%
The intersection between the alert node's chain and the adversarial chain is
within $s$ slots from the wallclock. This means that density at the intersection
point is just chain length (\cref{lemma:tip-density-is-chain-length}), and
hence the property follows from \cref{lemma:adversarial-before-k}.
\end{proof}

\subsection{The original genesis rule}
\label{genesis:original}

The Density rule is simplification of the rule as presented in the paper
\cite{cryptoeprint:2018:378}.  The original rule is shown in
\cref{genesis:maxvalid-bg}, and paraphrased below:

\begin{definition}[Genesis chain selection rule, original version]
\label{genesis:originalrule}
A candidate chain is preferred over our current chain if

\begin{itemize}
\item The intersection between the candidate chain and our chain is \textbf{no
more than $k$} blocks back, and the candidate chain is strictly \textbf{longer}
than our chain.

\item If the intersection \emph{is} \textbf{more than $k$} blocks back, and the
candidate chain is \textbf{denser} (contains more blocks) than our chain in
a region of $s$ slots starting at the intersection.
\end{itemize}
\end{definition}

As we saw in \cref{lemma:rule-equivalence}, the Density rule is equivalent
to the Longest Chain rule if the intersection is within $k$ blocks, so the
original rule and the simplified form are in fact equivalent.

For completeness sake, we should note that this equivalence only holds for
suitable choice of $s$. If $s$ is much smaller (for example, the paper uses $s =
\frac{1}{4}(k/f)$ in some places), then we might have a situation such as the
following, where we have two chains $A$ and $B$; $A$ is denser than $B$ at the
intersection with $B$, but $B$ is longer:
%
\begin{center}
\begin{tikzpicture}
\path (0, 0) coordinate (tip) node{$\bullet$};
\draw (tip) -- ++(1.0,  0.5) -- ++(2.5, 0) coordinate(C1) node[right]{$A$};
\draw (tip) -- ++(1.0, -0.5) -- ++(3.5, 0) coordinate(C2) node[right]{$B$};
\draw [red, very thick] (tip) -- ++(1.0,  0.5) -- ++(2.0, 0);
\draw [dashed]
     (tip)
  -- ++(0, 0.75)
  -- ++(3, 0)
  -- ++(0, -1.5)
  -- ++(-3, 0) node[pos=0.5, below]{$\underbrace{\hspace{3cm}}_{\text{$s$ slots}}$}
  -- cycle;
\path (tip) -- (C1) node[pos=0.5, above=0.5cm]{$\overbrace{\hspace{3.5cm}}^{\text{fewer than $k$ blocks}}$};
\path (tip) -- (C2) node[pos=0.5, below=1.1cm]{$\underbrace{\hspace{4.5cm}}_{\text{more than $k$ blocks}}$};
\draw (tip) + (-3,0) node{$\bullet$} -- (tip);
\end{tikzpicture}
\end{center}
%
In this case, the original rule ends up preferring either $A$ or $B$ depending
on the order in which we consider them, whereas the Density Rule would simply
pick $A$.

\begin{figure}
\hrule

\textbf{Parameters} \\[0.5em]
\begin{tabular}{ll}
$C_\mathit{loc}$ & Current chain \\
$\mathcal{N} = \{C_1, \ldots, C_M\}$ & All possible chains (including our own) \\
$k$ & Security parameter (\cref{consensus:overview:k}) \\
$s$ & Genesis window size (Genesis rule specific parameter) \\
$f$ & Active slot coefficient (\cref{optimum:f}) \\[1em]
\end{tabular}

\textbf{Algorithm}

\begin{lstlisting}[escapeinside={(*}{*)}, language={}, keywords={for,do,if,then,else,end,return}]
// Compare (*$C_\mathit{max}$*) to each (*$C_i \in \mathcal{N}$*)
Set (*$C_\mathit{max} \leftarrow C_\mathit{loc}$*)
for (*$i = 1$*) to (*$M$*) do
  if (*$(C_i \text{ forks from } C_\mathit{max} \text{ at most } k \text{ blocks})$*) then
    if (*$|C_i| > |C_\mathit{max}|$*) then // Condition A
      Set (*$C_\mathit{max} \leftarrow C_i$*).
  else
    Let (*$j \leftarrow \max \Bigl\{ j' \ge 0 \mathrel{\Bigl\lvert} C_\mathit{max} \text{ and } C_i \text{ have the same block in } \mathtt{sl}_{j'} \Bigr\} $*)
    if (*$|C_i[0 : j + s]| > |C_\mathit{max}[0 : j + s]|$*) then // Condition B
      Set (*$C_\mathit{max} \leftarrow C_i$*).
return (*$C_\mathit{max}$*)
\end{lstlisting}

\hrule
\caption{\label{genesis:maxvalid-bg}Algorithm \texttt{maxvalid-bg}}
\end{figure}

\section{Fragment selection}
\label{genesis:fragment-selection}

While the algorithms described in the literature on Shardagnostic compare entire
chains, we will want to compare \emph{fragments} of chains: if at all possible
we would prefer not to have to download and verify entire chains before we can
make any decisions. As we have discussed in
\cref{genesis:background:joining-late}, comparing fragments (prefixes) of chains
using the Longest Chain rule does not actually make much sense, but in this
section we will see that the situation is fortunately much better when we use
the Density rule.

\begin{definition}[Preferred fragment]
Let $\mathcal{S}$ be set of chain fragments, all anchored at the same point
(that is, the fragments share a common ancestor), corresponding to some set of
chains $\mathcal{C}$. Then $A$ is a preferred fragment in $\mathcal{S}$ if and
only if $A$ is a fragment of a preferred chain in $\mathcal{C}$.
\end{definition}

We will now establish the (necessary and sufficient) condition for fragment
preference to be decidable. First, if we have to choose between two chains, we
must see enough of those chains to do a density comparison.

\begin{definition}[Known density]
We say that a chain fragment has a \emph{known density} at some point $p$
if either of the following conditions hold:

\begin{enumerate}
\item The fragment contains a block after at least $s$ slots:
\begin{center}
\begin{tikzpicture}[yscale=0.75]
\path (0,0) -- (9,0); % adjust bounding box
\path (0,0) -- (1,0) node[pos=0.5]{$\cdots$};
\draw
     (1,0)
  -- (3,0)   node{$\bullet$} node[above left]{$p$} coordinate(p)
  -- (3.5,0) node{$\bullet$}
  -- (4,0);
\path
     (4,0)
  -- (5,0) node[pos=0.5]{$\cdots$};
\draw
     (5,0)
  -- (5,0) node{$\bullet$}
  -- (6,0) node{$\bullet$}
  -- (7,0) node[red]{$\bullet$}
  -- (8,0) node[right]{$\cdots$};
\draw [dashed]
     (p)
  -- ++(0, 1)
  -- ++(3.5, 0) node[pos=0.5,above]{$s$ slots}
  -- ++(0, -2)
  -- ++(-3.5, 0)
  -- cycle;
\end{tikzpicture}
\end{center}

\item The chain (not just the fragment\footnote{We can distinguish between these
two cases because nodes report the tip of their chain as part of the chain sync
protocol, independent from the headers that we have downloaded from those
nodes.}) terminates within the window:
\begin{center}
\begin{tikzpicture}[yscale=0.75]
\path (0,0) -- (9,0); % adjust bounding box
\path (0,0) -- (1,0) node[pos=0.5]{$\cdots$};
\draw
     (1,0)
  -- (3,0)   node{$\bullet$} node[above left]{$p$} coordinate(p)
  -- (3.5,0) node{$\bullet$}
  -- (4,0);
\path
     (4,0)
  -- (5,0) node[pos=0.5]{$\cdots$};
\draw
     (5,0)
  -- (6,0) node[red]{$\bullet$};
\draw [dashed]
     (p)
  -- ++(0, 1)
  -- ++(3.5, 0) node[pos=0.5,above]{$s$ slots}
  -- ++(0, -2)
  -- ++(-3.5, 0)
  -- cycle;
\end{tikzpicture}
\end{center}
\end{enumerate}
\end{definition}

\begin{definition}[Look-ahead closure]
\label{lookahead-closure}
Let $\mathcal{S}$ be a set of chain fragments all anchored at the same point. We
say that $\mathcal{S}$ is \emph{look-ahead closed} if whenever there are two
fragments $A, B \in \mathcal{S}$, the densities of $A$ and $B$ are known at
their intersection.
\end{definition}

\begin{lemma}[Look-ahead closure is sufficient for fragment selection]
\label{lemma:fragment-selection}
Let $\mathcal{S}$ be a look-ahead closed set of chain fragments. Then
we can always choose a preferred fragment in $\mathcal{S}$.
\end{lemma}

\begin{proof}[Proof (sketch)]
In order to be able to pick a chain, we need to resolve forks. In order to
resolve forks using the Density Rule, we need to know the density, but that
is precisely what is guaranteed by look-ahead closure.
\end{proof}

\Cref{lemma:fragment-selection} is relevant because it reflects how the
consensus layer uses chain selection:

\begin{enumerate}
\item We maintain a fragment of the chain for each upstream peer we track
(\cref{chainsyncclient}). The block fetch client
(\cref{chainsyncclient:plausiblecandidates}) then picks a preferred fragment and
downloads that.
\item When the chain database needs to construct the current chain
(\cref{chainsel}), it constructs a set of chain fragments through the volatile
DB, all anchored at the tip of the immutable database, picks a preferred
fragment, and adopts that as the node's current chain.
\end{enumerate}

However, \cref{lemma:fragment-selection} is less useful than it might seem:
the look-ahead closure requirement means that in the worst case, we still need
to see entire chains before we can make a decision: every new intersection point
requires us to see $s$ more slots:
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\path (0,0) coordinate(I);
%
\node at (I) {$\bullet$};
\draw (I) -- ++(2.5, 0);
\draw (I) -- ++(1,-1) -- ++(0.5,0) coordinate(A) -- ++(2,0);
\draw [dashed]
     (I)
  -- ++(0,0.25)
  -- ++(2,0)
  -- ++(0,-1.5)
  -- ++(-2,0) node[pos=0.5,below]{$\underbrace{\hspace{2cm}}_\text{$s$ slots}$}
  -- cycle;
%
\node at (A) {$\bullet$};
\draw (A) -- ++(2.5, 0);
\draw (A) -- ++(1,-1) -- ++(0.5,0) coordinate(B) -- ++(2,0);
\draw [dashed]
     (A)
  -- ++(0,0.25)
  -- ++(2,0)
  -- ++(0,-1.5)
  -- ++(-2,0) node[pos=0.5,below]{$\underbrace{\hspace{2cm}}_\text{$s$ slots}$}
  -- cycle;
%
\node at (B) {$\bullet$};
\draw (B) -- ++(2.5, 0);
\draw (B) -- ++(1,-1) -- ++(0.5,0) coordinate(C) -- ++(2,0) node[above=0.5cm, right]{$\cdots$};
\draw [dashed]
     (B)
  -- ++(0,0.25)
  -- ++(2,0)
  -- ++(0,-1.5)
  -- ++(-2,0) node[pos=0.5,below]{$\underbrace{\hspace{2cm}}_\text{$s$ slots}$}
  -- cycle;
\end{tikzpicture}
\end{center}
%
Moreover, due to the header/body split
(\cref{nonfunctional:network:headerbody}), when we are tracking the headers from
an upstream peer, we cannot (easily) verify headers that are more than $3k/f$
slots away from the intersection between our chain and their chain (see also
\cref{low-density}). In the next section we will therefore consider how we can
drop the look-ahead closure requirement.

In case it is not obvious why we must only compare density at intersection
points, in the remainder of this section we will consider an example that will
hopefully clarify it. Suppose a malicious node with some stake intentionally
skips their slot, after which the chain continues to grow as normal:
%
\begin{center}
\begin{tikzpicture}
\draw
       (0,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node[above]{$\times$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$};
\end{tikzpicture}
\end{center}
%
It is now trivial for the adversary to create an alternative chain that
\emph{does} have a block in that slot; if other nodes switch to the denser chain
the moment they see a window of $s$ slots that is denser, they would adopt the
adversary's chain; after all, it has one more block in the window than the
honest chain does:
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw
       (0,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$} coordinate(s-anchor)
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$} coordinate(branch)
  -- ++(1,0) node[above]{$\times$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$};
\draw
       (branch)
  -- ++(1, -1) node{$\bullet$}
  -- ++(3,  0) node{$\bullet$};
\draw [dashed]
     (s-anchor)
  -- ++(0,1)
  -- ++(3.5,0)
  -- ++(0,-2.5)
  -- ++(-3.5,0) node[below, pos=0.5]{$\underbrace{\hspace{3.5cm}}_{\text{$s$ slots}}$}
  -- cycle;
\end{tikzpicture}
\end{center}
%
Instead, we must wait until we make such a comparison until we have reached
the intersection point:
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw
       (0,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$} coordinate(branch)
  -- ++(1,0) node[above]{$\times$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$};
\draw
       (branch)
  -- ++(1, -1) node{$\bullet$}
  -- ++(3,  0) node{$\bullet$};
\draw [dashed]
     (branch)
  -- ++(0,1)
  -- ++(3.5,0)
  -- ++(0,-2.5)
  -- ++(-3.5,0) node[below, pos=0.5]{$\underbrace{\hspace{3.5cm}}_{\text{$s$ slots}}$}
  -- cycle;
\end{tikzpicture}
\end{center}
%
Since the adversary does not have sufficient stake, their chain will be less
dense and we will therefore not select it. If the adversary creates another fork
earlier on the chain, then we will resolve that fork when we encounter it using
a window of $s$ slots \emph{anchored at that fork}, and then later resolve the
second fork using a \emph{different} window of $s$ slots, anchored at the second
fork:
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw
       (0,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$} coordinate(s-anchor)
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$} coordinate(branch)
  -- ++(1,0) node[above]{$\times$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$}
  -- ++(1,0) node{$\bullet$};
\draw
       (branch)
  -- ++(1, -1) node{$\bullet$}
  -- ++(3,  0) node{$\bullet$};
\draw
       (s-anchor)
  -- ++(1, -1) node{$\bullet$};
\draw [dashed]
     (s-anchor)
  -- ++(0,1)
  -- ++(3.5,0)
  -- ++(0,-2.5)
  -- ++(-3.5,0) node[below, pos=0.5]{$\underbrace{\hspace{3.5cm}}_{\text{$s$ slots}}$}
  -- cycle;
\draw [dotted]
     (branch) ++ (0, 0.1)
  -- ++(0,1)
  -- ++(3.5,0)  node[above, pos=0.5]{$\overbrace{\hspace{3.5cm}}^{\text{$s$ slots}}$}
  -- ++(0,-2.5)
  -- ++(-3.5,0)
  -- cycle;
\end{tikzpicture}
\end{center}

\pagebreak

\section{Prefix selection}
\label{genesis:prefix-selection}

\subsection{Preferred prefix}

When a set $\mathcal{S}$ of chain fragments is not look-ahead closed, we may
not be able to pick a best fragment. For example, in
%
\begin{equation*}
\mathcal{S} = \left\{ \;
\begin{tikzpicture}[baseline=0pt, xscale=0.5,yscale=0.5]
\draw [very thick, red] (-2,0) -- (0,0);
\draw (0,0) -- (1, 1) -- (6,  1) node[right]{$A$};
\draw (0,0) -- (1,-1) -- (7, -1) node[right]{$B$};
\draw [dashed] (-2,0) -- ++(0,1.5) -- ++(5,0) node[pos=0.5,above]{$\overbrace{\hspace{2cm}}^\text{$s$ slots}$} -- ++(0,-3) -- ++(-5,0) -- cycle;
\end{tikzpicture}
\right\}
\end{equation*}
%
we cannot choose between $A$ and $B$; what we \emph{can} say however is that no
matter which of $A$ and $B$ turns out to be the better fragment, the common
prefix of $A$ and $B$ (shown in red) will definitely be a prefix of that
fragment. This example provides the intuition for the definition of a preferred
prefix:

\begin{definition}[Preferred prefix]
Given a set $\mathcal{S}$ of chain fragments, all anchored at the same point, a
preferred prefix is a prefix $\Pi$ of one of the fragments in $\mathcal{S}$,
such that $\Pi$ is guaranteed to be a prefix of a preferred fragment in the
lookahead-closure of $\mathcal{S}$.
\end{definition}

In other words, we may not be able to pick the best fragment out of
$\mathcal{S}$, but we \emph{can} pick a prefix which is guaranteed to be a
prefix of whatever turns out to be the best fragment. Obviously, the empty
fragment is always a valid choice, albeit not a particularly helpful one.
Ideally, we would choose the \emph{longest} preferred prefix. Fortunately,
constructing such a prefix is not difficult.

\subsection{Algorithm}

We will now consider how we can choose the longest preferred prefix.

\begin{definition}[Prefix selection]
\label{prefix-selection}
Let $\mathcal{S}$ be a set of chain fragments all anchored at the same point $a$,
such that all fragments have known density at point $a$. Then we can construct
the longest preferred prefix in two steps:
%
\begin{enumerate}
\item \emph{Resolve initial fork.}
Suppose $\mathcal{S}$ looks like this:
%
\begin{center}
\begin{tikzpicture}
\node at (0,0) [left] {$a$};
\node at (0,0) {$\bullet$};
%
\draw (0,0) -- (1, 0.5) coordinate(x);
\draw (0,0) -- (1,-0.5) coordinate(y);
\draw [dotted] (0,0) -- (1,-1) node[below right]{$\cdots$};
%
\draw (x) -- ++(2,  0.25) -- ++(0, -0.5) node[pos=0.5,right]{$\vec{x}$} -- cycle;
\draw (y) -- ++(2,  0.25) -- ++(0, -0.5) node[pos=0.5,right]{$\vec{y}$} -- cycle;
%
\draw [dashed]
     (0,0)
  -- ++(0, 1)
  -- ++(2.5, 0) node[pos=0.5,above]{$s$ slots}
  -- ++(0, -2.5)
  -- ++(-2.5, 0)
  -- cycle;
\end{tikzpicture}
\end{center}
%
where (without loss of generality) the common prefixes are non-empty. Suppose
one of the $\vec{x}$ has the highest density\footnote{ If two fragments in
different forks have  have exactly the same density, we need a tie-breaker in
order to be able to make progress. The genesis paper does not prefer either
chain in such a scenario, switching only if another chain is strictly denser. We
can therefore follow suit, and just focus on one of the two chains arbitrarily.}
at point $a$; let's call it $x_i$. That means if we ever were to adopt any of
the $\vec{y}, \ldots$, and then compared our chain to $x_i$, we would find that
$x_i$ is denser at the intersection point (which is precisely what we are
comparing in this window here), and therefore switch to it. This means we can
focus our attention on the $\vec{x}$. (\Cref{greedy-chain-selection}
justifies the greedy nature of this algorithm in a bit more depth.)

\item \emph{Adopt common prefix.}
Most of the time, the density of the $\vec{x}$ will yet not be known at point
$b$. This means we do not yet know which $x_i$ will turn out to be the best, but
we \emph{do} know that whichever it turns out to be, it will have the common
prefix from $a$ to $b$, so we choose this as the longest preferred prefix:

\begin{center}
\begin{tikzpicture}
\path (0,0)   node{$\bullet$} node[left]{$a$};
\path (1,0.5) node{$\bullet$} node[below]{$b$};
%
\draw [very thick, red] (0,0) -- (1, 0.5) coordinate(x);
%
\draw (x) -- ++(2,  0.25) -- ++(0, -0.5) node[pos=0.5,right]{$\vec{x}$} -- cycle;
%
\draw [dashed]
     (0,0)
  -- ++(0, 1)
  -- ++(2.5, 0) node[pos=0.5,above]{$s$ slots}
  -- ++(0, -1.5)
  -- ++(-2.5, 0)
  -- cycle;
\end{tikzpicture}
\end{center}
%
One useful special case is when all of the $\vec{x}$ terminate within the
window. In this case, their density \emph{is} known, and we can just pick the
densest fragment as the preferred prefix; the preferred prefix is then in fact
the preferred fragment.
\end{enumerate}
\end{definition}

Prefix selection fits very naturally with the needs of the consensus layer
(\emph{cf.} the description of how consensus uses fragment selection in
\cref{genesis:fragment-selection}):
%
\begin{enumerate}
\item When blockfetch applies prefix selection to the set of fragments of
the upstream peers we are tracking, the computed prefix is precisely the set
of blocks that blockfetch should download.
\item When the chain database applies prefix selection to the set of fragments
through the volatile database, the computed prefix is precisely the chain that
we should adopt as the node's current chain.
\end{enumerate}

\begin{lemma}
If the intersection point between the chains of our upstream peers is at most
$k$ blocks away from their tips, prefix selection will choose the longest chain.
\end{lemma}

\begin{proof}[Proof (sketch)]
The proof is very similar to the proof of \cref{lemma:rule-equivalence}.
If the intersection is at most $k$ blocks away, it will be less than $s$ slots
away; therefore prefix selection will be able to see the chains to their tip,
the density of all chains will be known, and the densest fragment equals the
longest one.
\end{proof}

\begin{figure}[p]
\hrule
\vspace{0.5em}

Step 1 in \cref{prefix-selection} makes greedy decisions, discarding the less
dense chains at every step without looking further ahead. To justify this, it is
instructive to consider an apparent counter-example to the validity of this
strategy. Consider three chains $A, B, C$ in the following example:
%
\begin{center}
\begin{tikzpicture}
\draw
     (0, 0) node {$\bullet$} coordinate (a)
  -- (2, 1) node {$\bullet$} coordinate (b) node[above left]{$ab$};
\draw
     (a)
  -- ++(1, -0.5) node {$\bullet$}
  -- ++(2,  0  ) node {$\bullet$}
  -- ++(4,  0  ) node[right]{$C$};
\draw
     (b)
  -- ++(0.5, 0.5) node {$\bullet$}
  -- ++(0.5, 0  ) node {$\bullet$}
  -- ++(0.5, 0  ) node {$\bullet$}
  -- ++(3.5, 0  ) node[right]{$A$};
\draw
     (b)
  -- ++(0.5 , -0.5)
  -- ++(1.25,  0  )
  -- ++(0.5 ,  0  ) node {$\bullet$}
  -- ++(0.5 ,  0  ) node {$\bullet$}
  -- ++(0.5 ,  0  ) node {$\bullet$}
  -- ++(0.5 ,  0  ) node {$\bullet$}
  -- ++(1.25,  0  ) node[right]{$B$};
%
\draw [dashed]
     (a)
  -- ++(0, 2)
  -- ++(4, 0)
  -- ++(0, -3)
  -- ++(-4, 0) node[pos=0.5,below]{$\underbrace{\hspace{4cm}}_\text{$s$ slots}$}
  -- cycle;
\draw [dashed]
     (b)
  -- ++(0, 1.5)
  -- ++(4, 0) node[pos=0.5,above]{$\overbrace{\hspace{4cm}}^\text{$s$ slots}$}
  -- ++(0, -2.5)
  -- ++(-4, 0)
  -- cycle;
\end{tikzpicture}
\end{center}
%
Depending on the order in which we consider the chains, we might pick any of
these three chains:
%
\begin{center}
\begin{tabular}{l|l}
\textbf{order} & \textbf{selected chain} \\ \hline
$B, C, A$ & $A$ \\
$C, A, B$ & $B$ \\
$A, B, C$ & $C$ \\
\end{tabular}
\end{center}
%
In practice this situation cannot occur, however. The Genesis analysis tells us
that there will be single chain (the honest chain) which will be denser than any
other chain (within the genesis window) at every intersection point. This is
precisely what justifies the greedy nature of the algorithm: it will always hone
in on the honest chain. In the counter-example above, there is no single chain
with this property, and hence cannot arise.
\caption{\label{greedy-chain-selection}Greedy chain selection}
\end{figure}

\subsection{Prefix selection on headers}

When the chain sync client is deciding which of the chains of its upstream peers
to download, it does so based on chains of \emph{headers}. It does not download
block bodies and so cannot verify them. As such, it is basing its decisions
based on header validity \emph{only}.  However, this is then only used to tell
the block fetch client which blocks to \emph{download}
(\cref{chainsyncclient:plausiblecandidates}); it does not necessarily mean that
those blocks will be \emph{adopted}. When the chain database performs chain
selection (\cref{chainsel}), it will verify the blocks and discard any that turn
out to be invalid. If any blocks \emph{are} invalid, then the chain sync client
will disconnect from the nodes that provided them, which in turn may change
which prefix is chosen by prefix selection.

Indeed, the only reason to even validate headers at all is to avoid a denial
of service attack where an adversary might cause us to waste resources.
It may be worth reconsidering this risk, and balancing it against the costs
for the implementation; the only reason we need forecasting, and a special
treatment of low density chains (\cref{low-density}) is that we insist we want
to validate headers independent from blocks.

\pagebreak
\subsection{Known density}

\Cref{prefix-selection} requires known density at the anchor $a$ of the set, so
that it can resolve the initial fork. This means we have to wait until
we have downloaded enough headers from each peer: the last header we downloaded
must either be outside the $s$ window, or else it must be the last header on that
peer's chain (in which case the peer will tell us so). If a peer tells us they have
more headers but then do not provide them, we should disconnect from them
after some time-out to avoid a potential denial of service attack.

If that header is outside the genesis window, we may not be able to validate it:
since it is more than $s$ slots away from the anchor point (and we only have a
ledger state at the anchor point), it falls outside the ledger's forecast range.
However, this does not matter: the presence of this header only tells us that we
have seen everything we need to see to compute the density within the window; an
invalid header after that window cannot increase the density \emph{within} the
window.

We can make one useful exception to the known density requirement:
if there \emph{is} no initial fork, we do not need to know the density at
all and can go straight to step (2). This will allow a node to sync faster:
consider a new node that is joining the network. In most cases, all of the
node's upstream peers will report the \emph{same} blocks for all but the last
few blocks on the chain. Since there is no fork to resolve, we can start
adopting each block the moment it is reported by all peers.

\subsection{Propagation delays}

Propagation delays will be a bit slower than before. To see why, let's consider
the most ideal situation where all nodes are on exactly the same chain, and
the next slot leader produces a new block. Let
%
\begin{center}
\begin{tikzpicture}[block/.style={rectangle,draw=black,minimum size=7mm}]
\node at (0,0) (A) [block] {$n$};
\draw [->] (A.east) -- ++(2,0) node[pos=0.9,above]{$m$};
\end{tikzpicture}
\end{center}
%
denote an upstream peer and a downstream peer. The upstream peer's tip is $n$,
and is sending headers to the downstream peer; the most recent header that the
downstream peer validated is $m$. When $n = m$, the downstream peer has ``seen''
the upstream peer's entire chain; in the terminology of this chapter, the
density of the upstream peer is then \emph{known}.

Consider a diamond topology with four peers $A \ldots D$, nodes $B$ and $C$
following node $A$, and node $D$ following nodes $B$ and $C$; suppose all nodes
are up to date:

\begin{center}
\begin{tikzpicture}[block/.style={rectangle,draw=black,minimum size=9mm}]
\node at ( 0,  0) (A) [block] {$n$};
\node at (-1, -2) (B) [block] {$n$};
\node at ( 1, -2) (C) [block] {$n$};
\node at ( 0, -4) (D) [block] {$n$};
\node at (A.south west) [above left] {$A$};
\node at (B.south west) [above left] {$B$};
\node at (C.south east) [above right] {$C$};
\node at (D.south west) [above left] {$D$};
\draw [->] (A.south) -- (B.north) node[pos=0.9,above left]{$n$};
\draw [->] (A.south) -- (C.north) node[pos=0.9,above right]{$n$};
\draw [->] (B.south) -- (D.north) node[pos=0.8,left]{$n$};
\draw [->] (C.south) -- (D.north) node[pos=0.8,right]{$n$};
\end{tikzpicture}
\end{center}

Now consider what happens when node $A$ produces a new block:\footnote{Exact
details of the chain sync protocol may mean that this example as it  stands is
not quite possible: the only way for $D$ to become aware of $C$'s new tip is
when it receives the header. However, the point still remains; have $A$ switch
to a fork instead and repeat the reasoning.}

\begin{center}
\begin{tabular}{c@{\hspace{1cm}}c@{\hspace{1cm}}c}
\begin{tikzpicture}[block/.style={rectangle,draw=black,minimum size=9mm}]
\node at ( 0,  0) (A) [block] {$n'$};
\node at (-1, -2) (B) [block] {$n$};
\node at ( 1, -2) (C) [block] {$n$};
\node at ( 0, -4) (D) [block] {$n$};
\node at (A.south west) [above left] {$A$};
\node at (B.south west) [above left] {$B$};
\node at (C.south east) [above right] {$C$};
\node at (D.south west) [above left] {$D$};
\draw [->] (A.south) -- (B.north) node[pos=0.9,above left]{$n$};
\draw [->] (A.south) -- (C.north) node[pos=0.9,above right]{$n$};
\draw [->] (B.south) -- (D.north) node[pos=0.8,left]{$n$};
\draw [->] (C.south) -- (D.north) node[pos=0.8,right]{$n$};
\end{tikzpicture}
&
\begin{tikzpicture}[block/.style={rectangle,draw=black,minimum size=9mm}]
\node at ( 0,  0) (A) [block] {$n'$};
\node at (-1, -2) (B) [block] {$n'$};
\node at ( 1, -2) (C) [block] {$n'$};
\node at ( 0, -4) (D) [block] {$n$};
\node at (A.south west) [above left] {$A$};
\node at (B.south west) [above left] {$B$};
\node at (C.south east) [above right] {$C$};
\node at (D.south west) [above left] {$D$};
\draw [->] (A.south) -- (B.north) node[pos=0.9,above left]{$n'$};
\draw [->] (A.south) -- (C.north) node[pos=0.9,above right]{$n'$};
\draw [->] (B.south) -- (D.north) node[pos=0.8,left]{$n$};
\draw [->] (C.south) -- (D.north) node[pos=0.8,right]{$n$};
\end{tikzpicture}
&
\begin{tikzpicture}[block/.style={rectangle,draw=black,minimum size=9mm}]
\node at ( 0,  0) (A) [block] {$n'$};
\node at (-1, -2) (B) [block] {$n'$};
\node at ( 1, -2) (C) [block] {$n'$};
\node at ( 0, -4) (D) [block] {$n$};
\node at (A.south west) [above left] {$A$};
\node at (B.south west) [above left] {$B$};
\node at (C.south east) [above right] {$C$};
\node at (D.south west) [above left] {$D$};
\draw [->] (A.south) -- (B.north) node[pos=0.9,above left]{$n'$};
\draw [->] (A.south) -- (C.north) node[pos=0.9,above right]{$n'$};
\draw [->] (B.south) -- (D.north) node[pos=0.8,left]{$n'$};
\draw [->] (C.south) -- (D.north) node[pos=0.8,right]{$n$};
\end{tikzpicture}
\\
(i) & (ii) & (iii)
\end{tabular}
\end{center}

Initially node $A$ announces its header to $B$ and $C$ (i), which download it
and verify it. Since their only peer is $A$, the density of ``all'' peers is now
known and so both $B$ and $C$ adopt this new block and make it available to $D$
(ii).

Suppose node $D$ now downloads that header from $B$ and verifies it (iii).
Prior to the implementation of genesis, the chain selection rule would have
concluded that this chain is longer, therefore preferred, and node $D$ would be
able to adopt the block before interacting with $C$.

Prefix selection however will \emph{wait}: it hasn't yet verified the header
from $C$, and so it considers the density of $C$'s chain's to be unknown, and
hence it cannot make a decision. \emph{This is the correct behaviour}: the
old implementation would select $C$'s chain immediately, but it would do so
for a terrible reason: sure, it's longer, but only \emph{because we downloaded
more of that chain}; this is the circular reasoning we described above in
\cref{genesis:background:joining-late}.

In principle it would be possible for $D$ to \emph{skip} the verification of
any number of headers if those headers have already been verified from another
peer. However, this would be optimising the code for the best case, with
performance degrading when the peers are on different chains; this is something
we avoid throughout (\cref{nonfunctional:best-is-worst}).

% \item When we say that the density is \emph{known}, it either means that the
% peer reports the next header to be outside the genesis window, or has given us
% the header corresponding to tip of their chain and we validated it. However,
% consider what happens when the node is up to date, and a new block is produced
% (by some other node). Strictly speaking we must now wait for \emph{all} peers to
% have provided us with this header, and have validated all those headers, before
% we would consider the density to be known and prefix selection can make progress
% (if we are up to date, all headers we receive will be within the genesis
% window). This is however unnecessary: when node $A$ provides us with a header,
% and then node $B$ reports the exact same tip, we know that node $B$'s density
% cannot exceed node $A$s, and so we can go ahead and select node $A$'s chain.
% \end{enumerate}

\section{Avoiding long rollbacks}
\label{genesis:avoiding-long-rollbacks}

\subsection{Representative sample}

At the end of \cref{genesis:background:density-rule} we mentioned that if
we have a situation such as
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw (-2,0) node{$\bullet$} -- (0,0);
\draw (0,0) node{$\bullet$} -- (10,0) node{$\bullet$} node[right] {honest chain};
\draw
     (0,0)
  -- (0,-1)
  -- (4,-1)
    node{$\bullet$}
    node[right]{adversary's chain}
    node[pos=0.5, below]{$\underbrace{\hspace{4cm}}_{\text{$> k$ blocks}}$};
\end{tikzpicture}
\end{center}
%
and we happen to adopt the adversary's chain first and later compare it to the
honest chain, the Density Rule will prefer the honest chain and so we should
switch, at the cost of a rollback of more than $k$ blocks.

Such long rollbacks are problematic; we depend on having a maximum rollback of
$k$ blocks in many ways
(\cref{consensus:overview:k,storage:components,chainsyncclient:validation,chainsyncclient:forecasting}
and others), and we will want to \emph{continue} to depend on it. However, we
needed this long rollback in this example only because we \emph{first} adopted
the adversary's chain, and \emph{then} compared it to the honest chain. In the
consensus layer even as it is today, we already don't do chain selection in such
a chain-by-chain manner; as we saw in \cref{genesis:fragment-selection} and
again in \cref{genesis:prefix-selection}, we instead pick the best out of all
currently known candidates. This means that as long as we are \emph{aware} of
both chains before we adopt either one, we will just pick the honest chain
straight away, and we avoid the rollback.

This then is the solution to avoiding longer-than-$k$ rollbacks: as long as we
are not yet up to date with the main chain, we must make sure that we connect to
a representative sample $\RequiredPeers$ of upstream peers that the probability
that \emph{none} of them will serve us the honest chain is negligible
(presumably aided by a probabilistic way of choosing upstream peers in the
network layer), and avoid doing prefix selection until we have reached
this threshold.

Of course, the need to avoid being eclipsed is not unique to genesis. When we
are up to date we can however tolerate relatively long periods in which we do
not see the honest chain; an exact analysis of the relation between the length
of such an eclipse and the maximum chain divergence we can handle is beyond the
scope of this chapter, but it seems safe to say that being eclipsed for short
periods of time should be unproblematic. If however we are behind and are
catching up with the chain, being eclipsed for even just one minute is already
problematic: an attacker can easily feed us $k$ blocks in that period, at which
point we would be unable to switch to the honest chain.

\subsection{Becoming alert}
\label{genesis:becoming-alert}

The only remaining decision to make is when we can \emph{drop} this requirement:
at which point is the state of the node comparable to the state of an alert node
(a node that has consistently been online)?

Every block that we adopt through prefix selection that is more than $s$ slots
away from the wall clock must be a block on the honest chain, because at every
fork we will only adopt blocks from the denser fragment. Moreover, all honest
parties (all alert nodes) will agree on blocks that are more than $s$ slots away
from their tip. This means that we will not need to roll back at all: any block
that we adopt which is more than $s$ slots away from the wall-clock is a block
that we can be sure about.

It is tempting to conclude from \cref{lemma:adversarial-within-s} that as soon
we have reached $s$ slots from the wallclock, we can drop the requirement to be
connected to $\RequiredPeers$ nodes and restrict rollback to  $k$ blocks. This
is however not the case. Recall what the situation looks like:
%
\begin{center}
\begin{tikzpicture}[yscale=0.5]
\draw
     (0,0)
  -- (6,0) coordinate (s-back) node{$\bullet$}
  -- (7,0) coordinate (k-back) node{$\bullet$};
\draw
     (k-back)
  -- ++(2, 1)
  -- ++(0, -2) node[pos=0.5,right=0.5]{honest chains}
  -- cycle;
\draw
     (s-back)
  -- ++(1, -2)
  -- ++(1, 0) node[right=1.5cm]{adversary's chain};
\path
     (k-back)
  -- ++(0, 1)
  -- ++(2, 0) node[pos=0.5,above=-0.15]{$\overbrace{\hspace{2cm}}^{\text{$\le k$ blocks}}$};
\path
     (s-back)
  -- ++(0, 1.75)
  -- ++(3, 0) node[pos=0.5,above]{$\overbrace{\hspace{3cm}}^{\text{$\le s$ slots}}$};
\draw [very thick] (9, -2.5) -- (9, 2.5) node[above]{now};
\end{tikzpicture}
\end{center}
%
\Cref{lemma:adversarial-within-s} tells us that the adversary cannot construct a
chain that forks off more than $k$ blocks from an alert node's chain and is
longer than that chain. It does \emph{not} tell us that it cannot contain more
than $k$  blocks after the intersection.\footnote{A worst-case adversary with
near 50\% stake would be able to construct $0.5 \times 3k = 1.5k$ blocks in
$3k/f$ slots. The reasoning would be simpler if the adversary has at most 33\%
stake, as then they could indeed only construct $k$ blocks in $3k/f$ slots.}
This means that if we dropped the requirement that we see all chains and then
see and adopt the adversary's chain, we would be stuck, as switching to the
honest chain would involve a rollback of more than $k$ blocks.

\Cref{lemma:adversarial-within-s} would only help us if we are somehow
guaranteed that we have adopted one of the alert nodes' chains. But how can we
tell? Fortunately, here we have a rare example of reality serendipitously lining
up with theory. In the theoretical model, nodes collect entire chains broadcast
by their peers, and then use the density rule to select the best one. Once they
have done that, they have selected the same chain that any of the alert nodes
might have selected, and so from this point forward their state is effectively
indistinguishable from the state of an alert node.

In reality of course we cannot collect entire chains, and so we introduced
the concept of prefix selection in order to be able to apply the Density rule
incrementally. However, notice what happens when we have reached $s$ slots
from the wallclock: once we have filled our look-ahead window, \emph{we will
have seen every chain to its very tip}. Every fragment terminates within the
window, which means that prefix selection will not just pick the preferred
\emph{prefix}, and not just the preferred \emph{fragment}, but in fact the
preferred \emph{chain}. This means that just like the theory assumes, we have
selected the best chain out of all possible chains, which means we can
conclude we are now completely up to date and can resume normal operation.

\begin{definition}[Recovering ``alert'' status]
\label{recover-alert-status}
When prefix selection can see all available chains to their tip, and we have
selected and adopted the best one, the node is up to date.
\end{definition}

(TODO\todo{TODO}: Christian tells me that the genesis proof also depends on
such a ``final chain selection''. Would be good to refer to that, but I'm
not sure where that happens in the paper.)

\subsection{Avoiding DoS attacks}
\label{genesis:becoming-alert:DoS}

Malicious nodes cannot abuse \cref{recover-alert-status} in an attempt to
prevent us from concluding we are up to date: as we saw, all blocks that get us
to within $s$ slots from the wallclock come from the honest chain, and once we
have reached $s$ slots from the wallclock, an adversary cannot present us with a
chain that exceeds the window, since blocks with a slot number after the
wall-clock are invalid.

We do have to be careful if we allow for clock skew however: if a malicious node
presents us with a header past the $s$ window (and hence past the wallclock,
though within permissible skew), we would not be able to conclude that we have
become alert. This would not stop prefix selection from doing its job---after
all, a header after the window means that we now have a known density---and so
we would continue to adopt blocks from the honest chain; however, the malicious
node could keep presenting us with a header that is just out of reach,
preventing us from ever concluding we are up to date and hence from producing
new blocks. The only reason we allow for clock skew at all, however, is to avoid
branding nodes as adversarial whereas in fact its just that our clocks are
misaligned. This must therefore be a best-effort only: allow for clock skew, but
not if this would exceed $s$ slots from the intersection point.

\subsection{Falling behind}

\Cref{recover-alert-status} gives us a way to discover that we are up to date.
Deciding when we are \emph{not} up to date is less critical. One option is to
simply use the inverse of \cref{recover-alert-status} and say we are not up to
date when one of our peers provides us with a chain that we cannot see until its
tip.  Another option is to assume we are not up to date when we boot up the
node, and then only conclude that have somehow fallen behind again if we notice
that our tip is more than a certain distance away from the wallclock (at most
$s$ slots). This may need some special care; if nodes stop producing blocks for
a while, we might end up in a state in which we both conclude that we are up to
date (because we can see all of our peer's chains to their tip) and not up to
date (because our tip is too far from the wallclock). However, this scenario
needs special case anyway; we will come back to it in \cref{low-density}.

\subsection{Block production}

In principle, block production can happen even when the node is not up to date.
There is however not much point: any blocks that the node will produce while it
is not up to date are likely to be discarded almost immediately after
production, because the node will prefer the existing (honest) chain over the
tiny fork that it itself created. Moreover, blocks that we produce while we are
not up to date may in fact be helpful for an adversary. We should therefore
disable block production while the node is not up to date.

\section{Implementation concerns}

\subsection{Chain selection in the chain database}
\label{genesis:chain-database}

We mentioned in \cref{genesis:prefix-selection} that the prefix selection
interface works equally well for the chain sync client and the chain database.
They are however not doing the same thing; this is a subtle point that deserves
to be spelled out in detail.

It comes back to the difference between perceived chain length and actual chain
length (\cref{genesis:background:joining-late}). In the chain sync client this
is a meaningful and useful difference: since we are tracking chains from
individual peers, it makes sense to distinguish between having seen the tip of
that particular chain, or only seeing a prefix of that chain. However, unless we
completely change the API to the chain database, the chain database just sees
individual blocks, without knowing anything about their provenance;  it  does
therefore not know if those blocks are the tip of ``their'' chains; it's not
even clear what that would mean.

Of course, when the chain database is constructing fragments of chains through
its volatile database, it knows if a particular block is the tip of any
constructed fragment. However, that is \emph{perceived} chain length: it might
be a tip just because we haven't downloaded any more blocks yet. The difference
is important. Consider two forks like this:
%
\begin{center}
\begin{tikzpicture}[yscale=0.75]
\draw (0, 0) -- (4, 0) coordinate(i);
\draw (i) -- ++(1,  1) -- ++(6,0) node[right]{$A$};
\draw (i) -- ++(1, -1) -- ++(6,0) node[right]{$B$};
\draw [dashed]
     (i)
  -- ++( 0,  2)
  -- ++( 4,  0) node[pos=0.5,above]{$\overbrace{\hspace{4cm}}^\text{$s$ slots}$}
  -- ++( 0, -4)
  -- ++(-4,  0)
  -- cycle;
\path
     (i)
  -- ++(0, 1)
  -- ++(4, 0) node[pos=0.5,above]{$\overbrace{\hspace{4cm}}^\text{denser}$};
\path
     (i)
  -- ++(0, -1)
  -- ++(4,  0) node[pos=0.5,below]{$\underbrace{\hspace{4cm}}_\text{$> k$ blocks}$};
\end{tikzpicture}
\end{center}
%
Chain $A$ is denser in the window, but $B$ nonetheless has more than $k$ blocks
in the window (this is entirely possible; a chain of normal density would have
$3k$ blocks in the window).  The chain sync client knows about both nodes, knows
that the chains extend past the window, will wait until it has seen sufficient
blocks from both chains (that is, for the first header outside the window), then
do a density comparison, find that $A$ is denser, and choose to download the
blocks from chain $A$.

But the chain database cannot replicate any of that reasoning. When blocks  from
either chain $A$ or chain $B$ are added, as far as the chain database is
concerned, those \emph{are} the tips of those chains. This means that is not
doing a density comparison, but a chain length comparison. What's worse, if more
than $k$ blocks from chain $B$ are added before it sees any blocks from chain
$A$, then it from that point forward be unable to switch to chain $A$, as this
would involve a rollback of more than $k$ blocks.

This is not necessarily problematic: since the chain sync client has more
context, it will make the right decision, and only present blocks from chain $A$
to the database. Indeed, as we saw in \cref{genesis:becoming-alert}, we will
in fact download \emph{only} blocks from the honest chain until we are $s$
slots away from the wallclock, at which point we do one final chain selection,
and we are up to date. At this point the Density Rule is \emph{anyway} just
selecting the longest chain, so the fact that the chain database is effectively
doing longest chain selection \emph{always} does not matter.

It does however mean that the block fetch client becomes an important
``guardian'' of the chain database; they become more tightly coupled than they
are currently. This is unfortunate, but not disastrous; it ``only'' makes the
system more difficult to understand. Solving this problem would require
rethinking how the chain database works; this is well outside the scope of this
chapter.

There is one advantage to this.  \Cref{genesis:prefix-selection} describes how
the chain database could \emph{in principle} use prefix selection: compute all
paths through the volatile database and then use prefix selection to construct a
prefix that the node should adopt as its current chain. While this is a useful
\emph{specification} of what the chain database must do, in practice we will
probably need an equivalent of \cref{focusonnewblock} that will allow us to
avoid looking at the entire volatile database every time a new block is added to
the database. If we however decide that the chain database is just selecting
based on chain length \emph{anyway}, then the existing lemma and existing
implementation can be used as-is.

\subsection{Abstract chain selection interface}

The current chain selection API compares two chains at a time, and only looks at
the tips of those two chains (\cref{consensus:overview:chainsel}). This will
obviously have to change; depending on how exactly we want to modify the chain
database (\cref{genesis:chain-database}), we must either replace the existing
interface with prefix selection as the primitive operation, or else add prefix
selection as an additional concept.

One of the reasons we currently only look at the tips of chains is because this
simplified treatment of chain selection in the hard fork combinator. This by
itself might not be too difficult to change; for example, we could set the
\lstinline!SelectView! of the hard fork block to be an $n$-ary sum of the
\lstinline!SelectView!s of the various eras. However, it is not a-priori clear
what it would mean to apply, say, the Optimum rule in one era on the chain,
and the Genesis rule in another. This will require some careful thought,
though we can probably just sidestep the entire issue and pretend we were
using the Genesis rule all along.

\subsection{Possible optimisations}
\label{genesis:optimisations}

Chain selection while we are not up to date has some properties that might
enable us to implement some performance optimisations. Here we just list some of
the possibilities:

\begin{itemize}

\item When a node is operational, we try to line up its average-case performance
requirements with its worst-case performance requirements, since this avoids
an attack vector: if the average-case performance would be significantly better
than the worst-case, it is likely that nodes would be optimised for the average
case (for instance, run on hardware that can handle the average case, but not
necessarily the worst case); then if a malicious node can intentionally cause
the worst-case, they might be able to bring down parts of the network.

For this reason we don't normally share work between various peers; when
multiple upstream peers all send us the same header, we verify the header
each time. This means that the average case (most upstream chains are the same)
and the worst case (every upstream chain is different) are equal.

However, it is less important that we can predict accurately how long it takes
a node (that isn't up to date) to sync with the network. Such a node is anyway
not producing blocks; here, faster is simply better. This means that while we
we are not up to date  we could share the validation work across upstream peers:
when two peers send us the same header, we do not need to validate it twice.

This is \emph{especially} important when we are not up to date, because due to
the requirement to have at least $\RequiredPeers$ upstream peers, we might be
connecting to more peers than usual. Moreover, under normal circumstances we
expect all of these peers to present us with exactly the same chain (and
finally, these cryptographic checks are expensive).

\item Similarly, since we expect all upstream nodes to report the same chain,
if we receive a bunch of headers from peer 1, we can just ask peer 2
whether they have the most recent of those headers on their chain, thereby
skipping over large chunks of the chain altogether.

\item Since we only ever fetch blocks strictly in order, we can simplify
the interaction with the block fetch client: it might be easier to generate
longer fetch ranges, as well as spread the load more evenly across the peers.

\item We saw in \cref{genesis:becoming-alert} that any blocks that we download
while syncing which are more than $s$ slots away from the wall clock, will be
blocks from the common prefix of the honest chains and will not have to be
rolled back. It might therefore be possible to bypass the volatile database
entirely. However, how this works when we switch back from being up to date
to not being up to date would require careful thought.

\end{itemize}


%% rnaastex.cls is the classfile used for Research Notes. It is derived
%% from aastex61.cls with a few tweaks to allow for the unique format required.
\documentclass{rnaastex}

%% Define new commands here
\newcommand\latex{La\TeX}

\newcommand{\vM}{\mathsf{M}}
\newcommand{\vD}{\mathsf{D}}
\newcommand{\vd}{\mathsf{d}}
\newcommand{\vrr}{\mathsf{r}}
\newcommand{\vm}{\mathsf{m}}
\newcommand{\vR}{\mathsf{R}}
\newcommand{\vC}{\mathsf{C}}
\newcommand{\vA}{\mathsf{A}}
\newcommand{\va}{\mathsf{a}}
\newcommand{\vF}{\mathsf{F}}
\newcommand{\vf}{\mathsf{f}}
\newcommand{\vy}{\mathsf{y}}
\newcommand{\trans}{\mathsf{T}}

\begin{document}

\title{Inference of the K2 PSF}

%% Note that the corresponding author command and emails has to come
%% before everything else. Also place all the emails in the \email
%% command instead of using multiple \email calls.
\correspondingauthor{Michael Gully-Santiago}
\email{igully@gmail.com}

\author[0000-0002-4020-3457]{Michael Gully-Santiago}
\affiliation{Kepler/K2 Guest Observer Office \\
NASA Ames Research Center \\
Moffett Blvd \\
Mountain View, CA 94035, USA}
\affiliation{Bay Area Environmental Reaearch Institute}

\author{Gosemite Team and TBD}
\affiliation{TBD}

%% See the online documentation for the full list of available subject
%% keywords and the rules for their use.
\keywords{methods: statistical, techniques: image processing, methods: data analysis}

%% Start the main body of the article. If no sections in the
%% research note leave the \section call blank to make the title.
\section{}

We infer PSFs in Kepler data.  Here is a citation: \cite{2015Natur.521..332O}.

\section{PSF photometry for Kepler}
Astronomical photometry has conventionally followed two paths: aperture photometry or Point Spread Function (PSF) photometry.  Point spread function photometry excels in crowded regions and in low Signal-to-Noise ($S/N$) ratio regimes.  Kepler/K2 mission time-series imaging of star clusters and extended objects will benefit from PSF photometry.  The large computational cost of PSF photometry has hampered its application to crowded fields in Kepler/K2, slowing the unbiased analysis of existing valuable data. In this research note, we lay the foundation for the application of GPU-acceleration applied to PSF photometry, demonstrating a speed-up over conventional computational methods.  We apply our method to both synthetic data, recovering unbiased estimates of our inputs, and real K2 data.  In the latter case we derive the Kepler PSF itself, the individual fluxes of stars, and the covariance among all inferred stars.  Recent advances in GPU acceleration hardware and associated software libraries offer avenues for performance improvements in other domains in astrophysical data analysis.

\section{Methodology}
We make these basic assumptions about our problem structure:

\begin{enumerate}
  \item All sources are point sources.
  \item There are exactly $N_{\star}$ sources.
  \item All sources share the same PSF.
  \item All pixels share the same response function.
  \item The PSF is drawn from a Gaussian Mixture Model with $N_{GMM} = 3$ unknown components.
  \item Each source has 3 parameters-- $x_{\star,i}, y_{\star,i}$ center locations and flux amplitude, $f_i$.
  \item Each PSF component has 5 parameters-- relative $(x_j, y_j)$ location; amplitude $z_j$; and bivariate covariance matrix $C_j$, which is comprised of 3 parameters $(\phi_a, \phi_b, \phi_c)$.
  \item No PSFs are saturated.
  \item All pixels have known, homoscedastic Gaussian read noise $\sigma_{r}$.
  \item The flat field is uniform, or equivalently has already been corrected perfectly.
  \item There are $N_{pix}$ pixels in the scene.
\end{enumerate}

We fix the first Gaussian Mixture Model components $x_1,y_1 \equiv 0, 0$, enforce $z_{j+1} < z_{j}$, and enforce normalization $\int \mathcal{K} \,dx\,dy = 1$. The above assumptions yield a total number of parameters, $N_{\mathrm{param}} = 3 N_\star + 6 N_{GMM} - 3$.

We identified two nearly identical procedures for calculating the likelihood.  In procedure I, we instantiate $N_\star$ delta functions in a sub-sampled pixel grid, convolve a model PSF with the delta functions, and sum the sub-sampled pixels in a Kepler Pixel.  In procedure II, instantiate $N_\star$ model PSFs, evaluating the model at each pixel center.  We anticipate that these two procedures produce the same results, but procedure II is likely to possess smoother derivatives than procedure I, resulting in higher numerical performance.  Procedure II can only be carried out assuming statement number 4 above holds.  We adopted procedure II.  The source of the PSFs can be either source detection in the local image (\emph{e.g. Source Extractor}), or an outside deep catalog in a similar bandpass (\emph{e.g. Gaia}).  We performed source detection for the current note.

The models are:

\begin{eqnarray}
\mathcal{K}_{\mathrm{PSF}} = \sum_{j=0}^{N_{\mathrm{GMM}}} z_j \mathcal{G}(x-x_j, y-y_j, C_j) \\
\hat I(x,y) = \sum_i^{N_\star} f_i \cdot \mathcal{K}_{\mathrm{PSF}}(x-x_{\star, i}, y-y_{\star, i})
\end{eqnarray}

We can rewrite the sums as matrix products.  We assign $ \vd $ as the $N_{pix} \times 1$ column vector mapping all data pixel values from the 2D image onto a 1D vector.  The vector $ \va $ is the 2D PSF model for the $i^{th}$ star mapped onto an $N_{pix} \times 1$ one-dimensional column vector.  The matrix $\vA$ represents the column-wise concatenation of the $N_{\star}$ models $\va_{i}$, yielding an $N_{pix} \times N_{\star}$ matrix.  The vector $\vf$ contains the $N_{\star}$ PSF flux amplitudes $f_i$.  We can analytically compute the ``profile likelihood'' flux amplitudes:

\begin{eqnarray}
\vf^{*} = (\vA^{\trans} \vC^{-1} \vA)^{-1}(\vA^{\trans} \vC^{-1} \vd)
\end{eqnarray}

Computing the $\vf^{*}$ analytically eliminates $N_\star$ linear parameters (the $f_i$'s) from the $N_{\mathrm{param}}$ parameters, and delivers the covariance among the inferred amplitudes.  We derive the non-linear parameters by minimizing $\chi^2$:

\begin{eqnarray}
\vm &=& \vA \cdot \vf^{*} \\
\vrr &=& \vd - \vm \\
  \ln{p(\vd | \vm)} &=& -\frac{1}{2} \left( \vrr^\trans \vC^{-1} \vrr + \ln{\det{\vC}} + N_{pix} \ln{2\pi} \right)
\end{eqnarray}

where $\vC$ is the pixel noise covariance matrix.  For our current assumption of homoscedastic read noise $\vC = \sigma_r \mathcal{I}$ where $\mathcal{I}$ is the diagonal identity matrix.

\section{Application- PSF photometry on synthetic data}

\begin{itemize}
\item Application- PSF photometry
\item Synthetic data generation
\item Application to synthetic data
\item (Stretch goal): Application to real patch of FFI data
\item (Stretch goal): Performance with CPU
\item (Stretch goal): Performance with GPU
\end{itemize}

\section{Appendix}

Here we clarify the notation and offer additional mathematical transformations used in this note.  The normalized Gaussian mixture model takes the form:

\begin{eqnarray}
\mathcal{G} &=& \frac{1}{2 \pi \sqrt{\det{C_j} }} e^{-\frac{1}{2} \vec{r}^{\trans} C_j^{-1} \vec{r}}
\end{eqnarray}

with $\vec{r}=\vec{x}+\vec{y}$ representing the distance vector.  The Kernel and data positions refer to their respective reference frames, with Kernels representing continuous variables and data spaces representing pixel coordinates.  The bivariate Gaussian mixture model covariance matrix can be written down as:
\begin{eqnarray}
   C_j=
  \left[ {\begin{array}{cc}
   e^{2 \phi_a} & \;\;\phi_c e^{\phi_a} \\
   \phi_c e^{\phi_a} & \;\;\phi_c^2 +e^{2\phi_b} \\
  \end{array} } \right]
\end{eqnarray}

which enforces positive scale factors and positive semidefinite matrices.  Enforcing normalization $\int \mathcal{K} \,dx\,dy \equiv 1$ results in $\sum_{j=0}^{N_{\mathrm{GMM}}} z_j = 1$.  We apply several transformations for computational expediency:

\begin{eqnarray}
z_j - z_{j-1} &=& e^{-\theta_j} \\
z_1 &=& \frac{1}{1 + \sum_{j=2}^{N_{\mathrm{GMM}}} z_j} \\
\theta_{j} >0
\end{eqnarray}

%%%%%%%FIGURE%%%%%%%
%\begin{figure}[h!]
%\begin{center}
%        \includegraphics[scale=0.85,angle=0]{figa.pdf}
%\caption{Demo of our K2 PSF inferred from data\label{fig:1}}
%\end{center}
%\end{figure}

\acknowledgments

This research has made use of NASA's Astrophysics Data System.  The reproducible Jupyter Notebook that generated the figures in this document are freely available \href{https://github.com/gully/quigly}{on GitHub}.

\begin{thebibliography}{}

\bibitem[Olling et al.(2015)]{2015Natur.521..332O} Olling, R.~P., Mushotzky, R., Shaya, E.~J., et al.\ 2015, \nat, 521, 332

\end{thebibliography}

\end{document}


\documentclass[12pt]{article}
\usepackage{amsfonts}

\newcommand{\real}{{\rm Re}}
\newcommand{\imag}{{\rm Im}}

\begin{document}

\section{Folded Profile Scattered Power Correction}

This document describes a 2-bit scattered power correction algorithm
that can be used to correct folded pulsar profiles.  The algorithm
requires that the signal in each frequency channel has not been
significantly altered, such that the mean undigitized power can be
trivially computed from the mean digitized power.  This typically
means that either the data have not been coherently dedispersed or the
dispersion smearing in each frequency channel is less than the time
resolution of the folded profile.
%
The digitized power $\hat\sigma^2$ is given by equation (A5) of
Jenet \& Anderson (1998); hereafter JA98,
%
\[
\hat\sigma^2 = \sum_\Phi {\mathcal P}(\Phi)f(\Phi),
\label{eqn:A5}
\]
where $\Phi$ is the fraction of samples that fall between the chosen
thresholds, $f(\Phi)$ is the digitized power as a function of $\Phi$,
given by equation (A4) of JA98, and ${\mathcal P}(\Phi)$ is the
discrete probability distribution for $\Phi$, given by equation (A6)
of JA98.
%
The parameter $\Phi$ is eliminated by the summation in equation~(A5);
however, both $f(\Phi)$ and ${\mathcal P}(\Phi)$ are also
parameterized by the expectation value, $\langle\Phi\rangle$.
%
Therefore, equation~(A5) represents the relationship between the mean
digitized power and the mean value of $\Phi$.
%
That is, given the digitized power $\hat\sigma^2$, equation (A5) of
JA98 can be inverted to compute $\langle\Phi\rangle$, which can in
turn be used to estimate the mean undigitized power $\sigma^2$ and the
mean scattered power $A$ via equations (45) and (43) of JA98,
respectively.

Equation~(A5) can be inverted using the Newton-Raphson method and the
partial derivatives of equations (A4) through (A6) with respect to
$\langle\Phi\rangle$.
%
These are simplified in the case of two-bit sampling by noting that
equation (A4) reduces to
%
\[
f(\Phi) = \langle\Phi\rangle y_3^2(\Phi)  + (1-\langle\Phi\rangle) y_4^2(\Phi)
\]
%
where $y_3(\Phi)$ and $y_4(\Phi)$ are given by equations (41)
and (40), respectively, of JA98.

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Carlos Santos             %
% ECE 351-51                %
% Lab 3                     %
% 2/18/2020                 %
%                           %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[12pt]{article}

% Language and font encoding
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{caption}
\usepackage{float}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{rotating}
\usepackage{setspace}

% Sets page size and margins
\usepackage[a4paper,top=3cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}

% Useful packages
\usepackage[colorinlistoftodos]{todonotes}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}
\usepackage{listings}
\usepackage{gensymb}
\usepackage{graphicx} %package to manage images
\usepackage{listings}
\usepackage{color}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=tb,
  language=Python,
  aboveskip=3mm,
  belowskip=3mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\small\ttfamily},
  numbers=none,
  numberstyle=\tiny\color{gray},
  keywordstyle=\color{blue},
  commentstyle=\color{dkgreen},
  stringstyle=\color{mauve},
  breaklines=true,
  breakatwhitespace=true,
  tabsize=3
}

%Line Spacing
\setstretch{1.5}

%Info for Title Page
\title{ECE 351 Lab 3 Report \\ Section 52}
\date{February 28, 2020}
\author{Carlos Santos}
\begin{document}

%Make a Title Page
\vspace{\fill}
\maketitle
\vspace{\fill}
\clearpage

%Introduction
\section*{Introduction}

The purpose of this lab is to become familiar with convolutions and its properties. To do that we will use the Python programming language.

%Equations
\section*{Equations}

\begin{equation}
    f_1t = u(t-2) - u(t-9)
\end{equation}
\begin{equation}
    f_2t = e^{-t} * u(t)
\end{equation}
\begin{equation}
    f_3t = r(t-2)[u(t-2) - u(t-3)] + r(4-t)[u(t-3) - u(t-4)]
\end{equation}

%Methodology
\section*{Methodology}

\begin{enumerate}
    \item The first thing  I did was to bring over my ramp and step function code from the previous lab 2.
    \item I then test the lab 2 code with the new functions given in this lab 3.
    \item Once that was done it was time to write the convolution code.
    \item The code below is what we came up with together as a class.
\end{enumerate}

\begin{lstlisting}
// Task 1 Code
#############################################################
#Section 4.1, 4.2, 4.3

def convo(t1,t2):

    t1_len = len(t1)
    t2_len = len(t2)

    Et1 = np.append(t1,np.zeros((1,t2_len - 1))) # extend the first function
    Et2 = np.append(t2,np.zeros((1,t1_len - 1)))# extend the first function

    res = np.zeros(Et1.shape) # works when we start at zero

    for i in range((t1_len + t2_len) - 2 ): # First Loop
        res[i] = 0

        for j in range(t1_len): # Second Loop
            if(i-j+1  > 0):
                try:
                    res[i] += (Et1[i] * Et2[i-j+1])
                except:
                    print(i,j)

    return res

\end{lstlisting}

%Results
\section*{Results}

\begin{figure}[H]
\caption{User Defined Functions}
\centering
\includegraphics[width=.8\textwidth]{User_Defined_Functions.png}
\end{figure}

\begin{figure}[H]
\caption{Check with sig.convo}
\centering
\includegraphics[width=.8\textwidth]{check_with_sig_convo.png}
\end{figure}

%Questions
\section*{Questions}

\begin{enumerate}
    \item Did you work alone or with classmates on this lab? If you collaborated to get to the solution,
what did that process look like?
    \begin{enumerate}
        \item For this lab I worked with my classmates.  In order to get to the solution we had to think outside the box a bit. We broke down what convolution is into small parts that a computer can calculate. Then the pieces were put together and we prayed to the Python gods that it would work. Unfortunately it didn't work but we were close.
    \end{enumerate}
    \item What was the most difficult part of this lab for you, and what did your problem-solving
process look like?
    \begin{enumerate}
        \item The most difficult thing for me was taking a concept we know like convolution and translating that into code. To make this easier I broke down the idea of convolution into smaller pieces and then coded up the small pieces. For example I know that the duration of convolving two equations is the sum of their total times. In coding this is represented by arrays that need to be extended and have the same sizes.
    \end{enumerate}
    \item Did you approach writing the code with analytical or graphical convolution in mind? Why
did you chose this approach?
    \begin{enumerate}
        \item I approached writing this code with graphical convolution in mind. I did this because I like to picture what the code is supposed to do and then code it up. I find that easier than an analytical approach.
    \end{enumerate}
    \item Leave any feedback on the clarity of lab tasks, expectations, and deliverables
    \begin{enumerate}
        \item N/A
    \end{enumerate}
\end{enumerate}

%Conclusion
\section*{Conclusion}

Convolution is a complex thing to wrap our minds around. Luckily we have machines that can do complex convolutions for us. The only catch is we have to capture our ideas as code for the machines. In the end why do convolutions by hand when a computer can do it for us?

\end{document}