\section{Introduction}
\label{sec:intro}
\citet{Manson:2005:JMM:1047659.1040336} identify the central problem in the
design of software relaxed memory models: ``The memory model must strike a
balance between ease-of-use for programmers and implementation flexibility
for system designers.''

%In order to sharpen the criteria to evaluate memory models, we first outline desiderata  that concretize the above prescription.

There are two aspects to ``ease of use.''  First, programs should support
\emph{compositional} and \emph{local} reasoning; in this paper, we emphasize
temporal safety
properties~\cite{PnueliSafety,Misra:1981:PNP:1313338.1313770,StarkSafety,Abadi:1993:CS:151646.151649}.
Second, relaxing memory consistency should not change the behavior of correctly
synchronized programs; this property is known as \emph{sequential consistency
  for data race free programs} (\drfsc)~\cite{DBLP:journals/tpds/AdveH93,
  DBLP:conf/isca/AdveH90}.
 % Second, the expectation of
% \emph{sequential consistency for data race free programs}
% (\drfsc)~\cite{DBLP:journals/tpds/AdveH93, DBLP:conf/isca/AdveH90} permits
% the programmer to forget about relaxed memory for correctly synchronized
% programs.

% There are two aspects to ``ease of use.''  First, programmers should be able
% to reason about program fragments \emph{compositionally} and \emph{locally};
% in this paper we emphasize compositional reasoning about temporal safety
% properties~\cite{PnueliSafety,Misra:1981:PNP:1313338.1313770,StarkSafety,Abadi:1993:CS:151646.151649}.
% Second, programmers should be able to forget about relaxed memory in
% correctly synchronized programs; this property is known as \emph{sequential
%   consistency for data race free programs}
% (\drfsc)~\cite{DBLP:journals/tpds/AdveH93, DBLP:conf/isca/AdveH90}.

There are also two aspects of ``implementation flexibility.''  First, relaxed
atomic access should not require hardware synchronization (at least for the
word size of the machine).  Second, the model
should facilitate compiler transformations, such as the reordering of independent
statements; ideally the model should support all %valid
optimizations of
synchronization\hyp{}free single\hyp{}threaded code.
% A
% canonical example is that independent statements should commute.

Sailing between this Scylla and Charybdis has proven very difficult.  Three
lines of code can leave the top experts in the field flabbergasted.  The
solutions that have been proposed are understandable to mechanical proof
assistants, but humans have been left behind.

In this paper, we combine two ideas that humans can understand: \emph{preconditions}
\cite{Hoare:1969:ABC:363235.363259} and \emph{labelled partial orders} (aka \emph{pomsets})
\cite{GISCHER1988199,Plotkin:1997:TSP:266557.266600}.  The resulting model
mostly satisfies the desiderata.  We sacrifice only implementability on
``non-\mca'' processors, such as \ppc\ and \armseven{}. As a result, however,
there is only one order relation to visualize.

Perhaps you believe the problem has already been solved?  Let us try to
convince you otherwise.
% Eventually,
% the enthusiasm of proposing a new model yields to weariness as the
% complications mount.

To get a sense of the difficulties involved,
% \citeauthor{DBLP:journals/toplas/Lochbihler13}'s example shows a surprising feature of
% relaxed memory models: the presence of an execution may depend on code that was not executed. Consider:
consider that the existence of an execution in a relaxed memory model may
depend on code that was \emph{not} executed. Let $r$--$s$ be registers and
$b$, $x$--$z$ be shared memory locations.  Consider the following program, where
all memory locations are initialized to $0$:
\begin{align*}
  \tag{\ref{OOTA?}}
    \PW{y}{x}
  \PAR&
  \PR{y}{r}\SEMI\IF{r}\THEN
  \PW{x}{r}\SEMI \PW{z}{r} \ELSE \PW{x}{1} \FI
\intertext{Most programmers would be surprised to learn that this program allows an
execution that sets $z$ to $1$. To see why, imagine that a compiler does type
inference and finds that $x$ and $y$ are booleans, with value either $0$ or
$1$.  This enables the program to be optimized to the following:}
  %\taglabelp{OOTA?}
    \PW{y}{x}
  \PAR&
  \PR{y}{r}\SEMI\IF{r}\THEN
  \PW{x}{1}\SEMI \PW{z}{1} \ELSE \PW{x}{1} \FI
\intertext{Since $\PW{x}{1}$ occurs in both branches of the conditional, the compiler can
then lift it, and reorder with the independent read of $y$, yielding:}
  %\taglabelpp{OOTA?}
    \PW{y}{x}
  \PAR&
  \PW{x}{1}\SEMI
  \PR{y}{r}\SEMI\IF{r}\THEN \PW{z}{1} \FI
\intertext{Then $z$ is $1$ at then end of an execution where the first thread is interleaved
immediately after executing $\PW{x}{1}$.
Without the conditional in \eqref{OOTA?}, it is obvious that the program should not write $1$:}
  \tag{\ref{OOTA3}}
  %\PW{x}{0}\SEMI \PW{y}{0}\SEMI
  %(\bLoc \GETS \aLoc \PAR \aLoc \GETS \bLoc)
  \PW{\bLoc}{\aLoc} \PAR&
  \PR{y}{r}\SEMI \PW{x}{r}  \SEMI \PW{z}{r}
\end{align*}
In \ref{OOTA3}, the constant $1$ arises ``Out Of Thin Air'' (\oota)
\cite{DBLP:conf/esop/BattyMNPS15}.  As a result, any model of relaxed memory
that supports common compiler optimizations, as above, must take into account
code that was not executed.  This is why many models of relaxed memory
include some form of speculative execution, with the goal of allowing the
outcome $z{=}1$ for \eqref{OOTA?}, but not \ref{OOTA3}.

The control flow variant of \ref{OOTA3} is:
\begin{displaymath}
  \label{cyc}\tag{\textsc{oota2}}
  \IF{x}\THEN \PW{y}{1} \FI \!\PAR\! \IF{y}\THEN \PW{x}{1} \SEMI \PW{z}{1}\FI
\end{displaymath}
This program is data-race-free. Thus, allowing an execution that writes $1$
would violate \drfsc{}.

\oota{} behaviors can be quite subtle.
Consider the following variants of \eqref{OOTA?}:
\begin{align}
  \tag{\ref{OOTA!}}
    \PW{y}{x}
  \PAR&
  \PR{y}{r}\SEMI \IF{r}\THEN
  \PW{x}{r}\SEMI \PW{z}{r} \ELSE \PW{x}{2} \FI
  \\
  \tag{\ref{OOTA4}}
  \PW{y}{x}
  \PAR&
  \PR{y}{r} \SEMI \IF{b}\THEN  \PW{x}{r} \SEMI \PW{z}{r} \ELSE \PW{x}{1} \FI
  \PAR
  \PW{b}{1}
\end{align}
Following the reasoning above, \ref{OOTA!} has an execution where
$z{=}2$, but it does not have an execution where $z{=}1$.   Neither does
\ref{OOTA4}.  In this case, it not sound to assume that  $1$ is
written on both sides of the conditional, invalidating the first optimization
given for \eqref{OOTA?} above.
% Note that the only
% difference between \eqref{OOTA?} and \eqref{OOTA!} is the code in the branch
% that was not taken.

% Many models of relaxed
% memory include some form of speculative execution in order to allow $1$ in
% \eqref{OOTA?}, but not \eqref{OOTA!} and \ref{OOTA3}.
% Any model of relaxed memory that supports common compiler optimizations
% (such as the transformations above) must take into account code that was
% not executed. This is why many models of relaxed memory include some form
% of speculative execution.

% Any model of relaxed memory that supports common compiler optimizations, as
% above, must take into account code that was not executed.  This is why many
% models of relaxed memory include some form of speculative execution, with the
% goal of allowing $1$ in \eqref{OOTA?}, but not \eqref{OOTA!} and
% \ref{OOTA3}.
% Many models of relaxed memory include some form of speculative execution in
% order to allow $1$ in \eqref{OOTA?}, but not \eqref{OOTA!} and
% \ref{OOTA3}.

\citeauthor{DBLP:conf/java/Pugh99} [\citeyear{DBLP:conf/java/Pugh99},
\textsection2.3] initiated the modern study of relaxed memory by noting that
Java 1.1 failed to validate Common Subexpression Elimination (CSE) in the
presence of aliasing. For example, given that $\aReg_2{\neq}\bReg$, is it valid
to transform the program on the left to that on the right?
\begin{align*}
  %\tag{\ref{CSE}}
  ({r_1\GETS \aLoc \SEMI
  s\GETS \bLoc \SEMI
  r_2\GETS \aLoc\SEMI\aCmd})
&&
  ({r_1\GETS \aLoc \SEMI
    r_2\GETS r_1\SEMI
    s\GETS \bLoc \SEMI\aCmd})
\end{align*}
The resulting Java Memory Model (JMM) \cite{Manson:2005:JMM:1047659.1040336}
greatly advanced the state of the art.

\citeauthor{DBLP:journals/toplas/Lochbihler13}'s
monumental study of the JMM
revealed a surprising limitation. Consider the following program
\citep[Fig.~8]{DBLP:journals/toplas/Lochbihler13}, where again
all memory locations are initialized to $0$:
%\vspace{-.5ex}
\begin{gather}
  \tag{\textsc{oota5}}\label{OOTA1}
  \PW{y}{x}
  \PAR
  \PR{y}{\aReg} \SEMI \IF{b} \THEN \LET{\aReg}{\NEW \classD} \SEMI \PW{x}{\aReg} \SEMI \PW{z}{\aReg} \ELSE \LET{\bReg}{\NEW \classC} \SEMI \PW{x}{\aReg} \FI
  \PAR %\\[-1ex] \PAR&
  \PW{b}{1}
\end{gather}
\ref{OOTA1} ``is type correct if it declares $x$, $y$ and $r$ of type
$\classD$. However, it has a legal execution where they reference a $\classC$
object.''  The JMM allows $(\PR{y}{\aReg})$ to see the object created by
$\NEW$, by bouncing it through $(\PW{x}{\aReg})$ and $(\PW{y}{x})$.  In the
\emph{commitment order} of the JMM, this allows the address of the
allocated object to be read $(\PR{y}{\aReg})$ before its type is determined
$(\IF{b})$.

% See \ref{OOTA4} in
% \textsection\ref{sec:logic} for a simple variant of \eqref{OOTA?} with this
% behavior.

% By allowing such \emph{bait-and-switch} behaviors, the JMM fails to support
% compositional reasoning for temporal safety properties:  the individual
% threads of \ref{OOTA1} satisfy the invariant
% % \emph{allocation at type $\classD$ is preceded by reading zero for $b$}.
% % They also satisfy:
% \emph{allocation at type $\classC$ is preceded by reading $0$ for
%   $b$}, but the composed program does not.  This lack of compositionality

This type of \emph{bait-and-switch} behavior forced
\citeauthor{DBLP:journals/toplas/Lochbihler13} to partition memory by type in
order to prove type safety.  This formal device means that memory cannot be
used at different types over time, making practical memory reclamation
impossible.  Even partitioning memory to achieve type safety, there are
implications for the Java security architecture
\cite[\textsection5.4]{DBLP:journals/toplas/Lochbihler13}.

In both \ref{OOTA4} and \ref{OOTA1}, the \oota{} outcome occurs by
\emph{baiting} with the \texttt{else} branch, then \emph{switching} to the
\texttt{then} branch, based on a coin flip $(\IF{b})$.  As confirmed by
\cite{kang,soham}, the promising semantics \cite{DBLP:conf/popl/KangHLVD17}
and related models
\citep{DBLP:conf/esop/JagadeesanPR10,DBLP:journals/pacmpl/ChakrabortyV19,Manson:2005:JMM:1047659.1040336}
all allow \oota{} behaviors of \ref{OOTA4}.\footnote{Call the threads
  \texttt{s}, \texttt{t}, and \texttt{u}.  To get the result in the promising
  semantics, first execute \texttt{u} to get message \texttt{<b:1@1>}.  Then
  \texttt{t} promises \texttt{<x:1@1>}, which it can fulfill by reading
  \texttt{b}$=$\texttt{0}.  Then execute \texttt{s} to get message \texttt{<y:1@1>}.
  Then execute \texttt{t}, reading \texttt{b}$=$\texttt{1} and \texttt{y}$=$\texttt{1} and
  fulfill the promise by writing \texttt{<x:1@1>}. The execution is exactly
  the same in our speculative semantics \cite{DBLP:conf/esop/JagadeesanPR10},
  removing timestamps and replacing the word \emph{promise} by \emph{speculation}.}  Due
to the similarity of \ref{OOTA4} and \ref{OOTA1}, it is reasonable to
conclude that these models \emph{cannot support both type safety and
  realistic memory reclamation}.

% The JMM is defined using an operational model that incorporates the idea of
% restarting executions.  Similar models have used speculation
% \cite{DBLP:conf/esop/JagadeesanPR10} or promises
% \cite{DBLP:conf/popl/KangHLVD17}.

% % Whereas the JMM was designed with these two aims,
% , invalidating compositional reasoning for temporal safety
% properties.
% % , as in
% % \ref{OOTA1}.  % (Specifically, they
% % allow the variant \ref{OOTA4} given in \textsection\ref{sec:logic}.)
% % See
% % \textsection\ref{sec:promising} for confirmation from the authors themselves.
% This means that these models \emph{cannot support both type safety and
%   realistic memory reclamation}.

% Due to the shortcomings of the JMM, many subsequent models preferred an
% axiomatic approach.
The C11 Memory Model \cite{Batty:2011:MCC:1926385.1926394} does not attempt
to validate CSE, at least not for relaxed atomic access (consider the case
where $x$ and $y$ are aliased above).  C11 \emph{does} allow the
transformation for \emph{plain} access, but this comes with the threat of
\emph{undefined behavior} should any plain access ever possibly engage in a
data race \cite{undefined}.  C11 also allows \oota{} behaviors, exploiting
causality cycles.  Undefined/\oota{} behavior is antithetical to the goals of
safe languages.
% Thus the folklore belief that ``every substantial C program has
% undefined behavior.''

%to permit us to reason separately about individual threads validating safety properties.
% ``Out Of Thin Air'' (\oota) executions invalidate the
% composability  of safety properties.

% None of the extant memory models validate both
% ``implementation flexibility'' and ``ease of use.''  This paper provides a
% solution.  The solution is remarkably simple, enriching labelled partial
% orders (aka \emph{pomsets}) with preconditions.

Strong models, including Sequential Consistency
(SC)~\citep{Lamport:1979:MMC:1311099.1311750}, RC11
\citep{DBLP:conf/pldi/LahavVKHD17}, and others
\citep{Dolan:2018:BDR:3192366.3192421,DBLP:conf/pldi/LahavVKHD17,DBLP:conf/lics/JeffreyR16,Boehm:2014:OGA:2618128.2618134},
support compositional reasoning.  However, all of these models invalidate
reordering of independent statements.  All require fences after relaxed
reads, even on \armeight.
% \cite{Dolan:2018:BDR:3192366.3192421,BoehmOOTA,DBLP:conf/lics/JeffreyR16} forbid breaking of the program order from reads to writes and thus require extra fences after read actions in hardware implementations.
% \citep{Boehm:2014:OGA:2618128.2618134} show that the RC11 model %\cite{DBLP:conf/pldi/LahavVKHD17}
% forces a dependency or a fence between a relaxed atomic read and a subsequent relaxed atomic write.

\myparagraph{Our Model}

In our approach, a program is a set of executions.  Each execution is a
\emph{pomset}: a partial order over a set of read and write events.  The
order is intended to be read as a \emph{dependency} relation.  The dependency
relation is dynamic, varying between executions.  Events that are not related
in an execution are \emph{independent} and can be seen by a sequential
observer in either order.

Cross thread dependencies arise from conflicting actions on the same
variable: Roughly, we order any two actions on the same location, at least
one of which is a write.  In the parlance of hardware memory
models~\citep{alglave}: $\rcoe$, $\rfre$, and $\rrfex$ are included in the
global dependency ordering.  Thus, our model realizes \emph{multi-copy
  atomicity} (\mca): when a write becomes visible to one thread it must
become visible to all\nofootnote{\mca\ is traditionally explored in hardware
  memory models.  \tso\ (see, e.g.~\cite{DBLP:journals/cacm/SewellSONM10})
  and recent architectures, such as \armeight\ (see,
  e.g.~\cite{DBLP:journals/pacmpl/PulteFDFSS18}, RISC-V.  ), are \mca, but
  not older architectures, such as \ppc\ (see,
  e.g.~\cite{DBLP:conf/pldi/SarkarSAMW11}) or \armseven\ (see,
  e.g.~\cite{DBLP:conf/popl/AlglaveFIMSSN09}).}
\citep{DBLP:journals/pacmpl/PulteFDFSS18}.  As envisioned in
\cite[\textsection3.3]{AlglaveThesis}, this allows us to capture cross-thread
dependencies in a single partial order.

Our key insight is that \emph{\mca{} permits a single, global notion of time,
  manifest in the pomset order}.

Within a thread, the dependency calculation can be viewed as the computation
of \emph{preserved program order}, called $\rppo$ in hardware models.  In our
software model, $\rppo$ captures the \emph{essential dependencies} between
events in the same thread.  Consider the following program fragments:
\begin{align*}
  & \aCmd_1: \PW{\aLoc}{1} \SEMI \PW{\bLoc}{1}
  \\[-1ex] & \aCmd_2: \aReg \GETS \aLoc \SEMI \IF{\aReg} \THEN\ \PW{\bLoc}{1} \ELSE \PW{\bLoc}{1}  \FI
  \\[-1ex] & \aCmd_3: \PW{\aLoc}{1} \SEMI \aReg \GETS \aLoc \SEMI \IF{\aReg} \THEN \PW{\bLoc}{1} \FI
  %\\[-1ex] &\aCmd_4:  \PW{\bLoc}{1} \SEMI \PW{\aLoc}{1}
\end{align*}
Each of these fragments satisfy the \citeauthor{Hoare:1969:ABC:363235.363259} triple $\hoare{\TRUE}{\aCmd_i}{\bLoc =1}$; thus, in each
case, the write of $y$ is independent of any code that precedes it in program
order. While $\aCmd_1$ reflects syntactic independence, $\aCmd_2$ reflects
the independence derived by case analysis, and $\aCmd_3$ reflects the
independence deduced from partial evaluation, in the restricted form of
constant propagation.

Our key insight is to that \emph{logic is better than syntax} to capture such
dependencies.

The logical perspective provides a clear intuition as to why certain compiler
transformations should be valid.  Such intuitions are not always readily
available in relaxed memory models.  For example, \emph{value range
  analysis}---used in the discussion of \eqref{OOTA?}---is difficult in many
models.  As another example, models such as \armeight{} distinguish
\emph{internal} reads, which are fulfilled by a write of the same thread,
from \emph{external} ones, which are fulfilled cross-thread.  Unlike
external reads, internal reads are not necessarily recorded in
the dependency relation.  As exemplified by $\aCmd_3$, this allows a compiler
to reorder the fulfilling write with subsequent code that depends on the
read.  Neither value range analysis nor internal reads require special
treatment in our model.

% The logical perspective provides a clear intuition as to why our model validates
% the compiler transformations discussed above \eqref{OOTA?}.
% reordering of independent statements, Irrelevant read introduction, RaR, RaW,
% WaW and Read Reordering and motivates the expressivity wrt the \jmm\
% causality test cases.  For example, consider Test Case 17 from Pugh.
% \[
% \begin{array}{ll}
%  &\aReg_3 \GETS \aLoc \SEMI \IF{\aReg_3 \neq 42} \THEN \PW{\aLoc}{42}  \FI \SEMI \aReg_1 \GETS \aLoc \SEMI \bLoc \GETS \aReg_1  \\
%  \PAR & \aReg_2 \GETS \bLoc \SEMI \aLoc \GETS \aReg
% \end{array}
% \]
% \jmm\ permits the behavior $\aReg_1 = \aReg_2 = \aReg_3 =42$.

% We deduce:
% \[
% \hoare{\TRUE}{\aReg_3 \GETS \aLoc \SEMI \IF{\aReg_3 \neq 42} \THEN \PW{\aLoc}{42}  \FI \SEMI \aReg_1 \GETS \aLoc }{\aReg_1 =42}
% \]
% and so:
% \[
% \hoare{\TRUE}{\aReg_3 \GETS \aLoc \SEMI \IF{\aReg_3 \neq 42} \THEN \PW{\aLoc}{42}  \FI \SEMI \aReg_1 \GETS \aLoc  \SEMI \bLoc \GETS \aReg_1  }{\bLoc =42}
% \]
% validating the transformation of the first thread to:
% \[ \PW{\bLoc}{42} \SEMI \aReg_3 \GETS \aLoc \SEMI \IF{\aReg_3 \neq 42} \THEN \PW{\aLoc}{42}  \FI \SEMI \aReg_1 \GETS \aLoc \SEMI \]
% from which the required execution follows.

% Our approach has two key ingredients.

% First, we focus on \emph{multi-copy atomicity} (\mca), which holds that when
% a write becomes visible to one thread it must become visible to
% all\nofootnote{\mca\ is traditionally explored in hardware memory models.
%   \tso\ (see, e.g.~\cite{DBLP:journals/cacm/SewellSONM10}) and recent
%   architectures, such as \armeight\ (see,
%   e.g.~\cite{DBLP:journals/pacmpl/PulteFDFSS18}), are \mca, but not older
%   architectures, such as \ppc\ (see, e.g.~\cite{DBLP:conf/pldi/SarkarSAMW11})
%   or \armseven\ (see, e.g.~\cite{DBLP:conf/popl/AlglaveFIMSSN09}).}
% \citep{DBLP:journals/pacmpl/PulteFDFSS18}.  As envisioned in
% \cite[\textsection3.3]{AlglaveThesis}, this allows us to capture cross-thread
% dependencies in a single partial order.  The key insight is that \emph{\mca{}
%   permits a single, global notion of time, manifest in the pomset order}.
% This is a dramatic simplification over other models.
% % acyclicity of the pomset providing a global notion of time.  Just as \mca{}
% % dramatically simplifies the programmer model for
% % hardware, % \citep{DBLP:journals/pacmpl/PulteFDFSS18},
% % this global notion of time dramatically simplifies our model for the language
% % level.
% % As far as we are aware, ours is the first language-level model to capture \mca.   The appeal of \mca{} in hardware is the dramatically simpler programmer model \citep{DBLP:journals/pacmpl/PulteFDFSS18}.  We believe it has the same appeal at the language-level.

% Second, we weaken the program-order within a thread to capture only
% \emph{essential dependencies}.  These are represented in the pomset order.
% This reduction from program order to pomset order is similar to the reduction
% to \emph{preserved} program order (\textsf{ppo}) in hardware models.  However, rather
% than calculating dependencies syntactically, we compute them using
% %Previous language models have used syntactic notions of dependency \cite{Batty:2011:MCC:1926385.1926394}.
% classical Hoare logic. The key insight is that \emph{logic is better than syntax}.
% Consider the following program fragments: %\begingroup \allowdisplaybreaks
% \begin{align*}
%   & \aCmd_1: \PW{\aLoc}{1} \SEMI \PW{\bLoc}{1}
%   \\[-1ex] & \aCmd_2: \PR{\aLoc}{\aReg} \SEMI \IF{\aReg} \THEN\ \PW{\bLoc}{1} \ELSE \PW{\bLoc}{1}  \FI
%   \\[-1ex] & \aCmd_3: \PW{\aLoc}{1} \SEMI \PR{\aLoc}{\aReg} \SEMI \IF{\aReg} \THEN \PW{\bLoc}{1} \FI
%   %\\[-1ex] &\aCmd_4:  \PW{\bLoc}{1} \SEMI \PW{\aLoc}{1}
% \end{align*}
% %\endgroup
% All these fragments satisfy  $\hoare{\TRUE}{\aCmd_i}{\bLoc =1}$; thus, in each case, the write of $y$ is independent of
% any code that precedes it in program order.  This allows a compiler or processor to reorder the write with respect to the code that precedes it\nofootnote{IN RELATED Thus, the model fully reaps the benefits of viewing a memory model in terms of (sequential) program transformations, eg. see~\citep{Saraswat:2007:TMM:1229428.1229469,DBLP:conf/fm/LahavV16,
% DBLP:conf/popl/DemangeLZJPV13,DBLP:conf/esop/FerreiraFS10}, without explicitly being formalized as such.}.

In \textsection\ref{sec:model} and \textsection\ref{sec:refine}, we define the model.  We show that the model:
\begin{itemize}
\item validates expected litmus cases and compiler optimizations
  (\textsection\ref{sec:props}-\ref{sec:refine}).

\item captures all C11 concurrency features  %, including relaxed, release-acquire and SC atomics, fences, and RMW
  (\textsection\ref{sec:variants}),

\item allows compositional reasoning for temporal safety, disallowing \oota{} behavior %properties
  (\textsection\ref{sec:logic}),

\item is implementable on \armeight/\tso\ {\em without} extra synchronization for
  relaxed access\nofootnote{Compilation to \armseven\ or \ppc\
    requires extra synchronization.} (\textsection\ref{sec:arm}), and

\item  satisfies the \emph{local} \drfsc\ criterion \cite{Dolan:2018:BDR:3192366.3192421} (\textsection\ref{sec:sc}).

\end{itemize}
We conclude by discussing relating work (\textsection\ref{sec:related}) and limitations (\textsection\ref{sec:limits}).

\citet{Batty17} observed that ``the current crop of relaxed memory models can
only be used to calculate the behavior of a whole program\ldots'' and argued
that instead, we should ``consider a program as an aggregate of components
over different models, composed together.''  Our work is inspired by this
call for \emph{compositionality} in models of relaxed concurrency.

Our model is compositional in the normal sense of a denotational semantics:
for example, the denotation $\sem{C_1\!{\PAR}\! C_2}$ is computed from
$\sem{C_1}$ and $\sem{C_2}$.  The model obeys laws such as scope
extrusion---$\sem{\aCmd\!{\PAR}\! \VAR\aLoc\SEMI\bCmd} =
\sem{\VAR\aLoc\SEMI(\aCmd\!{\PAR}\!\bCmd)}$ when $\aLoc\not\in\free(\aCmd)$---and
case analysis---$\sem{\aCmd} = \sem{\IF{\aExp}\THEN\aCmd\ELSE\aCmd\FI}$.
This kind of algebraic reasoning is not supported by current models.

Our model also supports compositional reasoning about \emph{data races}
(\textsection\ref{sec:sc}), isolating races in space and time.  Spatial
separation ensures that a race on one location does not invalidate \drfsc{}
at other locations.  Temporal separation ensures that \drfsc{} can be applied
within a properly synchronized region, unaffected by races that precede or
follow.

Finally, our model supports compositional reasoning about \emph{temporal
  safety properties} (\textsection\ref{sec:logic}).  Consider that each
thread of \ref{OOTA4} satisfies the following invariant: \emph{A write of $1$
  to $y$ must be preceded by a read of $1$ from $x$, and if $1$ is written to
  $z$ then a write of $1$ to $x$ must be preceded by a read of $1$ from $y$.}
Compositionality allows us to conclude that whole program satisfies this
property.  As noted above, this reasoning \emph{fails} in models based on
promises, speculations, or commitments.
%\cite{DBLP:conf/popl/KangHLVD17,DBLP:conf/esop/JagadeesanPR10,DBLP:journals/pacmpl/ChakrabortyV19}.
% demonstrating a lack of compositionality
Compositionality for temporal safety is a \emph{verifiable} criterion for claiming that a model
rejects \oota{} executions.
% define $\notar$ to be the ability to combine thread local reasoning and
% compositionally prove temporal safety properties.

% The executions of a program are computed compositionally, by induction on the
% structure of the program.  Thus, our model falls into the style of relaxed
% memory models advocated by~\citet{Batty17}.  Combined with a single global
% order on events, compositionality helps explain how our model disallows
% \oota{} execution.  After all, compositionality states that parallel
% composition does not create unexpected
% behaviors.  % The fact that the dependency relation is a partial order forbids
% % \oota\ behaviors.
% % The simplest \oota{} litmus test is the following variant of \eqref{OOTA?}
% % and \ref{OOTA1}---without $z$ or
% % the conditional.  Again all memory locations are initialized to $0$:
% % % asks whether either of the following threads
% % % can see non-zero values:
% % \begin{gather*}
% %   \tag{\ref{OOTA3}}
% %   %\PW{x}{0}\SEMI \PW{y}{0}\SEMI
% %   %(\bLoc \GETS \aLoc \PAR \aLoc \GETS \bLoc)
% %   \bLoc \GETS \aLoc \PAR \PR{y}{r}\SEMI \PW{x}{r}
% % \end{gather*}
% We rule out $z=1$ in \ref{OOTA3}--\ref{OOTA4} as follows: The write of
% $\bLoc$ on the left hand side is dependent on the read to $\aLoc$.  If the
% program writes $z$, then the write of $\aLoc$ on the right is dependent on
% the read to $\bLoc$.  Since a non-zero read can only be satisfied externally,
% both reads are part of the dependency order, leading to a cycle.  Such a
% cycle contradicts the assumption that dependency forms a partial order.  We
% formalize this argument in \textsection\ref{sec:logic}.

% In the main paper, we present the model, examples, the results concerning
% compositional reasoning  and optimization, and a discussion of
% related work.
%The details of \armeight/\tso-compilation may be found in the appendix.

% On the last point, we demonstrate a further completeness result.  Whereas the JMM aims to validate {\em all} sequential optimizations, it is clear that is impossible.  For example, the introduction of redundant reads  but not valid concurrently. Thus, $\aReg \GETS \REF{\aLoc} \SEMI \IF{\aReg != \aReg} \PW{\cLoc}{1} \FI$ cannot be replaced by $\PR{\aLoc}{\aReg} \SEMI \PR{\aLoc}{\bReg}  \SEMI
% \IF{\aReg != \bReg} \THEN \PW{\cLoc}{1} \FI$.

% Our model does the best possible under these constraints.   Call a program fragment ``linear'' if it does at most one read and at most one write on any location in any execution.  Thus, the context is unable to interfere with the atomic execution of the command; dually, neither can the atomic execution of the command interfere with the context.  We show that if sequential and synchronization free $\aCmd$ and $\bCmd$ are sequentially equivalent, and furthermore $\bCmd$ is ``'linear'' in this sense, then $\aCmd$ can be validly replaced by $\bCmd$.
% Similarly, for redundant writes; $\PW{\aLoc}{1}$ cannot be
% replaced by $\PW{\aLoc}{1} \SEMI \PW{\aLoc}{1} $ in a model
% with coherence.

%\paragraph*{Rest of the paper. }  We begin with an informal introduction to the modeling ideas in \textsection\ref{sec:model:intro}, developing the precise formalities in \textsection\ref{sec:model}.   \textsection\ref{sec:sc} proves the DRF theorem, whereas \textsection\ref{sec:arm} provides a compilation into \armeight\ and \tso.  Single threaded optimizations, and the associated completeness theorems are addressed in \textsection\ref{sec:opt}.  \textsection\ref{sec:logic} describes a temporal logic, and a compositional proof principle for proving safety properties.  \textsection\ref{sec:examples}.develops more illustrative examples.  We address related work in \textsection\ref{sec:ldrf} and conclude after.  An appendix contains details of proofs and further examples.

% We give an informal introduction to the model in \textsection\ref{sec:model:intro} before presenting the precise formalities in \textsection\ref{sec:model}.
% \textsection\ref{sec:sc} proves the DRF theorem, whereas \textsection\ref{sec:arm} provides a compilation into \armeight\ and \tso.  Single threaded optimizations, and the associated completeness theorems are addressed in \textsection\ref{sec:opt}.  \textsection\ref{sec:logic} describes a temporal logic, and a compositional proof principle for proving safety properties.
% We %present additional examples in \textsection\ref{sec:examples} and
% end with
% a discussion of related work in \textsection\ref{sec:ldrf}.
% An appendix
% contains details of proofs and further examples.

\endinput

To the reader interested in models that forbid load buffering, we provide a way to adapt our model to forbid the relaxing of the program order from reads to writes, thus modeling~\cite{Dolan:2018:BDR:3192366.3192421,BoehmOOTA}.  Our new contributions for such a reader are an approach to validating data-sensitive compiler optimizations and compositional reasoning of temporal properties.

We illustrate the last criterion with two examples.
First, consider the well-known ``Out Of Thin Air'' (\oota) litmus test, with all variables initialized to $0$:
\begin{equation}
  %\PW{x}{0}\SEMI \PW{y}{0}\SEMI
  (\PW{y}{x} \PAR \PW{x}{y})
\end{equation}
Informally, both threads satisfy the invariant that conjoins ``A write of 1  to x  requires a prior read  of 1 from y'' and ``A write of 1  to y  requires a prior read  of 1 from x ''.  If composition holds, the full program satisfies the invariant.  Since the variable declaration closes the program from other writes to $x,y$, we  deduce the conjunction of  ``A write of 1  to x  requires a prior write  of 1 to x'' and ``A write of 1  to y  requires a prior write  of 1 from x'' . Thus, we deduce that ``A write of 1  to x  requires a prior write  of 1 to x'', and consequently ``there is no write of 1 to x''.

provides an  {\em objectively  falsifiable} measurement of \oota\ in a memory model.


\chapter{Analysis}

In this section, we analyse \zclaim from different perspectives.
In \cref{sec:sec_analysis}, we show that the protocol fulfills certain properties under which we consider it to be secure.

In \cref{sec:attack_vs}, we present several common attack vectors on blockchain protocols and discuss their (in-)feasibility on \zclaim, providing thresh\-old parameters guaranteeing the security of the system where appropriate.

Finally, in \cref{sec:privacy}, we analyse privacy concerns related to the vault intermediary system.

\section{Security analysis}
\label{sec:sec_analysis}

In this section, we present a set of properties under which \zclaim is considered secure.
We specify the conditions that need to hold in order to attain each of these properties.
Finally, we show that \zclaim satisfies these conditions.

We presuppose the proper choice of security parameters $k^I$, $k^B$, $\Delta^I$ and $\Delta^B$ as discussed in \cref{sec:setup} and the security of the Zcash/Zerocash protocol, which has been extensively shown in~\cite{sasson2014zerocash}.
We also assume that the relay system is secure and that if a block header is accepted by the relay system, the possibility of a future chain reorganisation is vanishingly low.

\subsection{Notation}

We denote any participant in the protocol by $P \in \mathcal{P}$, who may be a vault $V \in \mathcal{V}$ or a user $U \in \mathcal{U}$ taking on the role of issuer or redeemer.
The amount of currency X of any of the backing currency Zcash (ZEC), the issuing chain's native currency (\emph{ICN}) and the issued currency (\emph{ICZ}) that a participant can spend is denoted by $X(P)$, e.g.\ $ICN(U)$, to which we may refer to as $U$'s ICN \emph{balance}.
Specifically, $ZEC(P)$ is the sum of the value of all unspent Zcash notes of which $P$ has knowledge containing $\dpa_P$, the diversified payment address associated with $P$.
Same goes for ICZ.
Funds locked as collateral by participant $P$ are denoted by $X^{col}(P)$.

In addition to the aforementioned balances, $ZEC^{obl}(V)$ are the \emph{ZEC obligations} associated with vault $V$ and is defined as the sum of the ZEC value of all (hidden) amounts of ICZ issued or redeemed in \mint and \burn transactions facilitated by $V$.
This also represents the amount that vaults can release to redeemers and does not reflect their ZEC balance, since $ZEC(V)$ includes the fees accumulated by $V$ in past transactions while $ZEC^{obl}(V)$ does not.
Furthermore, it is not possible to verify that $ZEC(V)$ matches $ZEC^{obl}(V)$ in any way as $V$ may perform arbitrary shielded transactions at any time.

Upon submission of a valid POB (or similarly for POCs), in which a vault proves in zero knowledge that $c \geq ZEC^{obl}(V) \cdot \xr \cdot \sstd$ where $c \leq ICN^{col}(V)$, $V$ is said to have some \emph{blocked collateral} $ICN^{bcol}(V) = ZEC^{obl}(V) \cdot \xr \cdot \sstd$ and some free collateral $ICN^{fcol}(V) = c - ICN^{bcol}(V)$.
The vault's blocked collateral are the funds backing their ZEC obligations, which they can only decrease by releasing funds to users (or by burning ICZ, see discussion on rebalancing in \cref{sec:rebalancing_exit}).

$X_t(P)$ denotes P's balance at time $t$, where one step in time corresponds to a transaction on I.
$ZEC_t(P)$ is not defined.

Finally, $\xr_t$ denotes the latest ZEC to ICN exchange rate as provided by the exchange rate oracle \oxr at time $t$, where 1 ZEC = $\xr_t$ ICN.
We assume a 1:1 peg between the issued currency and the backing currency, i.e.\ 1 ZEC = 1 ICZ.

\subsection{Goals}
\label{sec:security_goals}

We deem \zclaim \emph{secure} if it achieves the following properties:
\begin{itemize}
    \item \textbf{Soundness} The amount of issued currency in circulation is equal to the amount of ZEC obligations, i.e.\ for all $t \geq 0$
    \begin{equation}\label{eq:icz_equals_obl}
        \sum_{U \in \mathcal{U}} ICZ_t(U) = \sum_{V \in \mathcal{V}} ZEC^{obl}_t(V)
    \end{equation}
    and $ZEC^{obl}(V)$ is derived from protocol transactions to and from $V$ on Zcash.
    Specifically,
    \begin{lemma}\label{le:lock_before_mint}
        For every Mint transfer of (hidden) value $\val_{IM}$ on I containing $\dpa_V$ as the vault's diversified payment address, there is an Output transfer on Zcash creating a note of value $\val_Z$ to $\dpa_V$ s.t.
        \begin{equation}
            \left \lfloor{\val_Z \cdot (1 - f)}\right \rfloor = \val_{IM}
        \end{equation}
    \end{lemma}
    and
    \begin{lemma}\label{le:release_before_burn}
        For every Burn transfer of value $\val_{IB}$ on I containing $\dpa_V$ as the vault's diversified payment address and $\cm_Z$ as the requested release note commitment, there is an Output transfer on Zcash creating a note of value $\val_Z$ with note commitment $\cm_Z$ s.t.
        \begin{equation}
            \val_Z = \left \lfloor{\val_{IB} \cdot (1 - f)}\right \rfloor
        \end{equation}
    \end{lemma}
    according to the fee policy defined in \cref{sec:fees}.

    These are the transfers that make up a vault's ZEC obligations, and evidently the sum of all Mint transfers minus all Burn transfers is equal to the circulating supply.
    Hence \cref{eq:icz_equals_obl} is in fact satisfied by the definition of $ZEC^{obl}(V)$.

    \item \textbf{Coverage} The total amount of ZEC obligations are backed by a proportional amount of the issuing chain's native currency according to the prevailing exchange rate, i.e.\ for all $t \geq 0$
    \begin{equation}\label{eq:coverage}
        \left(\sum_{V \in \mathcal{V}} ZEC^{obl}_t(V)\right) \cdot \xr_t \cdot \smin \leq \sum_{V \in \mathcal{V}} ICN^{bcol}_t(V)
    \end{equation}
    where $\smin$ is the minimum collateralisation ratio as defined in \cref{sec:constants}.

    \item \textbf{Fairness} An honest participant following best practices will not incur any loss of funds as long as they can receive and broadcast transactions from and to chains Z and I.

    For the condition on this claim, availability of both chains must be guaranteed, which we derive from the security assumption of \xclaim that transactions broadcast by users are received by (honest) consensus participants within a known maximum delay $\Delta_{tx}$.
    In case of network failure for a significant amount of time on the user/vault side, they may in effect incur loss of funds.
    It is thus the participants' responsibility to ensure they remain online throughout the duration of a subprotocol.

    As for the claim itself, we examine the cases where a participant may incur loss of funds.
    This risk exists both in the issue and redeem subprotocols.
    We concern ourselves with transactions that incur a change in any of the balances of the two parties involved (except for slashing, which we cover separately) i.e.\ in which a monetary transaction takes place.
    These are, on the one hand, \lock and \release transactions, and on the other, \mint and \burn transactions.

    Concerning the issue subprotocol, the following statements must be proven:
    \begin{lemma}\label{le:mint_after_lock}
        After executing a lock operation, a user is able to mint the locked amount minus fees of ICZ.
    \end{lemma}
    \begin{lemma}\label{le:lock_before_mint2}
        If a \mint transaction involving a vault $V$ is confirmed on I (thus increasing their ZEC obligations), $V$ has received the amount being minted plus fees in a previous \lock transaction.
    \end{lemma}
    Note that \cref{le:lock_before_mint2} is equivalent to \cref{le:lock_before_mint} with the added requirement that $V$ must have knowledge of the note values.

    Analogously, for redeeming we must show that:
    \begin{lemma}\label{le:burn_after_release}
        After executing a release operation, a vault is able to trigger the inclusion of an associated pending \burn transaction decreasing their ZEC obligations by the released amount plus fees.
    \end{lemma}
    \begin{lemma}\label{le:release_before_burn2}
        If a \burn transaction involving a user $U$ is confirmed on I, $U$ has received the amount being burned minus fees in ZEC in a previous \release transaction.
    \end{lemma}
    Again, \cref{le:release_before_burn2} is tantamount to \cref{le:release_before_burn} with addition of the knowledge requirement.

    Consideration must additionally be paid to the slashing mechanism, in order to ensure that slashing of an honest party cannot be instigated by a malicious one.

    Finally, we provide a set of practices that vaults may follow in order to prevent liquidation and hence loss of funds.
\end{itemize}

\subsection{Argumentation}

We show that \cref{le:lock_before_mint,le:release_before_burn} are always satisfied and, furthermore, that the recipients of the Zcash notes referenced therein have knowledge of the note values, hence satisfying \cref{le:lock_before_mint2,le:release_before_burn2}.
Further, we cover \cref{le:mint_after_lock,le:burn_after_release}, and show the validity of \cref{eq:coverage}.
Finally we discuss slashing and liquidation.

\subsubsection{Issuing}

The zk-SNARK $\pim$ in Mint transfers as defined in \cref{sec:mint} guarantees that \cref{le:lock_before_mint} holds through the \textbf{Note commitment integrity} condition together with the Merkle path validity success requirement (there exists such a note on Zcash), and the \textbf{Locked value}, \textbf{Minted value} and \textbf{Fee rounding commitment integrity} conditions (the note has the specified value).

Mint transfers also satisfy \cref{le:mint_after_lock} through \textbf{Trapdoor commitment integrity}, which guarantees that only the user that authored the \lock transaction can create a \mint transaction.

Furthermore, the challenge mechanism guarantees that the vault has knowledge of the note values, satisfying \cref{le:lock_before_mint2}, since \mint transactions will only be included on I if they are not successfully challenged.
The vault can challenge the pending transaction through disclosure of the shared secret, proven to be correct in $\pic$.
If the note values have not been correctly encrypted to the vault, the transaction will be discarded, which ensures \textbf{Fairness}.

So far we have proven \cref{le:lock_before_mint,le:mint_after_lock,le:lock_before_mint2}.

\subsubsection{Redeeming}

Equally, the conjunction of $\pib$ as defined in \cref{sec:burn} and the \confirmRedeem transaction guarantees that \cref{le:release_before_burn} holds through the conditions in \pib and the Merkle path validity success requirement of \confirmRedeem transactions.

\Cref{le:burn_after_release} is rather trivial as all it takes is to provide a Merkle path showing the existence of the note commitment in the note commitment tree, which not only the vault but any participant can do.

As for redeeming, the challenge mechanism works somewhat differently: it is the recipient of the note (the redeemer) who constructs it, and the sender must have knowledge of the values in order to be able to create it.
Thus \cref{le:release_before_burn2} is satisfied by construction, and we can move on to the challenge mechanism: in this case, it serves to ensure the vault has received the correct note values.

A vault may successfully challenge the transaction if and only if the provided note commitment cannot be generated from the decrypted values, thus it cannot punish honest redeemers but also a redeemer cannot ask a vault to release a note it cannot create, causing it to be slashed.
Hence \textbf{Fairness} is ensured when redeeming too.

Now, we have also proven \cref{le:release_before_burn,le:burn_after_release,le:release_before_burn2}, in addition to \cref{le:lock_before_mint,le:mint_after_lock,le:lock_before_mint2} from before, which concludes the proof for \textbf{Soundness}.
It remains to prove \textbf{Coverage} and to discuss loss of funds in liquidations and challenge operations.

\subsubsection{Proofs of balance}

From \cref{eq:coverage} it is evident that if we can show that
\[
    ZEC^{obl}_t(V) \cdot \xr_t \cdot \smin \leq ICN^{bcol}_t(V)
\]
for any vault, then~\eqref{eq:coverage} holds for the entire system.

We recall here that $ICN^{bcol}(V) = ZEC^{obl}(V) \cdot \xr \cdot \sstd$, which nevertheless only holds for some $\xr$ used in the POB.
We shall denote this exchange rate by $\xr_V$.

Technically, it also only holds for $ZEC^{obl}(V)$ at the time of its last POB, though the only problematic case is that in which a vault's ZEC obligations increase since that time, i.e.\ they issue funds.

However, vaults may only become available to issue by submitting a POC (and must provide POCs instead of POBs until issuing is completed), in which they prove collateralisation for their ZEC obligations plus the maximum amount they can issue in one round.
Thus we may assume without loss of generality that $ZEC^{obl}(V)$ stays constant.

For any $\xr_t > \xr_V$ then it must hold that
\[
    ZEC^{obl}(V) \cdot \xr_t \cdot \smin \leq ZEC^{obl}(V) \cdot \xr_V \cdot \sstd
\]
\[
    \xr_t \cdot \smin \leq \xr_V \cdot \sstd
\]
which is the liquidation trigger as defined in \cref{sec:liquidation}.

Liquidation reduces $V$'s ZEC obligations until they meet \sstd again as described in the aforementioned section.

Thus we see that~\eqref{eq:coverage} does indeed always hold and is enforced by reducing the circulating supply if needed through liquidation auctions.
This concludes the proof for the \textbf{Coverage} property.

\subsubsection{Challenging}

Challenge operations result in the challenged party's warranty collateral $ICN_w$ being transferred to the challenging party if successful (for vaults, this amount is deducted from their staked collateral).
However, by definition these are only successful if the note plaintext in the challenged transaction has not been correctly encrypted to the challenger.
Thus it is not possible to cause loss of funds through these operations to an honest participant that adheres to the protocol.

\subsubsection{Liquidation}
\label{sec:analysis_liquidation}

In the event of liquidation, vaults are effectively slashed the liquidation penalty on the funds sold through the liquidation auction.
The amount of funds sold depends on the collateralisation constants and is always a fixed fraction of the vault's total collateral.

As mentioned in \cref{sec:liquidation}, with the collateralisation constants as suggested, liquidation is triggered on a vault if the exchange rate increases by 33\% with respect to $\xr_{V}$, the last exchange rate they used in a proof of balance.

In order to avoid liquidation, $V$ needs to provide a POB using a higher $\xr$ before the aforementioned threshold is reached.
If at any time $t$ where $T_{p} < t < T_{liq}$ and $\xr_{V} < \xr_t < \xr_{T_{liq}}$, it holds that
\begin{equation}\label{eq:pob_requirement}
    ZEC^{obl}_t(V) \cdot \xr_t \cdot \sstd \leq ICN^{col}_t(V)
\end{equation}
the vault can submit a new POB using $\xr_t$, hence avoiding liquidation at $T_{liq}$.
Naturally, this holds for later steps in time: if the exchange rate further increases, the vault will need to continue providing POBs in order to protect themselves from liquidation.
Vaults should choose a sensible $\xr_t > \xr_{V}$ for which to submit a new POB before they approach the liquidation threshold.

However, if \cref{eq:pob_requirement} is no longer satisfied, i.e.\ the change in the exchange rate at time $t$ has caused $V$'s collateralisation rate to fall below $\sstd$, $V$ first needs to top up their collateral or rebalance in order to be able to submit a POB using $\xr_t$.

If they hold ICN such that $ZEC^{obl}_t(V) \cdot \xr_t \cdot \sstd \leq ICN^{col}_t(V) + ICN_t(V)$, this is straightforward (barring the case where the exchange rate further increases such that \cref{eq:pob_requirement} again does not hold after collateral has been topped up) and is accomplished in the time it takes the vault to construct the two transactions and forward them to the network, which is negligible.

However, if this is not the case, $V$ may first have to acquire some $ICN$ to top up their collateral such that they can again submit a POB.
How long it takes them to do so depends on many external factors, but it may plausibly be up to several days: they may first have to procure liquid assets, transfer them to an exchange, trade them for ICN, withdraw the ICN from the exchange and finally lock them as collateral.
In this case, it may be fair to say that all is lost and vaults should follow best practices in order to avoid this sort of situation.

Another option vaults have to satisfy \cref{eq:pob_requirement} is to rebalance, but this is a strictly longer process than topping up their collateral since it also requires ICN and one further transaction on $I$, but also exchanging ICN for ICZ.
Thus if the goal is for them to protect themselves from liquidation as fast as possible, they should always choose to top up.
In addition to this, vaults may choose to start rebalancing such that they do not need to top up again if the exchange rate further increases or that they may unlock some collateral again.

Best practices vaults can follow in order to protect themselves from liquidation may thus include:
\begin{itemize}
    \item Maintaining a collateralisation rate somewhat higher than $\sstd$, i.e.\ overcollateralising, by strategically advertising themselves only to issue or redeem, rebalancing or increasing their collateral when necessary.
    For example, $\sigma_{sft} = 2$.
    \item Holding some amount of ICN not locked as collateral that they may use to top up their collateral or rebalance, or ensuring that they may acquire such amount quickly in case of emergency.
    \item Submitting POBs to address changes in the exchange rate above a certain threshold $r_{\xr} = \frac{\xr_t}{\xr_V}$.
    This should be a value close to $1.0$, e.g.\ $r_{\xr} \leq 1.05$.
\end{itemize}

Vaults may modify these parameters according to their risk tolerance, but with the suggested values we observe that for liquidation to be triggered it would be necessary that $\frac{\xr_t}{\xr_V} > \frac{\sigma_{sft}}{\smin} - r_{\xr} \approx 1.62$ i.e.\ the exchange rate increases by 62\% before $V$ is able to acquire new funds or rebalance.

Such a dramatic increase is unlikely to happen in a short time window, but in the context of cryptocurrencies, where high volatility is the norm rather than the exception~\cite{Lahmiri2018LongrangeMDvolatility,CAPORALE2019143volatility}, it is impossible to set a definite threshold for the increase in the exchange rate above which the likelihood of it taking place is insignificant.
Ultimately, it is left to vaults to assess the risks themselves.

Finally, liquidation may also be triggered if a vault fails to comply with issue or redeem requests too often, which should however not happen under the availability assumptions of participants under which \textbf{Fairness} applies.

Thus if we make the assumption that \xr will not increase fast enough to trigger liquidation on vaults following best practices, this concludes the argumentation in favour of \textbf{Fairness}.

\section{Attack vectors and points of failure}
\label{sec:attack_vs}

We discuss here a range of attacks and points of failure in \zclaim and offer mitigation strategies.
Familiarity with the security analysis of \xclaim~\cite[Section~VII]{zamyatin2019xclaim} is assumed.
Where not specified otherwise, the discussion on specific vulnerabilities offered there also holds for \zclaim.

\subsection{Inference attacks}
\label{sec:inference_attacks}

As discussed in \cref{sec:privacy}, vaults may guess the users' identity through the amounts in \lock and \release transactions in which they are involved.
The knowledge of this amount is of course per se insufficient to this end, but it can lay the ground for an inference attack leading to de-anonymisation if combined with other information they may have access to.

For instance, if an easily identifiable amount a user locks with a vault matches a recent transparent-to-shielded transaction on Zcash, the vault may deduce that there is a high likelihood those two transactions were performed by the same user.
This is similar to the heuristic used by \textcite{quesnelle2017linkability} to identify what he calls \textcquote[1]{quesnelle2017linkability}{round-trip transactions, where the same, or nearly the same number of coins are sent from a transparent address, to a shielded address, and back again to a transparent address. [He argues that] such behavior exhibits high linkability, especially when they occur nearby temporally}.
If the vault is correct in their assumption, they may be able to infer the user's identity from previous and future activity associated with the disclosed transparent address.

Another possibility is the case where a vault has privileged access to other information on one or many users, such as through data from a cryptocurrency exchange.
This may render them capable of effortlessly matching the real-world identities of exchange users with specific \lock or \release transactions if, for example, the guessed/observed amount matches a recent withdrawal from the exchange.

In both of these scenarios, an attacker may leverage this information to attempt extortion or defamation of the involved parties.

The splitting strategy defined in \cref{sec:splitting_strategy} aims to prevent this sort of attacks.
\zclaim's privacy has been analysed in this context in \cref{sec:privacy}.

\subsection{Chain relay poisoning}
\label{sec:relay_poisoning}

Chain relay poisoning involves an adversary triggering a chain reorganisation, or \emph{reorg} for short, of $N \geq k^B$ such that a previously accepted transaction at depth $N$ is invalidated.
This is known as an $N$-confirmation double spend and yields similarities to selfish mining~\cite{eyal2014majority} in that it involves misleading honest nodes through an amassing of computational power.

We recall here that the security parameter $k^B$ is based on the assumption that an adversary's computational power is bounded by $\alpha \leq 33\%$ and denotes the depth at which the likelihood of an adversary triggering a reorg is negligible.

However, in \xclaim a poisoning attack may be successful well below this threshold $\alpha$ if the relay system is deprived of recent block header data~\cite[Section VII-A]{zamyatin2019xclaim}.

We expand here on this observation.
This is in fact a common vulnerability to cross-chain interoperability schemes and is not only the case if the relay system is devoid of real-world data~\cite{2017hijackingbtc}, but may involve more intricate attacks in which \emph{relayers} are isolated from the rest of their peers in the network and misled to accept the attacker's chain as the longest.
This is more commonly known as an eclipse attack~\cite{heilman2015eclipse}.

Such attacks along with mitigation strategies have been discussed previously in the literature~\cite{heilman2015eclipse,wust2016ethereum,xu2020eclipsed,alangot2020decentralized}, and we refer the reader in particular to analyses on Bitcoin and Bitcoin-based blockchains (such as Zcash is) for mitigation strategies~\cite{heilman2015eclipse,alangot2020decentralized}.

\subsection{Exchange rate poisoning}
\label{sec:er_poisoning}

Similarly, if \oxr is manipulated to provide an erroneous price feed, this may allow a prepared adversary to steal funds in several ways.

For instance, an exchange rate much higher than the actual value would allow vaults to issue ZEC or unlock collateral such that they become undercollateralised when the exchange rate returns to normal.
An economically rational vault would have no incentive to bring back their collateralisation rate above the safety value and may choose to undergo liquidation and retain the ZEC for which they hold obligations, effectively violating \textbf{Soundness} and jeopardising the stability of the protocol.

On the other hand, an artificially low exchange rate may trigger mass liquidation and allow users to buy vaults' collateral at an unfair price, hence stealing funds from them.

It is thus of paramount importance to guarantee the reliability of the exchange rate oracle.
Blockchain oracles aim to solve this exact problem~\cite{peterson2015augur,8726819astraea,ellis2017chainlink}, aggregating exchange rates from different sources, leveraging economic incentives to reinforce the veracity of these sources and providing dispute mechanisms in case of discrepancies.
These systems are, without question, safer than relying on a single source to provide an exchange rate, though they may still fail under certain circumstances~\cite{lo2020reliabilityoracles}.

Further measures can be taken to prevent such an attack, such as sanitising data provided by the oracle or employing so-called circuit breakers~\cite{2019circuitbreakers}, which involve halting operations in case of unusual price movements.

Lastly, it must be noted that relying on a decentralised oracle constitutes a further cross-chain integration and as such opens another door to poisoning attacks as discussed in the previous section, which can be addressed through the same countermeasures.

\subsection{Replay attacks on inclusion proofs}

\zclaim prevents replay attacks on \lock and \release transaction inclusion proofs, in which a user reuses a past \lock transaction to mint ICZ or a vault reuses a \release transaction to decrease their ZEC obligations, stealing funds from the other party, as follows.

The nonce $\nlock$ provided by the issuing chain in lock permits must be used to generate the note commitment trapdoor in \lock transactions as specified in \cref{sec:lock}, which is enforced in the zero knowledge proof in Mint transfers $\pim$ through the \textbf{Trapdoor commitment integrity} condition.
This ensures that every \lock transaction is uniquely associated with the corresponding Issue procedure.

As for \release transactions, protection from replay attacks is implicit since the note commitment is generated in advance by the redeemer.
Therefore a vault can only replay a release inclusion proof if the redeemer has chosen the same note values, most notably the same note commitment trapdoor \rcm as in a previous Burn transfer.

However, the likelihood of the same trapdoor being sampled twice randomly from $\ncm.\gent$ as specified in \cref{sec:release} is insignificant.
If a redeemer purposely reuses the same note values, they only derive negative utility from their actions.

\subsection{Counterfeiting}
\label{sec:counterfeiting}

In \xclaim, counterfeiting is defined as the issuing of issued currency that is not backed by funds locked with vaults.
This is enforced through the principle of \textbf{Auditability}.
In a nutshell, since the vaults' actions are observable by anyone, a vault removing funds from the pool of funds locked with them can be reported and will be punished by slashing their collateral as a consequence.
This also includes vaults reusing these funds to mint more issued currency.

It is here that a central difference between \zclaim and \xclaim arises: we do not make such restriction.
Instead, \textbf{Coverage} and \textbf{Soundness} guarantee that the total amount of issued currency in circulation is always backed by an equivalent amount of the vaults' collateral.
A vault may very well reuse ZEC locked with them to issue more ICZ; they will still be unable to unlock their collateral until they have released ZEC to redeemers or burnt ICZ themselves.

Counterfeiting would hence imply issuing ICZ which is not locked by ICN, which is impossible as per the aforementioned properties.

\todo{%\subsection{Composability attacks}
Would like to get this in here}
%~\cite{zindros_2019summa,zamyatin2019sok}
%higher potential gain extracted from attack
%together w/ relay  poisoning attacks
%k chosen with respect to financial cost of carrying out an attack larger than benefit?
%consider case where \zclaim is instantiated on two different chains

\subsection{Extortion}
Extortion by vaults, which would involve vaults setting extreme fees to redeem hence making it unfeasible, is prevented in \zclaim under the suggested fee policy, which defines a fixed fee for both the issue and redeem procedures.

\subsection{Black swan events}
\label{sec:black_swan}
A black swan event is an extremely rare event with potentially catastrophic consequences for parties exposed to a previously unknown or neglected risk.

In a financial setting, this term is usually employed to describe a severe market-wide crash or extreme, sudden devaluation of an asset due to unforeseen circumstances.

Such events are a rather common occurrence in the cryptocurrency space~\cite{fry2016negative,sophonex_2019blackswan}, perhaps challenging their definition.
This is due to several factors, foremost the speculative nature of cryptocurrencies~\cite{cheah2015speculative} and the prevailing high volatility in their valuation.
Furthermore, even though advances in blockchain technology are made at a rapid pace, the technology is still in its early stages and as such is susceptible to attacks and exploits of varying nature.
Finally, the regulatory framework surrounding cryptocurrencies is still being developed in many countries~\cite{loc2018regulation} and is a hot topic of debate~\cite{yeung2019regulation}, as it may facilitate the widespread adoption of certain types of cryptocurrencies while hindering the development of others.

This is to say that the chance of sudden, extreme devaluation of either one of Zcash or the issuing currency is non-negligible and must be kept in mind.
In case of a drop in the valuation of Zcash, \zclaim would continue to operate normally and it is quite clear what the consequences would be: any participant holding either ZEC or ICZ will suffer a loss unrelated to the protocol.

If, on the other hand, the issuing currency suffered this fate, the consequences would be similar to those discussed in \cref{sec:er_poisoning} and \zclaim would eventually no longer meet the security properties defined in \cref{sec:security_goals}.
\textcite{zamyatin2019xclaim} hence assume a minimum exchange rate $\xr_{min}$ \textcquote[3]{zamyatin2019xclaim}{below which adhering to protocol rules no longer represents the equilibrium strategy of rational adversaries}, and a delay $\Delta_{\xr_{min}} < \Delta^I$ such that honest participants can include a transaction on I before this threshold is reached.
However, it remains unclear what strategy participants may follow in order to avoid financial loss in this situation.

\section{Privacy against vaults}
\label{sec:privacy}

We argue that the monetary values of transactions in which vaults are involved do not leak the total value prior to splitting.
We also consider the information they may learn by observing network traffic and the case where several vaults collude with each other or are operated by the same adversary.

The consequences of such information leakage are discussed in \cref{sec:inference_attacks}.

\subsection{Adversarial model}

We assume an adversary $\mathcal{A}$ constrained by the following assumptions:
\begin{itemize}
    \item $\mathcal{A}$ has perfect knowledge of the \zclaim protocol, i.e.\ also of the splitting strategy.
    \item $\mathcal{A}$ can observe transactions happening on both Zcash and $I$, but cannot filter network packets sent by individual users.
    \item $\mathcal{A}$ controls no more than 1/3 of all vaults that are available to issue or redeem, independently of each other, at any given time.
\end{itemize}

\subsection{Obfuscation of total transacted value}

Using the splitting strategy defined in \cref{sec:splitting_strategy}, knowledge of one or even several amounts reveals no information about the total \vtot other than the evident fact that it is larger than the total observed value.
Indeed, the individual amounts sent to vaults are independent of the total: the latter only determines which amounts out of the fixed set of possible values get sent.
Hence, as long as $\mathcal{A}$ controls only a fraction of all vaults, the unknown amounts are equally distributed among all possible unobserved values.

However, if $\mathcal{A}$ learns the number of transactions $k$, this is no longer the case.
We have as per the splitting strategy that $k \leq k_{max} = (b-1) (i_{max}-i_{min} + 1)$.
Now, if $\mathcal{A}$ not only has knowledge of $k_{obs}<k$ amounts but also of the value $k<k_{max}$, the odds of them deducing \vtot increase from $(2^{k_{max}-k_{obs}})^{-1}$ to $\left(\binom{k_{max}-k_{obs}}{k-k_{obs}}\right)^{-1}$, which is strictly larger and grows as $k$ approaches either $k_{max}$ or $k_{obs}$.

One approach to counter this issue is to split \vtot into a small, random number of amounts and then apply the splitting strategy for each of these individually, such that $\mathcal{A}$ has no knowledge of $k_{max}$ (across all amounts).

\subsection{Transaction linkability}
\label{sec:linkability}

As we have seen, if $\mathcal{A}$ learns the number of transactions $k$ into which \vtot has been split, they may gain a considerable advantage in their attempts to learn \vtot.

We explore the circumstances under which $k$ could be leaked.

The first and most evident risk is low network traffic, as it may lead to the $k$ \lock or \burn transactions being submitted within a shorter delay with respect to other than other transactions on the network.
We note that if there is enough traffic across \zclaim, this issue does not arise.

Furthermore, this risk can be mitigated by not submitting all sub-transactions simultaneously but within a short random delay between one another.

Furthermore, redeemers should use a different destination diversified payment address \dpa per \burn transaction, which may be derived from the same incoming viewing key.
Otherwise, $\mathcal{A}$ may easily link \burn transactions with each other in which vaults controlled by $\mathcal{A}$ are requested to redeem, hence allowing $\mathcal{A}$ to learn the note values.

% An adversary $\mathcal{A}$ with perfect knowledge of the protocol (i.e.\ also of our mitigation strategy) should not be able to guess the value \vtot with a certain degree of confidence.
% We assume that $\mathcal{A}$ controls an unknown number of vaults and thus may learn any number of amounts $k_{obs} \leq k$.
% \todo{Assume fraction of malicious vaults $<1/3$ and employ this assumption elsewhere in this section}

% We now have to consider what it means for $\mathcal{A}$ to guess the amount with a certain degree of confidence.
% Assuming that $\mathcal{A}$ knows the actual distribution of values sent through the system $P(v)$, we only care whether they can assign the original value a significantly higher probability than what could be guessed without evidence (i.e.\ if most values sent are around $\val_m$, and $\mathcal{A}$ guesses a value around $\val_m$, they will be right most of the time, but they still do not have any specific knowledge about the actual value).
% \todo{certain degree of confidence still not defined, significantly higher probability vague}

% \subsection{Approaches}

% \subsubsection{Uniform sampling}
% \todo[inline]{Shall I just get rid of the whole talk about distributions? Maybe remove uniform sampling section}

% Taking a Bayesian approach, all $x_i$ that $\mathcal{A}$ can ascribe to the same transaction constitute the evidence $X$.

% The question now becomes what the distribution $P(X \mid V)$ from which to sample the values should be, such that $P(V \mid X) = \frac{P(V) P(X \mid V)}{P(X)}$ does not reveal significantly more information about the value than $P(V)$.

% It is obvious that some information about the total will always be leaked through knowledge of even one of its parts; at the very least, $\mathcal{A}$ will know that the total is equal or larger to the latter.
% Thus the worst they can do is guessing that $P(V \mid X = x) = P(V \mid V \ge x)$, i.e.\ the prior distribution truncated to values greater than or equal to the observed value.

% The larger the amount that $\mathcal{A}$ has knowledge of, the narrower this distribution will be, revealing more information about the original value.
% From these observations it follows that we need to set an upper bound $x_{max}$ for each term and also try to limit the number of terms $\mathcal{A}$ may learn.
% We suppose that we can split \vtot into an arbitrary number of parts since the same vault can be sent multiple transactions.
% In practice, the number of vaults should be much larger than the number of transactions into which the total is split.
% We ignore the issue this causes with rising fees and possibly higher latency for larger amounts for the moment.

% Thus we set $x_{max}$ small enough s.t.\ $\left. P(V) \right|_{x}^{\vmax} \ge \alpha$ for all values of $x$, where $\alpha$ represents a significant fraction of all transactions that will always be above the observed value.

% \paragraph{Minimal example}

% We assume $V$ to be distributed uniformly over the set of all possible values, i.e.\ $V \sim U((0, \vmax])$, $k = 2$ and that $\mathcal{A}$ learns one of the two values.

% Now assuming $X_1$ is also sampled uniformly across all possible values s.t.\ it is split at least once, i.e.~$X_1 \sim U((0, v))$, we have that $X_2 = v - X_1 \sim U((0, v))$ too and so this is a suitable distribution.
% Here, $\mathcal{A}$ can learn that
% \[
%     P(V \mid X_i) = \frac{P(V) P(X_i \mid V)}{P(X_i)} \text{, for } i \in \{1, 2\}
% \]
% where $P(X_i \mid V) = f(x_i) =
% \begin{cases}
%     1/v & x_i < v\\
%     0 & x_i \ge v
% \end{cases}\,$
% , thus
% \[
%     P(V \mid X_i)
%     =
%     \frac{\frac{1}{\vmax} f(x_i)}{\int_{0}^{\vmax}\frac{1}{\vmax} f(x_i)dv}
%     =
%     \begin{cases}
%         \frac{1}{v \log{\frac{\vmax}{x_i}}} & v > x_i\\
%         0 & v \le x_i
%     \end{cases}
% \]

% We can see that this means the total amount is always more likely to be closer to the observed value than it is to be much larger.
% The posterior distribution also becomes flatter but narrower the larger the observed value.

% This is exactly the kind of information we would like to prevent $\mathcal{A}$ from learning.

% \paragraph{Multiple variables}

% We try to answer the following question: how can we sample $k$ values from a uniform distribution such that their sum equals \vtot?

% The distribution that we are looking for is
% \[
% P(X_i \mid \sum_{j=1}^{k} X_j = v) = \frac{P(X_i) P(\sum_{j=1}^{k} X_j = v \mid X_i)}{P(\sum_{j=1}^{k} X_j = v)}
% \]

% If we set $X_i' = \frac{X_i}{x_{max}}$ and $v' = \frac{v}{x_{max}}$,
% we can choose $X' \sim U(0, 1)$ and this becomes for the first value
% \[
%     P(X_1' = x_1' \mid \sum_{j=1}^{k} X_j' = v') = \frac{P(\sum_{j=1}^{k} X_j' = v' \mid X_1' = x_1')}{P(\sum_{j=1}^{k} X_j' = v')}
% \]
% \[
%     = \frac{P(\sum_{j=2}^{k} X_j' = v' - x_1')}{P(\sum_{j=1}^{k} X_j' = v')}
% \]

% Both numerator and denominator follow an Irwin-Hall distribution, defined as the sum of a number of independent random variables each following a uniform distribution.
% An Irwin-Hall distribution on $n$ random variables has probability density function
% $
%     f_X(x;n)=\frac{1}{2(n-1)!}\sum_{k=0}^n (-1)^k{\binom{n}{k}} (x-k)^{n-1}\text{sgn}(x-k)
% $
% thus the above distribution can be expressed as
% \[
%     (k-1)
%     \frac{
%         \sum_{j=0}^{k-1} (-1)^j {\binom{k-1}{j}} (v'-x_1'-j)^{k-2}\text{sgn}(v'-x_1'-j)
%     }{
%         \sum_{j=0}^{k} (-1)^j {\binom{k}{j}} (v'-j)^{k-1}\text{sgn}(v'-j)
%     }
% \]

% The denominator is just a normalising factor dependent on $v'$ and $k$.
% The whole term is simply an Irwin-Hall distribution with mean $v' - \frac{k-1}{2}$ truncated and normalised to values between 0 and 1.
% A plot of this distribution for $v' = 5$ and different values of $k$ can be found on \cref{fig:dists}.\todo{Proper labels, more plots, discussion on v \& k}

% \begin{figure}
% \centering
% \includegraphics[width=0.8\textwidth]{img/distributions.png}
% \caption[Probability distribution of first sampled value]{Probability distribution of first sampled value $P(X_1 \mid \sum_{j=1}^{k} X_j = 5)$ for different values of $k$}
% \label{fig:dists}
% \end{figure}

% For the $t$-th value, we obtain an Irwin-Hall distribution on $k-t$ random variables with mean $v' - \sum_{j=1}^{t-1} x_j - \frac{k-t}{2}$.
% We have
% \[
%     P(X_t' = x_t' \mid \sum_{j=1}^{k} X_j' = v', X_1'=x_1', ...,X_{t-1}'=x_{t-1})'
% \]
% \[
%     = P(X_t' = x_t' \mid \sum_{j=t+1}^{k} X_j' = v' - \sum_{j=1}^{t} x_j', X_1'=x_1', ..., X_{t-1}'=x_{t-1}')
% \]
% \[
%     = \frac{P(\sum_{j=t+1}^{k} X_j' = v' - \sum_{j=1}^{t} x_j')}{P(\sum_{j=t}^{k} X_j' = v' - \sum_{j=1}^{t-1} x_j')}
% \]

% For $t = k$, $X_t$ converges to a single value, for $t = k-1$ it follows a uniform distribution and for $t = k-2$ a triangular distribution (as before all truncated and normalised).

% \todo[inline]{\url{https://stackoverflow.com/a/8068956} looks like a much easier approach to this problem}

% Naturally, if $\mathcal{A}$ has knowledge of the number of transactions $k$, they will learn some information from the observed amount(s) since the distribution $P(V \mid X)$ is a truncated Irwin-Hall distribution as discussed above.
% \todo{have some more work done on this but unsure whether to expand or remove altogether}

%coordinate along line
%sample from n-1-dimensional space
%if value $k$ is unknown to adversary, the distribution of possible values would be $P(V=v) = %\frac{\sum_{k=m}^{n} P(\sum_{i=1}^{k} X_i = v|X)}{n-m + 1}$, where $m=\left %\lceil{\frac{v-\sum_{x_i \in X} x_i}{\vmax}}\right \rceil + |X|$ and $n=\left %\lfloor{\frac{v-\sum_{x_i \in X} x_i}{\val_{min}}}\right \rfloor + |X|$, a sum of I-H %distributions across a larger set of possible values for \vtot.
%We do not try to analyse this distribution...


% !TEX root =../thesis-letomes.tex

\chapter{Search Strategies}
\section{Evolution Strategies} \label{sec:ES}
OpenAI released a blog post \cite{Karpathy} with accompanying paper \cite{Salimans2017} in March 2017, where they announced ``Our finding continues the modern trend of achieving strong results with decades-old ideas''. Intuitively ES in a ``guess and check'' optimization process, where we start with some random parameters and then repeatedly 1) tweak the guess a bit randomly, and 2) move our guess slightly towards whatever tweaks worked better.

The team applied evolution strategies (ES) to two reinforcement learning (RL)\footnote{Reinforcement learning is an area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.} scenarios and compared the performance with typical deep RL problems:

\begin{enumerate}
    \item MuJoCo\footnote{MuJoCo stands for Multi-Joint dynamics with Contact, and is a ``\textit{physics engine aiming to facilitate research and development in robotics, biomechanics, graphics and animation, and other areas where fast and accurate simulation is needed}''. See \url{http://www.mujoco.org}} (see \cref{fig:mujoco}) trained faster with ES than A3C\footnote{Asynchronous Advantage Actor Critic (A3C), published in 2016 \cite{Mnih2016}, is a class of deep RL algorithms that is still close to state of the art among deep RL algorithms.}.
    \item Atari game playing (see \cref{fig:atari}) trained to similar performance with ES in 1 hour compared to 1 day with A3C. Atari games have also become a popular problem space for deep RP researchers
\end{enumerate}

\begin{figure}[H]
    \centering
    \subfloat[MuJoCo: Creatures resembling a dog, 4-legged spider and worm learn to move forward. (Source: \cite{Karpathy})]{
        \includegraphics[width=0.46\linewidth]{fig/MuJoCo}
        \label{fig:mujoco}
    }
    \hfill
    \subfloat[Atari games are popular to RL algorithm research. Here: Breakout (left) and Space Invaders (right) (Source: \cite{Deshpande}).]{
        \includegraphics[width=0.46\linewidth]{fig/Atari}
        \label{fig:atari}
    }
    \caption{Two Applications used for testing RL algorithms: MuJoCo and Atari.}
    \label{fig:rl}
\end{figure}

To achieve the signal needed for learning, i.e. exploration, noise must be somehow injected into the system:
- For RL, noise is injected into the agent's \emph{actions}.
- For ES, noise is injected directly into the \emph{parameters}.

We are basing our machine learning efforts on the Evolution Strategy (ES) algorithm outlined by Salimans et al. \cite{Salimans2017} using ES -- it being a gradient estimator--makes sense in optimization scenarios where calculating an exact gradient is expensive, or even impossible. The idea is to sample around a starting point, and use a weighted average fitness score to pick a direction to move each cycle. It is generally applicable to unsupervised problems, such as ours; a reinforcement learning problem. There are several flavors of the concept under the ES umbrella term: Covariance Matrix Adaptation (CMA-ES), Natural Evolution Strategy (NES), and Exponential NES, to name a few. The one used in \cite{Salimans2017} and by extension in our project is NES. Its exact implementation details will follow:

\subsection{Theoretical Advantages and Disadvantages of ES Compared to RL}
The advantages of ES is outlined in \cite{Karpathy} and can be summarized as follows:
\begin{enumerate}
    \item No need for back propagation.
    \begin{itemize}
        \item Makes code shorter and 2-3x faster
        \item Memory saving: not necessary to keep episode recordings for later update.
        \item No need to worry about exploding gradients in recurrent neural networks.
        \item We can explore larger classes of policy functions, including networks that are not differentiable.
    \end{itemize}
    \item Highly parallelizable.
    \begin{itemize}
        \item ES workers communicate scalars, RL workers communicate entire parameter vectors $\rightarrow$ easy to obtain linear speedups with more CPU cores.
    \end{itemize}
    \item Higher robustness.
    \begin{itemize}
        \item Has fewer hyperparameters. Several hyperparameters that are difficult to set in RL implementations are side-stepped in ES (although \cite{Karpathy} does not go into so much detail with this).
    \end{itemize}

    \item Structured exploration.
    \begin{itemize}
        \item Due to the noise being injected in parameter space, not actions, ES can use deterministic policies and achieve consistent exploration (instead of "jittering on the spot").
    \end{itemize}
    \item Credit assignment over long time scales
    \begin{itemize}
        \item By studying both ES and RL gradient estimators mathematically we can see that \emph{ES is an attractive choice especially when the number of time steps in an episode is long, where actions have long-lasting effects}, or if no good value function estimates are available.
    \end{itemize}
\end{enumerate}

\subsubsection{ES Challanges}

A core problem in ES is that adding noise in parameters must lead to different outcomes to obtain some gradient signal. It turns out that virtual batch normalization can help alleviate this problem \cite{Karpathy}.

Another problem is that it doesn't really work on \emph{supervised learning}, i.e. in settings where we know the/a solution/truth and want to use that to train a neural network. Supervised learning problems include image classification, speech recognition, or most other tasks in the industry, where one can compute the exact gradient of the loss function with backpropagation. For example, in the preliminary experiments it was found that using ES to estimate the gradient on the MNIST digit recognition task can be as much as 1,000 times slower than using backpropagation\cite{Karpathy}.

Therefore, \emph{it is only in RL settings, where one has to estimate the gradient of the expected reward by sampling, where ES becomes competitive}.

\subsubsection{ES Advantages Summary}
In conclusion we are interested in ES as an optimization algorithm for LETO search mainly because it has advantages in:
\begin{itemize}
    \item Reduced code complexity
    \item Ease of scaling (to large-scale distributed settings)
    \item It does not suffer in settings with sparse rewards, such as is the case in searching for LETOs.
\end{itemize}

\subsection{NES Algorithm}
In formal terms: \emph{``Let F denote the objective function acting on parameters θ. NES algorithms represent the population with a distribution over parameters \(p_\psi (\theta)\)—itself parameterized by $\psi$—and proceed to maximize the average objective value \(E_{\theta \sim p_\psi}\) over the population by searching for $\psi$ with stochastic gradient ascent. Specifically, using the score function estimator for $\nabla_\psi E_{\theta \sim p_\psi} F(\theta)$, [...] NES algorithms take gradient steps on $\psi$ with the following estimator:''} \cite[p.~2]{Salimans2017}
\begin{equation}
    \nabla_\psi E_{\theta \sim p_\psi} F(\theta)
    = E_{\theta \sim p_\psi} \{F(\theta)~\nabla_\psi~log~p_\psi(\theta)\}
\end{equation}
In our problem, \(F\) is the score returned by our environment (the Earth-Moon or Earth-Mars-Sun system)(required $\Delta v$ for our spacecraft's path) with \(\theta\) representing a single path in the environment and \(\psi\) being the parametrization of that path (example launch parameters: starting position of spacecraft, angle of burn vector at time of maneuver, and magnitude of that burn vector). The algorithm mutates the launch parameters \(\psi\), checks how the resulting path \(\theta\) performs in the environment \(F\) and moves around the resulting n-dimensional optimization space with gradient descent along its estimated gradient, defined by a normally distributed weighted point cloud \(\epsilon\).

\begin{figure}
    \centering
    \subfloat[Starting at some random point, the first iteration computes the fitness of a surrounding point cloud ($\epsilon$), and takes a step of size $\alpha$ in the direction of the weighted mean of $\epsilon$.]{
        \includegraphics[width=0.46\linewidth]{fig/ES_basic_0}
        \label{fig:esbasic0}
    }
    \hfill
    \subfloat[this process continues each iteration, until we find ourselves at a local maximum (this example version is doing gradient \textit{ascent})]{
        \includegraphics[width=0.46\linewidth]{fig/ES_basic_9}
        \label{fig:esbasic9}
    }
    \caption{the fundamental ES algorithm illustrated in a toy problem. Generated from code based on \cite{Salimans2017}}
    \label{fig:esbasic}
\end{figure}

In genetic algorithm terms, the input parameters $\psi$ are the genotype, the resultant path $\theta_\psi$ is the phenotype, and the fitness is of course  $F(\theta_\psi)$. \cite{Tomassini2005}. Since we simply input $\psi$ to the simulator that delivers $F(\theta_\psi)$, and $\theta$ is not really within the purview of the ES module, we will generally skip $\theta_\psi$, and simply write $F(\psi)$ instead, unless the distinction between geno- and phenotype is relevant. This also reflects the reality more precisely, with regards to our code.
\subsection{Modifications}
There are a number of modifications to the core ES idea of gradient descent with an estimated gradient, that are required for the algorithm to actually work in anything but the most ideal context.

\subsubsection{Fitness Shaping}
Fitness shaping refers to the transformations one can perform on the output of $F$, generally in order to smoothen the optimization algorithm's movement. One such transformation that was particularly critical for our problem was rank transformation, i.e. for each gradient estimation distribution $F(\psi_t + \sigma\epsilon_i)$, we replace the value of $F_i$ with its index in the sorted list of $F_{0..n}$. This shaping is to prevent outliers from overly influencing the gradient estimate. If a value is extreme in one way or the other, it would otherwise either pull or push the next iteration $\psi_{t+1}$ proportionately. With rank transformation, this sensitivity to the random sampling of $\sigma\epsilon$ largely disappears. \cite{Wierstra2011}.

Another fitness shaping approach that we tried was sub-par sample flattening. Since we are more interested in being pulled \textit{towards} good solutions than being pushed away from bad ones, we ignore the half of the estimation distribution $\sigma\epsilon$ that performs worse than the mean. We tried this both in and out of conjunction with rank transformation, but it did not perform particularly in either case, having a negligible influence.

\subsubsection{Antithetical Sampling}
Another measure to reduce the impact of variance in the sampling of $\sigma\epsilon$ is antithetical sampling. This means that for every point $\psi_t + \sigma\epsilon_i$ we sample, we evaluate its complement, $\psi_t - \sigma\epsilon_i$ as well. \cite{Salimans2017}. This means that there are always an equal number of points on either side of the distribution, and the estimated gradient is not thrown off by extreme stochastic outcomes in this regard.

\section{Fan Search}
In \cite{Saxe2015}, the search strategy was significantly simpler than here, and it serves as the point of comparison for our search efforts. The way it was done is that the same three parameters contained in $\psi$ were tested in a three-dimensional fan arrangement. For example, the starting position $\psi_{pos}$ was defined from 0 to $2\pi$, the burn angle $\psi_{ang}$ was defined between 0 and $-\frac{\pi}{4}$, and the burn vector's magnitude $\psi_{burn}$ was defined between 3 and 3.8. The search strategy was then to evaluate all superpositions of these three intervals, with e.g 100 values linearly spaced throughout each 'fan'. The set of values with the best score ($\psi\star$) would then be selected for refinement, where a smaller fan (tighter intervals) would be created, centered around $\psi\star$. The process would then repeat for some number of steps, constantly zeroing in on the best candidate.

For anyone with optimization experience it will be obvious that this method is very sensitive to local minimum issues. It doesn't multi-start, or compute any kind of gradient, definite or estimated. It relies heavily on empirically defined starting parameters for determining the initial search intervals, so it's very specific to this problem. It is, in short, not really universal, or particularly intelligent, and was thus seen as an obvious candidate for improvement in this project.

\section{Comparison of ES With Random Guessing Baseline}
In order to benchmark the performance of our ES implementation, we compared it with a baseline alternative; picking a random point within the problem space, and evaluating it, keeping it if it is the best we have seen. To find out which is best, there are two interesting metrics we can look at:
\begin{itemize}
    \item Given a fixed budget of fitness evaluations, which algorithm finds the lowest value.
    \item Given some threshold value that is considered "good enough", which algorithm reaches that in the shortest time..
    \item ..or in the case where it is interesting to have multiple good candidate solutions, which algorithm outputs most of them in a given time.
\end{itemize}
Since the CPU-bound version of our algorithm takes several seconds to evaluate one fitness function, and the GPU version has had most of the pythonic\footnote{``Pythonic'' is a term often thrown around in the python programming community meaning ``doing things the way that is considered best practice in the Python'' that are in harmony with \href{https://www.python.org/dev/peps/pep-0020/}{The Zen of Python}, see \url{https://docs.python-guide.org/writing/style/}} convenience compromised, we pre-computed the fitness values in a promising cutout of the problem space. The 1024x1024 grid seen in \cref{fig:golf_course_s1024} took approximately 40 hours to compute on an HPC node, and with it, we were able to rapidly iterate our algorithm, both for the purpose of running it with multi-start to alleviate randomness, and to test the various potential accuracy and convergence-speed optimizations that have been applied in the literature.
\begin{figure}[ht]
    \centering
    \includegraphics[width=\linewidth]{fig/golf_course_s1024.png}
    \caption{1024x1024 resolution fitness map of the flattened problem space. burnDV is applied delta-v and position is the angular position around Earth ($\text{position} = 0$ is the line between Earth to Moon). A certain fractal quality seems to be present, consistent with \cite{Topputo2014}}
    \label{fig:golf_course_s1024}
\end{figure}


\documentclass{tufte-handout}

\title{Assignment 3: Extended Kalman Filter}

\author[Kautilya Chenna]{Kautilya Chenna}

%\date{28 March 2010} % without \date command, current date is supplied

%\geometry{showframe} % display margins for debugging page layout

\usepackage{graphicx} % allow embedded images
    \setkeys{Gin}{width=\linewidth,totalheight=\textheight,keepaspectratio}
    \graphicspath{{graphics/}} % set of paths to search for images
\usepackage{amsmath}  % extended mathematics
\usepackage{booktabs} % book-quality tables
\usepackage{units}    % non-stacked fractions and better unit spacing
\usepackage{multicol} % multiple column layout facilities
\usepackage{lipsum}   % filler text
\usepackage{fancyvrb} % extended verbatim environments
    \fvset{fontsize=\normalsize}% default font size for fancy-verbatim environments

% Standardize command font styles and environments
\newcommand{\doccmd}[1]{\texttt{\textbackslash#1}}% command name -- adds backslash automatically
\newcommand{\docopt}[1]{\ensuremath{\langle}\textrm{\textit{#1}}\ensuremath{\rangle}}% optional command argument
\newcommand{\docarg}[1]{\textrm{\textit{#1}}}% (required) command argument
\newcommand{\docenv}[1]{\textsf{#1}}% environment name
\newcommand{\docpkg}[1]{\texttt{#1}}% package name
\newcommand{\doccls}[1]{\texttt{#1}}% document class name
\newcommand{\docclsopt}[1]{\texttt{#1}}% document class option name
\newenvironment{docspec}{\begin{quote}\noindent}{\end{quote}}% command specification environment

\begin{document}

\maketitle% this prints the handout title, author, and date

\section{Exercise 1: Bayes Filter and Extended Kalman Filter}
\begin{fullwidth}
\textsc{Question 1}: Describe briefly the two main steps of the Bayes filter in your own words.
\end{fullwidth}

First step is the prediction step where we use the motion model to update our belief and then we have the correction step where we use the observation model or the sensor model to correct our belief about the state.

\begin{fullwidth}
\textsc{Question 2}: Describe briefly the meaning of the following probability density functions.
\end{fullwidth}

$p(x_t \mid x_{t-1},u_t)$: Motion model

$p(z_t\mid x_t)$: Sensor model

$bel(x_t)$: belief of the state

\begin{fullwidth}
\textsc{Question 3}: Specify the distributions that correspond to the above mentioned three terms in the EKF.
\end{fullwidth}

All three distributions are Gaussian, since EKF assumes that the the distributions be Gaussian.

\begin{fullwidth}
	\textsc{Question 4}: Explain in a few sentences all of the components of the EKF algorithm
\end{fullwidth}

\end{document}