\section{AMBA Protocol Case Study}\label{amba:sec}

We demonstrate how to synthesize a parameterized implementation of the AMBA AHB,
with guaranteed correctness for any number of masters.
To this end, we translate the LTL specification of the
AMBA AHB (as found in~\cite{BarbaraThesis}) into a version that is suitable
for parameterized synthesis in token rings, and address several challenges
with respect to theoretical applicability and practical feasibility:
\li
\- We show how to localize global input and output signals (those that cannot be assigned to one particular master).
   This is necessary since our approach is based on the replication of components that act only on local information.
\- We extend the cutoff results to fully asynchronous timing model and systems with two process templates.
\- We describe further optimizations that make synthesis feasible,
   in particular based on the insight that the AMBA protocol features three
   different types of accesses, and the control structures for these accesses
   can be synthesized step-by-step.
\il

\subsection{Description of the AMBA Protocol} \label{sec:amba}

ARM’s \emph{Advanced Microcontroller Bus Architecture} (AMBA)~\cite{AMBAspec}
is a communication bus for a number of masters and
clients on a microchip. One of the crucial parts of AMBA is
the \emph{Advanced High-performance Bus} (AHB),
a system bus for the efficient connection of processors, memory, and devices.

For convenience, the input signals are depicted in \textcolor{red}{red} color
and the outputs are \textcolor{blue}{blue}.

The bus arbiter ensures that only one master accesses the bus at any time.
Masters send \hbusreq\ to the arbiter if they want access, and receive
\hgrant\ if they are allowed to access it.
Masters can also ask for different
kinds of \emph{locked transfers} that cannot be interrupted.

The exact arbitration
protocol for AMBA is not specified. Our goal is to synthesize a protocol
that guarantees safety and liveness properties. According to the
specification, any device that is connected to the bus will react to an input
with a delay of one time step. I.e., we are considering Moore machines. In
the following, we introduce briefly which signals are used to
realize the arbiter of this bus for masters.

\parbf{Requests and grants}
The identifier of the master which is
currently active is stored in the $n+1$-bit signal \hmaster[$n$:0],
with $n$ chosen such that the number of masters fit into $n+1$ bits.
To request the
bus, master $i$ raises signal \hbusreqi. The arbiter decides who will be
granted the bus next by raising signal \hgranti. When
the client raises \hready, the bus access starts at the next tick, and there
is an update \hmaster[$n$:0] := i, where \hgranti\ is currently active.

\parbf{Locks and bursts}
A master can request a locked access by raising both \hbusreqi\ and \hlocki. In this case,
the master additionally sets \hburst[1:0] to either \ambasignal{single}
(single cycle access), \hburstfour\ (four cycle burst) or \hincr\ (unspecified
length burst). For a \hburstfour\ access, the bus is
locked until the client has accepted 4 inputs from the master (each signaled by
raising \hready). In case of a \hincr\ access, the bus is
locked until \hbusreqi\ is lowered. The arbiter raises \hmastlock\ if the bus
is currently locked.

\parbf{LTL specification}\ak{where $\G \neg g_1 \land g_2$?}
The original natural-language specification~\cite{AMBAspec} has been translated into a formal specification in the GR(1) fragment of LTL before in~\cite{BarbaraThesis,Bloem12,GodhalCH13}.
Figure~\ref{amba:fig:AMBAspec} shows the environment assumptions and system guarantees from~\cite{BarbaraThesis} that serve as the basis for our parameterized specification.
The full specification is
$ (A1 \land \ldots \land A4) \rightarrow (G1 \land \ldots \land G11)$.
%
\begin{figure}[t]
  \fbox{%
  \begin{minipage}{\textwidth}
  \input{token-systems/amba-spec}
  \end{minipage}}
\caption{Specification of the AMBA AHB~\cite{BarbaraThesis}, in the GR(1) fragment of LTL.
  The inputs are:
  \hburst, \hbusreq[i], \hready, and \hlock[i].
  The outputs are:
  \hmastlock, \hmaster, \hstart, \hdecide, \hlocked, and \hgrant[i].}
\label{amba:fig:AMBAspec}
\end{figure}
%

\parbf{Challenges}
The AMBA specification has global inputs and outputs (those are without ``$[i]$''),
distinguishes 0 from non-0 processes (G10.1 and G11),
has the assume-guarantee form $\A(\forall i. A1\land...\land A4 \impl \forall i.G1\land...\land G11)$
(thus not in the prenex-indexed form),
has the process quantification inside a temporal operator in G10.2 $\G(\forall i... \impl ...)$,
and requires a synchronous mode of execution (all processes transit simultaneously).
Thus we cannot apply the cutoff results (Theorem~\ref{tok_rings:thm:cutoffs} on page~\pageref{tok_rings:thm:cutoffs})
for parameterized synthesis.
The next section shows how to handle this.

% We will show how we can use our approach for parameterized synthesis in token
% rings for synthesizing component implementations for the AMBA AHB
% specification, such that every component serves the requests of one master,
% and the composition of an arbitrary number of components is correct by
% construction. To illustrate our solution, we first review the parameterized
% synthesis approach, and then mention a number of problems that need to be
% solved for the AMBA case study.

%\subsection{Additional Definitions}
%

\subsection{Handling the AMBA Specification}\label{amba:sec:handling-amba}

  This section shows how to rewrite the AMBA specification into a form
  admissible to the parameterized synthesis.
  We not only rewrite the specification,
  but also extend the cutoff results~\cite{Emerso03} to the resulting class of specifications.
  Note that the resulting specification is not the same as the original AMBA (but closely resembles it),
  due to constraints of the token-ring architecture.
  (For example, token rings cannot ensure immediate granting of a client,
   because the token has to travel to the corresponding process first.)
  The resulting specification describes a round-robin arbiter
  with different granting schemes and one special process.

%%%%The AMBA specification (Figure~\ref{amba:fig:AMBAspec}) has the form
%%%%$
%%%%(A1 \land \ldots \land A4) \impl (G1 \land \ldots \land G11).
%%%%$
%%%%The specification poses
%%%%the following challenges to existing parameterized synthesis approach~\cite{JB12}
%%%%and cutoff results~\cite{Emerso03,AJKR14}:

\subsubsection{Special 0-process: two process templates $(A,B)$}

  The specification distinguishes between master number 0 and all other masters.
  We support this by synthesizing two different process implementations,
  $A$ for the 0-process and $B$ for non-0 processes:
  the $A$-process serves master 0 and the $B$-processes serve the other masters.
  We denote a token-ring system composed of one $A$ process and $n$ copies of $B$
  using the notation $(A,B)^{(1,n)}$.
  The modified parameterized synthesis problem is to find $(A,B)$ such that
  $\forall n: (A,B)^{(1,n)} \models \Phi$.
  Later we will separate the specification into two parts:
  one will talk about process $A$, another will talk about $B$-processes.

\subsubsection{Localizing global outputs}

  The AMBA specification has global outputs \hmastlock, \hmaster, \hstart, \hdecide, and \hlocked.
  They depend on the global state of the system,
  which is not handled by the work on parameterized model checking of token ring systems~\cite{Emerso03,AJKR14}.
  To overcome this,
  we introduce local versions of the global outputs and build global outputs from them:
  \li
  \- $\hmaster = i$ whenever $\hmasteri$ is high, and
  \- for every global output $glob$ from $\{\hmastlock, \hstart, \hdecide, \hlocked\}$,
     $glob = \exists{i}.\ \toki \land glob_i$.
  \il

  We replace each global output with its local version, e.g., $\hstart$ is replaced by $\hstarti$.
  Note that the limited communication interface (via token passing)
  does not make AMBA specification unrealizable,
  although processes cannot access the value of global outputs when they do not possess the token.
  Intuitively, this is because the token is the shared resource that guarantees mutual exclusion of grants, and therefore the values of these global signals should always be controlled by the process that has the token. In particular, outputs \hdecide\ and \hstart\ are used to decide when to raise a grant and when to start and end a bus access\footnote{The original AMBA specification~\cite{AMBAspec} does not have these signals---they were introduced to simplify the formalization of the specification~\cite{BarbaraThesis}.}, which should only be done when the token is present.
  Similarly, signals \hmastlock\ and \hmaster\ should be controlled by the process that currently controls the bus (and hence has the token).

  Finally, we mentioned many times that the token should be used to ensure the mutual exclusion of grants.
  Let us explicitly add this requirement into the specification,
  namely we add G12: $\forall i.\,\hgranti \rightarrow \toki$.
  (The original formula contains only an \emph{implicit} mutual exclusion property:
   G4 defines how \hmaster\ is updated by the \hgranti\ signals,
   which can only be satisfied if \hgranti\ are mutually exclusive.)

\subsubsection{Splitting the specification into two \& other small rewritings}

  Once we localized global outputs,
  we can talk about splitting the AMBA specification into two parts.
  At first, each part will be in the assume-guarantee form,
  where the assumptions talk about all the processes (the only $A$ and all $B$),
  but the guarantees will be separated into
  (i) guarantees for the $B$-processes and
  (ii) guarantees for the process $A$.

  After the localization, guarantees G10.1 and G10.2 become:
  \[\begin{array}{rrlr}
  \forall \mathrm i\neq 0\!:\!\!\!\!\! & \always & \neg \hgranti \impl (\neg \hgranti \weakuntil \hbusreqi) & (G10.1)\\[3pt]
  & \always & (\hdecide[0] \land \forall \mathrm i.\neg \hbusreqi) \impl \nextt \hgrant[0] & (G10.2) \\[3pt]
  \end{array}\]
  Thus, G10.1 is used for $B$-processes,
  while G10.2 is used for the process $A$.
  Let us talk more about G10.2, because it has two issues.

  The first issue with G10.2 is that it requires an immediate reaction to a situation when no process receives a bus request.
  This is unrealizable in token rings,
  because mutual exclusion of the grants requires possession of the token,
  and the token transmission takes time.
  We modify G10.2 to allow the process $A$ to wait for the token and then immediately react:
  $$
  \G
  \big(
  \neg \tok[0] \land \X \tok[0] \land \forall \mathrm i. \neg \hbusreqi
  \impl
  \nextt \hgrant[0]
  \big).
  \footnote{Maybe you expected to see
    $\always \ (\hdecide[0] \land (\forall \mathrm i: \neg \hbusreqi) \land \nextt \tok[0])
    \impl \nextt \hgrant[0])$,
    but this makes the specification unrealizable due to the token ring requirement
    $\always (\tok \impl \eventually \send)$:
    the environment falsifies it by making $\forall \mathrm i: \neg \hbusreqi$ true whenever the process raises $\hdecide$ (and hence the process should continue granting and cannot release the token).
    To regain realizability one could add additional assumptions (something like $\G\F (\hdecide\land\neg\norequests)$).
    Instead, we decided to change slightly the specification.%
  }
  $$

  The second issue with G10.2 is the quantifier $\forall{i}$ \emph{inside} the temporal operator $\G$
  (such specifications were not studied in parameterized model checking of token rings).
  It requires the process $A$ to know about inputs of all $B$-processes,
  as it needs to react to a situation where \hbusreqi\ is low for every process.
  To get rid of the nesting $\G(\forall i...)$,
  we introduce a new global input $\norequests$,
  and add the assumption $\forall{i}.\G (\hbusreqi \impl \neg \norequests)$.
  Then G10.2 becomes:
  $$
  \G \
  (\neg \tok[0] \land \X \tok[0] \land \norequests)
  \impl \X \hgrant[0]).
  $$
  This strengthens the specification,
  because the environment can set $\neg\norequests$
  even when there are no requests.
  This concludes the discussion of G10.2.

  The last asymmetric property is the guarantee G11. We split it into two parts:
  \li
  \- G11.1: $\neg \hgranti \land \neg \hmastlocki$ for $B$-processes and
  \- G11.2: $\tok[0] \impl \hgrant[0] \land \hmaster[0] \land \neg \hmastlock[0]$ (for $A$).
  \il
  % In the general case, if allow index quantifiers inside temporal operators, then parameterized model checking becomes undecidable even for systems with no synchronization at all~\cite[Appendix A]{Igor}.
  % However, for the special case of G10.2 their undecidability proof breaks.\ak{so what?}
  % since G10.2 contains only a single index quantifier inside the topmost $\always$ operator.

\subsubsection{Localizing global inputs}

  The AMBA specification in Figure~\ref{amba:fig:AMBAspec} uses global inputs \hburst,\hready,
  and \norequests\ that we introduced in the previous section.

  First, we introduce local versions $\hburst[i]$, $\hready[i]$, and $\norequests[i]$,
  and add the assumption $\forall i\neq j.\ loc_i = loc_j$ for $loc \in \{\hburst, \hready,\norequests\}$.
  This rewriting does not change the specification.
  The specification becomes
  \begin{align*}
  &\A\Big(\forall{i \neq j}_{i,j \in \{A,B_1,...,B_n\}}. \Phi({i,j}) ~\impl~ \forall{k \in \{B_1,...,B_n\}}. \Psi(k)\Big) ~\land\\
  &\A\Big(\forall{i \neq j}_{i,j \in \{A,B_1,...,B_n\}}. \Phi'({i,j}) ~\impl~ \Psi'(A)\Big),
  \end{align*}
  where each $\Phi(i,j)$ and $\Phi'(i,j)$ talk about propositions of processes $i$ and $j$,
  and $\Psi(k)$ and $\Psi'(A)$ talk about propositions of process $k$ and $A$ respectively.
  Note that $\Psi \neq \Psi'$ because we split the guarantees for $B$-processes and the process $A$
  (the assumptions also slightly differ, so we use $\Phi$ and $\Phi'$).
  Now the specification does neither have global inputs nor global outputs.

  Second, we drop the newly introduced assumptions.
  This means that the original global inputs $\hburst$, $\hready$, and $\norequests$ may have different values for different processes,
  i.e., they are not ``global'' anymore.
  This strengthens the specification,
  because dropping the assumptions enables more environment behaviors (and the formula is universal $\A(...)$).
  The resulting specification becomes
  \begin{equation}\label{eq:amba:assume-guarantee-spec}
  \begin{aligned}
  &\A\big(\forall{i \in \{A,B_1,...,B_n\}}. \Phi(i) ~\impl~ \forall{j \in \{B_1,...,B_n\}}. \Psi(j)\big) ~\land\\
  &\A\big(\forall{i \in \{A,B_1,...,B_n\}}. \Phi'(i) ~\impl~ \Psi'(A)\big).
  \end{aligned}
  \end{equation}
  Figures~\ref{fig:AMBASpecNewI}~and~\ref{fig:AMBASpecNew0} define $\Phi$, $\Psi$, $\Phi'$, and $\Psi'$.
  We stress that $\Phi(i)$ and $\Phi'(i)$
  has to be conjoined over all $B$-processes \emph{and} the process $A$ to form the assumptions.

  \begin{figure}%[t]
    \fbox{%
    \begin{minipage}{\textwidth}
    \input{token-systems/amba-spec-new}
    \end{minipage}
    }
    \caption{Parameterized AMBA specification for $B$-processes:
      assumptions $\Phi(i)$ and guarantees $\Psi(i)$.
      G10.2 is omitted, since it is only needed for the process $A$.%
    }
    \label{fig:AMBASpecNewI}
  \end{figure}
  \begin{figure}%[t]
    \fbox{%
    \begin{minipage}{\textwidth}
    \input{token-systems/amba-spec-new0}
    \end{minipage}}
    \caption{Parameterized AMBA specification for the process $A$:
      modifications wrt.\ Figure~\ref{fig:AMBASpecNewI}.
      The index $0$ denotes the process $A$.%
    }
    \label{fig:AMBASpecNew0}
  \end{figure}

\subsubsection{Resulting parameterized specification and cutoffs}

  We still cannot apply the cutoff results to the specification in Eq.\ref{eq:amba:assume-guarantee-spec},
  because it is in the assume-guarantee form (and thus is not prenex-indexed)
  and has the synchronous timing model.

  To handle the synchronous timing model, we synthesize a more general case of fully asynchronous systems
  (those work under all ranges of schedulers from the synchronous to the interleaving one).
  This represents a more difficult synthesis task,
  but if the synthesizer finds such a system,
  then the system works in the synchronous setting too
  (because we have universal properties).

  To handle the assume-guarantee issue,
  we localize the assumptions as described in Eq.\ref{tok_rings:eq:localised} on page~\pageref{tok_rings:eq:localised},
  by strengthening
  $\A(\forall i. \phi(i) \impl \forall j. \psi(j))$ into
  $\A\big(\forall i. (\phi(i)\land\GF\token_i \impl \psi(i))\big)$
  where $A_i$ in Eq.\ref{tok_rings:eq:localised} is $\phi(i)$.
  The final specifications are (in LTL):
  \begin{equation}\label{eq:amba:final-spec}
  \boxed{
  \begin{aligned}
    \text{for $B$-processes:}
    &~~\forall{i \in \{B_1,...,B_n\}}. \Phi(i)\land\GF\token_i \impl \Psi(i)
    \text{ and } A_{loc} = \Phi,\\
    %
    \text{for the process $A$:}
    &~~\Phi'(A) ~\impl~ \Psi'(A) \text{ and } A_{loc} = \Phi',
  \end{aligned}
  }
  \end{equation}
  where $\Phi$, $\Phi'$, $\Psi$, and $\Psi'$ are defined in Figures~\ref{fig:AMBASpecNewI}~and~\ref{fig:AMBASpecNew0}
  (Recall that $A_{loc}$ is a formula over process propositions
   such that $\G[\token \land A_{loc} \impl \F \tsnd)]$,
   see definitions on page~\pageref{page:tok_rings:defs:process_template}.)

  Let us prove cutoffs for specifications of the above form.
  The parameterized synthesis problem can be separated into two: find $(A,B)$ such that
  \begin{align*}
  \begin{aligned}
  &\forall n: \largesys ~\models~ \forall{i \in \{B_1,...,B_n\}}. \Phi(i) ~\impl~ \Psi(i) ~\land\\
  &\forall n: \largesys ~\models~ \Phi'(A) \impl \Psi'(A),
  \end{aligned}
  \end{align*}
  where $A_{loc}$ is either $\Phi$ or $\Phi'$.

  \begin{theorem}
    Given two process templates,
    $A=(\Ipr,\Opr,Q^A,Q_0^A,\delta^A,out^A,A^A_{loc})$ and
    $B=(\Ipr,\Opr,Q^B,Q_0^B,\delta^B,out^B,A^B_{loc})$,
    and let $\Iglob=\emptyset$ (no global inputs).
    Assume that initially the process $A$ has the token.
    Then a cutoff is $(1,1)$ (one $A$-process and one $B$-process) for the following PMCPs:
    \li
    \-[(1)] $\forall n: (A,B)^{(1,n)} \models A^A_{loc} \land \GF\token_A ~\impl~ \psi(A)$,
    \-[(2)] $\forall n: (A,B)^{(1,n)} \models \forall i: A^B_{loc,i} \land \GF\token_i ~\impl~ \psi(B_i)$.
    \il
    where
    $A^B_{loc,i}$ is $A^B_{loc}$ with all propositions subscripted with $i$,
    $\psi(p)$ is an LTL formula over propositions of a process $p \in \{A,B_1,...,B_k\}$.
  \end{theorem}

  \begin{proof}[Proof idea]
    The proof is inspired by the original proof~\cite{Emerso03}.

    \parbf{Item (1)}
    Fix an arbitrary $n>1$ and let $\varphi(A) = A^A_{loc}\land \GF \token_A \impl \psi(A)$.
    We prove that
    $$
    (A,B)^{(1,1)} \models \varphi(A) ~~\Iff~~ \largesys \models \varphi(A).
    $$

    Consider direction $\Rightarrow$. After contra-positioning:
    $$
    (A,B)^{(1,1)} \not\models \varphi(A) ~~\Leftarrow~~ \largesys \not\models \varphi(A).
    $$
    Given a system run of $(A,B)^{(1,1)}$ that satisfies $\neg\varphi(A)$,
    we build a system run of $(A,B)^{(1,n)}$ that satisfies $\neg\varphi(A)$.
    The construction is in Figure~\ref{amba:fig:simulation_proof_1}.
    \begin{figure}[t]\centering{
      \includegraphics[scale=0.7]{token-systems/figures/simulation_proof}}
      \caption{Constructing a run of a cutoff system from a run of a large system.
        Vertical lines depict (local) paths of the processes,
        the horizontal lines mean the token transmission.
        The process $A$ starts with the token.%
      }
      \label{amba:fig:simulation_proof_1}
    \end{figure}
    We copy the behaviors of processes $A$ and $B_1$ until before $B_1$ sends the token.
    At this moment, we postpone sending the token by $B_1$ and stutter\footnote{
      To ``stutter a process $p$'' means ``not to schedule it''.
      As a result, a stuttered process neither reads inputs nor changes its state.
      In the figures it is shown by repeating a state.%
    }
    it,
    while the process $A$ continues execution until it gets into state \textcolor{orange}{$\CIRCLE$}
    ready to receive the token.
    Then $B_1$ transmits the token to process $A$.
    After that we move process $B_1$ into state \textcolor{blue}{$\blacklozenge$},
    while $A$ stutters in \textcolor{orange}{$\CIRCLE$}.
    Now we are in the original situation and repeat the construction.
    Since the property talks about process $A$ only, the resulting run satisfies it.
    Finally, we assumed that the processes of the large system pass the token infinitely often.
    If some process $B_x \in \{B_1,...,B_n\}$ holds the token forever,
    then we use its behavior for $B_1$ in the cutoff system
    (this may require to insert stuttering steps into behaviors of $B_1$ and $A$ of the cutoff system,
     to synchronize their (finitely many) token transmissions).

    Consider direction $\Leftarrow$. After contra-positioning:
    $$
    (A,B)^{(1,1)} \not\models \varphi(A) ~~\Rightarrow~~ \largesys \not\models \varphi(A).
    $$
    Given a system run of $(A,B)^{(1,n)}$ that satisfies $\neg\varphi(A)$,
    we build a system run of $(A,B)^{(1,1)}$ that satisfies $\neg\varphi(A)$.
    Figure~\ref{amba:fig:simulation_proof_2} shows how to construct a run of a system
    that has one more $B$ process than the cutoff system.
    By repeating the construction we can add the necessary $n-1$ $B$-processes.
    The construction works as follows.
    \begin{figure}[t]\centering{
      \includegraphics[scale=0.7]{token-systems/figures/simulation_proof2}}
      \caption{Constructing a run of a system $(A,B)^{(1,2)}$ from a run of a cutoff system $(A,B)^{(1,1)}$.
        Vertical lines depict (local) paths of the processes,
        the horizontal lines mean the token transmission.
        The process $A$ starts with the token.%
      }
      \label{amba:fig:simulation_proof_2}
    \end{figure}
    The new process $B_2$ copies the behavior of $B_1$ until before $B_1$ receives the token
    (i.e., up to the state \textcolor{green}{$\rule{0.6em}{0.6em}$}).
    Then it stutters in \textcolor{green}{$\rule{0.6em}{0.6em}$} awaiting for the token from process $B_1$.
    After that it copies $B_1$ behavior from state \textcolor{green}{$\CIRCLE$} till \textcolor{blue}{$\blacklozenge$},
    while processes $A$ and $B_1$ stutter.
    Then $B_2$ sends the token to $A$ and we return to the original situation.
    Finally, the case of $B_1$ or $A$ holding the token forever is straightforward.

    \parbf{Item (2)}
    Consider the case $\forall i. \varphi(B_i)$.
    First, we use the symmetry argument: for every $n$,
    $$
    \largesys \models \forall i.\varphi(B_i) ~\Iff~ \largesys \models \varphi(B_1).
    $$
    It holds because, for every $B_i$ and system run that satisfies $\neg\varphi(B_i)$,
    we can construct a run that satisfies $\neg\varphi(B_1)$.
    The latter is possible because all $B$-processes start without the token and $\varphi$ is 1-indexed%
    \footnote{%
      In contrast,
      the symmetry argument will not work for properties of the form $\forall i. \varphi(A,B_i)$,
      because $B_1$, $B_n$, and $B_{x \in \{2,...,n-1\}}$ have different ``relation'' to $A$.
      For example, take the formula $\forall i. \G(\token_A \impl \token_A \W \token_i)$.
      The (wrongly applied) symmetry argument would produce $\G(\token_A \impl \token_A \W \token_1)$,
      which says that the token moves from $A$ to $B_1$ (trivially true in every system),
      but the original formula does not hold.%
    }.

    After applying the symmetry argument,
    we can use the very same constructions as in item (1),
    see Figures~\ref{amba:fig:simulation_proof_1}~and~\ref{amba:fig:simulation_proof_2}.
    Let us only note the case when the token is stuck in some process.
    As for the construction in Figure~\ref{amba:fig:simulation_proof_2},
    this is simple: the token will be stuck in $A$ or in $B_1$ in the large system too.
    Consider the case in Figure~\ref{amba:fig:simulation_proof_1},
    when the token gets stuck in some process $B_d$ for $d \neq 1$.
    This is the only place in the proof where we use the peculiar structure of the formula to verify:
    $A^B_{loc,1} \land \GF\token_1 \impl \psi(B_1)$.
    Recall that the contra-position negates it and gives
    $A^B_{loc,1} \land \GF\token_1 \land \neg \psi(B_1)$.
    Thus, in the large system the process $B_1$ receives the token infinitely often,
    and we can simply ignore the case\footnote{%
      We \emph{did} consider the case in other proof branches,
      to avoid relying on the peculiarity of the formula.
      We conjecture that in the case of (more general) properties of the form $\forall i. \varphi(B_i)$ (without $\GF\token_i$),
      the cutoff increases to $(1,2)$.%
    }.
  \end{proof}

  Let us note that without the assumption ``$A$ starts with token'' the constructions break.
  We conjecture that in this case a cutoff increases to $(1,2)$.

\subsection{Experiments} \label{amba:sec:experiments}

  In this section,
  we describe optimizations that are crucial for the synthesis of the parameterized AMBA,
  and present synthesis timings and resulting implementations.
  Most of the optimizations were already described in Section~\ref{tok_rings:sec:optimizations}.
  One interesting and not previously described optimization is ``Decompositional synthesis'',
  where the specification is synthesized incrementally, starting from a subset of the properties.
  It is this optimization that allowed us to synthesize the AMBA.

\parbf{Prototype}
  Our prototype is based on our tool \textsc{Party}~\cite{party},
  a synthesizer of parameterized token rings.
  \textsc{Party} is written in Python,
  uses LTL3BA~\cite{LTL3BA} for automata translation and Z3~\cite{Moura08} for SMT solving.
  The prototype and specification files can be found at \small{\url{https://github.com/5nizza/Party/}} (branch `amba-gr1').
  The experiments were run on a x86\_64 machine with $2.6$GHz CPU, $12$GB RAM, Ubuntu OS.

\parbf{Synchronous hub abstraction (Section~\ref{tok_rings:sec:optimizations})}
  Synchronous hub abstraction can be applied to 1-indexed specifications.
  It lets the environment simulate all but one process, and always schedules this process.
  Thus, the synthesizer searches for a process template in the synchronous setting
  with additional assumptions on the environment, namely:
  (i) the environment sends the token to the process infinitely often, and
  (ii) the environment never sends the token to the process if it already has it.
  The synchronous hub abstraction is \emph{sound and complete} for 1-indexed properties.
  After applying this optimization \emph{any} monolithic synthesis method
  can be applied to the resulting specification
  (in the form of Eq.\ref{tok_rings:eq:hub-abstraction} on page~\pageref{tok_rings:eq:hub-abstraction}).

\parbf{Hardcoding states with and without the token~\cite[Section4]{Khalimov13}}
  The number of states with and without the token in a process template
  defines the degree of the parallelism in a token ring.
  Parallelism increases with the number of states that do not have the token.
  In the AMBA case study, any grants related action depends on having the token.
  Thus we divide the states in the process template:
  (a) one state does not have the token, while
  (b) all other states have the token.
  We do this by hardcoding the $\tok$ output function.

\parbf{Decompositional synthesis of different grant schemes}
  The idea of the decompositional synthesis is:
  synthesize a subset of the properties,
  then synthesize a larger subset using the model from the previous step as the basis.
  %   As will be noted in the experiments section, all previous optimizations still do not allow us to synthesize AMBA.
  %   It is a successive synthesis approach of properties together with hardcoding token states (next paragraph) that have made it.
  Consider an example of the synthesis of the non-0-process of AMBA.
  The flow is:
  \begin{enumerate}
  \- Assume that every request is a locked request of type \hburstfour, i.e.,
     add the assumption $\always(\hlocki\land\hburst=\hburstfour)$ to the specification.
     This implicitly removes guarantee G2 and assumption A1 from the specification.
     Synthesize the model. The resulting model has $10$ states
     (states $t0,..,t9$ with their outputs, as well as transitions between them that satisfy the added assumption,
      are in Figure~\ref{amba:fig:ith-model}).

  \- Use the model found in the previous step as the basis:
     assert the number of states, values of output functions in these states, transitions for inputs that satisfy the previous assumption.
     Transitions for inputs that violate the assumption from step 1 are not asserted, and thus are left to be synthesized.

     Now relax the assumptions: allow locked and non-locked \hburstfour\ requests, i.e., replace the previous assumption with $\always(\hburst=\hburstfour)$.
     Again, this implicitly removes G2 and A1.
     In contrast to the last step, now guarantee G3 is not necessarily `activated' if there is a request.

     Synthesize the model.
     This may require increasing the number of states (and it does, in the case of non-0 process)---add new states and keep assertions on all the previous states.

  \- Assert the transitions of the model found, as in the previous step.\\
     Remove all added assumptions and consider the original specification.
     Synthesize the final model.
\end{enumerate}

Although for AMBA this approach was successful, it is not clear how general it is.
For example, it does not work if we start with locked \hburstfour\ and \hready\ always high,
and then try to relax it.
Also, the separation into sets of properties to be synthesized was done manually.

%
\begin{figure}[t]\centering{
\includegraphics[width=\textwidth]{token-systems/figures/ith-model}}
\caption{Synthesized model of non-$0$-processes (after manual simplification).
Circle green state ($t0$) is without the token, other states are with the token.
The initial state is $t0$.
States are labeled with their active outputs.
Edges are labeled with inputs, a missing input variable means ``don't care''.
`Burst4' means $\hburst=\hburstfour$, `incr' means $\hburst=\hincr$, `single' means neither of them.
In the first step of decompositional synthesis states $t0,..,t9$ were synthesized, in the second $t10,..,t12$ were added, in the final step state $t13$ was added.}
\label{amba:fig:ith-model}
\end{figure}
%
\begin{figure}[t]\centering{
\includegraphics[width=\textwidth]{token-systems/figures/0-model}}
\caption{Synthesized model of $0$-processes (after manual simplification).
Circle state ($t0$) is without the token, other states are with the token.
The initial state is $t1$.
States are labeled with their active outputs.
Edges are labeled with inputs, a missing input variable means ``don't care''.
`Burst' means $\hburst=\hburstthree$ (recall we decreased the length of bursts for 0 process), `incr' means $\hburst=\hincr$, `single' means neither of them.
In the first step of decompositional synthesis states $t0,..,t10$ were synthesized, in the second only transitions were synthesized, but no new states added, in the final step $t11$ was added.}
\label{amba:fig:zero-model}
\end{figure}
%\todo{can we reuse ith models for the zero-process?}

%\parbf{Optimization of SMT Encoding}\label{sec:simpleGR1}
%%
%\newcommand{\Atm}{\ensuremath{{A_{\neg\varphi}}}}
%Recall from Section~\ref{sec:approach} that SMT based bounded synthesis, given an automaton \Atm\ of the negation of specification $\varphi$ and an unknown process template $P=(\Locals_P, \LocalsI, \ActionsProc, \delta_P, out)$ with a fixed number of states, encodes the product automaton $\Atm \times P$ into SMT constraints such that $\Atm \times P$ contains no reachable loops with an accepting state of the \Atm\ iff SMT constraints are satisfiable.
%Below is a general assertion from which the SMT query is composed:
%$$
%\bigwedge{q\in Q_P}
%\bigwedge{\trans{a}{b}{i,o}\in \delta_\Atm}: \ \ \
%\rho(a,q) \ge 0 \land o=out(q)
%\impl
%\rho(b,\delta_P(q,i))
%%
%% [> \text{ if } b \text { is accepting, else } \ge]
%\vartriangleright
%\rho(a,q)
%$$
%where $\vartriangleright$ is `$>$' if $b$ is an accepting state of \Atm, else `$\ge$'.
%In words: for any state of the process template, and any transition of the automaton, if the current state of the product automaton is reachable, then the next state should also be reachable and the ranking function should be as stated.
%
%The specification of AMBA we synthesize is derived from GR(1) specification.
%As a consequence it contain assumptions (A3, A6) of the form
%$\always\alpha(i)$ where $\alpha(i)$ is a Boolean formula over current inputs, and many guarantees (G1, G4, G5, G6, G7, G8, G12, G10.2) of the form $\always\beta(i,o,o')$ where $\beta(i,o,o')$ is a Boolean formula over current inputs and outputs and next outputs.
%Instead of using the standard approach via automaton translation described above, we:
%\begin{enumerate}
%  \item encode assertions of the form $\always\alpha(i)$ directly into SMT constraints, namely add $\alpha(i)$ to the the premise of the SMT rule.
%  Thus, the premise becomes
%  `$\rho(a,q)\ge 0 \land o=out(q) \mathbf{\land \alpha(i)} \impl ...$'
%
%  \item for all guarantees of the form $\always\beta(i,o,o')$ add SMT constraints of the form:
%  $$
%  \bigwedge{q \in Q_P} \bigwedge{i \in \ActionsProc}: \ \
%  \alpha(i) \impl \beta(i,out(q),out(\delta_P(q,i)))
%  $$
%\end{enumerate}
%The first optimization is sound and complete, the second one: introduces incompleteness if used for model checking (since a given model may have unreachable state that violate the assertion), and seems to be complete for the synthesis.
%
%For AMBA specification in Figure~\ref{amba:fig:AMBAspecNewI} and \ref{amba:fig:AMBAspecNew0} this optimization means that only guarantees G2, G3, G9, G10.1, G11 require the standard flow via automata translation.
%
%Does this optimization help in the synthesis?
%Preliminary experiments (considering the first step of the decompositional synthesis of non-0 process) show:
%\begin{itemize}
%\item
%With the optimization the automaton for the negated specification has 24 states, without -- 42 states.
%
%\item
%The synthesis time with optimization is 16 minutes, without -- 57 minutes.
%Interesting to note that the optimized and non-optimized versions spent the same time (2 minutes) checking satisfiability of the last query (with the model size of 10), so the main difference is in checking unsatisfiable queries -- Z3 identifies unsatisfiability of optimized queries faster (14 vs. 53 minutes).
%A similar behavior happens for a version of the same specification with reduced lengths of bursts ($3/4\impl 2/3$): total times are 3/6 minutes, but the last query took 1m40s/30s for optimized/non-optimized version.
%
%\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\parbf{Results}
Synthesis times are in Tables~\ref{amba:tab:non-zero-process}~and~\ref{amba:tab:zero-process},
the model synthesized for non-0 process is in Figure~\ref{amba:fig:ith-model}.
The table has timings for the case when all optimizations described in this section are enabled --- it was not our goal to evaluate the optimizations separately, but to find a combination that works for the AMBA case study.

For the $0$-process we considered a simpler version with burst lengths reduced to 2/3 instead of the original 3/4 ticks.
With the original length, the synthesizer could not find a model within 2 hours (it hanged checking 11 state models, while the model has at least 12 states).

Without the decompositional approach,
the synthesizer could not find a model for for non-0 process of the AMBA specification within 5 hours.

\begin{table}[tb]
\footnotesize
\centering
\begin{minipage}[b]{0.45\textwidth}
\centering
% \setlength{\tabcolsep}{4pt}
\begin{tabular}{ l|cc }
Addit.\ assumptions                          & time & \#states \\
\hline
\rule{0pt}{3ex} \specialcellC{$\always \hlock$ \\ $\always \hburst=\hburstfour$}
                       & 16min.  & 10  \\
\hline
\rule{0pt}{2ex} \specialcellC{$\always \hburst=\hburstfour$} & 13sec.  & 13  \\
\hline
\rule{0pt}{2ex}
-- (Full Specification) & 1min.  & 14 \\
\end{tabular}
\caption{Results for non-$0$ process.\\~}
\label{amba:tab:non-zero-process}
\end{minipage}
%
\hspace{0.5cm}
%
\begin{minipage}[b]{0.45\textwidth}
\centering
% \setlength{\tabcolsep}{4pt}
\begin{tabular}{ l|cc }
Addit. assumptions & time & \#states \\
\hline
\rule{0pt}{3ex} \specialcellC{$\always \hlock$ \\ $\always \hburst=\hburstfour$}
                       & 3h.  & 11  \\
\hline
\rule{0pt}{2ex} \specialcellC{$\always \hburst=\hburstfour$} & 1min.  & 11  \\
\hline
\rule{0pt}{2ex}
-- (Full Specification) & 1m30s.  & 12 \\
\end{tabular}
\caption{Results for $0$-process \\ (bursts reduced: $3/4 \rightarrow 2/3$).}
\label{amba:tab:zero-process}
\end{minipage}
\end{table}

%\ak{`decide' is not faithfully simulated in token passing simulations?}

\subsection{Discussion}

We have shown that parameterized synthesis in token rings can be used to
solve benchmark problems of significant size, in particular the well-known
AMBA AHB specification that has been used as a synthesis benchmark for a long
time. To achieve this goal, we slightly extended the cutoff results that
parameterized synthesis is based on, and used a number of optimizations in
the translation of the specification and the synthesis procedure itself to
make the process feasible.

This is the first time that the AMBA case study, or any other realistic case,
has been solved by an automatic synthesis
procedure for the parameterized case. However, some of the steps in the
procedure are manual or use an ad-hoc solution for the specific problem at
hand, like the limited extension of cutoff results for global inputs, the
construction of suitable functions to convert local to global outputs, or the
decompositional synthesis for different grant schemes. Generalizing and
automating these approaches is a possible future work.

Our synthesized implementation is such
that the size of the parallel composition grows only linearly with the number of
components. Thus, for this case study our approach does not only solve the
problem of increasing synthesis time for a growing number of components, but
also the problem of implementations that need an exponential amount of memory
in the number of components. We pay for this small amount of memory with a
less-than-optimal reaction time, as processes have to wait for the token in
order to grant a request. This restriction could be remedied by extending the
parameterized synthesis approach to different system models, e.g., processes
that coordinate by guarded transitions~\cite{EmersonK03},
or communicate via broadcast messages~\cite{EsparzaFM99}.


\subsection{Truth tables}


\chapter{Basics: Types and Statements}
\begin{uclidlisting}[htbp]
    \lstinputlisting[language=uclid,style=uclidstyle]{../examples/tutorial/ex2.1-alu.ucl}
    \caption{Model of a simple ALU}
\label{ex:alu}
\end{uclidlisting}

This chapter will provide an overview of \uclid{}'s type system and modelling features. Let us start with Example~\ref{ex:alu}, a model of a simple arithmetic logic unit (ALU).

\section{Types in \uclid{}}

Types supported by \uclid{} are of the following kinds:
\begin{enumerate}
    \item \keyword{integer}: the type of mathematical integers.
    \item \keyword{boolean}: the Boolean type. This type has two values: \keyword{true} and \keyword{false}.
    \item \keyword{bv\textit{W}}: The family of bit-vector types parameterized by their width ($W$).
    \item \keyword{enum}: enumerated types.
    \item \keyword{float}: single precision floating point type with an $8$ bit exponent and $23$ bit signficiant and $1$ sign bit.
    \item Tuples and records.
    \item Array types.
    \item Uninterpreted types.
    \item Groups: A finite group of variables of a single type. Groups are not modifiable once declared, i.e., they are a static size.
\end{enumerate}

An enumerated type is used in line~2 of Example~\ref{ex:alu}. This declares a \textit{type synonym}: \codelike{cmd_t} is an alias for the enumerated type consisting of three values: \codelike{add}, \codelike{sub} and \codelike{mov_imm}. The input \codelike{cmd} is then declared to be of type \codelike{cmd_t} on line~6.

The input \codelike{valid} is of type \keyword{boolean}. Register indices \codelike{r1} and \codelike{r2} are bit-vectors of width 3 (\keyword{bv3}), while \codelike{immed}, \codelike{r1val} and \codelike{r2val} are bit-vectors of width 8 (\keyword{bv8}).

Line~3 declares a type synonym for a \keyword{record}. It declares \codelike{result_t} as consisting of two fields: a Boolean field \codelike{valid} and a bit-vector field \codelike{value}. The output \codelike{result} is declared to be of type \codelike{result_t} on Line~9.

The final point of interest in this example, type-wise, is line~10. The state variable \codelike{regs} is declared to be of type array: indices to the array are of type \codelike{bv3} and elements of the array are of type \codelike{bv8}. This is used to model an 8-entry register file, where each register is a bit-vector of width 8.

Uninterpreted functions can be declared in \uclid{} using the
\codelike{function} declaration. These functions are typed,
mapping a tuple of typed arguments to a return type.
For example, the following uninterpreted function models
the mapping of an instruction to an opcode in a simple
CPU model given later in this tutorial as Example~\ref{ex:cpu-cpu}.
\begin{lstlisting}[language=uclid,style=uclidstyle]
  function inst2op   (i : word_t) : op_t;
\end{lstlisting}

Two further types of functions without definitions can be declared in \uclid{} using the
\codelike{function} declaration.
First a synthesis function, declared using the synthesis keyword. Given a SyGuS solver using the command line option \codelike{-y "solver path"},
\uclid{} will construct
a SyGuS-IF file that synthesises bodies for any synthesis function in the \uclid{} file
\begin{lstlisting}[language=uclid,style=uclidstyle]
  synthesis function inst2op   (i : word_t) : op_t;
\end{lstlisting}

The second function extension is an oracle function, declared using the oracle keyword. An oracle function is declared along with the name of an external executable binary which implements the function. Given an Satisfiability Modulo Theories and Oracles (SMTO) solver using the command line option \codelike{-s "solver path"}, or a Synthesis Modulo Oracles (SyMO) solver using the command line option \codelike{-y "solver path"},
\uclid{} will construct
an SMTO/SyMO file for the verification/synthesis query including the oracle function. For more details on Synthesis and Satisfiability Modulo Oracles see \url{https://arxiv.org/abs/2107.13477}.
\begin{lstlisting}[language=uclid,style=uclidstyle]
  oracle function inst2op  [binaryName] (i : word_t) : op_t;
\end{lstlisting}

{\em Symbolic constants} can be declared using a
\codelike{const} declaration, as follows:
\begin{lstlisting}[language=uclid,style=uclidstyle]
  const w0 : word_t;
\end{lstlisting}

Finally, a useful feature of UCLID is groups. Groups are declared using the keyword \codelike{group}. All members of a group must have the same type, but they can be literals or variables that are already declared and in scope at the point of the group declaration.
\begin{lstlisting}[language=uclid,style=uclidstyle]
  group MyGroup :  integer = {0, 1, 2, 3};
  group AnotherGroup: integer = {a, b, c, 7};
\end{lstlisting}

\section{Statements in \uclid{}}

Computation in \uclid{} can be either procedural (sequential) or parallel (concurrent). Procedural computation is performed by defining a \keyword{procedure} (and in the \codelike{init} block) while parallel computation occurs in the \keyword{next} block.

\subsection{Parallel vs. Procedural Assignments}
Assignments inside procedures and the \codelike{init} block are called \textbf{procedural assignments} and must be of the form \codelike{variable = expression;} Assignments inside \codelike{next} block are \textbf{parallel assignments} and must be of the form \codelike{variable' = expression;}. Mathematically, parallel assignments compute the next state of the transition system described by the model.

An example showing the use of sequential assignments is the following:

%\begin{uclidlisting}[htbp]
\begin{lstlisting}[language=uclid,style=uclidstyle]
  x = 1;
  x = x + 2;
  x = x + 3;
\end{lstlisting}

	In this example, \codelike{x} is assigned sequentially.  Recall that these procedural assignments \emph{must} appear inside a procedure or in the \codelike{init} block. Executing these three statements will result \codelike{x} having the value 6.

    In contrast, the following sequence of parallel assignments is \textbf{not} allowed and will result in a compiler error.
\begin{lstlisting}[language=uclid,style=uclidstyle]
  // Error, will not compile.
  x' = 1;
  x' = x + 2;
  x' = x + 3;
\end{lstlisting}
Only a single parallel assignment to a state/output variable is allowed in a code block. Furthermore, since parallel assignments are computed in data-flow order, the order in which they are specified does not matter. This means that the following two snippets of code are equivalent:

\begin{lstlisting}[language=uclid,style=uclidstyle]
next {
  x' = x + 1;
  y' = x' + 1;
}
\end{lstlisting}

\begin{lstlisting}[language=uclid,style=uclidstyle]
next {
  y' = x' + 1;
  x' = x + 1;
}
\end{lstlisting}

    \uclid{} will determine that since \codelike{y'} depends on the value of \codelike{x'}, \codelike{x'} has to be computed first. This value is then used in the computation of \codelike{y'}. This is regardless of the order in which these assignments appear in the \codelike{next} block.

    Note also that the assignment to \codelike{x'} uses the value of the variable \codelike{x} \emph{at the beginning of the current step} of the transition system (i.e., the ``old'' value of \codelike{x}). In contrast the assignment to \codelike{y'} uses the ``new'' value of \codelike{x}, which is the value of x at the \emph{end} of this step of the transition system. It is important to think carefully about which version of a variable (\codelike{var} or \codelike{var'}) must be used in a particular assignment.
%\caption{Example of sequential assignment.}
%\label{ex:seq-assignment}
%\end{uclidlisting}

    %\lstinputlisting
\subsection{Procedures}
Example~\ref{ex:alu} demonstrates how sequential computation is used in concert with parallel computation. %The \codelike{procedure} \codelike{set\_init\_state} (lines~13--18) is used to initialize the values of the registers (state variable \codelike{regs}), the variables \codelike{cnt} and \codelike{result}. Since this procedure updates the module's state variables, a \codelike{modifies} clause (line~14) is required to explicitly specify that these updates are intended. The procedure is called on line~21 in the \codelike{init} block. Updates to a state variable not mentioned in a modifies clause will result in a compilation error.
In Example~\ref{ex:alu}, consider \codelike{procedure} \codelike{exec\_cmd} which executes a single ALU command and returns (line 23) a single value of type \codelike{result\_t}. The procedure is invoked on line~39 in the \codelike{next} block, and its return value is assigned to the output variable \codelike{result}. Note we are again using the notation \codelike{result'} to refer to parallel assignment. Since this procedure updates the \codelike{reg} state variable, a \keyword{modifies} clause must be used to declare this fact (line 24).

We emphasize that assignments inside procedures do not assign primed
variables. However, if a state variable is defined to be modified by
a procedure (mentioned in its modifies clause), then its next-state
value is the value that variable has upon return from that procedure.
Put another way, the post-state of the procedure determines the
next-state assignment of all state variables modified by it.
In our example, \codelike{procedure} \codelike{exec\_cmd} modifies
the state variable \codelike{regs}, and thus determines its next-state
value.

\subsection{Macro Definitions}

\uclid{} also supports the definition of macro expressions using
the \keyword{define} statement.
Line~20 of Example~\ref{ex:alu} illustrates this construct.
Macro definitions are useful, as in C, to define expressions
over arguments that are instantiated in multiple places, or
which help make the code more readable.

\subsection{For Loops}
The procedure \codelike{set\_init\_state} uses a \keyword{for} loop to initialize each value in the array \codelike{regs} to the bit-vector value 1.\footnote{\codelike{1bv8} here refers to the bit-vector value 1 of width 8.} The loop iterates over the values between 0 and 7 (both-inclusive).

The range over which a \keyword{for} loop iterates must be defined by two numeric literals.

Alternatively, the loop iterates may be defined by two macro definitions whose expressions are numeric literals. However, this requires the for statement to be declared with a typed iterator.

For example, if the following macro definitions are at the module level,
\begin{lstlisting}[language=uclid,style=uclidstyle]
  define begin() : bv3 = 0bv3;
  define end()   : bv3 = 7bv3;
\end{lstlisting}
we may alternatively write the for loop in \codelike{set\_init\_state} in the following way:
\begin{lstlisting}[language=uclid,style=uclidstyle]
  for (i : bv3) in (begin(), end()) { regs[i] = 1bv8; }
\end{lstlisting}

\subsection{If and Case Statements}
Also worth pointing out are the \keyword{if} statement that appears on line~27, and the \keyword{case} statement that appears on line~29. Syntax for \keyword{if} statements should be familiar.

\keyword{case} statements are delimited by \keyword{case} and \keyword{esac} and contain within them a list of boolean expressions and associated statement blocks. These expressions are evaluated in the order in which appear, and if any of them evaluate to \codelike{true}, the corresponding block is executed. If none of the case-expressions evaluate to \codelike{true}, nothing is executed. The keyword \keyword{default} can be used as a ``catch-all'' case like in C/C++.

\subsection{Expressions}

The syntax for expressions in \uclid{} is similar to languages like C/C++/Java. Index \codelike{i} of array \codelike{regs} is accessed using the syntax \codelike{regs[i]}. Field \codelike{value} in the record \codelike{result} is accessed as \codelike{result.value}.

\subsection{Quantifiers}

\uclid{} supports universal and existential quantifiers, which can be used in any place where a predicate would be permitted, for instance, in assumptions:
\begin{lstlisting}[language=uclid,style=uclidstyle]
  assume (forall (ri : robind_t) :: ops[ri] == no_op);
  assume (exists (i : integer) :: (i > 0) && (i < 11) && numbers[i] != some_int);
\end{lstlisting}
\uclid{} Also supports finite universal quantification, where we universally quantify over a finite group type. These quantifiers are expanded to a conjunction before being passed to the underlying SMT solver.
\begin{lstlisting}[language=uclid,style=uclidstyle]
  group myGroup : integer = {0, 1, 2, 3};
  property test : finite_forall (int : integer) in myGroup :: int < 4;
\end{lstlisting}

%-----------------------------------------------------
\section{An Illustrative Example}

This section briefly describes the execution semantics of Example~\ref{ex:alu}.

\subsection{Initialization}
Execution of the model in Example~\ref{ex:alu} starts with the \keyword{init} block. This block invokes \codelike{set\_init\_state} and assigns initial values to \codelike{regs}, \codelike{cnt} and \codelike{result.value}. The other variables (e.g. \codelike{r1val} and \codelike{r2val}) are not assigned to in the \keyword{init} block and will be initialized non-deterministically.

\subsection{Next State Computation}
The next state of each state variable in the model is computed according to the \keyword{next} block. Any variables not assigned to in the \keyword{next} block retain their ``old'' values.

The \keyword{input} variables of the model are assigned (possibly different) non-deterministic values for each step of the transition system. These values can be controlled by using assumptions. Indeed, the model uses the two assumptions on lines 43--44 to constrain the input to the ALU to always be an \codelike{add} operation, where both operands refer to register index 0.

\subsection{Verification}
As in Example~\ref{ex:fib-model}, the verification script in Example~\ref{ex:alu} unrolls the transition system for 5 steps and checks if the \keyword{invariant} on line~45 is violated in any of these steps.

\subsection{Running \uclid{}}

Running \uclid{} on Example~\ref{ex:alu} produces the following output.

\begin{Verbatim}[frame=single, samepage=true]
$ uclid examples/tutorial/ex2.1-alu.ucl
Successfully parsed 1 and instantiated 1 module(s).
6 assertions passed.
0 assertions failed.
0 assertions indeterminate.
Finished execution for module: main.
\end{Verbatim}

\uclid{} is able to prove that the \keyword{invariant} on line~45 holds for all states reachable within 5 steps of the initial state, under the assumptions specified in lines~43--44.


\section{Flow over a weir}

In this problem water runs up a broad sloping floodplain, which is split into 2 by a thin weir (riverwall). The landward side of the floodplain is initially dry but becomes wet as the water overtops the weir. We use the change in mass on the landward side of the weir to compute the flux over the weir according to \anuga{}, and compare with direct computation of the weir equation.

\anuga{} allows computation of flow over a weir using the riverwall structure (supported for Discontinuous Elevation Algorithms only as of 01/07/2014). The default method adjusts the edge flux over the weir to satisfy a basic weir relation with Villemonte's submergence correction. However at high submergence ratios, or when the depth of flow over the weir is large compared with the weir height, \anuga{} smoothly reverts to the shallow water solution (because the weir relations are not sensible in these situations). This is required so that e.g. a weir of 1cm height covered by flow of 1m is basically no different from the shallow water solution - weir relations by themselves will not achieve this. See the documentation of riverwalls for more information.

\subsection{Results}

The following figures show the discharge over the weir computed with ANUGA, and with the simple weir equation with Villemonte's submergence correction. For reference we also show other weir equations. Before 1100s the discharge is quite similar for all methods. There is still a small difference between \anuga{} and the simple-weir equation because the latter is computed only using 2 water level gauges (rather than by integration as in ANUGA), and because of the discretization of the model geometry.

At later times the water elevations on the landward and seaward sides of the wall are very similar, and submergence relations play a greater role in influencing the weir flow. The flow computed by \anuga{} reduces much more rapidly than do the other methods. This is due to the blending with the shallow water solution in \anuga{} at high submergence ratios (greater than s1). The user can adjust this behaviour in ANUGA by changing the s1, s2, h1, h2 parameters (see riverwall documentation). The Qfactor parameter can also be adjusted to increase/decrease the ideal weir flow.

We have not found much information on how other models treat the transition from weir to shallow water fluxes. However, according to the HecRAS 4.1 Technical Reference manual, HecRas switches to the energy equation when the submergence ratio is 0.95 (default), which corresponds to our default choice of s2=0.95. Our default submergence ratio at which blending begins is s1=0.9.

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{Stage.png}
\end{center}
\caption{Stage (above the riverwall crest) at 2 points either side of the riverwall}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[width=0.9\textwidth]{Fluxes.png}
\end{center}
\caption{Fluxes over the riverwall, computed with a range of methods.}
\end{figure}

\endinput


\section{Introduction}

Compiler implementation is a complex and expensive activity~\citep{Cooper2012}.
For this reason, there has been significant interest in using machine learning
to automate various compiler tasks~\citep{Allamanis2017a}. Most works have
restricted their attention to selecting compiler heuristics or making
optimization decisions~\citep{leather2020mlinc}. Whether learned or engineered
by human experts, these decisions naturally require reasoning about the program
and its behavior. Human experts most often rely upon \textit{data flow}
analyses~\citep{Kildall1973,Kam1976}. These are algorithms on abstract
interpretations of the program, propagating information of interest through the
program's control-flow graph until a fixed point is reached~\citep{Kam1977}. Two
examples out of many data flow analyses are: \emph{liveness} -- determining when
resources become dead (unused) and may be reclaimed; and \emph{available
expressions} -- discovering which expressions have been computed on all paths to
points in the program. Prior machine learning works, on the other hand, have
typically represented the entirety of the program's behavior as a fixed-length,
statically computed feature vector~\citep{Ashouri2018}. Typical feature values
might be the number of instructions in a loop or the dependency depth. The
demonstrable weakness of these techniques is that they are trivially confused by
the addition of dead code, which changes their feature vectors without changing
the program's behavior or its response to optimizations. Such learning
algorithms are unable to learn their own abstract interpretations of the program
and so cannot avoid these pitfalls or more subtle versions
thereof~\citep{Barchi2019a}.

\begin{figure}[t]
 \centering %
 \includegraphics[width=.85\linewidth]{images/overview}
 \caption{%
    Our proposed approach for compiler analyses driven by graph-based deep
    learning.%
 }%
 \label{figure:overview}%
\end{figure}

Recently, there have been attempts to develop representations that allow
finer-grain program reasoning. Many, however, are limited both by how inputs are
represented as well as how inputs are processed. Representations based on source
code and its direct artifacts (e.g.,
AST)~\citep{Alon2018a,Yin2018,Haj-Ali2019,cummins2017synthesizing} put
unnecessary emphasis on naming and stylistic choices that may not correlate with
the functionality of the code (e.g., Fig.~\ref{subfig:code2vec}). Approaches
based on intermediate representations
(IR)~\citep{Ben-nun2018,Mirhoseini2017,Brauckmann2020} remove such noise but
fail to capture information about the program that is important for analysis
(e.g.,  Fig.~\ref{subfig:cdfg} variables, Fig.~\ref{subfig:inst2vec}
commutativity). In both cases, models are expected to reason about the flow of
information in programs using representations that do not directly encode this
information. Clearly, a program representation is needed that enables machine
learning algorithms to reason about the execution of a program by developing its
own data flow analyses.

Since current approaches are ill-suited to program-wide data flow analysis, we
propose overcoming some of their limitations by making the program's control,
data, and call dependencies a central part of the program's representation
\emph{and} a primary consideration when processing it. We achieve this by seeing
the program as a graph in which individual statements are connected to other
statements through relational dependencies. Each statement in the program is
understood only in the context of the statements interacting with it. Through
relational reasoning~\citep{Battaglia2018}, a latent representation of each
statement is learned that is a function of not just the statement itself, but
also of the (latent) representations of its graph neighborhood. Notably, this
formulation has a striking similarity to the IRs used by compilers, and the
iterative propagation of information resembles the \emph{transfer functions} and
\emph{meet operators} in traditional data flow analyses~\citep{Kildall1973}.

Recently proposed techniques for learning over graphs have shown promise in a
number of domains~\citep{Li2018a}. With a suitable representation and
graph-based model, we extend these approaches to the domain of compiler
analysis, enabling downstream tasks built on top of such graph models to
natively incorporate reasoning about data flow into their decision making. This
improves performance on downstream tasks without requiring additional features,
although challenges with respect to generalization to large programs at
test-time remain and are discussed in detail. Figure~\ref{figure:overview}
illustrates our approach. We make the following contributions:

\begin{itemize}
\item
We propose a portable, language-independent graph representation of programs
derived from compiler IRs.
\programl\footnote{\url{https://github.com/ChrisCummins/ProGraML}}
simultaneously captures whole-program control-, data-, and call relations
between instructions and operands as well as their order and data types.
\programl is a compiler-agnostic design for use at all points in the
optimization pipeline; we provide implementations for LLVM and XLA IRs.
\item
We introduce a benchmark dataset that poses a suite of established compiler
analysis tasks as supervised machine learning problems.
\textsc{DeepDataFlow}~\cite{chris_cummins_2020_4247595} comprises five tasks
that require, in combination, the ability to model: control- and data-flow,
function boundaries, instruction types, and the type and order of operands over
complex programs. \textsc{DeepDataFlow} is constructed from 461k real-world
program IRs covering a diverse range of domains and source languages, totaling
8.5 billion data flow analysis classification labels.
\item
We adapt Gated-Graph Neural Networks (GGNN) to the \programl representation. We
show that, within a bounded problem size, our approach achieves $\ge 0.939$
F$_1$ score on all analysis tasks, a significant improvement over
state-of-the-art representations. We set a new state-of-the-art on two
downstream optimization tasks for which data flow analyses are important.
\end{itemize}


\section{Reflection security}\label{sec:refsec}

We now define the theoretical model for reflection attacks. We propose a
cryptographic game for a simulation-based model and present proofs of insecurity
under the necessary assumptions. We then move on to describe the properties of
the various elements in the game and describe existing compression attacks,
including Rupture, as instances of the theoretical model we introduce.

\subsection{The reflection game}\label{subsec:refsecgame}

Let $\mathcal{SE} = (Gen, \mathcal{E}, \mathcal{D})$ be a private-key encryption
scheme, $\mathcal{A}$ be an adversary and $\mathcal{S}$ be a simulator. Also let
some function $f(\cdot, \cdot)$, some function
$g$ of the plaintext, some function $Com(\cdot)$, and a distribution of secrets
$\mathcal{M}$. We call $f$ a rendering function and require that it
is polynomially computable and reversible. We call $Com$ the compression
function and require that it is a deterministic polynomially computable and
reversible bijection.

The game $\text{Game}_{\text{REF-SEC}}^{\mathcal{SE},\mathcal{A}}(\lambda)$ is
parameterized with the security parameter $\lambda$. The challenger produces a
$\lambda$-bit key $k \leftarrow Gen(1^\lambda)$ and initially
chooses a secret $s \leftarrow \mathcal{M}$.

The adversary is then allowed to run and make arbitrary calls to a reflection
oracle. The oracle is parameterized by $s$, the secret unknown to the adversary.
For the reflection oracle call, the adversary chooses a reflection string $r$
and sends it to the oracle. The oracle computes $m = f(s, r)$. Subsequently $m$
is encrypted as $c = \mathcal{E}_\kappa(m)$, and $c$ is sent back to the
adversary.

When the adversary decides to complete the game, they output a guess $y$. The
adversary is successful if $g(s) = y$. In the case of Rupture, a successful
attack results in the decryption of a single character that extends the known
prefix of secret $s$.

Formally, let the secret key adversarial game be defined as follows:

\begin{algorithm}
    \caption{The reflection security game.}\label{alg:refsec-game}
    \begin{algorithmic}
        \Function {$\text{Game}_{\text{REF-SEC}}^{\mathcal{SE},\mathcal{A}}$}{$\lambda$}:
            \State $k \leftarrow Gen(1^\lambda)$
            \State $s \leftarrow \mathcal{M}$
            \State $y \leftarrow \mathcal{A}^{\text{Reflect}^{k}_s(r)}(1^\lambda)$
            \If {$y = g(s)$}
                \State \Return $1$
            \Else
                \State \Return $0$
            \EndIf
        \EndFunction
    \end{algorithmic}
\end{algorithm}

Where the reflection oracle provided to the adversary is as follows:

\begin{lstlisting}[texcl,mathescape,basicstyle=\small]
\end{lstlisting}

\begin{algorithm}
    \caption{The reflection oracle.}\label{alg:ref-oracle}
    \begin{algorithmic}
        \Function {$\text{Reflect}^{k}_s$}{$r$}:
            \State $m = f(s, r)$
            \State $c \leftarrow \mathcal{E}_{k}(m)$
            \State \Return $c$
        \EndFunction
    \end{algorithmic}
\end{algorithm}

Let the simulator game be defined as follows:

\begin{algorithm}
    \caption{The simulated reflection security game.}\label{alg:sim-refsec-game}
    \begin{algorithmic}
        \Function {$\text{Game}_{\text{REF-SIM}}^{\mathcal{SE},\mathcal{S}}$}{$\lambda$}:
            \State $s \leftarrow \mathcal{M}$
            \State $s' = 0^{|s|}$
            \State $y \leftarrow \mathcal{S}^{\text{Reflect}^{k}_s(r)}(1^\lambda, {|Com(s)|})$
            \If {$y = g(s)$}
                \State \Return $1$
            \Else
                \State \Return $0$
            \EndIf
        \EndFunction
    \end{algorithmic}
\end{algorithm}

The reflection oracle provided to the simulator is as follows:

\begin{algorithm}
    \caption{The simulator's reflection oracle.}\label{alg:sim-ref-oracle}
    \begin{algorithmic}
        \Function {$\text{Reflect}^{k}_{s'}$}{$r$}:
            \State $m = f(s', r)$
            \State $c \leftarrow \mathcal{E}_{k}(m)$
            \State \Return $c$
        \EndFunction
    \end{algorithmic}
\end{algorithm}

Both the adversary and the simulator are given access to an encryption and decryption oracle, but
are not allowed to call the decryption oracle with the reflection oracle output.

The reflection game is depicted in Figure \ref{fig:refgame}.

    \begin{figure}[thpb]
        \centering
            \includegraphics[width=0.48\textwidth]{figures/reflection_game.png}
        \caption{Reflection Game}
        \label{fig:refgame}
    \end{figure}

\subsection{Adversarial advantage}\label{subsec:refsecadv}

Let us now define the advantage of the adversary against a simulator:
\begin{align*}
    \text{Adv}_{\mathcal{SE}, \mathcal{A}, \mathcal{S}}&(\lambda) &\defeq\\
    |\Pr[\text{Game}_{\text{REF-SEC}}^{\mathcal{SE},\mathcal{A}}(\lambda) = 1] &-\\
    \Pr[\text{Game}_{\text{REF-SIM}}^{\mathcal{SE},\mathcal{S}}(\lambda) = 1]| &
\end{align*}

\subsection{Adaptive reflection security}\label{subsec:adaptiverefsec}

Given a rendering function $f(\cdot, \cdot)$, a private-key encryption
scheme $\mathcal{SE}$ is \textit{reflection-secure} if:
\begin{align*}
    \forall \mathcal{M}:
    \forall g:
    \forall PPT \mathcal{A}:
    \exists PPT \mathcal{S}:\\
    \text{Adv}_{\mathcal{SE}, \mathcal{A}, \mathcal{S}}(\lambda) = negl(\lambda)
\end{align*}

\begin{lemma}[Semantic security]
    Let $(Gen, K, D)$ be a length-preserving reflection-secure encryption
    scheme. Then it is also semantically secure.
\end{lemma}

For a full proof, see Appendix A.


\documentclass[a4paper,10pt]{article}

\usepackage[utf8]{inputenc}
\usepackage{epsfig}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{array}
\usepackage{float}
\usepackage{ctable}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{amsfonts}
\usepackage{cite}
\usepackage[]{algorithm2e}

\usepackage{fullpage}

\usepackage{listings}
\lstset{breaklines=true}

\def\th{^{th}}
\newcommand{\bs}[1]{\boldsymbol{#1}}
\newcommand{\pop}{\mathcal{P}}

\def\th{^{th}}
\def\pd{\frac{\partial}{\partial q_i^k}}
\newcommand{\mcf}[1]{p\Big( \norm{\bs{#1}^k}_1 \Big)}
\def\pdy{\frac{\partial}{\partial y_i^k}}
\def\pdq{\frac{\partial}{\partial q_i^k}}
\newcommand{\diag}{\mathop{\mathrm{diag}}}
%\newcommand{\bs}[1]{\boldsymbol{#1}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\newcommand{\normb}[1]{\left\lVert \bs{#1} \right\rVert_1}

\usepackage{autonum}
\usepackage{placeins}

\title{Population Dynamics Toolbox}
\author{Carlos Barreto}

\begin{document}
\lstset{language=Matlab}
\maketitle

\begin{abstract}
 The \emph{population dynamics toolbox} (PDToolbox) contains a set of functions to implement evolutionary dynamics with multiple populations. We consider both small and large populations. For finite populations, we implement some revision protocols to model random interactions between agents. On the other hand, the evolution of a society with large populations is approximated by dynamical equations.

 This toolbox is designed to facilitate the implementation of any game with different evolutionary dynamics or revision protocols.  In particular, our attempt is to make an efficient implementation of the algorithms to compute the dynamical evolution of the society. Also, the toolbox counts with some functions to plot the state of the system and the evolution of each strategy.

 In Section \ref{sec:introduction} we start by introducing the notation used along the paper and we present some ideas of population games which lead to the \emph{mean dynamics} equation. In Section \ref{sec:protocols} we introduce some well known revision protocols and evolutionary dynamics. In Section \ref{sec:implementation} we introduce some details of the implementation of games with the toolbox and show an example of the implementation of the rock-paper-scissors game.
 Sections \ref{sec:combined} and \ref{sec:multi-pop} contain examples of the implementation of \emph{combined dynamics} and \emph{multi-population games}.

\end{abstract}

\tableofcontents

\input{introduction}

\input{evolutionary_dynamics}

\input{implementation}

\input{combined_dynamics}

\input{multipopulation}

\input{maximization}

\input{example_dr}

 \FloatBarrier
\section{Running Time Analysis}\label{sec:running_time}

In this section we investigate the running time of the evolutionary dynamics as a function of either the number of populations or the number of strategies per population. We use the demand response example from Section \ref{sec:dr_example} to make the experiments.
Our interest is to observe the time that takes to simulate the evolution of the dynamics during 10 seconds.

\subsection{Running Time as a Function of the Number of Populations}

Note that the time complexity of the algorithms, with respect to number of populations $P$, is $O(P \cdot T_f(n,P))$. The fitness function in the example satisfies $T_f(n,P)=O(nP)$. Hence, the complexity of the dynamics with respect to $P$ is $O(P^2)$.

Although the dynamics have the same time complexity, the running time of the dynamics has substantial differences.
Fig. \ref{fig:run_time_pop} shows the running time of the game for multiple number of populations. In this case each population has $24$ strategies and all populations have the same set of fitness functions.
From the simulations we observe that the
%running time increases roughly linearly for all dynamics. The
fastest simulations are made with replicator dynamics.
Moreover, there is a large difference between the running time of Replicator and the other dynamics.
%On the other hand, Smith and Logit dynamics have almost the same running time.
%
One reason is that the ODE solver run less iterations with Replicator dynamics, and consequently, it makes less calls to the dynamic's algorithm.

\subsection{Running Time as a Function of the Number of Strategies}

Fig. \ref{fig:run_time_str} shows the running time of the game for different number of strategies per population. In this case we  define $25$ populations. Note that the time complexity of the dynamics with respect to the number of strategies is $O(n)$ (or $O(n \log n)$ in the case of the Smith dynamics). In this case Replicator has the lowest running time, while smith have the largest running time. The large difference between Smith and the other dynamics is that the ODE solver makes much more iterations.

Note that, except for Smith dynamics, the running time is lower when we implement games with large number of strategies. Hence, the maximization problem in Section \ref{sec:maximization} might be solved faster using the single population case.

\begin{figure}
        \centering
        \begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=1\textwidth]{./images/running_time_pop.eps}
                \caption{Run time for different population numbers and constant number of strategies.}
                \label{fig:run_time_pop}
        \end{subfigure}
        ~
        \begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=1\textwidth]{./images/running_time_str.eps}
		%\includegraphics[width=1\textwidth]{./images/running_time_strategies.eps}
                \caption{Run time for different number  of strategies and constant number of populations.}
                \label{fig:run_time_str}
        \end{subfigure}
        \caption{Run time of four evolutionary dynamics: `rd', `bnn', `smith', and `logit'.}
        \label{fig:run_time}
\end{figure}

%Which case gives the largest simulation time?

\subsection{Running Time of Smith Dynamics }

In the toolbox we include two algorithms for the Smith dynamics. One of them relies on matrix multiplications and has running time $O(P(T_f(n,P) + n^2))$ (`smith.m'), while the other has time complexity $O(P(T_f(n,P) + n \log n))$ (`smith\_b.m'). Even though the time complexity of the first algorithm is higher, its simulation are faster under certain conditions. The reason is that Matlab is optimized to work with matrices. From the experiments we see that  `smith\_b.m' is faster only for large number strategies (see Fig. \ref{fig:smith_run_time}).

\begin{figure}
        \centering
        \begin{subfigure}[b]{0.45\textwidth}
                \includegraphics[width=\textwidth]{./images/running_time_smith_pop.eps}
                \caption{Run time for different population numbers and constant number of strategies.}
                \label{fig:smith_run_time_pop}
        \end{subfigure}%
        ~
        \begin{subfigure}[b]{0.45\textwidth}
                \includegraphics[width=\textwidth]{./images/running_time_smith_str.eps}
                \caption{Run time for different number  of strategies and constant number of populations.}
                \label{fig:smith_run_time_str}
        \end{subfigure}%

        \caption{Run time of two implementations of the Smith dynamics, namely `smith' and `smith\_b'.}
        \label{fig:smith_run_time}
\end{figure}

 \FloatBarrier

\iffalse

 \FloatBarrier

\section{Designing games}

In the previous section we show some examples of strategical situations that can be analyzed with game theory. In these cases, the structure of the game is given by the problem. However, we can modify the fitness function of each player in order to solve an optimization problem.

For example, let us consider the following optimization problem:

\begin{equation}\label{eq:opt_problem}
\begin{aligned}
& \underset{x}{\text{maximize}}
& & \sum_{i=1}^N U_i(x_i)  - C(|x|)\\
& \text{subject to}
& & 0 \leq q_i \geq m,  i =\{1,\ldots, N\}.
\end{aligned}
\end{equation}

This can be seen as a problem of allocating a finite resource to maximize a utility function.

Note that there are $N$ agents with that give a valuation $v_i(x_i)$ to the resource $x_i$.

However, the cost of assigning the resource is $C(|x|)$ .

The cost of assigning the resource might be distributed among the population.

Let us consider the following example:

\begin{equation}
U_i(x_i) =   \alpha_i  log(1+x_i)
\end{equation}

\begin{equation}
C(z) = \beta z^2 + b z
\end{equation}

define fitness functions as

\begin{equation}
f_i(x_1, x_2) = \frac{\alpha_i}{1+x_i} - 2 \beta |x| - b
\end{equation}

\fi

--

\bibliographystyle{plain}
\bibliography{references}

\end{document}

\iffalse

\begin{figure}[htb]
 \includegraphics[1\textwidth]{./images/}
 \caption{}
 \label{fig:}
\end{figure}

\fi


\section{Evaluation}
\label{sec:evaluation}

Let us now see how well on-demand evaluation works in practice.
We begin by empirically studying the bias and variance of the joint estimator proposed in \refsec{method} and find it is able to correct for pooling bias while significantly reducing variance in comparison with the simple estimator.
We then demonstrate that on-demand evaluation can serve as a practical replacement for the TAC KBP evaluations by piloting a new evaluation service we have developed to evaluate three distinct systems on TAC KBP 2016 document corpus.
%We find that we are able to obtain results of  quality in a cost effective manner.

\subsection{Bias and variance of the on-demand evaluation.}
Once again, we use the labeled system predictions from the TAC KBP 2015 evaluation and treat them as an exhaustively annotated dataset.
To evaluate the pooling methodology we construct an evaluation dataset using
instances found by human annotators and labeled instances pooled from 9
randomly chosen teams (i.e.\ half the total number of participating teams), and
use this dataset to evaluate the remaining 9 teams.
On average, the pooled evaluation dataset contains between 5,000 and 6,000 labeled instances and evaluates 34 different systems (since each team may have submitted multiple systems).
Next, we evaluated sets of 9 randomly chosen teams with our proposed simple and joint estimators using a total of 5,000 samples:
about 150 of these samples are drawn from $\sY$, i.e.\ the full TAC KBP 2015 evaluation data, and 150 samples from each of the systems being evaluated.

We repeat the above simulated experiment 500 times and compare the estimated precision and recall with their true values (\reffig{simulation}).
The simulations once again highlights that the pooled methodology is biased, while the simple and joint estimators are not.
Furthermore, the joint estimators significantly reduce variance relative to the simple estimators:
the median 90\% confidence intervals reduce from 0.14 to 0.06 precision and from 0.14 to 0.08 for recall.

\subsection{Number of samples required by on-demand evaluation}
Separately, we evaluate the efficacy of the adaptive sample selection method described in \refsec{joint} through another simulated experiment.
In each trial of this experiment, we evaluate the top 40 systems in random order.
As each subsequent system is evaluated, the number of samples to pick from the system is chosen to meet a target variance and added to the current pool of labeled instances.
To make the experiment more interpretable, we choose the target variance to correspond with the estimated variance of having 500 samples.
\reffig{simulation} plots the results of the experiment.
The number of samples required to estimate systems quickly drops off from the benchmark of 500 samples as the pool of labeled instances covers more systems.
This experiment shows that on-demand evaluation using joint estimation can scale up to an order of magnitude more submissions  than a simple estimator for the same cost.

\subsection{A mock evaluation for TAC KBP 2016}
We have implemented the on-demand evaluation framework described here as an evaluation service to which researchers can submit their own system predictions.
As a pilot of the service, we evaluated three relation extraction systems that also participated in the official 2016 TAC KBP competition.
Each system uses Stanford CoreNLP~\citep{manning2014stanford} to identify entities, the Illinois Wikifier~\citep{ratinov2011local} to perform entity linking and a combination of a rule-based system (P), a logistic classifier (L), and a neural network classifier (N) for relation extraction.
%distinct relation extraction systems (a rule-based system, a logistic classifier, and a neural network classifier) on 15,000 Newswire documents from 2016 TAC KBP evaluation.
We used 15,000 Newswire documents from the 2016 TAC KBP evaluation as our document corpus.
In total, 100 documents were exhaustively annotated for about \$2,000 and 500 instances from each system were labeled for about \$150 each.
Evaluating all three system only took about 2 hours.

%In total, 100 documents were exhaustively annotated for about \$2,000, and 1,000 instances from each system were labeled for about \$300 each, with 500 sampled to estimate macro-averaged relation scores and 500 sampled to estimate macro-averaged entity scores.
\reffig{evaluation-results} reports scores obtained through on-demand evaluation of these systems as well as their corresponding official TAC evaluation scores.
While the relative ordering of systems between the two evaluations is the same, we note that precision and recall as measured through on-demand evaluation are respectively higher and lower than the official scores.
This is to be expected because on-demand evaluation measures precision using each systems output as opposed to an externally defined set of evaluation entities.
Likewise, recall is measured using exhaustive annotations of relations within the corpus instead of annotations from pooled output in the official evaluation.

%We note that the rule-based system does better on \fone{} because it has significantly higher precision that the other systems, while the RNN system has the highest recall among the three systems.
%We also include TAC KBP evaluation scores for the official submissions that most closely represent these systems.
%The TAC KBP submissions are actually a combination of systems and include filtering and post-processing as required under the evaluation guidelines, e.g.\ submitting a unique instance for each relation.
%While it is hard to directly compare the two evaluation scores, we note that the our evaluation evaluates the rule-based system significantly higher:
%, which includes submitting a unique instance for each relation,

%\ac{@PL: Should we talk about who pays what on the platform, etc.? A reviewer mentioned this, but I don't think it's appropriate for the paper.}
% PL: yeah, leave it out