\documentclass[a4paper]{article}
\usepackage[width=15cm,height=25.7cm]{geometry}
\usepackage{xspace}
\usepackage{color}
\usepackage[leqno]{amsmath}
\usepackage{amssymb}
\usepackage{array}
\usepackage{tabularx}
\usepackage{booktabs}
\usepackage{url}
\usepackage{hyperref}
\usepackage{breakurl}
\usepackage{graphicx}
\usepackage{subfigure}
\def\url@leostyle{%
  \@ifundefined{selectfont}{\def\UrlFont{\small\sf}}{\def\UrlFont{\small\sf}}}
\makeatother
\urlstyle{leo}
\usepackage[normalem]{ulem}
\input{macros}

\title{A Cat tutorial: HSA Models}
\date{}
\author{Jade Alglave and Luc Maranget}
\begin{document}
\maketitle
\part{A reading of the HSA document}

\section{\label{coherence}Coherence}

For a given location~$L$,
the coherence order $\cohl{L}$ is defined as a total order on all loads and
stores to location~$L$. The ``single Coherent Order'' $\coh$ is
the union of all these orders for all locations.
In appendix~\ref{coh} we describe how to generate the set of all possible
$\coh$ ``orders'' in the \cat{} language.
For now, let us assume a variable \texttt{allCoh}
whose value is the set of all possible \coh{} orders.

The instruction \texttt{with $v$ from $S$} will, for each element~$e$ in $S$,
execute the rest of the model in an extended environment that binds
$v$ to~$e$. In our case of \coh{}, we write:
\begin{verbatim}
with coh from allCoh
\end{verbatim}
In effect, the construct performs a further enumeration of candidate
executions: the initial candidate is extended with one \coh{} order
bound to the variable~\texttt{coh}.

We can then  check the consistency of $\coh$ and~$\po$:
\begin{verbatim}
call consistent(coh,po) as CohPoCons
\end{verbatim}
See appendix~\ref{procedure} for the definition of the consistency check as
a procedure.

The other coherence check is the ``value of a load'' check, phrased
as: ``\emph{a load [\ldots] will always observe the most recent store in the coherent order of location~$L$}''.
Given the existence of initial writes, which are minimal amongst writes to
the same location in \coh{}, the above condition defines a relation from
writes to reads to the same location. This relation, \texttt{mincohWR},
is implemented in the \cat{} language as follows:
\begin{verbatim}
let cohWR = coh & (W * R)
let cohWW = coh & (W * W)
let mincohWR = cohWR \ (cohWW; cohWR)
\end{verbatim}
Where ``\verb+&+'' is intersection, ``\verb+*+'' is cartesian product
and ``\verb+\+'' is (set or relation) difference.
Then it remains to check that $\rf$ equals \texttt{mincohWR}.
We proceed by checking double inclusion:
\begin{verbatim}
(* Relation a includes relation b, ie b(x,y) => a(x,y) *)
procedure includes(a,b) = empty b \ a end
procedure equals(a,b) =
  call includes(a,b)
  call includes(b,a)
end

call equals(rf,mincohWR) as LoadCons
\end{verbatim}
Notice another usage of the difference operator on relations.

Finally, we may view the atomicity of RMW constructs as a coherence property
\marginpar{This property does not appear formally in the document?}
This property states that there cannot be any write in-between the read
and the write performed by a RWM operation.

There are two possible semantics of RMW ``operations'' as ``events''.
\begin{enumerate}
\item
A C-like semantics would consider RMW events that are both read and write
events. We would then state the atomicity of RMW's as follows:
\begin{verbatim}
let RMW = R & W
let rmw = id & (RMW * RMW)
let cohRW = coh & (R * W)
empty rmw & (cohRW;cohWW) as RmwCons
\end{verbatim}
Where \texttt{id} is the pre-defined identity relation on events.
As a consequence the relation \texttt{rmwid} is the identity restricted
to RMW events.
However, this condition is useless. Indeed, by construction,
\texttt{coh} is a total order and is thus irreflexive.
As a consequence, \texttt{cohRW;cohWW} which is included in \coh{}
cannot intersect the identity.
\item Or, and we prefer this solution, a RMW operation is represented
by two events a read and a write, which are related by a pre-defined
\texttt{rmw} relation.
We would then state the atomicity of RMW's as follows:
\begin{verbatim}
empty rmw & (cohRW;cohWW) as RmwCons
\end{verbatim}
In that case, we cannot get rid of the \texttt{RmwCons} check,
as \texttt{rmw} now relate different events.
\end{enumerate}

\section{No values out-of-thin-air, dependencies}
The document defines the ``local dependence order'' $\ldo$ informally
as the union of data, address and control dependencies (Sec 3.8).
Our simulator \herd{} provides pre-defined relations for those three
``local dependencies'' relations.
Hence we write
\begin{verbatim}
let ldo = data | addr | ctrl
\end{verbatim}
We notice that control dependencies to reads are part of \ldo.

The document  then define the ``global dependence order'' as
the irreflexive transitive closure of \ldo{} union \coh{},
which we would write \verb!let gdo = (ldo|coh)+!.
The document then states
``\emph{By rule, there cannot be a cycle in in \gdo{}}'',
which we interpret as a requirement: \gdo{} must be acyclic.
The intent of the global dependence order clearly is to rule out
values out of thin air, as illustrated by
the typical example ``\ltst{lb+ldos}'':
\begin{center}\fmt{lb+ldos}\end{center}

However, including the complete~\coh{} in the definition of~\gdo{} looks too
strong: the requirement will then for instance forbid \ltst{wrc+ldos}.
\begin{center}\hspace*{-2cm}\fmt{wrc+ldos}\end{center}

We  think, given the intent to forbid values out of thin air,
that \gdo{} should include a fragment of \coh{}: the \rf{} relation
when source and target belong to different units:
\begin{verbatim}
let rfe = rf & ext
let gdo = ldo | rfe
acyclic gdo as GdoCons
\end{verbatim}
In the code above, we first define the \rfe{} relation (\rf{} from different
units) by intersecting \rf{} and the pre-defined \ext{} relation that relates
all events from different units.
Also notice that we do not apply transitive closure to \gdo{}, as this
does not impact the acyclicity of a relation. Furthermore,
the \gdo{} relation will no longer appear in the remainder of the model.

\section{\label{sso}Scoped synchronization order}
The documents defines a hierarchy of five scope levels:
system, agent, work-group, wave and work-item,\footnote{We use some abbreviations:``wg'' for work-group and ``wi'' for and work-item.}
here listed from top to bottom.
The \herd{} simulator handles scopes by the means
of various objects: tags, scope relations,
and set of annotated events.
Those are defined by reading a prelude ``bell'' file,
supplied to \herd{} with the \texttt{-bell} option.
Appendix~\ref{bell} details the bell file for HSA.

Each test features a ``scope tree'' definition that specifies how
the test  units are organized according to the scope hierarchy.
Consider for instance the test \ltst{isa2}:
{\small
\begin{verbatim}
Bell isa2
{ 2:r1=-1; }
 P0                     | P1                         | P2                          ;
 w[] x 53               | r[atomic,scacq,agent] r0 y | r[atomic,scacq,system] r0 z ;
 w[atomic,screl,wg] y 1 | bne r0, 1, Exit1           | bne r0, 1, Exit2            ;
                        | w[atomic,screl,system] z 1 | r[] r1 x                    ;
                        | Exit1:                     | Exit2:                      ;
scopes: (agent (wg 0 1) (wg 2))
exists (1:r0=1 /\ 2:r0=1 /\ 2:r1=0)
\end{verbatim}
}
This test has three units, \myth{0}, \myth{1} and~\myth{2}.
According to the scope tree \verb+(agent (wg 0 1) (wg 2))+,
the first two units \myth{0} and \myth{1} are in the same work-group,
while the last unit~\myth{2} is in a different work-group.
Then, all units are in the same agent.
Some scopes remain unspecified: they implicitly contain only one
item --- \emph{e.g.} the system scope contains one agent, while
there are three work-item scopes that contain one unit each.

Hence, at runtime, there will be two ``scope instances'' of the work-group
level. Those scope instances are available to \cat{} model as a ``scope
relations'' whose names are the names of the corresponding scope levels.
A scope relation is an equivalence relation that relates events whose units
are in the same scope instance of the given level.
For instance, figure~\ref{isa2scopes} pictures the \wg{} relation
for the \ltst{isa2} test.\footnote{Reflexivity edges are omitted for clarity.}
\begin{figure}
\caption{\label{isa2scopes} The \wg{} scope relation of the \ltst{isa2} test}
\begin{center}\hspace*{-2cm}\fmt{isa2+scopes}\end{center}
\end{figure}
We could also have pictured the \agent{} relation. We refrain from doing so,
as there is a single agent scope instance that contains all units.
Thus, the \agent{} relation is the total relation over events,
and picturing it would clobber the diagram.

It is to be noticed that the bell file must define
an enumeration of name \texttt{scopes},
whose tags are the name of scope levels:
\begin{verbatim}
enum scopes = 'system || 'agent || 'wg || 'wave || 'wi
\end{verbatim}
Then, given an expression~$e$ whose value is the
tag \texttt{'\textit{lvl}}, the primitive \texttt{tag2scope($e$)} returns
the scope relation \textit{lvl}.

Scopes also appear as event annotations. Such annotations are the ones
of the instructions that generate them.
Sets of annotated events are also defined from bell file contents,
and are available to \cat{} models as pre-defined variables.
The name of one of those variables is the name
of the annotation with uppercase initial letter.
For instance, on figure~\ref{isa2scopes}, we have
$\texttt{Wg} = \{ b \}$,
$\texttt{Agent} = \{ c \}$,
$\texttt{System} = \{ d,e \}$.
Like scope relations, sets of annotated events can be accessed
from the scope tag by a primitive, \verb+tag2set+.
For instance, in our \ltst{isa2} example,
the expression \verb+tag2set('system)+ will evaluate to the set
of events $\{d,e\}$.

Section 3.9 ```Scoped synchronization order'' of the HSA document
states that two operations may ''\emph{both specify (directly or indirectly
through scope inclusion) scope instance~$S$}''.
We first notice that ``indirectly'' applies to the event annotations,
not to the scope instance that is fixed.
For instance, a memory operation whose scope annotation is agent also
specifies work-group.
More generally, the scope annotation~\textit{lvl} takes effect on
all scopes at levels \emph{lower} than~\textit{lvl}.

In practice, we shall define equivalence relations
\texttt{same-\textit{lvl}}, where \textit{lvl} is a scope level.
Two operations are related by \texttt{same-\textit{lvl}}
when they specify a common instance at level~\textit{lvl} in the HSA document
sense.
This not only means that the two operations
are related by the scope relation \textit{lvl} (\emph{i.e.}
they do belong to one common scope instance at level~\textit{lvl}),
but also that they are annotated by scope~\textit{lvl} or higher,
so that they take effect at level~\textit{lvl}.

Hence, we first define a function \texttt{all-events} that takes
a (scope) tag as argument and returns all the events annotated by
this scope or higher:
\begin{verbatim}
let rec all-events(tag) = match tag with
|| 'system -> tag2events(tag)
|| _ -> tag2events(tag) | all-events(wider(tag))
end
\end{verbatim}
The (omitted) \texttt{wider} function is defined in the bell file:
it simply returns
the tag that is immediately above its tag~argument.
That is, it associates \texttt{'system} to~\texttt{'agent}, \texttt{'agent}
to~\texttt{'wg}, etc.

Following our discussion, we then
define the function~\texttt{same-instance} that takes a scope
tag~\texttt{'\textit{lvl}} as argument and returns the
relation~\texttt{same-\textit{lvl}}:
\begin{verbatim}
let same-instance(lvl) =
 let evts = all-events(lvl) in
 tag2scope(lvl) & (evts * evts)
\end{verbatim}
The function simply performs the intersection of the scope relation
and of the cartesian product on events annotated with
scope level~\textit{lvl} or higher. Figure~\ref{isa2same} pictures two of
those ``same'' relations.\footnote{Reflexivity edges are omitted for clarity.
In particular $e \same{wg} e$ is omitted.}
\begin{figure}
\caption{\label{isa2same}The \same{wg} and~\same{agent} relations of the \ltst{isa2} test.}
\begin{center}\hspace*{-2cm}\fmt{isa2+instances}\end{center}
\end{figure}

We are now ready to define scope synchronization orders, which basically
formalize release-acquire synchronization, with scope restrictions.
From \herd{} pre-defined sets of annotated events we build a few
relevant sets:
\begin{verbatim}
let Release = Screl | Scar
let Acquire = Scacq | Scar
let Synchronizing = Acquire | Release (* Could be Screl|Scacq|Scar *)
\end{verbatim}
Notice that the \texttt{screl}, \texttt{scacq} and~\texttt{scar}
annotations apply to atomic operations and to fences only. As a consequence,
the above sets regroup atomic operations and fences only.
We then directly follow the description of Sec. 3.9, up to a few innocent
changes:
\begin{verbatim}
let acq-rel =
  ((W & Release) * (R & Acquire)) & coh
| ((F & Release) * Acquire) &
  ((po & (_ * W)); coh; (po? & (R * _)))
| (Release * (F & Acquire)) &
  ((po? & (_ * W)); coh; (po & (R * _)))

let sso s = same-instance(s) & acq-rel
\end{verbatim}
\marginpar{The proposed model enforces atomic accesses for the intermediate
$A$ and~$B$ operations such that $A \coh B$ in the fence-to-fence case.
Although this is a race, the HSA document does not specify this atomicity
condition.}
The definition of \texttt{acq-rel} regroups the three top-level items
of the description of the HSA document.
It uses set and relation constructs intensively, most of which have
already been introduced, except the option postfix
operator~\texttt{$e$?} that yields,
\texttt{$e$|id} (\emph{i.e.} $e$ union the identity relation),
and the universe set ``\verb+_+'' that contains all events.
Our (minor) changes are:
\begin{itemize}
\item Some simplifications apply,
as our \texttt{Acquire} and \texttt{Release} sets
contain synchronizing operations~only.
\item Similarly, we make no specific provision for
RMW events as they either belong to the pre-defined sets of reads and writes,
\verb+R+ and \verb+W+, or are represented by a read event (in~\verb+R+)
and a write event (in \verb+W+) --- See our Sec.~\ref{coherence}.
\item We have made the definition a bit more symmetric (and redundant)
by having the third item to produce fence-to-fence order,
as the second item does.
\item We have factored out the condition ``\emph{$X$ and~$Y$ both specify the same instance~$S$}'' (implemented by the call ``\verb+same-instance(s)+'').
\end{itemize}

Finally the HSA-happens-before order~\hhb{}
is defined by following the HSA~document,
as the transitive closure of the union of the program order and of
the union of scope synchronization order for all scopes.
\begin{verbatim}
let union-scopes f = fold (fun (s,y) -> f s | y) (scopes,0)

let hhb = (po | union-scopes sso)+
\end{verbatim}
The function \texttt{union-scopes} takes a function~$f$ from scope
tags to relations as argument (as \texttt{sso} is)
and returns the union of the \texttt{$f(\textit{lvl})$} for all scope tags.
It refers to the \texttt{scopes} tag set, which is implicitly defined
by the \verb+enum scopes = +\ldots{} definition, and to the
\verb+fold+ (over sets) function, defined in appendix~\ref{coh}.
Figure~\ref{isa2sso} pictures the scope synchronization orders
for scopes agent and work-group, all transitive edges being omitted.
\begin{figure}
\caption{\label{isa2sso}Scope synchronization orders for scopes agent (\textsf{sso-agent}) and work-group (\textsf{sso-wg}).}
\begin{center}\hspace*{-2cm}\fmt{isa2+sso}\end{center}
\end{figure}
The figure also pictures the \hhb{} and \coh{} relations.

The HSA document defines three validity conditions on~\hhb.
The \hhb{} relation must be acyclic (equivalently irreflexive, as \hhb{} is
transitive), consistent with \coh{}, and consistent with sequentially consistent
orders (see next section).
We express the first two conditions as follows:
\begin{verbatim}
irreflexive hhb as HhbCons
call consistent (hhb,coh) as HhbCohCons
\end{verbatim}
Notice that Figure~\ref{isa2sso} demonstrates a case of inconsistency
of \hhb{} and~\coh{},
as the union of their pictured representations without transitivity edges is
cyclic. And, indeed, our \ltst{isa2} test
is similar\footnote{We have replaced \texttt{while} loops by \texttt{if} constructs.} to the test
``Race-free transitive synchronization through multiple scopes''
of the HSA document. It should thus be forbidden.

\section{Sequentially consistent synchronization order}
Sec. 3.10 of the HSA document states: ``\emph{there is a
total (apparent) order of all synchronizing operations with,
release, acquire, or acquire-release semantics in a single scope instance}''.
Given a scope instance~$S$, we write $\SCS{S}$ for this total order,
and $\SCLVL{lvl}$ for the union of  $\SCS{S}$ orders for all scope instances
at level~\textit{lvl}. We also abbreviate
sequentially consistent synchronization order as ``SC order''.

We have already defined the set \texttt{Synchronizing} in the previous section.
As to scope instances at level \textit{lvl} we represent then as the
equivalence relation~\symr{\it{lvl}}, which is available
in \cat{} from a scope tag as \texttt{tag2scope('\textit{lvl})}.
We here need a more direct representation of scope instances, as sets.
As a matter of fact,
for a given scope level~\textit{lvl}, scope instances at level~\textit{lvl}
are the equivalence classes of the relation~\symr{\itshape{}lvl}.
The \cat{} primitive \texttt{classes} takes an equivalence relation as argument
and returns its equivalence classes as a set of sets of events.
Hence we can compute the sets~$S$ for a given level as follows:
\marginpar{Luc for Jade, all our models use \texttt{same-instance} in place
of \texttt{tag2scope}\ldots}
\begin{verbatim}
let sync-instances(lvl) =
  classes ((Synchronizing * Synchronizing) & tag2scope(lvl))
\end{verbatim}

Then, the HSA  document clearly states that the total order $\SC{$S$}$
extends \po{}: ``\emph{Given synchronisation operations $X$ and~$Y$, if $X \po Y$ and $X$ and~$Y$ specify the same scope instance~$S$ (directly or indirectly
through inclusivity), then $X \SC{$S$} Y$}''.
However it is unclear if this applies to any pair of synchronizing operations
that belong to~$S$, or only to those whose scope annotations take effect
at the level of~$S$.
Nevertheless, as $\SC{$S$}$ is total and later required to be consistent
with $\po$ it does not matter much. We thus choose the first, more simple,
interpretation. Hence given a scope instance~$S$, we compute the set
of $\po$ linearisation on~$S$ as follows:
\begin{verbatim}
let preSC = po
let makeSCinstance(S) = linearisations(S,preSC)
\end{verbatim}
The \texttt{linearisation($S$,$r$)} primitive that computes all topological sorts of the graph $(S \times r)$ is introduced in appendix~\ref{coh}.
Finally, we compute the set of all possible $\SCLVL{lvl}$ relations as:
\begin{verbatim}
let makeSCscope(lvl) = cross (map makeSCinstance (sync-instances(lvl)))
\end{verbatim}
The function~\texttt{map} is map over sets. In the code above it serves
to compute the set of the sets of all possible $\SCS{S}$ orders
for all scope instances~$S$ at level~\textit{lvl}.
The function~\texttt{cross} takes a set of sets $\{S_1,S_2,\ldots,S_n\}$ as argument and returns the set of all sets built by picking one element
in each~$S_i$. Those two functions are introduced in appendix~\ref{coh}.

Finally, we iterate over all possible choices of $\SCLVL{lvl}$ for the five
HSA scope levels as follows:
\begin{verbatim}
with SWI from makeSCscope('wi)
call consistent(SWI,coh) as ScCohCons
call consistent(SWI,hhb) as ScHhbCons
with SWAVE from makeSCscope('wave)
call consistent(SWAVE,coh) as ScCohCons
call consistent(SWAVE,hhb) as ScHhbCons
with SWG from makeSCscope('wg)
call consistent(SWG,coh) as ScCohCons
call consistent(SWG,hhb) as ScHhbCons
with SAGENT from makeSCscope('agent)
call consistent(SAGENT,coh) as ScCohCons
call consistent(SAGENT,hhb) as ScHhbCons
with SSYSTEM from makeSCscope('system)
call consistent(SSYSTEM,coh) as ScCohCons
call consistent(SSYSTEM,hhb) as ScHhbCons
\end{verbatim}
Notice that we also check the consistency of SC orders with \coh{} and \hhb{},
as required, but not with~\po{}.
Indeed, by construction the ``order'' \SCLVL{lvl} includes~\po and
the two relation are thus consistent.

It remains to check that the SC orders are pairwise consistent:
\marginpar{Luc to Jade, unclear whether omitting the \texttt{ScSc}
checks will impact valid candidates or not.}
\begin{verbatim}
call consistent(SWI,SWAVE) as ScSc
call consistent(SWI,SWG) as ScSc
call consistent(SWI,SAGENT) as ScSc
call consistent(SWI,SSYSTEM) as ScSc
call consistent(SWAVE,SWG) as ScSc
call consistent(SWAVE,SAGENT) as ScSc
call consistent(SWAVE,SSYSTEM) as ScSc
call consistent(SWG,SAGENT) as ScSc
call consistent(SWG,SSYSTEM) as ScSc
call consistent(SAGENT,SSYSTEM) as ScSc
\end{verbatim}
As an example

\section{Races}
The HSA document provides two definition of conflicts, ordinary
and special.
Ordinary conflicts are defined as follows:
``\emph{Two operations $X$ and~$Y$ conflict, iff they access one or
more common byte locations, at least one is a write, and at least one is
an ordinary data operation}''.
Having noticed that ``conflicts'' form a symmetric relation, we paraphrase
the definition:
\begin{verbatim}
let at-least-one a = (a * _) | (_ * a)

let ordinary-conflicts = loc & at-least-one(W) & at-least-one(Ordinary)
\end{verbatim}
We use one pre-defined relation:
\verb+loc+ that relates accesses to the same location
(\herd{} does not handle mixed-size accesses yet),
and two pre-defined sets of events:
\verb+W+ the set of write operations, and \verb+Ordinary+ the set of
ordinary data operations.

The HSA document defines special conflicts as follows:
\begin{quote}\em
 Two special operations $X$ and~$Y$ conflict iff $X$ and $Y$ access
the same byte location and:
\begin{itemize}
\item[35.] $X$ and~$Y$ are different sizes (e.g., 32-bit vs. 64-bit), or
\item[36.] At least one is a write (or a read-modify-write),
and $\neg(\textit{Match}(\textit{SI}(X), \textit{SI}(Y))$.
\end{itemize}
\end{quote}
We ignore condition~35, by lack of mixed-size accesses,
and remind that \verb+W+, our pre-defined set of write operations,
includes the write performed by RMW operations.
Condition~36. refers to a (negated) \textit{Match} predicate
and to a~\textit{SI} function.
Both are defined in Sec. ``Scope instances'' of the HSA document.

The function~\textit{SI} returns the set of scope instances specified by
an operation, and the \textit{Match} predicate
tests the non-emptiness of intersection.
In our Sec.~\ref{sso} we have defined the relation~\texttt{same-instance('\textit{lvl})} that relates events that specify a common scope instance at level~\textit{lvl}. Hence, we represent~\textit{Match} by the relation~\verb+matches+
that relates events specifying a common scope instance at some level.
In effect, we quantify over scope levels rather than on pairs of operations
and  define \verb+matches+ as the union of \texttt{same-instance('\textit{lvl})}
for all scope levels. In \cat{} we write:
\begin{verbatim}
let matches = union-scopes same-instance

let special-conflicts =
  (loc & (Atomic * Atomic) & at-least-one(W)) \ matches
\end{verbatim}
The function~\verb+union-scopes+ that returns the union of
the application of a function on all scope tags is defined
in our Sec.~\ref{sso}.

The definition of HSA conflicts lacks a common additional condition:
accesses have to be by different units, which we consider in
the definition of conflicts below, by the means of the pre-defined~\verb+ext+
relation that relates operations by different units:
\begin{verbatim}
let conflicts = ((ordinary-conflicts|special-conflicts) & ext) \ at-least-one(I)
\end{verbatim}
We have also considered that initial writes (pre-defined set~\verb+I+)
do not enter in conflicts.
As a matter of fact \herd{} consider the initial value of a location
to come from an explicit initial write operation, for the $\rf$ relation to be
defined on all read operations.

We then define races as conflicts that are not ordered by HSA happens-before,
in either way:
\begin{verbatim}
let hsa-race = conflicts \ (hhb | hhb^-1)
\end{verbatim}
We used the postfix~$r\texttt{\^{}-1}$ operator that evaluates to
the inverse of relation~$r$.

It remains to inform the \herd{} simulator about race occurrence.
We do so with the \texttt{flag} construct, which apply to all checks.
The normal behaviour for a check is to stop model execution when invalid.
By contrast, a failing flagged check
does not stop execution but instead ``flags'' it with an arbitrary flag
(here \verb+undefined+).
Those flags are recorded and handed over to the \herd7{} machinery
at the end of model execution --- hence for valid executions that passed
all (unflagged) checks.
The simulator \herd{} can then decide that the simulated program is undefined,
as soon as one of the valid executions has been flagged as \verb+undefined+.
\begin{verbatim}
flag ~empty hsa-race as undefined
\end{verbatim}
Observe that the execution is flagged when the \verb+hsa-race+ relation is
\emph{not} empty.

\appendix
\section{\label{procedure}Checks and procedures}
A model text is a list of instructions.
Checks are significant instructions that, depending on check outcome,
will let execution continue or stop it.
For instance we can test that the two relations
\coh{} and \po{} are consistent by the instruction:
\begin{verbatim}
irreflexive coh;po as CohPoCons
\end{verbatim}
Here, execution will continue when \coh{} and~\po{} are consistent,
\emph{i.e.} when the sequence relation $\coh; \po$ is irreflexive.
Further notice that checks have an optional name, introduced by the \texttt{as}
keyword, for documentation and control purposes (see \herd{} option \texttt{-skipchecks}).

In the case of the consistency check it may be interesting to abstract the
details of the check by defining a procedure:
\begin{verbatim}
procedure consistent(a,b) =
  irreflexive a;b
end
\end{verbatim}

A procedure is called by using the explicit ``\texttt{call}'' keyword.
As an example the consistency of \coh{} and~\po{} can be checked as follows:
\begin{verbatim}
call consistent(coh,po) as CohPoCons
\end{verbatim}

\section{\label{coh}Generating \coh}

The ``coherence'' relation is a fairly classical one.
It can be generated by \herd{} from the $\coinit$ pre-defined relation.
This relation expresses a few constraints on the writes:
it relates initial writes to all other writes and
all non-final writes to final writes, for each  location~$L$.
Those constraints on writes are introduced by \herd{} initial machinery
that enumerates all candidate executions, considering all possible
final writes\footnote{As a matter of fact, we assume that each location
holds a well defined value at the end of program execution, and we
enumerate all such final values.} in turn.

In \herd{}, we can generate all orders on a certain set of events with the
\texttt{linearisations($S$,$R$)} primitive that takes two arguments:
a set of events~$S$ and a relation~$R$.
The primitive considers $R_S$, the restriction of~$R$ to $S$ (written
\texttt{$R$ \& ($S$ * $S$)} in the \cat{} language)
If $R_S$ is acyclic, the call will return the set of all total orders
that extend $R_S$. Otherwise, \emph{i.e.} if $R_S$ has a cycle,
the primitive returns the empty set.
Hence, assuming $S_L$ to be the set of all memory events to location~$L$,
one can generate the set of all possible $\cohl{L}$ by the call
\texttt{linearisations($S_L$,co0)}.
\begin{verbatim}
let makeCohL(s) = linearisations(s,co0)
\end{verbatim}

In fact, we want to generate the set of all possible $\coh$ relations,
\emph{i.e.} all the unions of all the possible $\cohl{L}$ orders for all
locations~$L$. To that end we use another \cat{} primitive:
\texttt{partition($S$)} that takes a set of events as argument and
returns a set of set of events $T = \{S_1,\ldots,S_n\}$, where each
$S_i$ is the set of all events in $S$ that act on location $L_i$,
and, of course $S$ is the union $\bigcup_{i=1}^{i=n} S_i$.

Combining the effect of the \texttt{partition} and \texttt{linearisations}
primitives is possible in the \cat{} language.
We first define a, quite classical,
\texttt{map} functions that, given a set~$S= \{e_1,\ldots,e_n\}$
and a function $f$, returns the set $\{f(e_1),\ldots,f(e_n)\}$:
\begin{verbatim}
let map f  =
  let rec do_map es = match es with
  || {} -> {}
  || e ++ es -> f e ++ do_map es
  end in
  do_map
\end{verbatim}
Notice that \texttt{map} is written in curried style.
The code above uses a few of \cat{} set and relation construct
``\texttt{++}'' is set addition (in infix notation),
``\verb+{}+ is the empty set.
It also uses set pattern matching that permits recursion over sets,
by considering the empty and non-empty cases, in a manner similar
to list pattern matching.

Then, we generate the set of all possible \cohl{L} for all locations~$L$ as
follows:
\begin{verbatim}
let allCohL = map makeCohL (partitions(M))
\end{verbatim}

Now, \texttt{allCohL} is a set of set of relations, one element of which
is the set of all possible $\cohl{L}$ orders for a specific~$L$.
It remains to generate all possible unions of the $\cohl{L}$ for
all possible choices of those and all possible locations~$L$.
It can be done by another \cat{} function~\texttt{cross}, that takes
a set of sets $S = \{S_1, S_2, \ldots, S_n\}$
as argument and returns all possible unions
built by picking elements from each of the $S_i$:
$$
\left\{\, e_1 \cup e_2 \cup \cdots \cup e_n \mid
e_1 \in S_1, e_2 \in S_2, \ldots, e_n \in S_n \,\right\}
$$
\begin{verbatim}
let fold f =
  let rec fold_rec (es,y) = match es with
  || {} -> y
  || e ++ es -> fold_rec (es,f (e,y))
  end in
  fold_rec

let rec cross ess = match ess with
  || {} -> { 0 }
  || es ++ ess ->
      let yss = cross ess in
      fold
        (fun (e,r) -> map (fun ys -> e | ys) yss | r)
        (es,{})
  end
\end{verbatim}
In the code above,
one may notice the use of the union operator ``\verb+|+''
and of the empty relation ``\verb+0+''.

Finally we generate all possible $\coh$ ``orders'' by:
\begin{verbatim}
let allCoh = cross allCohL
\end{verbatim}

\section{\label{bell}The bell file for HSA}

\end{document}


\section{Exploiting Subprograms}\label{sect:foreground}
\subsection{BGP Strategy}
What emerges from the details of BGP's successful examples is a stepwise strategy:
\begin{inparaenum}

\item Capture the behavior of \st{s} within a program in a trace matrix $T$.

\item  Regress $T$ as feature data on the desired program outputs and derive a model $M$.

\item Assign a value of merit $w$ to each \st within $M$. Use this merit to determine whether it should be inserted into an archive. Use a modified crossover that draws subprograms from the archive.

\item Integrate model error and complexity into program fitness so that subprogram behavior influences selection.
\end{inparaenum} \\

% WEKA Citation M. Hall, E. Frank, G. Holmes, B. Pfahringer,
%P. Reutemann, and I. H. Witten. The weka data mining software: An update. SIGKDD Explor. Newsl., 11(1):10–18, Nov. 2009.

One example of the strategy is realized in \cite{krawiecGecco2014} where, in Step~(2), the fast RepTree (\REPTREE - Reduced  Error  Pruning  Tree) algorithm  of decision tree classification from the WEKA Data Mining software library\cite{Hall:2009:WDM:1656274.1656278} is used for regression modeling.  \REPTREE builds a decision/regression tree using information gain/variance. In Step~(3) merit is measured per Equation~\ref{eq:subtree_weight} where $|U(p)|$ is the number of subprograms (equivalently distinct columns of the trace) used in the model and $e$ is model error.

\begin{equation}
\label{eq:subtree_weight}
w = \frac{1}{(1 + e)|U(p)|}
\end{equation}

\subsection{Exploring Model Bias}
Following our motivation to understand the impact of model bias on useful subprogram identification and program fitness, we first explore an alternative realization of BGP's strategy by using the \SCIKIT optimized version of the CART decision tree algorithm\footnote{\url{http://scikit-learn.org/stable/modules/tree.html\#tree-algorithms-id3-c4-5-c5-0-and-cart}}.  CART (Classification and Regression Trees) is very similar to C4.5, but it differs in that it supports numerical target variables (regression) and does not compute rule sets. CART constructs binary trees using the feature and threshold that yield the largest information gain at each node.  With the \SCIKIT implementation we derive a model we denote by $M_S$ and contrast it to deriving $M$ which for clarity we now denote as $M_R$ with $M$ subscripted S for \SCIKIT and R for REPTree.

\subsection{Identifying Useful Subprograms}
\newcommand{\FULL}{\textbf{FULL}\xspace}
\newcommand{\DRAW}{\textbf{DRAW}\xspace}
Next, following our goal to investigate alternate ways to identify useful subprograms based upon the observation that prior work derives useful subtree fitness from a model that references just the feature set of \textit{one} program, we realize Steps~(1) and (4) alternately.  In Step~(1) we first select a set of programs  $C$ from the population. We then form a new kind of trace matrix, $T_c$, by column-wise concatenating all $T${'s} of the programs in $C$. In a version we call \FULL,  $T_c$ is then passed through Step~(2). Steps~(3) and (4) are omitted because in each generation only a single machine learning model is built so all of the selected trees put into the archive in a single generation have the same weight.  This realization of the strategy allows us to experiment with different programs in $C$, trying $C$ containing all the programs in the population, for diversity and, conversely trying elitism, holding only a top fraction of the  population by fitness.

% $x\%$ by fitness (as computed by NSGA2 pareto crowding ordering) where $x$ ranges from 25\% of the population to a less elite shares of $50\%$ and $75\%$.

An alternate implementation, that we name \DRAW,  is to draw random subsets from the combined trace of the programs in $C$, and build a model on each.  This would create many candidate subtrees with different error and complexity, and possibly contribute a more robust archive, if it can be populated with subtrees that are frequently selected by the machine learning model.  We modify Step~(4) because subprograms in $M$ come from the set $T_C$, not solely one program. The information we can integrate into program fitness is whether a program contributed a feature to $M$. For this we use $w_c$, see Equation~\ref{eq:subtree_contrib} which weighs how many times a program's subprogram appeared in the model normalized by the number of features in the model. Let $p'$ be the number of subtrees in the model from program $p$.  We define the weight as $w_c$ per Equation~\ref{eq:subtree_contrib}

\begin{equation}
\label{eq:subtree_contrib}
w_c = 1 - \frac{p'}{U(M)}
\end{equation}

Implementation details of \FULL and of \DRAW are provided the next section.

%Initially $U$ will be the entire population.


\section{Results}
In this section we performed a series of simulations. First, we validated
our implementation of \gls{esn} by replicating the results from Pathak et al.
2017 \cite{pathak_using_2017}. Then we conducted a total of 42 experiments to
test the influence of each weather parameter on forecast accuracy. These experiments are outlined in Table \ref{tab:tasks}.
\input{benchmark}
\subsection{\gls{esn} Forecasting: Demand}
\input{48_demand_results}
\input{04_demand_results}
\subsection{\gls{esn} Forecasting: Solar Energy}
\input{48_solar_results}
\input{04_solar_results}
\subsection{\gls{esn} Forecasting: Wind Energy}
\input{48_wind_results}
\input{04_wind_results}


% Introduction (your goal);

\section{Introduction}
	One of the major domains of Artificial Intelligence is search, i.e. how a rational agent, given some sensor information and possibly some domain knowledge, searches for a goal state. Within search, there are two types of search problems. In uninformed search, the rational agent doesn't have any heuristics, i.e. it has no idea of which states are ``closer'' to the goal state. In informed search, the rational agent does have this information.
	One classic example of informed search is A*, wherein the rational agent uses both backward cost and a heuristic function to prioritize nodes in the fringe. A* search operates under the assumption that the entire search space is known. Therefore, in applications such as robot exploration, where the entire search space is not known, the rational agent must search, carry out the generated search plan, and re-adjust (if necessary) based on discovered obstacles or other environmental factors.

	The goal of this research project is to understand the different types of searches that can be utilized in a ``goal-directed robot-navigation task'' in unknown terrain. In particular, this research project aims to compare these searches in terms of their intuition, implementation, and performance. The three methods that will be compared are the Naive Re-planning A* (NRA*), Lifelong Planning A* (LPA*), and D* Lite.


\chapter{Scientific computing: a story}
\label{chapter:context}
    %% TODO Historique : https://fr.wikipedia.org/wiki/Superordinateur#Historique_des_records

    \section{First computers, from carbon to silicon}%
    \label{sec:first_computers}

        % TODO talk about the human computers, followed by the first (machine) computer created by Cray and IBM, also
        % Von Neuman, analog computers, etc.
        Science has always been tightly associated to computations, hence it is no surprise that the first computers
        were not machines, but humans. Already in the second century AD, Ptolemy, a scientist living in Alexandria,
        wrote the Almagest. This book aggregated the state of the art in mathematics and astronomy and remained a
        reference for centuries. It contained several tables that were computed by the scientist, including a
        trigonometric table (called \emph{table of chords}).

        In 1757, three French astronomers, Clairaut, Lalande and Lepaute, started working on the prediction of the next
        appearances of Halley's comet~\cite[Chapter~1]{human_computers}. Using the recent theories of Newton, they had
        to numerically solve the three-body problem: they computed the orbits of Saturn and Jupiter around the Sun,
        taking into account the attraction force between the two planets. They carried this computation by splitting the
        orbits into tiny steps, computing the new planet locations for each step. They used these sequences of coordinates
        to compute the orbit of Halley's comet around the Sun, by taking into account the effect of the two giant
        planets on the comet and neglecting the effect of the comet itself on the three bodies. In the end, they were successful in
        predicting the next appearance of the comet in the beginning of 1759, making an error of only one month. Their
        work constitutes one of the first recorded division of labor applied to computations, Lalande and Lepaute
        computed the orbits of the two giant planets while Clairaut computed the orbit of the comet.

        Gaspard de Prony, a French civil engineer, went a step further in this endeavor of organized
        computation~\cite[Chapter~2]{human_computers}. In 1791, he was named director of the \emph{Bureau du Cadastre}.
        At the time, the French revolutionary government was preparing reforms for their outdated system of weights and
        measures, which will eventually result in the creation of the metric system. The reforms proposed to measure
        angles in grades instead of degrees, dividing a right angle into 100 grades instead of 90 degrees. Prony was
        tasked to prepare trigonometric tables for this decimal grade system. He organized his staff in a hierarchy with
        three levels:
        \begin{itemize}
            \item The first class of workers, a handful of renowned scientists like Carnot or Legendre, oversaw the
                operations. They had to research the appropriate formulas for computing approximations of trigonometric
                functions with basic arithmetical operations.
            \item The second class, subsequently named \emph{planning committee}, was a team of eight experienced
                computers. Their task consisted in translating the trigonometric equations produced by the first class
                into a sequence of additions and subtractions. They prepared worksheets where all the basic operations
                were written with a blank space for the result.
            \item The third class consisted in nearly ninety computers. Many of them were former servants or wigmakers
                that lost their jobs with the Revolution and did not know any mathematics besides the addition and
                subtraction. Their job was to compute the results to fill the blank spaces left by the second class of
                workers.
        \end{itemize}

        The idea of constructing a machine capable of doing computations is not recent. Already in 1642, Blaise Pascal,
        a French mathematician, invented and built a mechanical calculator that could perform the four arithmetical
        operations. The calculator was not commercially viable at the time, so only twenty machines were built. Much
        later, in the first half of the nineteenth century, Charles Babbage~\cite[Chapters~2-3]{human_computers}, an
        English mathematician, invented two very innovative machines. The difference engine, for computing tables of
        polynomial functions, and the analytical engine, a general purpose computer that would subsequently be qualified
        as \emph{Turing-complete}. Unfortunately, due to a lack of funding, he was never able to build his inventions.
        In the same period, a French inventor named Thomas de Colmar designed and manufactured a digital
        mechanical calculator, called arithmometer. Capable of doing the four arithmetical operations, it was the first
        machine of its kind to be reliable enough for a practical use.  Similarly, Herman Hollerith, an American
        inventor, created the tabulating machine. Initially built to process the 1890 US Census data, it worked by
        reading and summarizing information stored in punched cards. It decreased considerably the duration and cost of
        the whole census organization. These two last inventions~\cite[Chapter~6]{human_computers} were commercial
        successes and started an era of mechanical computations that lasted until the second half of the twentieth
        century.

        Gradually, computing became more and more important and recognized as a discipline. The apparition of modern
        statistics, mainly due to the work of Francis Galton and Karl Pearson in the early twentieth century, led to a
        growing need for computation power. The First World War itself was an important catalyst, as the American,
        French and English governments hired entire computing laboratories to create ballistic
        tables~\cite[Chapter~10]{human_computers}. By the time, electromechanical computers were used everywhere, for
        their efficiency and reliability largely superior to human computers. There was still an important need for
        human labor, not only for operating these machines, but because some complex operations were still carried by hand.

        The first working general purpose programmable computer, named \emph{Z3}, was designed and built more than two
        decades later in Germany by Konrad Zuse~\cite{sep-computing-history}. Completed in 1941, it was an
        electromechanical machine, using both (mechanical) relays and (electronic) vacuum tubes. Its programs, written
        on external tapes, could use loops but not conditional branches. In 1944, the British government built the first
        fully electronic computer, named \emph{Colossus}. Made of vacuum tubes, its primary function was to break the
        German ciphers during the war. Later, in 1945, the first US electronic computer was unveiled, called
        \emph{ENIAC}. Both the Colossus and ENIAC computers were programmed by plugboards and switches, instead of
        reading from a tape like the Z3. Interestingly, the Z3 and Colossus machines used binary arithmetic while the
        ENIAC used decimal representation.

        An important breakthrough came with the notion of stored-program computer, \ie the idea of storing the program
        instructions in memory. Although the original ideas can be traced to Turing himself and his 1936 article, the
        real implementation in electronic computers came several years later. The first stored-program computer was
        the \emph{Manchester Baby}, built at the University of Manchester in 1948~\cite{sep-computing-history}.
        Similarly, the successor of the ENIAC, called \emph{EDVAC} was also a stored-program computer. Yet, at the time,
        computers were still using vacuum tubes. Although this was a great reliability improvement to the mechanical
        parts used before, the vacuum tubes had a very large electricity consumption which started to become problematic
        (the ENIAC consumed \NSI{150}{\kilo\watt}, the EDVAC \NSI{56}{\kilo\watt}). They also required many
        human operators for their daily usage.

        The invention of the transistor in 1947 by three Bell Labs physicists had a huge impact on the whole electronic
        world. It achieves the same functionality as a vacuum tube (amplifying and switching an electrical signal),
        but with a much lower power consumption, much smaller size and easier to mass produce. Hence, it is no
        surprise if the industry quickly started to use this new technology. The first transistor computer was made in
        1953 in the University of Manchester. Later, in 1955, IBM announced the first commercial transistor computer,
        the IBM 608~\cite{ibm608}, made of \Num{3000} transistors. Compared to its predecessor, switching to transistors
        allowed IBM to reduce the computer physical size by \NSI{50}{\percent} and its power consumption by
        \NSI{90}{\percent} while multiplying its computing speed by \Num{2.5}, reaching a performance of \Num{4500}
        additions per second.

    \section{Exponential growth}%
    \label{sec:exponential_growth}

        % TODO Moore's law, huge performance gains, helped to get great scientifical breakthrough
        % Not terminated yet, we "need" to go further. Side note: when will it be enough? Question of the environmental
        % impact (we make great progress in terms of efficiency, like flops/W, but the growth is even faster (so for
        % instance the total consumptions are still increasing)).
        In 1965, Gordon Moore, who will later become the CEO and co-founder of Intel, predicted an exponential growth of
        the number of transistors in a chip~\cite{moore:1965}, based on an extrapolation of the current pace of
        technological progress. He estimated that the number of transistors was doubling every year:
        \begin{quote}
            The complexity for minimum component costs has increased at a rate of roughly a factor of two per year.
            Certainly over the short term this rate can be expected to continue, if not to increase. Over the longer
            term, the rate of increase is a bit more uncertain, although there is no reason to believe it will not
            remain nearly constant for at least 10 years.
        \end{quote}

        Ten years later, Moore revised his forecast to a doubling every two year~\cite{moore:1975}. This prediction,
        which revealed to be true, is now known as \emph{Moore's law}.

        \begin{figure}[htbp]
            \centering
            \includegraphics[width=\textwidth]{img/context/49_years.pdf}
            \caption{\label{fig:context:49_years}
            Evolution of the processor characteristics between 1971 and 2020. Plot inspired from the work of Pedro Bruel,
            generated with data from Wikipedia~\cite{wiki2021chronology,wiki2021transistor}.}
        \end{figure}

        One of the main contributors to this exponential growth of the number of transistors is the exponential
        decrease of their size, as plotted in Figure~\ref{fig:context:49_years}. While the IBM 608 calculator
        commercialized in the 50s had transistors ``no bigger than a paper clip''~\cite{ibm608}, the latest processors
        commercialized in 2020 have \NSI{5}{\nano\meter} transistors.

        In 1974, Dennard~\etal listed a set of rules for scaling simultaneously the transistor density, clock frequency
        and power dissipation of processors~\cite{dennard}, which would eventually be named \emph{Dennard scaling}. The
        effect of this scaling on the device is summarized in Table~\ref{tab:dennard}. A scaling of \(\kappa\) will
        result in a clock frequency multiplied by a factor \(\kappa\) and a number of transistors multiplied by
        \(\kappa^2\) while the power density of the chip remains constant, \ie if the size of the processor does not
        change, it will have the same power consumption and generate the same amount of heat.

        \begin{table}[htpb]
            \centering
            \caption{Dennard scaling with a factor \(\kappa\) (table reproduced from~\cite[Table 1]{dennard}).}
            \label{tab:dennard}
            \begin{tabular}{l|c}
                Device or Circuit Parameter & Scaling Factor\\
                \hline
                Device dimension \(t_{ox}, L, W\) & \(1/\kappa\)\\
                Doping concentration \(N_a\) & \(\kappa\)\\
                Voltage \(V\) & \(1/\kappa\)\\
                Current \(I\) & \(1/\kappa\)\\
                Capacitance \(C=\epsilon A/t\) & \(1/\kappa\)\\
                Delay time/circuit \(VC/I\) & \(1/\kappa\)\\
                Power dissipation/circuit \(VI\) & \(1/\kappa^2\)\\
                Power density \(VI/A\) & 1\\
            \end{tabular}
        \end{table}

        Unfortunately, Dennard scaling came to an end 15 years ago. Indeed, it is no longer possible to scale the
        operating voltage and the gate oxide thickness~\cite{Bohr_2007}, as transistors have reached scales where power
        leakage is no longer negligible. With a voltage that cannot be scaled anymore, the power density cannot remain
        constant and reaches alarming levels of more than
        \NSI{4}{\watt/\milli\metre\squared}~\cite{Hennessy_2019,Hennessy_youtube}, as illustrated in
        Figure~\ref{fig:context:power_density}.

        \begin{figure}[htpb]
            \centering
            \includegraphics[width=\linewidth]{img/context/power_density.pdf}
            \caption{Evolution of the power density in the last 20 years (plot reproduced from ~\cite[Figure
            3]{Hennessy_2019}).}%
            \label{fig:context:power_density}
        \end{figure}

        Consequently, it has become impossible to increase further the CPU frequency, which has reached a limit of about
        \NSI{5}{\giga\hertz} since 2006 after an exponential growth of several decades, as shown by
        Figure~\ref{fig:context:49_years}. One of the responses to keep the race for performance was to design multicore
        processors. Nowadays, nearly all laptops and smartphones have several cores, while high-end servers can have two
        or four sockets with up to 64 cores per socket.

        Yet, it appears that using several cores was only a short-term help for increasing the CPU
        performance. With the end of Dennard scaling, the power density increases with each generation of processor.
        Hence, to stay within a safe thermal design power (TDP), it is no longer possible to power at the nominal
        voltage all the components of the processors (notion named \emph{dark silicon}). Esmaeilzadeh~\etal predict
        that with the \NSI{8}{\nano\metre} processor generation, more than \NSI{50}{\percent} of their transistors might
        be unpowered at any given time~\cite{Esmaeilzadeh_2011}. With these constraints, Hennessy and
        Patterson~\cite{Hennessy_2019,Hennessy_youtube} estimate that the single-processor performance is now only
        growing by a mere \NSI{3}{\percent} per year, which is a much slower pace than the \NSI{50}{\percent} yearly
        growth rate that the industry got used to for decades.

        % First, performance gains are limited by Amdahl's law, stating that the maximal speedup obtained by running an
        % application on \(P\) cores is \(\frac{1}{(1-f) + f/P}\) where \(f\) is the fraction of the application that can
        % be run in parallel. For instance, an application whose \NSI{95}{\percent} of the execution can be made in
        % parallel will have a speedup of at most \(20\), even with an infinite number of cores.

    \section{Scientific computing today}%
    \label{sec:scientific_computing_today}

        \subsection{High performance computing}%
        \label{sub:hpc}

            Computations are now used in every scientific fields, from small statistical calculations to large numerical
            simulations. Science as a whole has immensely benefited from this exponential performance improvement. For
            instance, the price for sequencing an entire human genome has dropped from \SI{100000000}[\$]{} in 2001 to
            \SI{1000}[\$]{} in 2020~\cite{genome_sequencing}.

            To reach the highest performance, a single processor is not enough, and the benefit of doing computations in
            parallel was already well known at the time of human computers, as discussed in Section~\ref{sec:first_computers}.
            For this reason, the largest computations are now performed on supercomputers, large machines made of many
            processors connected through a fast network. The 500 fastest non-classified supercomputers of the world are
            ranked biannually in the Top500 list~\cite{top500}. As shown by Figure~\ref{fig:context:top500}, they suffered from
            the same frequency staling as the rest of the industry in 2006, yet their performance kept an exponential
            increase, in part explained by the exponential increase of their total number of cores.

            \begin{figure}[htbp]
                \centering
                \includegraphics[width=\textwidth]{img/context/top500.pdf}
                \caption{\label{fig:context:top500}
                Evolution of the Top500~\cite{top500} supercomputers between 1993 and 2020.  The line denotes the median, the inner ribbon
                contains the \([\NSI{10}{\percent}, \NSI{90}{\percent}]\) interval, the outer ribbon contains all the
                values.\\ Data compiled by Dan Lenski~\cite{top500_compiled} and plotted by ourselves.}
            \end{figure}

            The largest supercomputers now have thousands of processors, millions of cores and power consumptions of several
            megawatts. At these scales, and with the hardware and software complexity required to reach the highest
            performance, it is extremely difficult to have an accurate picture of the whole platform. Even if all the
            individual components are deterministic at a microscopic level, a supercomputer observed at a macroscopic level
            can arguably be seen as a stochastic object. Hence, it is very common for the operators or the users of these
            machines to stumble on performance anomalies, \ie the \emph{observed} performance is different from the
            \emph{expected} performance. For instance, Petrini~\etal~\cite{Petrini_2003} found a severe but previously
            undetected performance problem on the supercomputer they were using, responsible for a \NSI{100}{\percent}
            performance variation and caused by operating system perturbations. Likewise,
            Tuncer~\etal~\cite[Section~1]{Tuncer_2017} list several examples of performance problems observed in operation:
            \begin{itemize}
                \item The amount of variation in application running time can reach \NSI{100}{\percent} on real-life
                    systems~\cite{Bhatele_2013,Skinner}.
                \item Orphan processes left over from previous jobs consuming system resources~\cite{Brandt_2010}.
                \item Firmware bugs, affecting the CPU usage on the server and making the user programs
                    fail~\cite{cisco_bug}.
                \item Memory leaks in applications, eventually leading to job failure~\cite{Agelastos_2015}.
                \item CPU throttling of the nodes for thermal control~\cite{Brandt2015EnablingAO}.
                \item Resource contention~\cite{Bhatele_2013,dorier:hal-00916091}.
            \end{itemize}
            If unnoticed, these kinds of performance anomalies can severely impact the conclusions than an experimenter
            would draw based on performance measures obtained on the machine.

        \subsection{A reproducibility crisis}%
        \label{sub:reproducibility_crisis}

            For several years there has been an ongoing concern of a reproducibility crisis in science. An online survey
            done in 2016 found that among the \Num{1576} scientists who responded, \NSI{70}{\percent} have failed at
            least once to reproduce the work of another scientist, \NSI{50}{\percent} have even failed to reproduce
            their own work~\cite{nature_survey}. This crisis is particularly concerning in medical science, where some
            researchers estimate that a large part of claimed research findings are
            false~\cite{Ioannidis_2005,freedman}. In this field, the main sources of non-reproducibility are a wrong
            usage of statistics, poor experimental methods or even frauds~\cite{science_misconduct}. In other
            disciplines such as computational biology or computational physics, reproducibility issues mainly come
            from numerical instability and the growing complexity of software environments~\cite{dong2021}.

            Computer science is not immune to these concerns of statistical misusage, numerical errors or frauds.  A
            recurrent problem is also the unavailability of the software programs, methods and data on which are based the
            research articles. For instance, Collberg~\etal examined 601 papers from ACM conferences and journals, they
            were able to personally build the software of only 194 of them while they simply could not access the code
            of 176 of these papers~\cite{collberg2015repeatability}. Likewise, Papadopoulos~\etal propose eight
            methodological principles for better reproducibility and show that most of the published articles
            they surveyed did not follow all these principles~\cite{Papadopoulos_2019}.

            What may be more surprising is that computer science suffers from the same experiment reproducibility
            problems than the other sciences: experimental biases exist and may even be larger. For instance,
            Mytkowicz~\etal show that simply changing the environment variable size can account for a performance
            variability of more than \NSI{30}{\percent}~\cite{Mytkowicz_2009}. They propose two methods for detecting
            and avoiding such measurement bias, based on experiment randomization. Similarly, Curtsinger and Berger have
            implemented a software for repeatedly re-randomizing the layouts of code, stack and heap at runtime, to
            avoid the bias introduced by a particular binary layout~\cite{stabilizer}. Again, the growing problem of
            experimental reproducibility we are describing here is tightly coupled to the growing software and hardware
            complexity.

    \section{This thesis}%
    \label{sec:this_thesis}

        In this thesis, we argue that a large part of computer science, and in particular high performance computing, is
        an experimental science. It is commonplace for computer scientists to run experiments, \eg for comparing the
        performance of several algorithms. With the kind of variability described in
        Section~\ref{sec:scientific_computing_today}, such experiments could easily lead to false conclusions. We
        therefore advocate for using similar methods to the natural sciences, which have been faced with the same issues
        for decades.

        Our contribution towards this goal of improving the confidence in the experimental results is multiple:
        \begin{itemize}
            \item In Part~\ref{part:prediction}, we propose a methodology for predicting the performance of an
                application through simulation. Similarly to biology or physics, making an experiment in simulation
                can be of great help to rule out part of the experimental bias. It can also enable the researchers to
                test scenarios that would be too costly in reality.
                \begin{itemize}
                    \item We contextualize this work in Chapters~\ref{chapter:prediction:related_work}
                        and~\ref{chapter:prediction:hpl}.
                    \item We explain how to improve the efficiency of the simulation in
                        Chapter~\ref{chapter:prediction:emulation}.
                    \item The models we used for the predictions are presented in
                        Chapter~\ref{chapter:prediction:modeling}.
                    \item Chapter~\ref{chapter:prediction:validation} is a thorough validation of the simulations, where
                        we compare the predicted performance with the observations made on real experiments.
                    \item We illustrate an important use case of such simulations in
                        Chapter~\ref{chapter:prediction:sensibility} by performing several sensibility studies.
                \end{itemize}
            \item In Part~\ref{part:experiment}, we present additional tools and methods to help experimenters increase
                the quality of their work.
                \begin{itemize}
                    \item Automation is a great way to improve both reliability and productivity. It helped tremendously
                        throughout the last century for scientific computations, as shown in
                        Section~\ref{sec:first_computers}. Machines make much fewer mistakes than humans (if any) and
                        are much faster. For these reasons, we implemented several programs during this thesis. We
                        automatized the execution of experiments with an experiment engine, presented in
                        Chapter~\ref{chapter:experiment:testbed}. We also implemented several tools, listed in
                        Appendix~\ref{chapter:zenodo}, for automating some parts of the analyses as well as cumbersome
                        daily tasks.
                    \item Several times while conducting the research in this thesis, our performance models were not in
                        agreement with the
                        reality we observed. In some instances, our models were too simple, and we needed to add
                        complexity to capture additional real-world phenomena, as described in
                        Chapters~\ref{chapter:prediction:modeling} and~\ref{chapter:prediction:validation}. But, most of
                        the time, we suffered from an experimental bias (or the lack thereof). These difficulties are
                        described in depth in Chapter~\ref{chapter:experiment:difficulties}.
                    \item With an experimental work that spans several years, like this thesis, it is very likely that the
                        object of the experimental study will evolve. In the case of high performance computing, the
                        machine can
                        have hardware or software upgrades, or even defects, that can all affect significantly the
                        measured performance. In Chapter~\ref{chapter:experiment:tests}, we present the performance
                        non-regression tests we implemented and used in the second half of this thesis. They helped us
                        to detect numerous platform changes that could have led to wrong conclusions if unnoticed.
                \end{itemize}
        \end{itemize}

        % \todo{
        % Performance anomalies $\rightarrow$ reproductibility problem (eg. if we compare two implementations, how can we be sure
        % that the difference we observe really come from the two implementations).
        % $\rightarrow$ Need a way for the users to have more confidence in their experiments.
        % $\rightarrow$ Several contributions towards this goal:
        % (1) Simulation of application, to remove the need of doing experiments.
        % (2) Automation of experiments (peanut), to reduce a lot the possibility of human mistakes.
        % (3) List several experimental difficulties, some of them very unsettling, to be aware of what can happen.
        % (4) Performance non regression tests of the platform, to detect significant changes.}

    % \section{Increased complexity}%
    % \label{sec:increased_complexity}

        % TODO increased complexity everywhere (HW and SW), deterministic at micro-level but the macro-level is random
        % no complete understanding of the whole thing, we need experimental CS, we need models (very similarly to what
        % is done in physics or biology)
    %     Some horror stories~\cite{Petrini_2003}\dots


\documentclass[12pt]{article}

\usepackage{headerfooter}
\usepackage{epsfig}
\usepackage{verbatimfiles}
\usepackage{fullpage}
\usepackage{amsmath}

\newcommand{\HRule}{\rule{\linewidth}{.3mm}}

\bibliographystyle{plain}
\begin{document}

\newcommand{\pd}[2]{\frac{\partial{#1}}{\partial{#2}}}

\begin{center}
{\bf \Large Beam-Column Elements in OpenSees}

{\bf August 22, 2001}

{\bf Michael H. Scott}

{\bf PEER, University of California, Berkeley}
\end{center}

This document provides a brief description of the interaction between a beam-column
element and the SectionForceDeformation and CoordTransformation classes in OpenSees.
Material and geometric nonlinearities are abstracted, or separated, from the
element formulation by using the SectionForceDeformation and CoordTransformation
classes. As a result, an element can be programmed in the ``basic system'' to account
for material nonlinearities, then use one of many transformation types to pick
up geometric nonlinearities.
A displacement based, distributed plasticity formulation is presented as an
example of how a beam-column element is formulated in the basic system.

\section{Geometric Nonlinearity}
In general, the transformation of nodal displacements, {\bf u}, in the global system
to deformations, {\bf v}, in the basic system is described by a nonlinear
function,

\begin{equation}
\label{eq:v=v(u)}
{\bf v} = {\bf v}({\bf u}).
\end{equation}

In a similar manner, the transformation of basic forces, {\bf q}, to forces
{\bf p} in the global frame of reference is given by

\begin{equation}
\label{eq:p=p(q,u)}
{\bf p} = {\bf p}({\bf q}({\bf u}), {\bf u}),
\end{equation}

\noindent where {\bf p} is implicitly a function of {\bf u} via the basic forces,
{\bf q}, as well as an explicit function of {\bf u}. The explicit dependence on
{\bf u} takes into account {\em approximate} geometric nonlinearities such as
P-$\Delta$. These transformations are shown schematically in
figure~\ref{fig:Transformation}.

\begin{figure}[htpb]
\begin{center}
\leavevmode
\hbox{%
\epsfxsize=6.0in
\epsffile{./fig_files/BeamTransf.eps}}
\end{center}
\caption{Transformations between global and basic systems}
\label{fig:Transformation}
\end{figure}

As seen in figure~\ref{fig:BeamClass}, a beam-column element acquires geometric
nonlinearity from the CoordTransformation class, and material nonlinearity from the
SectionForceDeformation class.

\section{Material Nonlinearity}
At every cross-section along the element length, a force-deformation
relationship holds, providing section stress resultants, {\bf s}, as a
function of section deformations, {\bf e},

\begin{equation}
\label{eq:s=s(e)}
{\bf s}(x) = {\bf s}({\bf e}(x)).
\end{equation}

\noindent Linearizing the force-deformation relationship with respect to
deformations reveals the section tangent stiffness, ${\bf k}_s$,

\begin{align}
\Delta{\bf s} &= \pd{\bf s}{\bf e} \Delta{\bf e} \\
\Delta{\bf s} &= {\bf k}_s \Delta{\bf e},
\end{align}

\noindent where ${\bf k}_s = \pd{\bf s}{\bf e}$, the partial derivative of the
section stress resultants with respect to the section deformations.

A beam-column element obtains material nonlinearity through use of the
SectionForceDeformation class, as seen in figure~\ref{fig:BeamClass}.

\section{Class Hierarchy}
Figure~\ref{fig:BeamClass} shows the class interaction between a beam-column
element and the CoordTransformation and SectionForceDeformation classes.
An element can use any one of Linear, PDelta, or Corotational transformations;
and any one of ElasticSection or FiberSection constitutive models. When a new
transformation or section class is added to the framework, the element can use
the new class without modification.

\begin{figure}[htpb]
\begin{center}
\leavevmode
\hbox{%
\epsfxsize=6.5in
\epsffile{./fig_files/BeamClass.eps}}
\end{center}
\caption{DispBeamColumn class diagram}
\label{fig:BeamClass}
\end{figure}

\section{Displacement Based Element Formulation}
This section describes the formulation of a displacement based, distributed plasticity
beam-column element. The governing compatibility and equilibrium equations are covered
along with the consistent element stiffness. Bending deformations are assumed to be
small, and shear deformations are neglected.

\subsection{Compatibility}
For displacement based elements, there is a strong form of compatibility between
basic displacements, {\bf v}, and section deformations {\bf e}, satisfied
pointwise along the element length,

\begin{equation}
\label{eq:e=av}
{\bf e}(x) =
\left[ \begin{array}{c} \varepsilon(x) \\ \kappa(x) \end{array} \right] =
{\bf a}(x) {\bf v},
\end{equation}

\noindent where {\bf a} is the strain-displacement matrix. The section deformations
are the axial strain, $\varepsilon$, and curvature, $\kappa$. Assuming linear axial
displacement and transverse displacement based on cubic Hermitian polynomials,
the shape functions in the basic system are

\begin{equation}
\label{eq:N}
{\bf N}(x) =
\left[ \begin{array}{c} N_1(x) \\ \\ N_2(x) \\ \\ N_3(x) \end{array} \right] =
\left[ \begin{array}{c} \frac{x}{L} \\ \\
L\left( \frac{x}{L} - 2\frac{x^2}{L^2} + \frac{x^3}{L^3} \right) \\ \\
L\left( -\frac{x^2}{L^2} + \frac{x^3}{L^3} \right)
\end{array}
\right].
\end{equation}

\noindent The strain-displacement matrix contains the shape function derivatives.
Axial strain is the first derivative of the axial displacement, and
curvature is the second derivative of the transverse displacement,

\begin{equation}
{\bf a}(x) = \left[ \begin{array}{ccc}
N_{1,x} & 0 & 0 \\ \\
0 & N_{2,xx} & N_{3,xx}
\end{array}
\right].
\end{equation}

\noindent Using the shape functions defined in equation~\ref{eq:N}, the
strain-displacement matrix is then,

\begin{equation}
{\bf a}(x) = \frac{1}{L} \left[ \begin{array}{ccc}
1 & 0 & 0 \\ \\
0 & -4+6\frac{x}{L} & -2+6\frac{x}{L}
\end{array}
\right].
\end{equation}

\noindent The basic displacements, {\bf v}, can be obtained by invoking
the method {\em getBasicTrialDisp()}.
After computing section deformations from basic displacements via
equation~\ref{eq:e=av}, the
method {\em setTrialSectionDeformation()} may be invoked with the updated
deformations, {\bf e}.

\subsection{Equilibrium}
Using the principle of virtual displacements (virtual work),
equilibrium between element end forces, {\bf q}, and section stress
resultants, {\bf s},
is satisfied weakly, or in an average sense, along the element length,

\begin{equation}
\label{eq:q}
{\bf q} = \int_0^L {\bf a}(x)^T {\bf s}(x) \: dx,
\end{equation}

\noindent where the section stress resultants are the axial force, $P$, and bending
moment, $M$,

\begin{equation}
{\bf s}(x) =
\left[ \begin{array}{c} P(x) \\ M(x) \end{array} \right].
\end{equation}

To obtain the current value of section stress resultants, {\bf s}, the method
{\em getStressResultant()} must be invoked. To perform the transformation from
basic to global resisting force (equation~\ref{eq:p=p(q,u)}), the method
{\em getGlobalResistingForce()} should be invoked.

\subsection{Element Stiffness}
To solve the structural system of equations, the element stiffness must be assembled
along with the resisting force. The element stiffness is obtained by taking the
partial derivative of equation~\ref{eq:p=p(q,u)} with respect to displacements,
{\bf u}.

\begin{align}
{\bf k} &= \pd{\bf p}{\bf q}\pd{\bf q}{\bf u} + \left.\pd{\bf p}{\bf u}\right|_{\bf q} \\
&= \pd{\bf p}{\bf q} \pd{\bf q}{\bf v} \pd{\bf v}{\bf u} +
\left.\pd{\bf p}{\bf u}\right|_{\bf q} \\
{\bf k} &= \label{eq:stiff} \pd{\bf p}{\bf q} {\bf k}_b \pd{\bf v}{\bf u} +
\left.\pd{\bf p}{\bf u}\right|_{\bf q}
\end{align}

\noindent The basic element stiffness, ${\bf k}_b$, is the partial derivative
of the basic forces, {\bf q}, with respect to the basic displacements, ${\bf v}$.
Differentiating equation~\ref{eq:q} gives,

\begin{align}
{\bf k}_b &= \pd{\bf q}{\bf v} \\
&= \int_0^L {\bf a}(x)^T \pd{\bf s}{\bf v} \: dx \\
&= \int_0^L {\bf a}(x)^T \pd{\bf s}{\bf e} \pd{\bf e}{\bf v} \: dx \\
{\bf k}_b &= \label{eq:kb} \int_0^L {\bf a}(x)^T {\bf k}_s(x) {\bf a}(x) \: dx
\end{align}

The section tangent stiffness matrix, ${\bf k}_s$, is returned upon invoking the
method {\em getSectionTangent()}. After computing the basic stiffness, ${\bf k}_b$,
the method {\em getGlobalStiffMatrix()} should be invoked to perform the
transformation in equation~\ref{eq:stiff}. The remaining partial derivatives in
equation~\ref{eq:stiff} are computed by the {\em getGlobalStiffMatrix()} method.

\subsection{Numerical Quadrature}
In general, the element integrals, equations~\ref{eq:q} and~\ref{eq:kb}, cannot
be evaluated in closed form due to nonlinearities in the section constitutive model.
These integrals must be approximately evaluated by numerical quadrature,

\begin{align}
\label{eq:qapprox} {\bf q} &\approx
\sum_{i=1}^{N_s} {\bf a}(x_i)^T {\bf s}(x_i) \: W_i \\
\label{eq:kbapprox}{\bf k}_b &\approx
\sum_{i=1}^{N_s} {\bf a}(x_i)^T {\bf k}_s(x_i) {\bf a}(x_i) \: W_i ,
\end{align}

\noindent where $N_s$ is the number of integration points, i.e., the number of section
sample points along the element length.

Integration points, $\xi_i$, and weights, $\omega_i$, are typically
defined over a fixed domain such as $\left[-1,1\right]$ or $\left[0,1\right]$, then
mapped to the element domain $\left[0,L\right]$, where $L$ is the element length.
Assuming points and weights defined on $\left[-1,1\right]$, the following
relationships hold,

\begin{align}
x_i &= \frac{L}{2} \left( \xi_i+1 \right) \\
W_i &= \frac{L}{2} \: \omega_i .
\end{align}

\noindent After mapping the points and weights to the element domain,
equations~\ref{eq:qapprox} and~\ref{eq:kbapprox} can be evaluated.

\end{document}


\chapter{Conclusions}
\label{ch:conlusions}

In this chapter, we take the time to revisit what has been achieved in this thesis and to sum up our original contributions to the reinforcement learning literature. Moreover, we draw some conclusions about the impact that these modern optimization techniques can have on the financial industry and the challenges that must be overcome to make it possible. Finally, we suggest some possible axis for future research.

\section{What Has Been Achieved}
In this thesis, we presented an innovative application of some state-of-the-art policy gradient algorithms to the classical asset allocation problem with transaction costs.\\ In Chapter \ref{ch:discrete_time_stochastic_optimal_control} we introduced the basic concepts of stochastic optimal control in discrete time, which is the standard theoretical framework used to model sequential decision problems. In particular, we presented both the traditional risk-neutral formulation, in which the goal of the agent is to maximize its total returns, and the less common risk-sensitive formulation, in which the agent wishes instead to optimize the trade-off between the rewards and the risk required to achieve them. This chapter laid the theoretical foundations and introduced the notation used in the rest of the thesis.\\
In Chapter \ref{ch:reinforcement_learning} we presented the main ideas of \glsfirst{RL}, a general class of algorithms in the field of \glsfirst{ML} that allows an agent to learn how to behave in a stochastic and possibly unknown environment only by trial-and-error. This algorithms are typically based on the feedback mechanism between the stochastic environment and the agent, which receives a numerical reward for his actions. After having discussed the characteristic features of the \gls{RL} problem, we provided a high-level overview of the different typologies of algorithms.\\
In Chapter \ref{ch:policy_gradient} we gave an in-depth presentation of Policy Gradient algorithms for the risk-neutral control problem. After having introduced the key ideas of these methods, we provided a thorough review of the state-of-the-art algorithms that can be found in the literature. For episodic environments, most of these methods are based on the well-known likelihood-ratio technique from stochastic optimization  that allows to rewrite the policy gradient as an expected value. For non-episodic environments, the main result from which most of the learning algorithms are derived is the policy gradient theorem.\\
In Chapter \ref{ch:risk_sensitive_policy_gradient} we discussed policy gradient methods for the risk-sensitive control problem, which is still an active field of research. In particular, we presented an extension of the policy gradient theorem to the risk-sensitive framework, both in the average reward and in the discounted reward formulations. To the best of our knowledge, this is the first time that a risk-sensitive policy gradient theorem is proved for a general discounted Markov decision process, in which rewards might as well depend on the next state of the system.\\
In Chapter \ref{ch:parameter_based_policy_gradient} we proposed an original parameter-based version of the policy gradient theorem, both for the risk-neutral and the risk-sensitive framework. This extension allowed us to derive efficient online learning algorithms similar in spirit to the well-known \gls{PGPE} algorithm, which was originally conceived only for episodic environments. Moreover, these new algorithms can be easily enhanced using a critic or the natural policy gradient idea. This chapter undoubtedly represents the most innovative contribution of this thesis to the \gls{RL} literature.\\
In Chapter \ref{ch:financial_applications_of_reinforcement_learning}, after a brief discussion about why finance represents an extremely challenging field of research, we provided a bibliographical survey of successful financial applications of \gls{RL} techniques. In particular, we focused on the classical asset allocation problem with transaction cost, which has been used to numerically test the algorithms proposed in the previous chapters.\\
In Chapter \ref{ch:numerical_results} we presented the numerical results for the asset allocation problem. We showed that the learning algorithms proposed in the previous chapters perform extremely well on synthetic data, consistently outperforming a simple buy-and-hold strategy and automatically adapting to transaction costs. On the other hand, the algorithms encountered more difficulties on historical data and we tried to provide an explanation for this behavior.\\

\section{Final Remarks}
The main contribution of this thesis to the \gls{RL} literature is the development of some original, online and parameter-based policy gradient algorithms both for the traditional risk-neutral framework and the less common risk-sensitive framework. These algorithms are highly versatile and can be easily combined with some powerful techniques such as the use of a critic or the use of a natural policy gradient.\\
The numerical application of this innovative algorithms to the classical financial problem of determining a profitable long-short trading strategy confirmed their potential. For a synthetic asset, the strategies learned by the algorithms outperformed the simple Buy \& Hold strategy, even when investment decisions were only based on extremely basic autoregressive features. Contrarily to standard prediction-based trading systems, the learning algorithms were able to adapt as expected to the introduction of transaction costs by reducing the frequency of reallocation and of short positions.\\
The algorithms encountered more difficulties in learning a profitable strategy from historical data. In this case, the algorithms failed to converge to a profitable trading strategy probably because of the low signal-to-noise ratio of the historical price series considered and the low capabilities of representation of the controller used. However, even if we considered a more complex controller success would not be a-priori guaranteed. This is due to the inherent complexity of financial markets. Indeed, while identifying patterns in historical data is not particularly hard, finding signals that work reliably in the real world is. Man AHL, a quant unit of Man Group Plc, needed three years of work to gain enough confidence in a machine learning strategy to devote client money to it. Therefore, it is clear that there is still much work to be done to be able to beat the markets with a strategy produced by a \gls{RL}  algorithm.\\
Despite the difficulties encountered in applying the algorithms developed in this thesis to historical data, we feel confident that in the future these techniques will have a large impact on financial markets. The first reason is that these algorithms have already proven successful in solving very difficult decision problems, such as mastering the game of Go or controlling robots in complex environments. The second reason is the versatility of these techniques, which can be applied with only minor modifications to very different domains. Finally, their modularity allows them to be combined with other modern techniques, such as \glsfirst{DL}. These features have attracted the interest of the financial community, with investment banks and hedge funds investing large amount of resources to develop innovative solutions  based on \gls{ML} methods to classical problems.

\section{Further Developments}
To conclude, let us suggest some research directions that could be explored to improve the work presented in this thesis.\\
The first idea is to develop more complex features so as to allow the agent to take more informed decisions. A first approach would be to construct the indicators typically used in the technical analysis of stock prices. These measures embed the expert knowledge acquired by financial analysts over decades of activity and could help in guiding the agent towards better decisions. Another approach would be to employ some \gls{DL} techniques to learn more powerful features directly from data. A first possibility would be to extract new features using a deep auto-encoder in a completely unsupervised way that would then be fed into a simple parametric controller. A second possibility would be to directly replace the simple autoregressive controller considered in this thesis with a deep neural network. Using a deep neural network was one of the key ingredient of many of the recent successes of \gls{RL}. For a modern presentation of these techniques, the interested reader may refer to \cite{bengio2015deep}.\\
The second idea would be to consider high-frequency data, such as intra-day prices and traded volumes. It is well documented that high-frequency data show some characteristic patterns that could be captured by a deep reinforcement learning trading system. An example of application of \gls{RL} to high-frequency data can be found in \cite{nevmyvaka2006reinforcement}, where the authors applied some basic algorithms such as Q-learning to the optimal trade execution problem in a limit-order book. It would thus be interesting to apply more advanced techniques to this problem.\\
As the \gls{RL} and \gls{DL} literatures grows at a frenetic rhythm, new research directions keep opening up and new applications in different fields provide plenty of food for thought. Given the versatility of these techniques, we are convinced that a breakthrough application to finance is not far away.