\documentclass[letterpaper,twocolumn,10pt]{article}
\usepackage{usenix,epsfig,comment,amssymb,listings}
\setlength{\parskip}{1.8mm}
\setlength{\parindent}{0mm}
\begin{document}
%\date{}
\title{\Large \bf Probabilistic Agent Based Modelling: Mathematical foundations}

\author{
{\rm Daniel Tang}\\
Leeds Institute for Data Analytics
}

\maketitle

% Use the following at camera-ready time to suppress page numbers.
% Comment it out when you first submit the paper for review.
%\thispagestyle{empty}

\abstract
In order to do modelling well we need to do calibration, data assimilation and uncertainty quantification properly. All of these problems can be posed in terms of probabilistic inference. At present, tools to do probabilistic inference on agent based models are lacking. In this paper I introduce the idea of an agent based dynamical system and develop a mathematical foundation for thinking about and working with probabilistic inference in these systems.

\section{Introduction: The problem}

In this paper we will consider the following problem: Suppose we have a complex system that can be described in terms of interacting agents. Suppose also that we have some (possibly incomplete) knowledge of the behaviour of those agents. Suppose we also have a set of real-world, noisy observations of the aggreagate behaviour of the system and are able to define an observation operator for each observation type that defines the conditional probability of making an observaton given the state of the system. We would like to gain information about a the posterior distribution of a (hitherto unobserved) observable at some time in the past, present or future, given our knowledge of the behaviour of the agents, our observations and our observation operators. That information may be an expectation value, a modal value, or a distribution chosen from a family that minimises the KL divergence. (or the modal trajectory in time)

\subsection{Defining agents}
We will proceed by specifying our knowledge of the behaviour of an agent as a computer program written from the point of view of an agent. In order to express any incompleteness in our knowledge of the behaviour, we suppose there is a set of global paramaters, which we do not know but which we have a prior distribution for, that may  infulence an agent's behaviour. In addition, we may make the agent's behaviour stochastic in order to express our uncertainty in the agent's behaviour. By doing this, we can track the effect of our uncertainty in the agent's behaviour on the uncertainty in the aggregate behaviour of the system.

\subsection{Defining observation operators}
Similarly, we will define observation operators as computer programs that take a state of the system as input and return an observation (i.e. a real number or vector of reals). The observation operator will itself be stochastic in order to represent the noise in the observation.

We can connect real-world observations to observations of the model by means of an \texttt{observe(x,v)} command which should be interpreted as ``we observe the variable \texttt{x} to have value \texttt{v}''.

In this way, our whole problem can be specified in a single computer program that looks very much like a normal simulation which, if executed, would generate an instance of our data and an instance of the observable we're interested in. The values generated would probably not match our actual data, but there is a positive, if small, probability that they would match.

Our problem can now be cast as one of doing inference on this computer program. That is, finding the expectation value of the observable of interest given that the generated data matches our real-world observations.

\section{Agent based dynamical systems}
We introduce the concept of an \textit{agent based dynamical system} which consists of a set of interacting \textit{agents}. Each agent has a state and can interact with other agents. At any given time, $t$, each agent has a propensity, $r$, to perform an action such that, in the infinitesimal slice of time, $[t, t+dt]$, there is a probability $rdt$ that the agent performs the action. The only actions an agent can perform are to create a new agent or iterate over agents that have a given property. Surprisingly, we will show that this will be enough to express all agent behaviours.

\section{Object oriented implementation of an agent based dynamical system}
In order to implement an agent based dynamical system in an object oriented programming language, we have a class, \texttt{ABM}, which is templated on the agent state type \texttt{T}. This class represents a probability distribution over ABM states and has three methods:

\begin{tabular}{|l|p{4.5cm}|}
\hline
\texttt{add(s)} & add an agent in state \texttt{s} to the ABM and return the modified ABM\\
\hline
\texttt{integrate(B,T)} & integrate the ABM over T units of time, using behaviour \texttt{B} for the agents\\
\hline
\texttt{observe(o,v)} & observe that the observable \texttt{o} has value \texttt{v} and return the resulting \texttt{ABM}\\
\hline
\texttt{evaluate(e)} & Calculate the expectation value of the function e\\
\hline
\end{tabular}

A \texttt{Behaviour} class consists of a set of \texttt{actions} and \texttt{interactions}. An \texttt{action} is a triplet $(t,r,l)$, where $t$ is the trigger which defines which agents the action applies to, as a function of the agent's state, $r$ is the rate at which the action happens once it is triggered and $l$ is a lambda function that performs the action. $l$ takes the current state of the acting agent, $k$, and a distribution that represents the agent's environment (i.e the distribution of the other agents in the system, given that there is an agent in state $k$) and returns the ABM distribution after the action is performed.  An \texttt{interaction} is very much like an action but the trigger is a function of two agent states and $l$ can refer to both of those states.

\section{A probabilistic interpretation of an \texttt{ABM}}

An \texttt{ABM} object represents a probability distribution over the states of an agent based model. To be mathematically precise, an \texttt{ABM} belongs to a symmetric Fock space on the state space of the agents, $T$, which we'll write as $\mathcal{F}(T)$. However, a Fock space has infinite dimensionality so it is not at all obvious how we could go about representing such a mathematical object as a data structure of manageable size for numerical computation.

We'll start by naming two special distributions: the distribution that is definitely the empty set (i.e. with probability $1$ there are no agents in the system), which we'll write as $\emptyset$, and the state that has zero probability for all states, which we'll write as $\mathbf{0}$.

Surprisingly, we can express everything we need with these two states and just two Fock space operators; the \textit{annihilation operator}, $a_k$, and the \textit{creation operator}, $a_k^\dag$.

The creation operator, $a_k^\dag$, represents the addition of an agent in state $k$ to every state in the probability distribution. By definition, the probability of the ABM being empty after the action of a creation operator is zero.

Creation operators always commute with other creation operators
\begin{equation}
a_k^\dag a_q^\dag = a_q^\dag a_k^\dag
\label{acommute}
\end{equation}

We define the annihilation operator $a_k$ as the operator that has the following properties
\begin{equation}
a_k a_q = a_q a_k
\label{ccommute}
\end{equation}
\begin{equation}
a_k a_q^\dag - a_q^\dag a_k = \delta_{kq}
\label{symcommute}
\end{equation}
\begin{equation}
a_k\emptyset = \mathbf{0}
\label{adagzero}
\end{equation}
where $\delta_{kq}$ is the Kronecker delta function\footnote{To be precise, the $1$ and $0$ of the Kronecker delta function should be replaced by the identity operator and the operator that always returns $\mathbf{0}$, but for notational simplicity we'll allow the free interpretation of numbers as operators that multiply all probabilities of an ABM distribution by that number.}.

For a given (non-probabilistic) ABM, we define the \textit{occupation number} of agent state $s$ to be the number of agents that are in that state. The annihilation operator $a_s$ multiplies the probability of each ABM state by the occupation number of $s$ and removes one agent in state $s$.

This behaviour follows from the properties above. Using a recursive proof, suppose that $a_k a_k^\dag S = nS$ for some integer $n$ then, from equation \ref{symcommute},
\[
a_k a_k^\dag a_k^\dag S = (1 + a_k^\dag a_k)a_k^\dag S = (n+1)a_k^\dag S
\]
We can start the recursion using equations \ref{symcommute} and \ref{adagzero}
\[
a_k a_k^\dag \emptyset = (1 + a_k^\dag a_k)\emptyset = 1\emptyset
\]

We can now interpret the \texttt{add(s)} operation as the creation operator $a^\dag_s$. When the same operator is applied multiple times, we will use the power notation, e.g. $a_s a_s = a^2_s$.

A probability distribution over an ABM can be represented as a sum of creation operators on the empty set. The result looks something like a polynomial:
\[
P = \sum_{\mathbf{n}\in \mathbb{N}^{|k|}} p_\mathbf{n} \prod_k a_k^{\dag n_k}\emptyset
\]
where $\mathbf{n} = (n_1...n_{|k|})$ and $p_\mathbf{n}$ is the probability that there are $n_1$ agents in state $1$, $n_2$ agents in state $2$ etc.

For ease of notation, from now on we will omit the trailing $\emptyset$ on distributions when this is not ambiguous.

An \texttt{models.predatorPrey.generatorPoly.Action} object represents an operator which takes an ABM distribution and returns a rate of change of an ABM distribution, and can be defined as
\[
A(t,r,l) = \int_{t(k)} (l_k - a_k^\dag)a_kr \,dk
\]
where the integral reverts to a sum in the case of discrete state variables.

An \texttt{Interaction} object is very similar to an \texttt{models.predatorPrey.generatorPoly.Action} but with a double integral
\[
I(t,r,l) = \int \int_{t(j,k)} (l_{jk} - a_j^\dag a_k^\dag)a_j a_k r\;  dj\; dk
\]

A \texttt{Behaviour} object is just the sum of its \texttt{Actions} and \texttt{Interactions}:
\[
H = \sum_n A_n + \sum_m I_m
\]
So, a \texttt{Behaviour} is also an operator from an ABM distribution to a rate of change. In this way, we can write the equation of motion of a probabilistic ABM as
\[
\frac{\partial P}{\partial t} = HP
\]
and we see that the \texttt{Behaviour} takes the place of the Hamiltonian when seen as a dynamical system. Given this, we can now express the meaning of the \texttt{integrate} method as
\begin{equation}
P(T) = \int_0^t HP \, dt = e^{tH}P(0)
\label{mastereq}
\end{equation}
where we define the exponential of an operator to be
\[
e^{tH} = \sum_{k=0}^\infty \frac{(tH)^k}{k!}
\]
and $H^n$ is just $n$ repeated applications of $H$ (taking care to respect the order of non-commutative operators).

Let the dot product of two ABM probability distributions be defined as
\[
\left(\sum_\mathbf{n} c_\mathbf{n} a^{\dag \mathbf{n}}\right) \cdot \left(\sum_\mathbf{m} d_\mathbf{m} a^{\dag \mathbf{m}}\right) = \sum_\mathbf{k} c_\mathbf{k}d_\mathbf{k}
\]
where $\mathbf{n} = (n_1...n_N)$ and we use the convenience notation
\[
a^{\dag \mathbf{n}} = \prod_i a_i^{\dag n_i}
\]
so that, for example
\[
a^{\dag \mathbf{n}} \cdot P = p_\mathbf{n}
\]
is just the probability that $P$ is in the state with occupation number $\mathbf{n}$.

If we let an observable, $P(v|s)$, be defined as the conditional probability of making a particular observation, $v$ given the state, $s$, of the ABM,  then we can define the \texttt{observe} method as the operator $O_{P(v|s)}$ such that
\[
O_{P(v|s)}P = \frac{1}{P(v)} \int_S s P(v|s) (s\cdot P) \,ds
\]
where the integration ranges over the ABM states. The observation operator has the effect of applying a Bayesian update to a distribution in order to reflect how knowledge of the observation's value updates our belief about the distribution of the system. The prior probability $P(v)$ is often not known so we define an unnormalised version of the observation operator $\hat{O}_{P(v|s)}$ which omits it (i.e. takes its value to be $1$).

WThe \texttt{evaluate} method is defined as
\[
\mathbf{E}[f]P = \int_S f(s) (s\cdot P) \,ds
\]

With the above interpretations, we can see how a distribution over an ABM, which belongs to an infinite dimensional Fock space, can be represented in a data structure whose size is of the order of the size of the computer program that defines the ABM's behaviour, and we can see how such a representation can be systematically derived from the program.

\section{Some analytic examples}
As a demonstration, let's look at some very simple ABMs which, with the help of the formalism we have developed, we can solve analytically.

\subsection{Single agent}
We start with the simplest possible agent based dynamical system: a system which is either empty or there is a single, stateless agent. Suppose that at time $t=0$ the agent is definitely present and between time $t$ and $t+dt$ the agent leaves the system with probability $dt$. So, we would expect the probability that the agent exists to start at $1$ and reduce exponentially according to $e^{-t}$ as time progresses.

 Here's the program that represents this system
\begin{lstlisting}
  ABM abm
  abm.add(void)

  Behaviour b
  b.add({true}, {1.0}, (state, abm) -> {
    return abm
  })
  return b.integrateTime(abm, T)
\end{lstlisting}
remember that the ABM passed to the action in the bahaviour is the ABM minus the agent whose behaviour we're considering. So in this case, it's the empty ABM and returning this is equivalent to the agent removing itself from the system.

Starting from the first line of the program, we create a new ABM which is definitely empty, so is in the state $\emptyset$. We then add an agent, after which the ABM will be in state $a^\dag \emptyset$. Next we create a behaviour and add an action to it. The action is applicable to all agents (i.e. the only agent in the system) and occurs at a rate of 1 per unit time. The act itself is just the ``do nothing'' action, which is just the identity operator, $1$. So, the behaviour can be expressed as the operator
\[
H = (1 - a^\dag)a
\]
so, the equation of motion is
\[
\frac{\partial P}{\partial t} = HP = (1 - a^\dag )aP
\]
Solving this for $P(0) = a^\dag\emptyset$ gives
\[
P(t) = e^{t(1-a^\dag)a}a^\dag\emptyset
\]
\[
 = \sum_{k=0}^\infty \frac{\left(t(1-a^\dag)a\right)^k}{k!}a^\dag\emptyset
\]
\[
= \sum_{k=0}^\infty \frac{(-t)^k\left((a^\dag-1)a\right)^k}{k!}a^\dag\emptyset
\]
but since $a^na^\dag\emptyset = \mathbf{0}$ for $n>1$
\[
\left((a^\dag-1)a\right)^ka^\dag\emptyset = (a^\dag-1)aa^\dag\emptyset
\]
for $k>0$, so
\[
P(t) = \left(1 + \sum_{k=0}^\infty \frac{(-t)^k}{k!}(a^\dag-1)\right)\emptyset
\]
where the initial $1$ is a correction to the sum in the case that $k=0$. We can now turn the sum back into an exponential to give
\[
P(t) = (1 + e^{-t}(a^\dag -1))\emptyset =  (1 - e^{-t})\emptyset + e^{-t}a^\dag\emptyset
\]
So, the probability that the agent exists (i.e. state $a^\dag\emptyset$) reduces exponentially as time progresses, exactly as we expected, and the probability of the empty state increases exponentially so that the sum over both states remains at $1$.

\subsection{Many agents with state}

We can now show how our operators extend very elegantly to systems with more than one agent and add state to the agents.
We can see immediately that our creation operator works as we would expect:
\[
a_k^\dag a_k^{\dag n} = a_k^{\dag n+1}
\]
This also applies to distributions. Suppose there's a $0.5$ probability that the system is empty and a $0.5$ probability that there's one agent, so $P = 0.5 + 0.5a^\dag$. So
\[
a^\dag P = a^\dag (0.5 + 0.5a^\dag) = 0.5a^\dag + 0.5a^{\dag 2}
\]
So, after applying the creation operator, there's an equal probability of there being either one or two agents, i.e. the operator simultaneously adds an agent to all possible states.

\subsection{Poisson model}
Suppose we start with a system with one agent in state 0 (i.e. $P(0) = a_0^\dag$) and that that single agent adds new agents in state $1$ at a constant rate, $r$. This is just a Poisson process, so we would expect the distribution over the number of agents in state $1$ to be distributed according to the Poisson distribution.

The program for this model would be

\begin{lstlisting}
  ABM abm
  abm.add(0)

  Behaviour b
  b.add(state -> state==0, {r},
       (state, abm) -> {
         abm.add(0)
         abm.add(1)
         return abm
       })
  return b.integrateTime(abm, T)
\end{lstlisting}
Using our rules for constructing the equation for behaviour, for this program we have
\[
\frac{\partial P}{\partial t} = (a_0^\dag a_1^\dag - a_0^\dag)a_0r
\]
This can again be integrated analytically:
\[
P(t) = e^{tH}P(0) = e^{-rt (1-a^\dag)a_0^\dag a_0}a_0^\dag
\]
but since there's always exactly one agent in state $0$, $a_0^\dag a_0$ is just the identity operator so
\[
P(t)  = e^{-rt}e^{rta_1^\dag}a_0^\dag = e^{-rt}\sum_k \frac{(rt)^k}{k!}a_1^{\dag k} a_0^\dag
\]
so, the probability of there being $k$ agents in state $1$ at time $t$ (i.e. the coefficient of the $k^{th}$ power of $a_1^\dag$) is
\[
c_k = \frac{(rt)^ke^{-rt}}{k!}
\]
which is just the Poisson distribution, as we expected.

\subsection{Binomial model}
Suppose we start with $n$ agents and each agent has a propensity, $r$, to annihilate itself. We would expect the distribution over the number of remaining agents to follow a binomial distribution, with an exponentially decaying probability of survival, since we are effectively taking $n$ independent flips of a weighted coin to decide whether each agent has annihilated itself or not.

The program for this, and the resulting behaviour, would be the same as in the example we gave for the single agent case, but this time we add more agents to the initial state.

\begin{lstlisting}
  ABM abm
  for(i in 1...n) {
	  abm.add(void)
  }

  Behaviour b
  b.add({true}, {r}, (state, abm) -> {
    return abm
  })
  return b.integrateTime(abm, T)
\end{lstlisting}

The behaviour is expressed as
\[
H = (1 - a^\dag)ar
\]
The boundary condition this time is $P(0) = a^{\dag n}$. The solution to this is
\[
P(t) = \sum_m{n\choose m}e^{-rtm}(1-e^{-rt})^{n-m}a^{\dag m}
\]
The coefficients of $a^\dag$ (i.e. the probabilities that there are $m$ agents) make up a Binomial distribution, which, again, is what we expected.

The above examples, although very simple, demonstrate how we can start with a computer program which specifies the behaviour of a single agent, convert this into an operator, expressed in terms of creation and annihilation operators, which can then be used to specify the rate of change of a probability distribution over the whole system, then integrate that operator to find the probability distribution of the system at some time in the future.

\subsection{Interacting agents: SIR model}

Now let's get agents to interact. Suppose we have a conpartmental model of infectious disease spread. Suppose there's a population of $n$ people. Each person can be one of suseptible, infected or recovered. Infected people have a propensity, $\beta$, to meet and infect a suseptible person, in which case the suseptible person becomes infected. Also, infected people have a propensity, $\gamma$ to recover.

The program for this would look something like
\begin{lstlisting}
  ABM abm
  for(i in 1...ns) {
	  abm.add(S)
  }
  for(i in 1...ni) {
	  abm.add(I)
  }
  for(i in 1...nr) {
	  abm.add(R)
  }

  Behaviour b
  b.add((s1,s2) -> {s1==S && s2==I},
        {beta},
        (state1, state2, abm) -> {
  	abm.add(I) // replace self
  	abm.add(I) // replace other
  	           // as an infected
    return abm
  })
  b.add(s -> {s == I}, {gamma},
        (state, abm) -> {
    abm.add(R) // agent recovers
  })
  return b.integrateTime(abm, T)
\end{lstlisting}

If we let $a^\dag_S$, $a^\dag_I$ and $a^\dag_R$ represent suseptible, infected and recovered people respectively, then the Hamiltonian of the above behaviour is
\[
H = (a_I^{\dag 2} - a_I^\dag a_S^\dag)a_I a_S\beta + (a_R^\dag - a_I^\dag)a_I\gamma
\]
Although this is a differential equation, it's worth pointing out explicitly that it is representing discrete interactions between agents. So, it is still valid for small numbers of agents and correctly captures things like disease eradication, unlike the standard differential version of the SIR model, which is only accurate for large numbers of agents.

Although this cannot be solved analytically, we can see that it has the expected behaviour. For example, if there are no infected agents, the $a_I$ operators will return $0$ and the rate of change will be zero.

\subsection{Arithmetic}

If an agent has numerical variables in its state, we can represent arithmetic on those variables with a change of state operation. For example, if an agent has two integers, $a$ and $b$ in its state, we can represent the operation $a = a + b$ as
\[
H = (a^\dag_{[a+b,b]} - a_{[a,b]}^\dag)a_{[a,b]}
\]
The same method can be used to apply any operator on any number of variables.

\section{Transformations}

It will often be useful to transform the ABM distriution between different representations so we'll build up a way of doing this now.

First of all, we'll construct the expectation operator which returns the sum of all probabilities in a distribution. We can formally define this as
\[
\mathbf{E}[\, ] = \emptyset \cdot e^{\int_S a_s \, ds}
\]
which, if we take the expansion of the exponential, can be seen to have the effect of integrating over the whole Fock space.

We'll now transform to a moment representation, where the moment of a distribution $P$ is defined as
\[
m(s_1^{j_1}...s_n^{j_n})[P] = \frac{\mathbf{E}[a_{s_1}^{j_1}...a_{s_n}^{j_n}P]}{(j_1!)...(j_n!)}
\]
for convenience, we'll use the shorthand notation
\[
m(s_1^{j_1}...s_n^{j_n})[P] = m(S^J)[P] = \frac{\mathbf{E}[a_S^JP]}{J!}
\]
To help our intuitive understanding of the moments, we note that the moment, $m(s)$, is just the average $s$-occupation number. We now extend the idea of occupation number to pairs of states $(s_1,s_2)$ so that the $(s_1,s_2)$-occupation number is the number of distinct pairs of agents whose states are $s_1$ and $s_2$ respectively. So the $(s_1,s_2)$-occupation number of state $...s_1^ns_2^m...$ is $mn$. If we're after pairs of agents in the same state, $m(s^2)$ from a state $...s^n...$, then there are $\frac{n(n-1)}{2}$ \textit{distinct} pairs. In the same way, we can extend occupation number to any number of agent states. The moments can now be seen to be the expectations of the occupation numbers.

So, if there is an interaction that requires a set of agents in given states, the moment will tell you the expectation value of the rate at which that interaction will occur in the whole system if the agent-based interaction rate is 1.

We can now build an moment heirarchy transform which transforms a distribution that is expressed in terms of the creation operators into a distribution that is expressed in terms of a \textit{moment}-space creation operator, $A_s^\dag$
\[
\mathbf{M}[P] = \sum_{n=0}^\infty \int_{s_1...s_n,m_1...m_n} m(a_{s_1}^{j_1}...a_{s_n}^{j_n})[P] A_{s_1}^{\dag j_1}...A_{s_n}^{\dag j_n}\emptyset
\]
We can now derive the relationship between the moment-space and Fock-space creation and annihilation operators.
\[
\mathbf{M}[a_x^\dag P] = \sum_{n=0}^\infty \int m(a_S^J a_x^\dag)[P] A_S^{\dag J}\emptyset
\]
but by the commutation relation on Fock-space creation operators,
\[
a_x^j a_x^\dag = a_x^{(j-1)}a_x^\dag a_x + a_x^{(j-1)}
\]
so
\[
a_x^j a_x^\dag = a_x^\dag a_x^j + j a_x^{(j-1)}
\]
but for $j>0$
\[
\frac{\mathbf{E}[j a_x^{(j-1)}P]}{j!} = \frac{\mathbf{E}[a^{(j-1)}P]}{(j-1)!} = m(a^{j-1})[P]
\]
so
\[
\mathbf{M}[a_x^\dag P] = \sum_{n=0}^\infty \int m(a_x^\dag a_S^J )[P] A_S^{\dag J}\emptyset  + \sum_{n=0}^\infty \int_{j_x > 0} m(a_x^{-1}a_S^J)[P] A_S^{\dag J}\emptyset
\]
But since $\mathbf{E}[a_x^\dag  a_{s_1}...a_{s_n}P] = \mathbf{E}[a_{s_1}...a_{s_n}P]$
\[
\mathbf{M}[a_x^\dag P] = \mathbf{M}[P] + A_x^\dag \sum_{n=0}^\infty \int_{j_x>0} m(a_x^{-1}a_S^J)[P] A_x^{\dag -1}A_S^{\dag J}\emptyset
\]
\[
= (1+A_x^\dag)\mathbf{M}[P]
\]
Similarly with the annihilation operator
\[
\mathbf{M}[a_x P] = \sum_{n=0}^\infty \int \frac{\mathbf{E}[a_S^J a_x P]}{J!} A_{S}^{\dag J}\emptyset
\]
\[
= \sum_{n=0}^\infty \int \frac{\mathbf{E}[a_S^J a_x P]}{J!} A_{S}^{\dag J} A_x A_x^\dag \emptyset
\]
which, if we define the moment-annihilation operator to have the commutation relation
\[
A_kA_q^\dag - A_q^\dag A_k = \delta_{kq}
\]
in the same way as the Fock-space annihilation operator, then
\[
= \sum_{n=0}^\infty \int \frac{\mathbf{E}[a_S^J a_x P]}{J!}  \frac{1}{j_x+1}A_x A_{S}^{\dag J} A_x^\dag \emptyset
\]
\[
= A_x \sum_{n=0}^\infty \int m(a_S^J a_x) [P]  A_{S}^{\dag J} A_x^\dag \emptyset
\]
so
\[
\mathbf{M}[a_x P] = A_x\mathbf{M}[P]
\]

This provides a very convenient, easily automated way of going from a computer program of an ABM to the dynamics of the moments of the model. At its simplest, this provides a way of automatically generating the moment closure for an ABM.

\section{The path integral and conditional probability}

The time integrated probability
\begin{equation}
P(t) = e^{tH}P(0)
\label{timeint}
\end{equation}
can be given an alternative interpretation as a path integral. To do this, we start by expressing $P(0)$ as a sum
\[
P(0) = \int (a_{s_1}^\dag...a_{s_n}^\dag\emptyset \, ds_1...ds_n \cdot P(0))a_{s_1}^{\dag}...a_{s_n}^{\dag}
\]
\[
 = \int (a_S^\dag \cdot P(0)) a_S^\dag \, dS
\]
so
\[
P(t) = e^{tH}\int (a_S^{\dag} \cdot P(0)) a_S^{\dag} \, dS
\]
if we expand into another sum
\[
P(t) = \int a_C^\dag \cdot \left(e^{tH}\int a_S^{\dag} \cdot P(0) a_S^{\dag} \, dS\right) a_C^\dag \, dC
\]
rearranging
\[
P(t) = \int \int \left(a_C^\dag \cdot e^{tH} a_S^{\dag}\right)\left( a_S^{\dag} \cdot P(0) \right) a_C^\dag \, dS\, dC
\]
The first term above can be interpreted as the probability that the system ends in state $C$ at time $t$ given that it started in state $S$ at $t=0$.
\[
P(C|S) = \left(a_C^\dag \cdot e^{tH} a_S^{\dag}\right)
\]
we can now split the exponential
\[
a_C^\dag\cdot e^{tH} a_S^{\dag} = a_C^{\dag}\cdot e^{0.5tH} e^{0.5tH} a_S^{\dag}
\]
\[
 = \int_I (a_C^{\dag}\cdot e^{0.5tH}a_I^{\dag}) (a_I^{\dag} \cdot e^{0.5tH}a_S^{\dag})\,dI
\]
\[
= \int_I P(C|I)P(I|S) \,dI
\]
[this comes out of the linearity of the operators and the definition of dot product]

Using the same principle, we can split it into as many pieces as we please
\[
P(C_N|C_1) = \int_{C_{1...N}} \prod_{n=1}^{N-1} (a_{C_{n+1}}^\dag \cdot e^\frac{tH}{N}a_{C_n}^\dag) \,dC_{1...N}
\]

\begin{comment}
\subsection{Continuous states and the uncertainty principle}

So far, we've only considered agents with a finite number of discrete states. We may want to endow the agent with a real number as part of its state, in which case the agent can be in any one of an uncountable infinity of states and we'd need a polynomial in an infinite number of variables to represent this.

The idea of using a polynomial to represent a probability distribution carries over to the infinitesimal case quite naturally. The transition is made easier if we think of the coefficients as a function from monomials to coefficient values. For example, for a monovariate polynomial
\[
P = \sum_n c_n a^{\dag n} = \sum_n f(a^{\dag n})a^{\dag n}
\]
In this format we can go over to the infinitesimal limit by replacing the monomials with sets of real numbers. The coefficients can now be represented as a function from sets of real numbers to probability densities (i.e. a PDF over $\mathcal{P}(\mathbb{R})$, the power set of the real numbers). The probability distribution now becomes
\[
P = \int_{\mathbf{z} \in \mathcal{P}(\mathbb{R})} f(\mathbf{z})
\]
Note that because we chose sets rather than bags to represent monomials, we can now no longer represent multiple agents in the same state, but because each state is infinitesimally small the probability of finding more than one agent in a state is of second order.

Multiplication of a state by $\mathbf{z}_n = \left\{a^\dag_n\right\}$ becomes
\[
\mathbf{z_n}P \equiv \int_{\mathbf{z} \in \mathcal{P}(\mathbb{R})} f(\mathbf{z} \setminus\mathbf{z_n})
\]
while differentiation becomes
\[
\frac{\partial P}{\partial \mathbf{z_n}}  \equiv \int_{\mathbf{z} \in \mathcal{P}(\mathbb{R})} f(\mathbf{z} \cup \left\{a^\dag_n\right\})
\]
[TODO: make these integrals more formally correct. Define as differential ring/algebra?]

Consider an agent with a 1-dimensional, real valued state. We'd naturally represent a probability distribution over this value as a density function $P:\mathbb{R} \rightarrow \mathbb{R}$. For such a density function, we define the information of the function to be
\[
I = \int_{-\infty}^\infty P(x) \log(P(x)) dx
\]
We can escape the infinitude of variables by noting that in all realistic scenarios, the amount of information we have about an agent is finite. That is, whatever observations we make of a variable, there always remains some inherent uncertainty. This is analogous to Heisenberg's uncertainty principle: the more precicely we measure the location of a particle, the more momentum it must have. In our case, momentum is replaced with information and we are only interested in cases of finite information.

Given this, we can represent any probability density that has finite information as a probability mass function over a finite number of variables. For example, if the PDF has a compact support, a convenient representation would be as a Bernstein polynomial, a Deselby polynomial or the spectral modes.

[TODO: find good bases that make the mapping simple from agent to aggregate behaviour]

\subsection{Mixed real/integer states}
 We can create a fully general system state that can deal with mixed real and integer agent states by defining the state to be a function from bags of state vectors to real coefficients.
 \[
 P = \int_{\Psi\in\mathcal{P}(\psi)} f(\Psi)
 \]
\end{comment}
\section{Making observations and doing inference}

If we have a prior $P(S)$ over system states and an observation operator $O$ that goes from a probability distribution over a system state to a probability distribution over the reals and we observe that the obervation operator has value $o$ then we can define the posterior
\[
P(S|OS = o) \propto P(OS=o|S)P(S)
\]
i,e, we multiply the prior probability distribution over $S$ by the likelihood of the observation. We can represent likelihoods as polynomials in a similar way to probability distributions
\[
P(o|.) = \sum_n P(o|a^{\dag n})a^{\dag n}
\]
However, multiplication of two probability distributions is not the same as polynomial multiplication. Instead, it is multiplication of the coefficients:
\[
\left(\sum_n c_n a^{\dag n}\right) \times \left(\sum_m d_m a^{\dag m}\right) = \sum_n c_nd_n a^{\dag n}
\]

Going back to the example of a single agent that has a unit propensity to annihilate itself. If we start with a prior that there's a 0.5 probability that the agent exists, then observe that it doesn't exist at time $t$, what's the posterior probability that it existed at time $t=0$? The equation of motion is
\[
\frac{\partial P}{\partial t} = HP = \left(\frac{\partial}{\partial a^\dag} - a^\dag \frac{\partial }{\partial a^\dag}\right)P
\]
Integrating this for the case $P(0) = (1-p) + pa^\dag$
\[
P(t) = e^{\frac{\partial}{\partial a^\dag} - a^\dag \frac{\partial }{\partial a^\dag}}S
\]
\[
= 1-pe^{-rt} + e^{-rt}pa^\dag
\]
So,
\[
P(o|.) = 1 + (1-e^{-rt})a^\dag
\]
So
\[
P(.|o) \propto (1-p) + (1-e^{-rt})pa^\dag
\]
normalising
\[
\int P(.|o) dp = (1-0.5e^{-rt})
\]
\[
P(.|o) = \frac{(1-p) + (1-e^{-rt})pa^\dag}{(1-0.5e^{-rt})}
\]
So, if the prior $P(0) = 0.5 + 0.5a^\dag$ then the posterior is given by
\[
P(.|o) = \frac{0.5 + 0.5(1-e^{-rt})a^\dag}{(1-0.5e^{-rt})}
\]

\section{Numerical methods}

So far, we've considered simple systems that have analytic solutions. However, almost all of the systems of interest will not have an analytic solution. We'll now introduce some techniques to help us to numerically approximate solutions.

Because the number of coefficients in the probabilistic state of a system expands exponentially with the number of variables in an agent's state and with the number of agents, for most real world models it isn't feasible to explicitly represent the probabilistic state as a list of coefficients. There are three ways around this. We can represent the state symbolically, we can approximate the state or we can abstract over the state.

\subsection{Symbolic representation of probability distributions}

In almost all cases of interest a probability distribution, although very high dimensional and complex, will be the result of a (potentially large but manageable) number of operations. What's more, we can construct these states from the empty set of agents $P=1$. So, instead of representing a system state as a list of monomial coefficients we can represent it as a list of operations on $1$:
\[
P = \prod_n O_n1
\]
all of which are constructed from the creation and annihilation operators.

When thought of like this, we never have to explicitly represent a probability distribution as a polynomial or as any other kind of representation. Instead, all we ever need to deal with is creation and annihilation operators on probability distributions.

We can reduce the necessary behaviour of these operators to an abstract algebra that obeys the following axiomatic rules:
\begin{equation}
a^\dag 0 = 0
\label{zerozero}
\end{equation}
where $d_{kq}$ is the Kronecker delta function.

Using these rules, we can show that everything acts as expected. Starting with the number operator, $a^\dag a$, we have, from equations \ref{adagzero} and \ref{zerozero}
\[
a^\dag a 1 = 0
\]
and more generally, if $a^\dag a S = nS$ then, from equation \ref{symcommute}
\[
a^\dag a a^\dag S = a^\dag(a^\dag a +1)S = (n+1)a^\dag S
\]
so the number operator acts as expected.

Given this we can show that the annihilation operator also acts as expected. Suppose again that $a^\dag a S = nS$ then
\[
a a^\dag S = (a^\dag a+1)S = (n+1)S
\]

Equations \ref{ccommute} and \ref{acommute} ensure that these results extend to multivariate cases.

\begin{comment}

\subsection{State approximation: Projecting to lower dimensional spaces}

We can also often approximate a high-dimensional probabilistic state in a lower dimensional space that is more practical to work with.

\subsection{State abstraction: Projection to lower dimensional spaces without approximation}

\section{Integrating over propensities: The path integral}

\section{Data assimilation}
\section{ABM as a computational paradigm}

We define an ABM as a set of (Turing complete) computational processes, which we call \textit{agents}. Without loss of generality, we assume that each agent is executing the same program, although their internal states may differ. In addition, there is an \textit{environment} which has a finite state and a \textit{public API} through which any agent may perform actions or query for information. Similarly, the each agent has a public API through which the environment may act on the agent or gain information about the state of the agent. The environment can also create or annihilate agents.

Computation proceeds in three modes
\begin{description}

\item[Synchronous mode] In which each agent gets to perform computations and act on the environment through repeated execution of a \textit{timestep} function. For each timestep of the whole model, the state of the environment is only updated once per timestep.

\item[Sequential mode] In which each agent has a unique identity which has an ordering. Computation proceeds through repeated execution of each agent's timestep function \textbf{in order of their identity}. This time the environment is updated immediately as agents act upon it.

\item[Asymchronous mode] In which each agent is a separate process which communicates with the environment through asynchronous message passing. We will not consider this mode in this paper.

\end{description}

It is immediately clear that an ABM is a Turing complete computer, so can be considered to be a paradigm for computation. This paradigm is particularly convenient when we are modelling domains that we would naturally think in terms of multiple, interacting agents.

Since both sequential and synchronous modes are Turing complete, it follows that they are equivalent. A constructive proof of this is as follows: A synchronous mode ABM can calculate an sequential mode ABM by having the environment assignin a unique ID to each agent. At each timestep the environment passes a \textit{right of execution} to a single agent. Upon execution of an agents timestep function, the agent may only act if it is currently holding a \textit{right of execution}. Once used, the agent passes the right of execution back to the environment, which then passes it on to the next agent in the ordering of IDs. Similarly, a sequential mode ABM can calculate a synchronous mode ABM by having each agent hold a boolean state which specifies whether the next timestep is a calculation step or an action step. On the calculation step, each agent calculates its actions for that step, while on an action step each agent performs the pre-caclulated action. The boolean state is flipped at the end of each step. Since, in synchronous mode, agent's actions must commute the resulting computation is independent of the ordering of the agents.

Given this equivalence, we will consider only synchronous mode ABMs in the remainder of this paper.

\section{Probabilistic semantics for ABMs}

\subsection{Fixed number of agents, no interaction}

Let's start with a vert simple ABM: an agent-based simulation of a Markov process.

The state of an ABM can be defined as a pair $(E ,\Psi)$, where $E$ is the state of the environment and $\Psi$ is a bag of states of the agents\footnote{where a \textit{bag} is like a set but it may contain multiple members with the same state.}.
\end{comment}

%{\footnotesize \bibliographystyle{acm}
%\bibliography{sample}}

%\theendnotes

\end{document}


\documentclass{article}
\usepackage{a4wide}
\usepackage{amsmath}
\usepackage[colorlinks=true,linkcolor=blue,citecolor=blue]{hyperref}
\usepackage[sort&compress,comma,authoryear]{natbib}
\newcommand{\Prob}[0]{\mbox{Prob}}
\newcommand{\ul}[1]{\underline{#1}}
\newcommand{\ol}[1]{\overline{#1}}
\usepackage{bm}
\usepackage{latexsym}

\title{DEB model description: 'hep'}
%\author{S.A.L.M. Kooijman and K. Lika and S. Augustine and N. Marn and others?}

\begin{document}

\maketitle

This document specifies the typified DEB model \citep{MarqAugu2018} 'hep'.
The the 'hep typified model is a DEB model for hemi-metabolic insects, such as ephemeropterans, odonata and possibly other insect groups. It characterics are:
\begin{itemize}
\item morphological life stages: egg, larva, (sub)imago; functional stages: embryo, juvenile, adult, imago
    the embryo still behaves like model std
\item acceleration starts at birth and ends at puberty
\item  puberty occurs during the larval stage
\item emergence occurs when reproduction buffer density hits a threshold
\item the (sub)imago does not grow or allocate to reproduction. It mobilises reserve to match constant (somatic plus maturity) maintenance
\end{itemize}
Nymphs of \href{https://en.wikipedia.org/wiki/Mayfly}{Mayflies (ephemeropterans)} transform into flying \href{https://en.wikipedia.org/wiki/Mayfly#Subimago}{subimagos}, which subsequently transform into \href{https://en.wikipedia.org/wiki/Imago}{imagos} within minutes till 3 days (depending on species and temperature).
Imagos live several days, and, like subimagos, don't feed, implying that nymphs allocate to reproduction.
Ephemeropterans are presently the only insects that moult (once) while having wings.
The hep model also applies to Odonata and possibly some other groups as well.

To capture this pattern the hep model assumes that type $\cal M$ acceleration \citep[section 3.3]{Kooy2014} occurs between birth (event $b$) and puberty (event $p$), and transformation to subimago occurs when the reproduction buffer exceeds a threshold (event $j$), called $[E_R^j]$.
So (isomorphic) growth occurs in the adult nymph, but not in the (sub)imago.
We assume that the heating length is zero, i.e.\ $L_T = 0$, so $l_T = 0$ as well.

Allocation to reproduction at constant food amounts to $\frac{d} {dt} E_R = \dot{p}_R =  (1 - \kappa) \dot{p}_C  - \dot{p}_J $, with reserve mobilisation $\dot{p}_C = [E_m] L^3 (\frac{\dot{v} s_{\cal M}} {L} + \dot{k}_M) \frac{e g} {e + g}$, maturity maintenance $\dot{p}_J = \dot{k}_J E_H^p$, acceleration factor $s_{\cal M} = L_p/ L_b = l_p/ l_b$.
If emergence would not kick in, $L \rightarrow L_\infty = s_{\cal M} f L_m$ and the reproduction buffer density grows at constant rate $\frac{d} {dt} [E_R] \rightarrow \frac{1 - \kappa} {\kappa} [\dot{p}_M] - \frac{\dot{p}_J} {L_\infty^3}$.
The change in reproduction buffer density is $\frac{d} {dt} [E_R] = [\dot{p}_R] - 3 \dot{r}_B (L_\infty/ L - 1) [E_R]$.
Since reproduction buffer density starts at zero at event $p$, we are sure that it hits any finite threshold, but we need to find the first event as long as $(1 - \kappa) [\dot{p}_M] L_\infty^3 > \kappa \dot{p}_J$, to allow for puberty to be reached.
%In the hex model (see next section), the reproduction buffer density has a maximum, due to acceleration, and therefore does not hit a threshold value for sure.

At constant food density, scaled length changes in scaled time as $\frac{d} {d \tau} l = r_B(l_\infty - l)$ or $l(\tau) = l_\infty - (l_\infty - l_p) \exp(r_B \tau)$ with $r_B = (3 + 3 f/g)^{-1}$ and $l_\infty = s_{\cal M} f$.
The dimensionless scaled reproduction buffer density $v_R = \frac{\kappa} {1 - \kappa} \, \frac{[E_R]} {[E_G]}$ changes as
$\frac{d} {d \tau} v_R = \frac{f g s_{\cal M}/ l + f} {g + f} - \frac{k v_H^p} {l^3} - 3 r_B v_R (f s_{\cal M} / l - 1)$.
So scaled time at emergence is $\tau_j = \int_0^{v_R^j} \frac{d} {d v_R} \tau$ and $l(0) = l_p$ and $l(\tau_j) = l_j$ with  $v_R^j = \frac{\kappa} {1 - \kappa} \, \frac{[E_R^j]} {[E_G]}$.
The number of eggs at emergence is $N = \kappa_R [E_R^j] L_j^3/ E_0 = (1 - \kappa) \kappa_R v_R^j l_j^3/ u_E^0$

\section{Background}

\input{model_intro}

\input{std_description}

\bibliographystyle{apalike}
\bibliography{debmodels}

\end{document}


% - ``Aggregation of Multi-Armed Bandits learning algorithms for Opportunistic Spectrum Access'', see https://hal.inria.fr/hal-01705292

The aggregation approach is especially interesting if we know that the problem the application will face is one of a few known kinds of problems.
In such cases, if there are $N$ different sorts of problems, and if the practitioner has prior knowledge on it, she can use the naive approach by selecting algorithm $\cA_i$ which should perform well on problem $i$, for $i\in[N]$.
Then she can use the aggregation of $\cA_1,\dots,\cA_N$ when facing an unknown problem.
%
% As we said, a third possible approach is to select \emph{on the run} the best algorithm for a specific situation, for which the so called \emph{expert aggregation algorithms} framework can be useful.
%
To the best of the authors' knowledge, aggregation algorithms, such as \ExpQ{} which dates back from 2002 \cite{Auer02},
have actually never been used in practice for stochastic MAB problems.
%
For instance, if we look at the applications of MAB for Opportunistic Spectrum Access, like it was proposed in \cite{Jouini09,Jouini10,Jouini12},
online selection of MAB algorithms was never discussed,
and usually the considered algorithm is simply $\UCB_1$, with no discussion regarding this choice.
% and we show that it appears empirically sub-optimal when applied to simple stochastic problems.

% We present an improved variant of \ExpQ, called \Aggr.
% For synthetic MAB problems, with Bernoulli or other distributions for $K$ arms, simulation results are presented to demonstrate its empirical efficiency.
% We combine classical algorithms, such as Thompson sampling, Upper-Confidence Bounds algorithms (\UCB{} and variants), and Bayesian or \klUCB.
% %
% Our algorithm offers good performance compared to state-of-the-art algorithms
% (\ExpQ{}, \CORRAL{} or \LearnExp{}) \cite{Agarwal16,Singla17},
% and appears as a robust approach to select on the run the best algorithm for any stochastic MAB problem, being more realistic to real-world radio settings than any tuning-based approach.

% \TODOL{This chapter is basically a raw include from my paper ``Aggregation of Multi-Armed Bandits learning algorithms for Opportunistic Spectrum Access'', see https://hal.inria.fr/hal-01705292}

% ----------------------------------------------------------------------
\subsection{Aggregating bandit algorithms}\label{sub:25:aggregation}
% \paragraph{Notations:}\label{sub:25:aggregation}
%
We assume to have $N \geq 2$ MAB algorithms, $\Alg_1, \dots, \Alg_N$,
and let $\Alg_{\mathrm{aggr}}$ be an aggregation algorithm,
which runs the $N$ algorithms in parallel (with the same slotted time), and use them to choose its arms based on a voting from their $N$ decisions.
%
$\Alg_{\mathrm{aggr}}$ depends on a pool of algorithms and a set of parameters.
We would like that $\Alg_{\mathrm{aggr}}$
performs almost as well as the best of the $\Alg_a$, with a good choice of its parameters, independently of the MAB problem.
Ideally $\Alg_{\mathrm{aggr}}$ should perform similarly to the best of the $\Alg_a$.
%
To simplify the presentation, we restrict in Algorithm~\ref{algo:25:Aggr} to bandit algorithms that give deterministic recommendations: one arm is chosen with probability $1$ and the others with probability $0$.
%
However, both \ExpQ{} and \Aggr{} can be adapted to aggregate randomized bandit algorithms, \ie, algorithms that output a probability distribution $q(t) = (q_k(t))_{k\in[K]}$ over the $K$ arms at each time step, and draw the next selected arm according to this distribution.
For instance, Thompson sampling uses its posterior distribution, or \UCB{} can use a distribution with a mass of $1/n$ on the $n \geq 1$ arms of maximal index, and a mass of $0$ on the other arms.
% in case of tie in the $\argmax$.

The aggregation algorithm maintains a probability distribution $\pi^{t}$ on the $N$ algorithms $\Alg_a$,
% (\ie, $\pi^{t}\in\Delta_N$ the simplex of dimension $N$)
starting from a uniform distribution.
The probability of trusting the decision made by algorithm $\Alg_a$ at time $t$ is thus $\pi^t_a$.
$\Alg_{\mathrm{aggr}}$ then simply performs a weighted vote on its algorithms: it first decides whom to trust by sampling $a \in [N]$ from $\pi^t$, then follows $\Alg_a$'s decision: $a \sim \pi^t, A_{\mathrm{aggr}}(t) = A(t) = A_a(t)$.
We prove below that it is equivalent to first let all the algorithms decide their arm, \ie, $A_a(t)$, and then to compute
$\forall k\in[K], p^{t}_k \eqdef \sum_{a = 1}^{N} \pi^{t}_a \times \indic(\{A_a(t) = k\})$, the weighted probability that one of the $N$ aggregated algorithms chose arm $k$, and finally to sample $A(t)\sim\pi^t$.

\begin{proposition}\label{prop:25:equivalenceOfSamplingSchemes}
\begin{leftbar}[propositionbar]  % XXX leftbar propositionbar, comment if needed
	The two following sampling schemes for the aggregated algorithm $\Alg_{\mathrm{aggr}} \left[\Alg_1, \dots, \Alg_N\right]$ are equivalent,
	\ie, they give the same distribution on the action chosen by $\Alg_{\mathrm{aggr}}$, $A_{\mathrm{aggr}}(t) = A(t)$.
	%
	\begin{enumerate}
		\item Sample $a(t)\sim \pi^t$ first, then trusts $\Alg_{a(t)}$'s decision: $A_{\mathrm{aggr}}(t) = A(t) = A_{a(t)}(t)$,
		\item Let $A_a(t)$ be the arm chosen by algorithm $\Alg_a$ (sampled from $q_a^t$), for each $a\in[N]$, and compute $\forall k\in[K], p^{t}_k \eqdef \sum\limits_{a = 1}^{N} \pi^{t}_a \times \indic(A_a(t) = k)$. Then, play $A_{\mathrm{aggr}}(t) = A(t) \sim p^t$.
	\end{enumerate}
	% \begin{align}\label{eq:25:equivalenceOfSamplingSchemes}
	% 	 \\
	% \end{align}
\end{leftbar}  % XXX leftbar propositionbar, comment if needed
\end{proposition}
\begin{proof}
	For any fixed time step $t\in[T]$,
	we denote respectively $P(t)$ and $P'(t)$ the distribution of action of the aggregated algorithm $\Alg_{\mathrm{aggr}}$ at time $t$, respectively under the first and second sampling scheme.
	Let $q_a^t\in\Delta_K$ be the distribution of the chosen action by algorithm $\Alg_a$  (for $a\in[N]$), that depends on the past observations (as well as some external randomness, see Section~\ref{par:2:externalrandomness}).
	% \begin{enumerate}
		% \item
		The first sampling scheme gives
		$P_k(t) = \bP(A_{\mathrm{aggr}}(t) = k) = \bP(\cupdot_{a=1}^N A_{\mathrm{aggr}}(t) = A_a(t) = k, a(t) = a) = \sum_{a=1}^N \bP(A_a(t) = k) \bP(a(t) = a) = \sum_{a=1}^N q_{a,k}(t) \pi_a^t$,
		by independence of $a(t)\sim \pi^t$ and $A_i(t) \sim q_i(t)$ (for all $i$).
		% \item
		The second sampling scheme gives
		$P'_k(t) = \bP(A_{\mathrm{aggr}}(t) = k) = \E[p_k^t]$ by definition,
		and $\E[p_k^t] = \sum_{a = 1}^{N} \pi^{t}_a \E[\indic(A_a(t) = k)] = \sum_{a = 1}^{N} \pi^{t}_a \bP(A_a(t) = k) = \sum_{a=1}^N q_{a,k}(t) \pi_a^t$.
	% \end{enumerate}
	Thus we showed that $P_k(t) = P'_k(t)$, so the two sampling schemes are equivalent, as claimed.
\end{proof}

The main questions are then to know what observations (\ie, arms and rewards) should be given as feedback to which algorithms,
and how to update the trusts at each step, and our proposal \Aggr{} differs from \ExpQ{} on these very two points.
%
The considered aggregation algorithms are special cases of the well-known \emph{multiplicative weights update algorithm}.

% ----------------------------------------------------------------------
\section{The \Aggr{} algorithm}\label{sub:25:Aggr}

The \Aggr{} algorithm is detailed in Algorithm~\ref{algo:25:Aggr}.
%
At every time step, after having observed a reward $r(t)=Y_{A(t),t}$ for its chosen action $A(t)$,
the algorithm updates the trust probabilities from $\pi^t$ to $\pi^{t+1}$ by
a multiplicative exponential factor (using the learning rate and the \emph{unbiased} reward).
%
Only the algorithms $\Alg_a$ who advised the last decision get their trust updated, in order to trust more the ``reliable'' algorithms.

% \begin{small}
	\begin{figure}[h!]
		\centering
		\begin{framed}
		% Documentation at http://mirror.ctan.org/tex-archive/macros/latex/contrib/algorithm2e/doc/algorithm2e.pdf if needed
		% Or https://en.wikibooks.org/wiki/LaTeX/Algorithms#Typesetting_using_the_algorithm2e_package
		\begin{algorithm}[H]
			% XXX Input, data and output
			\KwIn{$N$ bandit algorithms, $\Alg_1, \dots, \Alg_N$, with $N \geq 2$}
			% \KwIn{Number of arms, $K \geq 2$}
			% \KwIn{Time horizon, $T \geq 1$, \textbf{not} used for the learning}
			\KwIn{A sequence of learning rates, $(\eta_t)_{t \geq 1}$, \eg, $\eta_t= \log(N) / (t K)$}
			\KwData{Initial uniform distribution, $\pi^{0} = \cU([N])$}
			\KwResult{$\Alg_{\mathrm{aggr}}=\Aggr\left[\Alg_1, \dots, \Alg_N\right]$}
			% XXX Algorithm
			\For(\tcp*[f]{At every time step})
            {$t = 1, \dots, T$}{
				%
				% \tcp{First, run each algorithm $\Alg_a$}
				\For(\tcp*[f]{Can be parallel}){$a = 1, \dots, N$}{
					$\Alg_a$ updates its internal state (e.g., \UCB{} indexes)\;
					It chooses $A_a(t) \in [K]$\;
				}
				%
				% \tcp{The more algorithms have chosen arm $j$, the more probable it is to be chosen.}
				Let $p^{t}_k \eqdef \sum\limits_{a = 1}^{N} \pi^{t}_a \times \indic(A_a(t) = k), \;\forall k\in[K]$\;
				% \tcp*{Sum trusts}
				Then $\Alg_{\mathrm{aggr}}$ chooses arm $A(t) \sim p^{t}$\;
				% $\P(A(t)_{\mathrm{aggr}} = j) = p^{t}_j$
				% \tcp*{Sample once}
				%
				Give \emph{original} reward $(A(t), r(t))$ to \emph{each} $\Alg_a$ (maybe \emph{not} on its chosen arm)\;
				Compute an \emph{unbiased} estimate of the reward, $\widehat{r}(t) = r(t) / p^{t}_{A(t)}$\;
				% \tcp{Use bandit feedback to update $\pi^{t}$}
				% \tcp{Finally, use the instantaneous bandit feedback (reward $\ell_{A(t),t}$) to update $\pi^{t}$}
				% \For(\tcp*[f]{Can be parallel}){$a = 1, \dots, N$}{
				\For{$a = 1, \dots, N$}{
					% \eIf{$\Alg_a$ was trusted, \ie, $A_a(t) = A(t)$}{
					\uIf{$\Alg_a$ was trusted, \ie, $A_a(t) = A(t)$}{
						$ \pi^{t+1}_{a} = \exp(\eta_t \widehat{r}(t)) \times \pi^{t}_{a} $
						\tcp*{Trusted more}
					}
					\uElse{
						% $ \pi^{t+1}_{a} = \exp(- \eta_t r(t) \times \pi^{t}_{a} $
						$ \pi^{t+1}_{a} = \pi^{t}_{a} $
						\tcp*{Do not update the trust now}
					}
				}
				Renormalize the new weights $\pi^{t+1}$: $\pi^{t+1} \eqdef \pi^{t+1} / \sum_{a=1}^{N} \pi^{t+1}_{a}$.
				% \tcp*{Project it back to $\Delta_N$}
			}
			\caption{The \Aggr{} algorithm, aggregating $N$ MAB algorithms $\Alg_1,\dots,\Alg_N$.}
			\label{algo:25:Aggr}
		\end{algorithm}
		\end{framed}
	\end{figure}
% \end{small}

\paragraph{Unbiased estimate of the rewards.}
%
The reward estimate is unbiased in the following sense. If one had access to the samples $Y_{k,t}$ for all arms $k$, the reward incurred by algorithm $a$ at time $t$ would be $r^a(t) = Y_{A_a(t),t}$.
But we are not in the full information setting (see \cite{CesaLugosi06}), but in the bandit setting, so only one of the samples $Y_{k,t}$ can be observed, $r(t)=Y_{A(t),t}$.
This quantity can only be observed for those algorithms $a$ for which $A_{a}^{t}=A(t)$.
However, by dividing by the probability of observing this recommendation, one obtains an unbiased estimate of $r^a(t)$. More precisely, if we define the estimate by
\begin{equation}
	\widehat{r^a}(t) \eqdef \frac{Y_{A_a(t),t}}{p_{A_a(t)}^{t}} \indic(A_a(t) = A(t)),
\end{equation}
then it satisfies $\E_{O_t}[\widehat{r^a}(t) | O_t] = Y_{A_a(t),t}$, for all $a$, where the expectation is taken conditionally to the history of observations up to round $t$, $O_t$.
%
Observe that $\widehat{r^a}(t) = 0$ for all algorithms $a$ such that $A_a(t) \neq A(t)$.

% - Precise some tricky points
An important feature of \Aggr{} is the feedback provided to each underlying bandit algorithm, upon the observation of arm $A(t)$. Rather than updating only the trusted algorithms (that is the algorithms which would have drawn arm $A(t)$) with the observed reward $r(t)$, we found that updating each algorithm with the (original) reward observed for arm $A(t)$ improves the performance drastically.
%
As expected, the more feedback they get, the faster the underlying algorithms learn, and the better the aggregation algorithm is.
This intuition is backed up by theory explained in \cite{Maillard11}.

Regarding the update of $\pi^t$, one can note that the trust probabilities are not all updated before the normalization step,
and an alternative would be to
increase $\pi_{a}$ if $A_a(t) = A(t)$ and to \emph{decrease it otherwise}.
It would not be so different, as there is a final renormalization step, and empirically this variation has little or no impact on the performance of \Aggr{}.

% ----------------------------------------------------------------------
\paragraph{Comparison with \ExpQ.}\label{sub:25:Exp4}
%
% - explain, but don't give pseudo-code ?
The \ExpQ{} algorithm dates from \cite{Auer02NonStochastic}, and for instance it is also well explained in Section~4.2 of \cite{Bubeck12}.
It is similar to \Aggr{}, presented in Algorithm~\ref{algo:25:Aggr},
but differs in the two following points.
%
The first difference is that $a\sim\pi^t$ is sampled first and the arm chosen by $\Alg_a$ is trusted, whereas \Aggr{}
needs to listen to the $N$ decisions to perform the updates on $\pi^{t}$.
Then, \ExpQ{} gives back an observation (\ie, a pair arm, reward) only to the last trusted algorithm
whereas \Aggr{} gives it to all algorithms.
%
The second difference is that after having computed the reward estimate $\widehat{r^a}(t)$, \ExpQ{} updates the estimated cumulative reward for each algorithm,
$\widetilde{R}_a(t) = \sum_{s=1}^{t} \widehat{r^a}(t) \times \indic(A^{s}_{a} = A^{s}_{\mathrm{aggr}})$.
%
Instead of updating $\pi^{t}$ multiplicatively as we do for our proposal, \ExpQ{} recomputes it, proportionally to
$\exp(\eta_t \widetilde{R}_a(t))$.
Note that there is no difference for the case of constant learning rates, and it does not differ much for decreasing learning rates as well.

We also note that \ExpQ{} uses \emph{losses} instead of rewards, with $\ell(t) = 1 - r(t)$, but this is only a change of vocabulary.
A learner can equivalently play to maximize its cumulated rewards or to minimize its cumulated losses.
Most works on using expert advice and game theory usually prefer to think about losses, as for instance the book \cite{CesaLugosi06}.

\paragraph{How to choose $\eta_t$.}
%
The sequence of non-negative learning rates $(\eta_t)_{t \geq 1}$ used by \ExpQ{} can be arbitrary.
It can be constant but should be non-increasing \cite[Theorem 4.2]{Bubeck12}.
% helps to choose a good sequence.
If the horizon $T$ is known (and fixed), they advise to use $\eta_t \eqdef \eta = 2 \log(N) / (T K)$.
However, for real-world communication problems, it can be considered unrealistic to assume a fixed and known time horizon.
For more discussion on this hypothesis, we refer to Section~1 of our article \cite{Besson2018DoublingTricks}.
% that we shortly present in Appendix~\ref{app:2:DoublingTricks}.
%
Thus we prefer an alternative horizon-free choice of learning rates,
$\eta_t \eqdef \log(N) / (t K)$ suggested by \cite{Bubeck12}.
It is non-increasing, and it is obtained by minimizing the upper-bound on the regret derived in \cite[pp48]{Bubeck12}.
%
We compared both approaches empirically, and the second one usually performs better.
We also stick to this sequence $(\eta_t)_{t \geq 1}$ of decreasing learning rates for \Aggr.

% ----------------------------------------------------------------------
\section{Experiments on simulated MAB problems}\label{sub:25:numExp}

% \TODOL{Je pense refaire complètement les expériences de ce morceau.

% - C'est un peu bête de montrer des problèmes chelous genre Gaussien et "mixed" quand tout le reste de la thèse s'intéresse vraiment aux Bernoulli.

% - C'est un peu naze de montrer en log-y ou en log-log, alors que le reste de la thèse s'intéresse uniquement à des plots x-y, ou des histogrammes.

% - Pour être plus consistant avec les sections suivantes, je pourrais inclure juste 1/2 courbes de regret, et un tableau de résultats pour d'autres problèmes ?
% }

% - explain the problem studied in the simulation
We focus on \emph{i.i.d.} MAB problems, with $K = 9$ arms.
Similar behaviors are observed for other values, \eg, trying up-to $K = 100$ gave the same results.
For Bernoulli problem, the first one uses $\boldsymbol{\mu}=[0.1,\dots,0.9]$, and is considered as a ``simple'' problem.
The second one is divided in three groups:
2 very bad arms ($\mu = 0.01, 0.02$), 5 average arms ($\mu = 0.3$ to $0.6$) and 3 very good arms ($\mu = 0.78, 0.8, 0.82$), and it is considered as a ``harder'' problem.
The horizon is set to $T = 20000$ (but its value is unknown to all algorithms), and simulations are repeated $1000$ times, to estimate the expected regret.
%
This empirical estimation of the expected regret $R_T$ is plotted below,
as a function of $T$, comparing some algorithms $\Alg_1,\dots,\Alg_N$ (for $N=6$), and their aggregation with \Aggr{} (displayed in orange bold),
using the parameter-free learning rate sequence, $\eta_t \eqdef \log(N) / (t K)$.

\paragraph{Lower-bound.}
%
The Lai \& Robbins' logarithmic lower-bound \cite{LaiRobbins85} is also plotted. It corresponds to the second case in Theorem~\ref{thm:2:secondLogTLowerBound}. It is crucial to note that the lower-bound is only asymptotic, and as such one should not be surprised to see regret curves smaller than the lower-bound (\eg, for the easier Bernoulli problem in Figure~\ref{fig:25:EasyBernoulli})

\paragraph{Comparing the algorithms against the same realizations.}
%
Note that for each of the $N=1000$ simulations, we choose to generate all the rewards beforehand, \ie, one full matrix $(Y_{k,t})_{k\in[K], t\in[T]}$, for each of the $N$ repetitions, in order to compare the algorithms on the same realizations of the MAB problem.
%
Similar plots and similar results are obtained if this choice is not made, but it makes more sense to compare them against the same randomization. Note that this choice is the default in SMPyBandits, but the other possibility is implemented as well, by changing \texttt{cache\_rewards} to \texttt{true} or \texttt{false} in the \texttt{configuration} dictionary.
We refer to the online documentation for explanations on all these details.
% See Section~\ref{sub:3:miniConfigurationFileSMPyBandits} for explanations on this \texttt{configuration.py} file, or the online documentation.

We compare the \Aggr{} algorithm,
as well as other aggregation algorithms, \ExpQ{} from \cite{Bubeck12},
\CORRAL{} from \cite{Agarwal16} and \LearnExp{} from \cite{Singla17}, all with their default parameters.
%
The aggregated algorithms consist in:
a naive uniform exploration (to have at least one algorithm with bad performances, \ie, linear regret, that is \emph{not} included in the plots to reduce clutter),
\UCB{} with $\alpha=1/2$, three \klUCB{} algorithms (resp. with Bernoulli, Gaussian and exponential $\kl$ functions), and Bayes-UCB and Thompson sampling with uniform prior.

Figures~\ref{fig:25:EasyBernoulli} and~\ref{fig:25:HarderMixed} are in semilog-$y$ scale, this helps to see that the best algorithms can be an \emph{order of magnitude} more efficient than the worst, and the \Aggr{} performs similarly to the best ones, when the other aggregation algorithms are usually amongst the worst.
Figure~\ref{fig:25:HarderMixed_semilogx} is in semilog-$x$ scale to show that the regret of efficient algorithms are indeed logarithmic.

\begin{figure}[h!]  % [htbp]
	\centering
	\includegraphics[width=1.09\linewidth]{2-Chapters/2-Chapter/Images/plots-aggregation/main_semilogy____env1-4_932221613383548446.pdf}
	\caption[On a ``simple'' Bernoulli problem (semilog-$y$ scale). \Aggr{} is in bold red.]{On a ``simple'' Bernoulli problem (semilog-$y$ scale). \Aggr{} is in \textbf{bold \textcolor{red}{red}}.}
	\label{fig:25:EasyBernoulli}
\end{figure}

\begin{figure}[h!]  % [htbp]
	\centering
	\includegraphics[width=1.09\linewidth]{2-Chapters/2-Chapter/Images/plots-aggregation/main____env2-4_932221613383548446.pdf}
	\caption{On a ``harder'' Bernoulli problem, they all have similar performances, except \LearnExp, and our proposal \Aggr{} outperforms its competitors.}
	\label{fig:25:HardBernoulli}
\end{figure}

\begin{figure}[h!]  % [htbp]
	\centering
	\includegraphics[width=1.09\linewidth]{2-Chapters/2-Chapter/Images/plots-aggregation/main____env3-4_932221613383548446.pdf}
	\caption{On an ``easy'' Gaussian problem, only \Aggr{} shows reasonable performances, thanks to Bayes-UCB and Thompson sampling.}
	\label{fig:25:EasyGaussian}
\end{figure}

\begin{figure}[h!]  % [htbp]
	\centering
	\includegraphics[width=1.09\linewidth]{2-Chapters/2-Chapter/Images/plots-aggregation/main_semilogy____env4-4_932221613383548446.pdf}
	\caption{On a harder problem, mixing Bernoulli, Gaussian, Exponential arms, with 3 arms of each type with the \emph{same mean}.}
	\label{fig:25:HarderMixed}
\end{figure}

\begin{figure}[h!]  % [htbp]
	\centering
	\includegraphics[width=1.09\linewidth]{2-Chapters/2-Chapter/Images/plots-aggregation/main_semilogx____env4-4_932221613383548446.pdf}
	\caption{The semilog-$x$ scale clearly shows the logarithmic growth of the regret for the best algorithms and our proposal \Aggr, even in a hard ``mixed'' problem (\emph{cf}. Figure~\ref{fig:25:HarderMixed}).}
	\label{fig:25:HarderMixed_semilogx}
\end{figure}

For Bernoulli problems (Figures~\ref{fig:25:EasyBernoulli} and~\ref{fig:25:HardBernoulli}), UCB with $\alpha=1/2$, Thompson sampling, Bayes-UCB and \klUCB{}$^+$ (with the binary $\kl$ function) all perform similarly, and \Aggr{} is found to be as efficient as all of them.
For Gaussian and exponential arms, rewards are truncated into $[0,1]$, and the variance of Gaussian distributions is fixed to $\sigma^2 = 0.05$ for all arms, and can be known to the algorithms (the \kl{} function is adapted to this one-dimensional exponential family).
%
Figure~\ref{fig:25:EasyGaussian} uses only Gaussian arms, with a large gap between their means and a relatively small variance, giving an ``easy'' problem.
%
And Figure~\ref{fig:25:HarderMixed} shows a considerably harder ``mixed'' problem, when the distributions are no longer in the same one-dimensional exponential family and so the Lai \& Robbins' lower-bound no longer holds.

For each of the 4 problems considered, the \Aggr{} algorithm  is the best of all the aggregation algorithms,
and its regret is close to the best of the aggregated algorithms.
Especially in difficult problems with mixed distributions, \Aggr{} showed to be more efficient that \ExpQ{} and orders of magnitude better than the other reference aggregation algorithms \LearnExp{} and \CORRAL{} (see Figures~\ref{fig:25:HarderMixed} and~\ref{fig:25:HarderMixed_semilogx}).

\paragraph{Reproducibility.}
%
The experiments in this section are based on our library SMPyBandits,
and the page \href{https://SMPyBandits.GitHub.io/Aggregation.html}{\texttt{SMPyBandits.GitHub.io/Aggregation.html}} gives instructions to reproduce them.

% ----------------------------------------------------------------------
% \subsection{No theoretical guarantees.}\label{sec:25:theory}
\section{Conclusion -- Towards theoretical guarantees}\label{sec:25:theory}

The \Aggr{} does not have satisfying theoretical guarantees in terms of regret $R_T$, unlike many bandit algorithms.
%
Another notion, the \emph{adversarial regret}, denoted by $\overline{R_T}$,
measures the difference in terms of rewards,
between the aggregation algorithm $\Alg_{\mathrm{aggr}}$ and the best aggregated algorithm $\Alg_a$. This is in contrast with the (classical) regret, which measure the difference with the best fixed-arm strategy (Definition~\ref{def:2:regret}).
Thus, even if the aggregated algorithms have logarithmic (classical) regret, having an adversarial regret scaling as $\sqrt{T}$ does not permit to obtain a logarithmic (classical) regret for the aggregation algorithm.
%
%
Under some additional hypotheses,
\cite[Theorem 4.2]{Bubeck12} proves that
\ExpQ{} has a bounded adversarial regret, % $\overline{R_T}$
$\overline{R_T} \leq 2 \sqrt{T N \log(K)}$,
with the good choice of the learning rate sequence $(\eta_t)_{t \geq 1}$.

The proposed algorithm follows quite closely the architecture of \ExpQ,
and a similar bound for \Aggr{} is expected to hold.
%
This would be a first theoretical guarantee, but not satisfactory as we saw above that simple algorithms (like \UCB) have regrets scaling as $\log(T)$ \cite{Auer02,Bubeck12}, not $\sqrt{T}$.
%
Regret bounds in several different settings are proven for the \CORRAL{} algorithm \cite{Agarwal16}, but no logarithmic upper-bound can be obtained from their technique, even in the simplest setting of stochastic bandits.
%
However, \Aggr{} seems to have a (finite-horizon) logarithmic regret in all the experiments we performed,
for both Bernoulli and non-Bernoulli problems (\eg, Gaussian, exponential and Poisson distributions).
% Further theoretical developments were left as future work.

% % ----------------------------------------------------------------------
% \paragraph{Conclusion about our \Aggr{} algorithm.}\label{sub:25:conclusion}

% We presented the use of aggregation algorithms in the context of Opportunistic Spectrum Access for Cognitive Radio,
% especially for the real-world setting of unknown problem instances,
% when tuning parameters before-hand is no longer possible and an adaptive algorithm is preferable.
% % - \ExpQ and \Aggr works fine
% % Both algorithms \ExpQ{} and \Aggr{} have been detailed, and we explained why
% Our proposed \Aggr{} was presented in details,
% and we also highlighted its differences with \ExpQ.
% % as well \color{red} as the intuition that it seems more suited for purely stochastic problems\color{black}.
% %
% % - \Aggr{} really help to select the best algorithm against a certain problem, on the fly without any prior knowledge of the problem neither any prior knowledge on which algorithms is the best
% We presented experiments on simple MAB problems coming from previous work on bandits for OSA \cite{Jouini10},
% and the simulations results showed that \Aggr{} is able to identify on the fly the best algorithm to trust for a specific problem, as expected.
% Experiments on problems mixing different families of distributions were also presented, with similar conclusions in favor of \Aggr.
% % It is not presented in this section, but our proposed algorithm also works well in dynamic scenarios, in which the distribution of the arms can change abruptly at some time,
% % and appears to be more robust than simple non-aggregated algorithms.

% \ExpQ{} has theoretical guarantees in terms of adversarial regret, and even if the same result could hold for \Aggr, results in terms of classical regret are yet to be proven.
% Empirically, \Aggr{} showed to always have a logarithmic
% regret $R_T$ if it aggregates algorithms with logarithmic regrets (like UCB, \klUCB, Thompson sampling, Bayes-UCB etc).
% It usually succeeds to be close to the best of the aggregated algorithms, both in terms of regret and best arm pull frequency.
% As expected, the \Aggr{} is never able to outperform any of the aggregated algorithms, but this was an over-optimistic goal.
% %
% What matters the most is that, empirically, \Aggr{} is able to quickly discover \emph{on the fly} the best algorithms to trust, and then performs almost as well as if it was following it from the beginning.
% %
% %
% Our \Aggr{} algorithm could probably be rewritten as an Online Mirror Descent (for more details, see for instance \cite{Hazan2016introduction,Zimmert2018}), as it is the case for both \ExpQ{} and \CORRAL.
% But this direction does not appear very useful, as in the case of \CORRAL{}  the analysis cannot bring a logarithmic regret bound, even by aggregating asymptotically optimal algorithms.
% % We will continue investigating regret bounds for \Aggr,
% % and other directions include possible applications to the non-stochastic case (\eg, rested or restless Markovian problems, like it was very recently studied in \cite{Luo18}).

% \paragraph{Reproducibility.}
% %
% The experiments in this section are based on our library SMPyBandits,
% and the page \href{https://SMPyBandits.GitHub.io/Aggregation.html}{\texttt{SMPyBandits.GitHub.io/Aggregation.html}} gives instructions to reproduce them.

% ----------------------------------------------------------------------------
% \bibliographystyle{ieeetr}
% \bibliography{IEEE_WCNC__2018__Paper__Lilian_Besson__07-17}


\subsection{Basic Properties of Graphene}

Graphene is a two dimensional hexagonal atomic structure (also called honeycomb structure) made of carbon atoms. After its discovery in 2004 \cite{novoselov}, graphene has sparked increasing interest among scientists due to its extraordinary mechanical \cite{strength}, optical \cite{optical} and electrical \cite{carrier} properties. This includes being a semiconductor without a band gap \cite{Lherbier2012}, high carrier mobility\cite{carrier} and an enormous intrinsic tensile strength \cite{strength}.

\subsubsection{Graphene Lattice}

Graphene is a two dimensional allotrope of carbon where atoms are densely packed in a hexagonal pattern with a distance of about \SI{1.42}{\angstrom} as shown in figure~\ref{fig:spm}. Each sp$^2$-hybridized carbon atom has a covalent $\sigma$-bond to three other carbon atoms formed from three of the valence electrons while the 4th valence electron does not participate in the covalent bonding but forms a conduction $\pi$-band.

\begin{figure}[!h]
  \centering
  \includegraphics[width=0.7\textwidth]{./images/graphene-spm.png}
  \caption{Graphene's two dimensional hexagonal structure as seen through scanning tunnerling microscopy. The distance between two atoms is about \SI{1.42}{\angstrom} (adapted from \cite{graphene-spm}). One hexagon is exemplarily highlighted.}
  \label{fig:spm}
\end{figure}

As seen in figure~\ref{fig:lattice}a graphene's lattice consists of a two-atom triangular Bravais lattice which is spanned by the two basis vectors $\mathbf{a}_1$ and $\mathbf{a}_2$. For solid state physics it is often interesting to look at the corresponding lattice in $k$-space. Graphene's reciprocal lattice is also hexagonal with a hexagonal unit cell with $\Gamma$ as its center (as seen in figure~\ref{fig:lattice}b). Other symmetric points are $K_+$ and $K_-$ at the corners of the first Brillouin zone. In addition to them there is the $M$ point describing the center between $K_+$ and $K_-$.

\begin{figure}[!h]
  \centering
  \begin{subfigure}[t]{0.45\textwidth}
    \caption{}
    \includegraphics[width=\textwidth]{./images/cell.png}
  \end{subfigure}
  ~
  \begin{subfigure}[t]{0.45\textwidth}
    \caption{}
    \includegraphics[width=\textwidth]{./images/k-cell.png}
  \end{subfigure}
  \caption{\textbf{(a)} The basis vectors $\mathbf{a}_1$ and $\mathbf{a}_2$ build a triangular Bravais lattice with a two atom basis. The Wigner-Seitz cell is shaded in gray (adapted from \cite{maffucci2014}). \textbf{(b)} The reciprocal lattice with the unit cell shaded in gray. $\Gamma$ is the cell's center; $K_+$, $K_-$ and $M$ are high-symmetry points (adapted from \cite{maffucci2014}).}
  \label{fig:lattice}
\end{figure}

\newpage

\subsubsection{Electronic Properties}

Most of graphene's special electronic properties are caused by the fourth valence electron which causes the $\pi$ orbitals to hybridize and form a delocalized electron cloud above and below the graphene lattice enabling ballistic transport of electrons and a high carrier mobility \cite{carrier}. These orbitals are depicted in figure~\ref{fig:orbitals}.

\begin{figure}[!h]
  \centering
  \includegraphics[width=0.3\textwidth]{./images/orbitals.png}
  \caption{Valence orbitals of carbon atoms in graphene. Covalent bonds are formed from $\sigma$-orbitals while the perpendicular $\pi$-orbitals create the valence and conduction band (adapted from \cite{orbitals}).}
  \label{fig:orbitals}
\end{figure}

Due to the symmetry of the system the $\pi$ orbitals can be treated independently as the main contributor to the conduction and valence band. The band structure can be calculated with a tight-binding model as \cite{dispersion}

\begin{equation}
  E(\mathbf{k})=\pm\sqrt{\gamma_0^2\left(1+4\cos^2\frac{k_ya}{2}+4\cos\frac{k_ya}{2}\cdot\cos\frac{k_x\sqrt{3}a}{2}\right)},
  \label{eq:dispersion}
\end{equation}

with the hopping energy $\gamma_0\approx\SI{2.8}{eV}$~\cite{dispersion}, the lattice constant $a\approx\SI{2.46}{Å}$, "$+$" for the conduction band and "$-$" for the valence band.

A plot of the band structure in figure~\ref{fig:bands} shows that valence and conduction band touch in the $K$-points. Looking a bit closer at the band structure near the $K$ points in figure~\ref{fig:bands} allows to use a linear approximation of the valence and conduction band. Both bands touch in a single point at the intersection.

\begin{figure}[!h]
  \centering
  \includegraphics[width=0.8\textwidth]{./images/dispersion.png}
  \caption { Conduction and valence band touch in the points $K_-$ and $K_+$. At these points the conduction and valence bands can be linearly approximated (figure from \cite{graphene}). }
  \label{fig:bands}
\end{figure}

Evaluating equation~(\ref{eq:dispersion}) at the K points gives

\begin{equation}
  E(\mathbf{K})=0,
\end{equation}

confirming that conduction and valence band touch in the K points and therefore graphene is a semiconductor without a band gap.

\subsubsection{Other Properties}

Other interesting properties of graphene often exceed those of any other material. It has an intrinsic tensile strength of 130 GPa and a Young's modulus as high as 1TPa~\cite{strength}. It is also impermeable to gases~\cite{Bunch2008} and absorbs light by a factor of $\pi\alpha$~\cite{optical}, where $\alpha$ is the fine structure constant.