\documentclass{article}

\usepackage{fancyhdr}
\usepackage{extramarks}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{syntax}
\usepackage{stmaryrd}
\usepackage{mathpartir}
\usepackage{tipa}

%
% Basic Document Settings
%

\topmargin=-0.45in
\evensidemargin=0in
\oddsidemargin=0in
\textwidth=6.5in
\textheight=9.0in
\headsep=0.25in

\linespread{1.1}

\pagestyle{fancy}
\chead{Globally Ordered Type System}
\lfoot{\lastxmark}
\cfoot{\thepage}

\renewcommand\headrulewidth{0.4pt}
\renewcommand\footrulewidth{0.4pt}

\setlength\parindent{30pt}

\newcommand{\Z}{\mathbb{Z}}
\newcommand{\Zt}{$\Z$}

% Create a relational rule
% [#1] - Additional mathpartir arguments
% {#2} - Name of the rule
% {#3} - Premises for the rule
% {#4} - Conclusions for the rule
\newcommand{\relationRule}[4][]{\inferrule*[lab={\sc #2},#1]{#3}{#4}}

\newcommand{\rel}[1]{\ensuremath{\llbracket {#1} \rrbracket}}
\newcommand{\ttt}{\texttt}
\newcommand{\transform}{\rightsquigarrow}
\newcommand{\proj}{\pi}
\newcommand{\ttuple}{(\tau_1, \ldots, \tau_n)}
\newcommand{\etuple}{(e_1,\ldots,e_n)}
\newcommand{\bool}{\mathrm{bool}}
\newcommand{\integer}{\mathrm{int}}
\newcommand{\option}{\mathrm{option}}
\newcommand{\uoption}[1]{(\bool,#1)}
\newcommand{\opt}[1]{\texttt{opt-#1}}
\newcommand{\varOf}[1]{\texttt{varOf}(#1)}

\newcommand{\oktype}{valid}

\begin{document}
\section*{The System}
We begin by defining a grammar of "base types". These are the types which are eligible to be stored in global variables. The particular set of base types is not important, but we believe the following ones give a good overview.

\begin{grammar}
	<$T$ (base types)> ::= Unit | Int | Bool | T * T
\end{grammar}

Everything in the rest of this document is defined with respect to an ordered set $\mathbb{G} = \{g_1, \dots, g_n\}$ of \emph{global variables}, each of which has an associated base type $T_i$. Our goal is to enforce that $g_1$ is always used before $g_2$, $g_2$ before $g_3$, and so on. For simplicity, we require that each global variable be used; however, we will later relax this restriction by adding subtyping.

In our model, a global variable is considered \emph{used} until it is actually accessed. Note that this is different from a normal ordered type system, where any appearance of the variable is considered a use. To distinguish appearances from memory accesses, we adopt ref-cell-like syntax for dereferencing or updating global variables. Note that this is different from the user model in actual dpt, where global variables are viewed as simple values. Instead of the ref-cell operators, dpt contains several built-in functions with appropriate type signatures.

The \emph{effects} $\epsilon$ in this system specify which global variables have been used so far. Effects are represented as an integer from $1$ to $n+1$, where effect $i$ means that all global variables up to but not including $g_i$ have been used so far (that is, we expect to use $g_i$ next).

We will also wish to allow users to define functions which are polymorphic in effect; for example, a function which takes in any global variable, dereferences it, and returns the value. Such a function would have an "effect signature" of something like $\alpha \rightarrow \alpha + 1$. To accommodate this, we will add polymorphic variables and a $+$ operation to our effect grammar.

Our effects are also intertwined with the language's value types $\tau$, in two ways. First, we have a special $\texttt{ref} (T, \epsilon)$ type for representing the value of a global variable. Second, function types must be altered to include their effect on the global variables. We do so by adding input and output effects to function types as well, using the form $(\tau, \epsilon) \rightarrow (\tau, \epsilon)$.

All in all, we have the following grammar of types and effects.

\begin{grammar}
	<$\epsilon$ (effects)> ::= 1 | 2 | \dots | n + 1 | $\alpha$ | $\epsilon + \epsilon$

	<$\tau$ (types)> ::= T | $\tau * \tau$ | ref ($\tau, \epsilon$) | $\forall \alpha.(\tau, \epsilon) \rightarrow (\tau, \epsilon)$
\end{grammar}

Note that we use the convention that when representing effects, $\epsilon$ represents any effect, Latin letters represent only integer effects, and Greek letters represent effect variables.

Next, we define a simple functional language to use this system on:

\begin{grammar}
	<$x$ (variables)> ::= alphanumeric

	<$\Gamma$ (environments)> ::= Maps from variables to values

	<$v$ (values)> ::= () | $\Z$ | True | False | $(v, v)$ | \textless$\Gamma, \alpha, (x : \tau, \epsilon), e $\textgreater\ | $g_1$ | \dots | $g_n$

	<$e$ (expressions)> ::= $v$ | $x$ | $e + e$ | $(e,e)$ | fst $e$ | snd $e$ | let $x$ = $e$ in $e$ | if $e$ then $e$ else $e$ | !$e$ | $e := e$
	\alt $\texttt{fun}\ [\alpha]\ (x:\tau, \epsilon)  \rightarrow e$ | $e[\epsilon]$ $e$
\end{grammar}

Note that we treat the global variables as values in their own right in this system.

The $\texttt{fun}\ [\alpha]$ syntax defines an effect-polymorphic function whose effects may use the universally quantified effect variable $\alpha$. The $e[\epsilon]$ $e$ syntax applies an effect-polymorphic function, instantiating $\alpha$ with $\epsilon$.

\clearpage
\section*{Typing Programs}
\subsection*{Effect Equivalence}
One may notice that our effect grammar allows us to write logically equivalent effects in different ways: for example, $1+1$ or $2$. To avoid this complication, we define an equivalence relation $\equiv$ on effects as follows, using $\oplus$ to represent integer addition:

\begin{mathpar}
	\relationRule{effect-refl}{
		\
	}{
		\epsilon \equiv \epsilon
	}

	\relationRule{effect-sym}{
		\epsilon_2 \equiv \epsilon_1
	}{
		\epsilon_1 \equiv \epsilon_2
	}

	\relationRule{effect-trans}{
		\epsilon_1 \equiv \epsilon_2\\
		\epsilon_2 \equiv \epsilon_3
	}{
		\epsilon_1 \equiv \epsilon_3
	}

	\relationRule{int-plus}{
		i \oplus j = k
	}{
		i+j \equiv k
	}
\\
	\relationRule{plus-comm}{
		\
	}{
		\epsilon_1 + \epsilon_2 \equiv \epsilon_2 + \epsilon_1
	}

	\relationRule{plus-assoc}{
		\
	}{
		(\epsilon_1 + \epsilon_2) + \epsilon_3 \equiv \epsilon_1 + (\epsilon_2 + \epsilon_3)
	}

	\relationRule{plus-cong}{
		\epsilon_1 \equiv \epsilon_3\\
		\epsilon_2 \equiv \epsilon_4
	}{
		\epsilon_1 + \epsilon_2 \equiv \epsilon_3 + \epsilon_4
	}
\end{mathpar}

\subsection*{Effect Substitution}
In order to define our typing relation, we will need to be able to perform substitution on types to eliminate $\alpha$s. To do so, we define a standard capture-avoiding substitution relation $[\epsilon/\alpha]$, defined below.

\begin{align*}
	\alpha[\epsilon/\alpha] & = \epsilon\\
	\beta[\epsilon/\alpha] & = \beta\ (\mbox{if }\beta \neq \alpha)\\
	i[\epsilon/\alpha] & = i\\
	(\epsilon_1 + \epsilon_2)[\epsilon/\alpha] & = \epsilon_1[\epsilon/\alpha] + \epsilon_2[\epsilon/\alpha]
\\\hfill\\
	T[\epsilon/\alpha] & = T\\
	(\tau_1 * \tau_2)[\epsilon/\alpha] & = \tau_1[\epsilon/\alpha] * \tau_2[\epsilon/\alpha]\\
	\texttt{ref } (\tau, \epsilon_1)[\epsilon/\alpha] & = \texttt{ref } (\tau[\epsilon/\alpha], \epsilon_1[\epsilon/\alpha])\\
	\left(\forall \alpha.(\tau_1, \epsilon_1) \rightarrow (\tau_2, \epsilon_2)\right)[\epsilon/\alpha] & = \forall \alpha.(\tau_1, \epsilon_1) \rightarrow (\tau_2, \epsilon_2)\\
	\left(\forall \beta.(\tau_1, \epsilon_1) \rightarrow (\tau_2, \epsilon_2)\right)[\epsilon/\alpha] & = \forall \beta.(\tau_1[\epsilon/\alpha], \epsilon_1[\epsilon/\alpha]) \rightarrow (\tau_2[\epsilon/\alpha], \epsilon_2[\epsilon/\alpha])\ (\mbox{if }\beta \neq \alpha \mbox{ and } \beta \mbox{ does not appear in } \epsilon)
\end{align*}

\subsection*{Effect Ordering}
We define a standard $\leq$ order on effects so we can compare them, using the $\preceq$ symbol to indicate integer less-than-or-equals.

\begin{mathpar}
	\relationRule{leq-refl}{
		\
	}{
		\epsilon \leq \epsilon
	}

	\relationRule{leq-trans}{
		\epsilon_1 \leq \epsilon_2\\
		\epsilon_2 \leq \epsilon_3
	}{
		\epsilon_1 \leq \epsilon_3
	}

	\relationRule{leq-equiv}{
		\epsilon_1 \equiv \epsilon_2
	}{
		\epsilon_1 \leq \epsilon_2
	}

	\relationRule{leq-int}{
		i \preceq j
	}{
		i \leq j
	}

	\relationRule{leq-plus-1}{
		\epsilon_1 \leq \epsilon_3\\
		\epsilon_2 \leq \epsilon_4
	}{
		\epsilon_1 + \epsilon_2 \leq \epsilon_3 + \epsilon_4
	}

	\relationRule{leq-plus-2}{
		\
	}{
		\epsilon_1 \leq \epsilon_1 + \epsilon_2
	}
\end{mathpar}

\section*{The Typing Relation}
Type inference in this system simultaneously reasons about effects and regular types. As a result, the typing relation has an extra input and output, representing the state of the global variables before and after the expression, respectively.

Our typing relation will thus have the form $\Gamma, \epsilon \vdash e\ \colon \tau, \epsilon$. As usual, $\Gamma$ represents our current collection of bound variables; we treat it as a set. Our goal is that our program $p$ satisfies $\emptyset, 1 \vdash p\ \colon \tau, n+1$ for some $\tau$.

When writing rules, we use the convention that the metavariable $x$ matches all \emph{non-global} variables. We will always denote global variables using symbols of the form $g_i$. For the rest of these rules we will treat effects as being equal if they are equivalent.

Definition: a type $\tau$ is said to be \emph{\oktype} if all sub-parts of $\tau$ are \oktype, and if $\tau = \forall \alpha.(\tau_1, \epsilon_{in}) \rightarrow (\tau_2, \epsilon_{out})$ implies $\epsilon_{in} \leq \epsilon_{out}$.

For a given value environment $\Gamma$, we define $\Gamma_\tau$ to be any typing environment with the same elements as $\Gamma$ and such that $\forall y \in \Gamma. \emptyset, 1 \vdash \Gamma[y]\ \colon \Gamma_\tau[y], 1$.

$\texttt{fun } [\alpha, \beta] (f : (\forall \gamma. (\tau, \alpha) \rightarrow (\tau', \beta)) = f\ ...$

\begin{mathpar}
	\relationRule{int}{
		n \in \Z
	}{
		\Gamma, \epsilon \vdash n\ \colon \texttt{Int}, \epsilon
	}

	\relationRule{true}{
		\
	}{
		\Gamma, \epsilon \vdash \texttt{True}\ \colon \texttt{Bool}, \epsilon
	}

	\relationRule{false}{
		\
	}{
		\Gamma, \epsilon \vdash \texttt{False}\ \colon \texttt{Bool}, \epsilon
	}

	\relationRule{Unit}{
		\
	}{
		\Gamma, \epsilon \vdash ()\ \colon \texttt{Unit}, \epsilon
	}

	\relationRule{global variable}{
		\
	}{
		\Gamma, \epsilon \vdash g_i\ \colon \texttt{ref}(T_i, i), \epsilon
	}

	\relationRule{Closure}{
		\tau\mbox{ is \oktype}\\
		\Gamma_\tau[x := \tau], \epsilon_f \vdash e\ \colon \tau_1, \epsilon_1
	}{
		\Gamma_0, \epsilon \vdash\ <\Gamma, \alpha, (x : \tau, \epsilon_f), e >\ \colon \forall \alpha.(\tau, \epsilon_f) \rightarrow (\tau_1, \epsilon_1), \epsilon
	}
\end{mathpar}

Now we can write rules for non-value expressions:

\begin{mathpar}
	\relationRule{local variable}{
		\Gamma[x] = \tau
	}{
		\Gamma, \epsilon \vdash x\ \colon \tau, \epsilon
	}

	\relationRule{plus}{
		\Gamma, \epsilon \vdash e_1\ \colon \texttt{Int}, \epsilon_1\\
		\Gamma, \epsilon_1 \vdash e_2\ \colon \texttt{Int}, \epsilon_2
	}{
		\Gamma, \epsilon \vdash e_1 + e_2\ \colon \texttt{Int}, \epsilon_2
	}

	\relationRule{pair}{
		\Gamma, \epsilon \vdash e_1\ \colon \tau_1, \epsilon_1\\
		\Gamma, \epsilon_1 \vdash e_2\ \colon \tau_2, \epsilon_2
	}{
		\Gamma, \epsilon \vdash (e_1, e_2)\ \colon \tau_1 * \tau_2, \epsilon_2
	}

	\relationRule{fst}{
		\Gamma, \epsilon \vdash e\ \colon \tau_1 * \tau_2, \epsilon_1\\
	}{
		\Gamma, \epsilon \vdash \texttt{fst } e\ \colon \tau_1, \epsilon_1
	}

	\relationRule{snd}{
		\Gamma, \epsilon \vdash e\ \colon \tau_1 * \tau_2, \epsilon_1\\
	}{
		\Gamma, \epsilon \vdash \texttt{snd } e\ \colon \tau_2, \epsilon_1
	}

	\relationRule{let}{
		\Gamma, \epsilon \vdash e_1\ \colon \tau_1, \epsilon_1\\
		\Gamma[x := \tau_1], \epsilon_1 \vdash e_2\ \colon \tau_2, \epsilon_2
	}{
		\Gamma, \epsilon \vdash \texttt{let } x = e_1 \texttt{ in } e_2\ \colon \tau_2, \epsilon_2
	}

	\relationRule{if}{
		\Gamma, \epsilon \vdash e_1\ \colon \texttt{Bool}, \epsilon_1\\
		\Gamma, \epsilon_1 \vdash e_2\ \colon \tau, \epsilon_2\\
		\Gamma, \epsilon_1 \vdash e_3\ \colon \tau, \epsilon_2
	}{
		\Gamma, \epsilon \vdash \texttt{if } e_1 \texttt{ then } e_2 \texttt{ else } e_3\ \colon \tau, \epsilon_2
	}
\end{mathpar}

Now let's finally write some rules that actually change $\epsilon$.

\begin{mathpar}
	\relationRule{deref}{
		\Gamma, \epsilon \vdash e\ \colon \texttt{ref}(\tau, \epsilon_1), \epsilon_1
	}{
		\Gamma, \epsilon \vdash !e\ \colon \tau, \epsilon_1+1
	}

\relationRule{update}{
	\Gamma, \epsilon \vdash e_1\ \colon \tau, \epsilon_1\\
	\Gamma, \epsilon_1 \vdash e_2\ \colon \texttt{ref}(\tau, \epsilon_2), \epsilon_2
}{
	\Gamma, \epsilon \vdash e_2 := e_1\ \colon \texttt{Unit}, \epsilon_2+1
}
\end{mathpar}

And finally, the function rules:
\begin{mathpar}
	\relationRule{abs}{
		\tau\mbox{ is \oktype}\\
		\Gamma[x := \tau], \epsilon_f \vdash e\ \colon \tau_1, \epsilon_1
	}{
		\Gamma, \epsilon \vdash \texttt{fun } [\alpha] (x : \tau, \epsilon_f) \rightarrow e\ \colon \forall \alpha.(\tau, \epsilon_f) \rightarrow (\tau_1, \epsilon_1), \epsilon
	}

	\relationRule{app}{
		\Gamma, \epsilon \vdash e_1\ \colon \forall \alpha.(\tau_2, \epsilon_2) \rightarrow (\tau', \epsilon'), \epsilon_1\\
		\Gamma, \epsilon_1 \vdash e_2\ \colon \tau_2[\epsilon_\alpha/\alpha], \epsilon_2[\epsilon_\alpha/\alpha]\\
	}{
		\Gamma, \epsilon \vdash e_1[\epsilon_\alpha]\ e_2\ \colon \tau'[\epsilon_\alpha/\alpha], \epsilon'[\epsilon_\alpha/\alpha]
	}
\end{mathpar}

\subsection*{Skipping Variables}
In practice, the requirement that we \emph{must} use each global variable more restrictive than we would like. We would like to programs in which some global variables are skipped. We do this by adding a subtyping relation on effects and types.

\begin{mathpar}
	\relationRule{sub-refl}{
		\
	}{
		\tau <: \tau
	}

	\relationRule{sub-pair}{
		\tau_1 <: \tau_3\\
		\tau_2 <: \tau_4\\
	}{
		(\tau_1, \tau_2) <: (\tau_3, \tau_4)
	}

	\relationRule{sub-function}{
		\tau_1' <: \tau_1\\
		\tau_2 <: \tau_2'\\
		\epsilon_1' \leq \epsilon_1 \leq \epsilon_2 \leq \epsilon_2'
	}{
		\forall \alpha.(\tau_1, \epsilon_1) \rightarrow (\tau_2, \epsilon_2) <: \forall \alpha.(\tau_1', \epsilon_1') \rightarrow (\tau_2', \epsilon_2')
	}
\end{mathpar}

Finally, we augment our typing relation with the single rule
\begin{mathpar}
	\relationRule{subtyping}{
		\tau_1 <: \tau_2\\
		\epsilon_1 \leq \epsilon_2\\
		\Gamma, \epsilon \vdash e\ \colon \tau_1, \epsilon_1
	}{
		\Gamma, \epsilon \vdash e\ \colon \tau_2, \epsilon_2
	}
\end{mathpar}

Note that we only really have two uses for this new typing rule: replacing functions, and incrementing the effect.

\section*{Properties of the Typing Relation}

\subsection*{Transitivity of subtyping}
Theorem: If $\tau_1 <: \tau_2$ and $\tau_2 <: \tau_3$ then $\tau_1 <: \tau_3$.
\\

\noindent Proof: Straightforward structural induction on the proof that $\tau_1 <: \tau_2$, using inversion on the proof that $\tau_2 <: \tau_3$.

\subsection*{A shorthand for values}
Definition: We will write $v : \tau$ as shorthand for $\forall\ \Gamma, \epsilon.\ \Gamma, \epsilon \vdash v\ \colon \tau, \epsilon$.
\\

\noindent Theorem: If $\Gamma, \epsilon \vdash v\ \colon \tau, \epsilon'$ then $v : \tau$.
\\

\noindent Proof: Straightforward structural induction on the typing derivation. If we used a value rule we may immediately reapply it. In the PAIR and SUBTYPING cases the result follows quickly by induction.

\subsection*{Monotonicity}
In our later proofs, we will need to use a monotonicity property stating that the effect only ever increases when we make typing judgements. Before we can prove that, though, we need some lemmas about substitution and the \oktype\ property.
\\

\noindent Lemma M-1: For all $\epsilon, \epsilon_1, \epsilon_2, \alpha$, if $\epsilon_1 \equiv \epsilon_2$, then $\epsilon_1[\epsilon/\alpha] \equiv \epsilon_2[\epsilon/\alpha]$

\noindent Proof: Straightforward structural induction on the proof that $\epsilon_1 \equiv \epsilon_2$.
\\

\noindent Lemma M-2: For all $\epsilon, \epsilon_1, \epsilon_2, \alpha$, if $\epsilon_1 \leq \epsilon_2$, then $\epsilon_1[\epsilon/\alpha] \leq \epsilon_2[\epsilon/\alpha]$

\noindent Proof: Straightforward structural induction on the proof that $\epsilon_1 \leq \epsilon_2$, using lemma M-1 in the \texttt{leq-equiv} case.
\\

\noindent Lemma M-3: if $\tau$ is \oktype, then for all $\epsilon, \alpha$, $\tau[\epsilon/\alpha]$ is also \oktype.

\noindent Proof: Straightforward structural induction on $\tau$, using lemma M-2 in the function case.
\\

\noindent Lemma M-4: if $\tau_1 <: \tau_2$, then $\tau_1$ is \oktype\ if and only if $\tau_2$ is \oktype.

\noindent Proof: Straightforward structural induction on the proof that $\tau_1 <: \tau_2$.
\\

\noindent Theorem: If $\Gamma, \epsilon_1 \vdash e\ \colon \tau, \epsilon_2$ and all elements of $\Gamma$ are \oktype\, then $\tau$ is \oktype\ and $\epsilon_1 \leq \epsilon_2$.

\noindent We don't actually care about $\tau$ being \oktype\ (although it's a nice result), but we need it during the proof
\\

\noindent Proof of Theorem: Straightforward induction on the typing derivation. The interesting bits are:
\begin{itemize}
	\item In the CLOSURE rule, we show inductively that all elements of $\Gamma_\tau$ are \oktype.
	\item In the LOCAL VARIABLE rule, we use the fact that all elements of $\Gamma$ are \oktype.
	\item In the GLOBAL VARIABLE rule, use the fact that all base types are valid.
	\item In the ABS case we need to do a little work to show that the output is \oktype.
	\item In the APP case we use lemmas M-2 and M-3.
	\item In the SUBTYPING case we use lemma M-4.
\end{itemize}

\subsection*{Canonical Forms}
Lemma C-1: If $\tau_1 <: \tau_2$, then:
\begin{itemize}
	\item If either $\tau_1$ or $\tau_2$ is a base type or $\texttt{ref}$ type then $\tau_1 = \tau_2$
	\item If either $\tau_1$ or $\tau_2$ is a pair type, then $\tau_1 = (\tau_a, \tau_b)$ and $\tau_2 = (\tau_c, \tau_d)$ where $\tau_a <: \tau_c$ and $\tau_b <: \tau_d$.
	\item If either $\tau_1$ or $\tau_2$ is a function type, then $\tau_1 = \forall \alpha. (\tau_a, \epsilon_a) \rightarrow (\tau_b, \epsilon_b)$ and $\tau_2 = \forall \alpha. (\tau_c, \epsilon_c) \rightarrow (\tau_d, \epsilon_d)$ where $\tau_c <: \tau_a$, $\tau_b <: \tau_d$, and $\epsilon_a \leq \epsilon_c \leq \epsilon_b \leq \epsilon_d$.
\end{itemize}

\noindent Proof: Inversion of the typing relation.
\\

\noindent Theorem: For all values $v$, if $v\ \colon \tau$ then
\begin{itemize}
	\item If $\tau = \texttt{Int}$ then $v \in \Z$
	\item If $\tau = \texttt{Bool}$ then $v = \texttt{True}$ or $v = \texttt{False}$
	\item If $\tau = \texttt{Unit}$ then $v = ()$
	\item If $\tau = \tau_1 * \tau_2$ then $v = (v_1, v_2)$ where $v_1 : \tau_1$ and $v_2 : \tau_2$.
	\item If $\tau = \texttt{ref} (\tau_1, \epsilon)$ then $\epsilon = i \in \Z$, $v = g_i$ and $\tau_1 = T_i$.
	\item If $\tau = \forall \alpha.(\tau_1, \epsilon_1) \rightarrow (\tau_2, \epsilon_2)$ then $v =\ <\Gamma, \alpha, (x : \tau', \epsilon'), e>$ for some $\Gamma, \tau', \epsilon', e$.
\end{itemize}

\noindent Proof: Case analysis on the proof that $v$ has type $\tau$ for some arbitrary $\Gamma, \epsilon$. Each $\tau$ has only two (or three, for booleans) that apply to a value and produce it: the base rules, and the subtyping rule. In the subtyping cases, we use Lemma C-1. In the pair case, we use induction, and in the case where $\tau$ is a pair type and the subtyping rule was used, we Lemma C-1 twice in addition to induction.

\subsection*{The Substitution Lemma}

\noindent Definition: If $\Gamma$ is a map, let $\Gamma \backslash x$ indicate the map which is identical to $\Gamma$ but contains no binding for $x$.
\\

\noindent Definition: Let $e[v/x]$ be the standard capture-avoiding substitution relation. It differs from the effect one in that it operates on expressions/values/variables instead of effects and effect variables. We also define it to be the identity function if $e$ is a value.
\\

\noindent Theorem: If $\Gamma[x] = \tau$, $v : \tau$, and $\Gamma, \epsilon \vdash e\ \colon \tau, \epsilon'$, then $\Gamma\backslash x, \epsilon \vdash e[v/x]\ \colon \tau, \epsilon'$

\noindent Proof: Structural induction on first typing proof. The interesting cases are:

\begin{itemize}
	\item In the VAR case $e = z$, either $z = x$ in which case $e[v/x] = v$ and the premise $v : \tau$ solves it, or $z <> x$ in which case $z \in \Gamma\backslash x$ and we simply use the VAR rule.
	\item In the LET case $e = \texttt{let } z = e_1 \texttt{ in }e_2$ either $z = x$ in which case $\Gamma\backslash z[x := \tau_1] = \Gamma[x := \tau_1]$ and we do not substitute in $e_2$, or $z <> x$ in which case $\Gamma\backslash z[x := \tau_1] = \Gamma[x := \tau_1]\backslash z$ and we do substitute in $e_2$. In either case the result follows immediately.
	\item Similar reasoning works for the ABS case.
	\item The subtyping case is immediate, but it bears noting that we don't change any of the types or effects that appear in the original derivation.
\end{itemize}

\clearpage
\section*{Operational Semantics}

Now we'll define an operational semantics for our language. We are modeling a situation in which global variables are ordered in a pipeline, such that each global variable must be accessed after the ones before it in the pipeline.

Our big-step operational semantics relation will have the form $(\Gamma, G, n, e) \Rightarrow (G', n', e')$, where $\Gamma$ is an environment mapping variables to values, $G$ is an array $[v_1; \dots; v_n]$ of the current values for each global variable, $n$ is an integer indicating the current pipeline stage, and $e$ is an expression to be evaluated. We treat $G$ as a map from integers to values, so $G[1] = v_1$, etc. We say $G$ is well-typed if $G[i] : T_i$ for $1 \leq i \leq n$.

As usual, the metavariable $v$ will refer exclusively to values, while $e$ represents any expression. We use $\oplus$ to denote regular integer addition.

\subsection*{Big-step}

\begin{mathpar}
	\relationRule{value}{
		\\
	}{
		(\Gamma, G, n, v) \Rightarrow (G, n, v)
	}

	\relationRule{variable}{
		\Gamma[x] = v
	}{
		(\Gamma, G, n, x) \Rightarrow (G, n, v)
	}

	\relationRule{sum}{
		(\Gamma, G, n, e_1) \Rightarrow (G_1, n_1, i_1)\\
		(\Gamma, G_1, n_1, e_2) \Rightarrow (G_2, n_2, i_2)\\
		i_1, i_2 \in \Z
	}{
		(\Gamma, G, n, (e_1, e_2)) \Rightarrow (G_2, n_2, i_1 \oplus i_2)
	}

	\relationRule{pair}{
		(\Gamma, G, n, e_1) \Rightarrow (G_1, n_1, v_1)\\
		(\Gamma, G_1, n_1, e_2) \Rightarrow (G_2, n_2, v_2)
	}{
		(\Gamma, G, n, (e_1, e_2)) \Rightarrow (G_2, n_2, (v_1, v_2))
	}

	\relationRule{fst}{
		(\Gamma, G, n, e) \Rightarrow (G_1, n_1, (v_1, v_2))
	}{
		(\Gamma, G, n, \texttt{fst } e) \Rightarrow (G_1, n_1, v_1)
	}

	\relationRule{snd}{
		(\Gamma, G, n, e) \Rightarrow (G_1, n_1, (v_1, v_2))
	}{
		(\Gamma, G, n, \texttt{snd } e) \Rightarrow (G_1, n_1, v_2)
	}

	\relationRule{let}{
		(\Gamma, G, n, e_1) \Rightarrow (G_1, n_1, v_1)\\
		(\Gamma[x := v_1], G_1, n_1, e_2) \Rightarrow (G_2, n_2, v_2)\\
	}{
		(\Gamma, G, n, \texttt{let } x = e_1 \texttt{ in } e_2) \Rightarrow (G_2, n_2, v_2)
	}

	\relationRule{if-true}{
		(\Gamma, G, n, e_1) \Rightarrow (G_1, n_1, \texttt{True})\\
		(\Gamma, G_1, n_1, e_2) \Rightarrow (G_2, n_2, v)\\
	}{
		(\Gamma, G, n, \texttt{if } e_1 \texttt{ then } e_2 \texttt{ else } e_3) \Rightarrow (G_1, n_2, v)
	}

	\relationRule{if-false}{
		(\Gamma, G, n, e_1) \Rightarrow (G_1, n_1, \texttt{False})\\
		(\Gamma, G_1, n_1, e_3) \Rightarrow (G_2, n_2, v)\\
	}{
		(\Gamma, G, n, \texttt{if } e_1 \texttt{ then } e_2 \texttt{ else } e_3) \Rightarrow (G_1, n_2, v)
	}

	\relationRule{deref}{
		(\Gamma, G, n, e_1) \Rightarrow (G_1, n_1, g_i)\\
		n_1 <= i
	}{
		(\Gamma, G, n, !e) \Rightarrow (G_1, i+1, G_1[i])
	}

	\relationRule{update}{
		(\Gamma, G, n, e_1) \Rightarrow (G_1, n_1, v_1)\\
		(\Gamma, G_1, n_1, e_2) \Rightarrow (G_2, n_2, g_i)\\
		n_2 <= i
	}{
		(\Gamma, G, n, e_2 := e_1) \Rightarrow (G_2[i := v_1], i+1, ())
	}

	\relationRule{abs}{
		\
	}{
		(\Gamma, G, n, \texttt{fun } [\alpha]\ (x : \tau, \epsilon) \rightarrow e) \Rightarrow (G, n, < \Gamma, \alpha, (x : \tau, \epsilon), e>)
	}

	\relationRule{app}{
		(\Gamma, G, n, e_1) \Rightarrow (G_1, n_1, < \Gamma', \alpha, (x : \tau, \epsilon), e>)\\
		(\Gamma, G_1, n_1, e_2) \Rightarrow (G_2, n_2, v_2)\\
		(\Gamma'[x := v_2], G_2, n_2, e) \Rightarrow (G_3, n_3, v_3)\\
	}{
		(\Gamma, G, n, e_1\ e_2) \Rightarrow (G_3, n_3, v_3)
	}
\end{mathpar}

\subsection*{Big-step weakening}

During our soundness proof, we will need to use the following theorem, which states that if a program evaluates from a state then it also evaluates from every earlier state.
\\

\noindent Theorem: If $(\Gamma, G, n, e) \Rightarrow (G', n', v)$ and $m \leq n$ then $(\Gamma, G, m, e) \Rightarrow (G', m', v)$ for some $m' \leq n'$.
\\

\noindent Proof: Straightforward induction on the proof of evaluation. The only mildly interesting cases are DEREF and UPDATE, where we use transitivity of $\leq$ and the fact that $i+1 \leq i+1$.

\subsection*{Big-step soundness}

Since our language (and also dpt!) doesn't allow recursion or looping, we can show not only that well-typed programs do not get stuck, but that they terminate (i.e. well-typed programs are normalizing).
\\

\newcommand{\okvalue}{good}
\noindent Definition: A value is \emph{\okvalue} if whenever $v : \forall \alpha.(\tau_{in}, \epsilon_{in}) \rightarrow (\tau_{out}, \epsilon_{out})$, we know that $v =\ <\Gamma, \alpha, (x : \tau_{in}, \epsilon_{in}), e>$ and for all $G, i, v_x$ where $G$ is well-typed and $v_x : \tau_{in}[i/\alpha]$, $(\Gamma_\tau[x := v_x], G, \epsilon[i/\alpha], e) \Rightarrow (G', i', v)$ where $G'$ is well-typed, $i' \leq \epsilon_{out}[i/\alpha]$, and $v : \tau_{out}[i/\alpha]$ and $v$ is \okvalue.
\\

\noindent Definition: for all typing environments $\Gamma$, let $\Gamma_v$ denote any evaluation environment such that for all $x \in \Gamma$, $\Gamma_v[x] = v_x$ where $v_x : \Gamma[x]$ and $v_x$ is \okvalue.
\\

\noindent Desired Theorem: \\\indent If $\emptyset, 1 \vdash e\ \colon \tau, n+1 $ then for all $G$ there exist some $G', n', v$ such that $(\emptyset, G, 1, e) \Rightarrow (G', n', v)$
\\

\noindent Unfortunately this won't give us a useful induction hypothesis, so we must prove a generalization instead:
\\

\noindent Generalized Theorem: \\\indent If $\Gamma, i \vdash e\ \colon \tau, j$ then for all well-typed $G$ there exist some $G', i', v$ such that $v$ is \okvalue, $v : \tau$, $i' \leq i$, $G'$ is well-typed, and $(\Gamma_v, G, i, e) \Rightarrow (G', i', v)$.
\\

\noindent Fortunately it is immediately obvious that the general theorem implies the desired one. For now we'll prove it assuming there's no polymorphism; that is, no effects anywhere in the program or judgements include any type variables. To prove it with polymorphism, we would probably have to instantiate any type variables in the judgement and do so in a consistent way, and then say that $e$ evaluates for all such instantiations.

Roughly, something like "Let $[\alpha_1, \dots, \alpha_k]$ be the set of type variables appearing inside $e$. For any set of integer effects $\mathbb{I} = \{i_1, \dots i_k\}$, let $e_\mathbb{I}$ denote $e[i_1/\alpha_1]...[i_k/\alpha_k]$, and similarly for other things. Then if $\Gamma_\mathbb{I}, (\epsilon_1)_\mathbb{I} \vdash e_\mathbb{I} : \tau_\mathbb{I}, (\epsilon_2)_\mathbb{I}$ then $(\Gamma_v, G, (\epsilon_1)_\mathbb{I}, e_\mathbb{I}) \Rightarrow (G', i', v)$ where $i' \leq (\epsilon_2)_\mathbb{I}$, and $v : \tau_\mathbb{I}$ and $G'$ is well-typed and $v$ is \okvalue.
\\

\noindent Proof of Generalized Theorem: Induction on the proof that $\Gamma, i \vdash e\ \colon \tau, j$.
\\

Case $\texttt{Int/True/False/Global Variable}$. In these cases, $e$ is a value, so we may apply the value rule to show that $(\Gamma_v, G, i, e) \Rightarrow (G, i, v)$, and we can show $v : \tau$ using the appropriate rule appearing in the title of the case. Note that we won't need to use subtyping when we do so.
\\

Case $\texttt{Local Variable}$. Our premise gives us that $\Gamma[x] = \tau$. Hence by definition, $\Gamma_v[x] = v_x$, $v_x : \tau$, and $v_x$ is \okvalue. Thus we may use the \texttt{Variable} rule to show that $(\Gamma_v, G, i, e) \Rightarrow (G, i, v_x)$.
\\

Case $\texttt{Pair}$.
From our premise:
\begin{itemize}
	\item $e = e_1\ e_2$
	\item $\Gamma, i \vdash e_1\ \colon \tau_1, \epsilon_1$
	\item $\Gamma, \epsilon_1 \vdash e_2\ \colon \tau_2, j$
	\item IH1: $(\Gamma_v, G, i, e_1) \Rightarrow (G_1, i_1, v_1) \wedge i_1 \leq \epsilon_1 \wedge v_1 : \tau_1 \wedge G_1 \mbox{ is well-typed}$
	\item IH2: $(\Gamma_v, G_1, \epsilon_1, e_2) \Rightarrow (G_2, i_2, v_2) \wedge i_2 \leq \epsilon_2 \wedge v_2 : \tau_2 \wedge G_2 \mbox{ is well-typed}$
\end{itemize}

This is almost enough to apply the PAIR evaluation rule, but unfortunately that rule requires the output of the first to match the input of the second. Fortunately since we know that $i_1 \leq \epsilon_1$, we may use weakening to show that $(\Gamma_v, G_1, i_1, e_2) \Rightarrow (G_2, i_2', v_2)$ for some $i_2' \leq i_2$, and now we may use the PAIR rule to conclude that $(\Gamma_v, G, i, e_1) \Rightarrow (G_2, i_2', (v_1, v_2))$. Next, we note that $i_2' \leq i_2 \leq \epsilon_2$, and that $(v_1, v_2) : \tau_1 * \tau_2$ can be easily shown using the PAIR typing rule plus the fact that $v_1 : \tau_1$ and $v_2 : \tau_2$. Finally, note that $G_2$ being well typed is part of IH2, since $G_1$ is well-typed.
\\

Case PLUS: Same as the PAIR case, but we also need to use our canonical forms lemma to satisfy the prerequisite of SUM that both values be integers.
\\

Case FST: $e = \texttt{fst } e_1$. We know that $\Gamma, i \vdash e_1\ \colon \tau_1 * \tau_2, \epsilon_1$, so by induction $(\Gamma_v, G, i, e) \Rightarrow (G', i', v)$, where $v : \tau_1 * \tau_2$, $i' \leq i$, and $G'$ is well-typed. Our canonical forms lemma lets us conclude that $v = (v_1, v_2)$ and $v_1 : \tau_1$, so we may apply the FST rule to show that $(\Gamma_v, G, i, \texttt{fst } e) \Rightarrow (G', i', v_1)$. That was all that remained to be shown.
\\

Case SND: Analogous to FST.
\\

Case LET: $e = \texttt{let } x = e_1 \texttt{ in } e_2$. We know that $\Gamma, i \vdash e_1\ \colon \tau_1, \epsilon_1$ and $\Gamma[x := \tau_1], \epsilon_1 \vdash e_2\ \colon \tau_2, \epsilon_2$. Thus by induction we get
$$(\Gamma_v, G, i, e_1) \Rightarrow (G_1, i_1, v_1)\mbox{ and }(\Gamma[x := \tau_1]_v, G_1, \epsilon_1, e_2) \Rightarrow (G_2, i_2, v_2),$$
where $i_1 \leq \epsilon_1$, $i_2 \leq \epsilon_2$, $v_1 : \tau_1$ and $v_2 : \tau_2$. We note that by induction, $v_1$ is \oktype, and since $v_1 : \tau_1$ and $\tau_1 = \Gamma[x := \tau_1][x]$, $\Gamma_v[x := v_1]$ is a valid instance of $\Gamma[x := \tau_1]_v$. We also apply weakening to our second hypothesis as in the PAIR case so that it begins evaluation at $i_1$ instead of $\epsilon_1$. This gives us everything we need to apply the LET rule and show that $(\Gamma_v, G, i, e) \Rightarrow (G_2, i_2, v_2)$.
\\

Case IF: By induction we know that $e$ evaluates to a value $v$ and $v : \texttt{Bool}$, and hence by canonical forms $v = \texttt{True}$ or $v = \texttt{False}$. We can then apply weakening to the other hypotheses and then use IF-TRUE or IF-FALSE as appropriate.
\\

Case DEREF: We know that
\begin{itemize}
	\item DEREF: $\Gamma, i \vdash e\ \colon \tau, \epsilon_1+1$
	\item INV1: $e = !e_1$
	\item INV2: $\Gamma, i \vdash e_1\ \colon \texttt{ref}(\tau, \epsilon_1), \epsilon_1$
	\item IH: $(\Gamma_v, G, i, e_1) \Rightarrow (G', i', v) \wedge i' \leq \epsilon_1 \wedge v : \texttt{ref}(\tau, \epsilon_1) \wedge G' \mbox{ is well-typed}$.
\end{itemize}

By canonical forms, we conclude that $\tau = T_j$ and $v = g_j$ where $\epsilon_1 = j \in \Z$. Hence $i' \leq \epsilon_1 = j$, so we may apply the DEREF rule to show that $(\Gamma_v, G, i, !e_1) \Rightarrow (G_1, j+1, G_1[j])$. Then by reflexivity $j + 1 \leq \epsilon_1 + 1$, and since $G$ is well-typed $G_1$ is as well. Thus $G_1[j] : T_j = \tau$, which was all that remained to show.
\\

Case UPDATE: We know that
\begin{itemize}
	\item UPDATE: $\Gamma, i \vdash e_2 := e_1\ \colon \texttt{Unit}, \epsilon_2 + 1$
	\item INV1: $e = e_2 := e_1$
	\item INV2: $\Gamma, i \vdash e_1\ \colon \tau, \epsilon_1$
	\item INV3: $\Gamma, \epsilon_1 \vdash e_2\ \colon \texttt{ref }(\tau, \epsilon_2), \epsilon_2$
	\item IH1: $(\Gamma_v, G, i, e_1) \Rightarrow (G_1, i_1, v_1) \wedge i_1 \leq \epsilon_1 \wedge v_1 : \tau \wedge G_1 \mbox{ is well-typed}$.
	\item IH2: $(\Gamma_v, G_1, \epsilon_1, e_2) \Rightarrow (G_2, i_2, v_2) \wedge i_2 \leq \epsilon_2 \wedge v_2 : \texttt{ref}(\tau, \epsilon_2) \wedge G_2 \mbox{ is well-typed}$.
\end{itemize}
We immediately apply weakening to IH2 so that its input is $i_1$ instead of $\epsilon_1$. By canonical forms, $\tau = T_j$ and $v_2 = g_j$ where $\epsilon_2 = j \in \Z$. Thus since $i_2 \leq \epsilon_2 = j$, we may use the UPDATE rule to show that $(\Gamma_v, G, i, e_2 := e_1) \Rightarrow (G_2[j := v_1], j+1, ())$. It's immediate that $j+1 \leq \epsilon_2+1$ (by reflexivity) and that $() : \texttt{Unit}$. Last, we know that $G_2$ is well-typed and $v_1 : \tau = T_j$, so $G_2[j := v_1]$ is well-typed as well.
\\

Case ABS: We know that
\begin{itemize}
	\item INV1: $e = \texttt{fun } [\alpha]\ (x : \tau_{in}, \epsilon_{in}) \rightarrow e_{body}$
	\item ABS: $\Gamma, \epsilon \vdash e\ \colon \forall \alpha.(\tau_{in}, \epsilon_{in}) \rightarrow (\tau_{out}, \epsilon_{out})$
	\item INV2: $\Gamma[x := \tau], \epsilon_{in} \vdash e_{body}\ \colon \tau_{out}, \epsilon_{out}$
	\item IH1: $(\Gamma[x := \tau]_\tau, G, \epsilon_{in}, e_{body}) \Rightarrow (G', i', v) \wedge i' \leq \epsilon_{out} \wedge v : \tau_{out} \wedge v \mbox{ is \okvalue}$
\end{itemize}
For now, let's ignore polymorphism (So, in particular, all $\epsilon$s are integers, and substitution is a no-op).

We may immediate apply the ABS rule to show that if $v_{out} =\ <\Gamma_v, \alpha, (x : \tau_{in}, \epsilon_{in}), e_{body})>$ then
$(\Gamma_v, G, i, e) \Rightarrow (G, i, v)$. We can then use the CLOSURE rule to show that $v : \forall \alpha.(\tau_{in}, \epsilon_{in}) \rightarrow (\tau_{out}, \epsilon_{out})$, and of course $G$ is well-typed and $i \leq i$. So we need only show that $v$ is \okvalue.

First, $v$ is in fact a closure. Since substitution is a no-op, we need only show that $(\Gamma_v[x := v_x], G, \epsilon_{in}, e_{body}) \Rightarrow (G', i', v')$ plus all the other stuff, which falls out immediately from INV2 by induction. (Ok we also have to do the reasoning from the LET case involving $\Gamma_v$)
\\

Case APP: We know that
\begin{itemize}
	\item APP: $\Gamma, i \vdash e_1[\epsilon_\alpha] e_2\ \colon \tau'[\epsilon_\alpha/\alpha], \epsilon'[\epsilon_\alpha/\alpha]$
	\item INV1: $e = e_1[\epsilon_\alpha] e_2$
	\item INV2: $\Gamma, i \vdash e_1\ \colon \forall\alpha.(\tau_2, \epsilon_2) \rightarrow (\tau', \epsilon'), \epsilon_1$
	\item INV3: $\Gamma, \epsilon_1 \vdash e_2\ \colon \tau_2[\epsilon_\alpha/\alpha], \epsilon_2[\epsilon_\alpha/\alpha]$
	\item IH1: $(\Gamma_v, G, i, e_1) \Rightarrow (G_1, i_1, v_1) \wedge i_1 \leq \epsilon_1 \wedge v_1 : \forall\alpha.(\tau_2, \epsilon_2) \rightarrow (\tau', \epsilon') \wedge G_1 \mbox{ is well-typed}$.
	\item IH2: $(\Gamma_v, G_1, \epsilon_1, e_2) \Rightarrow (G_2, i_2, v_2) \wedge i_2 \leq \epsilon_2 \wedge v_2 : \tau_2[\epsilon_\alpha/\alpha], \epsilon_2[\epsilon_\alpha/\alpha] \wedge G_2 \mbox{ is well-typed}$.
	\item \okvalue: $v_1 =\ <\Gamma_v', \alpha, (x : \tau_2, \epsilon_2), e_{body}>$ and $(\Gamma_v'[x := v_2], G_2, \epsilon_{\tau_2}, e_{body}) \Rightarrow (G_3, i_3, v)$ where $G_3$ is well-typed, $i_3 \leq \epsilon'[i/\alpha]$, $v : \tau'[i/\alpha]$ and $v$ is \okvalue.
\end{itemize}
By applying weakening twice, we can use IH1 to fulfill the first premise of APP, IH2 for the second, and the evaluation part of "\okvalue" finishes it off.
\\

Case SUBTYPING:
By induction, $(\Gamma_v, G, i, e) \Rightarrow (G', i', v)$ where $G'$ is well-typed, $i' \leq \epsilon_1 \leq \epsilon_2$, $v : \tau_1$, and $v$ is \okvalue. The only thing to show is that $v : \tau_2$, which can be done with a single application of the SUBTYPING rule since we know that $\tau_1 <: \tau_2$.

\end{document}
%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:


\chapter{Background}
\label{sec:background}

\section{Termination Checking}
\label{sec:background:termination}

Proof assistants based on dependently typed languages must, if they are to be
consistent, ensure that all recursive programs terminate. The usual solution to
this problem, used by Coq, Agda, Idris, Lean and others, is to augment the type
checker with a separate termination checker based on the principle of structural
recursion. That principle says roughly: a program terminates if the input to any
recursive call is a subterm of the original input. For example, consider the
usual definition of addition for natural numbers:
\begin{code}
  plus : ℕ → ℕ → ℕ
  plus zero    m = m
  plus (suc n) m = suc (plus n m)
\end{code}

In the recursive call in the second equation, \icode{n} is a subterm of the
original input, \icode{suc~n}. Since this is the only recursive call, the
definition as a whole is structurally recursive and is accepted by Agda's
termination checker. This is justified because the natural numbers are
inductively defined, so any closed term of type \icode{ℕ} consists of finitely
many applications of the \icode{suc} constructor. If we \enquote*{peel off} one
constructor for every recursive call, we must eventually reach the \icode{zero}
case and the recursion ends.

Structural recursion as a basis for termination checking has many advantages. It
is conceptually simple, relatively easy to implement and surprisingly versatile:
many interesting definitions are naturally structurally recursive. However,
structural termination checkers are also inadequate in some important ways.

First, structural recursion is based on the syntactic notion of a subterm. This
makes termination checking non-compositional: we cannot, in general, replace a
term with another term of the same type in a recursive definition and still
expect the definition to typecheck. For a contrived example, take our previous
definition of \icode{plus} and replace the \icode{n} by \icode{id~n} in the
recursive call. The identity function \icode{id} just returns its argument, so
the two terms are obviously equivalent, but a naive termination checker would
not accept the modified definition -- after all, \icode{id~n} is not a syntactic
subterm of \icode{suc~n}. (Agda's termination checker is smart enough to
evaluate \icode{id~n} to \icode{n} and therefore accepts the modified
definition.)

Unfortunately, this non-compositionality bites not only in such contrived
situations. An often-cited example is the mapping function on rose trees (for
which we also need the standard list type and its mapping function):
\begin{code}
  data List (A : Set) : Set where
    []  : List A
    _∷_ : A → List A → List A

  mapList : ∀ {A B} → (A → B) → List A → List B
  mapList f []       = []
  mapList f (x ∷ xs) = f x ∷ mapList f xs

  data Tree (A : Set) : Set where
    leaf : A → Tree A
    node : List (Tree A) → Tree A

  mapTree : ∀ {A B} → (A → B) → Tree A → Tree B
  mapTree f (leaf x)  = leaf (f x)
  mapTree f (node xs) = node (mapList (mapTree f) xs)
\end{code}
Agda's termination checker does not accept \icode{mapTree}, and with good
reason: the recursive call in the second equation does not involve a subterm of
any input. We can argue that when we unfold \icode{mapList}, it becomes apparent
that \icode{mapTree} is always applied to an element of the list \icode{xs} and
these elements are obviously subterms of \icode{xs}. But Agda's syntactic check
is not smart enough to realise this. (Coq's termination checker employs a
heuristic that can deal with \icode{mapTree} by essentially inlining a
specialised version of \icode{mapList}.)

A second problem of termination checkers based on structural recursion is that
it is tempting to extend them with various heuristics to deal with more
complicated variants of structural recursion such as mutually recursive
definitions, lexicographic termination measures and nested inductive datatypes
like our rose trees. As a result, the termination checkers of Coq and Agda have
become quite complex, and with complexity come bugs: both Coq and Agda have
shipped versions with errors in their termination checkers that made the systems
inconsistent \cite{coqbug2013,agdabug2013}.

The final problem with structural recursion is that it does not easily
accommodate coinductive datatypes and corecursive definitions. Corecursive
functions produce values of coinductive types, which can be infinite. For
example, we can coinductively define the type of streams whose values are
infinite lists. A function which produces a stream obviously cannot terminate.
Instead, we must demand that the function is \emph{productive}, meaning that it
generates any finite prefix of its output in finite time. In other words, any
finite observation of an infinite stream must terminate. There are syntactic
methods, analogous to structural recursion, to ensure productivity, but they are
very inflexible. Sized types, on the other hand, when combined with copatterns
\cite{abel2016}, yield a practical productivity checker mostly for free.

To address these problems of structural termination checkers, we look to sized
types as a type-based termination checking mechanism.

\section{Sized Types}
\label{sec:background:sized}

Sized types are a termination checking methodology that does not rely on a
syntactic analysis. Instead, terms of inductive and coinductive types are
annotated at the type level with a size. For terms of inductive types, this size
may be thought of as an upper bound on the height of the term, viewed as a
constructor tree. For coinductive types, the size denotes the maximum depth to
which the term may be inspected. With this setup, checking a recursive
definition becomes easy: a recursive call is justified if the size of its
argument decreases. A corecursive call is justified if it increases the maximum
observation depth of its result.

Various type systems based on these principles have been proposed (see
\secref{conclusion:related} for references). This thesis investigates a calculus
whose sized types closely resemble Agda's, so the remainder of this section
gives a brief overview of Agda's system. More details and examples can be found
in Agda's manual \cite{agdamanual} and in \cite{abel2016}.

A size is a term of type \icode{Size<~n}, where \icode{n} is also a size.
Primitive sizes are variables, the successor of a size \icode{↑~n} and
\enquote*{infinity}, \icode{∞}. The size \icode{∞} plays a special role: it
designates \enquote*{fully defined} types, i.e.\ those whose values could have any
size.

For a size \icode{m} to have type \icode{Size<~n} it must be less than
\icode{n} according to an order \icode{<}. This order is mostly straightforward;
for example, we have \icode{n~<~↑~n} for all \icode{n}. However, we also have
\icode{n~<~∞} for all \icode{n} and in particular \icode{∞~<~∞}. As we will see,
this rule creates significant problems.

A sized inductive type is simply an inductive type with a parameter of type
\icode{Size} (which is the same as \icode{Size<~∞}). Continuing our rose tree
example, we define sized lists and a mapping function:
\begin{code}
  data Listₛ (A : Set) (n : Size) : Set where
    []   : Listₛ A n
    cons : (m : Size< n) → A → Listₛ A m → Listₛ A n

  mapListₛ : ∀ {A B} n → (A → B) → Listₛ A n → Listₛ B n
  mapListₛ n f []            = []
  mapListₛ n f (cons m x xs) = cons m (f x) (mapListₛ m f xs)
\end{code}
The typing of the \icode{cons} constructor reflects the intuition that the
height of \icode{cons~m~x~xs}, \icode{n}, is strictly greater than the height of
\icode{xs}, \icode{m}.\footnote{Agda also supports a different style of using
  sized types where \icode{cons} takes a \icode{Listₛ~A~n} as input and returns
  a \icode{Listₛ~A~(↑~n)} (making \icode{n} an index rather than a parameter of
  \icode{Listₛ}). This works just as well for the most part, but there are
  technical reasons to prefer our \enquote*{quantifier style}.} We can then
exploit this in \icode{mapListₛ}: in the recursive call, \icode{mapListₛ} is
applied to \icode{m}, which we know from the type of \icode{cons} is less than
\icode{n}. Thus, each recursive call decreases in size and the recursion is
justified. The definition also demonstrates that the order on sizes induces a
subtyping relation: the right-hand side of the second equation has type
\icode{Listₛ~A~m}, which is a subtype of \icode{Listₛ~A~n} for \icode{m~<~n}.

So far, we could have just as well used a structural termination checker. The
benefits of sized types become obvious when we turn to rose trees:
\begin{code}
  data Treeₛ (A : Set) (n : Size) : Set where
    leaf : A → Treeₛ A n
    node : (m : Size< n) → Listₛ (Treeₛ A m) ∞ → Treeₛ A n

  mapTreeₛ : ∀ {A B} n → (A → B) → Treeₛ A n → Treeₛ B n
  mapTreeₛ n f (leaf x)    = leaf (f x)
  mapTreeₛ n f (node m xs) = node m (mapListₛ ∞ (mapTreeₛ m f) xs)
\end{code}
The definition of \icode{Treeₛ} demonstrates the utility of \icode{∞}: we can
say that a \icode{node} should have a list of children without caring about the
size of that list. But more importantly, the termination checker now accepts
\icode{mapTreeₛ}: the recursive call is now at size \icode{m~<~n} and thus
justified. In effect, we have encoded in the type of \icode{node} our intuition
that the height of \icode{node~xs} is strictly greater than the height of any of
the elements of \icode{xs}.

Agda's sized types thus deliver on the compositionality promise: all information
the termination checker needs is encoded at the type level, so we can freely
abstract over terms. Sized types are also mostly straightforward to implement,
being just an extension of the type system.\footnote{I say \enquote{mostly}
  because Agda's current implementation includes some subtle checks to prevent
  inconsistent size assumptions, which is necessary to preserve decidability of
  type checking. Agda also has a sophisticated size inference engine (so we
  could have left all sizes in our example code implicit), but this engine is
  not part of the trusted computing base.}

Unfortunately, such convenience currently comes at the ultimate price: Agda's
implementation of sized types is, and has been for some time, inconsistent. The
culprit seems to be the highly dubious rule \icode{n~<~∞}, in particular
\icode{∞~<~∞}. This rule makes the \icode{<} relation obviously non-well-founded
(meaning there is an infinite descending chain \icode{∞~<~∞~<~\dots}) but Agda
assumes that \icode{<} is well-founded. This assumption can be exploited in
different ways \cite{agdabug2015,agdabug2016,agdabug2017,agdabug2018} to sneak
non-terminating programs past the termination checker.

It is currently unclear how to satisfactorily resolve this issue with Agda's
design. In this thesis, I adopt the obvious solution: changing the \icode{<}
relation so that \icode{∞~≮~∞}. Doing the same in Agda would, however, lead to
issues with the constructors and fields of sized data types. For example, we
would like to use the \icode{cons} constructor for sized lists at type
\begin{code}
  A → Listₛ A ∞ → Listₛ A ∞
\end{code}
but this is impossible with \icode{∞~≮~∞}. As a workaround, we can define a
consing operation for lists at \icode{∞} that is extensionally equal to
\icode{cons}, but this requires a pattern match on the input list -- an
inefficiency that should not be necessary. How best to \enquote*{rescue} Agda's
sized types thus remains an open question for now.

Another possible criticism of Agda's sized types concerns expressivity. Agda's
size arithmetic is restricted to the successor; we cannot add or multiply sizes.
This means that we cannot give a precise type to, for example, the list
appending function, whose output size should be the sum of its input sizes. This
lack of expressivity, however, is a conscious design decision. In return, we get
a size inference algorithm that can infer almost all sizes in a typical program.
This significantly lowers the cost of adopting sized types.


\subsection{Harvest and count cells}

\begin{enumerate}
\item Aspirate cells medium
\item Add 10 $mL$ of PBS
\item Aspirate the PBS
\item Add 10-12 $mL$ of trypsine 0.25 \% by flask
\item Get the cells in 50 mL falcon tube
\item Centrifuge 5 min at 800$g$
\item Aspirate the supernatant
\item Suspends cells in 1 $mL$ of medium
\item Count the cells
\item Dilute the cells to reach a final concentration of \py{rf"{to_latex(Cf)[0]}$\times 10^{to_latex(Cf)[1]}\, cellule/mL$"}
\item Add \py{Vol_depos}~$\mu L$ by well that should have cells on it
\end{enumerate}

\begin{table}[h]
\caption{Cells dilution}
\begin{pycode}

from tabulate import tabulate

infos = [
    ["Cells name", "Concentrations (cell/mL)", "Initial volume (mL)", "Final concentration (cell/mL)", "Final volume (mL)"]]

if "rows" in locals():
    for row in rows:
        infos.append(row)
else:
    infos = [
    ["Cells name", "Concentrations (cell/mL)", "Initial volume (mL)", "Final concentration (cell/mL)", "Final volume (mL)"],
    ["BMEC", rf"{to_latex(Ci)[0]}$\times 10^{to_latex(Ci)[1]}$", "$1$", rf"{to_latex(Cf)[0]}$\times 10^{to_latex(Cf)[1]}$", Vf],
]

table = r"\resizebox{\textwidth}{!}{" + tabulate(infos, headers="firstrow", tablefmt="latex_raw", stralign="center") + "}"
print(table)
\end{pycode}
\label{table-comptage-cellules}
\end{table}


% declare document class and geometry
\documentclass[12pt]{article} % use larger type; default would be 10pt
\usepackage[margin=1in]{geometry} % handle page geometry

% import packages and commands
\input{../header2.tex}

% title information
\title{Phys 221A -- Quantum Mechanics -- Lec05}
\author{UCLA, Fall 2014}
\date{\formatdate{20}{10}{2014}} % Activate to display a given date or no date (if empty),
         % otherwise the current date is printed

\begin{document}
\maketitle

[missed first hour, lots of stuff missing here]

\section{Spin precession (of a localized spin-$1/2$ particle)}

We have a 2 dimensional Hilbert space with Hamiltonian
\begin{eqn}
H = -\gamma (\hbar / 2) \v{\sigma} \cdot \v{B},
\end{eqn}
where $\gamma$ is the gyromagnetic ratio and
\begin{eqn}
\v{\sigma} = \set{ \pmat{0 & 1 \\ 1 & 0}, \pmat{0 & -i \\ i & 0}, \pmat{1 & 0 \\ 0 & -1} }
\end{eqn}
are the Pauli matrices. We can also write
\begin{eqn}
H = -\gamma \v{S} \cdot \v{B} = -\v{\mu} \cdot \v{B},
\end{eqn}
where $\v{S} = (\hbar/2) \v{\sigma}$ is the spin operator and $\v{\mu} = \gamma \v{S}$ is the magnetic moment.

For the electron, we have
\begin{eqn}
\mu = \gamma \hbar / 2 = -(g/2) \mu_B
\end{eqn}
where $g$ is the so-called ``g factor'' and $\mu_B = \abs{e} \hbar / (2mc)$ is the Bohr magneton. Classically it is expected that $g = 1$, which was actually measured (incorrectly) by Einstein et al. According to QED and the Dirac equation, $g \approx 2$ for a free electron. In fact it is very close to 2, actually around $g \approx 2.002$ which has been measured with great accuracy in agreement with theory. This measurement is one of the great triumphs of quantum field theory.

\subsubsection{Constant magnetic field}

Let $\v{B} = B \uv{z}$, then
\begin{eqn}
H = -\gamma (\hbar / 2) B \v{\sigma}_z = \pmat{-\mu B & 0 \\ 0 & +\mu B}.
\end{eqn}
So we have two energy eigenkets,
\begin{eqn}
\ket{\uparrow} = \pmat{1 \\ 0}, \qquad \ket{\downarrow} = \pmat{0 \\ 1}
\end{eqn}
with energies $E_\uparrow = -\mu B$, $E_\downarrow = \mu B$. Since $\mu < 0$, $E_\downarrow$ is the energy of the ground state. Furthermore, we have
\begin{eqn}
\Delta E = E_\uparrow - E_\downarrow = -2\mu B = \hbar \abs{\gamma} B
\end{eqn}
or $\Delta E = \hbar \omega_L$ where $\omega_L = \abs{\gamma} B$ is the Larmor frequency.

Now, for an arbitrary state $\ket{\psi}$ we can write
\begin{eqn}
\ket{\psi} = C_\uparrow \ket{\uparrow} + c_\downarrow \ket{\downarrow},
\end{eqn}
which time evolved becomes
\begin{align}
\ket{\psi(t)} &= e^{-(i / \hbar) E_\uparrow t} c_\uparrow \ket{\uparrow} + e^{-(i / hbar) E_\downarrow t} c_\downarrow \ket{\downarrow} \\
	&= e^{-(i / \hbar) E_\uparrow t} (c_\uparrow \ket{\uparrow} + e^{i\omega_L t} c_\downarrow \ket{\downarrow}) \\
	&\sim c_\uparrow \ket{\uparrow} + e^{i\omega_L t} c_\downarrow \ket{\downarrow}.
\end{align}
The expectation value of spin angular momentum is given by
\begin{eqn}
\v{s}(t) = \frac{\hbar}{2} \bra{\psi(t)} \v{\sigma} \ket{\psi(t)},
\end{eqn}
which satisfies the equation
\begin{eqn}
\od{\v{s}(t)}{t} = -\gamma \v{B} \times \v{s}(t) = \omega_L \uv{z} \times \v{s}(t).
\end{eqn}

\section{Heisenberg picture}

In the Schroedinger picture, the operators are time-independent while the state evolves in time. In the Heisenberg picture, we picture the situation the other way around, that is the states are time-independent and the operators evolve in time. In general, for an operator $O(t)$ the time average is given by
\begin{eqn}
\avg{O(t)} = \bra{\psi(t)} O(t) \ket{\psi(t)}.
\end{eqn}
In the Schroedinger picture, we solve for $\ket{\psi(t)}$. In the Heisenberg picture, we have
\begin{align}
\avg{O(t)} &= \bra{\psi} U^\dagger(t) O(t) U(t) \ket{\psi} \\
	&= \bra{\psi} O_H (t) \ket{\psi},
\end{align}
where
\begin{eqn}
O_H(t) = U^\dagger (t) O(t) U(t)
\end{eqn}
is the operator in the Heisenberg picture.

\end{document}


\newcommand{\uu}[1]{\underline{\underline{#1}}}
  \section*{List of Symbols}
  \begin{tabular}{l{0.5cm}l{7cm}}
    $\uu{\sigma}$ &Second rank stress tensor\\
    $\sigma_{eq}$ &Von Mises equivalent stress \\
    $f_{y}$       &Yield stress \\
    $\uu{s}$      &Second rank deviatoric stress tensor \\
    $J_{2}$       &Second invariant of the deviatoric stress tensor\\
    $J_{3}$       &Third invariant of the deviatoric stress tensor\\

    \multicolumn{2}{c}{} \\

    $p$                          &Equiv. creep (viscoplastic) strain \\
    $\uu{\dot \varepsilon}^{cr}$ &Second rank creep (viscoplastic) strain
    rate tensor\\

    \multicolumn{2}{c}{} \\

    $A$, $n$, $m$          &Temperature dependent isotropic creep parameters \\
    $w$                    &Weights \\

    \multicolumn{2}{c}{} \\

    $e$           &Sample's thickness \\
    $\phi_s$      &Sample's diameter \\
    $\phi_j$      &Jaws' diameter \\

    \multicolumn{2}{c}{} \\

    $\uu{I}$      &Identity matrix \\
    $Tr()$        &Trace \\
    $\left\langle \quad \right\rangle$  &Macaulay brackets \\
    $x^{\pm}$  &Positive and negative parts of variable $x$\\

    \multicolumn{2}{c}{} \\

    $F$  &Force \\
    $t$  &Time \\
  \end{tabular}


% !Mode:: "TeX:UTF-8"
% !TEX program  = xelatex
\section{Python Code}\label{A:python-1}
\Python{Sum of Progression}{code/2.py}

\begin{notepad}[Result.txt]\scriptsize
\begin{verbatim}
    //          r                        \
    ||        -----           for |r| < 1|
    ||        1 - r                      |
    ||                                   |
    ||/   r                              |
    ||| -----    for |r| < 1             |
    ||| 1 - r                            |
a_0*|<|                                  |
    |||  oo                              |
    ||< ___                    otherwise |
    ||| \  `                             |
    |||  \    n   otherwise              |
    |||  /   r                           |
    ||| /__,                             |
    \\\n = 1                             /
          / 2    \
          |n    n|
a_0*n + d*|-- + -|
          \2    2/
\end{verbatim}
\end{notepad}


\input{preamble}

\begin{document}
    \section{Subjects}
    \begin{itemize}
        \item Markov Decision Process
        \item Bellman Equations and Algorithms
    \end{itemize}

    \section{Notes}

    \subsection{Elements of reinforcement learning}

    \begin{description}
        \item[Exploration vs. exploitation] Should we explore new
        opportunities, or take advantage of the ones we have found that works?
        \item[Agent] How do we interact with the environment, who do we
        represent?
        \item[Environment] What actions can we take and what is currently
        happening?
        \item[Policy] How does the learning agent behave? A mapping from the
        perceived states of the environment to which actions to be taken when
        in those states.
        \item[Reward signal] On each time step, the environment send the agent
        a single number, a reward. The agents sole objective is to maximize the
        total reward it receives. The agent can change the outcome by the
        signal by taking actions or changing the environment, but can't change
        the function that generates the reward signal.
        \item[Value function] The value of a state is the total amount of
        reward an agent can expect to accumulate, \textit{starting from that
        state}. Rewards return the immediate desirability of a state, value
        returns the long-term desirability of a state.
        \item[Model of the environment] Something that mimics the behaviour of
        the environment, so we can infer how the environment will behave. (E.g.
        could be used to predict the next model). There are both model-based
        and model-free methods.
    \end{description}

    \subsection{Finite Markov Decision Processes}
    We can think of the interaction between an agent and the environment as the
    agent taking some action at time $t$ $A_t$, and the environment providing
    the agent with a reward $R_{t+1}$ and a new state $S_{t+1}$ which prompts a
    new action $A_{t+1}$ from the agent.

    At each time step $t$, the agent implements a mapping from states to
    probabilities of selecting each possible action $A_t \in A(S_t)$. This
    mapping is called the agent's policy and is denoted $\pi_t$ where
    $\pi_t(a|s)$ is the probability that $A_t=a$ if $S_t=s$. Reinforcement
    learning is about changing the policy as a result of experience.

    We seek to maximize the \textit{expected return} $\Ex({G_t})$, but how do
    we define $G_t$? The simplest case is simply the sum of the rewards:

    \begin{equation*}
        G_t = R_{t+1}+R_{t+2}+\cdots+R_T
    \end{equation*}
    If we are dealing with \textit{episodic tasks}, i.e. tasks that are
    independent like plays of a game or trips through a maze (we call these
    tasks ``episodes''). However, if we
    are dealing with \textit{continuing tasks}, for example a robot with a long
    life span or process-control tasks then the return, which we are trying to
    maximize is potentially infinite. Thus, we will use a definition of return
    that is slightly more complex conceptually but much simpler mathematically.
    We introduce the concept of \textit{discounting}, the agent will try to
    maximize the sum of the discounted rewards, where the rewards are
    decreasing in value so tasks that are performed immediately are worth more:
    \begin{equation*}
        G_t=R_{t+1}+\gamma R_{t+2}+ \gamma ^2 R_{t+3} + \cdots =
        \sum_{k=0}^{\infty}\gamma^k R_{t+k+1}
    \end{equation*}
    Here $0\leq \gamma \leq 1$ is called the discount rate. Then we know that
    if $\gamma < 1$ then the infinite sum has a finite value, as long as
    $R_k\neq \infty$. If $\gamma = 0$, then the agent is only concerned with
    maximizing immediate rewards. More often than not, we won't have $\infty$
    rewards, so we can simplify this to:
    \begin{equation*}
        G_t=\sum_{k=0}^{T-t-1}\gamma^k R_{t+k+1}
    \end{equation*}
    Where it is possible that $T=\infty$ or $\gamma = 1$ but both cannot be
    true (since the sum would then be $\infty$).

    \subsubsection{The Markov Property}
    We cannot expect an actor to know everything about the environment, as some
    information may be hidden. The environment will provide the actor with
    different sensations, and we might expect the actor to remember all the
    sensations it has experienced so far.

    We call a state signal which succeeds in retaining all relevant information
    to be \textit{Markov} or to have \textit{the Markov property}. For example,
    if we were playing chess, then the current configuration of all the pieces
    on the board would serve as a Markov state, because it summarizes all
    important details that led to it. Much of the information about the
    sequence is lost, but all that really matters for the future of the game is
    retained. It doesn't matter that we don't know how we ended up at that
    chess board configuration, the current configuration is all that is
    relevant.

    In order to formally define the Markov property in a mathematically simple
    way, we will assume that there are a finit number of states and reward
    values. This enables us to work in terms of sums and probabilities instead
    of integrals and probability densities, but it can easily be extended.

    Consider how an environment might respond at time $t+1$ to the action taken
    at time $t$. The most general, causal case is where the response depends on
    everything that has happened earlier:
    \begin{equation*}
        \Pr\left[S_{t+1} =s',
        R_{t+1}=r|S_0,A_0,R_1,S_1,A_1,\dots,R_{t-1},S_{t-1},A_{t-1},R_t,S_t,A_t
         \right]
    \end{equation*}
    However, if the state signal has the Markov property, then the environments
    response at $t+1$ only depends on the state and action at $t$ so we can
    define it as simply:
    \begin{equation*}
        p(s',r|s,a)=\Pr\left[S_{t+1}=s',R_{t+1}=r|S_t=s, A_t=a\right]
    \end{equation*}
    If this hold for all $s'$ and $r$ then we will be able to predict the
    result of an action having just $s$, $a$ just as good as we would be able
    to do it given the complete history of states and actions.

    It is usually useful to think of the state at each time step as an
    approximation to a Markov state even in non-Markov state signals because of
    the better performance Markov gives us in reinforcement learning, as long
    as one keeps in mind that it may not fully satisfy the Markov property.

    \subsubsection{Markov Decision Process}
    A reinforcement learning task that satisfies the Markov property is called
    a \textit{Markov decision process} or MDP. If there is a finite amount of
    states and actions, then we call it a \textit{finite MDP}. If you
    understand \textit{finite MDP} then you understand $90\%$ of modern
    reinforcement learning.

    A finite MDP, is defined by uts state and actions sets and by the one-step
    dynamics of the environment. We use the probability of going to $s'$ and
    receiving $r$ from $s$ and $a$ as previously:
    \begin{equation*}
    p(s',r|s,a)=\Pr\left[S_{t+1}=s',R_{t+1}=r|S_t=s, A_t=a\right]
    \end{equation*}
    This completely specify the dynamics of some finite MDP. We can then
    compute anything else we might want to know about the environment such as
    the expected rewards for state-action pairs:
    \begin{equation*}
        r(s,a)=\Ex\left[R_{t+1}|S_t=s, A_t=a\right]=\sum_{r\in R}r \sum_{s' \in
        S} p(s',r|s,a)
    \end{equation*}
    The state-transition probabilities:
    \begin{equation*}
        p(s'|s,a)=\Pr\left[S_{t+1}=s'|S_t=s, A_t=a\right]=\sum_{r\in
        R}p(s',r|s,a)
    \end{equation*}
    And the expected rewards for state--action--next-state:
    \begin{equation*}
        r(s,a,s')=\Ex\left[R_{t+1}|S_t=s,A_t=a,S_{t+1}=s'\right]=\frac{\sum_{r\in
         R}r\cdot p(s',r|s,a)}{p(s'|s,a)}
    \end{equation*}

    \subsubsection{Value Functions}
    We will need to estimate value functions for almost all reinforcement
    learning algorithms. The value functions are functions of states that
    estimate \textit{how good} it is to for the the agent to be in a given
    state (or rather, how good is it to perform some action in a given state).
    The notion of ``how good'' is defined in terms of what future rewards can
    we expect to receive.

    A policy $\pi$ is a mapping from each state $s \in S$ and action $a \in
    A(s)$ to the probability $\pi(a|s)$ of taking action $a$ when in state $s$.
    Informally, the following equation is the expected return when starting in
    state $s$ and following policy $\pi$ for a MDP:
    \begin{equation*}
        v_\pi(s)=\Ex_\pi\left[G_t\given
        S_t=s\right]=\Ex_\pi\left[\sum_{k=0}^{\infty}\gamma^k
        R_{t+k+1}\given S_t=s\right]
    \end{equation*}

    Similarly, we can define the value of taking action $a$ in state $s$ under
    $\pi$ as the expected return starting from $s$, taking the action $a$ and
    then following $\pi$:
    \begin{equation*}
        q_\pi(s,a)=\Ex_\pi\left[G_t\given
        S_t=s,A_t=a\right]=\Ex_\pi\left[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}\given
         S_t=s,A_t=a\right]
    \end{equation*}
    We call $q_\pi$ the action-value function for policy $\pi$. We can estimate
    $v_\pi$ and $q_\pi$ using the average of the actual returns, we will return
    to this later. For now, let's look at a fundamental property of $v_\pi$
    which is used throughout reinforcement learning. Namely, the property that
    they satisfy particular recursive relationships:

    \begin{align*}
        v_\pi(s)&= \Ex_\pi\left[G_t\given S_t=s\right]\\
            &= \Ex_\pi\left[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}\given S_t =
            s\right]\\
            &=
            \Ex_\pi\left[R_{t+1}+\gamma\sum_{k=0}^{\infty}\gamma^kR_{t+k+2}
            \given S_t=s\right]\\
            &= \sum_a \pi\left(a\given s\right) \sum_{s'} \sum_r p(s',r|s,a)
            \left[r+\gamma \Ex\left[\sum_{k=0}^{\infty}\gamma^k R_{t+k+2}
            \given S_{t+1}=s'\right]\right]\\
            &= \sum_a \pi\left(a\given s\right) \sum_{s',r} p(s',r|s,a)
            \left[r+\gamma v_\pi(s')\right]
    \end{align*}
    We can read this equation as a sum over the triples $a$, $s'$ and $r$. For
    each triple, we compute its probability $\pi\left(a\given s\right)p(s',
    r|s,a)$ weight the value in the bracket (which is the reward plus the
    expected reward from the next state $s'$), which gives us the expected
    value. This equation is called the \textit{Bellman equation for $v_\pi$}

    \subsubsection{Optimal value functions}
    In order to ``solve'' reinforcement learning, roughly means that we need to
    find a policy that achieves a lot of reward in the long run. For finite
    MDPs we can define it precisely as follows. A police $\pi$ is said to be
    better than or equals to a policy $\pi'$ if its expected return is better
    than or equal to that of $\pi'$. In other words $\pi \geq \pi' \iff
    v_\pi(s)\geq v_{\pi'}(s) \forall s\in S$. There will always be at least one
    policy which is better than or equal to all other policy which is the
    optimal policy, we denote any of the optimal policies as $\pi_*$. We can
    define their state-value function as:
    \begin{equation*}
        v_*(s)=\max_\pi v_\pi(s)
    \end{equation*}
    Which is the optimal state-value function. We can define the same for the
    optimal action-value function:
    \begin{equation*}
        q_*(s,a)=\max_\pi q_\pi(s,a)
    \end{equation*}
    For the state-action pair $(s,a)$, this function gives the expected return
    of taking action $a$ in state $s$ and then following an optimal policy. We
    can therefore write $q_*$ in terms of $v_*$ as follows:
    \begin{equation*}
        q_*(s,a)=\Ex\left[R_{t+1}+\gamma v_*(S_{t+1}) \given S_t=s, A_t=a\right]
    \end{equation*}

    \subsubsection{Dynamic programming}
    We can use dynamic programmin (DP) to compute the value functions explained
    earlier. Furthermore, we can easily obtain optimal policies once we have
    found the optimal value functions $v_*$ or $q_*$ which satisfy the Bellman
    equations:
    \begin{align*}
        v_*(s)&=\max_a \Ex\left[R_{t+1}+\gamma v_*(S_{t+1}) \given S_t=s, A_t =
        a\right]\\
            &= \max_a \sum_{s',r}p(s',r|s,a)\left[r+\gamma v_*(s')\right]
    \end{align*}
    or
    \begin{align*}
        q_*(s,a) &= \Ex\left[R_{t+1}+\gamma \max_{a'}q_*(S_{t+1},a') \given
        S_t=s, A_t=a \right]\\
            &= \sum_{s', r}p\left(s', r \given s,a\right)\left[r+\gamma
            \max_{a'}q_*(s',a')\right]
    \end{align*}
    We will see that we obtain DP algorithms by turning these Bellman equations
    into assigments, that is update rules which improve approximations of the
    desired functions.

    \subsubsection{Policy Evaluation}
    First, we will consider computing the state-value function $v_\pi$. This is
    called \textit{policy evaluation} recall that:
    \begin{align*}
    v_\pi(s) &= \Ex_\pi\left[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}\given S_t =
    s\right]\\
        &= \Ex_\pi\left[R_{t+1}+\gamma v_\pi(S_{t+1})
        \given S_t=s\right]\\
        &= \sum_a \pi\left(a\given s\right) \sum_{s',r} p(s',r|s,a)
        \left[r+\gamma v_\pi(s')\right]
    \end{align*}
    Now consider a sequence of approximate value functions $v_0,v_1,\dots$, the
    initial approximation $v_0$ is arbitrary except the terminal state has to
    be $0$. We can then obtain the approximations by using the Bellman equation
    for $v_\pi$:
    \begin{align*}
        v_{k+1}(s)&= \Ex_\pi\left[R_{t+1}+\gamma v_k(S_{t+1}) \given S_t = s
        \right]\\
            &=\sum_a \pi\left(a \given s\right)\sum_{s',r}p\left(s',r \given
            s,a\right)\left[r+ \gamma v_k(s')\right]
    \end{align*}
    This approximation does indeed converge to $v_\pi$ as $k\rightarrow \infty$

    \subsubsection{Policy Improvement}
    Now that we are able to determine $v_\pi$, we are able to evaluate our
    policies, and thus we will be able to improve upon them. For some state
    $s$, we would like to know whether or not we should change the policy such
    that we deterministically choose an action $a\neq \pi(s)$. We know the
    quality of following the current policy from $s$ ($v_\pi(s)$), so would it
    be better or worse to change to another policy? In order to answer this, we
    could select $a$ in $s$ and otherwise just follow the existing policy:
    \begin{align*}
        q_\pi(s,a)&=\Ex\left[R_{t+1}+\gamma v_\pi(S_{t+1}) \given S_t=s,
        A_t=a\right]\\
            &= \sum_{s',r}p\left(s',r \given s,a\right)\left[r+\gamma
            v_\pi(s')\right]
    \end{align*}
    If this is greater than $v_\pi(s)$ then it is better to select $a$ once in
    $s$, but then it would in fact be better to pick $s$ every time:
    \begin{equation*}
        q_\pi(s,\pi'(s)) \geq v_\pi(s) \implies v_{\pi'}(s) \geq v_\pi(s)
    \end{equation*}
    We can then use this to define a new greedy policy $\pi'$ given by:
    \begin{align*}
        \pi'(s)&=\arg\max_a q_\pi(s,a)\\
            &=\arg\max_a \Ex\left[R_{t+1} + \gamma v_\pi(S_{t+1}) \given S_t=s,
            A_t=a \right]\\
            &=\arg\max_a \sum_{s',r}p\left(s',r \given s,a\right)\left[r+\gamma
            v_\pi(s')\right]
    \end{align*}
    I.e. the greedy policy looks one step ahead and picks the action that looks
    best in the short term. Now suppose $\pi'$ is as good as, but not better
    than, the old policy $\pi$. The $v_\pi=v_{\pi'}$ and then it follows that:
    \begin{align*}
        v_{\pi'}(s)&=\max_a\Ex\left[R_{t+1}+\gamma v_{\pi'}(S_{t+1}) \given
        S_t=s, A_t=a \right]\\
            &= \max_a \sum_{s',r}p\left(s',r \given s,a \right)\left[r+\gamma
            v_{\pi'}(s')\right]
    \end{align*}
    But this was the optimal Bellman equation from earlier, therfore $v_{\pi'}$
    must be $v_*$ and thus $\pi'$ must be an optimal policy, thus this policy
    improvement algorithm must give us a strictly better policy until we hit
    the optimal one.

    So far, we have only considered deterministic policies, that is where
    $\pi(s)$ always evaluate to the same action. However, all of the ideas so
    far, easily extend to stochastic policies ($\pi(a|s)$).

    Now that we can both evaluate and improve our policies, we can just
    continue evaluating then improving and evaluating then improve and so on,
    monotonically improving untill we find a optimal solution.

    \subsubsection{Value iteration}
    This algorithm is fairly expensive, so we need to speed it up somehow. One
    way is implementation specific, where we simply make sure to pick up policy
    evaluation values from the last evaluation and not $0$.

    Value iteration provides us with a way to solve both policy evaluation and
    improvement in one step.

    \begin{align*}
    v_{k+1}(s)&=\max_a\Ex\left[R_{t+1}+\gamma v_k(S_{t+1}) \given
    S_t=s, A_t=a \right]\\
        &= \max_a \sum_{s',r}p\left(s',r \given s,a \right)\left[r+\gamma
        v_k(s')\right]
    \end{align*}
    Here, we find that $v_k$ still converges to $v_*$.

    \subsubsection{Drawback of DP}
    The drawback of DP as described here, is that it involves operations over
    the entire state set of the MDP, so if there are many states (like
    backgammon) then it is incredibly expensive.

    \subsubsection{Generalized Policy Iteration}
    In generalized policy iteration (GPI), we maintain both an approximate
    policy and an approximate value function. The value function is repeatedly
    altered to approximate the value function for the current policy, and the
    current policy is repeatedly improved with respect to the current value
    function. In a way they move against each other, as they create a moving
    target for each other, but they cause both policy and value function to
    approach optimality.

    This is a simple overview of the generalized policy iteration:
    \begin{itemize}
        \item While improving repeat
        \item Run policy evaluation for some time on some states
        \item Run policy improvement for som time on some states
    \end{itemize}
    Here we may choose not to visit all states, but in most cases we will still
    converge in polynomial time.

    \subsection{Monte Carlo algorithms}
    Up until now, we have assumed complete knowledge of the environment (finite
    states and actions). Monte Carlo algorithms, allows us to learn from just
    \textit{experience} -- we cample sequences of states, action and rewards
    without any prior knowledge of the environment's dynamics.

    In the Monte Carlo methods, we want to sovle the reinforcement learning
    problem based on averaging sample returns. Furthermore instead of computing
    the value functions (as we did in DP), we here want to learn the value
    functions instead.

    What if we looked at just the state-value function, how would we try to
    estimate that? The obvious solution here, is to average all the returns
    observed after visits to that state. As we observe more and more returns,
    the average should converge to the expected value. This is the underlying
    idea of all Monte Carlo algorithms.

    In order to ensure well-defined returns are available, we will only
    consider Monte Carlo for episodic tasks.

    \subsubsection{Estimating $v_\pi(s)$}
    $v_\pi(s)$ is the value of a state $s$ under policy $\pi$, given a set of
    episodes obtained by following $\pi$ and passing through $s$. Each
    occurrence of state $s$ in an episode is called a \textit{visit} to $s$. It
    may happen that we visit $s$ multiple times in an episode, so we will look
    at the \textit{first visit} to $s$. Let's first look at a theoretical
    algorithm which computes the value of $s$ in the first visit to $s$:
    \begin{algorithm}
        \caption{First-visit MC policy evaluation}
        \begin{algorithmic}
            \State \textit{// Initialize:}
            \State $\pi \gets$ policy to be evaluated
            \State $V \gets $ an arbitrary state-value function
            \State $Returns(s) \gets $ an empty list, for all $s \in S$
            \State
            \Loop \, forever
                \State Generate an episode using $\pi$
                \ForAll{state $s$ in the episode}
                    \State $G \gets $ return following the first occurence of
                    $s$
                    \State Append $G$ to $Returns(s)$
                    \State $V(s) \gets \average(Returns(s))$
                \EndFor
            \EndLoop
        \end{algorithmic}
    \end{algorithm}

    We could also have explored the \textit{every-visit MC method}, which has
    some different properties and extends more naturally to function
    approximation, but First-visit MC has been more widely studied.

    First-visit MC converges quadratically to $v_\pi(s)$ as the number of
    first-visits to $s$
    goes to infinity. An alternative, finite option, is to loop for some time
    or some number of episodes.

    \subsubsection{Estimation of action values}
    If a model is available, we can simply look ahead in the next possible
    states and pick the action that leads to the best state. If no model is
    available, then we have to estimate the values of the action (e.g. speed of
    vehicle) we should pick in order to get to the best state and maximize
    reward.

    In other words, one of the primary goals for Monte Carlo methods is to
    estimate $q_*$, which is the optimal value we get when we take action $a$
    from state $s$. In order to estimate this, we may run a First-visit MC
    method as before, just visiting state-action pairs instead of just states.
    Again, this will converge quadratically as before.

    The complication is that many state-action pairs may never be visited. If
    $\pi$ is a deterministic policy, then following $\pi$ one will only observe
    returns from one action from each state (as it is deterministic). Therefore
    the Monte Carlo estimates of the other actions from that state will not
    improve with experience. This is the general problem of \textit{maintaining
    exploration} i.e. we must assure that we are continually exploring other
    options in order to attempt to find a better one. One way to solve this, is
    to say that we start in a state-action pair, and every state-action pair
    has a nonzero probability of being selected as the start. We call this the
    assumption of \textit{exploring starts}.

    \textit{Exploring starts}, is not really useful when we learn by actual
    interacting with an environment as we cannot simply start from any state we
    want. In this case, the most common alternative approach is to only
    considering stochastic policies with only nonzero probabilities.

    \subsubsection{Approximating optimal policies}
    Let us consider a Monte Carlo version of classical policy iteration. We can
    perform policy evaluation as described in the previous section (estimation
    of action values). If we assume that:
    \begin{itemize}
        \item We observe an infinite number of episodes.
        \item The episodes are generated with exploring starts.
    \end{itemize}
    Then the Monte Carlo methods will compute each $q_{\pi_k}$ exactly for
    arbitrary $\pi_k$.

    Now we just need to figure out how to perform policy-improvements. We can
    improve on our policy with a simple greed policy with respect to the
    current value function. We can then compute for each $s \in S$ the policy
    $\pi$:
    \begin{equation*}
        \pi(s) = \arg\max_a q(s,a)
    \end{equation*}
    We can then construct each $\pi_{k+1}$ as the greedy policy with respect to
    $q_{\pi_k}$, and we get that the policy improvement theorem applies since:
    \begin{align*}
        q_{\pi_k}(s,\pi_{k+1}(s)) &= q_{\pi_k}(s, \arg\max_a q_{\pi_k}(s,a))
    \end{align*}
    This assures us that each $\pi_{k+1}$ is uniformly better than or just as
    good as $\pi_k$. If it is just as good as $\pi_k$ then they are both
    optimal policies.

    \subsubsection{Monte Carlo without infinite episodes}
    We made two unlikely assumptions in order to easily obtain this guarantee
    of convergence. One was that the episodes has exploring starts, the other
    was that we could run for infinitely many episodes. We will try to remove
    the latter assumption here and then return to exploring starts afterwards.

    There are two primary ways to combat the infinite episodes assumption, one
    approach is to estimate some bounds on magnitude and probability of error
    and run until these bounds are sufficiently small. This will guarantee
    correct convergence up to some level of approximation, however it is likely
    to require far too many episodes to be useful in practice.

    The second approach is to avoid trying to complete policy evaluation before
    returning for improvement and simply move the value function \textit{toward}
    $q_{\pi_k}$. One extreme example of this was the value iteration, in which
    only one iteration of iterative policy evaluation is performed between
    improvements. Or the in-place version where we alternate between
    improvement and evaluation steps for single states.

    For Monte Carlo policy evaluation it is natural to alternate between
    evaluation and improvement on an episode-by-episode basis. After each
    episode we evaluate the policy and then improve it at all the visited
    states. We can describe this algorithm as follows:

    \begin{algorithm}
        \caption{Monte Carlo ES (Exploring Starts)}
        \begin{algorithmic}
            \State \textit{// Initialize, for all } $s\in S, a\in A(s)$:
            \State $Q(s,a) \gets$ arbitrary
            \State $\pi(s) \gets$ arbitrary
            \State $Returns(s,a) \gets$ empty list
            \State
            \Loop\, forever
                \State Choose $S_0 \in S$ and $A_0 \in A(S_0)$ s.t. all pairs
                have probability $> 0$
                \State Generate an episode starting from $S_0, A_0$ following
                $\pi$
                \ForAll{pairs $s,a$ appearing in the episode}
                    \State $G \gets$ return following the first occurence of
                    $s,a$
                    \State Append $G$ to $Returns(s,a)$
                    \State $Q(s,a) \gets \average(Returns(s,a))$
                \EndFor
                \ForAll{states $s$ in the episode}
                    \State $\pi(s) \gets \arg \max_a Q(s,a)$
                \EndFor
            \EndLoop
        \end{algorithmic}
    \end{algorithm}

    \subsubsection{Monte Carlo without Exploring Starts}
    Now, how can we avoid the unlikely assumption of exploring starts? The only
    general way to ensure that all actions are selected infinitely often is for
    the agent to continue to select them. We have two approaches to ensuring
    that this happens, we call it \textit{on-policy} methods  and
    \textit{off-policy} methods. On-policy attempts to evaluate or improve the
    policy used to make decision whereas off-policy methods evaluate or improve
    a different policy. The ES method above is an example of an on-policy
    method. We will explore on-policy methods more here, and not go into
    details with off-policy.

    Here we will look at $\epsilon$-greedy policies, which with probability
    $1-\epsilon$ will simply follow the policy $\pi$, but with probability
    $\epsilon$ it will select a uniformly random action. So each action has
    probability $\frac{\epsilon}{|A(s)|}$ of being picked, except the estimated
    optimal action which also has probability $1-\epsilon$ of being picked,
    yielding the algorithm:

    \begin{algorithm}
        \caption{On-policy first-visit MC control (for $\epsilon$-soft
        policies)}
        \begin{algorithmic}
            \State \textit{// Initialize}
            \ForAll{$s \in S$ and $a \in A(s)$}
                \State $Q(s,a) \gets$ arbitrary
                \State $Returns(s,a) \gets$ empty list
                \State $\pi(a|s) \gets$ an arbitrary $\epsilon$-soft policy
            \EndFor

            \Loop\, forever
                \State Generate an episode using $\pi$
                \ForAll{pairs $s,a$ appearing in the episode}
                    \State $G \gets$ return following the first occurence of
                    $s,a$
                    \State Append $G$ to $Returns(s,a)$
                    \State $Q(s,a) \gets \average(Returns(s,a))$
                \EndFor
                \ForAll{visited states $s$ in the episode}
                    \State $A^* \gets \arg\max_a Q(s,a)$
                    \ForAll{$a \in A(s)$}
                        \State $\pi(a|s) \gets \begin{cases}
                        1-\epsilon + \frac{\epsilon}{|A(s)|},&\text{ if
                        }a=A^*\\
                        \frac{\epsilon}{|A(s)|},&\text{ if } a \neq A^*
                        \end{cases}$
                    \EndFor
                \EndFor
            \EndLoop
        \end{algorithmic}
    \end{algorithm}

    This holds under the policy improvement theorem, although I will not prove
    this here.
\end{document}