\section{Reinforcement Learning}
\label{sec:rl_theory}

\subsection{The Environment/Agent Interface}

In the real world learning happens by trial and error. Reinforcement learning is an attempt to formalize this study of ``learning by interaction''. The problem of learning by trial and error has a natural formulation by the environment/agent interface, which is graphically represented in figure \ref{fig:agent_enviroment_interface}. Note, this entire section draws heavily from the book Reinforcement Learning by \textcite{sutton_reinforcement_2018}, regarding naming conventions, algorithms and notation.

\begin{figure}
    \centering
    \includegraphics[scale=0.35]{figures/agent_environment_interface.png}
    \caption{Agent/Environment Interface}
    \label{fig:agent_enviroment_interface}
\end{figure}

The concept can be phrased the following way: An agent will over a series of discreet time steps $(t=1, 2, \cdots, T)$ take an action which will lead to some sort of reward (or in economic terms utility). For each time step the agent receives information about the state $S_t$ of the environment $\mathcal{E}$. The agent then takes an action $A_t$, which prompts the environment $\mathcal{E}$  to return a reward $R_{t+1}$, and a new state $S_{t+1}$. This process continues until the game terminates. The agent's sole purpose is to maximize the cumulative rewards throughout the game. It should be noted here that $R_t \in \R$ such that the agent is optimizing over a sum of scalars. The game will lead to a trajectory of states, actions and rewards that look like:

\begin{equation}
    S_0, A_0, R_1, S_1, A_1, R_2, S_2, A_2, \cdots ,R_{T-1}, S_{T-1}, A_{T-1}, R_{T}
\end{equation}

Certain assumptions is necessary to perform any sort of modelling. The most fundamental assumption reinforcement learning relies on, is that of the Markov decision process: A Markov decision process (MDP) abides to:

\begin{equation}\label{eq:mdp1}
    p(s_t, r_t \mid s_{t-1},  a_{t-1}) = p(s_t, r_t \mid s_{t-1},\cdots, s_{0}, a_{t-1}, \cdots, a_{0}) = P(S_t = s_t, R_t = r_t \mid S_{t-1} = s_t, A_{t-1} = a_t)
\end{equation}

Breaking equation \eqref{eq:mdp1} down reveals that the MDP follows a true probability distribution. That is, a joint probability distribution describes $R_t$ and $S_t$. Another important feature is that the probability distribution of $S_t$ and $R_t$ only depends on the last state and action. This turns out to be an instrumental assumption for doing any sort modelling, the implication being that the state $s_t$ contains all relevant information about the past, hence the name, \textit{Markov} Decision Process. First, the assumption implies, that the size of the probability distribution would not grow linearly as more and more states and actions was represented for the agent, yielding the computations more and more expensive. Second, it allows for backward induction and dynamic programming (a topic which will be discussed later).  Lastly one must ensure that when a system is modelled, the state represents all relevant information about the past. Multiple statements can be derived from \eqref{eq:mdp1}, but most importantly one can derive the expected reward:

\begin{equation}
    \E[R_t \mid A_{t-1} = a_{t-1}, S_{t-1} = {s_{t-1}}] = \int_{r_t} r_t \int_{s_t} p(r_t, s_t \mid a_{t-1} s_{t-1}) d s_t d r_t
\end{equation}

The agent's goal is to maximize the cumulative rewards, which can be formulated as:

\begin{equation}\label{eq:cum_rewards}
   G_t = R_{t+1}, R_{t+2}, R_{t+3}, \cdots R_{T}
\end{equation}

The formulation in \eqref{eq:cum_rewards} can be problematic with continuing tasks. That is if $T \rightarrow \infty$. Therefore discounting of rewards is usually implemented, which have the nice economic implications, that agents in the real world tend to be impatient, and therefore more realistically model real human agents:

\begin{equation}
    G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots = \sum_{k=0}^{T - t} \gamma^k R_{t+k+1}
\end{equation}

With $\gamma$ being the discount rate, yielding a geometric series, that is known to converge, if $R_k$ is bounded.

\subsection{Value Function, Q-Function, Policy Function and the Bellman Equation}

The value function represents the expected discounted cumulative returns from following a policy, $\pi$. The value function can be said to approximate the value of a strategy:

\begin{equation}\label{eq:value_function1}
    v_{t}^{\pi}(s_t) = \E_t [G_t \mid S_t = s_t] = \E_t \lsp \sum_{k=0}^{T - t} \gamma^k R_{t+k+1} \bigg\vert S_t = s_t \rsp
\end{equation}

A couple of things to note about \eqref{eq:value_function1}: Since the expectation is taken over a sum, one can instead take the sum over the expectations, yielding it possible to calculate the individual expected returns from following a policy, and using those to calculate the value function. Another concept that closely resembles the value-function is the Q-function:

\begin{equation}
    q_t^{\pi} (s_t, a_t) = \E_t [G_t \mid S_t = s_t, A_t = a_t ] = \E_t \lsp \sum_{k=0}^{T - t} \gamma^k R_{t+k+1} \bigg\vert S_t = s_t, A_t = a_t\rsp
\end{equation}

The Q-function only differs from the value function by also conditioning on the action, and not only the state. The value function and the Q-function shares the property, that it maps the expected value of a state (or state action pair) to a scalar value, where this value represents the cumulative, discounted rewards of following a certain policy. This has the nice property of allowing the agent to choose an action which maps to the highest expected value, $G_t$.

The Bellman equation can be expressed, using the expression for the value function:

\begin{equation}
    v^{\pi}_{t} (s_t) = \E_t [G_t \mid S_t] = \E_t  \lsp R_{t+1} + \gamma G_{t+1} \mid S_t \rsp = \E_t \lsp R_{t+1} + \gamma v_{t+1}^{\pi}(S_{t+1}) \rsp
\end{equation}

One should consider that $\E [v_{t+1}^\pi(s_{t+1})]$ does not imply $v_{t+1}^{\pi}(\E[s_{t+1}])$, which has the consequence of considerable computational markup when solving a model using the Bellman Equation as an update rule.

Lastly some information about the policy function.  A policy function defines how the agent chooses an action:

\begin{equation}
    \pi_t : \statespace \mapsto \actionspace
\end{equation}

In general reinforcement learning algorithms falls in two distinct categories: Algorithms that directly estimate the policy function or algorithms that work by estimating the value function (or Q-function), and use this to find the optimal policy. In this paper the primary focus will be on the latter category of algorithms.

\subsection{Relationship to Dynamic Programming}\label{sec:dynamic_programming}

Dynamic programming, invented by Richard Bellman, allows for a way to find the optimal policy, $\pi^{*}$, and the associated value function, $v^{\pi^{*}}$. Dynamic programming has a set of practical and formal requirements. First, the size of the state space should be limited. This is due to the fact, that number of computations will increase exponentially with the number of states, what Richard Bellman described as ``The Curse of Dimensionality''. Secondly it requires that the entire MDP is known. Here one should distinguish between an environment and a model of an environment. In this paper the model and the environment coincide, but this is not always the case. An example could be a self driving car. Because the self driving car does not have a perfect model of the environment, dynamic programming cannot be used as an algorithm for navigating the environment. In other words, unless the joint probability distribution of states and actions can be formulated explicitly, dynamic programming is not feasible. In general, it can be said that dynamic programming (and also reinforcement learning) is to use the value function to structure the search for good policies  \parencite{sutton_reinforcement_2018}.  The optimal value function  implies that one knows the optimal policy:

\begin{equation}
    v_t^{*}(s_t) = \underset{a}{\max}  \E \lsp R_{t+1} + \gamma v^{*}_{t+1}(S_{t+1}) \mid S_t = s_t, A_t = a\rsp
\end{equation}

Dynamic programming problems can be solved by two different approaches: \textit{value function iteration} and \textit{policy function iteration}.

\textit{Policy function iteration} consists of two steps: 1) an evaluation step. Which calculates the value of a policy, and 2) a policy improvement step.
The first step can be considered a prediction step. By following a given policy the associated value function can be estimated. This is done by sweeping through the state space calculating the expected value of the state following the policy. This process continues until the algorithm has converged. In this context convergence implies the difference between the previous estimation of the value function and the current estimation of the value function, only differs below some threshold. The second step, policy improvement, works by searching through the state space, seeing if diverging from the current policy yields higher expected cumulative returns. An alternative phrasing is: Assume that the policy used for the evaluation step is optimal. Then there should be no other strategy that would yield a higher value-function for all possible states. This can be expressed more formally as:

\begin{equation}
     \E \lsp R_{t+1} + \gamma v^{*}_{t+1}(S_{t+1}) \mid S_t = s_t, A_t = \pi_t^*(s_t) \rsp \geq \E \lsp R_{t+1} + \gamma \tilde{v}_{t+1}(S_{t+1}) \mid S_t = s_t, A_t =\tilde{\pi}_t(s_t) \rsp \qquad \forall s_t \in \statespace
\end{equation}

Where $\tilde{\pi}$ is any arbitrary policy. If at any point in the state space one can find a policy that yields a higher value function than the current, then one should switch policy, which yields an updated, superior policy! The process alternates between policy evaluation and policy improvement, until no better policy can be found. A graphical representation of the process can be considered as shown in equation \eqref{eq:policyevaluation} where $\overset{\textbf{E}}{\longrightarrow}$ denotes a policy evaluation and $\overset{\textbf{I}}{\longrightarrow}$ denotes policy improvement:

\begin{equation}
    \label{eq:policyevaluation}
    \pi^0 \overset{\textbf{E}}{\longrightarrow}
    v^{\pi^0} \overset{\textbf{I}}{\longrightarrow} \pi^1 \overset{\textbf{E}}{\longrightarrow} v^{\pi^1} \overset{\textbf{I}}{\longrightarrow} \cdots \overset{\textbf{I}}{\longrightarrow} \pi^* \overset{\textbf{E}}{\longrightarrow} v^{\pi^*}
\end{equation}

The second approach \textit{value function iteration} computes a max over the value function implying only a single sweep through the state space in each iteration of the loop. The max operation yields value function iteration a considerably faster solution method. Value function iteration can be considered using the Bellman equation as an update rule \parencite{sutton_reinforcement_2018}. The algorithm for value function iteration is described below:

\begin{algorithm}[H]
\SetAlgoLined
\KwResult{Yielding $\pi^*, v^*$}
 Algorithm parameter $\theta > 0$ determining accuracy of estimation\;
 Initialize $V(s)\quad \forall s \in \statespace$ except $V(terminal) = 0$\;
 \While{$\Delta > \theta$}{
    $\Delta \la 0$ \;
    \ForEach{$s \in \statespace$}{
        $v \la V(s)$ \;
        $V(s) \la \underset{a}{\max} \E [R_{t+1}  + \gamma V(S_{t+1}) \mid A_t = a, S_t = s] $ \;
        $\Delta \la \max (\Delta, \mid  v - V(s) \mid )$
    }
 }
 \caption{Value Function Iteration}
\end{algorithm}

 So for each sweep through the state space a single sweep of policy evaluation and a single sweep of policy improvement is performed. Again this algorithm terminates when the difference between the value function of the last sweep and the current value function is below some threshold.

 In economics a dynamic programming solution will usually have the addition of using backwards induction. This is due to the fact the age evolves deterministically. The model assumes an agent acting over $T$ time steps, terminating when the agent reaches a certain age. So in a sense, the agent moves in a deterministic fashion towards the termination of the environment. This implies that one can solve such a model by only doing a single sweep through the state space! The agent will maximize its value function in the terminating period. Now using this value associated with the terminating period, the agent can consider his actions in $T-1$, remembering the Bellman equation can be used as an update rule:

 \begin{equation}
     V_{T-1}(s_{t-1}) = \underset{a}{\max}\E [R_T + \gamma V_T (S_T) \mid S_{T-1} = s_{T-1}, A_{T-1} = a]
 \end{equation}

 In other words, one can model all possible states that the agent can encounter in each time step of the model, making it possible to find the optimal value function $v^{\pi^*}$ and policy function $\pi^{*}$ by a single sweep through the state space using a combination of dynamic programming and backwards induction. The implementation of value function iteration in this paper will be explored in section \ref{sec:solution_methods}. In the reinforcement learning literature, this concept of updating the estimate of the value of a state based on value estimates of other states, is called bootstrapping\footnote{In economics bootstrapping will usually refer to a non-parametric, sample based approach for doing inference of a parameter estimate. These things are unrelated.}.

 Before moving on to other reinforcement learning techniques, the concept of \textit{Generalized Policy Iteration} (GPI) is introduced. GPI is the concept of letting policy iteration and policy improvement interact with the intention of having the policy converge to the optimal policy. In practice the policy evaluation will be done with respect to the current policy. Policy improvement will be greedy with respect to the current value function. As described by \textcite{sutton_reinforcement_2018}: The value function stabilizes only when it is consistent with the current policy, and the policy stabilizes
only when it is greedy with respect to the current value function.
Thus, both processes stabilize only when a policy has been found that is greedy with respect to its own evaluation function. This implies that the Bellman optimality equation holds, and thus that the policy and the value function are optimal.

 \subsection{Overview of Reinforcement Learning Techniques}

Two different reinforcement learning methods will be presented in section \ref{sec:solution_methods}. Here I present some basic information that makes the reader able to digest the material presented. First it is important to address why not only use dynamic programming. Dynamic programming requires that a perfect model of the environment is accessible. This is due to the fact, that when calculating the expected value function for each action, the probability distribution of the reward and the next state, needs to formulated in an explicit form. The other techniques presented here does not have the same requirement. Secondly DP methods require that the size of the state space must be limited. The implementations presented later does not have the same requirements, allowing for approximating the value function and/or the policy function. Below I explain two different methods of learning \textit{Monte Carlo Methods} and \textit{Temporal Difference Learning} these being the two foundations of the reinforcement learning methods presented later. It is also useful to distinguish between \textit{control} and \textit{prediction}. Prediction can be considered the step of estimating the value function, whereas control relates to how to approximate optimal policies. Finally, one should make a distinction between \textit{off-policy} and \textit{on-policy} methods. \textcite{sutton_reinforcement_2018} describes the difference as: On-policy methods attempt to evaluate or improve the policy that is used to make decisions, whereas off-policy methods evaluate or improve a policy different from that used to generate the data.

\subsubsection{Monte Carlo Methods}

Before delving into MC-methods it is appropriate to introduce some more terminology: When talking about an \textit{episode} it should be understood as an agent moving through the environment from start to termination. When talking about a \textit{step} it should be understood as going from one state to the next in the environment.

First consider the problem of Monte Carlo prediction. The best way to get a sense of prediction with Monte Carlo methods is to present an algorithm:

\begin{algorithm}[H]
\SetAlgoLined
\KwResult{Yielding $v^{\pi}$}
 Input: policy $\pi$ to be evaluated\;
 $Returns(s) \la$ an empty list $s \in \statespace$\;
 \While{Forever}{
    Generate an episode: $S_0, A_0, R_1, S_1, A_1, \cdots S_{T-1}, A_{t-1}, R_T$\;
    $G \la 0$ \;
    \ForEach{step in episode, $t = \{T -1 , T-2 , \cdots, 0\}$}{
        $G \la \gamma G + R_{t+1}$\;
        \If{$S_t \notin \{S_{t-1}, S_{t-2}, \cdots S_0 \}$}{
            Append $G$ to $Returns(S_t)$ \;
            $V(S_t) \la average(Returns(S_t))$ \;
        }
    }
 }
 \caption{First-Visit MC prediction, for estimating $v^{\pi}$}
 \label{alg:mcfirstvisit}
\end{algorithm}

Algorithm \ref{alg:mcfirstvisit} shows how the general concept of estimating the value function for given policy. The agent follows the policy until the game terminates. Starting from the terminating state, a reversed experience replay is performed using the discounted rewards to approximate the value function for each state the agent visited. However, the algorithm above relies on a model of the environment. If not such a model is present, one will need to estimate the value of each action. Instead of considering the value function, the Q-function (state-action pair) is used for Monte Carlo estimation. It is assumed that the policy stays constant. That is, the policy does not update as more and more episodes are experienced, which defeats the purpose of learning how to interact with the environment. This can be addressed by updating the policy, and then discard old experience. Another possibility is to accept the non-stationarity of the data collected, and update the policy using the data from a previous policy, hoping that with time the algorithm will converge. This is in fact generalized policy iteration.

Consider now how to do Monte Carlo Control, i.e., approximating the optimal policy. Just as with policy iteration the pattern followed is:

\begin{equation}
    \label{eq:montecarlocontrol}
    \pi^0 \overset{\textbf{E}}{\longrightarrow}
    q^{\pi^0} \overset{\textbf{I}}{\longrightarrow} \pi^1 \overset{\textbf{E}}{\longrightarrow} q^{\pi^1} \overset{\textbf{I}}{\longrightarrow} \cdots \overset{\textbf{I}}{\longrightarrow} \pi^* \overset{\textbf{E}}{\longrightarrow} q^{\pi^*}
\end{equation}
Policy evaluation is done as described above. Policy improvement is done by making the policy greedy with respect to the current estimated Q-function \parencite{sutton_reinforcement_2018}. Since the Q-function instead of the value function is used, no model is needed to construct a greedy policy \parencite{sutton_reinforcement_2018}. The greedy policy is the one that for each $s \in \statespace$ deterministically chooses an action with maximal action-value:

\begin{equation}
    \pi(S_t) = \underset{a}{\argmax}  Q( S_t, a )
\end{equation}

Now this algorithm rests on the assumption of \textit{exploring starts} and on the assumption of \textit{infinite episodes}. The assumption of infinite episodes is to ensure convergence, and in practice the algorithm is usually run until the algorithm has converged by some high number of episodes. The more problematic assumption is that of exploring starts. This is due to the fact, that in reality one cannot assume that there is a non-zero probability of starting in all states $s \in \statespace$, and proceed the episode from that starting point. This is an essential assumption, because otherwise, one could not know the value function of unexplored states without visiting them.

The problem of not visiting every state is closely related to the exploration vs. exploitation trade-off. If an algorithm acts greedy (only exploits) new and better policies will never be discovered. The exploration of the state space can be done by either on-policy control or off-policy control. The exploration will in this paper be done by $\epsilon$-greedy algorithms. $\epsilon$-greedy algorithms chooses with probability $1 - \epsilon$ a (uniformly) random action each step, else it greedily chooses the action that corresponds to the highest Q-value. The next section will give an example of both on-policy and off-policy methods.

\subsubsection{Temporal Difference Learning}

Temporal Difference (TD) learning combines ideas taken from Dynamic Programming and Monte Carlo Methods. It takes from Monte Carlo methods, that you do not need a perfect model of the environment. It takes from Dynamic Programming the bootstrapping, i.e., it uses learned estimates to update, without needing the final outcome. Just as Monte Carlo methods, Temporal difference methods has a prediction and control element.

TD prediction can be summarized in the equation:

\begin{equation}
    V(S_t) \leftarrow V(S_t) + \alpha \lsp R_{t+1} + \gamma V(S_{t+1}) - V(S_t) \rsp
\end{equation}

The equation states that $V(S_t)$ should be updated according to the return in a given period + the value function of the next period $V(S_{t+1})$. This method is called TD(0), since it only uses 1 step to update the value function. In essence TD methods learn a guess from a guess. Compared to MC methods the TD methods can learn at each step; update its estimate at each point in time. It is also proven, that for any policy $\pi$ kept fixed, then a TD(0) algorithm will converge to $v^{\pi}$ \parencite{sutton_reinforcement_2018}. Even though, an open mathematical question, in practice it is found that TD methods converge faster than MC methods \parencite{sutton_reinforcement_2018}.

Control with temporal difference learning, can also be separated into on-policy control and off-policy control. The most used on-policy control for TD methods is called SARSA (state, action, reward, state, action). Just as with MC methods a need to trade off exploitation and exploration is present. The agent want to see new parts of the state space (exploration), but should also use (exploit) what is assumed to be the optimal choice given the value function associated with the current value function. Consider the policy generating the data being an $\epsilon$-greedy algorithm. In this case, instead of using the value function, consider the case of using the state-action pair function $Q(S_t, A_t)$, yielding the update rule:

\begin{equation}
    Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \lsp R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t) \rsp
\end{equation}

This update is made after each step visiting a non-terminal state of the environment.
Now notice here that the update rule uses the realized state $S_{t+1}$ and $A_{t+1}$. In other words the data that is used to update the estimate is the data generated from following the epsilon greedy policy!

Off-policy control can be done by using Q-learning. Q-learning has the slight twist on the update rule compared to SARSA that:

\begin{equation}
    Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \lsp R_{t+1} + \gamma \underset{a}{\max} Q(S_{t + 1}, a) - Q(S_t, A_t) \rsp
\end{equation}
Actions is still chosen by an $\epsilon$-greedy policy, the difference here being the update rule uses a different policy than the actual policy followed. This is due to the fact that the Q-function is updated under the assumption of greedily choosing an action in time $t+1$ conditional on the state. Q-learning and double Q-learning will be explored in depth later when describing the algorithms used in this paper.

\subsection{Landmarks}

Finally, a brief overview of environments/games where reinforcement learning have been instrumental to the current hype of reinforcement learning. The first big success of reinforcement learning was made by Tesauro in 1992 creating an agent capable of learning to play backgammon trough self play \parencite{sutton_reinforcement_2018}. Using an artificial neural network to approximate the value function, and using a temporal difference algorithm, the algorithm was capable of playing expert level backgammon. In 2011 the IBM Watson algorithm won in jeopardy using the same methods as Tesauro did for his backgammon agent \parencite{sutton_reinforcement_2018}. In 2013 the company DeepMind (now acquired by google), showed that it was possible for a reinforcement learning algorithm to learn to play video games. Here an important feat was, that it was fed the raw image input and used an artificial neural network to transform this image into a representation of the state space allowing for navigating in the environment \parencite{mnih_playing_2013}. In 2016 DeepMind created AlphaGo and a year later AlphaGo Zero, which learned to master the game of Go. This was assumed in a long time to be a hard problem for learning algorithms due to its very large state and action space \parencite{silver_general_2018}. The first iteration used expert players to learn the game, while the AlphaGo zero used only self play. These examples show, that the capabilities of these algorithms to learn to navigate complicated environments, might leave a way for high dimensional dynamic economic models to be solved using reinforcement learning methods.


%%
%% Author: novitoll
%% 2/17/18
%%

% Preamble
\documentclass[11pt]{article}
\usepackage{amsmath}
\usepackage{graphicx}
\graphicspath{ {code/} }

\title{CVT: Lecture 3}
\date{2018-02-16}
\author{Novitoll}

% Document
\begin{document}
    \maketitle
    \pagenumbering{arabic}

    \section{Task}
    \begin{enumerate}
        \item Get the text image
        \item Gray-scale it, White -> black, black <- white --> gray
        \item Blur, Make it common
        \item Thresholding binarize
        \item Morphology closing, reduce noize
        \item Find contours, bounding box
        \item According to the coordinates of contours, classify text from image
    \end{enumerate}

    Note: Cropping text lines from text image by boundaries is not effective as it's not stable to the noises
    That's why gradients (transition of W to B, B to W) are more useful in real world data

    \section{Gradients}

    Apply kernel [-1, 0, 1]  as the filter to find gradients:

    \includegraphics[scale=0.7]{gradients.png}

    Popular kernels:

    Sobel - sum gathered in the center of matrix, can be transposed to calculate grads vertically
    Scharr - more restricted on sides, can be applied vertically
    Laplacian - gradients, calculates vertically and horizontally at once

    \includegraphics[scale=0.7]{filters.png}

    Example of the input with the noise and how gradients of vertical projection + Sobel smoothing works

    \includegraphics[scale=0.7]{output-gradient.png}

\end{document}


\documentclass[a4paper]{llncs}

\usepackage{etex}
\usepackage{nag}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

\usepackage{amsmath,amssymb,stmaryrd}
\usepackage{cite}
\usepackage{tikz}
\usetikzlibrary{matrix,positioning,backgrounds,shapes,automata,fit}
\usepackage{mathpartir}

\usepackage[
  pdftex,
  pdfusetitle,
  pdfborder={0 0 0}]{hyperref}

\usepackage{fixme}
\fxsetup{
 status=draft
}

\renewcommand*\sectionautorefname{Section}
\renewcommand*\subsectionautorefname{Section}
\renewcommand*\subsubsectionautorefname{Section}
\newcommand{\etal}{\emph{et~al.}}

\include{isamarkup}
\include{isadecl}

\DeclareIsaConst{ert}{\isaconst{ert}}
\DeclareIsaConst{cost}{\isaconst{cost}}
\DeclareIsaConst{K}{\isaconst{K}}

\hyphenation{Isa-belle hy-poth-esis co-data-type co-data-types
  co-al-ge-bra co-al-ge-bras co-al-ge-bra-ic co-in-duct co-in-duct-ive
  co-in-duct-ion co-re-cur-sor co-re-cur-sors co-re-cur-sion co-re-cur-sive
  meas-sur-able}

\pagestyle{plain}

\title{Formalising Semantics for Expected Running Time of Probabilistic Programs \\ (Rough Diamond)}
\author{Johannes Hölzl}
\authorrunning{J. Hölzl}
\titlerunning{Expected Running Time of Probabilistic Programs}
\institute{Fakultät für Informatik, TU München, \email{hoelzl@in.tum.de}}

\begin{document}

\maketitle

\begin{abstract}

We formalise two semantics observing the expected running time of pGCL programs.
The first semantics is a denotational semantics providing a direct computation of the running time, similar to the weakest pre-expectation transformer.
The second semantics interprets a pGCL program in terms of a Markov decision process (MDPs), i.e.~it provides an operational semantics.
Finally we show the equivalence of both running time semantics.

We want to use this work to implement a program logic in Isabelle/HOL to verify the expected running time of pGCL programs.
We base it on recent work by Kaminski, Katoen, Matheja, and Olmedo.
We also formalise the expected running time for a simple symmetric random walk discovering a flaw in the original proof.

\end{abstract}

\section{Introduction}\label{introduction}

We want to implement expected running time analysis in Isabelle/HOL based on Kaminski~\etal~\cite{kaminski2016ert}.
They present semantics and proof rules to analyse the expected running time of probabilistic guarded command language (pGCL) programs.
pGCL is an interesting programming language as it admits probabilistic and non-deterministic choice, as well as unbounded while loops~\cite{mciver2004arp}.

Following \cite{kaminski2016ert}, in \autoref{sec:mdp} we formalise two running time semantics for pGCL and show their equivalence: a denotational one expressed as expectation transformer of type \isa{(\tv{s} \t{=>} \t{ennreal}) \t{=>} (\tv{s} \t{=>} \t{ennreal})}, and a operational one defining a Markov decision process (MDP).
This proof follows the equivalence proof of pGCL semantics on the expectation of program variables in \cite{hoelzl2016mdp} derived from the pen-and-paper proof by Gretz \etal~\cite{gretz2014pgclsem}.

Based on these formalisations we analyse the simple symmetric random walk, and show that the expected running time is infinite.
We started with the proof provided in \cite{kaminski2016ert}, but we discovered a flaw in the proof of the lower $\omega$-invariant based on the denotational semantics.
Now, our solution combines results from the probability measure of the operational semantics and the fixed point solution from the denotational semantics.

Both proofs are based on our formalisation of Markov chains and MDPs~\cite{hoelzl2016mdp}. The formalisation in this paper is on BitBucket%
\footnote{\url{https://bitbucket.org/johannes2011/avgrun}}.

\section{Preliminaries}

The formulas in this paper are oriented on Isabelle's syntax: type annotations are written \isa{t \hastype \tv{t}}, type variables can be annotated with type classes \isa{t \hastype \tv{t} \hastype \mathit{tc}} (i.e.~\isa{t} has type \isa{\tv{t}} which is in type class \isa{\mathit{tc}}), and type constructors are written in post-fix notation: e.g.~\isa{\t{set}{\tv{a}}}. We write \isa{\t{int}} for integers, \isa{\t{ennreal}} for extended non-negative real numbers: $[0, \infty]$, \isa{\t{stream}{\tv{a}}} for infinite streams of \isa{\tv{a}}, \isa{\t{pmf}{\tv{a}}} for probability mass functions (i.e.~discrete distributions) on \isa{\tv{a}}. The state space is usually the type variable \isa{\tv{s}}.
On infinite streams \isa{\c{sdrop}~n~\omega} drops the first $n$ elements from the stream $\omega$: \isa{\c{sdrop}~0~\omega = \omega} and \isa{\c{sdrop}~(n+1)~(s{\cdot}\omega) = \c{sdrop}~n~\omega}.

\paragraph{Least fixed points}
A central tool to define semantics are least fixed points on complete lattices: \isa{\tv{a}\t{=>}(\tv{b}\hastype\t{completelattice})}, \isa{\t{bool}}, \isa{\t{enat}}, and \isa{\t{ennreal}}.
Least fixed points are defined as \isa{\c{lfp}~f = \bigsqcap \{ u \mid f\,u \le u \} }.
For a monotone function $f$, we get the equations \isa{\c{lfp}\,f = f\,(\c{lfp}\,f)}.
Fixed point theory also gives nice algebraic rules: the rolling rule ``rolls'' a composed fixed point: \isa{g\,(\c{lfp}\,(\lambda x.~f\,(g\,x))) = \c{lfp}\,(\lambda x.~g\,(f\,x)))} for monotone $f$ and $g$, and the diagonal rule for nested fixed points: \isa{\c{lfp}\,(\lambda x.~\c{lfp}\,(f\,x)) = \c{lfp}\,(\lambda x.~f\,x\,x)}, for $f$ monotone in both arguments.

To use least fixed points in measure theory, countable approximations are necessary.
This is possible if the function $f$ is sup-continuous: $f\,(\bigsqcup_i C\,i) = \bigsqcup_i f\,(C\,i)$ for all chains~$C$.
Then $f$ is monotone and \isa{\c{lfp}\,f = \bigsqcup_i f^i\,\bot}.
For our proofs we also need an induction and a transfer rule%
\footnote{In our formalisation, the transfer rule is stronger: expectation requires measurability, hence we restrict the elements to which we apply $\alpha$ by some predicate~$P$.}:
%
\begin{mathpar}
\inferrule{\isa{\c{mono}\,f} \\
  \isa{\forall x \le \c{lfp}\,f.~P\,x \c{-->} P\,(f\,x)} \\
  \isa{\forall S.~(\forall x \in S.~P\,x) \c{-->} P\,(\bigsqcup S)}
}{\isa{P\,(\c{lfp}\,f)}} \and
\inferrule{\isa{\c{supcontinuous}\,f, g, \textrm{and}~\alpha} \\
\isa{\alpha\,\bot \le \c{lfp}\,g} \\ \alpha \circ f = g \circ \alpha}%
{\isa{\alpha (\c{lfp}\,f) = \c{lfp}\,g}}
\end{mathpar}

\paragraph{Markov chains (MCs) and Markov decision processes (MDPs)}
An overview of Isabelle's MC and MDP theory is found in~\cite{hoelzl2016mdp, hoelzl2013thesis}.
A MC is defined by a transition function \isa{K \hastype \tv{a}\t{=>}\t{pmf}{\tv{a}}}, inducing an expectation: \isa{\mathbb{E}^K_s[f]} is the expectation of $f$ over all traces in $K$ starting in $s$.
A MDP is defined by a transition function \isa{K \hastype \tv{a}\t{=>}\t{set}{\t{pmf}{\tv{a}}}}, inducing the maximal expectation: \isa{\hat{\mathbb{E}}^{K}_s[f]} is the supremum of all expectation of $f$ over all traces in $K$ starting in $s$.
Both expectations \isa{\mathbb{E}^K_s[f]} and \isa{\hat{\mathbb{E}}^K_s[f]} have values in \isa{\t{ennreal}}, which is a complete lattice.
Both are sup-continuous on measurable functions (called \emph{monotone convergent} in measure theory), which allows us to apply the transfer rule when $f$ is defined as a least fixed point.
Also both expectations support an iteration rule, i.e. we can compute them by first taking a step in $K$ and then continue in the resulting state $t$:
%
\[ \isa{\mathbb{E}^K_s[f] = \int_t \mathbb{E}^K_t[\lambda \omega.~f(t\cdot\omega)] \mathsf{d}K_s} \quad \textrm{and} \quad
\isa{\hat{\mathbb{E}}^K_s[f] = \bigsqcup_{D \in K_s} \int_t \hat{\mathbb{E}}^K_t[\lambda \omega.~f(t\cdot\omega)] \mathsf{d}D}. \]
%
Where $t\cdot\omega$ is the stream constructor and $\int f\mathsf{d}D$ is the integral over the pmf~$D$.

\section{Probabilistic Guarded Command Language (pGCL)}

The probabilistic guarded command language (pGCL) is a simple programming language allowing probabilistic assignment, non-deterministic choice and arbitrary While-loops. A thorough description of it using the weakest pre-expectation transformer (wp) semantics is found in McIver and Morgan~\cite{mciver2004arp}. Gretz~\etal~\cite{gretz2014pgclsem} shows the equivalence of wp with a operational semantics based on MDPs.
Hurd~\cite{hurd2005pgcl} and Cock~\cite{cock2012pgcl} provide a shallow embedding of pGCL in HOL4 and Isabelle/HOL.
We follow the definition in Kaminski~\etal~\cite{kaminski2016ert}.

\begin{figure}[t]
\[ \isa{ \begin{array}{lcl@{~|~}l}
\t{pgcl}{\tv{s}} & = & \c{Empty} ~~~~|~~~~ \c{Skip} ~~~~|~~~~ \c{Halt}
& \c{Assign}~(\tv{s} \t{=>} \t{pmf}{\tv{s}}) \\
& | & \c{Seq}~(\t{pgcl}{\tv{s}})~(\t{pgcl}{\tv{s}})
& \c{Par}~(\t{pgcl}{\tv{s}})~(\t{pgcl}{\tv{s}}) \\
& | & \c{If}~(\tv{s} \t{=>} \t{bool})~(\t{pgcl}{\tv{s}})~(\t{pgcl}{\tv{s}})
& \c{While}~(\tv{s} \t{=>} \t{bool})~(\t{pgcl}{\tv{s}})
\end{array} } \]
\caption{pGCL syntax}\label{fig:syntax}
\end{figure}
%
In \autoref{fig:syntax} we define a datatype representing pGCL programs over an arbitrary program state of type~\isa{\tv{s}}.
\isa{\c{Empty}} has not running time.
\isa{\c{Halt}} immediately aborts the program.
\isa{\c{Seq}} is for sequential composition.
\isa{\c{Par}} is for non-deterministic choice, i.e.~both commands are executed and then one of the results is chosen.
\isa{\c{Assign}}, \isa{\c{If}}, and \isa{\c{While}} have the expected behaviour, and all three commands require one time step.
A probabilistic choice is possible with \isa{\c{Assign}~u}, where $u$ is a probabilistic state transformer (\isa{\tv{s} \t{=>} \t{pmf}{\tv{s}}}).
The expected running time of \isa{\c{Assign}~u} weights each possible running time with the outcome of $u$.
The assignment is deterministic is $u$ is a Dirac distribution, i.e.~assigning probability $1$ to exactly one value.
We need the datatype to have a deep embedding of pGCL programs, which is necessary for the construction of the MDP.

\begin{figure}[t]
\begin{flalign*} \isa{ \begin{array}{lccl}
\multicolumn{4}{l}{%
\c{ert} \hastype \t{pgcl}{\tv{s}} \t{=>} (\tv{s} \t{=>} \t{ennreal}) \t{=>} (\tv{s} \t{=>} \t{ennreal})} \\
\c{ert}~\c{Empty} & f & = &
f \\
\c{ert}~\c{Skip} & f & = &
1 + f \\
\c{ert}~\c{Halt} & f & = &
0 \\
\c{ert}~(\c{Assign}~u) & f & = & 1 +
\displaystyle \lambda x.\, \int_y f\,y ~d(u\,x) \\
\c{ert}~(\c{Seq}~c_1~c_2) & f & = &
\c{ert}~c_1~(\c{ert}~c_2~f) \\
\c{ert}~(\c{Par}~c_1~c_2) & f & = &
\c{ert}~c_1~f \sqcup \c{ert}~c_2~f \\
\c{ert}~(\c{If}~g~c_1~c_2) & f & = &
1 + \lambda x.\, \c{if} g~x \c{then} \c{ert}~c_1~f~x \c{else} \c{ert}~c_2~f~x \\
\c{ert}~(\c{While}~g~c) & f & = &
\c{lfp}~(\lambda W\,x.\, 1 + \c{if} g~x \c{then} \c{ert}~c~W\,x \c{else} f~x)
\end{array} } \end{flalign*}
\caption{Expectation transformer semantics for pGCL running times}\label{fig:ert}
\end{figure}

\begin{figure}[t]
\[ \isa{ \begin{array}{lccl}
\multicolumn{4}{l}{%
\c{K} \hastype (\t{pgcl}{\tv{s}} \t{*} \tv{s}) \t{=>} \t{set}{\t{pmf}{(\t{pgcl}{\tv{s}} \t{*} \tv{s})}}} \\
\c{K} (\c{Empty},&s) & = &
\c{det}{\c{Empty}}{s} \\
\c{K} (\c{Skip},&s) & = &
\c{det}{\c{Empty}}{s} \\
\c{K} (\c{Halt},&s) & = &
\c{det}{\c{Halt}}{s} \\
\c{K} (\c{Assign}~u,&s) & = &
\{ [\lambda s'.\, (\c{Empty}, s')]~(u~s) \} \\
\c{K} (\c{Seq}~c_1~c_2,&s) & = & \left[\lambda (c', s').\, \left(\left\{ %
\begin{array}{ll}
c_2 & \textrm{if}~c'= \c{Empty} \\
\c{Halt} & \textrm{if}~c'= \c{Halt} \\
\c{Seq}~c'~c_2 & \textrm{otherwise}
\end{array} %
\right\}, s'\right)\right]~\c{K}~(c_1, s) \\
\c{K} (\c{Par}~c_1~c_2,&s) & = &
\c{det}{c_1}{s} \cup \c{det}{c_2}{s} \\
\c{K} (\c{If}~g~c_1~c_2,&s) & = &
\c{if} g\, s \c{then}~ \c{det}{c_1}{s} ~\c{else}~ \c{det}{c_2}{s} \\
\c{K} (\c{While}~g~c,&s) & = &
\c{if} g\,s \c{then}~ \c{det}{\c{Seq}~c~(\c{While}~g~c)}{s} ~\c{else}~ \c{det}{\c{Empty}}{s}
\end{array} } \]
%
\[ \isa{ \begin{array}{lllccl@{\qquad\qquad}lllccl}
\multicolumn{12}{l}{%
\c{cost} \hastype (\tv{s} \t{=>} \t{ennreal}) \t{=>} \t{pgcl}{\tv{s}} \t{=>} \tv{s} \t{=>} \t{ennreal} \t{=>} \t{ennreal}} \\
\c{cost}~f&\c{Empty} & s & \c{_} & = & f s &
\c{cost}~\c{_}&(\c{Seq}~\c{Empty}~\c{_}) & \c{_} & x & = & x \\
\c{cost}~\c{_}&\c{Halt} & \c{_} & \c{_} & = & 0 &
\c{cost}~f&(\c{Seq}~c~\c{_}) & s & x & = & \c{cost}~f~c~s~x \\
\c{cost}~\c{_}&(\c{Par}\;\c{_}\,\c{_}) & s & x & = & x &
\c{cost}~\c{_}&\c{_} & \c{_} & x & = & 1 + x
\end{array} } \]
\begin{center}
  \isa{\c{det}{c}{s}} is the singleton set of the singleton distribution $(c, s)$.\\
  $[f]\mu$ maps $f$ over all elements of $\mu$
\end{center}
%
\caption{MDP semantics for pGCL running times}\label{fig:mdp}
\end{figure}

\paragraph{Expected Running Time}

The denotational semantics for the running time is given as an expectation transformer, which is similar to the denotational semantics for the expectation of program variables as weakest pre-expectation transformers.
Again we follow the definition in Kaminski~\etal~\cite{kaminski2016ert}.
In \autoref{fig:ert} we define the expectation transformer \isa{\c{ert}} taking a pGCL command $c$ and an expectation $f$, where $f$ assigns an expected running time to each terminal state of $c$.
This gives a simple recursive definition of the \isa{\c{Seq}} case, for the expected running time of a pGCL program we will set $f = 0$.
We proved some validating theorems about expectation transformer \isa{\c{ert}}, i.e.~continuity and monotonicity of \isa{\c{ert}~c}, closed under constant addition for \isa{\c{Halt}}-free programs, sub-additivitiy, etc.

\paragraph{MDP Semantics}\label{sec:mdp}

For the operational small-step semantics we introduce a MDP constructed per pGCL program, and compute the expected number of steps until the program terminates.
In \autoref{fig:mdp} we define the MDP by its transition function $K$ and the per-state cost function \isa{\c{cost}~f~c~s~x}.
The per-state cost \isa{\c{cost}~f~c~s~x} computes the running time cost associated with the program $c$ at state $s$.
Here the program is seen as a list of statements, hence we walk along a list of \isa{\c{Seq}} and only look at its left-most leaf.
If the program is \isa{\c{Empty}} the MDP is stopped and we return $f~s$ containing further running time cost we want to associated to a finished state $s$ (in most cases this will be $0$, but it is essential in the induction case of \autoref{thm:erteq}).
When the execution continues we also add~$x$, c.f.~the definition of \isa{\c{coststream}}.

The transition function $K$ induces now a set of trace spaces, one for each possible resolution of the non-deterministic choices introduced by \isa{\c{Par}}.
We write $\hat{\mathbb{E}}^{K}_{(c, s)}[f]$ for the maximal expectation of \isa{f \hastype \t{stream}{(\t{pgcl}{\tv{s}} \t{*} \tv{s})} \t{=>} \t{ennreal}} when the MDP starts in $(c, s)$.
We define the cost of a trace as the sum of \isa{\c{cost}} over all states in the trace:
%
\[ \isa{\c{coststream}~f~((c, s){\cdot}\omega) \stackrel{\c{lfp}}{=} \c{cost}~f~c~s~(\c{coststream}~f~\omega)} \]
%
Finally the maximal expectation of \isa{\c{coststream}} computes \isa{\c{ert}}:%
\begin{theorem}\label{thm:erteq}
$ \isa{\hat{\mathbb{E}}^{K}_{(c, s)}[\c{coststream}~f] = \c{ert}~c~f~s} $
\end{theorem}
\begin{proof}[Induction on $c$] The interesting cases are \isa{\c{Seq}} and \isa{\c{While}}.
For \isa{\c{Seq}} we prove the equation
\isa{\hat{\mathbb{E}}^{K}_{(\c{Seq}~a~b, s)}[\c{coststream}~f]
  = \hat{\mathbb{E}}^{K}_{(a, s)}[\c{coststream}~(\lambda s.~\hat{\mathbb{E}}^{K}_{(b, s)}[\c{coststream}~f])]},
by fixed point induction in both directions.
For \isa{\c{While}} we prove
\[\isa{\hat{\mathbb{E}}^{K}_{(\c{While}~g~c, s)}[\c{coststream}~f] =
\c{lfp}~(\lambda F\;s.\;1+\c{if} g\,s \c{then} \hat{\mathbb{E}}^{K}_{(c, s)}[\c{coststream}~f] \c{else} f\;s)\;s}\]
by equating it to a completely unrolled version using fixed point induction and then massaging it in the right form using the rolling and diagonal rules. \qed
\end{proof}

\section{Simple Symmetric Random Walk}\label{sec:ssrw}

As an application for the expected running time analysis Kaminski~\etal~\cite{kaminski2016ert} chose the simple random walk.
As difference to \cite{kaminski2016ert} we do not use $\omega$-invariants to prove the infinite running time, but the correspondence of the program with a Markov chain (there is no non-deterministic choice).

The simple symmetric random walk (\isa{\c{srw}}) is a Markov chain on $\mathbb{Z}$, in each step $i$ it goes uniformly to $i + 1$ or $i - 1$ (i.e. in both cases with probability $1/2$).
Surprisingly, but well known (and formalised by Hurd~\cite{hurd2002thesis}), it reaches each point with probability $1$.
Equally surprising, the expected time for the srw to go from $i$ to $i + 1$ is infinite!
Kaminski~\etal~\cite{kaminski2016ert} prove this by providing a lower $\omega$-invariant.
Unfortunately, this proof has a flaw: in Appendix~B.1 of \cite{kaminski2016ert-ext} (the extended version of \cite{kaminski2016ert}), the equation $1 + \llbracket x > 0 \rrbracket \cdot 2 + \llbracket 1 < x \le n + 1 \rrbracket \cdot \infty + \llbracket 0 < x \le n - 1 \rrbracket \cdot \infty
= 1 + \llbracket x > 0 \rrbracket \cdot 2 + \llbracket 0 < x \le n + 1 \rrbracket \cdot \infty$
does not hold for $n=0$ and $x=1$.
The author knows from private communication with Kaminski~\etal~that it still is possible to use a lower $\omega$-invariant.
Unfortunately, the necessary invariant gets much more complicated.

After discovering the flaw in the proof, we tried a more traditional proof.
The usual approach in random walk theory uses the generating function of the first hitting time.
Unfortunately, this would require quite some formalizations in combinatorics, e.g.~Stirling numbers and more theorems about generating functions than available in \cite{hoelzl2016mdp}.
Finally, we choose an approach similar to \cite{hurd2002thesis}, i.e.~we set up a linear equation system and prove that the only solution is infinity.

Now, \isa{\c{srw} \hastype \t{int} \t{=>} \t{pmf}{\t{int}}} is the transition function for the simple symmetric random walk.
The expected time to reach $j$ when started in $i$ is written \isa{H~i~j \stackrel{\c{def}}{=} \mathbb{E}^{\c{srw}}_{i}[f~j]}, where \isa{f~j~(k\cdot\omega) \stackrel{\c{lfp}}{=}~ \c{if} j = k \c{then} 0 \c{else} 1 + f~j~\omega} is the first hitting time.
Now we need to prove the following rules: (I) \isa{H~j~i = H~j~k + H~k~i} if \isa{i \le j \le k},
(II) \isa{H~(i+t)~(j+t) = H~i~j}, (III) \isa{H~i~j = H~j~i} and (VI) \isa{H~i~j = (\c{if} i = j \c{then} 0 \c{else} 1 + (H~i~(j+1) + H~i~(j-1)) / 2)}. From these rules we can derive $H~i~j = \infty$ for $i \not= j$.

Rule (VI) is derived the expectation transformer semantics.
But it is not clear to us how to prove rule (I) by only applying fixed point transformations or induction.
Instead we prove (I) in a measure theoretic way:

\begin{align}
H~j~k + H~k~i
  & = \mathbb{E}^{\isa{\c{srw}}}_j[f~j + H~k~i] \nonumber\\
  & = \sum_n (n + H~k~i)\cdot \Pr_j(f~k = n) \label{eq:fin}\\
  & = \isa{\sum_n \mathbb{E}^{\c{srw}}_j[\lambda \omega.~(n + f~i~(\c{sdrop}~n~\omega)) \cdot
     \llbracket f~k~\omega = n\rrbracket]} \nonumber \\
  & = \isa{\sum_n \mathbb{E}^{\c{srw}}_j[f~i]} = H~j~i \label{eq:f}
\end{align}
%
\autoref{eq:fin} requires that $f~k$ is finite with probability $1$, we do a case distinction: if it is not finite a.e.~the result follows from $H~j~i \ge H~j~k = \infty$.
\autoref{eq:f} is now simply proved by induction on $n$.
The proofs for Equations~\ref{eq:fin} and~\ref{eq:f} essentially operate on each trace~$\omega$ in our probability space, making them inherently dependent on the trace space.

\begin{theorem}[The running time of \isa{\c{srw}} is infinite.]
$ H~i~j = \infty ~~\textrm{if}~~ i \not= j. $
\end{theorem}

\section{Coupon Collector}

Another example we formalised is the coupon collector example from \cite{kaminski2016ert}.
The idea is to compute the expected time until we collect $N$ different coupons from a uniform, independent and infinite source of coupons.
The left side of \autoref{fig:cc} shows our concrete implementation $\mathtt{CC}_N$, the right side is its refinement (there is no array $cp$ necessary).
By fixed point transformations we show that the (refined) inner loop's running time has a Geometric distribution, and hence the expected running time for $\mathtt{CC}_N$ is:
$ \isa{\c{ert}~\mathtt{CC}_N~0~s = 2 + 4N + 2N\sum_{i = 1}^{N} \frac{1}{i} } $ for $N>0$.

\begin{figure}

\[
\begin{array}{lr@{}c@{}ll}
x := 0, cp := [\overbrace{F, \ldots, F}^{N~\textrm{times}}], i := 0
& & & &
c := 0, b := F
\\
\mathtt{WHILE}~x < N~\mathtt{DO}
& x&{\rightarrow}&c &
\mathtt{WHILE}~c < N~\mathtt{DO}
\\
\quad \mathtt{WHILE}~cp[i]~\mathtt{DO}~ i :\sim \mathrm{Unif}\{0, \ldots, N\}
& \quad cp[i]&{\rightarrow}&b \quad &
\quad \mathtt{WHILE}~b~\mathtt{DO}~ b :\sim \mathrm{Bern}(x/N)
\\
\quad cp[i] := T, x := x + 1
& & |cp| = x & &
\quad b := T, c := c + 1
\\
\end{array}
\]

\caption{The Coupon Collector in pGCL and its refinement}\label{fig:cc}
\end{figure}

\section{Related Work}

The first formalisation of probabilistic programs was by Hurd~\cite{hurd2002thesis} in \texttt{hol98}, formalising a trace space for a stream of probabilistic bits.
Hurd~\etal~\cite{hurd2005pgcl} is different approach, formalising the weakest pre-expectation transformer semantics of pGCL in HOL4.
Both formalisations are not related.
Audebaud and Paulin-Mohring~\cite{audebaud2009randomizedalgos} use a shallow embedding of a probability monad in Coq.

\pagebreak
\noindent
Cock~\cite{cock2012pgcl} provides a VCG for pGCL in Isabelle/HOL.
Hölzl and Nipkow~\cite{hoelzl2012casestudies, hoelzl2013thesis} formalises MCs and analyses the expected running time of the ZeroConf protocol.
On the basis of \cite{hoelzl2013thesis} formalises MDPs and shows the equivalence of the weakest pre-expectation transformer (based on the pen-and-paper proof in \cite{gretz2014pgclsem}).

Unlike \autoref{thm:erteq}, these formalisations either define denotational semantics~\cite{hurd2005pgcl, audebaud2009randomizedalgos, cock2012pgcl}, or operational semantics~\cite{hurd2002thesis, hoelzl2012casestudies, hoelzl2013thesis}, none of them relate both semantics.

\section{Conclusion and Future Work}

While formalising the random walk example in \cite{kaminski2016ert} we found an essential flaw in the proof in\cite{kaminski2016ert-ext}.
Our solution seams to indicate, that for the verification of expected running times an $\omega$-invariant approach is not enough.
While the expectation transformer gives us a nice verification condition generator (e.g.~\cite{cock2012pgcl}), the trace space might be required to get additional information i.e.~fairness and termination.
The equivalence between the expectation transformer semantics and the MDP semantics provides the required bridge between both worlds.
Also we might require a probabilistic, relational Hoare logic (maybe based on \cite{lochbihler2016proboracles}) to automate tasks like \autoref{fig:cc}.

\bibliographystyle{lncs}

\bibliography{itp2016}

\end{document}


As discussed in Sec.~\ref{sec:pp_physics_jets}, color confinement prevents the existence of free quarks and gluons.
A quark or gluon produced at the LHC typically undergoes hadronization and presents itself in the detector as a collection of collimated particles.
Jets are built from PF candidates, using the anti-$k_{\text{T}}$ clustering algorithm~\cite{Cacciari:2008gp,Cacciari:2011ma} with a distance parameter of 0.4.
The input PF candidates have charged hadron subtraction (CHS) applied, meaning charged hadrons associated with vertices other than the primary vertex of that event are removed.
CHS reduces the contribution of particles originating from pileup vertices.
Once the jets are built from PF candidates, three steps are taken to correct the jets' energies.

First, a pileup offset correction is applied to remove additional jet contributions from pileup not removed by CHS.
The pileup contributions not remove by CHS are primarily charged hadrons not matched to a good vertex and PF photons.
The individual jet energies are corrected by a multiplicative factor derived in simulation, parametrized in bins of jet area ($A$), $\rho, \pT,$ and $\eta$.
Typical values for the pileup offset corrections are shown in Fig.~\ref{fig:evt_jet_pu_corrections}.
\begin{figure} [h!]
    \centering
    \begin{tabular}{c c}
        \includegraphics[width=0.48\linewidth]{figures/event_reconstruction_and_selection/jetmet8Tev_Figure_009-c.png} &
        \includegraphics[width=0.48\linewidth]{figures/event_reconstruction_and_selection/jetmet8Tev_Figure_009-d.png}
    \end{tabular}
    \caption{Pileup offset correction values as a function of jet \pT (left) and jet $|\eta|$ (right). Taken from~\cite{Khachatryan_2017_jets}.}
    \label{fig:evt_jet_pu_corrections}
\end{figure}

Second, jet energy scale corrections designed to correct for the detector response to jets are derived in simulation, again in bins of jet area ($A$), $\rho, \pT,$ and $\eta$.
The goal of this step is to correct the reconstructed jet energy to match that of the true jet energy (only available in simulation).
Typical values for the jet energy scale corrections are shown in Fig.~\ref{fig:evt_jet_jec_corrections}.
\begin{figure} [h!]
    \centering
    \begin{tabular}{c c}
        \includegraphics[width=0.48\linewidth]{figures/event_reconstruction_and_selection/jetmet8Tev_Figure_014-a.png} &
        \includegraphics[width=0.48\linewidth]{figures/event_reconstruction_and_selection/jetmet8Tev_Figure_014-b.png}
    \end{tabular}
    \caption{Jet energy scale correction values as a function of jet \pT (left) and jet $|\eta|$ (right). Taken from~\cite{Khachatryan_2017_jets}.}
    \label{fig:evt_jet_jec_corrections}
\end{figure}

Third, remaining differences between data and simulation are corrected with a residual correction applied to data, derived as a function of \pT and $\eta$.
A variety of event topologies ($\gamma$ + jets, \Zee + jets, \Zuu + jets, and di-jet) are used to derive these corrections.
In each topology, the underlying strategy is the same: exploit the momentum conservation in the transverse plane betwen a well-measured reference object and the jet to be corrected.
The fact that the reference object ($\gamma$, \Zee, \Zuu, a well-measured central jet) is well-measured allows us to infer the true energy of the jet to be corrected.
The full details of these procedures are described in Ref.~\cite{Khachatryan_2017_jets}.

Jets used in the \ttH analysis are first corrected with the procedures described in this section.
They are further required to have $\pT > 25$ GeV and $|\eta| < 2.4$ and must pass a loose pileup jet ID criteria.
The loose pileup jet ID criteria is based on a BDT designed to discriminate between jets originating from pileup interactions and those originating from the primary vertex in the event.
The BDT is trained with variables describing the jets' shape as well as additional track information.
Jets are finally also required to not be overlapping with any photons or leptons in the event, requiring $\Delta R(\text{jet}, \text{photon/lepton}) > 0.4$.

\subsubsection*{b-Tagged Jets}
Hadronic jets at the LHC typically result from the hadronization of either a quark or gluon (with the exception of top quarks, which decay before they are able to hadronize).
Jets originating from a light-flavor quark (u,d,s) or a gluon are typically indistinguishable in the CMS detector.
However, jets originating from c or b quarks often have distinguishing features.
While hadrons containing only light-flavor quarks (as would typically be produced by the hadronization of a light-flavor quark or a gluon) often reach the calorimeters before decaying, hadrons containing b quarks tend to decay on a length scale of a few millimeters when produced at typical LHC energies.
Hadrons containing charm quarks frequently decay even sooner than this.
The resolution of the tracker is sufficient to distinguish the vertices of these decays, called ``secondary vertices'', from the primary vertices in the event.

Jet flavor tagging algorithms attempt to exploit information about the secondary vertices associated with a given jet to determine the flavor of the quark (or gluon) it originated from.
Machine learning algorithms are often used to classify jet flavor, using information about the secondary vertices, tracks, and pf candidates associated with a given jet.
Recently, algorithms built with deep neural networks have shown significantly improved jet flavor tagging performance over more traditionally used methods, such as those based on boosted decision trees~\cite{Guest_2016}.
The DeepCSV~\cite{Sirunyan_2018_deepcsv} algorithm is one such DNN-based tagger.
For a given jet, the algorithm assigns multiple flavor scores, indicating its degree of certainty that the jet originated from a quark of that flavor.
DeepCSV outputs scores corresponding to its degree of certainty that the jet originated from a b quark, c quark, light flavor quark (u,d,s) or gluon, and a \bb pair (four scores).
The performance of DeepCSV (purple) and other commonly used jet flavor algorithms is shown in Fig.~\ref{fig:evt_jet_btag}.
\begin{figure} [h!]
    \centering
    \includegraphics[width=\linewidth]{figures/event_reconstruction_and_selection/deepcsv_Figure_016.png}
    \caption{Misidentification rate as a function of b-tagging efficiency, shown for b vs. c jet discrimination (dotted lines) and b vs. light jet discrimination (solid lines). Taken from~\cite{Sirunyan_2018_deepcsv}.}
    \label{fig:evt_jet_btag}
\end{figure}

Jet flavor tagging is particularly useful for the \ttH analysis, as two b quarks are produced in the decay of the \ttb pair.
As the multi-jet, \gjets, and \dipho backgrounds primarily feature jets originating from light flavor quarks or gluons, the ability to select b-tagged jets allows for rejection of a significant component of the overall background.


\documentclass{memoir}
\usepackage{notestemplate}

%\logo{~/School-Work/Auxiliary-Files/resources/png/logo.png}
%\institute{Rice University}
%\faculty{Faculty of Whatever Sciences}
%\department{Department of Mathematics}
%\title{Class Notes}
%\subtitle{Based on MATH xxx}
%\author{\textit{Author}\\Gabriel \textsc{Gress}}
%\supervisor{Linus \textsc{Torvalds}}
%\context{Well, I was bored...}
%\date{\today}

\begin{document}

% \maketitle

% Notes taken on ??

\section{Minimal Polynomials}
\label{sec:minimal_polynomials}

\begin{prop}
	Let \(\alpha \) be an algebraic element over \(F\).
	\begin{enumerate}[(a).]
		\item Then there exists a monic irreducible polynomial of minimal degree \(m_{\alpha ,F}(x) \in F[x]\) which has \(\alpha \) as a root.
		\item A polynomial \(f(x) \in F[x]\) has \(\alpha \) as a root if and only if \(m_{\alpha ,F}(x) \mid f(x)\) in \(F[x]\).
		\item The polynomial \(m_{\alpha ,F}(x)\) with the property in (a) is unique.
	\end{enumerate}
\end{prop}
We can see the minimal polynomial must be irreducible, because otherwise one of its factors would have \(\alpha \) as a root and hence has degree smaller than \(m_{\alpha ,F}(x)\), contradicting our hypothesis. The divisibility \(m_{\alpha ,F}(x) \mid f(x)\) follows from the division algorithm in \(F[x]\). The divisibility and minimality conditions together give uniqueness.
\begin{cor}
	If \(K / F\) is a field extension, and \(\alpha \) is algebraic over both \(F\) and \(K\), then \(m_{\alpha ,K}(x)\) divides \(m_{\alpha ,F}(x)\) in \(K[x]\).
\end{cor}
This directly follows as \(m_{\alpha ,F}(x)\) has a root \(\alpha \) in \(K\) and hence (b) gives us divisibility.
\begin{defn}
	The polynomial \(m_{\alpha ,F}(x)\) is called the \textbf{minimal polynomial of \(\alpha \) over \(F\)}. The degree of \(m_\alpha (x)\) is called the \textbf{degree of \(\alpha \)}.\\

In other words, the minimal polynomial of \(\alpha \) over \(F\) is a monic irreducible polynomial over \(F\) that has \(\alpha \) as a root. Alternatively, it is a monic polynomial over \(F\) of minimal degree with \(\alpha \) as a root-- both imply the other.
\end{defn}

\begin{prop}
	Let \(\alpha \) be algebraic over \(F\). Then
	\begin{align*}
		F(\alpha ) \cong F[x] / (m_{\alpha }(x))
	\end{align*}
	So that \([F(\alpha ):F] = \textrm{deg}m_{\alpha }(x) \equiv \textrm{deg}\alpha\).
\end{prop}

\begin{prop}
	An element \(\alpha  \in F\) is algebraic over \(F\) if and only if the simple extension \(F(\alpha ) / F\) is finite.\\

	If \(\alpha \in K\) with \([K:F] = n\), then \(\textrm{deg}(\alpha) \leq n\).
\end{prop}
This follows by applying linear dependence to powers \(\alpha^i\) with \(i = 0,1,\ldots,n\).
\begin{cor}
	If \(K / F\) is finite, then \(K / F\) is algebraic.
\end{cor}

\begin{exmp}
	Take \(F\) to be a field with \(\textrm{char}(F) \neq 2\). Consider \(K/F\) of degree 2, which is hence algebraic. Let \(\alpha  \in K / F\) so that \(\alpha \) is a root of a polynomial over \(F\) of degree 1 or 2. Because \(\alpha \not\in F\), the polynomial must has degree 2.\\

	This implies that \(m_{\alpha ,F}(x) = x^2+bx+c\) for \(b,c \in F\). This implies that \(F(\alpha )\) has the same dimension of \(K\) and hence \(K = F(\alpha )\) (as \(K\) is a field extension of \(F(\alpha )\). This implies that \(K = F(\sqrt{b^2-4ac} )\) and so any degree 2 extension of a field \(F\) with characteristic not equal to \(2\) is of the form \(F(\sqrt{D} )\) for \(D\) a non-square element of \(F\).\\

	Conversely, for such a field, \([F(\sqrt{D} ) : F] = 2\) and hence extensions of the form \(F(\sqrt{D} ) / F\) are called \textbf{quadratic extensions of \(F\)}.
\end{exmp}
\end{document}

Abstract Algebra - Introductory/Algebra II/Notes/source/Lecture16 - MinimalPoly.tex


\documentclass[9pt,a4paper]{article}

\usepackage{amsmath,amsfonts,mathrsfs, amssymb, amsthm}
\usepackage{calc}
\usepackage[utf8]{inputenc}
\usepackage{comment}
\usepackage{nicefrac}
\usepackage{todonotes}
\usepackage{abstract}
\usepackage{subfig}
\usepackage{rotating}
\usepackage{setspace}

\usepackage{ulem}
\normalem

\usepackage{titlesec}
\titleformat{\section}{\large\bfseries}{\thesection}{1em}{}
\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection}{1em}{}

\usepackage[english]{babel}
\usepackage[round]{natbib}

%\setlength{\parindent}{0in}
\usepackage{paralist}
\usepackage{fancyhdr,lastpage}

\usepackage{color,hyperref}

%\setstretch{baselinestretch}

\usepackage{tabularx}
\newcolumntype{R}[1]{>{\raggedleft\arraybackslash}p{#1}}

\newcommand{\tab}{\hspace{4 pt}~}

\newcommand{\st}{\text{s.t.} \quad}
\newcommand{\free}{\text{ (free) } }
\newcommand{\botcomma}{\; , \,}

\newcommand{\fixedx}{\mathbf{\overline{x}_i}}
\newcommand{\upfixedx}{^{\mathbf{(\overline{x}_i)}}}
\newcommand{\one}{\mathbf{1}}
\newcommand{\zero}{\mathbf{0}}
\newcommand{\upone}{^{\mathbf{(1)}}}
\newcommand{\upzero}{^{\mathbf{(0)}}}

\newcommand{\on}{^{\,\text{on}}}
\newcommand{\off}{^{\,\text{off}}}

\DeclareMathOperator*{\argmin}{\arg \min}

\title{\Large An exact solution method for \\ binary equilibrium problems with compensation \\
	and the power market uplift problem
\author{\normalsize Daniel Huppmann, Sauleh Siddiqui \\
	\small huppmann@iiasa.ac.at, siddiqui@jhu.edu
}
\date{\normalsize Mathematical cheat sheet, \today}
}

% Linearize variables, relax constraints
% We call it "binary quasi-equilibrium"

\begin{document}
\maketitle

This document summarizes the mathematical formulation
for easier reference of the GAMS implementation
provided under an open-source license on GitHub
(\url{http://danielhuppmann.github.io/binary_equilibrium/}).
All equation numbers in this document are identical
to the published version of the manuscript.

\vspace{4 pt} \noindent
Please cite as:
\newline
D.~Huppmann and S.~Siddiqui.
An exact solution method for binary equilibrium problems with compensation
and the power market uplift problem.
\newline
\emph{European Journal of Operation Research}, 266(2):622-638, 2018 \\
doi: \href{https://dx.doi.org/10.1016/j.ejor.2017.09.032}{10.1016/j.ejor.2017.09.032}

\section*{Theoretical formulation (Section 3.4)}

\setcounter{equation}{13}
\begin{subequations} \label{model:overall:problem}
\begin{align}
\min_{
\substack{x_i,y_i,
\widetilde{y}\upfixedx_i,\widetilde{\lambda}\upfixedx_i \\  \kappa\upfixedx_i,\zeta\upfixedx_i}}
\quad F\Big(\big(x_i,y_i\big)_{i \in I}\Big) + G\Big(\big(\zeta\upfixedx_i\big)_{i \in I}\Big) \hspace{3 cm} \label{model:overall:problem:objective}
\end{align}
\vspace{-0.5 cm}\begin{align}
% first order conditions of y_i (specific formulation 1)
\st \nabla_{y_i} \, f_i\Big(\one,\widetilde{y}_i\upone,y_{-i}\Big) + \big(\widetilde{\lambda}_i\upone\big)^T \nabla_{y_i} \, g_i\Big(\one,\widetilde{y}_i\upone\Big) &= 0 \label{model:overall:problem:KKT:1}\\
%
0 \leq - g_i\Big(\one,\widetilde{y}_i\upone \Big)& \ \bot \ \widetilde{\lambda}_i\upone \geq 0 \label{model:overall:problem:KKT:constraints:1} \\
% first order conditions of y_i (specific formulation 0)
\nabla_{y_i} \, f_i\Big(\zero,\widetilde{y}_i\upzero,y_{-i}\Big) + \big(\widetilde{\lambda}_i\upzero\big)^T \nabla_{y_i} \, g_i\Big(\zero,\widetilde{y}_i\upzero\Big) &= 0 \label{model:overall:problem:KKT:0}\\
%
0 \leq - g_i\Big(\zero,\widetilde{y}_i\upzero \Big)& \ \bot \ \widetilde{\lambda}_i\upzero \geq 0 \label{model:overall:problem:KKT:constraints:0} \\
% incentive compatibility
f_i\Big(\one,y_i\upone,y_{-i}\Big) + \kappa_i\upone - \zeta_i\upone - \kappa_i\upzero + \zeta_i\upzero&= f_i\Big(\zero,y_i\upzero,y_{-i}\Big) \label{model:overall:problem:incentive} \\
% translation
\kappa_i\upone + \zeta_i\upone&\leq  x_i \, \widetilde{K}  \label{model:overall:problem:duals1} \\
\kappa_i\upzero + \zeta_i\upzero&\leq \big(1-x_i\big) \, \widetilde{K}  \label{model:overall:problem:duals0} \\
% % % translation of individual optimal solution to other players
\widetilde{y}_i\upzero - x_i \, \widetilde{K} \leq y_i &\leq \widetilde{y}_i\upzero + x_i \, \widetilde{K}  \label{model:overall:problem:translate0} \\
\widetilde{y}_i\upone - \big(1-x_i \big) \, \widetilde{K} \leq
y_i &\leq \widetilde{y}_i\upone + \big(1-x_i \big) \, \widetilde{K} \label{model:overall:problem:translate1} \\
x_i \in \{0,1\}, \big(y_i,\widetilde{y}_i\upfixedx \big) \in \mathbb{R}^{3m}, \big(\lambda_i\upfixedx,
& \kappa_i\upfixedx,\zeta_i\upfixedx\big) \in \mathbb{R}_+^{2k+4}
 \nonumber
\end{align}
\end{subequations}

\vfill
\newpage
\setcounter{table}{1}
\begin{table}
	\begin{center}
		\begin{small}
			\begin{tabular}{p{0.1 cm}p{1.9 cm}@{ ... }p{8.5 cm}}
				\hline
				\hline
				\multicolumn{3}{l}{\textbf{Sets \& Mappings}} \\
				&$n,m \in N$	& nodes \\
				&$t \in T$		& time step, hours \\
				&$i \in I$		& generators, power plant units \\
				&$j \in J$		& load, demand units \\
				&$l \in L$		& power lines \\
				&$i \in I_n, j \in J_n$ & generator/load unit mapping to node $n$ \\
				&$n(i), n(j)$ & node mapping to generator $i$/load unit $j$ \\
				&$\phi \in \Phi$	& set of dispatch options (schedules) for each generator \\
				&$t \in T_\phi$	& hours in which a generator is active in dispatch option $\phi$ \\
				\hline
				\multicolumn{3}{l}{\textbf{Primal variables}} \\
				&$x_{ti}$		& on/off decision for generator $i$ in hour $t$ \\
				&$z\on_{ti},z\off_{ti}$ & inter-temporal start-up/shut-down decision \\
				&$y_{ti}$		& actual generation by generator $i$ in hour $t$ \\
				&$y\on_{ti}$		& generation if binary variable is fixed at $\fixedx$ \\
				&$d_{tj}$		& demand by unit $j$ in hour $t$ \\
				&$\delta_{tn}$	& voltage angle \\
				\hline
				\multicolumn{3}{l}{\textbf{Dual variables}} \\
				&$\alpha\on_{ti},\beta\on_{ti}$	& dual to minimum activity/maximum generation capacity \\
				&$\nu_{tj}$			& dual to maximum load constraint \\
				&$\mu^+_{tl},\mu^-_{tl}$ & dual to voltage angle band constraints \\
				&$\xi^+_{tn},\xi^-_{tn}$ & dual to thermal line capacity constraints \\
				&$\gamma_t$	& dual to slack bus constraints \\
				\hline
				\multicolumn{3}{l}{\textbf{Switch and compensation variables}} \\
				&$p_{tn}$		& locational marginal price \\
				&$\kappa\on_{ti},\kappa\off_{ti}$	& switch value (defined per time step)\\
				&$\zeta_{i}$ & compensation payment (defined over entire time horizon) \\
				\hline
				\multicolumn{3}{l}{\textbf{Parameters}} \\
				&$c^G_{i}$		& linear generation costs \\
				&$c\on_i,c\off_i$		& start-up/shut-down costs \\
				&$c^D_{\phi}$		& commitment costs in dispatch option $\phi$ (start-up, shut-down)\\
				&$g^{min}_i$		& minimum activity level if power plant is online \\
				&$g^{max}_i$		& maximum generation capacity \\
				&$x^{init}_i$		& power plant status at start of model horizon ($t=0$)\\
				&$u^D_{tj}$			& utility of demand unit $j$ for using electricity \\
				&$d^{max}_{tj}$	& maximum load of unit $j$ \\
				&$f^{max}_l$		& thermal capacity of power line $l$ \\
				&$B_{nk},H_{lk}$			& line/node susceptance/network transfer matrices \\
				\hline
				\hline
			\end{tabular}
		\end{small}
		\caption{Notation for the nodal power market problem} \label{table:example:notation}
	\end{center}
\end{table}

\section*{The power market application (Section 4)} \label{sec:example}
\setcounter{equation}{15}

\subsubsection*{The generator's optimization problem}
Each generator~$i \in I$ seeks to maximize her profits from generating and selling electricity over the time horizon $t \in T$:
\begin{subequations} \label{example:generator:optimization}
\begin{align}
\min_{x_{ti},y_{ti},z\on_{ti},z\off_{ti}} &\quad  - p_{tn(i)} y_{ti} + c^G_{i} y_{ti} + c\on_{i} z\on_{ti} + c\off_{i} z\off_{ti}  \label{example:generator:objective} \\
&\st x_{ti} g_i^{min} \leq y_{ti} \leq x_{ti} g_i^{max} \quad \big(\alpha\on_{ti},\beta\on_{ti}\big)  \label{example:generator:con:generation} \\
& \hspace{0.8 cm} x_{ti} - x_{(t-1)i} = z\on_{ti} - z\off_{ti} \label{example:generator:con:intertemporal} \\
& \hspace{1 cm} x_{ti} \in \{0,1\}, \quad y_{ti},z\on_{ti},z\off_{ti}  \in \mathbb{R}_+ \nonumber
\end{align}
\end{subequations}

\subsubsection*{Demand for electricity and network constraints}
The other side of the market is a player seeking to maximize the welfare (utility) of consumers while guaranteeing feasibility of the transmission system, given locational marginal prices~$p_{tn}$.
A set of units~$j \in J$ consume electricity (load~$d_{tj}$), each located at a specific node~$n(j)$. The sets~$I_n$ and~$J_n$ are the generators and load units located at node~$n$, respectively. There are a set of power lines~$l \in L$ connecting the nodes; the direct-current load flow (DCLF) characteristics are captured using the susceptance matrix~$B_{nm}$ (node-to-node) and network transfer matrix~$H_{nl}$ (node-to-line mapping). This approach is equivalent to a power transfer distribution factor (PTDF) matrix.
\setcounter{equation}{17}

\begin{subequations}\label{example:ISO:optimization}
\begin{align}
\min_{d_{tj}, \delta_{tn}} \quad \sum_{j \in J} p_{tn(j)} \big(d_{tj} + \sum_{m \in N} B_{nm} \delta_{tm} \big) - u^D_{tj} d_{tj} &
\label{example:ISO:optimization:objective} \\
	\st d^{max}_{tj} - d_{tj} \geq 0 &\quad (\nu_{tj}) \label{example:ISO:optimization:demand:max} \\[4 pt]
f^{max}_l - \sum_{n \in N} H_{ln} \delta_{tn} \geq 0 &\quad (\mu^+_{tl}) \label{example:ISO:optimization:flow:pos} \\[-2 pt]
f^{max}_l + \sum_{n \in N} H_{ln} \delta_{tn} \geq 0 &\quad (\mu^-_{tl})\label{example:ISO:optimization:flow:neg} \\[-4 pt]
\pi - \delta_{tn} \geq 0 &\quad (\xi^+_{tn}) \label{example:ISO:optimization:angle:pos} \\[2 pt]
\pi + \delta_{tn} \geq 0 &\quad (\xi^-_{tn}) \label{example:ISO:optimization:angle:neg} \\[2 pt]
\delta_{t\hat{n}} = 0  & \quad (\gamma_t) \label{example:ISO:slackbus}
\end{align}
\end{subequations}

\subsubsection*{The bi-level multi-objective program}
\begin{subequations}
\begin{align}
\min \ \sum_{t \in T} \bigg[
\sum_{i \in I} c^G_{i} y_{ti} + c\on_{i} z\on_{ti} + c\off_{i} z\off_{ti}
- \sum_{j \in J} & u^D_{tj} d_{tj}
\bigg]
+ \sum_{i \in I} \zeta_i
	\tag{20a} \label{example:MO:optimization} \\[5 pt]
% subject to energy balance constraint at each node
\st \sum_{j \in J_n} d_{tj} - \sum_{i \in I_n} y_{ti} + \sum_{m \in N} B_{nm} \delta_{tm} = 0 &
\tag{20b} \label{example:MO:MBC} \\
% demand stationarity condition and network feasibility constraints
0 \leq - u^D_{tj} + p_{tn(j)} + \nu_{tj} &\quad \bot \quad d_{tj} \geq 0
	\tag{19a} \label{example:ISO:KKT:demand} \\[2 pt]
0 = \sum_{m\in N} B_{mn} p_{tm}
+ \sum_{l \in L} H_{ln} \big( \mu^+_{tl} - \mu^-_{tl}\big) \quad & \nonumber \\[- 4 pt]
+ \xi^+_{tn} - \xi^-_{tn} -
\genfrac\{\}{0pt}{0}{\gamma_t \quad \text{if }n = \hat{n}}{0 \quad \text{else} \hspace{0.55 cm}}
& \quad \botcomma \quad \delta_{tn} \free \tag{19b} \label{example:ISO:KKT:delta} \\
0 \leq d^{max}_{tj} - d_{tj} &\quad \bot \quad \nu_{tj} \geq 0
	\tag{19c} \label{example:ISO:KKT:demand:max} \\
0 \leq f^{max}_l - \sum_{n \in N} H_{ln} \delta_{tn} &\quad \bot \quad \mu^+_{tl} \geq 0
	\tag{19d} \label{example:ISO:KKT:flow:pos} \\
0 \leq f^{max}_l + \sum_{n \in N} H_{ln} \delta_{tn} &\quad \bot \quad  \mu^-_{tl} \geq 0
	\tag{19e}  \label{example:ISO:KKT:flow:neg} \\
0 \leq \pi - \delta_{tn} &\quad \bot \quad \xi^+_{tn} \geq 0
	\tag{19f}  \label{example:ISO:KKT:angle:pos} \\
0 \leq \pi + \delta_{tn} &\quad \bot \quad \xi^-_{tn} \geq 0
	\tag{19g} \label{example:ISO:KKT:angle:neg} \\[2 pt]
0 = \delta_{t\hat{n}}  &\quad \botcomma \quad \gamma_t \free
	\tag{19h} \label{example:ISO:KKT:slackbus} \\
% generator optimality conditions (for x=1)
0 = c^G_{i} - p_{tn(i)} + \beta\on_{ti} - \alpha\on_{ti} &\quad \botcomma \quad y\on_{ti} \free
	\tag{17a} \\
0 \leq - g^{min}_{ti} + y\on_{ti} &\quad \bot \quad \alpha\on_{ti} \geq 0
	\tag{17b} \label{example:generator:KKT:min} \\
0 \leq g^{max}_{ti} - y\on_{ti} &\quad \bot \quad \beta\on_{ti} \geq 0
	\tag{17c} \label{example:generator:KKT:max} \\
% binary equilibrium formulation
x_{(t-1)i} + z\on_{ti} - z\off_{ti} = x_{ti} &
	\tag{21a} \label{example:BNE:generator:intertemporal} \\
\beta\on_{ti} g^{max}_{ti} - \alpha\on_{ti} g^{min}_{ti} - \kappa\on_{ti} + \kappa\off_{ti} =0 &
	\tag{21b} \label{example:BNE:generator:profits} \\
|\kappa\on_{ti}| \leq x_{ti} \, \widetilde{K} &
	\tag{21c} \label{example:BNE:kappa_on} \\
|\kappa\off_{ti}| \leq (1-x_{ti}) \, \widetilde{K} &
	\tag{21d} \label{example:BNE:kappa_off} \\
\sum_{t \in T} \bigg[\kappa\on_{ti}
- c\on_{i} z\on_{ti} -  c\off_{i} z\off_{ti}  \bigg] + \zeta_{i}
\geq & \quad \nonumber \\
\sum_{t \in T_\phi} \bigg[\beta\on_{ti} g^{max}_{ti} - \alpha\on_{ti} g^{min}_{ti}& \bigg]
- c^D_{\phi i}
\quad \forall~\phi \in \Phi
	\tag{21e}\label{example:BNE:generator:incentive} \\
0 \leq y_{ti} &\leq x_{ti} \, g^{max}_{ti}
	\tag{21f} \\[2pt]
y\on_{ti} - (1-x_{ti}) \, g^{max}_{ti} \leq y_{ti} &\leq y\on_{ti} + (1-x_{ti}) \, g^{max}_{ti}
	\tag{21g}
\end{align}
\end{subequations}
\end{document}