\chapter{Course Introduction}
\label{cha:intro}

\sloppy
%\lecture{1 --- Wednesday, February 19th}
%{Spring 2020}{Rasmus Kyng}{Course Introduction}

\section{Overview}
 This course will take us quite deep into modern approaches to
 graph algorithms using convex optimization techniques.
%
 By studying convex optimization through the lens of graph algorithms,
 we'll try to develop an understanding of fundamental
 phenomena in optimization.
 %
 Much of our time will be devoted to flow problems on graphs.
 We will not only be studying these problems for their own sake,
 but also because they often provide a useful setting for thinking more broadly about optimization.

 The course will cover some traditional discrete approaches to various graph
 problems, especially flow problems, and then contrast these approaches
 with modern, asymptotically faster methods based on combining convex
 optimization with spectral and combinatorial graph theory.

\section{Electrical Flows and Voltages - a Graph Problem from Middle School?}

We will dive right into graph problems by considering how electrical
current moves through a network of resistors.

First, let us recall some middle school physics.
If some of these things don't make sense two you, don't worry, in less
than paragraph from here, we'll be make to safely doing math.

Recall that a typical battery that buy from Migros has two
endpoints, and produces what is called a \emph{voltage
  difference} between these endpoints.

One end of the battery will have a positive charge (I think that means an excess of
positrons\footnote{I'm joking, of course! Try Wikipedia if you want to know
more. However, you will not need it for this class.}), and the other a
negative charge.
If we connect the two endpoints with a wire, then a current will flow
from one end of the battery to the other in an attempt to even out
this imbalance of charge.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_battery9volts.png}
  \captionof{figure}{A 9 volts battery with a wire attached.}
  \label{fig:battery-volt}
\end{figure}

We can also imagine a kind of battery that tries to send a certain
amount of current the wires between its endpoints, e.g. 1 unit of charge per
unit of time.
This will be a little more convenient to work with, so let us focus on
that case.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_battery1ampere.png}
  \captionof{figure}{A 1 ampere battery with a wire attached.}
  \label{fig:battery-current}
\end{figure}

A \emph{resistor} is a piece of wire that connects two
points $u$ and $v$, and is completely described by a single number $r$
called its \emph{resistance}.

If the voltage difference between the endpoints of the resistor is
$x$, and the resistance is $r$ then this will create a flow of charge per unit of time of $f = x / r$.
This is called Ohm's Law.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_ohmslawx-vs-f.png}
  \captionof{figure}{Ohm's Law for a resistor with resistance $r = 1$.}
  \label{fig:ohmslaw}
\end{figure}

Suppose we set up a bunch of wires that route electricity from our
current source $s$ to our current sink $t$ in some pattern:

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_graphpath.png}
  \captionof{figure}{A path of two resistors.}
  \label{fig:graphpath}
\end{figure}

We have one unit of charge flowing out of $s$ per unit of time, and
one unit coming into $t$.
Because charge is conserved, the current flowing into any other point
$u$ must equal the amount flowing out of it.
This is called Kirchoff's Current Law.

To send one unit of current from $s$ to $t$, we must be sending it
first form $s$ to $u$ and then from $u$ to $t$.
So the current on edge $(s,u)$ is 1 and the current on $(u,t)$ is 1.
By Ohm's Law, the voltage difference must also be 1 across each of the two
wires.
Thus if the voltage is $x$ at $s$, it must be $x+1$ at $u$ and $x+2$
at $t$. What is $x$? It turns out it doesn't matter: We only care
about the differences. So let us set $x = 0$.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_graphpath-labelled.png}
  \captionof{figure}{A path of two resistors.}
  \label{fig:graphpathlabelled}
\end{figure}

Let us try one more example:

\begin{figure}[H]
  \centering
  \includegraphics[width=0.5\linewidth]{fig/lecture1_graphtriangle.png}
  \captionof{figure}{A network with three resistors.}
  \label{fig:graphtriangle}
\end{figure}

How much flow will go directly from $s$ to $t$ and how much via $u$?

Well, we know what the net current flowing into and out of each vertex
must be, and we can use to set up some equations.
Let us say the voltage at $s$ is $x_s$, at $u$ is $x_u$ and at $t$ is $x_t$.
\begin{tight_itemize}
\item Net current at $s$:   $-1 = (x_s-x_t) + (x_s-x_u)$
\item  Net current at $u$:   $\phantom{-}0 = (x_u-x_s) + (x_u-x_t)$
\item Net current at  $t$:\,   $\phantom{-}1 = (x_t-x_s)+(x_t-x_u)$
\end{tight_itemize}
The following is a solution: $x_s = 0$, $x_u = \frac{1}{3}$, $x_t
= \frac{2}{3}$.
And as before, we can shift all the voltages by some constant $x$ and
get another solution $a = x+0$, $x_u = x+\frac{1}{3}$, $x_t
= x+\frac{2}{3}$. You might want to convince yourself that these are the only solutions.

\paragraph{Electrical flows in general graphs.}
Do we know enough to calculate the electrical flow in some other
network of resistors?
To answer this, let us think about the network
as a graph.
Consider a undirected graph $G = (V,E)$ with $\abs{V} = n$ vertices and
$\abs{E} = m$ edges, and let us assume $G$ is connected.
Let's associate a resistance
$\rr(e) > 0$ with every edge $e \in E$.

To keep track of the direction of the flow on each edge, it will be
useful to assign an arbitrary direction to every edge. So let's do
that, but remember that this is just a bookkeeping tool that helps
us track where flow is going.

A \emph{flow} in the graph is a vector $\ff: \R^E$.
%
The \emph{net flow} of $\ff$ at a vertex $u \in V$ is defined as
$\sum_{v \to u} \ff(v,u) - \sum_{u  \to v} \ff(u,v).$
% As a second accounting trick, we can define $\ff(v,u) \defeq
% -\ff(u,v)$,
% and then write the net flow as $\sum_{(u,v)\in E} \ff(u,v)$.

We say a flow routes the demands $\dd \in \R^V$ if the net flow at
every vertex $v$ is $\dd(v)$.

We can assign a voltage to every vertex $\xx \in R^V$.
Ohm's Law says that the electrical flow induced by these voltages will
be $\ff(u,v) = \frac{1}{\rr(u,v)} (\xx(u) - \xx(v))$.

Say we want to route unit of current from vertex $s \in V$ to vertex
$t \in V$.
As before, we can write an equation for every vertex saying that the
voltage differences must produce the desired net current:
\begin{tight_itemize}
\item Net current at $s$:
  $\phantom{----,,,}-1 = \sum_{(s,v)} \frac{1}{\rr(s,v)} (\xx(s) - \xx(v))$
\item Net current at $u \in V\setminus\setof{s,t}$:
  $\phantom{..} 0 = \sum_{(u,v)} \frac{1}{\rr(u,v)} (\xx(u) - \xx(v))$
\item Net current at  $t$:  $\phantom{------,} 1=\sum_{(t,v)} \frac{1}{\rr(t,v)} (\xx(t) - \xx(v))$
\end{tight_itemize}
This gives us $n$ constraints, exactly as many as we have voltage
variables.
However we have to be a little careful when trying to conclude that a
solution exists, yielding voltages $\xx$ that gives induce an
electrical flow routing the desired demand.

You will prove in the exercises (Week 1, Exercise 3) that a solution $\xx$ exists. The proof requires two
important observations: Firstly that the graph is connected, and
secondly that summed over all vertices, the net demand is zero,
i.e. as much flow is coming into the network as is leaving it.

\paragraph{The incidence matrix and the Laplacian matrix.}
To have a more compact notation for net flow constraints, we also
introduce the \emph{edge-vertex incidence matrix} of the graph,
$\BB \in \R^{V \times E}$.
\[
  \BB(v,e) =
  \begin{cases}
    1 & \text{ if } e = (u,v) \\
    -1 &\text{ if } e = (v,u) \\
    0 &\text{ o.w.}
  \end{cases}
\]
%
Now we can express the net flow constraint that $\ff$ routes $\dd$ by
\[
  \BB \ff = \dd
  .
\]
This is also called a conservation constraint. In our examples so far,
we have $\dd(s) = -1$, $\dd(t) = 1$ and $\dd(u) = 0$ for all $u\in V\setminus\setof{s,t}$.

If we let $\RR = \diag_{e \in E} \rr(e)$
then Ohm's law tells us that $\ff = \RR^{-1}\BB^{\trp} \xx$.
Putting these observations together, we have $\BB \RR^{-1}\BB^{\trp}
\xx= \dd$.
The voltages $\xx$ that induce $\ff$ must solve this system of linear
equations, and we can use that to compute both $\xx$ and $\ff$.
It is exactly the same linear
equation as the one we considered earlier.
We can show a that for a connected graph, a solution $\xx$
exists if and only if the
flow into the graph equals the net flow out, which we can express as
$\sum_{v} \dd(v) = 0$ or $\vecone^\trp \dd = 0$.
You will show this as part of Exercise 3.
This also implies that an electrical flow routing
$\dd$ exists if and only if the net
flow into the graph equals the net flow out, which we can express as
$\vecone^\trp \dd = 0$.

The matrix $\BB \RR^{-1}\BB^{\trp}$ is called the \emph{Laplacian} of
the graph and is usually denoted by $\LL$.

\paragraph{An optimization problem in disguise.}
So far, we have looked at electrical voltages and flows as arising
from a set of linear equations -- and it might not be apparent that
this has anything to do with optimization.
But transporting current through a resistor requires energy, which will be
dissipated as heat by the resistor (i.e. it will get hot!).
If we send a current of $f$ across a resistor with a potential drop of
$x$, then the amount of energy spent per unit of time by the resistor will be
$f \cdot x$. This is called Joule's Law.
Applying Ohm's law to a resistor with resistance $r$,
we can also express this energy per unit of time
as $f \cdot x = x^2 / r = r \cdot f ^2$.
Since we aren't bothering with units, we will even forget about time,
and refer to these quantities as ``energy'', even though a physicist
would call them ``power''.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.6\linewidth]{fig/lecture1_flow-vs-energy.png}
  \captionof{figure}{Energy has a function of flow in a resistor with
    resistance $r = 1$.}
  \label{fig:flow-vs-energy}
\end{figure}

Now, another interesting question would seem to be: If we want to find
a flow routing a certain demand $\dd$, how should have flow behave in
order to minimize the the electrical energy spent routing the flow?
We can phrase this as an optimization problem:
\begin{align*}
  \min_{\ff \in \R^E} & \sum_e \rr(e) \ff(e)^2
  \\
  \textrm{s.t. } & \BB \ff = \dd .
\end{align*}
We call this problem \emph{electrical energy-minimizing flow}.
It turns out, that the flow $\ff^*$ that minimizes the electrical
energy among all flows that satisfy $\BB \ff = \dd$ is precisely the
electrical flow.

\paragraph{A pair of problems.}
What about our voltages, can we also get them from some optimization
problem?
%
Well, we can work backwards from the fact that our voltages solve the
equation $\LL \xx = \dd$.
%
Consider the function $c(\xx) = \frac{1}{2} \xx^{\trp} \LL \xx -
\xx^{\trp}\dd$.
We should ask ourselves some questions about this function $c : \R^V
\to \R$. Is it continuous and continuously differentiable? The answer to this is
yes, and that is not hard to see.
Does the function have a minimum?
This is maybe not immediately clear, but the minimum does indeed exist.

When this is minimized, the derivative of $c(\xx)$ with respect to each coordinate of $\xx$
must be zero. This condition yields exactly the system of linear
equations $\LL \xx = \dd$.
You will confirm this in Exercise 4.

Based on our derivative condition for the optimum, we can also express the electrical voltages as the solution to
an optimization problem, namely
\begin{align*}
\min_{\xx \in \R^V} & \frac{1}{2} \xx^{\trp} \LL \xx -
\xx^{\trp}\dd
\end{align*}
As you are probably aware, having the derivative of each coordinate equal zero is not a
sufficient condition for being at the optimum of a
function\footnote{Consider the function in one variable $c(x) =
  x^3$.}.
It is also interesting to know whether \emph{all} solutions to $\LL
\xx = \dd$ are in fact minimizers of $c$. The answer is
yes, and will see some very general tools for proving statements like
this in Chapter~\ref{sec:cvopt}.
% Although, we have not yet proven the electrical flow minimizes
% electrical energy $\sum_e \rr(e) \ff(e)^2$, I have promised you that
% this is true.
% Let us denote the electrical flow and voltages by $\fftil$ and $\xxtil$,
% and recall that $\fftil = \RR^{-1} \BB ^\trp \xxtil$.
% Based on this, we can make an interesting observation:
% \[
%   \sum_e \rr(e) \ff(e)^2
%   =
%   \ff^{\trp} \RR \ff
%   =
  % \]
  % \frac{1}{2} \xxtil^{\trp} \LL \xxtil - \xxtil^{\trp}\dd
  % =

Altogether, we can see that routing electrical current through a
network of resistors leads to a \emph{pair} of optimization problems,
let's call them $\ff^*$ and $\xx^*$,
and that the solutions to the two problems are related, in our case
through the equation $\ff^* = \RR^{-1}\BB^{\trp} \xx^*$.
In Exercise 5, you will explore this relationship more.
\\

This turns out to be an instance of a much broader phenomenon, known
as Lagrangian duality, which allows us to learn a lot about many
optimization problems by studying two related pairs of problems.

\paragraph{Solving $\LL \xx = \dd$.}
Given a graph $G$ with resistances for the edges, and some net flow
vector $\dd$, how quickly can we compute $\xx$?
%
Broadly speaking, there are two very different families of algorithms
we could use to try to solve this problem.

Either, we could solve the linear equation using something like
\emph{Gaussian Elimination} to compute an exact solution.

Alternatively,
we could start with a guess at a solution, e.g. $\xx_0
= \veczero$, and then we could try to make a change to $\xx_0$ to reach
a new point $\xx_1$ with a lower value of $c(\xx) = \frac{1}{2} \xx^{\trp} \LL \xx -
\xx^{\trp}\dd$, i.e. $c(\xx_1) < c(\xx_0)$.
If we repeat a process like that for enough steps, say $t$, hopefully we
eventually reach $\xx_t$ with $c(\xx_t)$ close to $c(\xx^*)$, where
$\xx^*$ is a minimizer of $c(\xx)$ and hence $\LL \xx^* = \dd$.
Now, we also need to make sure that $c(\xx_t) \approx c(\xx^*)$
implies that $\LL \xx_t \approx \dd$ in some useful sense.

One of the most basic algorithms in this framework of ``guess and adjust''
is called \emph{Gradient Descent}, which we will study in two weeks.
The rough idea is the following: if we make a very small step from $\xx$ to
$\xx + \ddelta$, then a multivariate Taylor expansion suggests that
$c(\xx + \ddelta) - c(\xx) \approx \sum_{v \in V} \ddelta(v)
\frac{\partial c(\xx)}{\partial\xx(v)} $.

If we are dealing with smooth convex function,
this quantity is negative if we let $\ddelta(v) = -\epsilon \cdot \frac{\partial
  c(\xx)}{\partial\xx(v)}$ for some small enough $\epsilon$ so the
approximation holds well.
So we should be able to make progress by taking a small step in this
direction.
That's Gradient Descent!
The name comes from the vector of partial
derivatives, which is called the gradient.
% But first we need to become familiar with some basic terminology about
% optimization and some convex geometry.
% As we will see later in this course, understanding electrical problems
% from an optimization perspective is crucial to developing fast
% algorithms.

As we will see later in this course, understanding electrical problems
from an optimization perspective is crucial to developing fast
algorithms for computing electrical flows and voltages, but to do very
well, we also need to borrow some ideas from Gaussian Elimination.

What running times do different approaches get?
\begin{enumerate}
\item Using Gaussian Elimination, we can find $\xx$ s.t. $\LL \xx =
  \dd$ in $O(n^3)$ time and with asymptotically faster algorithms based on matrix
  multiplication, we can bring this down to roughly $O(n^{2.372})$.
\item Meanwhile Gradient Descent will get a running time of $O(n^3m)$
  or so -- at least this is a what a simple analysis suggests.
\item However, we can do much better: By combining ideas from both
  algorithms, and a bit more,
  we can get $\xx$ up to very high
  accuracy in time $O(m \log^c n)$ where $c$ is some small constant.
  % This type of result was orignally shown by Spielman and Teng in a
  % breakthrough result in 2004.
\end{enumerate}

\section{Convex Optimization}

Recall our plot in Figure~\ref{fig:flow-vs-energy} of the energy required to
route a flow $f$ across a resistor with resistance $r$, which was
$\energy(f) = r\cdot f^2$.
%
We see that the function has a special structure: the graph of the function sits
below the line joining any two points $(f, \energy(f))$ and $(g,\energy(g))$.
A function $\energy : \R \to \R$ that has this property is said to be convex.

Figure~\ref{fig:flow-vs-energy-convex} shows the energy as a function of flow, along with two
points $(f, \energy(f))$ and $(g,\energy(g))$. We see the function sits below the
line segment between these points.

\begin{figure}[H]
  \centering
  \includegraphics[width=0.6\linewidth]{fig/lecture1_flow-vs-energy-convex.png}
  \captionof{figure}{Energy has a function of flow in a resistor with
    resistance $r = 1$. The function is convex.}
  \label{fig:flow-vs-energy-convex}
\end{figure}
We can also interpret this condition as saying that for all $\theta \in [0,1]$
\[
  \energy(\theta f + (1-\theta)g) \leq \theta \energy(f) +
  (1-\theta) \energy(g)
  .
  \]
This immediately generalizes to functions $\energy:\R^m \to \R$.

A \emph{convex set} is a subset of $S \subseteq \R^m$ s.t.
if $\ff, \gg \in S$ then for all $\theta \in [0,1]$ we have
$\theta \ff + (1-\theta) \gg \in S$.

Figure~\ref{fig:convex_sets} shows some examples of sets that are and
aren't convex.

\begin{figure}[t]
\begin{centering}
                \includegraphics[trim = 0mm 0mm 0mm 0mm, height=60mm]{fig/lec1_convex_sets.pdf}
                 \caption{A depiction of convex and non-convex sets.  The sets $A$ and $B$ are convex since the straight line between any two points inside them is also in the set.  The set $C$ is not convex.}\label{fig:convex_sets}
                 \end{centering}
\end{figure}

Convex functions and convex sets are central to optimization,
because for most problems of minimization a convex function over a
convex set, we can develop fast algorithms \footnote{
 There are some convex optimization problems that are NP-hard.
  That said, polynomial time algorithms exist for almost any convex
  problem you can come up with.
  The most general polynomial time algorithm for convex optimization
  is probably the
  \href{https://en.wikipedia.org/wiki/Ellipsoid_method}{Ellipsoid Method}.
}.

So why convex functions and convex sets?
One important reason is that
for a convex function defined over a convex feasible set,
any local minimum is also a global minimum,
and this fact makes searching for an optimal solution
computationally easier.
In fact, this is closely related to why Gradient Descent works well
on many convex functions.

Notice that the set $\setof{ \ff: \BB \ff = \dd}$ is convex, i.e. the
set of all flows that route a fixed demand $\dd$ is convex.
It is also easy to verify that $\energy(\ff) = \sum_e \rr(e) \ff(e)^2$
is a convex function, and hence finding an electrical flow is an
instance of convex minmization:

\section{More Graph Optimization Problems}

\paragraph{Maximum flow.}
Again, let $G=(V,E)$ be an undirected, connected graph with $n$
vertices and $m$ edges.
Suppose we want to find a flow
$\ff \in \R^E$ that routes $\dd$, but instead of trying to minimize
electrical energy, we try to pick an $\ff$ that minimizes the largest
amount of flow on any edge, i.e. $\max_e \abs{\ff_e}$  -- which we
also denote by $\norm{\ff}_{\infty}$.
We can write this problem as
\begin{align*}
\min_{\ff \in \R^E } & \norm{\ff}_{\infty} \\
\textrm{s.t. } & \BB \ff = \dd
\end{align*}
This problem is known as the Minimum Congested Flow Problem\footnote{This
  version is called undirected, because the graph is undirected, and
  \emph{uncapacitated} because we are aiming for the same
  bound on the flow on all edges.}.
It is equivalent to the more famous Maximum Flow Problem.

The behavior of this kind of flow is very different than electrical
flow. Consider the question of whether a certain demand can be routed
$\norm{\ff}_{\infty} \leq 1$.
Imagine sending goods from a source $s$ to a destination $t$ using a
network of train lines that all have the
same capacity and asking whether the network is able to route the
goods at the rate you want: This boils down to whether routing exists
with $\norm{\ff}_{\infty} \leq 1$, if we set it up right.

We have a very fast, convex optimization-based algorithm for Minimum Congested Flow:
In $m \epsilon^{-1} \log^{O(1)} n$ time, we can find a flow $\fftil$
s.t. $\BB \fftil = \dd$ and $\norm{\fftil}_{\infty} \leq
(1+\epsilon)\norm{\ff^*}_{\infty}$, where $\ff^*$ is an optimal solution, i.e. an
actual minimum congestion flow routing $\dd$.

But what if we want $\epsilon$ to be very small, e.g. $1/m$? Then this running
time isn't so good anymore.
But, in this case, we can use another algorithm, that finds an optimal
flow $\ff^*$ \emph{exactly}, in time\footnote{And there's even a paper on arXiv.org that brings this further
  down to $m^{11/8} \log^{O(1)} n$.} $m^{10/7} \log^{O(1)} n $.

Just as the electrical flow problem had a dual voltage problem, so
maximum flow has a dual voltage problem, which is know as the
$s$-$t$ minimum cut problem.

\paragraph{Maximum flow, with directions and capacities.}
We can make the maximum flow problem harder by introducing directed
edges: To do so, we allow edges in both directions to exist between a
vertex to exist, and we require that that flow on a directed edge is
always non-negative. So now $G=(V,E)$ is a directed graph.
We can also make the problem harder by introducing capacities.
We define a capacity vector $\cc \in \R^E \geq \veczero$ and try to minimize $\norm{\CC^{-1} \ff}_{\infty}$, where $\CC =
\diag_{e \in E}\cc(e)$.
Then our problem becomes
\begin{align*}
\min_{\ff \in \R^E } & \norm{\CC^{-1} \ff}_{\infty} \\
  \textrm{s.t. } &  \BB \ff = \dd\\
                     &  \ff \geq \veczero.
\end{align*}
For this capacitated, directed maximum flow problem, our best
algorithms run in about $O( m \sqrt{n} )$ time\footnote{Provided the
  capacities are integers satisfying a condition like $\cc \leq n^{100} \vecone$.}, even if we are willing to
accept fairly low accuracy solution.
If the capacities are allowed to be exponentially large, the best
running time we can get is $O(m n)$.
For this problem, we do not yet know how to improve over classical
combinatorial algorithms using convex optimization.

\paragraph{Multi-commodity flow.}
We can make the even harder still, by simultaneously trying to route
to types of flow (imagine pipes with Coke and Pepsi).
Our problem now looks like
\begin{align*}
\min_{\ff_1, \ff_2 \in \R^E } & \norm{\CC^{-1} (\ff_1 + \ff_2)}_{\infty} \\
  \textrm{s.t. } &  \BB \ff_1 = \dd_1 \\
                              & \BB \ff_2 = \dd_2\\
                     &  \ff_1, \ff_2 \geq \veczero.
\end{align*}
Solving this problem to high accuracy is essentially as hard as
solving a general linear program! We should see later in the course
how to make this statement precise.

If we in the above problem additionally require that our flows must be
integer valued, i.e. $\ff_1, \ff_2 \in \N_{0}$, then the problem becomes NP-complete.

\paragraph{Random walks in a graph.}
Google famously uses\footnote{At least they did at some point.} the
PageRank problem to help decide how to rank their search results.
This problem essentially boils down to computing the \emph{stable
distribution} of a random walk on a graph.
Suppose $G=(V,E)$ is a directed graph where each edge outgoing edge
$(v,u)$, which we will define as going from $u$ to $v$, has a
transition probability $p_{(v,u)} > 0$ s.t. $\sum_{z \leftarrow u}
p_{(z,u)} = 1$.
We can take a step of a random walk on the vertex set by starting at some vertex
$u_0 = u$, and then randomly picking one of it the outgoing edges
$(v,u)$ with probability $p_{(v,u)}$  and move to
the chosen vertex $u_1 = v$.  Repeating this procedure, to take a step from
the next vertex $u_1$, gives us a
\emph{random walk} in the graph, a sequence of vertices $u_0, u_1, u_2
\ldots, u_k$.

We let $\PP \in \R^{V \times V}$ be the matrix of transition
probabilities given by
\[
  \PP_{vu} =
  \begin{cases}
    p_{(v,u)} & \text{ for } (u,v) \in E \\
    0 & \text{ o.w.}
  \end{cases}
\]

Any probability distribution over the vertices can be specified by a vector
$\pp \in \R^V$ where $\pp \geq
\veczero$ and $\sum_v \pp(v) = 1$.
We say that probability distribution $\ppi$ on the vertices is a \emph{stable distribution} of the random walk
if $\ppi = \PP \ppi$.
A strongly connected graph always has exactly one stable distribution.

How quickly can we compute the stable distribution of a general random
walk? Under some mild conditions on the stable
distribution\footnote{Roughly something like $\max_{v} 1/\ppi(v) \leq n^{100}$.}, we can find
a high accuracy approximation of $\ppi$ in time $O(m \log^cn )$ for
some constant $c$.

This problem does not easily fit in a framework of convex
optimization, but nonetheless, our fastest algorithms for it use ideas
from convex optimization.

\section*{Topics in this Course}
In this course, we will try to address the following questions.
\begin{enumerate}
\item What are the fundamental tools of fast convex optimization?
\item What are some problems we can solve quickly on graphs using optimization?
\item What can graphs teach us about convex optimization?
\item What algorithm design techniques are good for getting algorithms
  that quickly find a crude approximate solution? And what techniques
  are best when we need to get a highly accurate answer?
\item What is special about flow problems?
\end{enumerate}

% \pagebreak
% \section{Exercises}

% NEW
% \begin{enumerate}
% \item sub-level set of a convex function is a convex set (show).
% \item is it true that a function whose sub-level sets are all convex
%   is necessarily convex?
% \item \todo{exercise A}
%   \begin{itemize}
%   \item kernel of the Laplacian
%   \item show that solutions (voltage and flow) exist when
%     graph is connected and $\dd^\trp \vecone = 0$.
%   \item prove matrix symmetric, consider quadratic form,
%     show positive semi-definite.
%   \end{itemize}
% \item \todo{exercise B}
%     define gradient. do the voltage gradient computation
% \item \todo{exercise C: more duality}
% {\color{olive}
%   \\
% Recall $\ff^* = \RR^{-1}\BB^{\trp} \xx^*$ and $\BB \ff^* = \dd$,
% we can conclude
% $(\ff^* )^\trp R \ff^*  = (\ff^* )^\trp \BB^\trp \xx^* = \dd^\trp \xx^* = (\xx^*)^\trp \LL
% \xx^*$
% (Maybe don't directly point this out to students!)

% now we can see that
% \begin{align*}
% \max_{\xx \in \R^V} \xx^{\trp}\dd - \frac{1}{2} \xx^{\trp} \LL \xx
%   &=
% \min_{\ff \in \R^E} \frac{1}{2}\sum_e \rr(e) \ff(e)^2 \\
%   & \qquad \textrm{ s.t. }
%   \BB \ff = \dd
% .
% \end{align*}

% Maybe also point out to students that the stuff they have proved can
% easily be generalized a larger class of quadratic problems.
% }
% \item GRADED? gaussian elimination for Laplacians is closed ?
% \end{enumerate}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "agao21_script"
%%% TeX-engine: luatex
%%% End:


\documentclass[12pt]{amsart}
\usepackage[margin=0.5in]{geometry}
  % see geometry.pdf on how to lay out the page. There's lots.
\usepackage{bsymb}
\usepackage{../unitb}
\usepackage{calculational}
\usepackage{ulem}
\usepackage{hyperref}

\newcommand{\REQ}{\text{REQ}}

\begin{document}
  % \input{main9/machine_m0.tex}
\begin{machine}{m0}

  \newset{\REQ}

We need a variable to keep track of all the requests to mutate the
data structure.

  \[ \variable{ req,req_0 : \set [\REQ] } \]

... and new events:
% % \hide{
  \newevent{req}{req}
  \newevent{handle}{handle}
  % \newevent{req}{req\_b}
  % \newevent{handle}{handle\_b}

\[\param{req}{r : \REQ }\]
\with{sets}
\begin{align*}
  & \evguard{req}{grd0}{ \neg r \in req }  \\
  & \evbcmeq{req}{act0}{ req }{ req \bunion \{ r \} } \\
  & \evbcmeq{req}{act1}{ req_0 }{ req }
\end{align*}
\[\param{handle}{r : \REQ }\]
\begin{align*}
  \evguard{handle}{grd0}{ r \in req }  \\
  \evbcmeq{handle}{act0}{ req }{ req \setminus \{ r \} } \\
  \evbcmeq{handle}{act1}{ req_0 }{ req }
\end{align*}

% \section{Requirements}
%   \dummy{ R : \set[\REQ] }
\begin{align*}
  \constraint{co0}{ req_0' = req \1\lor (req_0' = req_0 \1\land req' = req) }
\end{align*}
\begin{align*}
  & \progress{prog0}
    { \neg req = \emptyset }
    { \neg req_0 \subseteq req }
 \refine{prog0}{ensure}{handle}{ using \ref{handle} }
  & \progress{prog1}
    { V = req}
    { req \subset V
      \1\lor req = \emptyset \1\lor \neg req \subseteq req_0}
 \refine{prog1}{ensure}{handle}{ using \ref{handle} }
\end{align*}
\begin{align*}
  \dummy{ R,V : \set [\REQ] } \\
  \cschedule{handle}{m0:sch0}{ \neg req = \emptyset } \\
  \initialization{m0:in0}{ req = \emptyset }
\end{align*}
\end{machine}

% \input{main9/machine_m1}
\begin{machine}{m1}
  \refines{m0}
\[ \indices{handle}{ b : \Bool } \]
\[ \dummy{ b : \Bool } \]
\[ \variable{ ch : \Bool } \]
\begin{align*}
  & \cschedule{handle}{m1:sch0}{ b = ch } \\
  & \witness{handle}{b}{b = ch} \\
  & \progress{m1:prog1}{ b = ch }{ b = ch }
\end{align*}
\replace{handle}{m1:sch0}{m1:prog1}
  \begin{liveness}{m1:prog1}
    \progstep{\true}{req = \emptyset \1\lor \neg req \subseteq req_0}
      {induction}{}{ \var{req}{down}{\emptyset} }
  \begin{flatstep}
    \progstep
      {V = req}
      {req \subset V
        \1\lor req = \emptyset \1\lor \neg req \subseteq req_0}
      {discharge}{}{}
      \begin{step}
        \trstep{handle}{ \index{b}{ \true }  }
          { req = V \1\land \neg req = \emptyset  }
        \safstep
          { V = req }
          { req \subset V \1\lor \neg req \subseteq req_0 }
          {}
      \end{step}
  \end{flatstep}
  \end{liveness}
\end{machine}

% % \input{main9/machine_m2}
% \begin{machine}{m2}
%   \refines{m0}

% We now partitial $req$ into requests for operation A ($reqA$) and
% requests for operation B ($reqB$).

% \[ \variable{ reqA, reqB : \set [\REQ] } \]
% \begin{align*}
%   \invariant{m1:inv0}{ reqA \bunion reqB = req } \\
%   \invariant{m1:inv1}{ reqA \binter reqB = req }
% \end{align*}
% And consequ
% \begin{align*}
%   \initialization{m1:in0}{ reqA = \emptyset \land reqB = \emptyset }
% \end{align*}
% \end{machine}

\end{document}


\title          {\TeX\ macros for proof boxes}
\author         {Paul Taylor}

\documentclass{article}
\usepackage{boxproof}
\usepackage{a4wide}
\usepackage{daymonthyear}

\def\meta#1{\mbox{$\langle\hbox{#1}\rangle$}}
\def\macrowitharg#1#2{{\tt\string#1\bra\meta{#2}\ket}}

{\escapechar-1 \xdef\bra{\string\{}\xdef\ket{\string\}}}

\def\intro#1{{#1}{\cal I}}
\def\elim#1{{#1}{\cal E}}

\showboxbreadth 999
\showboxdepth 999
\tracingoutput 1

\let\imp\to
\def\elim#1{{{#1}{\cal E}}}
\def\intro#1{{{#1}{\cal I}}}
\def\lt{<}
\def\eqdef{=}
\def\eps{\mathrel{\epsilon}}
\def\biimplies{\leftrightarrow}
\def\flt#1{\mathrel{{#1}^\flat}}
\def\setof#1{{\left\{{#1}\right\}}}
\let\implies\to
\def\KK{{\mathsf K}}
\let\squashmuskip\relax

%=======================================================================
\begin{document}
\maketitle

\section{Introduction}
The proof
\begin{proofbox}
  \(\"1"\:\alpha\leftrightarrow\psi(x,\top)\=\\
        \:\Some\beta.\psi(x,\beta)\=\mathsf{total}\\
    \[\exists\beta\kern-1em\:\psi(x,\beta)\\
      \(\:\beta\=\\
        \:\beta=\top\=(*)\\
        \:\psi(x,\!\top)\=\mathsf{subs}\\
        \:\alpha\=\elim\leftrightarrow(\ref{1})\\
      \*\:\alpha\=\\
        \:\psi(x,\!\top)\=\elim\leftrightarrow(\ref{1})\\
        \:\beta=\top\=\mathsf{func}\\
        \:\beta\=(*)\\
      \)\:\alpha=\beta\=\intro\leftrightarrow\\
        \:\psi(x,\alpha)\=\mathsf{subs}\\
    \]  \:\psi(x,\alpha)\=\elim\exists\\
    \*  \:\psi(x,\alpha)\=\\
      \(\:\alpha\=\\
        \:\alpha=\top\=(*)\\
        \:\phi(x,\!\top)\=\mathsf{subs}\\
      \*\:\psi(x,\!\top)\=\\
        \:\alpha=\top\=\mathsf{func}\\
        \:\alpha\=(*)\\
      \)\:\alpha\leftrightarrow\psi(x,\!\top)\=\intro\leftrightarrow\\
    \)  \:\psi(x,\alpha)\leftrightarrow(\alpha\leftrightarrow\psi(x,\!\top))
          \=\intro\leftrightarrow\\
\end{proofbox}
 is produced by
\begin{verbatim}
\begin{proofbox}
  \(\"1"\:\alpha\leftrightarrow\psi(x,\top)\\
        \:\Some\beta.\psi(x,\beta)                      \=\mathsf{total}\\
    \[\exists\beta\kern-1em\:\psi(x,\beta)\\
      \(\:\beta\\
        \:\beta=\top                                    \=(*)\\
        \:\psi(x,\!\top)                                \=\mathsf{subs}\\
        \:\alpha                         \=\elim\leftrightarrow(\ref{1})\\
      \*\:\alpha\\
        \:\psi(x,\!\top)                 \=\elim\leftrightarrow(\ref{1})\\
        \:\beta=\top                                     \=\mathsf{func}\\
        \:\beta\=(*)\\
      \)\:\alpha=\beta                           \=\intro\leftrightarrow\\
        \:\psi(x,\alpha)                                 \=\mathsf{subs}\\
    \]  \:\psi(x,\alpha)                                  \=\elim\exists\\
    \*  \:\psi(x,\alpha)\\
      \(\:\alpha\\
        \:\alpha=\top                                   \=(*)\\
        \:\phi(x,\!\top)                                \=\mathsf{subs}\\
      \*\:\psi(x,\!\top)\\
        \:\alpha=\top                                   \=\mathsf{func}\\
        \:\alpha                                        \=(*)\\
      \)\:\alpha\leftrightarrow\psi(x,\!\top)   \=\intro\leftrightarrow\\
    \)  \:\psi(x,\alpha)\leftrightarrow
          (\alpha\leftrightarrow\psi(x,\!\top)) \=\intro\leftrightarrow\\
\end{proofbox}
\end{verbatim}

  Syntax as follows:
  each line is of the form
  \begin{center}
     \meta{variables}
     \meta{name}
     \verb/\:/ \meta{formula}
     \verb/\=/ \meta{reason}
     \verb/\-/ \meta{use}
     \verb/\\/
  \end{center}
  where
  \begin{itemize}
  \item \meta{variables} is something like ``$x,y$'' ---
    it's for variables declared at the beginning of $\intro\forall$- and
    $\elim\exists$-boxes.
  \item \meta{name} is a command \verb/\label{fred}/ which
    defines \verb/fred/ to be the label text, which may be used anywhere
    as \verb/\ref{fred}/ --- see {\em The \LaTeX book.}
    Local labels are also available, using \macrowitharg\lbl{name}
    or \verb/\"/\meta{name}\verb/"/; these obey the scoping rules of
    the boxes.
    You may also refer to the previous line as \verb/\ref{-}/.
  \item \meta{formula} is the proposition being asserted.
  \item \meta{reason} is \verb/\intro\land(\ref{john},\ref{mary})/
         or \verb/\elim\forall(\ref{jim})/.
  \item \meta{use} is provided for linear logic, to record the step
         which uses this one. How this accords with theory I don't yet know.
  \end{itemize}
  Note that the parts are separated by \verb/\:/, \verb/\=/ and \verb/\\/;
  these correspond to
  \begin{center}
     {\bf let } \meta{name} = \meta{expression} : \meta{type}
  \end{center}
  in a declarative language.
  The \verb/\:/, \verb/\=/ and \verb/\-/ fields are optional and may occur
  in any order. If any of them is repeated the last is taken.
  If none of them is present the \meta{variables} field is also ignored.

  Proof {\em boxes\/} are ``wrapped up'' as follows:
  \begin{itemize}
  \item the whole proof in \verb/\begin{proofbox}/...\verb/\end{proofbox}/;
  \item single-column boxes ($\intro\forall$, $\intro\imp$, $\elim\exists$),
      in \verb/\[/...\verb/\]/.
  \item multiple-column boxes are of two kinds:
    \begin{itemize}
     \item separate ($\intro\land$) boxes: \verb/\(/...\verb/\*/...\verb/\)/.
     \item stuck together ($\elim\lor$) boxes:
        \verb/\(/...\verb/\+/...\verb/\)/.
    \end{itemize}
  \end{itemize}
  You may put more than two columns in \verb/\(/...\verb/\)/ and even mix
  the \verb/\+/ and \verb/\*/ separators.

  The whole proof is enclosed in \verb/\proofbox/...\verb/\endproofbox/
  or \verb/\begin{proofbox}/...\verb/\end{proofbox}/, but the \LaTeX\
  environment form {\em must not\/} be used for nested boxes.

  If the proof occurs in paragraph mode (ie in vertical or
  unrestricted horizontal mode) then it is set as a display, using the
  full width of the page. Otherwise it uses only the required width.

  A lot of the internals are potentially configurable, but there is not
  yet a user interface suitable for doing this. This will be provided
  in the next version.

%=======================================================================
\section{Redefinable macros}
 WARNING: most of these commands will be hidden and replaced with
 optional arguments to \verb/\proofbox/ in a future version.
 Do not rely on them.

 We provide three different ways of numbering the lines of the proof:
 \begin{itemize}
 \item \verb/\runningproofline/: a global running sequence (default),

 \item \verb/\nestedproofline/: a hierarchical system with dots,

 \item \verb/\nestedproofline/: a fully hierarchical system which
 also includes the column number
 (\verb/\proof@columns/) as a letter (ASCII quote plus number).

 \end{itemize}
 \verb/\theproofline/ is the default.

 The macro \verb/\proofboxmakelabel#1/ is used to print the line label.
 We only put it in the leftmost box.
 It is printed in small non-ranging Arabic numerals
 ({\the\scriptfont1 0123456789}).
 Right-justify it in \verb/\prooflinenowidth/ if it will fit,
 otherwise let it stick out on the right, {\em i.e.}~left-justify it.

 Kill the numbers altogether with \verb/\proofboxnonumbers/.

 How to make the left column of the proof box:
 use the variables field, a space if necessary and the line label.

 How to make the middle column of the proof box:
 left justify the formula field.

 How to make the right column of the proof box:
 use the reason and use fields.

 Make the four edges of a rectangular box and the separator
 between \verb/\+/ columns.

 Use dotted lines: \verb/\dottedproofbox/.

 Leave the boxes open at the bottom: \verb/\openproofbox/

%=======================================================================
\section{Miscellaneous logical notations}

These macros are now in my \texttt{logicsym.sty}

 Print the names of the introduction and elimination rules, for example:
  \begin{quote}
     \verb/\elim\forall/ $\elim\forall$ \qquad
     \verb/\intro\land/ $\intro\land$ \qquad
  \end{quote}
 Recall that in \TeX\ the logical connectives and quantifiers are called
  \begin{center}
    \verb/\lor/ $\lor$\quad
    \verb/\land/ $\land$\quad
    \verb/\lnot/ $\lnot$\quad
    \verb/\forall/ $\forall$\quad
    \verb/\exists/ $\exists$
  \end{center}
 The following provide macros for the \verb/\implies/ {\em relation\/}
 and for the binary {\em operation\/} which yields the abstract
 \verb/\implic/ation between formulae.
 The point is that \TeX\ spaces them and breaks the lines differently:
  \begin{center}
    \verb/A\implies B/ $A\implies B$\quad{\em versus\/}\quad
    \verb/A\implic B/ $A\implic B$
  \end{center}
 There are forward and reverse, single and Double versions.

 Handle the spacing after a variable (and optionally its type)
 bound by a quantifier symbol. For example
  \begin{quote}
    \verb/\All x:X. \phi(x)/\quad prints as\quad $\All x:X.\phi(x)$\quad
     instead of\quad $\forall x:X.\phi(x)$
  \end{quote}
 We provide some commonly used forms; \verb/\iota/ ($\iota$) is Russell's
 description operator and should really be inverted.
 There are several notations for substitution.
 After writing $a[x:=b]$ throughout my book I~thought I~might change to
 $[b/x]^*a$.
 This macro reads the source in the first form and prints in the second.
 If you use it you can, like me, defer the decision about
 which notation to use until the final stages, doing
 \begin{quote}
    \verb/\renewcommand{\Subst}{\plainsubstitution}/
 \end{quote}
 if you finally decide on making substitution act on the right.
 This is already an improvement on the literal text, because it
 automatically enlarges the brackets according to the text inside.
 \verb/\Subst/ itself is (following my book)
 defined in terms of the action of a context morphism (\verb/\CtxtMor/)
 on a term. Again you can do
 \begin{quote}
    \verb/\renewcommand{\CtxtMor}{\plaincontextmorphism}/
 \end{quote}
 for something simpler.
 This macro interprets its argument as a comma-separated list
 of items in the form $x:=b$, which it switches to $b/x$.
 The simple versions.

%=======================================================================
\section{Some very easy logic exercises}

The following examples are taken from Krysia Broda's
{\em Solutions to Problems~5\/} (KB-Logic-B1-90) and took me a little
under an hour to type in.

page 1: (a)

\begin{proofbox}
   \: P\land Q \\
   \: P         \= \elim\land\\
\end{proofbox}

\begin{proofbox}
   \lbl{1}\: P \\
   \[
      \: Q \\
      \: P \= (\ref{1}) \\
   \]
   \: Q\to P \= \intro\to \\
\end{proofbox}

page 2: (c)

\begin{proofbox}
   \lbl{2}\: P\\
   \[
      \lbl{3}\: Q\\
      \: P\land Q\= \intro\land(\ref{2},\ref{3}) \\
   \]
   \: Q\to(P\land Q) \= \intro\to \\
\end{proofbox}

(g)

\begin{proofbox}
   \lbl{6}\: P\to(Q\to R)\\
   \[
      \lbl{4}\: P\to Q\\
      \[
         \lbl{5}\: P\\
         \lbl{8}\: Q \= \elim\to(\ref{4},\ref{5})\\
         \lbl{7}\: Q\to R \= \elim\to(\ref{6},\ref{5})\\
         \: R \= \elim\to(\ref{7},\ref{8})\\
      \]
      \: P\to R \= \intro\to \\
   \]
   \: (P\to Q)\to(P\to R) \= \intro\to \\
\end{proofbox}

page 3: (h)

\begin{proofbox}
   \lbl{10}\: P\to(Q\to R)\\
   \[
      \lbl{9}\: P\land Q\\
      \lbl{11}\: P \= \elim\land1(\ref{9})\\
      \lbl{12}\: Q\to R \= \elim\to(\ref{10},\ref{11})\\
      \lbl{13}\: Q \=\elim\land2(\ref{9})\\
      \: R \=\elim\to(\ref{12},\ref{13})\\
   \]
      \: P\land Q\to R \= \intro\to \\
\end{proofbox}

(i)

\begin{proofbox}
   \lbl{16}\: P\land Q\to R \\
   \[
      \lbl{14}\: P \\
      \[
         \lbl{15}\: Q \\
         \lbl{17}\: P\land Q \= \intro\land(\ref{14},\ref{15})\\
         \: R \= \elim\to(\ref{16},\ref{17})
      \]
      \: Q\to R \= \intro\to \\
   \]
   \: P\to(Q\to R) \= \intro\to \\
\end{proofbox}

(j)

\begin{proofbox}
   \lbl{18}\: P\to Q\\
   \lbl{20}\: \lnot Q\\
   \[
      \lbl{19}\: P \\
      \lbl{21}\: Q \= \elim\to(\ref{18},\ref{19})\\
      \: \bot \= \elim\lnot(\ref{20},\ref{21})\\
   \]
   \: \lnot P \= \intro\lnot \\
\end{proofbox}

page 5: (k)

\begin{proofbox}
   \lbl{22}\: \lnot P\\
   \[
      \lbl{23}\: P\\
      \[
         \: \lnot Q\\
         \: \lnot P \land P \= \intro\land(\ref{22},\ref{23})\\
         \: \bot \= \elim\lnot(\ref{22},\ref{23})\\
      \]
      \: \lnot\lnot Q \= \intro\lnot\\
      \: Q \= \lnot\lnot\\
   \]
   \: P\to Q \= \intro\to \\
\end{proofbox}

page 8: (o)

\begin{proofbox}
   \lbl{24}\: P\to Q \\
   \[
      \lbl{26}\: \lnot Q \\
      \[
         \lbl{25}\: P \\
         \lbl{27}\: Q \= \elim\to(\ref{24},\ref{25})\\
         \: \bot \= \elim\lnot(\ref{26},\ref{27})\\
      \]
      \: \lnot P \= \intro\lnot\\
   \]
   \: \lnot Q\to \lnot P \= \intro\to \\
\end{proofbox}

(p) The $\lnot\lnot$ rule is unnecessary!

\begin{proofbox}
   \lbl{28}\: P\to Q \\
   \[
      \: \lnot\lnot P\\
      \lbl{29}\: P \= \lnot\lnot\\
      \[
         \lbl{31}\: \lnot Q \\
         \lbl{30}\: Q \= \elim\to(\ref{28},\ref{29})\\
         \: Q\land\lnot Q \= \intro\land(\ref{30},\ref{31})\\
         \: \bot \= \elim\lnot\\
      \]
      \: \lnot\lnot Q \= \intro\lnot\\
   \]
   \: \lnot\lnot P\to\lnot\lnot Q \= \intro\to \\
\end{proofbox}

%=======================================================================
\section{Some more exercises}

\begin{proofbox}
   \: P\lor\lnot P \\
   \(
      \: P \\
      \: Q\to P \=\hbox{by (e)}\\
      \: (P\to Q)\lor(Q\to P) \= \intro\lor \\
   \+
      \: \lnot P\\
      \: P\to Q \=\hbox{by (k)}\\
      \: (P\to Q)\lor(P\to Q) \= \intro\lor \\
   \)
   \: (P\to Q)\lor(P\to Q) \= \elim\lor \\
\end{proofbox}

\begin{proofbox}
   \lbl{35}\: P\lor Q \\
   \[
      \lbl{30}\: (\lnot P)\land(\lnot Q) \\
      \(
         \lbl{31} \: P \\
         \lbl{32} \: \lnot P \=\elim\land1(\ref{30})\\
         \: \bot \= \elim\lnot(\ref{32},\ref{31})\\
      \+
         \lbl{33} \: Q \\
         \lbl{34} \: \lnot Q \= \elim\land2(\ref{30})\\
         \: \bot \= \elim\lnot(\ref{33},\ref{34})\\
      \)
      \: \bot \= \elim\lor(\ref{35})\\
   \]
   \: \lnot((\lnot P)\land(\lnot Q)) \= \intro\lnot \\
\end{proofbox}

\begin{proofbox}
\lbl{51} x \: \forall x'.x'<x\to p(x')\\
   \(
       \lbl{52}\: x=a \\
       \(
          \: a=c \\
               \: b<x \=\hbox{subst}(\ref{50})\\
               \: p(b) \= \elim\forall(\ref{51})\\
               \: b\neq b \= \elim\land\\
               \: \bot \= \hbox{refl}\\
       \+
          \: c<a \\
               \: c<x \=\hbox{subst}(\ref{52})\\
               \: p(c) \= \elim\forall(\ref{51})\\
               \: c\neq c \= \elim\land\\
               \: \bot \\
       \)
               \: \bot \= \elim\lor(\ref{53})\\
   \*
       \(
           \[
               \: x=b \\
                  \: a<x \\
                  \: p(a) \= \elim\forall\\
                  \: a\neq a\\
                  \: \bot \\
           \] \: x\neq b \= \intro\lnot \\
       \*
           \[
              \: x=c \\
                  \: b<x \\
                  \: p(b) \= \elim\forall\\
                  \: b\neq b\\
                  \: \bot
           \] \: x\neq c \= \intro\lnot \\
       \)
   \)
   \: x\neq a \land x\neq b \land x\neq c \= \intro\land\\
   \: p(x) \= \hbox{def} \\
\end{proofbox}

\begin{proofbox}
   \: a<b \\
   \lbl{40}\= b<c \\
   \: a<c \lor (a=c \lor c<a) \\
   \(
       \lbl{43}\: a=c \lor c<a \\
       \: p(x)\equiv(x\neq a\land (x\neq b \land x\neq c))\=\hbox{def}\\
       \[
          \lbl{41} x \: \forall x'.x'<x\to p(x')\\
          \(
              \lbl{42}\: x=a \\
            \(
               \: a=c \\
               \: b<x \=\hbox{subst}(\ref{40})\\
               \: p(b) \= \elim\forall(\ref{41})\\
               \: b\neq b \= \elim\land\\
               \: \bot \= \hbox{refl}\\
            \+
               \: c<a \\
               \: c<x \=\hbox{subst}(\ref{42})\\
               \: p(c) \= \elim\forall(\ref{41})\\
               \: c\neq c \= \elim\land\\
               \: \bot \\
            \)
            \: \bot \= \elim\lor(\ref{43})\\
         \*
            \(
               \[
                  \: x=b \\
                  \: a<x \\
                  \: p(a) \= \elim\forall\\
                  \: a\neq a\\
                  \: \bot \\
               \]
               \: x\neq b \= \intro\lnot \\
            \*
               \[
                  \: x=c \\
                  \: b<x \\
                  \: p(b) \= \elim\forall\\
                  \: b\neq b\\
                  \: \bot \\
               \]
               \: x\neq c \= \intro\lnot \\
            \)
       \)
       \: x\neq a \land x\neq b \land x\neq c \= \intro\land\\
       \: p(x) \= \hbox{def}\\
    \]
    \: \forall x.(\forall x'.x'<x\to p(x'))\to p(x) \= \intro\forall\\
   \: p(a) \= \elim\forall\\
   \: a\neq a \= \elim\land\\
   \: \bot\\
   \: a<c \= \elim\bot \\
\+
   \: a<c \\
\)
\: a<c \= \elim\lor\\
\end{proofbox}

%=======================================================================
\section{Krysia Broda's dragons exercise}

\def\happy#1{{{\mathsf{happy}}(#1)}}
\def\child#1#2{{{\mathsf{child}}(#1,#2)}}
\def\fly#1{{{\mathsf{fly}}(#1)}}
\def\dragon#1{{{\mathsf{dragon}}(#1)}}
\def\green#1{{{\mathsf{green}}(#1)}}
\def\parent#1#2{{{\mathsf{parent}}(#1,#2)}}

\def\all{\forall}
\def\some{\exists}
\def\imp{\Rightarrow}
\def\impby{\Leftarrow}

\advance\proofboxmargin-3em

\begin{proofbox}
\lbl{happy}\:\all x.\happy x\impby[\all y.\child y x\imp\fly y]
        \land\dragon x\\
\lbl{fly}\:\all x.\green x \land \dragon x \imp \fly x\\
\lbl{green}\:\all x.[\some y.\parent y x \land \green y]\imp\green x\\
\lbl{dragon}\:\all z.\all x.\child x z \land \dragon z \imp \dragon x\\
\lbl{parent}\:\all x.\all y.\parent x y \impby \child y x\\
\[\all x_0\lbl{Dragon}\:\dragon{x_0}\\
        \lbl{Green}\:\green{x_0}\\
        \[\all y_0\lbl{child}\:\child {y_0}{x_0}\\
                \(      \lbl{Parent}\:\parent{x_0}{y_0}
                                \=\elim\all(\ref{parent})\\
                        \:\parent{x_0}{y_0} \land\green{x_0}
                                \=\intro\land(\ref{Green})\\
\lbl{greenish}  \:\some z.\parent z {y_0} \land \green z
                                \=\intro\some(z:=x_0)\\
                        \:\green{y_0}\=\elim\all(\ref{green},x:=y_0)\\
                \*
                        \:\dragon{y_0}\=\elim\all(\ref{dragon},x:=y_0,
                                z:=x_0,\ref{Dragon})\\
                \)\lbl{greendragon}\:\green{y_0}\land\dragon{y_0}
                                \=\intro\land\\
                \:\fly{y_0}\=\elim\all(\ref{fly},x:=y_0,\ref{greendragon})\\
        \]\lbl{taught}\:\all y.\child y{x_0}\imp\fly y\=\intro\all\\
        \:\happy x\=\elim\all(\ref{happy},\ref{taught},\ref{Dragon})\\
\]\:\all x.\happy x \impby \green x \impby \dragon x\=\intro\all\\
\end{proofbox}
\noindent
Where the previous deduction is a premise of a rule, the reference is omitted.
Where the substitution is of the same letter (possibly with a subscript) it
is omitted, except for the $\alpha$-conversion (change of bound variable name)
in line~\ref{greenish}.

%=======================================================================
\section{Proof boxes from my book}

$$\proofboxformulawidth=18em\relax
\begin{proofbox}
\[\forall x:\lbl{ind box hyp}\:\forall x'.x'\prec x\imp\phi(x')
   \=\mbox{induction hypothesis}\\
\:\vdots\\
\lbl{ind box pred}\:u\prec x\=\mbox{various terms $u$}\\
\:\phi(u)\=\elim\forall(\ref{ind box hyp},\ref{ind box pred})\\
\:\vdots\\
\:\phi(x)\=\mbox{the property}\\
\]\:\forall x.(\forall x'.x'\prec x\imp\phi(x'))\imp\phi(x)\=\intro\forall\\
\:\forall x.\phi(x)\=\mbox{$\prec$-induction for $\phi$}\\
\end{proofbox}$$
$$\begin{proofbox}
\lbl{ind eg 0}\:\phi(0)\=z\\
\lbl{ind eg s}\:\forall n.\phi(n)\imp\phi(n+1)\=s\\
\lbl{ind eg 1-}\:\phi(0)\imp\phi(1)\=\elim\forall(\ref{ind eg s})\\
\lbl{ind eg 1}\:\phi(1)\=\elim\imp(\ref{ind eg 1-},\ref{ind eg 0})\\
\lbl{ind eg 2-}\:\phi(1)\imp\phi(2)\=\elim\forall(\ref{ind eg s})\\
\lbl{ind eg 2}\:\phi(2)\=\elim\imp(\ref{ind eg 2-},\ref{ind eg 1})\\
\lbl{ind eg 3-}\:\phi(2)\imp\phi(3)\=\elim\forall(\ref{ind eg s})\\
\lbl{ind eg 3}\:\phi(3)\=\elim\imp(\ref{ind eg 3-},\ref{ind eg 2})\\
\end{proofbox}$$
$$\begin{proofbox}
\lbl{mrwf1}\:\forall x.(\forall x'.x'\lt x\imp\phi(x'))\imp\phi(x)
\=\mbox{hypothesis}\\
\lbl{mrwf4}\:\psi(y)\eqdef\forall x.(fx=y)\imp\phi(x)\=\mbox{definition}\\
\[\forall y:\lbl{mrwf2}\:\forall y'.y'\prec y\imp\psi(y') \\
   \[\forall x:\lbl{mrwf3}\:fx=y\\
      \[\forall x':\:x'\lt x\\
          \:fx'\prec y\=\hbox{monotonicity}\\
          \:\psi(fx')\=\elim\forall(\ref{mrwf2})\\
          \:\phi(x')\=\elim\forall(\hbox{def }\ref{mrwf4},\ref{mrwf3})\\
      \]
      \:\forall x'.x'\lt x\imp\phi(x')\=\intro\forall\\
      \:\phi(x)\=\elim\forall(\ref{mrwf1})\\
   \]
   \:\forall x.(fx=y)\imp\phi(x)\=\intro\forall\\
  \:\psi(y)\=\hbox{def}(\ref{mrwf4})\\
\]
\:\forall y.(\forall y'.y'\prec y\imp\psi(y'))\imp\psi(y)\=\intro\forall\\
\:\forall y.\psi(y)\=(Y,{\prec})\hbox{-induction}\\
\end{proofbox}$$
$$\begin{proofbox}
\lbl{qwf1}\:\forall y.[\forall y'.y'\eps y\imp\phi(y')]\imp\phi(y)\\
\[\lbl{qwf2}\forall x:\:\forall x'.x'\prec x\imp\phi(fx')\\
\[\forall y':\:y'\eps fx\\
\lbl{qwf3}\:\exists x'.x'\prec x\land y'=fx'
\=\hbox{surj on pred}\\
\[\exists x':\:x'\prec x\\
\:y'=fx'\\
\:\phi(fx')\=\elim\forall(\ref{qwf2})\\
\:\phi(y')\=\hbox{substitution}\\
\]\:\phi(y')\=\elim\exists(\ref{qwf3})\\
\]\:\forall y'.y'\eps fx\imp\phi(y')\=\intro\forall\\
\:\phi(fx)\=\elim\forall(\ref{qwf1},y:=fx)\\
\]\:\forall x.[\forall x'.x'\prec x\imp\phi(fx')]\imp\phi(fx)\=\intro\forall\\
\end{proofbox}$$
$$\begin{proofbox}
\lbl{trwf1}\:\forall x_2,x.[x_2\ll x \biimplies
x_2\prec x \lor \exists x_1. x_2\ll x_1\prec x]\\
\lbl{trwf2}\:\forall x.[\forall x'.x'\prec x\imp\phi(x')]\imp\phi(x)
\=\mbox{hypothesis}\\
\lbl{trwf3}\:\psi(x)\eqdef\forall x_2.x_2\ll x\imp\phi(x_2)
\=\mbox{definition}\\
\lbl{trwf4}\:\forall x.\psi(x)\imp\phi(x)
\=\intro\forall(\elim\forall(\ref{trwf2},\ref{trwf3}))\\
\(\:\phi(x)\\
\:\forall x_2.x_2\ll x\imp\phi(x_2)\\
\[\forall x_1:\:x_2\ll x_1\\
\:x_2\ll x\\
\:\phi(x_2)\\
\:\forall x_2.x_2\ll x\imp\phi(x_2)\\
\:\psi(x_1)\\
\]\:\forall x_1.x_1\prec x\imp\psi(x_1)\\
\+\lbl{trwf5}\forall x:\:\forall x_1.x_1\prec x\imp\psi(x_1)\\
\[\forall x_2:\:x_2\ll x\\
\lbl{trwf7}\:x_2\prec x_1\lor\exists x_1.x_2\ll x_1\prec x
\=\elim\forall(\ref{trwf1})\\
\(\:x_2\prec x_1\\
\:\psi(x_2)\=\elim\forall(\ref{trwf5})\\
\:\phi(x_2)\=\elim\forall(\ref{trwf4})\\
\+\exists x_1:\:x_2\ll x_1\prec x\\
\:\psi(x_1)\=\elim\forall(\ref{trwf5})\\
\:\phi(x_2)\=\elim\forall(\ref{trwf3},x:=x_1)\\
\)\:\phi(x_2)\=\elim{\exists{\lor}}(\ref{trwf7})\\
\]\:\forall x_2.x_2\ll x\imp\phi(x_2)\=\intro\forall\\
\:\psi(x)\=\hbox{def}(\ref{trwf3})\\
\)\:\forall x.[\forall x_1.x_1\prec x\imp\psi(x_1)]\biimplies\psi(x)
\=\intro{\forall{\biimplies}}\hfill\\
\end{proofbox}$$
$$\begin{proofbox}
\:\forall U.[\forall V.V\flt\prec U\imp\phi(V)]\biimplies\phi(U)\\
\:\phi(\emptyset)\\
\[\forall x,U:\:\phi(U)\\
\lbl{fltindxhyp}\:
        [\forall y.y\prec x\imp[\forall V.\phi(V)\imp\phi(V\cup\setof y)]]\\
\[\forall V_0:\:V_0\flt\prec U\\
\:\phi(V_0)\=\mbox{premise}\\
\:\emptyset\flt\prec\setof x\imp\phi(V_0)\\
\[\forall y,V_1:\:V_1\flt\prec x\\
\lbl{fltindV0V1}\:\phi(V_0\cup V_1)\\
\lbl{fltindy<x}\:y\prec x\\
\:\phi(V_0\cup V_1\cup\setof y)
        \=\elim\forall(\ref{fltindxhyp},\ref{fltindy<x},V:=V_0\cup V_1,
        \ref{fltindV0V1})\\
\]
\:\forall y,V_1.V_1\flt\prec x\land y\prec x\land\phi(V_0\cup V_1)
        \imp\phi(V_0\cup V_1\cup\setof y)\=\intro\forall\\
\:\forall V_1.V_1\flt\prec\setof x\imp\phi(V_0\cup V_1)
        \=\mbox{$\KK$-induction}\\
\]
\:\forall V_0,V_1.V_0\flt\prec U\land V_1\flt\prec\setof x
        \imp\phi(V_0\cup V_1)\=\intro\forall\\
\:\forall V.V\flt\prec U\cup\setof x\imp\phi(V)\=\mbox{Lemma}\\
\:\phi(U\cup\setof x)\=\mbox{premise}\\
\]
\:\forall x.[\forall y.y\prec x\imp[\forall V.\phi(V)\imp\phi(V\cup\setof y)]]
        \imp[\forall U.\phi(U)\imp\phi(U\cup\setof x)]\=\intro\forall\\
\:\forall x.[\forall U.\phi(U)\imp\phi(U\cup\setof x)]
        \=\mbox{$\prec$-induction}\\
\:\forall U.\phi(U)\=\mbox{$\KK$-induction}\\
\end{proofbox}$$
$$\proofboxnonumbers\proofboxformulawidth=4em
\begin{array}{cc}
\begin{proofbox}\[\qquad\:\phi\=\qquad\\\:\vdots\\\:\psi\\\]
\:\phi\imp\psi\=\intro\imp\end{proofbox}&
\begin{proofbox}\[\forall x':\\\:\vdots\=\qquad\\\:\phi(x')\\\]
\:\forall x.\phi(x)\=\intro\forall\end{proofbox}\\
\begin{proofbox}\qquad\:\phi\lor\psi\\
\(\qquad\:\phi\=\qquad\\\:\vdots\\\:\alpha\\
\+\:\psi\=\qquad\\\:\vdots\\\:\alpha\\\)
\:\alpha\=\elim\lor\end{proofbox}&
\begin{proofbox}\:\exists x.\phi(x)\\
\[\exists x':\:\phi(x')\=\qquad\\\:\vdots\\\:\alpha\\\]
\:\alpha\=\elim\exists\end{proofbox}
\end{array}$$
$$\proofboxformulawidth=30em \begin{proofbox}
\(\:\alpha\biimplies\psi(x,\top)\\
\:\exists\beta.\psi(x,\beta)\=\mathsf{total}\\
\[\exists\beta:\:\psi(x,\beta)\\
\(\:\beta\\
\:\beta=\top\\
\:\psi(x,\top)\=\mathsf{subs}\\
\:\alpha\=\elim\biimplies\\
\*\:\alpha\\
\:\psi(x,\top)\=\elim\biimplies\\
\:\beta=\top\=\mathsf{func}\\
\:\beta\\
\)\:\alpha=\beta\=\intro\biimplies\\
\:\psi(x,\alpha)\=\mathsf{subs}\\
\]\:\psi(x,\alpha)\=\elim\exists\\
\*
\:\psi(x,\alpha)\\
\(\:\alpha\\
\:\alpha=\top\\
\:\phi(x,\top)\=\mathsf{subs}\\
\*\:\psi(x,\top)\\
\:\alpha=\top\=\mathsf{func}\\
\:\alpha\\
\)
\:\alpha\biimplies\psi(x,\top)\=\intro\biimplies\\
\)\:\psi(x,\alpha)\biimplies(\alpha\biimplies\psi(x,\top))\=\intro\biimplies\\
\end{proofbox}$$
$$\proofboxformulawidth=7em \proofboxmargin=2em
\begin{proofbox}
\[\lbl{pnn0h}\:\alpha\\
\lbl{pnn0r}\:\alpha\imp\alpha\\
\:\alpha\=\qquad\quad(\ref{pnn0h})\\
\]\:\alpha\imp(\alpha\imp\alpha)\imp\alpha\=\intro\forall\\
\proofboxnonumbers\:(0)\\
\end{proofbox}
\hskip 0pt minus3em\relax
\begin{proofbox}
\[\lbl{pnn1h}\:\alpha\\
\lbl{pnn1r}\:\alpha\imp\alpha\\
\:\alpha\=\elim\imp(\ref{pnn1r},\ref{pnn1h})\\
\]\:\alpha\imp(\alpha\imp\alpha)\imp\alpha\=\intro\forall\\
\proofboxnonumbers\:(1)\\
\end{proofbox}
\hskip 0pt minus3em\relax
\begin{proofbox}
\[\lbl{pnn2h}\:\alpha\\
\lbl{pnn2r}\:\alpha\imp\alpha\\
\lbl{pnn21}\:\alpha\=\elim\imp(\ref{pnn2r},\ref{pnn2h})\\
\:\alpha\=\elim\imp(\ref{pnn2r},\ref{pnn21})\\
\]\:\alpha\imp(\alpha\imp\alpha)\imp\alpha\=\intro\forall\\
\proofboxnonumbers\:(2)\\
\end{proofbox}
%  \hskip 0pt minus3em\relax
%  \begin{proofbox}
%  \[\lbl{pnn3h}\:\alpha\\
%  \lbl{pnn3r}\:\alpha\imp\alpha\\
%  \lbl{pnn31}\:\alpha\=\elim\imp(\ref{pnn3r},\ref{pnn3h})\\
%  \lbl{pnn32}\:\alpha\=\elim\imp(\ref{pnn3r},\ref{pnn31})\\
%  \:\alpha\=\elim\imp(\ref{pnn1r},\ref{pnn32})\\
%  \]\:\alpha\imp(\alpha\imp\alpha)\imp\alpha\=\intro\forall\\
%  \proofboxnonumbers\:(3)\\
%\end{proofbox}%Overfull \hbox (17.42128pt too wide) detected
$$
$$\vbox{\proofboxformulawidth=12em \relax
\begin{proofbox}
\lbl{pthora1}\:\phi\land\psi\\
\[\lbl{pthora2}\forall\alpha:\:\phi\implies(\psi\implies\alpha)\\
\lbl{pthora3}\:\phi\=\elim{{\land}1}(\ref{pthora1})\\
\lbl{pthora4}\:\psi\implies\alpha
\=\elim\implies(\ref{pthora2},\ref{pthora3})\\
\lbl{pthora5}\:\psi\=\elim{{\land}2}(\ref{pthora1})\\
\lbl{pthora6}\:\alpha
\=\elim\implies(\ref{pthora4},\ref{pthora5})\\
\]\:\forall\alpha.(\phi\implies(\psi\implies\alpha))\implies\alpha
\=\intro{\forall{\implies}}\\
\end{proofbox}}
\kern6em
%
% Underfull \hbox (badness 10000) in paragraph
% Overfull \hbox (282.68869pt too wide) detected
%
\vbox{\baselineskip=\proofboxbaselineskip\relax
$p_1:\phi\times\psi$\\
$p_2:\phi\to(\psi\to\alpha)$\\
$p_3=\pi_1(p_1):\phi$\\
$p_4=p_2(p_3):\psi\to\alpha$\\
$p_5=\pi_2(p_1):\psi$\\
$p_6=p_4(p_5):\alpha$\\
$p_7=\lambda\alpha.\lambda p_2.p_6
%  =\lambda\alpha\lambda p_2.p_2(\pi_1(p_1))(\pi_2(p_1))
%  :\Pi\alpha.(\phi\to(\psi\to\alpha))\to\alpha
$\\}$$

%=======================================================================
\section{From my JSL paper}

%--------------------------------------------------------------------------
%                 save the proof box so that we can find out its height
\setbox9=\vbox{%
\proofboxbaselineskip=1.42em
%
%  parameters for version 1:
   \proofboxmargin=80pt
   \proofboxformulawidth=\hsize
   \advance\proofboxformulawidth-2\proofboxmargin
\iffalse
   \def\isX{:X}%
   \def\isPfX{:\finpower(X)}%
\else
   \def\isX{}\def\isPfX{}%
\fi
%
\begin{proofbox}%%*** WARNING: the names of the labels dont match their numbers
\lbl{fltwf1}\:\All U.[\All V.V\flt\prec U\Implies\phi(V)]
\biimplies\phi(U)\\
%\lbl{fltwf2}\:\All V.V\flt\prec\emptyset\Implies\bot
%   \=\mbox{def}({\flt\prec})\\
\lbl{fltwf3}\:\phi(\emptyset)
    \=\elim\forall(\ref{fltwf1},\mbox{def}({\flt\prec}))\\ %\ref{fltwf2})\\
\lbl{fltwf4}\:\psi(x)\eqdef\forall U.\phi(U)\Implies\phi(U\cup\setof x)\\
\[\lbl{fltwf5}\lbl{fltindxhyp}\forall x\isX
   \:\forall y.y\prec x\Implies\psi(y)\\
%\:[\forall y.y\prec x\Implies[\All V.\phi(V)\Implies\phi(V\cup\setof y)]]\\
\[\lbl{fltwf8}\forall V_0\isPfX
\lbl{fltwf10}\:\phi(V_0)\\
\lbl{fltwf9}\:\theta(W)\eqdef W\flt\prec\setof x\Implies\phi(V_0\cup W)\\
\lbl{fltwf11}\:\theta(\emptyset)
   \=\mbox{def}(\ref{fltwf9},\ref{fltwf10})\\
\[\forall W\isPfX
   \:\theta(W)\\
  \forall y\isX\lbl{fltwf13}\: W\flt\prec\setof x\Implies\phi(V_0\cup W)
   \=\mbox{def}(\ref{fltwf9})\\
\[\lbl{fltwf14}\:W\cup\setof y\flt\prec\setof x
   \equiv W\flt\prec\setof x\land y\prec x\\
%\lbl{fltwf15}\:W\flt\prec\setof x\=\ref{fltwf14},\hbox{def}({\flt\prec})\\
\lbl{fltwf16}\:\phi(V_0\cup W)\=\elim\Implies(\ref{fltwf13},\ref{fltwf14})\\
%\lbl{fltwf17}\:y\prec x\=\ref{fltwf14},\hbox{def}({\flt\prec})\\
\lbl{fltwf18}\:\psi(y)\equiv\forall V.\phi(V)\Implies\phi(V\cup\setof y)
\=\elim\forall(\ref{fltwf5},\ref{fltwf14}),\mbox{def}(\ref{fltwf4})\\
\:\phi(V_0\cup W\cup\setof y)\=\elim\forall(\ref{fltwf18},\ref{fltwf16})\\
\]
\:W\cup\setof y\flt\prec\setof x\Implies\phi(V_0\cup W\cup\setof y)
   \=\intro\Implies\\
\:\theta(W\cup\setof y)\=\mbox{def}(\ref{fltwf9})\\
\]
\:\theta(\emptyset)\land
     \forall y.\All W.\big[\theta(W)\Implies\theta(W\cup\setof y)\big]
     \=\intro\land(\ref{fltwf11},\intro\forall)\\
\:\forall W.\theta(W)\=\mbox{$\KK$-induction for }\theta\\
\:\All W.W\flt\prec\setof x\Implies\phi(V_0\cup W)\=\mbox{def}(\ref{fltwf9})\\
\]
%\:\All V_0.V_0\flt\prec U\Implies\All W.\theta(W)\=\intro\forall\\
\lbl{fltwf24}\:\squashmuskip
\All V_0.\phi(V_0)\Implies
      \big(\All W.W\flt\prec\setof x\Implies\phi(V_0\cup W)\big)
        \=\intro\forall\\    %\mbox{Currying, def(\ref{fltwf9})}\\
\[\lbl{fltwf6}\forall U\isPfX
   \:\phi(U)\\
%   \:V_0\flt\prec U \=\elim\forall(\ref{fltwf1},\ref{fltwf6},\ref{fltwf8})\\
\lbl{fltwfa}\:U=\emptyset\lor U\neq\emptyset
   \=\hbox{Proposition~\ref{free semil}}\\
\lbl{fltwfb}\:
   U=\emptyset\Implies\All W.W\flt\prec(U\cup\setof x)\Implies\phi(W)
   \=\elim\forall(\ref{fltwf24},V_0=\emptyset,\ref{fltwf3})\\
\[\lbl{fltwfe}\forall V_0\isPfX\:V_0\flt\prec U\\
\lbl{fltwff}\forall W\isPfX\:W\flt\prec\setof x\\
\label{fltwfg}\:\phi(V_0)
   \=\elim{\forall{\Impliedby}}(\ref{fltwf1},\ref{fltwf6},\ref{fltwfe})\\
\lbl{fltwfh}\:\phi(V_0\cup W)
   \=\elim\forall(\ref{fltwf24},\ref{fltwfg},\ref{fltwff})\\
\]\:\All V_0,W.V_0\flt\prec U\land W\flt\prec\setof x\Implies\phi(V_0\cup W)
  \=\intro\forall\\
\lbl{fltwfk}\:
   U\neq\emptyset\Implies\forall V.V\flt\prec(U\cup\setof x)\Implies\phi(V)
  \=\hbox{Lemma \ref{flt power distr}}\\
\lbl{fltwf27}\:\forall V.V\flt\prec(U\cup\setof x)\Implies\phi(V)
        \=\elim\lor(\ref{fltwfa},\ref{fltwfb},\ref{fltwfk})\\
\:\phi(U\cup\setof x)\=\elim\forall(\ref{fltwf1},\ref{fltwf27})\\
\]\lbl{fltwf29}
\:\forall U.\phi(U)\Implies\phi(U\cup\setof x)\equiv\psi(x)
        \=\intro\forall,\hbox{def}(\ref{fltwf4})\\
\]
\:\forall x.[\All y.y\prec x\Implies\psi(y)]\Implies\psi(x)\=\intro\forall\\
\lbl{fltwf32}\:\forall x.\psi(x)\=\mbox{$\prec$-induction for }\psi\\
\:\phi(\emptyset)\land\All x.\All U.
        \big[\phi(U)\Implies\phi(U\cup\setof x)\big]
        \=\intro\land(\ref{fltwf3},\mbox{def}(\ref{fltwf4},\ref{fltwf32}))\\
\:\All U.\phi(U)\=\mbox{$\KK$-induction for }\phi      \\%\;\hfill\qedsymbol\\
\end{proofbox}%
}% end of \setbox9=\vbox

\box9

%=======================================================================

\nocite{*}
\bibliographystyle{plain}
%\bibliography{boxuser}

\begin{thebibliography}{1}

\bibitem{VickersSJ:reasprog}
K.~Broda, S.~Eisenbach, H.~Khoshnevisan, and Steven Vickers.
\newblock {\em Reasoned Programming}.
\newblock International Series in Computer Science. Prentice Hall, 1994.

\bibitem{TaylorP:intso}
Paul Taylor.
\newblock Intuitionistic sets and ordinals.
\newblock {\em Journal of Symbolic Logic}, 61:705--744, 1996.

\bibitem{TaylorP:prafm}
Paul Taylor.
\newblock {\em Practical Foundations of Mathematics}.
\newblock Number~59 in Cambridge Studies in Advanced Mathematics. Cambridge
  University Press, 1999.

\end{thebibliography}

%=======================================================================

\end{document}


\chapter{Sandbox}
This is inline tikz graphics \tikz \draw (0pt,0pt) -- (20pt,6pt); and continue text after this.

\begin{tikzpicture}
  \draw (-1.5,0) -- (1.5,0);
  \draw (0,-1.5) -- (0,1.5);
  %\draw (-1,0) .. controls (-1,1) and (1,1) .. (1,0);
  \draw (0,0) circle [radius=10pt];
\end{tikzpicture}

\begin{tikzpicture}
  \draw (0,0) .. controls (1,1) and (2,1) .. (2,0);
\end{tikzpicture}


\documentclass[11pt]{article}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{amsmath}
\usepackage{hyperref}
\DeclareMathOperator*{\argmin}{arg\,min}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=violet,
}
\newcommand{\numpy}{{\tt numpy}}    % tt font for numpy

\topmargin -.5in
\textheight 9in
\oddsidemargin -.25in
\evensidemargin -.25in
\textwidth 7in

\begin{document}

% ========== Edit your name here
\title{SAILON Evaluation Metrics: Rigorously Defined}
\author{Sam Showalter}
\maketitle

\medskip
\section{Experimental Setup}

The SAILON evaluation system is abstractly defined as follows. \textit{A priori}, a number of trials $T$ is defined, where each trial $t \in T$ contains a single batch $b$ of $n$ samples. The total number of samples across the entire evaluation is denoted $Tn = N$.

Each sample $\mathbf{x}$ is a video of one of $k$ possible actions, or an action unseen during training time as denoted the $k + 1$ novel class. These samples are independently fed through a network and novelty detection system, considered jointly in this setup as $f_\theta(.)$. In this setting, the output for any sample $\mathbf{x_i}$ fed through $f_\theta(.)$ is a scalar label $y_{\theta i} \in \{0,1, ... k, k+1\}$ where $k + 1$ represents a novel input sample.

In addition, there is a third metric set \textit{a-priori} on the trials of the evaluation. At an unspecified trial $t_a \in T$, the presence of novel instances begins and will remain for all subsequent trials. Before trial $t_a$, no novel samples are present, though the model $f_\theta(.)$ can at any time predict $y_{\theta i} = k + 1$.

The evaluation is run sequentially in a batch-wise fashion over trials. Accordingly, $\mathbf{x}_{ti}$ represents the $i^{th}$ sample of the $t^{th}$ batch, and $y_{ti}$ represents the corresponding label. In addition, a baseline model $f_\phi(.)$, whose predictions are denoted $\hat{y}_{\phi ti} \in \{0, 1, ... , k\}$, cannot predict novelty and is often referred to as the \textbf{baseline model}. All evaluation metrics are determined based on the relationship of predicted labels $\hat{y}_{\theta ti}$ to the ground-truth labels $y_{ti}$ of the evaluation and to the baseline predictions $\hat{y}_{\phi ti}$.

Implicitly, if a given trial $t$ at least one sample in its batch $b_t$ was predicted to be novel ($\hat{y}_{\theta ti} = 1$), the trial itself is tagged with binary label 1 for its batch $\hat{l}_t = 1$, signifying the agent believes novelty has started. After the first trial possessing novelty occurs, all future trials will be given a novelty label = 1 regardless of the predictions in its batch. The ground truth labels across trials is denoted $l_t$

\section{Trial-based Evaluation Metrics: Detection Performance}%
\label{sec:trial_based_evaluation_metrics}

First, trial-based evaluation metrics are evaluated, with an optimal performance defined as predicting novelty for the first time in batch $b_{ta}$ belonging to trial $t_a \in T$. Failure modes in terms of false positives and false negatives are defined as follows, where $t$ represents the $t^{th}$ trial and $\hat{l}_t$ represents the binary label defining if any samples in batch $b_t$ were classified as novel.

\begin{equation}
    \begin{aligned}
      TP &= \hat{l}_t = l_t \\
      FP &= \hat{l}_t > l_t \\
      FN &= \hat{l}_t < l_t
    \end{aligned}
\end{equation}

These metrics are then utilized to generate slightly more sophisticated evaluation metrics across trials, including:

\begin{equation}
    \begin{aligned}
      \text{Correctly Detected Trials (CDT)} &= \sum^{T}_{t = 1} \mathbbm{1}( (FP_{t} = 1) \ \& \ (TP_t >= 1)) \\
      \text{Trial Failures} &= (t_a - \argmin_t(l_t = 1)) \\
      \text{False Positive Trial \%} &= \frac{FP}{T} \\
      \text{False Negative Trial \%} &= \frac{FN}{T}
    \end{aligned}
\end{equation}

In summary, the primary goal of this family of evaluation metrics is to minimize the number of false positives. This is far more important to PAR than the other metrics. Trial Failures exclusive looks at how many trials too early you declared novelty.

\section{Sample-based Evaluation Metrics: Reaction Performance}%
\label{sec:sample_based_evaluation_metrics_reaction_performance}

The next family of evaluation metrics concerns sample-based evaluation metrics. These metrics conducted over batches (unless specified otherwise) and then averaged. The evaluation metrics are defined below.

\begin{equation}
    \begin{aligned}
      FP &= (\hat{y}_{\theta ti} = k+1) \ \& \ (y_{ti} \neq k+1)) \\
      FN &= (\hat{y}_{\theta ti} \neq k+1) \ \& \ (y_{ti} = k+1))
    \end{aligned}
\end{equation}

These can be computed in any way, but in particular there are several metrics that compare these FP, FN values in the pre- and post-novelty regime. The formulas are the same for the baseline model, but $\theta$ is replaced by $\phi$.

\begin{equation}
    \begin{aligned}
      \text{Accuracy} &= \frac{1}{T} \sum^{T}_{t=1} \frac{1}{n} \sum^{n}_{i=1} \mathbbm{1}(\hat{y}_{\theta ti} = y_{ti}) \\
      \text{Baseline Accuracy} &= \frac{1}{T} \sum^{T}_{t=1} \frac{1}{n} \sum^{n}_{i=1} \mathbbm{1}(\hat{y}_{\phi ti} = y_{ti}) \\
    \end{aligned}
\end{equation}

Per trial asymptotic performance is defined as the ``measure in the change in accuracy over post-novelty rounds''. No notation or formulae of any kind accompany this statement. However, some additional metrics include:

\begin{equation}
    \begin{aligned}
      \text{Novelty Response Performance (NRP)} &= \frac{( \text{Accuracy} | t \geq t_a)}{( \text{Accuracy} | t < t_a)}  \\
      \text{Overall Performance Task Improvement (OPTI)} &= \frac{( \text{Accuracy} | t \geq t_a)}{( \text{Baseline Accuracy} | t \geq t_a)} \\
      \text{Asymptotic Performance Task Improvement (APTI)} &= \frac{( \text{Accuracy per Trial} | t \geq t_a)}{( \text{Baseline Accuracy per trial} | t \geq t_a)} \\
      \text{Area Under Activity Monitor Operating Characteristic (AUAMOC)} &= \text{AUROC of TP and FP}
    \end{aligned}
\end{equation}

In words, \textbf{NRP} is the ratio of the average batch accuracy in batches after novelty over the average batch accuracy in batches before novelty - we want this as high as possible. \textbf{OPTI} is the ratio of post-novelty accuracy in our model over post-novelty accuracy of the baseline. We also want this to be as high as possible. \textbf{APTI} is a more granular \textbf{OPTI} that makes use of per-trial differences. \textbf{AUAMOC} is a simple AUROC curve plotted for global TP and FP, \textbf{threshold based on anomaly score}

\section{Oustanding items}%
\label{sec:oustanding_items}

\begin{enumerate}
  \item Top-k classification with anomaly score not a probability - how do they intend to do this?
\end{enumerate}

\end{document}


\chapter{Background}
\label{ch:02-background}

We can make human language manageable to computers by using a sort of different methods and techniques. This is the bread and butter of any NLP researcher. Here, following the NLP community, we will focus only in the techniques derived from the machine learning field. In this chapter, we will present some abstract machine learning models follow by some applications in NLP.

\section{Machine Learning}

\textit{Machine learning} is the branch of computer science that deals with programs that can improve with experience (i. e., learn). Machine learning is divided into three main subareas: \textit{supervised learning}, \textit{reinforcement learning} and \textit{unsupervised learning}.

We will work with supervised learning only. In this setting, we assume that the process of interest is defined by an unknown function $g:X\rightarrow Y$. We try to approximate $g$ by changing the parameters of a function $f$ through an optimization process. The optimization is done by defining an \textit{error function} that evaluates how well $f$ approximates $g$ in the part of $g$ that we have access: the training data $D = \{(\vect{x}^{(1)}, \vect{y}^{(1)}), \dots ,(\vect{x}^{(N)}, \vect{y}^{(N)})\}$ (where $g(\vect{x}^{(i)})=\vect{y}^{(i)}$).

$f$ can be a function from any family of models. One family that is having a lot of success for language tasks is the \textit{neural network} family.

\section{Neural Networks}

A neural network is a non-linear function $f(\vect{x}; \vect{\theta})$. It is defined by a collection of parameters $\vect{\theta}$ and a collection of non-linear transformations. It is usual to represent $f$ as a compositions of functions:

\begin{align}
f(\vect{x}; \vect{\theta}) &= f^{(2)}(f^{(1)}(\vect{x}; \vect{W}_1, \vect{b}_1); \vect{W}_2, \vect{b}_2)\\
&= softmax(\vect{W}_2 (\sigma(\vect{W}_1\vect{x} + \vect{b}_1)) + \vect{b}_2)
\end{align}

The output of these intermediary functions are referred as \textit{layers}. So in the example above, $\vect{x}$ (the output of the identity function) is the \textit{input layer}, $f^{(1)}(\vect{x}; \vect{W}_1, \vect{b}_1)$ is the \textit{hidden layer} and $f^{(2)}(f^{(1)}(\vect{x}; \vect{W}_1, \vect{b}_1); \vect{W}_2, \vect{b}_2)$ is the \textit{output layer}. Since each layer is a vector, we normally speak about the \textit{dimension} of a layer. For historical reasons we also say that each entry on a layer is a \textit{node} or a \textit{neuron}.  Models with a large number of hidden layers are called \textit{deep models}, for this reason the name \textit{deep learning} is used.

\par A neural network is a function approximator: it can approximate any Borel measurable function from one finite dimensional space to another with any desired nonzero amount of error. This theoretical result is know as the \textit{universal approximation theorem} \cite{Cybenko}. Without entering in the theoretical concepts, it suffice to note that the family of Borel mensurable functions include all continuous functions on a closed and bounded subset of $\mathbb{R}^n$.

Different deep learning architectures are used in NLP: \textit{convolutional architectures} have a good performance in tasks were it is required to find a linguistic indicator regardless of its position (e.g., document classification, short-text categorization, sentiment classification, etc); high quality word embeddings can be achieved with models that are a kind of \textit{feedforward neural network} \cite{Mikolov23}. But for a variety of works in natural language we want to capture regularities and similarities in a text structure. That is way \textit{recurrent} and \textit{recursive} models have been widely used in the field. Here we are focused on generative models and since recurrent models have been producing very strong results for language modeling \cite{goldberg15}, we will concentrate on them.

\section{Recurrent Neural Network}
\label{sec:RNN}

\textit{Recurrent neural network} (RNN) is a family of neural network specialized in sequential data $\vect{x}^{(1)}, \dots, \vect{x}^{(\tau)}$. As a neural network, a RNN is a parametrized function that we use to approximate one hidden function from the data. What makes RNN unique is a recurrent definition of one of its hidden layer:

\begin{equation}
\vect{h}^{(t)} = g(\vect{h}^{(t-1)}, \vect{x}^{(t)}; \vect{\theta})
\end{equation}

$\vect{h}^{(t)}$ is called \textit{state}, \textit{hidden state}, or \textit{cell}.

\par This recurrent equation can be unfolded for a finite number of steps $\tau$. For example, when $\tau =3$:

\begin{align}
\vect{h}^{(3)}& = g(\vect{h}^{(2)}, \vect{x}^{(3)}; \vect{\theta})\\
 & = g(g(\vect{h}^{(1)}, \vect{x}^{(2)}; \vect{\theta}), \vect{x}^{(3)}; \vect{\theta})\\
 & = g(g(g(\vect{h}^{(0)}, \vect{x}^{(1)}; \vect{\theta}), \vect{x}^{(2)}; \vect{\theta}), \vect{x}^{(3)}; \vect{\theta})\\
\end{align}

Using a concrete example consider the following classification model define by the equations:

\begin{equation}
f(\vect{x}^{(t)}, \vect{h}^{(t-1)}; \vect{V}, \vect{W}, \vect{U}, \vect{c}, \vect{b}) = \vect{\hat{y}}^{(t)}
\end{equation}
 \vspace{0.2cm}
\begin{equation}
\vect{\hat{y}}^{(t)} = softmax(\vect{V} \vect{h}^{(t)} + \vect{c})
\end{equation}
\vspace{0.2cm}
 \begin{equation}
\vect{h}^{(t)} = g(\vect{h}^{(t-1)}, \vect{x}^{(t)}; \vect{W},\vect{U}, \vect{b})
\end{equation}
\vspace{0.2cm}
\begin{equation}
\vect{h}^{(t)} = \sigma(\vect{W} \vect{h}^{(t-1)} + \vect{U} \vect{x}^{(t)} + \vect{b})
\end{equation}

This kind of model can create an output $\vect{\hat{y}}^{(t)}$ at each time $t$, or the model can produce a single output $\vect{\hat{y}}$ after processing an entire input sequence. This choice depends on the learning problem that is being modeled.

With the model's prediction at hand, we can use a loss function (like cross entropy for the classification problem) and apply the back-propagation algorithm to optimize the model. These models look complex, but it quite straightforward to compute the gradients \cite[p.~374]{DeepLearningbook}.

Although this kind of deep learning model is very useful, it presents a severe flaw. When computing the gradients there is a lot of repeated matrix multiplication using the recurrent weight matrix (in the example above, the matrix $\vect{W}$). Depending on some configurations of this matrix \textit{the gradients may vanish or explode exponentially with respect to the number of time steps}.

Thus, handing long-term dependencies became a problem when using RNNs. Different solutions were proposed, the most effective results came from some modifications of this model. We will present the two most famous modifications: \textit{the gated recurrent unit} and \textit{the long short-term memory}.

\section{Gated Recurrent Unit}
\label{sec:GRU}

To capture long-term dependencies on a RNN  the authors of the paper \cite{ChungGCB14}  proposed a new architecture called \textit{gated recurrent unit} (GRU). This model was constructed to make each hidden state  $\vect{h}^{(t)}$ to adaptively capture dependencies of different time steps. It work as follows, at each step $t$ one candidate for hidden state is formed:

\begin{equation}
\vect{\widetilde{h}}^{(t)} = tahn(\vect{W} (\vect{h}^{(t-1)} \odot  \vect{r}^{(t)}) + \vect{U} \vect{x}^{(t)} + \vect{b})
\end{equation}

where $\vect{r}^{(t)}$ is a vector with values in $[0, 1]$ called a \textit{reset gate}, i.e.,  a vector that at each entry outputs the probability of reseting the  corresponding entry in the previous hidden state $\vect{h}^{(t-1)}$. Together with $\vect{r}^{(t)}$ we define an \textit{update gate}, $\vect{u}^{(t)}$. It is also a vector with values in $[0, 1]$. Intuitively we can say that this vector decides how much on each dimension we will use the candidate update. Both $\vect{r}^{(t)}$ and $\vect{u}^{(t)}$ are defined by $\vect{h}^{(t-1)}$ and $\vect{x}^{(t)}$; they also have specific parameters:

\begin{equation}
\vect{r}^{(t)} = \sigma(\vect{W}_{r} \vect{h}^{(t-1)} + \vect{U}_{r} \vect{x}^{(t)} + \vect{b}_{r})
\end{equation}

\begin{equation}
\vect{u}^{(t)} = \sigma(\vect{W}_{u} \vect{h}^{(t-1)} + \vect{U}_{u} \vect{x}^{(t)} + \vect{b}_{u})
\end{equation}

At the end the new hidden state $\vect{h}^{(t)}$ is defined by the recurrence:

\begin{equation}
\vect{h}^{(t)} = \vect{u}^{(t)} \odot \vect{\widetilde{h}}^{(t)} + (1 - \vect{u}^{(t)}) \odot \vect{h}^{(t-1)}
\end{equation}

Note that the new hidden state combines the candidate hidden state $\vect{\widetilde{h}}^{(t)}$ with the past hidden state $\vect{h}^{(t-1)}$ using both $\vect{r}^{(t)}$ and $\vect{u}^{(t)}$ to adaptively copy and forget information.

\section{Long Short-Term Memory}
\label{sec:LSTM}

\textit{Long short-term memory} (LSTM) is one of the most applied versions of the RNN family of models. Historically it was developed before the GRU model, but conceptually we can think in the LSTM as an expansion of the model presented in the last session. Because of notation differences they can look different. LSTM is also based on parametrized gates; in this case three: the \textit{forget gate}, $\vect{f}^{(t)}$, the \textit{input gate}, $\vect{i}^{(t)}$, and the \textit{output gate}, $\vect{o}^{(t)}$. The gates are defined only by $\vect{h}^{(t-1)}$ and $\vect{x}^{(t)}$ with specific parameters:

\begin{equation}
\vect{f}^{(t)} = \sigma(\vect{W}_{f} \vect{h}^{(t-1)} + \vect{U}_{f} \vect{x}^{(t)} + \vect{b}_{f})
\end{equation}

\begin{equation}
\vect{i}^{(t)} = \sigma(\vect{W}_{i} \vect{h}^{(t-1)} + \vect{U}_{i} \vect{x}^{(t)} + \vect{b}_{i})
\end{equation}

\begin{equation}
\vect{o}^{(t)} = \sigma(\vect{W}_{o} \vect{h}^{(t-1)} + \vect{U}_{o} \vect{x}^{(t)} + \vect{b}_{o})
\end{equation}

Intuitively $\vect{f}^{(t)}$ should control how much informative will be discarded, $\vect{i}^{(t)}$ controls how much information will be updated, and $\vect{o}^{(t)}$ controls how munch each component should be outputted. A candidate cell, $\tilde{\vect{c}}^{(t)}$ is formed:

\begin{equation}
\tilde{\vect{c}}^{(t)} = tahn(\vect{W} \vect{h}^{(t-1)} + \vect{U} \vect{x}^{(t)} + \vect{b})
\end{equation}

And a new cell $\vect{c}^{(t)}$ is formed by forgetting some information of the previous cell $\tilde{\vect{c}}^{(t-1)}$ and by adding new values from $\tilde{\vect{c}}^{(t)}$ (scaled by the input gate)

\begin{equation}
\vect{c}^{(t)} = \vect{f}^{(t)} \odot \vect{c}^{(t-1)} + \vect{i}^{(t)} \odot \tilde{\vect{c}}^{(t)}
\end{equation}

The new hidden state, $\vect{h}^{(t)}$, is formed by filtering $\vect{c}^{(t)}$:

\begin{equation}
\vect{h}^{(t)} = \vect{o}^{(t)} \odot tanh(\vect{c}^{(t)})
\end{equation}

Until now we have presented general deep learning theory, now we will focus on the specificities of these models applied to natural language problems.

\section{Language model}

We call \textit{language model} a probability distribution over sequences of tokens in a natural language.

\[
P(x_1,x_2,x_3,x_4) = p
\]

This model is used for different NLP tasks such as speech recognition, machine translation, text auto-completion, spell correction, question answering, summarization and many others.

The classical approach to a language model was to use the chain rule of probability and a Markovian assumption, i.e., for a specific $n$ we assume that:

\begin{equation}
P(x_1, \dots, x_T) = \prod_{t=1}^{T} P(x_t \vert x_1, \dots, x_{t-1}) = \prod_{t=1}^{T} P(x_{t} \vert x_{t - (n+1)}, \dots, x_{t-1})
\end{equation}

This gave raise to models based on $n$-gram statistics. The choice of $n$ yields different models; for example, the
\textit{unigram} language model ($n=1$) is defined as:
\begin{equation}
P_{uni}(x_1, x_2, x_3, x_4) = P(x_1)P(x_2)P(x_3)P(x_4)
\end{equation}

where $P(x_i) = count(x_i)$ and $count$ is a function that counts tokens occurrence in a corpus.\\

Similarly the \textit{bigram} language model ($n=2$) is defined as:
\begin{equation}
P_{bi}(x_1,x_2,x_3,x_4) = P(x_1)P(x_2\vert x_1)P(x_3\vert x_2)P(x_4\vert x_3)
\end{equation}
where
\begin{equation}
P(x_i\vert x_j) = \frac{count(x_i, x_j)}{count(x_j)}
\end{equation}

With these basic statistics we can already define useful language models. It is observed that higher $n$-grams yields better performance. This comes with a price though, higher $n$-grams requires great amounts of memory \cite{Heafield}. For this motive $n$-grams based language models that are trained on large corpora uses at most $5$-grams.

Since \cite{Mikolov11} the landscape has change, instead of using one approach that is specific for the language domain, we can use a general model for sequential data prediction, a RNN. The RNN's ability to deal with unrestricted size sequence input permits to abandon the $n$-gram model's restricted context assumption.

To understand the language model task as a machine learning task, some details should be clear.

First, the learning task is to estimate the probability distribution

\begin{equation}
\label{languagedistri}
P(x_{n} = \text{word}_{j^{*}} | x_{1}, \dots ,x_{n-1})
\end{equation}

for any $(n-1)$-sequence of words $x_{1}, \dots ,x_{n-1}$.

Second, the function $f$ being used to approximate \ref{languagedistri} is trained on a corpus in the following way: each word $x_t$ of the corpus will be used as input to $f$ (in the form of an one-hot vector), and the immediate subsequent word, say $x_{t+1}$, will be used as a target. The training is done by minimizing the cross entropy loss between the model's output and the probability distribution given by the target.

One example is in order. Suppose our corpus $\corpus$ is compose only by the lines below:

\begin{quote}
Yes, here we go again, give you more, nothing lesser\\
Back on the mic is the anti-depressor\\
Ad-Rock, the pressure, yes, we need this\\
The best is yet to come, and yes, believe this\\
\end{quote}

From this corpus we can construct a vocabulary list $\Vocab$ as follows: after preprocessing the text we create a list of the most frequent words (in this case we can take all words) with the size $V$ (here $V=27$). Hence, we can treat each word in the text either by an index referring to the word position on $\Vocab$ or as a one-hot vector that codifies this index (e.g., "the" would be identified with $0$, "yes" with $1$, "we" with $2$, and so on).

Then, the dataset is the collection of words
\[
D = \{(<eos>, \text{Yes}), (\text{Yes}, \text{here}), (\text{here}, \text{we}),\dots,(\text{believe}, \text{this}), (\text{this}, <eos>)\}
\]
where $<eos>$ is the "end of sentence" token (also a member of $\Vocab$).

A simple recurrent language model $f(\vect{x}^{(t)}, \vect{\theta})$ is defined by the following equations:

\begin{equation}
\vect{e}^{(t)} = \vect{E}\vect{x}^{(t)}
\end{equation}
\vspace{0.2cm}
\begin{equation}
\vect{h}^{(t)} = \sigma(\vect{W}\vect{h}^{(t-1)}+ \vect{U}\vect{e}^{(t)}+ \vect{b})
\end{equation}
\vspace{0.2cm}
\begin{equation}
f(\vect{x}^{(t)}, \vect{\theta}) = \vect{\hat{y}}^{(t)} = softmax(\vect{V}\vect{h}^{(t)} + \vect{c})
\end{equation}

where $\vect{E} \in \mathbb{R}^{d,V}$ is the matrix of word embeddings, $\vect{x}^{(t)} \in \mathbb{R}^{V}$ is one-hot word vector at time step $t$, $\vect{y}^{(t)} \in \mathbb{R}^{V}$ is the ground truth at time step $t$ (also a one-hot word vector) and $d$ is the size of the word embeddings.

For each word $x_t$ let $j_t$ be the index of the subsequent word, so at each time $t$ the point-wise loss is:

\begin{align}
\label{lossCE}
L^{(t)}(\vect{\theta}) &= CrossEntropy(\vect{y}^{(t)},\vect{\hat{y}}^{(t)})\\
    &= - \log({\vect{\hat{y}}^{(t)}}_{j_t})\\
        &= - \log P(x_{t+1} = \text{word}_{j_t}|x_{1}, \dots, x_{t})\\
        &= - \log P(x_{t+1}|x_{1}, \dots, x_{t})
\end{align}

The loss $L$ is the mean of all point-wise losses

\begin{equation}
L(\vect{\theta})=\frac{1}{T}\sum_{t=1}^{T}L^{(t)}(\vect{\theta})
\end{equation}

With the loss function defined, we apply some optimization algorithm like \textit{stochastic gradient descent} to choose the optimal parameters for the language model:

\begin{equation}
\vect{\theta}^{*} = \argmin_{\vect{\theta}} L(\vect{\theta})
\end{equation}

Because of the historical connections with information theory, the \textit{perplexity} ($PP$) metric is often used to evaluate a language model. This metric can be thought as the weighted average branching factor of a language.

Given $\corpus = x_1, x_2, \dots, x_T$, we define the perplexity of $\corpus$ ($PP(\corpus)$) as:

\begin{align}
PP(\corpus) &= P(x_1, x_2, \dots, x_T)^{-\frac{1}{T}}\\
      &= \sqrt[T]{\frac{1}{P(x_1, x_2, \dots, x_T)}}\\
      &= \sqrt[T]{\prod_{i=1}^{T}\frac{1}{P(x_i \vert x_1,\dots, x_{i-1})}}
\end{align}

Using \ref{lossCE} we can relate cross entropy loss and perplexity:

\begin{align}
        L(\vect{\theta}) &=\frac{1}{T} \sum_{t=1}^{T} L^{(t)}(\vect{\theta})\\
          &=\frac{1}{T} \sum_{t=1}^{T} - \log P(x_{t+1}|x_{1}, \dots, x_{t})\\
          &=\frac{1}{T} \sum_{t=1}^{T} \log ((\frac{1}{P(x_{t+1}|x_{1}, \dots, x_{t})})\\
          &= \log\left( \sqrt[T]{\prod_{i=1}^{T}\frac{1}{P(x_i \vert x_1,\dots, x_{i-1})}} \right)\\
          &= \log(PP(\corpus))
\end{align}

Hence,

\begin{equation}
2^{L(\vect{\theta})} = PP(\corpus)
\end{equation}

Thus, by finding the parameters that minimize the cross entropy error we are also minimizing the perplexity of the language model.

\section{Sequence-to-Sequence}
\label{sec:Seq2seq}

There is one powerful application of RNN based language model. The authors from \cite{Sustskever} used two RNNs to create an end-to-end translation model that is now know as \textit{the encoder-decoder} or \textit{the sequence-to-sequence} (seq2seq) architecture. This architecture is define as follows: let $\vect{x}^{(1)}, \dots, \vect{x}^{(n)}$ be a source sentence in the one-hot representation,  let $\vect{y}^{(1)}, \dots, \vect{y}^{(m)}$ be the target sentence also in the one-hot format. $f_{enc}$ (the \textit{encoder}) is a RNN with the sole purpose of creating a vector representation of input language's sequences. $f_{dec}$ (the \textit{decoder}) is a language model for the target language. These models are trained together mapping source sentences to target sentences.

For example, suppose the training pair $(\vect{x}^{(1)}, \dots, \vect{x}^{(n)}, \vect{y}^{(1)}, \dots, \vect{y}^{(m)})$ is ("Nas tardes de fazenda há muito azul demais", "In the farm’s afternoons there is too much blue"). Here we want the model to translate one Portuguese sentence to an English one. We first encode the Portuguese sentence in the vector $\vect{s}$, i.e.,

\begin{equation}
\vect{s} = f_{enc}(\vect{x}^{(n)}, \vect{h}^{(n-1)})
\end{equation}

Then using the control English sentence as a target, at each time $t$ we compute the cross entropy error between the decoder prediction $f_{dec}(\vect{y}^{(t)}, \vect{\tilde{h}}^{(t-1)})$ and the target $\vect{y}^{(t+1)}$ ($\vect{y}^{(0)}$ is the $<eos>$ token). The decoder uses the vector representation of the source sentence $\vect{s}$ as an initial hidden state (i.e., $\vect{\tilde{h}}^{(0)} = \vect{s}$). The goal of this model is to to approximate the probability distribution over the tokens from the target language given the sentence of the source language, i.e.,

\begin{equation}
\vect{\tilde{h}}^{(t)} = f_{dec}(\vect{y}^{(t)}, \vect{\tilde{h}}^{(t-1)})
\end{equation}

\begin{equation}
\label{decpred}
p(y_t | y_1, \dots, y_{t-1}, x_1, \dots, x_{n}) = softmax(\vect{W}_{s}  \vect{\tilde{h}}^{(t)} + \vect{b}_s)
\end{equation}

One limitation of this architecture is that the source sentence, in some cases, has more features than the decoder embedding $\vect{s}$ can properly summarize. To address that some attention mechanisms were introduced.

\section{Attention}
\label{sec:Attention}

The attention-based models are models built on top of the seq2seq architecture. The encoding part continues the same as before, but now at each time $t$ a context vector $\vect{c}^{(t)}$ is defined to capture relevant source-side information to help the prediction of the current target word $\vect{y}^{(t)}$. Once $\vect{c}^{(t)}$ is constructed the attention hidden state is defined as:

\begin{equation}
\vect{\tilde{h}}^{(t)} = tahn(\vect{W}_c[\vect{c}^{(t)};\vect{h}^{(t)}])
\end{equation}

With the attention hidden state defined, the model's prediction is the same as the one defined in \ref{decpred}.

The core of this technique is the definition of $\vect{c}^{(t)}$. There are different strategies available, here we will focus only on one: \textit{global attention}.

Let $\vect{a} \in \mathbb{R}^{m,n}$. We will use this matrix as an alignment matrix, i.e., at the end of the training $\vect{a}_{ts}$ should reflect the probability of the source representation $\vect{h}^{(s)}$ be relevant for the output $\hat{y}^{(t)}$. We define $\vect{a}_{ts}$ as

\begin{equation}
\vect{a}_{ts} = \frac{exp(score(\vect{\tilde{h}}_t,\vect{h}_s))}{\sum_j exp(score(\vect{\tilde{h}}_t,\vect{h}_j))}
\end{equation}

Where $score$ is a content-based function that can have different implementations:

\begin{equation}
score(\vect{\tilde{h}}_t,\vect{h}_s) = \begin{cases}
\vect{\tilde{h}}_t ^{\top}\vect{h}_s\\
\vect{\tilde{h}}_t ^{\top}\vect{W}_a \vect{h}_s\\
\vect{v}_a ^{\top}tahn(\vect{W}_a[\vect{\tilde{h}}_t;\vect{h}_s])\\
\end{cases}
\end{equation}

At the end, a global context vector $\vect{c}^{(t)}$ is computed as the weighted average, according to $\vect{a}_t$ over all source states:

\begin{equation}
\vect{c}^{(t)} = \sum_{s} \vect{a}_{ts}\vect{h}^{(s)}
\end{equation}