\documentclass[twoside]{article}

\usepackage{mystyle}
\usepackage{tikz}

\newcommand{\I}{\mathcal{I}}

\title{Introduction to Matroids}
\author{Travis Westura}
\date{\today}

\begin{document}
\maketitle
\thispagestyle{empty}

\section{Introduction}

In this course we've seen several examples of algorithms that are ``greedy.''
A greedy algorithm is an algorithm that makes a ``locally best'' decision.
For example, in Dijkstra's algorithm we use a priority queue to keep track of nodes on the friontier and pop them off using the priority of shortest path distance.
And in Kruskal's algorithm for finding minimum weight spanning trees, at each step we select the minimum weight edge that does not form a cycle.

We can generalize the idea of ``locally make a best decision'' by using matroids.
The word \emph{matroid} should make you think of the word \emph{matrix}, which you use in linear algebra and multivariable or vector calculus.
We'll begin by reviewing the concepts of linear independence, then we'll discuss analogous concepts in graph theory.
Then we will relate these concepts by generalizing them and giving the definition of a matroid.
And finally we will use matroids to give a proof of the correctness of Kruskal's algorithm.

\section{Independence in Linear Algebra}

As you take Linear Algebra and Multivariable Calculus you will gain lots of experience working with vectors and matrices.
In these notes we'll denote vectors using boldface letters at the end of the alphabet, such as $\bv{u}$ and $\bv{v}$, and matrices using capital letters at the beginning of the alphabet, such as $A$ and $B$.
We'll denote by $\bv{0}$ the vector of all $0$'s, and we'll also write vectors as columns and matrices as rectangles containing numbers:
\begin{equation*}
  \bv{u} = \mat{3\\-1\\3}, \quad \bv{v} = \mat{v_1\\v_2\\v_3}, \quad \bv{0} = \mat{0\\0\\0}, \quad A = \mat{1 & 2 & 5\\-2 & 3 & 0}, \quad B = \mat{b_{1,1} & b_{1, 2} & b_{1, 3}\\b_{2, 1} & b_{2, 2} & b_{2, 3}}.
\end{equation*}
We say that vectors are elements of a set called a Vector Space, and we also have the ability to add vectors and to multiply them by scalars.
In multivariable calculus the most common example of vector spaces are $\R^2$ and $\R^3$, where vectors consist of tuples of $2$ and $3$ numbers, respectively.
We can add vectors and multiply them by scalars as follows:
\begin{equation*}
  \mat{u_1\\u_2} + \mat{v_2\\v_2} = \mat{u_1 + v_1\\v_2 + v_2}, \quad a\mat{u_1\\u_2} = \mat{a u_1\\ a u_2}.
\end{equation*}
A linear combination of vectors is a sum of scalar multiples of vectors.
For example
\begin{equation*}
  2\mat{2\\-1} + 3\mat{-3\\4} = \mat{-5\\10}.
\end{equation*}
We could say that the $3$rd vector depends on the first two vectors, since it is a linear combination of them.

A set of vectors\footnote{
  We can also treat the columns of matrices as vectors and define a notion of linear independence on them, although we should be careful that a column may be repeated, whereas a vector in a set is not repeated.
}
$\{\bv{v}_1, \bv{v}_2, \ldots, \bv{v}_n\}$ is \emph{linearly independent} if the only scalars ${a_1, a_2, \ldots, a_n}$ that satisfy
\begin{equation*}
  a_1\bv{v}_1 + a_2\bv{v}_2 + \cdots + a_n\bv{v}_n = \bv{0}
\end{equation*}
are ${a_1 = a_2 = \cdots = a_n = 0}$.This means there is no way to write one of the vectors as a combination of the others, unless we make all the coefficients~$0$.
That is, none of the vectors in the set depend on the others.
For example, the standard basis vectors in $\R^3$ are linearly independent,
\begin{equation*}
  \left\{\mat{1\\0\\0}, \mat{0\\1\\0}, \mat{0\\0\\1}\right\},
\end{equation*}
as none of them can be written as a linear combination of the others.
The following set of vectors is linearly dependent:
\begin{equation*}
  \left\{\mat{1\\2}, \mat{3\\2}, \mat{5\\6}\right\}, \quad 2\mat{1\\2} + \mat{3\\2} = \mat{2 + 3\\4 + 2} = \mat{5\\6}.
\end{equation*}

There is a limit to the number of vectors that we can add to a set while maintaining independence.
For example, in~$\R^3$, and set of $4$~vectors is linearly dependent.
An independent set to which we can't add any more vectors is called a \emph{maximal} independent set, with the word maximal meaning we have included as many vectors as possible.\footnote{
  A maximal linearly independent set of vectors is called a \emph{basis}.
}

Further, if we have an independent set of vectors, we can remove vectors and still have an independent set.
For example, the subset of standard basis vectors,
\begin{equation*}
  \left\{\mat{1\\0\\0}, \mat{0\\1\\0}\right\},
\end{equation*}
is still independent in~$\R^3$.

\section{Independence in Graphs}

Now let's discuss independence in graphs.
Recall that a graph~${G = (V, E)}$ consists of a set~$V$ of vertices and a set~$E$ of edges.
We will consider only undirected graphs for now.

Recall that a \emph{tree} is a connected graph that does not have any cycles.
A \emph{forest} is a graph in which each connected component is a tree.
That is, a forest is essentially a collection of trees grouped together.
A \emph{spanning tree} of an undirected\footnote{ There is also a notion of spanning tree in an undirected graph, called an \emph{arborescence}.} graph~$G$ is a subgraph that is a tree and that includes all of the vertices of~$G$.
The ``span'' part of a spanning tree refers to the tree containing all of the vertices.
In the following diagram, from left to right we have: a tree that does not span the graph, a cycle, two spanning trees, and a forest consisting of two trees.
\begin{center}
  \begin{tikzpicture}[every node/.style={circle,thick,draw}]
    \node (A) at (0, 0) {A};
    \node (B) at (1.5, 0) {B};
    \node (C) at (0, -1.5) {C};
    \node (D) at (1.5, -1.5) {D};

    \path[-] (A) edge[draw=gray,thin] (B);
    \path[-] (A) edge[draw=black,very thick] (C);
    \path[-] (B) edge[draw=gray,thin] (D);
    \path[-] (C) edge[draw=black,very thick] (D);
    \path[-] (A) edge[draw=gray,thin] (D);
  \end{tikzpicture}
  \quad
  \begin{tikzpicture}[every node/.style={circle,thick,draw}]
    \node (A) at (0, 0) {A};
    \node (B) at (1.5, 0) {B};
    \node (C) at (0, -1.5) {C};
    \node (D) at (1.5, -1.5) {D};

    \path[-] (A) edge[draw=gray,thin] (B);
    \path[-] (A) edge[draw=black,very thick] (C);
    \path[-] (B) edge[draw=gray,thin] (D);
    \path[-] (C) edge[draw=black,very thick] (D);
    \path[-] (A) edge[draw=black,very thick] (D);
  \end{tikzpicture}
  \quad
  \begin{tikzpicture}[every node/.style={circle,thick,draw}]
    \node (A) at (0, 0) {A};
    \node (B) at (1.5, 0) {B};
    \node (C) at (0, -1.5) {C};
    \node (D) at (1.5, -1.5) {D};

    \path[-] (A) edge[draw=gray,thin] (B);
    \path[-] (A) edge[draw=black,very thick] (C);
    \path[-] (B) edge[draw=black,very thick] (D);
    \path[-] (C) edge[draw=black,very thick] (D);
    \path[-] (A) edge[draw=gray,thin] (D);
  \end{tikzpicture}
  \quad
  \begin{tikzpicture}[every node/.style={circle,thick,draw}]
    \node (A) at (0, 0) {A};
    \node (B) at (1.5, 0) {B};
    \node (C) at (0, -1.5) {C};
    \node (D) at (1.5, -1.5) {D};

    \path[-] (A) edge[draw=gray,thin] (B);
    \path[-] (A) edge[draw=black,very thick] (C);
    \path[-] (B) edge[draw=black,very thick] (D);
    \path[-] (C) edge[draw=gray,thin] (D);
    \path[-] (A) edge[draw=black,very thick] (D);
  \end{tikzpicture}
  \quad
  \begin{tikzpicture}[every node/.style={circle,thick,draw}]
    \node (A) at (0, 0) {A};
    \node (B) at (1.5, 0) {B};
    \node (C) at (0, -1.5) {C};
    \node (D) at (1.5, -1.5) {D};
    \node (E) at (3, 0) {E};
    \node (F) at (3, -1.5) {F};

    \path[-] (A) edge[draw=gray,thin] (B);
    \path[-] (A) edge[draw=black,very thick] (C);
    \path[-] (B) edge[draw=gray,thin] (D);
    \path[-] (C) edge[draw=gray,thin] (D);
    \path[-] (A) edge[draw=black,very thick] (D);
    \path[-] (B) edge[draw=black,very thick] (E);
    \path[-] (E) edge[draw=black,very thick] (F);
    \path[-] (D) edge[draw=gray,thin] (F);
  \end{tikzpicture}
\end{center}
The independent sets of a graph are the forests, that is, the subgraphs that do not include cycles.
The dependent sets are the subgraphs that include cycles.
To see the analogy to linear algebra, here note that if we have a forest that is not a spanning tree, then there is still an edge we can add to it to obtain a larger subggraph without creating a cycle.
A spanning tree gives us a notion of maximal independent set, meaning that we have included as many edges as we possibly can.

And given a forest, removing edges does not create a cycle.
Thus we have the same property that removing elements from an independent set preserves independence.

\section{Definition of Matroids}

Now that we have some intuition about the concept of ``independence,'' let's give a definition of matroids.\footnote{
  We could define matroids in many ways.
  Matroids are referred to as \emph{cryptomorphic}, which means they have many equivalent but ostensibly unrelated definitions.}
Many mathematical definitions involve placing a set inside of parentheses with other sets or functions that further describe that set.
For example, a graph is given by two sets ${G = (V, E)}$.
A vector space is given by ${(V, k, +, \cdot)}$, where $V$ is a set of vectors, $k$ is a field of scalars, $+$ is an addition of vectors, and $\cdot$ is a multiplication between a scalar and a vector.
A familiar example is ${(\R^3, \R, +, \cdot)}$.
If you have taken CS~$2800$ or other more advanced courses, you will have seen groups, where are often described by writing ${(G, +)}$, where $G$ is a set and $+$ is a binary operation on that set.
Matroids have a similar definition, where we have a set that we write in parentheses together with some other information describing that set.
In this definition the additional information is a collection of subsets.

\begin{defn}[Matroid]
  A \emph{matroid}~$M$ is a pair~$(E, \I)$ consisting of a finite set~$E$ and a collection~$\I$ of subsets of~$E$, called the \emph{independent sets}, satisfying the following properties:
  \begin{enumerate}
    \item The collection is nonempty.
      That is, ${\I \ne \varnothing}$.
    \item All subsets of an independent set are independent.
      That is, if ${A \in \I}$ and ${B \subseteq A}$, then ${B \in \I}$.
    \item If $A$ and $B$ are independent sets and $A$ is larger than $B$, then we can take an element that is in $A$ but not in $B$ and add it to $B$ to construct an independent set.
    That is, if ${A, B \in \I}$ with ${|A| > |B|}$, then there exists ${x \in A \setminus B}$ such that ${B \cup \{x\} \in \I}$.
    This property is knows as the \emph{independent set exchange property}---we can ``exchange'' an element from one set to another.
  \end{enumerate}
  A subset of~$E$ that is not independent is called \emph{dependent}.
\end{defn}

Let's see how this definition fits in with our first two examples.
Recalling that a vector space has a set of vectors~$V$, we form a matroid~$(V, \I)$, where $\I$ is the collection of independent sets of vectors of~$V$.
Recall what we noted previously: given a set of independent vectors, removing vectors still results in an independent set.
And if there are two independent sets of vectors ${A = \{\bv{a}_1, \ldots, \bv{a}_n\}}$ and ${B = \{\bv{b}_1, \ldots, \bv{b}_m\}}$ with ${m < n}$, then clearly $B$ is not a maximal independent set, so we can add more vectors to it while maintaining independence.\footnote{
  Being formal about explicitly satisfying the thrid property requires slightly more linear algebra than I want to use right now, so I'll leave this verification as an exercise for those interested.
}

Next, given an undirected graph ${G = (V, E)}$, we form a matroid~$(E, \I)$ with the ground set given by the graph's edges and the independent sets given by the graph's forests.
Let's verify each part of the definition.
\begin{enumerate}
  \item The empty collection of edges forms a forest vacuously.
  \item If a graph does not form a cycle, then removing edges cannot yield a cycle.
  \item Consider two forests $A$ and $B$ where $A$ has more edges than $B$.
    Then there is some edge ${e \in A}$ such that ${B \cup \{e\}}$ does not have a cycle.
    The number of connected components in a graph ${(V, F)}$ is given by ${|V| - |F|}$.
    Note that individual vertices form their own component.
    If $A$ and $B$ are two forests with ${|A| > |B|}$, then there are more connected components in ${(V, B)}$ than in ${(V, A)}$.
    There there is some edge~$e$ in~$A$ that connects two of the components in~$B$.
    Thus ${B \cup \{e\}}$ forms a forest with one more edge than~$B$.
\end{enumerate}

\section{Kruskal's Algorithm and Matroid Optimization}

Given a weighted undirected graph, we construct a spanning tree by beginning by starting with the empty set and repeatedly adding the minimum weight edge that does not form a cycle.
That is, at each step, add the minimum weight edge that still produces an independent set.
The algorithm terminates when we have a spanning tree, or in the language of matroids, when we have a maximal independent set.

In the following example we start with the empty set of edges.
The minimum weight edge is between~$A$ and~$C$, the we first choose that edge.
Next we select the edge between~$A$ and~$D$.
We then can't select the edge between~$C$ and~$D$, as that edge would form a cycle.
Thus we select the edge between~$B$ and~$D$.
At this point there are no other edges we can add without creating a cycle, so we have a spanning tree and terminate out algorithm here.
\begin{center}
  \begin{tikzpicture}
    \node[shape=circle,draw=black,thick] (A) at (0, 0) {A};
    \node[shape=circle,draw=black,thick] (B) at (1.5, 0) {B};
    \node[shape=circle,draw=black,thick] (C) at (0, -1.5) {C};
    \node[shape=circle,draw=black,thick] (D) at (1.5, -1.5) {D};

    \path[-] (A) edge[draw=gray,thin] node[above] {5} (B);
    \path[-] (A) edge[draw=gray,thin] node[left] {1} (C);
    \path[-] (B) edge[draw=gray,thin] node[right] {4} (D);
    \path[-] (C) edge[draw=gray,thin] node[below] {3} (D);
    \path[-] (A) edge[draw=gray,thin] node[above right] {2} (D);
  \end{tikzpicture}
  \quad
  \begin{tikzpicture}
    \node[shape=circle,draw=black,thick] (A) at (0, 0) {A};
    \node[shape=circle,draw=black,thick] (B) at (1.5, 0) {B};
    \node[shape=circle,draw=black,thick] (C) at (0, -1.5) {C};
    \node[shape=circle,draw=black,thick] (D) at (1.5, -1.5) {D};

    \path[-] (A) edge[draw=gray,thin] node[above] {5} (B);
    \path[-] (A) edge[draw=black,very thick] node[left] {1} (C);
    \path[-] (B) edge[draw=gray,thin] node[right] {4} (D);
    \path[-] (C) edge[draw=gray,thin] node[below] {3} (D);
    \path[-] (A) edge[draw=gray,thin] node[above right] {2} (D);
  \end{tikzpicture}
  \quad
  \begin{tikzpicture}
    \node[shape=circle,draw=black,thick] (A) at (0, 0) {A};
    \node[shape=circle,draw=black,thick] (B) at (1.5, 0) {B};
    \node[shape=circle,draw=black,thick] (C) at (0, -1.5) {C};
    \node[shape=circle,draw=black,thick] (D) at (1.5, -1.5) {D};

    \path[-] (A) edge[draw=gray,thin] node[above] {5} (B);
    \path[-] (A) edge[draw=black,very thick] node[left] {1} (C);
    \path[-] (B) edge[draw=gray,thin] node[right] {4} (D);
    \path[-] (C) edge[draw=gray,thin] node[below] {3} (D);
    \path[-] (A) edge[draw=black,very thick] node[above right] {2} (D);
  \end{tikzpicture}
  \quad
  \begin{tikzpicture}
    \node[shape=circle,draw=black,thick] (A) at (0, 0) {A};
    \node[shape=circle,draw=black,thick] (B) at (1.5, 0) {B};
    \node[shape=circle,draw=black,thick] (C) at (0, -1.5) {C};
    \node[shape=circle,draw=black,thick] (D) at (1.5, -1.5) {D};

    \path[-] (A) edge[draw=gray,thin] node[above] {5} (B);
    \path[-] (A) edge[draw=black,very thick] node[left] {1} (C);
    \path[-] (B) edge[draw=black,very thick] node[right] {4} (D);
    \path[-] (C) edge[draw=gray,thin] node[below] {3} (D);
    \path[-] (A) edge[draw=black,very thick] node[above right] {2} (D);
  \end{tikzpicture}
\end{center}

We are interested in proving the correctness of this abstract algorithm.
For now we don't consider the implementation details.\footnote{
  For the details of an efficient implementation, look up the Union-Find data structure.
}
Kruskal's spanning tree algorithm is actually a case of the more general greedy algorithm that is used for weighted matroids.
A \emph{weighted matroid} is a matroid in which every element of the ground set has a nonnegative weight.
That is, a weighted matroid~${(E, \I)}$ has a weight~${w_e \ge 0}$ for each edge~${e \in E}$.
Finding a minimum weight spanning tree then becomes the problem of finding a minmum weight maximal independent set.\footnote{
  To use more terminalogy from linear algebra, a maximal independent set is called a \emph{base}.
  All maximal independent sets contain the same number of elements, and this number is called the \emph{rank} of the matroid.
}

In this general case the greedy algorithm works the same way.
Begin with the empty set.
Repeated select the minimum weight element of~$E$ that maintains independence.
Terminate when the only remaining elements of~$E$ would form a dependent set if added.

\begin{thm}[]
  Let ${(E, \I)}$ be a matroid with weights ${w_e \ge 0}$ for each~${e \in E}$.
  Then the greedy algorithm produces a maximal independent set of minimum weight.
\end{thm}
\begin{proof}
  Suppose that the greedy algorithm selects elements ${[e_1, e_2, \ldots , e_r]}$, in that order.
  These elements form an independent set, and their weights are in ascending order: ${w_{e_1} \le w_{e_2} \le \cdots \le w_{e_r}}$.
  To see this set is of minimum weight, suppose there is another maximal independent set consisting of ${[x_1, \ldots, x_r]}$ of lower weight.
  Then for some $i$ we must have ${w_{x_i} \le w_{e_i}}$.
  Use the least such $i$, which we note must be greater than~$1$, since the greedy algorithm picks the min cost edge at the first step.
  Now consider ${B = \{e_1, \ldots, e_{i-1}\}}$ and ${A = \{x_1, \ldots, x_i\}}$.
  Since ${|A| > |B|}$ and both sets are independent, we use the exchange property.
  There is some $j$ such that ${x_j \in A \setminus B}$ and ${B \cup \{x_j\}}$ is independent.
  But since ${w_{x_j} \le w_{x_i} < w_{e_i}}}$, we have a contradiction, as the greedy algorithm would have picked $x_j$ before $e_i$.
\end{proof}

There is also a converse to the previous result: if an independence system ${(E, \I)}$ is not a matroid, then the greedy algorithm fails for some choice of integer weights.
You'll learn more about Greedy Algorithms if you take the undergraduate algorithms course CS~$4820$, and you'll learn more about matroids if you take the graduate algorithms course CS~$6820$ or the combinatorics course sequence~Math~$4410$ and~$4420$.

\end{document}


\subsection{Asymmetric encryption}

Here we use different keys to encrypt and decrypt the file.

Consider two users who wish to send a message securely.

One option would be to use symmetric encrpytion. They would have to meet and share this key securely, however, as transferring it over an insecure network would mean it could be copied.

With public key encryption each user has a public and a private key. The private key is kept secure locally, while the public key can be broadcasted.

In order to encrypt the file, the recipient's public key is used, while both the private and public key are needed to decrypt the file.

As a result anyone can encrypt a file to send to the user, but only the user can read what is sent.

Public-key encryption can be used to facilitate symmetric encryption. If only one party has a public key then the other user can send a symmetric key securely using the public key.

Using this, asymmetric encrpyiton is only used at the start.

This is how HTTPS operates, where the website has a public key, but the client does not.

Each user still needs to trust that the public key is accurate. This could be done by hosting the public key on a secure location.

RSA is an algorithm used for public-key encrpytion, including for HTTPS handshakes and PGP.


\begin{lem}
% \emph{
If $A$ and $B$ are the adjacency matrices of simple graphs that are isomorphic to one another, then the minimum of rQAP is equal to the minimum of QAP.
\end{lem}

\begin{proof}
Because any feasible solution to GM is also a feasible solution to rQAP, we must only show that the optimal objective function value to rQAP can be no better than the optimal objective function value of QAP.  Let $A=PBP\T$, so that $\langle A, PBP\T\rangle=2m$, where $m$ is the number of edges in $A$.  If rQAP could achieve a lower objective value, then it must be that there exists a $D \in \mc{D}$ such that $\langle A, DBD\T\rangle > \langle A, PBP\T\rangle = 2m$ (remember that we are minimizing the negative Euclidean inner product). For that to be the case, it must be that $(DBD\T)_{uv} \geq 1$ for some $(u,v)$.  That this is not so may be seen by the submultiplicativity of the norm induced by the $\ell_{\infty}$ norm:
$\norm{Dx}_\infty \leq \norm{D}_{\infty,\infty} \norm{x}_\infty$.  Applying this twice (once for each doubly stochastic matrix multiplication) yields our result.
% Consider $d_i=\langle D, \text{col}_i(BD\T) \rangle$, where $\text{col}_i(\cdot)$ indicates the $i^{th}$ column of the matrix.  $d_i \leq 1$ for all $i \in [n]$, therefore, our result holds.
\end{proof}

\paragraph{Linear Assignment Problems} % (fold)
% \label{ssub:linear_assignment_problems}

% subsubsection linear_assignment_problems (end)

The standard way of writing a Linear Assignment Problem (LAP) is
\begin{equation*}
% \text{(LAP)} \qquad
\begin{array}{cl}
			\text{minimize}   & \sum_{u,v \in [n]} a_{u \pi(v)} b_{uv} \\
			\text{subject to}  &P \in \mc{P}.
\end{array} %\label{eq:LAP}
\end{equation*}
The LAP objective function, like the QAP objective function, enjoys a number of equivalent formulations, including
% , similar to the QAP objective function (cf Eq. \eqref{eq:equiv})
% \begin{multline} \label{eq:equiv}
% (a_{uv}-b_{\pi(u)\pi(v)})^2 = \norm{A - PBP\T}_F^2 \\ = tr \{ (A - PBP\T)\T (A - PBP\T)\}.
% \end{multline}
%
%
% which can be written equivalently in a number of ways using the notion of permutation matrix introduced in the main text, including
\begin{equation}
\text{(LAP)} \qquad
\begin{array}{cl}
			\text{minimize}   & \langle P, AB\T \rangle \\
			\text{subject to}  &P \in \mc{P}.
\end{array}\label{eq:LAP}
\end{equation}
%
% \begin{subequations} \label{eq:LAP2}
% \begin{align}
% 	&\argmin_{\PmcP} \norm{PA - B}_F =\\
% 	&\argmin_{\PmcP} \, tr(PA-B)\T (PA-B)=\\
% 	% &\argmin_{\PmcP} tr (A\T P\T PA) - tr(2PAB\T) + tr(B\T B)=\\
% 	&\argmin_{\PmcP}  -tr (P AB\T) = \argmin_{\PmcP}  -\langle P\T, AB\T \rangle = \label{eq:2c} \\
% 	% &\argmin_{\PmcP}  -\sum_{u,v \in [n]} p_{ij} a_{ij} b_{ji}
% 	% =\\% &\argmin_{\PmcP}  - \text{vec}(P)\T \text{vec}(AB\T).=\\
% 	&\argmin_{\PmcP}  -\langle P, AB\T \rangle, \label{eq:dotLAP}
% \end{align}
% \end{subequations}
% where $\langle \cdot,\cdot \rangle$ %the equality on the second to last line defines is the usual Euclidean inner product, i.e., $\langle X,Y\rangle \defn tr(X\T Y)= \sum_{ij} x_{ij} y_{ij}$.
% While the objective function and the first two constraints of LAP are linear, t
The binary constraints of LAP---like those of QAP---make solving even this problem computationally tricky.  Nonetheless, in the last several decades, there has been much progress in accelerating algorithms for solving LAPs, starting with exponential time, all the way down to $\mc{O}(n^3)$ for general LAPs, and even faster for certain special cases (e.g., sparse matrices) \cite{Jonker1987, Burkard2009}.

% That Eq.~\eqref{eq:dir} is a LAP is evident by considering Eq.~\eqref{eq:dotLAP}.
To see that Eq.~\eqref{eq:dir} is identical to Eq.~\eqref{eq:LAP},
simply let $A=\nabla_P^{(i)}$ and $B=I$ (the $n\times n$ identity matrix).

% The last form indicates that LAP is a linear programming problem (hence the name).  Yet, the constraints, $\mc{P}$, make it a bit trickier.  The feasible region $\mc{P}$ can be written as a set of three constraints: two linear equality constraint sets and a binary constraint.  The LAP objection function with constraints can explicitly be written:
% \begin{align}
% 		&\text{minimize}_P  &&\sum_{u \in \mc{V}} -p_{ij} a_{ij} b_{ji} \nonumber \\
% 		&\text{subject to } && \sum_{u \in \mc{V}} p_{ij} = 1 \, \forall u \in \mc{V} \nonumber \\
% 		& && \sum_{v \in \mc{V}} p_{ij} = 1 \, \forall v \in \mc{V}, \nonumber \\
% 		& &&p_{ij} \in \{0,1\} \, \forall u,v. \label{eq:rLAP}
% \end{align}
% Perhaps because LAP comes up in a wide variety of contexts, a large number of algorithms have been developed to solve LAP \cite{Burkard2009}.  These algorithms have become increasing efficient.
% One of the most popular algorithms, the so-called ``Hungarian algorithm'' has time complexity $\mc{O}(n^3)$ \cite{Jonker1987}.  Under certain conditions (for example, when $AB\T$ is sparse), faster implementations are also available.  As will be seen below, LAP is a key subroutine to our inexact QAP solution.

To solve a LAP, consider a continuous relaxation of LAP, specifically, relaxing the permutation matrix constraint to a doubly stochastic matrix constraint:
% A matrix $P$ is doubly stochastic precisely when $P$ satisfies the following three conditions:
% \begin{enumerate}
% \item	$P\mb{1} = \mb{1}$,
% \item	$P\T \mb{1}=\mb{1}$, %\\
% \item 	$P \in  \Real_+^{n \times n}$,
% \end{enumerate}
% where the third constraint relaxes the binary constraints of the permutation matrices with a non-negativity constraint.
% Let $\mc{D}$ be the set of doubly stochastic matrices.
% With this, we now state a relaxed LAP problem:
\begin{equation}
\text{(rLAP)} \qquad
\begin{array}{cl}
			\text{minimize}   & \langle P, AB\T \rangle \\
			\text{subject to}  &P \in \mc{D}.
\end{array}\label{eq:LAP}
\end{equation}
% \begin{subequations} \label{eq:rLAP}
% \begin{align}
% 		\text{(rLAP) } \quad &\underset{P}{\text{minimize}}  &&-\langle P, AB\T \rangle \\
% 		&\text{subject to } && P \in \mc{D}.
% 		% && \sum_{u \in \mc{V}} p_{ij} = 1 \, \forall u \in \mc{V} \nonumber \\
% 		% 		& && \sum_{v \in \mc{V}} p_{ij} = 1 \, \forall v \in \mc{V}, \nonumber \\
% 		% 		& &&p_{ij} \geq 0 \, \forall u,v, \label{eq:ALAP}
% \end{align}
% \end{subequations}
As it turns out, the minima of LAP and rLAP are equal to one anther \cite{Burkard2009}.
% solving rLAP is equivalent to solving LAP.
% \begin{prop}
% 	LAP and rLAP are equivalent, meaning that they have the same optimal objective function value.
% \end{prop}
% \begin{proof}
% 	Although this proposition is typically proven by invoking total unimodularity, we present a  proof here that we find simpler.	Let $P^L$ be a solution to LAP and let $P^r = \sum_{i\in[k]} \alpha_i P_i$ be a solution to rLAP for some positive integer $k$, permutation matrices $\{P_i\}_{i \in [k]}$, and positive real numbers $\{\alpha_i\}_{i \in[k]}$ such that $\sum_{i \in [k]} \alpha_i=1$.  Note that it must be the case that
% \begin{align}
% 	\langle P^L, AB\T \rangle \geq \langle P^r, AB\T \rangle,
% \end{align}
% because $P^L$ is also a solution for rLAP. Expanding $P^r$, we have
% \begin{align}
% 	\langle P^r, AB\T \rangle = \langle \sum_i \alpha_i P_i, AB\T \rangle = \sum_i \alpha_i \langle  P_i, AB\T \rangle.
% \end{align}
% But we know that
% \begin{align}
% 	\langle P^L, AB\T \rangle \leq \langle P, AB\T \rangle
% \end{align}
% for any $P \in \mc{P}$ because $P^L$ solves LAP, so
% 	%
% 	%
% 	% \begin{multline}
% 	% \langle P^L,AB\T \rangle = \langle  \sum_{i\in[k]} \alpha_i P_i^L, AB\T \rangle=  \sum_{i\in[k]} \alpha_i \langle  P_i^L, AB\T \rangle	 \\
% 	% \leq \sum_{i\in[k]} \alpha_i \langle P^r, AB\T  \rangle = \langle P^r, AB\T \rangle \leq \langle P^L, AB\T \rangle,
% 	% \end{multline}
% 	% % then we have a contradiction,
% 	% because $P^r$ is feasible in rLAP.
% 	\end{proof}
This relaxation motivates our approach to approximating QAP.

\begin{algorithm}
	\caption{\FAQ~ for finding a local optimum of rQAP} \label{alg:1}
\begin{algorithmic}[1]
	\REQUIRE graphs $A$ and $B$ as well as stopping criteria
	\ENSURE $\wh{P}$, an estimated permutation matrix
	\STATE Choose an initialization, $P^{(0)}=\mb{1}\mb{1}\T/n$ \label{step:init} %\COMMENT{although points in $\mc{D}$ would also be feasible}
	\WHILE{stopping criteria not met}
	\STATE Compute the gradient of $f$ at the current point via Eq.~\eqref{eq:grad}
	\STATE Compute the direction $Q^{(i)}$ by solving Eq.~\eqref{eq:dir} via the Hungarian algorithm
	\STATE Compute the step size $\alpha^{(i)}$ by solving Eq.~\eqref{eq:step}
	\STATE Update $P^{(i)}$ according to Eq.~\eqref{eq:update}  %$P^{(i+1)} \leftarrow P^{(i)} + \alpha^{(i)} Q^{(i)}$
	\ENDWHILE
	\STATE Obtain $\wh{P}$ by solving Eq.~\eqref{eq:proj} via the Hungarian algorithm.
\end{algorithmic}
\end{algorithm}

\begin{table}[h!]
\caption{Comparison of \FAQ~ with optimal objective function value and previous state-of-the-art for directed graphs.  The best (lowest) value is in \textbf{bold}. Asterisks indicate achievement of the global minimum.  The number of vertices for each problem is the number in its name (second column).}
\begin{center}
\begin{tabular}{|r|r|r||l|l|l|l|l|}
	\hline
	          \# &  Problem &      Optimal & \FAQ~ & \Epath~& \Grad~ \\
	\hline
	           1 &  lipa20a &     3683 & \textbf{3791} &     3885 &     3909 \\
	           2 &  lipa20b &    27076 & \textbf{27076}$^*$ &    32081 &    \textbf{27076}$^*$ \\
	           3 &  lipa30a &    13178 & \textbf{13571} 	&    13577 &    13668 \\
	           4 &  lipa30b &   151426 & \textbf{151426}$^*$ & \textbf{151426}$^*$ &   \textbf{151426}$^*$ \\
	           5 &  lipa40a &    31538 & \textbf{32109} 	&    32247 &    32590 \\
	           6 &  lipa40b &   476581 & \textbf{476581}$^*$ &   \textbf{476581}$^*$ &   \textbf{476581}$^*$ \\
	           7 &  lipa50a &    62093 & \textbf{62962} &    63339 &    63730 \\
	           8 &  lipa50b &  1210244 & \textbf{1210244}$^*$ &  \textbf{1210244}$^*$ &  \textbf{1210244}$^*$ \\
	           9 &  lipa60a &   107218 & \textbf{108488} &   109168 &   109809 \\
	          10 &  lipa60b &  2520135 & \textbf{2520135}$^*$ &  \textbf{2520135}$^*$ &  \textbf{2520135}$^*$ \\
	          11 &  lipa70a &   169755 & \textbf{171820} &   172200 &   173172 \\
	          12 &  lipa70b &  4603200 & \textbf{4603200}$^*$ &  \textbf{4603200}$^*$ &  \textbf{4603200}$^*$ \\
	          13 &  lipa80a &   253195 & \textbf{256073} &   256601 &   258218 \\
	          14 &  lipa80b &  7763962 & \textbf{7763962}$^*$ &  \textbf{7763962}$^*$ &  \textbf{7763962}$^*$ \\
	          15 &  lipa90a &   360630 & \textbf{363937} &   365233 &   366743 \\
	          16 &  lipa90b & 12490441 & \textbf{12490441}$^*$ & \textbf{12490441}$^*$ & \textbf{12490441}$^*$ \\
	\hline
	\end{tabular}
\end{center}
\label{tab:directed}
\end{table}%

\begin{table}[h!]
\caption{Comparison of \FAQ~ with optimal objective function value and the best result  on the undirected benchmarks.  Note that \FAQ~ restarted 100 times finds the optimal objective function value in 3 of 16 benchmarks, and that \FAQ~ restarted 3 times finds a minimum better than the previous state-of-the-art on all 16 particularly difficult benchmarks.}
\begin{center}
\begin{tabular}{|r|r|r||l|l|l|l|l|}
\hline
\# & Problem  &   Optimal    & \FAQ$_{100}$ & \FAQ$_{3}$ & previous min \\
\hline
1&    chr12c &   11156 &    \textbf{12176} &   13072 & 13072 \\
2&    chr15a &    9896 &    \textbf{9896}$^*$ &   17272 &  19086 \\
3&    chr15c &    9504 &    \textbf{10960} &   14274 &  16206 \\
4&   chr20b &    2298 &     \textbf{2786} &    3068 &    3068 \\
5&    chr22b &    6194 &    \textbf{7218} &    7876 &   8482 \\
6&    esc16b & 	292 & 		\textbf{292}$^*$ & 294 &    296 \\
7& 	   rou12 &  235528 &  \textbf{235528}$^*$ &  238134 &    253684 \\
8& 	   rou15 &  354210 &  \textbf{356654} &  371458 &    371458 \\
9&      rou20 &  725522 &  \textbf{730614} &  743884 &    743884 \\
10&    tai10a &  135028 &  \textbf{135828} &  148970 &    152534 \\
11&    tai15a &  388214 &  \textbf{391522} &  397376 &    397376 \\
12&    tai17a &  491812 &  \textbf{496598} &  511574 &    529134 \\
13&    tai20a &  703482 &  \textbf{711840} &  721540 &    734276 \\
14&    tai30a & 1818146 & \textbf{1844636} & 1890738 &  1894640 \\
15&    tai35a & 2422002 & \textbf{2454292} & 2460940 &  2460940 \\
16&    tai40a & 3139370 & \textbf{3187738} & 3194826 &  3227612 \\
    \hline
\end{tabular}
\end{center}
\label{tab:restarts}
\end{table}%


% !TEX root = scombinatorics.tex
\documentclass[scombinatorics.tex]{subfiles}
\begin{document}
\chapter{Vapnik-Chervonenkis theory}
\label{sauer}

\def\medrel#1{\parbox[t]{5ex}{$\displaystyle\hfil #1$}}
\def\ceq#1#2#3{\parbox[t]{20ex}{$\displaystyle #1$}\medrel{#2}{$\displaystyle #3$}}

%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%
\section{The Vapnik-Chervonenkis dimension}\label{VCdim}

If all subsets of $A\subseteq\U$ are definable, that is $\P A=\phi(A,b)_{b\in\V}$ we say that $A$ is \emph{shattered\/} by $\phi(x\,;z)$.
%Paraphrase the the definition,  $A$ is shattered by $\phi$ if fr every $C\subseteq A$ there is a $b_C\in\V$ such that $C=\phi(A,b_C)$.
The following is called the \emph{shatter function\/}

%\ceq{\hfill\emph{$\pi_\phi(n)$}}{=}{\max\Big\{|\phi(A,b)_{b\in\V}|\ \ :\ \ A\subseteq\U,\ |A|=n\Big\}.}

\ceq{\hfill\emph{$\pi_\phi(n)$}}{=}{\max\bigg\{|\phi(A,b)_{b\in\V}|\ \ :\ \ A\in {\U\choose n}\bigg\}}

So, $\pi_\phi(n)$ gives the maximal number of definable subsets that a set of cardinality $n$ can have.
Trivially, $\pi_\phi(n)\le2^n$ for all $n$.
Moreover, if $\pi_\phi(n)=2^n$ for some $n$, then $\pi_\phi(k)=2^k$ for every $k\le n$.

The \emph{Vapnik-Chervonenkis dimension\/} of $\phi(\U\,;b)_{b\in\V}$, or of $\phi(x\,;z)$, abbreviated by \emph{\vc-dimension}, is the maximal cardinality of a finite set $A\subseteq\U$ that is shattered by $\phi(x\,;z)$.
Equivalently, it is the maximal $k$ such that $\pi_\phi(k)=2^k$.
If such a maximum does not exist,
we say that the \vc-dimension is infinite or that $\phi(x\,;z)$ has \emph{\ip\/} (the independence property).
Otherwise, we say that $\phi(x\,;z)$ has \emph{\nip\/} (not the independence property).
We may also say: \textit{is\/} \ip, or \textit{is\/} \nip.

As $\U$ and $\V$ are usually clear from the context, we may say \vc-dimension of $\phi(x\,;z)$ for the \vc-dimension of $\phi(\U\,;b)_{b\in\V}$.

\begin{example}
  If $\phi(x\,;z)$ is either $\top$ or $\bot$, then it shatters only the empty set, therefore it has \vc-dimension $0$.\QED
\end{example}

\begin{example}
  If $\phi(x\,;z)$ has ladder dimension $n$ then it has \vc-dimension at most $n$. Hence stable formulas are \nip.\QED
\end{example}

\begin{example}
  If $\phi(\U\,;b)_{b\in\V}$ is a non trivial chain of sets, then its \vc-dimension is $1$.\QED
\end{example}

\begin{example}
  Let $\U=\RR$ and $\V=\RR^2$.
  Let $\phi(x\,;z_1,z_2)$ be the formula $z_1<x<z_2$.
  Then its \vc-dimension $2$.\QED
\end{example}

\begin{example}
  Let $\U=\V=\RR^2$.
  Let $\phi(x_1,x_2\,;z_1,z_2)$ be the formula $x_2<z_1\cdot x_1 + z_2$.
  Then its  \vc-dimension $3$ (by Radon's Theorem).\QED
\end{example}

\begin{example}\label{ex_vcdim_opt}
  If $\phi(\U\,;b)_{b\in\V}$ is the set of all subsets of $\U$ of cardinality $\le k$.
  Then its \vc-dimension is $k$ and

  \ceq{\hfill\pi_\phi(n)}{=}{\sum^k_{i=0} \binom{n}{i}.}

  Incidentally, we note that this is also the shatter function of the collection of all subsets of $\U$ of cardinality exactly $k$.
  In fact we always assume $\U$ is infinite (or at least very large, in this case, size $\ge2k$ suffices).\QED
\end{example}

The \vc-dimension of $\phi(x\,;z)^{\rm op}$ is called the \emph{dual \vc-dimension\/} or \emph{\vc-codimension\/} of $\phi(x\,;z)$.

\begin{proposition}\label{prop_bound_VCcodim}
  If $\phi(x\,;z)$ has \vc-dimension $k$, then its \vc-codimension is $\le2^{k+1}-1$.
\end{proposition}

\begin{proof}
  Suppose that the \vc-dimension of $\phi(x\,;z)^{\rm op}$ is $\ge 2^{k+1}$.
  We prove that the \vc-dimension of $\phi(x\,;z)$ is at least $k+1$.
  Let $B=\big\{b_I\,:\,I\subseteq [k+1)\big\}$ be a set of cardinality $2^{k+1}$ shattered by $\phi(x\,;z)^{\rm op}$.
  That is, for every $\J\subseteq \P[k+1)$ there is $a_\J$ such that

  \ceq{\hfill\phi(a_\J, b_I)}{\IFF}{I\in\J}\hfill for all $I\subseteq [k+1)$

  Let $a_i=a_{\{I\; :\ i\,\in\, I\}}$. Then from the equivalence above we obtain

  \ceq{\hfill\phi(a_i, b_I)}{\IFF}{i\in I}

  That is, $\phi(x\,;z)$ shatters $A=\big\{a_i\,:\, i\in [k+1)\big\}$.
\end{proof}

We prove that the bound in the proposition above is optimal.

\begin{proposition}
  For every $k$, there is a formula $\phi(x\,;z)$ that has \vc-dimension $k$ and \vc-codimension $2^{k+1}-1$.
\end{proposition}
\begin{proof}
  Let $k$ be given.
  We claim that there is a formula $\phi(x\,;z)$ with \vc-dimension $2^{k+1}-1$ and \vc-codimension $k$.
  As the dual of the dual is the primal, this claim is equivalent to the proposition.
  Let $\U$ an infinite set, let $\V=\P(\U)$.
  Fix an equivalence relation on $\U$ with $2^{k+1}-1$ many classes.
  We say that $b\in\V$ is compatible if it is union of equivalence classes.
  Define

  \ceq{\hfill\phi(x\,;z)}{=}{x\in z \ \wedge\ z\textrm{ is compatible}}%\A x_1,x_2\big[x_1\sim x_2\imp [x_1\in z\iff x_2\in z]\big]}

  (Recall that in the preliminaries to this notes, we have agreed that $\U$ and $\V$ are some fixed infinite sets. Without this constraint we could have taken as $\U$ a set of cardinality $2^{k+1}-1$; defined $\phi(x\,;z)=x\in z$; and forgot the equivalence relation altogether.)

  Clearly, $\phi(x\,;z)$ has \vc-dimension $2^{k+1}-1$.
  Note that the dimension of $\phi(x\,;z)^{\rm op}$ is at least $k$.
  Otherwise $\phi(x\,;z)$ would have dimension $\le2^k-1$ by Proposition~\ref{prop_bound_VCcodim}.

  So, it suffices to prove that the dimension of $\phi(x\,;z)^{\rm op}$ is exactly $k$.
  Assume for a contradiction that some $B=\big\{b_i\ :\ i\in[k+1)\big\}\subseteq\V$ is shattered by $\phi(x\,;z)^{\rm op}$.
  Then there is some set $A=\big\{a_I:I\subseteq[k+1)\big\}\subseteq\U$ such that

  \ceq{\hfill\phi(a_I, b_i)}{\IFF}{i\in I}

  The $b_i\in B$ are necessarily compatible sets, therefore any two distinct $a_I\in A$ are non equivalent.
  But this is impossible by cardinality reasons.
\end{proof}

\begin{exercise}
  Let $\phi(x\,;z)$ have \vc-dimension $k$.
  Assume that there is no $A\subseteq\U$ of cardinality $\le k+1$ such that $\phi(a\,;\V)_{a\in A}$ covers $\V$.
  Prove that $\phi(x\,;z)$ has \vc-codimension $\le2^{k+1}-2$.
  Prove that the bound is optimal.
  Hint: let $\U$ be an infinite set, $\V = \P(\U)$, and consider an equivalence relation $\sim$ s.t. $\U / {\sim} = C \cup \{c_0, \dots, c_k\}$ with $|C| = 2^{k+1}-2$ and take the same formula used in the previous Proposition, but with $b \in \V$ compatible if it is the union of equivalence classes that belongs to $C$ or if it coincides with $c_i$ for some $i \in \{0,\dots, k\}$.
  Prove that $\phi(x\,;z)$ has \vc-codimension $k$.\QED
\end{exercise}

% \begin{exercise}
%   Let $A\subseteq\U$ have finite cardinality $n$.
%   Let $\phi(x\,;z)$ have \vc-dimension $k$.
%   Define an equivalence relation on $\V$ as follows.

%   \ceq{\hfill b\sim_Ab'}
%   {\IFF}
%   {}

% \end{exercise}

%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%
\section{The Sauer-Shelah lemma}\label{sauer}

\def\ceq#1#2#3{\parbox[t]{15ex}{$\displaystyle #1$}\medrel{#2}{$\displaystyle #3$}}

According to Gil Kalai in~\cite{kalai}, Sauer-Shelah's Lemma can been described as an \textit{eigentheorem\/} because it is important in many different areas of mathematic (model theory, learning theory, probability theory, ergodic theory, Banach spaces, to name a few).
No wonder it has been discovered and rediscovered may times.

It has been proved independently by Shelah~\cite{shelah72}, Sauer~\cite{sauer}, and Vapnik-Cher\-vo\-nen\-kis~\cite{VC} around 1970 (Shelah gives credit to Micha Perles).
Saharon Shelah was working in model theory while Norbert Sauer, Vladimir Vapnik and Alexey Chervonenkis were in statistical learning theory.

\begin{void_thm}[Sauer-Shelah Lemma]\label{lem_sauer}
If $\phi(x\,;z)$ has \vc-dimension $k$ then for every $n\ge k$

\ceq{\hfill\pi_\phi(n)}{\le}{\bigsum^{k}_{i=0} \binom{n}{i}.}
\end{void_thm}

The set system presented in Example~\ref{ex_vcdim_opt} shows that the bound is optimal.

An alternative proof of the Sauer-Shelah Lemma derives it as corollary of a lemma by Alain Pajor~\cite{pajor}.

\begin{void_thm}[Pajor's Lemma]\label{lem_pajor}
  Let $A\subseteq\U$ be finite.

  \ceq{\hfill|\phi(A,b)_{b\in\V}|}{\le}{\Big|\{C\subseteq A\; :\; C \textrm{\ is\ shattered\ by\ }\phi(x\,;z)\}\Big|}.
\end{void_thm}

\begin{proof}
  If $A$ is empty then $|\phi(A,b)_{b\in\V}|=1$ and $\0$ is the only subset of $A$ that $\phi$ shatters, so the inequality holds trivially.
  Otherwise, pick an $a\in A$ and assume the lemma holds for $A'=A\sm\{a\}$.
  Define

  \ceq{\hfill\psi(x\,;y)}{=}{\phi(x\,;y)\ \wedge\neg \phi(a\,;y)\ \wedge\ \E y'\, \Big[ \phi(a\,;y')\ \wedge\ \phi(A'\,;y')=\phi(A'\,;y)\Big].}

  Notice that

  \ceq{\hfill\big|\phi(A,b)_{b\in\V}\big|}{=}{\bigg|\phi(A',b)_{b\in\V}\quad \cup\quad \Big\{\{a\}\cup\psi(A',b)\ :\ b\in\V\Big\}\bigg|.}

  as the two sets in the r.h.s.\@ are disjoint

  \ceq{\hfill\big|\phi(A,b)_{b\in\V}\big|}{=}{\big|\phi(A',b)_{b\in\V}\big|\ +\ \big|\psi(A',b)_{b\in\V}\big|.}

  By induction hypothesis,

  \ceq{\hfill\big|\phi(A',b)_{b\in\V}\big|}{\le}{\Big|\{C\subseteq A'\; :\; C \textrm{\ is\ shattered\ by\ }\phi(x\,;z)\}\Big|}\hfill(1)

  and

  \ceq{\hfill\big|\psi(A',b)_{b\in\V}\big|}{\le}{\Big|\{C\subseteq A'\; :\; C \textrm{\ is\ shattered\ by\ }\psi(x\,;z)\}\Big|}

  \ceq{}{=}{\Big|\{C\subseteq A'\; :\; C\cup\{a\} \textrm{\ is\ shattered\ by\ }\phi(x\,;z)\}\Big|.}\hfill(2)

  In fact, $C\subseteq A'$ is shattered by $\psi(x\,;y)$ if an only if $C\cup\{a\}$ it is shattered by $\phi(x\,;y)$.
  Clearly,

  \ceq{\hfill(1)+(2)}{=}{\Big|\{C\subseteq A\; :\; C \textrm{\ is\ shattered\ by\ }\phi(x\,;z)\}\Big|,}

  so the lemma follows.
\end{proof}

\begin{proof}[Proof of the Sauer-Shelah Lemma]
  Given $\phi(x\,;z)$ and $n\ge k$ as in the lemma

  \ceq{\hfill\pi_\phi(n)}{=}{\max_{|A|=n}\big|\phi(A,b)_{b\in\V}\big|}\smallskip

  \ceq{\hfill\pi_\phi(n)}{\le}{\max_{|A|=n}\big|\{C\subseteq A\; :\; C\textrm{\ shattered\ by\ }\phi(x\,;z)\}\big|}\hfill by  Pajor's Lemma\smallskip

  \ceq{}{\le}{\sum^k_{i=0}{n\choose i}}\hfill because $\phi(x\,;z)$ has \vc-dimension $k$
\end{proof}

We write $f(n)=O(g(n))$ if there is a constant $C$ such that $|f(n)|\le C g(n)$ holds for all (sufficiently large) $n$.

The \emph{\vc-density\/} of $\phi(x\,;z)$ is the infimum over all real number $r$ such that $\pi_\phi(n)= O(n^r)$.
It is infinite if no such $r$ exist.
The  \emph{dual \vc-density\/} is defined accordingly.

By the Sauer-Shelah lemma the \vc-density is at most as large as the \vc-dimension.
It could be smaller, however it is usually rather difficult to compute.

We conclude this section with a couple of inequalities that is useful to have at hand.

\ceq{\hfill\sum^k_{i=0}{n\choose i}}
{=}
{\sum^k_{i=0}\frac{n!}{i!\,(n-i)!}}

\ceq{}
{\le}
{\sum^k_{i=0}\frac{n^i}{i!}}

\ceq{}
{\le}
{\sum^k_{i=0}\frac{n^i\,k!}{i!(k-i)!}}

\ceq{}
{=}
{\sum^k_{i=0}n^i{k\choose i}}

\ceq{}
{=}
{(n+1)^k}
\hfill by the binomial theorem.\smallskip

There is a second bound, which is better when $k\ge 3$ and holds for $n>k$

\ceq{\hfill\sum^k_{i=0}{n\choose i}}
{\le}
{\Big(\frac{n}{k}\Big)^{\!k}\sum^k_{i=0}\Big(\frac{k}{n}\Big)^{\!i}{n\choose i}}
\hfill because $\dfrac{k}{n}<1$

\ceq{}
{\le}
{\Big(\frac{n}{k}\Big)^{\!k}\sum^n_{i=0}\Big(\frac{k}{n}\Big)^{\!i}{n\choose i}}

\ceq{}
{=}
{\Big(\frac{n}{k}\Big)^{\!k}\Big(1+\frac{k}{n}\Big)^n}
\hfill by the binomial theorem

\ceq{}
{\le}
{\Big(\frac{n\,e}{k}\Big)^{\!k}}
\hfill where $e$ is the base of the natural logarithm.

\end{document}


\pagebreak
\null
\newpage
\section{Discrete Laplacians}\label{sec:Chapter3: other discrete laplacians}
In this Chapter we describe different methods to approximate the Laplace-Beltrami operator on manifolds, with particular attention to what we will call the Heat Kernel Graph Laplacian.

Let $f$ be a sufficiently smooth function on a compact, closed infinitely differentiable manifold $\mathcal M$. The Laplacian eigenvalue problem on $\mathcal M$ is defined as
\begin{equation}\label{eq:continous eigenvalue problem}
\triangle_{\mathcal M}\psi = -\lambda \psi
\end{equation}
Being the Laplace-Beltrami operator self-adjoint and semi-positive definite, there exists a basis $\mathcal B=\{\psi_i\}_i$of the space $L^2(\mathcal M)$ such that $\triangle_\mathcal M \psi_i = -\lambda_i\psi_i,\ \lambda_0\leq\lambda_1\leq...,\lambda_i\leq\lambda_{i+1}...\leq+\infty$. See \cite{rosenberg_1997} for an introduction to the Laplace-Beltrami operator on manifolds.

There's been a lot of work in trying to calculate solutions to equation (\ref{eq:continous eigenvalue problem}), leading to different ways to approximate the Laplace-Beltrami operator through what we call a \textit{discrete Laplacian}. By discrete Laplacian we mean an operator $L$ that, once evaluated on a signal $f$ and on a vertex $x_i$ can be written as a matrix $\mathbf L$ in the following way:

\begin{equation}\label{eq:discrete laplacian}
\mathbf L f\left(\mathbf{x}_{i}\right)=\frac{1}{d_{i}} \sum_{j} w_{i j}\left(f\left(\mathbf{x}_{i}\right)-f\left(\mathbf{x}_{j}\right)\right)
\end{equation}

Note that with unit \textit{masses} $d_i=1$ we recover the same definition of Graph Laplacian. We need now to introduce some basic concepts of Differential Geometry, especially the definition of mean curvature of a manifold and its link with the Laplace-Beltrami operator.
\subsection{Notions of Differential Geometry}

 For this short introduction to basic concepts of Differential Geometry, set the manifold $\mathcal M$ to be a differentiable, two dimensional surface embedded in $\mathbb R^3$.
 \begin{figure}[h]
 	\centering
 	\includegraphics[width=0.7\textwidth]{figs/Chapter3/curvature.png}
 	\caption{\label{fig:curvature}Curvature normal of a manifold}
 \end{figure}
The \textit{curvature} of a curve on a plane is defined as the inverse of the radius $R$ of the tangent circle. For each point on the manifold $\mathcal M$, define its tangent plane $H$, orthogonal to the normal vector $\mathbf n$. For every unit vector $\mathbf e_\theta$ lying on the tangent plane $H$, where $\theta$ is an angle that measures the direction on the tangent plane of $\mathbf e_\theta$, the \textit{normal curvature} $\kappa(\theta)$ is defined as the curvature of the curve that is the intersection of the manifold $\mathcal M$ and the plane containing both $\mathbf n$ and $\mathbf e_\theta$. The \textit{mean curvature} $\overline \kappa $ is defined as the average on $\theta$ of the normal curvatures:

 \begin{equation}\label{eq:mean curvature}
 	\overline \kappa=\frac{1}{2 \pi} \int_{0}^{2 \pi} \kappa(\theta) d \theta
 \end{equation}

It can be proved that the Laplace-Beltrami operator applied on the identity function $\mathbf x \rightarrow \mathbf x, \ \forall \mathbf x\in \mathcal M$ is directly linked to the \textit{mean curvature normal} $\overline{\kappa}\mathbf n$ by the following formula:
\begin{equation}\label{eq:laplacian and curvature}
	\triangle_\mathcal M \mathbf x  = -2\overline{\kappa}\mathbf n
\end{equation}
This equation provides us a way to approximate the Laplace-Beltrami operator through the approximation of the mean curvature normal. This fact is exploited by methods presented in the next section.

\subsection{Discrete Laplacians from Differential Geometry}
\begin{figure}[h]
	\begin{center}
		\includegraphics[width=0.45\textwidth]{figs/Chapter3/MyDesbrun.png}
		\includegraphics[width=0.45\textwidth]{figs/Chapter3/Voronoi}
	\end{center}
	\caption{\label{fig:Desbrun}One term of curvature normal formula and one Voronoi cell constructed around the node $x_i$}
\end{figure}
Desbrun et al. \cite{Desbrun1999} construct a triangulation $\mathcal T_h$ with the vertices in the sampling $x_0, x_1, ..., x_{n-1}$ approximating the manifold $\mathcal M$, and then use the following discrete expression for the \textit{discrete normal curvature} $\overline{\kappa_h} \mathbf{n}$ of the manifold $\mathcal M$:
\begin{equation}\label{eq:curvature normal}
	-\overline{\kappa_h} =\frac{1}{4 A_i} \sum_{j \in N_{1}(i)}\left(\cot \alpha_{ij}+\cot \beta_{ij}\right)\left(x_{j}-x_{i}\right)
\end{equation}
where $A_i$ is the area of all the triangles of the mesh sharing the node $x_i$; $N_1(i)$ is the first ring of neighbors of the $i$th node; $\alpha_{i j},\ \beta_{i j}$ are the angles of the triangles of the mesh that lie on the opposite side to the edge $(i,j)$ with respect to the node $x_i$ (Figure \ref{fig:Desbrun}). Observe that for a flat surface the discrete curvature is equal to zero $\overline{\kappa_h}=0$. This is a geometric approach that relies on the intrinsic properties of the triangulation $\mathcal T_h$ and is based on the geometric meaning of the curvature normal $\overline{\kappa}$. Using equations (\ref{eq:curvature normal}) and (\ref{eq:laplacian and curvature}) it can be shown \cite{REUTER2009381} that this approach leads to a discrete Laplacian with masses
$$
d_i=\frac{A_i}{3}
$$
where $A_i$ is the area of all the triangles of the mesh with a vertex in $x_i$, and weights
$$
w_{i j}=\frac{\cot \left(\alpha_{i j}\right)+\cot \left(\beta_{i j}\right)}{2}
$$

\subsection{Linear Finite Element Method Laplacian}
The eigenvalue problem (\ref{eq:continous eigenvalue problem}) can be rewritten in the equivalent weak form
\begin{equation}\label{eq:weak}
	\langle \nabla f, \nabla v\rangle_{L^2(\mathbb S^2)} = \lambda \langle  f, v\rangle_{L^2(\mathbb S^2)} \quad \forall v \in L^2(\mathbb S^2)
\end{equation}
The Finite Element Method (FEM) is a numerical algorithm that allows to calculate a discrete approximation of the solution $f$ through a functional discretization of the weak eigenvalue problem (\ref{eq:weak}). We will discuss this method deeper in section \ref{sec:Chapter3: Using the Finite Element Method to approximate the Laplace-Beltrami operator on a manifold}. By projecting equation (\ref{eq:weak}) on a finite dimensional functional subspace of $L^2(\mathbb S^2)$ spanned by $n$ basis functions $\phi_i$, by writing $n$ times the equation (\ref{eq:weak eigenvalue problem}), setting each time the \textit{test} function $v$ equal to the $i$th basis function $\phi_i$ we obtain the generalized algebraic eigenvalue problem
$$
\begin{aligned}
&\text{Find }(f,\lambda)\text{ such that }\mathbf A\mathbf f = \lambda \mathbf B \mathbf f\\
&\begin{cases}
(\mathbf A)_{ij} &= \int_{\mathbb S^2}\nabla \phi_i(\mathbf{x})\cdot \nabla \phi_j(\mathbf{x})d\mathbf{x}\\
(\mathbf B)_{ij} &= \int_{\mathbb S^2} \phi_i(\mathbf{x}) \phi_j(\mathbf{x})d\mathbf{x}\\
(\mathbf f)_i &= f_i:\quad f(\mathbf x) = f_0\phi_0(\mathbf x)+ ... + f_{n-1}\phi_{n-1}(\mathbf x)
\end{cases}
\end{aligned}
$$
that can be solved through usual algebraic solvers.

\paragraph{FEM Laplacian as a Differential Geometry Laplacian.} Levy \cite{levy} showed that by using the \textit{lumped} mass matrix $\mathbf D_{ii} = \sum_j \mathbf B_{ij}$ the FEM Laplacian
$$
\mathbf D^{-1}\mathbf A
$$
is the same Laplacian of Desbrun et al. \cite{Desbrun1999} introduced in the previous section. Meyer et al. \cite{Meyer02discretedifferential-geometry} proposed another discrete Laplacian by modifying the masses of Desbrun et al. setting them to
$$
d_i=a_{V}(i),
$$
where \(a_{V}(i)\) is the area of the polygon obtained by joining the circumcenters of the triangles surrounding node $i$ (i.e. the Voronoi cell, figure \ref{fig:Desbrun}).

\subsection{Graph Laplacian for manifolds}\label{sec:Chapter1:theoretical foundations}
Belkin et al. \cite{NIPS2006_2989} prove convergence of eigenvectors of the \textit{Heat Kernel Graph Laplacian} $\mathbf L_n^t$ (HKGL) of a data point cloud to the eigenfunctions of the Laplace Beltrami operator $\Delta_\mathcal M$ on the manifold $\mathcal M$, when the data is sampled from a uniform distribution on $\mathcal M$.
For this result to hold, they suppose the manifold $\mathcal M$ to be compact, infinitely differentiable and without boundary. We just point out that being $\mathcal M$ compact, $\Delta_\mathcal M$ has a discrete spectrum. The graph they use to approximate the manifold is constructed as follows: given a sampling $ \mathcal P = \{x_i\in\mathcal M\}_{i=0}^{n-1}$ on a $k$-dimensional manifold $\mathcal M\subset \mathbb R^N$ they construct the full graph defined by the weights
$$
w_{ij}=\exp\left({-\frac{||x_i-x_j||^2}{4t}}\right)
$$
where $\norm\cdot$ is the Euclidean norm in the ambient space $\mathbb R^N$ and whose Laplacian matrix
$$
\mathbf L_n^t
$$ we call \textit{Heat Kernel Graph Laplacian}.

Observe that given a function $f: \mathcal P \rightarrow \mathbb R$ defined on the sampling $ \mathcal P$ and defined the vector $\mathbf f\in\mathbb R^n$ such that $\mathbf f_i = f(x_i)$, the Heat Kernel Graph Laplacian matrix acts on $\mathbf f$ in the following way:
\begin{equation}\label{eq:HKGL}
(\mathbf L_n^t \mathbf f)_i:=  \sum_{j=0}^{n-1} e^{-\frac{||x_i-x_j||^2}{4t}}\left(f(x_i)-f(x_j)\right)
\end{equation}
This graph construction is motivated by the fact that the HKGL is nothing else than the natural discretization of the continuous \textit{functional approximation to the Laplace-Beltrami operator} $L^t:  L^{2}(\mathcal{M}) \rightarrow L^{2}(\mathcal{M})$ whose eigenvectors and eigenvalues are proven to converge to the ones of $\Delta_\mathcal M$.
\vspace{0.5cm}
\begin{definition}{}(\cite[Belkin et al.]{Belkin:2005:TTF:2138147.2138189}Functional approximation to the Laplace-Beltrami operator)\\ \label{eq: my L^t} Let $\mu$ be the uniform probability measure on the manifold $\mathcal M$, where $\text{vol}(\mathcal M)$ is the volume of $\mathcal M$. We define the functional approximation to the Laplace-Beltrami operator to be the operator $L^t: L^{2}(\mathcal{M}) \rightarrow L^{2}(\mathcal{M})$ such that
	\label{def:Functional approximation to the Laplace-Beltrami operator}
	$$ L^tf(y) = \int_{\mathcal M}e^{-\frac{||y-x||^2}{4t}}\left(f(y)-f(x)\right)d\mu(x)$$
\end{definition}
We end this Chapter by stating the theorem of spectral convergence of the HKGL to the Laplace-Beltrami operator $\Delta_\mathcal M$, that makes it a really good candidate to construct rotation invariant graphs.
\vspace{0.5cm}
\begin{snugshade*}
	\begin{theorem}(Belkin et al., \cite{NIPS2006_2989})\label{theo:spectral convergence}
		Let \(\lambda_{n, i}^{t}\) be the $i$th eigenvalue of
		$$
		\frac{(4\pi t)^{-(k+2)/2}}{n}\mathbf L^t_n
		$$
		and \(\mathbf v_{n, i}^{t}\) be the corresponding eigenvector. Let \(\lambda_{i}\) and \(v_{i}\) be the corresponding eigenvalue and eigenfunction of \(\Delta\) respectively. Then there exists a sequence \(t_{n} \rightarrow 0,\) such that
		\begin{equation}
		\begin{array}{c}{\lim _{n \rightarrow \infty} \lambda_{n, i}^{t_{n}}=\lambda_{i}} \\
		{\lim _{n \rightarrow \infty}\left\|\mathbf v_{n, i}^{t_{n}}-v_{i}(\mathbf x)\right\|_{2}=0}\end{array}
		\end{equation}
		where the limits are in probability.
	\end{theorem}
\end{snugshade*}


\documentclass{scrartcl}

\input{../../shared.tex}
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]
\newtheorem{lemma}{Lemma}[section]
\usepackage{enumerate}

\begin{document}

\section*{Problem set: Gradient descent}%

\paragraph{Exercise (i)} (7P) Enter the missing code snippets in the jupyter notebook. Partial credit will be awarded.

\begin{definition}
  For a matrix $Q \in \R^{m \times d}$ its \textbf{operator norm} is defined as
  \begin{equation}
    \Vert Q \Vert_{op} := \sup_{x \in \R^d: \Vert x \Vert \le 1} \Vert Q x \Vert,
  \end{equation}
  but for convenience we will mostly write $\Vert Q \Vert$ (so no subscript) but mean the operator norm.
\end{definition}

\begin{lemma}%
  The operator norm of a matrix $Q$ fulfills
  \begin{equation}
    \Vert Qx \Vert \le \Vert Q \Vert_{op} \Vert x \Vert.
  \end{equation}
  In particular, every linear map (given by a matrix $Q$) is Lipschitz continuous with Lipschitz constant $\Vert Q \Vert$.
\end{lemma}

\paragraph{Exercise (ii)} (2P) Prove that the quadratic function
\begin{equation}
  f(x) = \frac{1}{2} x^T Q x +b^T x + c
\end{equation}
is \textbf{smooth} with parameter $\Vert Q \Vert$. Note that without loss of generality $Q$ can be assumed to symmetric, however, this should not change anything.

\paragraph{Exercise (iii)} (1P) Suppose that we have observations $(x_i, y_i)$ which are \textbf{centered}, meaning that $\sum_{i=1}^{n}x_i = 0 = \sum_{i=1}^{n}y_i$. Let $(b^*, w^*)$ be the global minimum of the least squares objective
\begin{equation}
  f(b, w) = \sum_{i=1}^{n} {(b + w^T x_i - y_i)}^2.
\end{equation}
Prove that $b^*=0$.

\end{document}


\chapter{Example: a Simple Parity Checker}\label{parity}

This chapter consists of a worked example: the specification and
verification of a simple sequential parity checker.  The intention is
to accomplish two things:

\begin{myenumerate}
\item To present a complete piece of work with \HOL.
\item To give a flavour of what it is like to use the \HOL\ system for
  a tricky proof.
\end{myenumerate}

Concerning (ii), note that although the theorems proved are, in fact,
rather simple, the way they are proved illustrates the kind of
intricate `proof engineering' that is typical.  The proofs could be
done more elegantly, but presenting them that way would defeat the
purpose of illustrating various features of \HOL. It is hoped that the
small example here will give the reader a feel for what it is like to
do a big one.

Readers who are not interested in hardware verification should be able
to learn something about the \HOL{} system even if they do not wish to
penetrate the details of the parity-checking example used here.  The
specification and verification of a slightly more complex parity
checker is set as an exercise (a solution is provided).

\section{Introduction}

The sessions of this example comprise the specification and
verification of a device that computes the parity of a sequence of
bits.  More specifically, a detailed verification is given of a device
with an input {\small\verb|in|}, an output {\small\verb|out|} and the
specification that the $n$th output on {\small\verb|out|} is
{\small\verb|T|} if and only if there have been an even number of
{\small\verb|T|}'s input on {\small\verb|in|}. A theory named
{\small\verb|PARITY|} is constructed; this contains the specification
and verification of the device. All the \ML{} input in the boxes below
can be found in the file {\small\verb|examples/parity/PARITY.sml|}. It
is suggested that the reader interactively input this to get a `hands
on' feel for the example. The goal of the case study is to illustrate
detailed `proof hacking' on a small and fairly simple example.

\section{Specification}
\label{example}
The first step is to start up the \HOL{} system.  We again use
\texttt{<holdir>/bin/hol}.  The \ML{} prompt is {\small\verb|-|}, so
lines beginning with {\small\verb|-|} are typed by the user and other
lines are the system's response.

To specify the device, a primitive recursive function
{\small\verb|PARITY|} is defined so that for $n>0$, {\small\tt PARITY}
$n f$ is true if the number of {\small\verb|T|}'s in the sequence
$f${\small\tt (}$1${\small\tt)}, $\ldots$ , $f${\small\tt
  (}$n${\small\tt)} is even.

\setcounter{sessioncount}{0}
\begin{session}
\begin{verbatim}
- val PARITY_def = Define`
    (PARITY 0 f = T) /\
    (PARITY(SUC n) f = if f(SUC n) then ~(PARITY n f) else PARITY n f)`;
Definition has been stored under "PARITY_def".
> val PARITY_def =
    |- (!f. PARITY 0 f = T) /\
       !n f. PARITY (SUC n) f =
             (if f (SUC n) then ~PARITY n f else PARITY n f)
    : thm
\end{verbatim}
\end{session}

\noindent

The effect of our call to {\small\verb|Define|} is to store the
definition of {\small\verb|PARITY|} on the current theory with name
{\small\verb|PARITY_def|} and to bind the defining theorem to the \ML\
variable with the same name.  Notice that there are two name spaces
being written into: the names of constants in theories and the names
of variables in \ML.  The user is generally free to manage these names
however he or she wishes (subject to the various lexical
requirements), but a common convention is (as here) to give the
definition of a constant {\small\tt CON} the name
{\small\verb|CON_def|} in the theory and also in \ML.  Another
commonly-used convention is to use just {\small\verb|CON|} for the
theory and \ML{} name of the definition of a constant
{\small\verb|CON|}.  Unfortunately, the \HOL{} system does not use a
uniform convention, but users are recommended to adopt one.  In this
case \ml{Define} has made one of the choices for us, but there are
other scenarios where we have to choose the name used in the theory
file.

The specification of the parity checking device can now be given as:

{\small\begin{verbatim}
   !t. out t = PARITY t inp
\end{verbatim}}

\noindent
It is {\it intuitively\/} clear that this specification will be
satisfied if the signal\footnote{Signals are modelled as functions
  from numbers, representing times, to booleans.}  functions
{\small\verb|inp|} and {\small\verb|out|} satisfy\footnote{We'd like
  to use \ml{in} as one of our variable names, but this is a reserved
  word for \ml{let}-expressions.}:

{\small\begin{verbatim}
   out(0) = T
\end{verbatim}}

\noindent and

{\small\begin{verbatim}
   !t. out(t+1)  =  (if inp(t+1) then ~(out t) else out t)
\end{verbatim}}

\noindent This can be verified formally in \HOL{} by proving the
following lemma:

{\small\begin{verbatim}
   !inp out.
    (out 0 = T) /\ (!t. out(SUC t) = (if inp(SUC t) then ~(out t) else out t))
    ==>
    (!t. out t = PARITY t inp)
\end{verbatim}}

\noindent The proof of this is done by Mathematical Induction and, although
trivial, is a good illustration of how such proofs are done.  The
lemma is proved interactively using \HOL's subgoal package.  The proof
is started by putting the goal to be proved on a goal stack using the
function {\small\verb|g|} which takes a goal as argument.

\begin{session}
\begin{verbatim}
- g `!inp out.
        (out 0 = T) /\
        (!t. out(SUC t) = (if inp(SUC t) then ~(out t) else out t)) ==>
        (!t. out t = PARITY t inp)`;
> val it =
    Proof manager status: 1 proof.
    1. Incomplete:
         Initial goal:
         !inp out.
           (out 0 = T) /\
           (!t. out (SUC t) = (if inp (SUC t) then ~out t else out t)) ==>
           !t. out t = PARITY t inp
\end{verbatim}
\end{session}

\noindent The subgoal package prints out the goal on the top of the goal stack.
The top goal is expanded by stripping off the universal quantifier
(with {\small\verb|GEN_TAC|}) and then making the two conjuncts of the
antecedent of the implication into assumptions of the goal (with
{\small\verb|STRIP_TAC|}).  The \ML{} function {\small\verb|expand|}
takes a tactic and applies it to the top goal; the resulting subgoals
are pushed on to the goal stack.  The message `{\small\verb|OK..|}' is
printed out just before the tactic is applied.  The resulting subgoal
is then printed.

\begin{session}
\begin{verbatim}
- expand(REPEAT GEN_TAC THEN STRIP_TAC);
OK..
1 subgoal:
> val it =
    !t. out t = PARITY t inp
    ------------------------------------
      0.  out 0 = T
      1.  !t. out (SUC t) = (if inp (SUC t) then ~out t else out t)
\end{verbatim}
\end{session}

\noindent Next induction on {\small\verb|t|} is done
using {\small\verb|Induct|}, which does
induction on the outermost universally quantified variable.

\begin{session}
\begin{verbatim}
- expand Induct;
OK..
2 subgoals:
> val it =
    out (SUC t) = PARITY (SUC t) inp
    ------------------------------------
      0.  out 0 = T
      1.  !t. out (SUC t) = (if inp (SUC t) then ~out t else out t)
      2.  out t = PARITY t inp

    out 0 = PARITY 0 inp
    ------------------------------------
      0.  out 0 = T
      1.  !t. out (SUC t) = (if inp (SUC t) then ~out t else out t)
\end{verbatim}
\end{session}

\noindent The assumptions of the two subgoals
are shown numbered underneath the horizontal lines of hyphens. The
last goal printed is the one on the top of the stack, which is the
basis case. This is solved by rewriting with its assumptions and the
definition of {\small\verb|PARITY|}.

\begin{session}
\begin{verbatim}
- expand(ASM_REWRITE_TAC[PARITY_def]);
OK..

Goal proved.
 [.] |- out 0 = PARITY 0 inp

Remaining subgoals:
> val it =
    out (SUC t) = PARITY (SUC t) inp
    ------------------------------------
      0.  out 0 = T
      1.  !t. out (SUC t) = (if inp (SUC t) then ~out t else out t)
      2.  out t = PARITY t inp
\end{verbatim}
\end{session}

The top goal is proved, so the system pops it from the goal stack (and
puts the proved theorem on a stack of theorems). The new top goal is
the step case of the induction. This goal is also solved by rewriting.

\begin{session}
\begin{verbatim}
- expand(ASM_REWRITE_TAC[PARITY_def]);
OK..

Goal proved.
 [..] |- out (SUC t) = PARITY (SUC t) inp

Goal proved.
 [..] |- !t. out t = PARITY t inp
> val it =
    Initial goal proved.
    |- !inp out.
         (out 0 = T) /\
         (!t. out (SUC t) = (if inp (SUC t) then ~out t else out t)) ==>
         !t. out t = PARITY t inp
\end{verbatim}
\end{session}

\noindent The goal is proved, \ie\ the empty list of subgoals is produced.
The system now applies the justification functions produced by the
tactics to the lists of theorems achieving the subgoals (starting with
the empty list).  These theorems are printed out in the order in which
they are generated (note that assumptions of theorems are printed as
dots).

The \ML{} function

{\small\begin{verbatim}
   top_thm : unit -> thm
\end{verbatim}}

\noindent
returns the theorem just proved (\ie\ on the top of the theorem stack)
in the current theory, and we bind this to the \ML{} name
\ml{UNIQUENESS\_LEMMA}.

\begin{session}
\begin{verbatim}
- val UNIQUENESS_LEMMA = top_thm();
> val UNIQUENESS_LEMMA =
    |- !inp out.
         (out 0 = T) /\
         (!t. out (SUC t) = (if inp (SUC t) then ~out t else out t)) ==>
         !t. out t = PARITY t inp
    : thm
\end{verbatim}
\end{session}

\section{Implementation}
\label{implementation}

The lemma just proved suggests that the parity checker can be
implemented by holding the parity value in a register and then
complementing the contents of the register whenever {\small\verb|T|}
is input. To make the implementation more interesting, it will be
assumed that registers `power up' storing {\small\verb|F|}. Thus the
output at time {\small\verb|0|} cannot be taken directly from a
register, because the output of the parity checker at time
{\small\verb|0|} is specified to be {\small\verb|T|}. Another tricky
thing to notice is that if {\small\verb|t>0|}, then the output of the
parity checker at time {\small\verb|t|} is a function of the input at
time {\small\verb|t|}. Thus there must be a combinational path from
the input to the output.

The schematic diagram below shows the design of
a device that is intended to implement this specification.
(The leftmost input to \ml{MUX} is the selector.)
This works by storing the parity of the sequence input so far in the
lower of the two registers.  Each time {\small\verb|T|} is input at
{\small\verb|in|}, this stored value is complemented. Registers are assumed to
`power up' in a state in which they are storing {\small\verb|F|}.  The second
register (connected to {\small\verb|ONE|}) initially outputs
 {\small\verb|F|} and
then outputs {\small\verb|T|} forever.  Its role is just to ensure that the
device
works during the first cycle by connecting the output {\small\verb|out|} to the
device {\small\verb|ONE|} via the lower multiplexer.  For all subsequent cycles
{\small\verb|out|} is connected to {\small\verb|l3|} and so either carries the
stored parity value (if the current input is {\small\verb|F|}) or the
complement of this value (if the current input is {\small\verb|T|}).

\begin{center}
%BEGIN IMAGE
\setlength{\unitlength}{5mm}
\begin{picture}(14,30)(0,0.5)
\put(8,20){\framebox(2,2){\small{\tt NOT}}}
\put(6,16){\framebox(6,2){\small{\tt MUX}}}
\put(2,16){\framebox(2,2){\small{\tt ONE}}}
\put(2,12){\framebox(2,2){\small{\tt REG}}}
\put(6,8){\framebox(6,2){\small{\tt MUX}}}
\put(8,4){\framebox(2,2){\small{\tt REG}}}

\puthrule(9,24){4}
\puthrule(3,15){8}
\puthrule(3,11){4}
\puthrule(7,7){2}
\puthrule(9,3){4}

\putvrule(3,11){1}
\putvrule(3,14){2}
\putvrule(7,2){5}
\putvrule(7,10){1}
\putvrule(7,18){8}
\putvrule(9,3){1}
\putvrule(9,6){2}
\putvrule(9,10){6}
\putvrule(9,18){2}
\putvrule(9,22){2}
\putvrule(11,10){5}
\putvrule(11,18){6}
\putvrule(13,3){21}

\put(6,26){\makebox(2,2){\small{\tt in}}}
\put(6,0){\makebox(2,2){\small{\tt out}}}
\put(9,18){\makebox(1.8,2){\small{\tt l1}}}
\put(13,18){\makebox(1.8,2){\small{\tt l2}}}
\put(9,12){\makebox(1.8,2){\small{\tt l3}}}
\put(11,12){\makebox(1.8,2){\small{\tt l4}}}
\put(4,11){\makebox(3,1){\small{\tt l5}}}

\put(10,23){\makebox(2,2){$\bullet$}}
\put(8,6){\makebox(2,2){$\bullet$}}
\put(2,14){\makebox(2,2){$\bullet$}}

\end{picture}
\setlength{\unitlength}{1mm}
%END IMAGE
%HEVEA \imageflush
\end{center}

The devices making up this schematic will be modelled with predicates
\cite{Why-HOL-paper}. For example, the predicate {\small\verb|ONE|} is true
of a signal {\small\verb|out|} if for all times {\small\verb|t|} the value of
{\small\verb|out|} is {\small\verb|T|}.

\begin{session}
\begin{verbatim}
- val ONE_def = Define `ONE(out:num->bool) = !t. out t = T`;
Definition stored under "ONE_def".
> val ONE_def = |- !out. ONE out = !t. out t = T : thm
\end{verbatim}
\end{session}

\noindent Note that, as discussed above, `{\small\verb|ONE_def|}'  is used both
as an \ML{} variable and as the name of the definition in the theory.
Note also how `{\small\verb|:num->bool|}' has been added to resolve
type ambiguities; without this (or some other type information) the
typechecker would not be able to infer that {\small\tt t} is to have
type {\small\tt num}.

The binary predicate {\small\verb|NOT|} is true of a pair of signals
{\small\verb|(inp,out)|} if the value of {\small\verb|out|} is always
the negation of the value of {\small\verb|inp|}. Inverters are thus
modelled as having no delay. This is appropriate for a
register-transfer level model, but not at a lower level.

\begin{session}
\begin{verbatim}
- val NOT_def = Define`NOT(inp, out:num->bool) = !t. out t = ~(inp t)`;
Definition stored under "NOT_def".
> val NOT_def = |- !inp out. NOT (inp,out) = !t. out t = ~inp t : Thm.thm
\end{verbatim}
\end{session}

\noindent The final combinational device needed is a multiplexer.
This is a `hardware conditional'; the input
{\small\verb|sw|} selects which of the other
two inputs are to be connected to the output {\small\verb|out|}.

\begin{session}
\begin{verbatim}
- val MUX_def = Define`
    MUX(sw,in1,in2,out:num->bool) =
      !t. out t = if sw t then in1 t else in2 t`;
Definition stored under "MUX_def".
> val MUX_def =
    |- !sw in1 in2 out.
         MUX (sw,in1,in2,out) = !t. out t = (if sw t then in1 t else in2 t)
    : thm
\end{verbatim}
\end{session}

The remaining devices in the schematic are registers.  These are
unit-delay elements; the values output at time {\small\verb|t+1|} are
the values input at the preceding time {\small\verb|t|}, except at
time {\small\verb|0|} when the register outputs
{\small\verb|F|}.\footnote{Time {\tt {\small 0}} represents when the
  device is switched on.}

\begin{session}
\begin{verbatim}
- val REG_def =
    Define `REG(inp,out:num->bool) =
              !t. out t = if (t=0) then F else inp(t-1)`;
Definition stored under "REG_def".
> val REG_def =
    |- !inp out. REG (inp,out) = !t. out t =
                 (if t = 0 then F else inp (t - 1))
    : thm
\end{verbatim}
\end{session}

The schematic diagram above can be represented as a predicate by
conjoining the relations holding between the various
signals and then existentially quantifying the internal lines.
This technique is explained elsewhere
(\eg\ see \cite{Camilleri-et-al,Why-HOL-paper}).

\begin{session}
\begin{verbatim}
- val PARITY_IMP_def = Define
   `PARITY_IMP(inp,out) =
      ?l1 l2 l3 l4 l5.
        NOT(l2,l1) /\ MUX(inp,l1,l2,l3) /\ REG(out,l2) /\
        ONE l4     /\ REG(l4,l5)        /\ MUX(l5,l3,l4,out)`;
Definition stored under "PARITY_IMP_def".
> val PARITY_IMP_def =
    |- !inp out.
         PARITY_IMP (inp,out) =
         ?l1 l2 l3 l4 l5.
           NOT (l2,l1) /\ MUX (inp,l1,l2,l3) /\ REG (out,l2) /\ ONE l4 /\
           REG (l4,l5) /\ MUX (l5,l3,l4,out)
    : thm
\end{verbatim}
\end{session}\label{parity-imp}

\section{Verification}

The following theorem will eventually be proved:
{\small\begin{verbatim}
   |- !inp out. PARITY_IMP(inp,out) ==> (!t. out t = PARITY t inp)
\end{verbatim}}
This states that {\it if\/} {\small\verb|inp|} and {\small\verb|out|}
are related as in the schematic
diagram (\ie\ as in the definition of {\small\verb|PARITY_IMP|}),
{\it then\/} the
pair of signals {\small\verb|(inp,out)|} satisfies the specification.

First, the following lemma is proved; the correctness of the parity
checker follows from this and {\small\verb|UNIQUENESS_LEMMA|} by the
transitivity of {\small{\tt\verb+==>+}}.

\begin{session}
\begin{verbatim}
- g `!inp out.
        PARITY_IMP(inp,out) ==>
        (out 0 = T) /\
        !t. out(SUC t) = if inp(SUC t) then ~(out t) else out t`;
> val it =
    Proof manager status: 2 proofs.
    2. Completed: ...
    1. Incomplete:
         Initial goal:
         !inp out.
           PARITY_IMP (inp,out) ==>
           (out 0 = T) /\
           !t. out (SUC t) = (if inp (SUC t) then ~out t else out t)
\end{verbatim}
\end{session}

The first step in proving this goal is to rewrite with definitions
followed by a decomposition of the resulting goal using
{\small\verb|STRIP_TAC|}. The rewriting tactic
{\small\verb|PURE_REWRITE_TAC|} is used; this does no built-in
simplifications, only the ones explicitly given in the list of
theorems supplied as an argument.  One of the built-in simplifications
used by {\small\verb|REWRITE_TAC|} is {\small\tt |-~(x~=~T)~=~x}.
{\small\verb|PURE_REWRITE_TAC|} is used to prevent rewriting with this
being done.
\begin{session}
\begin{verbatim}
- expand(PURE_REWRITE_TAC
           [PARITY_IMP_def, ONE_def, NOT_def, MUX_def, REG_def] THEN
         REPEAT STRIP_TAC);
OK..
2 subgoals:
> val it =
    out (SUC t) = (if inp (SUC t) then ~out t else out t)
    ------------------------------------
      0.  !t. l1 t = ~l2 t
      1.  !t. l3 t = (if inp t then l1 t else l2 t)
      2.  !t. l2 t = (if t = 0 then F else out (t - 1))
      3.  !t. l4 t = T
      4.  !t. l5 t = (if t = 0 then F else l4 (t - 1))
      5.  !t. out t = (if l5 t then l3 t else l4 t)

    out 0 = T
    ------------------------------------
      0.  !t. l1 t = ~l2 t
      1.  !t. l3 t = (if inp t then l1 t else l2 t)
      2.  !t. l2 t = (if t = 0 then F else out (t - 1))
      3.  !t. l4 t = T
      4.  !t. l5 t = (if t = 0 then F else l4 (t - 1))
      5.  !t. out t = (if l5 t then l3 t else l4 t)
\end{verbatim}
\end{session}

The top goal is the one printed last; its conclusion is
{\small\verb|out 0 = T|} and its assumptions are equations relating
the values on the lines in the circuit.  The natural next step would
be to expand the top goal by rewriting with the assumptions.  However,
if this were done the system would go into an infinite loop because
the equations for {\small\verb|out|}, {\small\verb|l2|} and
{\small\verb|l3|} are mutually recursive.  Instead we use the
first-order reasoner {\small\verb|PROVE_TAC|} to do the work:

\begin{session}
\begin{verbatim}
- expand(PROVE_TAC []);
OK..
Meson search level: .....

Goal proved.
 [......] |- out 0 = T

Remaining subgoals:
> val it =
    out (SUC t) = (if inp (SUC t) then ~out t else out t)
    ------------------------------------
      0.  !t. l1 t = ~l2 t
      1.  !t. l3 t = (if inp t then l1 t else l2 t)
      2.  !t. l2 t = (if t = 0 then F else out (t - 1))
      3.  !t. l4 t = T
      4.  !t. l5 t = (if t = 0 then F else l4 (t - 1))
      5.  !t. out t = (if l5 t then l3 t else l4 t)
\end{verbatim}
\end{session}
The first of the two subgoals is proved.  Inspecting the remaining
goal it can be seen that it will be solved if its left hand side,
{\small\verb|out(SUC t)|}, is expanded using the assumption:

{\small\begin{verbatim}
   !t. out t = if l5 t then l3 t else l4 t
\end{verbatim}}

    However, if this assumption is used for rewriting, then all the
    subterms of the form {\small\verb|out t|} will also be expanded.
    To prevent this, we really want to rewrite with a formula that is
    specifically about {\small\verb|out (SUC t)|}.  We want to somehow
    pull the assumption that we do have out of the list and rewrite
    with a specialised version of it.  We can do just this using
    {\small\verb|PAT_ASSUM|}.  This tactic is of type \ml{term -> thm
      -> tactic}.  It selects an assumption that is of the form given
    by its term argument, and passes it to the second argument, a
    function which expects a theorem and returns a tactic.  Here it is
    in action:

\begin{session}
\begin{verbatim}
- e (PAT_ASSUM ``!t. out t = X t``
       (fn th => REWRITE_TAC [SPEC ``SUC t`` th]));
<<HOL message: inventing new type variable names: 'a, 'b.>>
OK..
1 subgoal:
> val it =
    (if l5 (SUC t) then l3 (SUC t) else l4 (SUC t)) =
    (if inp (SUC t) then ~out t else out t)
    ------------------------------------
      0.  !t. l1 t = ~l2 t
      1.  !t. l3 t = (if inp t then l1 t else l2 t)
      2.  !t. l2 t = (if t = 0 then F else out (t - 1))
      3.  !t. l4 t = T
      4.  !t. l5 t = (if t = 0 then F else l4 (t - 1))
\end{verbatim}
\end{session}
The pattern used here exploited something called \emph{higher order
  matching}. The actual assumption that was taken off the assumption
stack did not have a RHS that looked like the application of a
function (\ml{X} in the pattern) to the \ml{t} parameter, but the RHS
could nonetheless be seen as equal to the application of \emph{some}
function to the \ml{t} parameter.  In fact, the value that matched
\ml{X} was {\small\verb|``\x. if l5 x then l3 x else l4 x``|}.

Inspecting the goal above, it can be seen that the next step is to
unwind the equations for the remaining lines of the circuit.  We do
this using the \ml{arith\_ss} simpset that comes with \ml{bossLib} to
help with the arithmetic embodied by the subtractions and \ml{SUC}
terms.

\begin{session}
\begin{verbatim}
- e (RW_TAC arith_ss []);
OK..

Goal proved.
 [.....]
|- (if l5 (SUC t) then l3 (SUC t) else l4 (SUC t)) =
   (if inp (SUC t) then ~out t else out t)

Goal proved.
 [......] |- out (SUC t) = (if inp (SUC t) then ~out t else out t)
> val it =
    Initial goal proved.
    |- !inp out.
         PARITY_IMP (inp,out) ==>
         (out 0 = T) /\
         !t. out (SUC t) = (if inp (SUC t) then ~out t else out t)
\end{verbatim}
\end{session}

\noindent The theorem just proved is named
{\small\verb|PARITY_LEMMA|} and saved in the current theory.

\begin{session}
\begin{verbatim}
- val PARITY_LEMMA = top_thm ();
> val PARITY_LEMMA =
    |- !inp out.
         PARITY_IMP (inp,out) ==>
         (out 0 = T) /\
         !t. out (SUC t) = (if inp (SUC t) then ~out t else out t)
\end{verbatim}
\end{session}

{\small\verb|PARITY_LEMMA|} could have been proved in one step with a
single compound tactic.  Our initial goal can be expanded with a
single tactic corresponding to the sequence of tactics that were used
interactively:

\begin{session}
\begin{verbatim}
- restart()
> ...
- e (PURE_REWRITE_TAC [PARITY_IMP_def, ONE_def, NOT_def,
                       MUX_def, REG_def] THEN
     REPEAT STRIP_TAC THENL [
       PROVE_TAC [],
       PAT_ASSUM ``!t. out t = X t``
                 (fn th => REWRITE_TAC [SPEC ``SUC t`` th]) THEN
       RW_TAC arith_ss []
     ]);
<<HOL message: inventing new type variable names: 'a, 'b.>>
OK..
Meson search level: .....
> val it =
    Initial goal proved.
    |- !inp out.
         PARITY_IMP (inp,out) ==>
         (out 0 = T) /\
         !t. out (SUC t) = (if inp (SUC t) then ~out t else out t)
\end{verbatim}
\end{session}

Armed with {\small\verb|PARITY_LEMMA|}, the final theorem is easily
proved.  This will be done in one step using the \ML{} function
{\small\verb|prove|}.

\begin{session}
\begin{verbatim}
- val PARITY_CORRECT = prove(
    ``!inp out. PARITY_IMP(inp,out) ==> (!t. out t = PARITY t inp)``,
    REPEAT STRIP_TAC THEN MATCH_MP_TAC UNIQUENESS_LEMMA THEN
    MATCH_MP_TAC PARITY_LEMMA THEN ASM_REWRITE_TAC []);
> val PARITY_CORRECT =
    |- !inp out. PARITY_IMP (inp,out) ==> !t. out t = PARITY t inp
\end{verbatim}
\end{session}

\noindent This completes the proof of the
parity checking device.

\section{Exercises}
\label{exercises}

Two exercises are given in this section: Exercise~1 is
straightforward, but Exercise~2 is quite tricky and might take a
beginner several days to solve.

\subsection{Exercise 1}

Using {\it only\/} the devices {\small\verb|ONE|}, {\small\verb|NOT|},
{\small\verb|MUX|} and {\small\verb|REG|} defined in
Section~\ref{implementation}, design and verify a register
{\small\verb|RESET_REG|} with an input {\small\verb|in|}, reset line
{\small\verb|reset|}, output {\small\verb|out|} and behaviour
specified as follows.
\begin{itemize}
\item If {\small\verb|reset|} is {\small\verb|T|} at time
  {\small\verb|t|}, then the value at {\small\verb|out|} at time
  {\small\verb|t|} is also {\small\verb|T|}.
\item If {\small\verb|reset|} is {\small\verb|T|} at time
  {\small\verb|t|} or {\small\verb|t+1|}, then the value output at
  {\small\verb|out|} at time {\small\verb|t+1|} is {\small\verb|T|},
  otherwise it is equal to the value input at time {\small\verb|t|} on
  {\small\verb|inp|}.
\end{itemize}
This is formalized in \HOL{} by the definition:

{\small\begin{verbatim}
   RESET_REG(reset,inp,out) =
    (!t. reset t ==> (out t = T)) /\
    (!t. out(t+1) = ((reset t  \/ reset(t+1)) => T | inp t))
\end{verbatim}}

\noindent Note that this specification is only partial; it doesn't specify the
output at time {\small\verb|0|} in the case that there is no reset.

The solution to the exercise should be a definition of a predicate
{\small\verb|RESET_REG_IMP|} as an existential quantification of a
conjunction of applications of {\small\verb|ONE|}, {\small\verb|NOT|},
{\small\verb|MUX|} and {\small\verb|REG|} to suitable line
names,\footnote{i.e.  a definition of the same form as that of
  {\small\tt PARITY\_IMP}
%BEGIN LATEX
on page~\pageref{parity-imp}.
%END LATEX
%HEVEA in section~\ref{parity-imp}
} together with a proof of:

{\small\begin{verbatim}
   RESET_REG_IMP(reset,inp,out) ==> RESET_REG(reset,inp,out)
\end{verbatim}}

\subsection{Exercise 2}

\begin{enumerate}
\item Formally specify a resetable parity checker that has two boolean
  inputs {\small\tt reset} and {\small\tt inp}, and one boolean output
  {\small\tt out} with the following behaviour:
  \begin{quote}
    The value at {\small\tt out} is {\small\tt T} if and only if there
    have been an even number of {\small\tt T}s input at {\small\tt inp}
    since the last time that {\small\tt T} was input at {\small\tt
      reset}.
  \end{quote}
\item Design an implementation of this specification built using {\it
    only\/} the devices {\small\verb|ONE|}, {\small\verb|NOT|},
  {\small\verb|MUX|} and {\small\verb|REG|} defined in
  Section~\ref{implementation}.
\item Verify the correctness of your implementation in \HOL.
\end{enumerate}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "tutorial"
%%% End: