\documentclass{article}
\usepackage[utf8x]{inputenc} % codifica scrittura
\usepackage[nochapters]{classicthesis} % nochapters

\usepackage[T1]{fontenc}
\usepackage[square,numbers]{natbib}
\usepackage{amsmath, amsthm, amssymb, amsfonts, tikz}
\usetikzlibrary{snakes}
\usepackage{titletoc}
\usepackage{verbatim}
\usepackage{hyperref}
\usepackage{boxedminipage}

\titlecontents{section}[3em]{}{\contentslabel{1em}}{}{\titlerule*[1.5pc]{.}\contentspage}
\titlecontents{subsection}[6em]{}{\contentslabel{2em}}{}{\titlerule*[1.5pc]{.}\contentspage}

\begin{document}

\title{\rmfamily\normalfont\spacedallcaps{Computer Aided Graphic
    Design course exercises}}

\author{\spacedlowsmallcaps{Massimo Nocentini}}
\date{\today}

\maketitle

\begin{abstract}
  This document contains some exercises and collects my work done during
  the CAGD course given by Prof. Alessandra Sestini and Prof. Costanza
  Conti at University of Florence.

  In particular, here we collect exercises requested by Prof.
  Costanza Conti about Bezier and BSplines curves. We implement numerical
  methods using Julia  language \cite{Julia} and everything (code for
  solving exercises and \TeX sources of this document) is under
  version control, available as open source Git repository
  \footnote{Hosted on \url{http://github.com/massimo-nocentini/cagd}},
  under \emph{MIT License}.
\end{abstract}

\tableofcontents

\newpage

\section{Bezier curves}

\subsection{Curve from simple set of control points}
In \autoref{fig:first-closed-curve} we report the very first Bezier
curve obtained using our implementation. The curve is obtained using control
points $(1,1), (3,4), (5,6),(7,8),(10,2),(1,1)$ in the given
order. This plot was the first test for our implementation of code
reported in Exercise 1 and requested in Exercise 2: it contains a
segmented curve in green which is the control polygon, and a Bezier
curve in red built using the given control points.
\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-one}
  \caption{Curve from simple set of control points}
  \label{fig:first-closed-curve}
\end{figure}

\subsection{Curve from parametric specification}
As required in Exercise 3, in \autoref{fig:curve-from-parametric-spec}
we report a Bezier curve matching the following parametric
specification:
\begin{displaymath}
  \left [  \begin{array}{c}
      x(u) \\
      y(u)
    \end{array} \right ] = \left [
    \begin{array}{c}
      1 + u + u^2 \\
      u^3
    \end{array} \right ]
\end{displaymath}
with $u\in[0,1]$. In order to find the control polygon we do simple
reductions with \emph{Maxima}:

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
a:v0*(1-t)^3 + v1*3*t*(1-t)^2 + v2*3*(t^2)*(1-t) + v3*t^3;
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o1) }
{t}^{3}\,v3+3\,\left( 1−t\right) \,{t}^{2}\,v2+3\,{\left( 1−t\right) }^{2}\,t\,v1+{\left( 1−t\right) }^{3}\,v0
\end{math}

\noindent
%%%%%%%%%%%%%%%
% %%% INPUT:
% \begin{minipage}[t]{8ex}{\color{red}\bf
% \begin{verbatim}
% (%i2)
% \end{verbatim}}
% \end{minipage}
% \begin{minipage}[t]{\textwidth}{\color{blue}
\begin{verbatim}
b:ratsimp(a,t);
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o2) }
{t}^{3}\,\left( v3−3\,v2+3\,v1−v0\right) +{t}^{2}\,\left( 3\,v2−6\,v1+3\,v0\right) +t\,\left( 3\,v1−3\,v0\right) +v0
\end{math}

\noindent
%%% INPUT:
\begin{verbatim}
b = 1 + t + t^2 + 0*t^3;
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o3) }
{t}^{3}\,\left( v3−3\,v2+3\,v1−v0\right) +{t}^{2}\,\left( 3\,v2−6\,v1+3\,v0\right) +t\,\left( 3\,v1−3\,v0\right) +v0={t}^{2}+t+1
\end{math}

\noindent
%%% INPUT:
\begin{verbatim}
solve([coeff(b, t,3) = 0, coeff(b, t,2) = 1, coeff(b, t,1) = 1,
        coeff(b, t,0) = 1], [v0,v1,v2,v3]);
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o4) }
[[v0=1,v1=\frac{4}{3},v2=2,v3=3]]
\end{math}
%%%%%%%%%%%%%%%

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
a:y0*(1-t)^3 + y1*3*t*(1-t)^2 + y2*3*(t^2)*(1-t) + y3*t^3;
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o5) }
{t}^{3}\,y3+3\,\left( 1−t\right) \,{t}^{2}\,y2+3\,{\left( 1−t\right) }^{2}\,t\,y1+{\left( 1−t\right) }^{3}\,y0
\end{math}
%%%%%%%%%%%%%%%

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
b:ratsimp(a,t);
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o6) }
{t}^{3}\,\left( y3−3\,y2+3\,y1−y0\right) +{t}^{2}\,\left( 3\,y2−6\,y1+3\,y0\right) +t\,\left( 3\,y1−3\,y0\right) +y0
\end{math}
%%%%%%%%%%%%%%%

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
b = t^3;
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o7) }
{t}^{3}\,\left( y3−3\,y2+3\,y1−y0\right) +{t}^{2}\,\left( 3\,y2−6\,y1+3\,y0\right) +t\,\left( 3\,y1−3\,y0\right) +y0={t}^{3}
\end{math}
%%%%%%%%%%%%%%%

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
solve([coeff(b, t,3) = 1, coeff(b, t,2) = 0, coeff(b, t,1) = 0,
        coeff(b, t,0) = 0], [y0,y1,y2,y3]);
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}

\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o8) }
[[y0=0,y1=0,y2=0,y3=1]]
\end{math}
%%%%%%%%%%%%%%%

Hence four control points are $\{(1,0), (\frac{3}{4},0), (2,0),
(3,1)\}$.
\begin{figure}
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-two}
  \caption{Curve from parametric specification}
  \label{fig:curve-from-parametric-spec}
\end{figure}

\subsection{Splitted curve on a given parameter $\hat{t}$}
As required in Exercise 4, in \autoref{fig:splitted-curve} we report
the control polygon for the original curve in red, and two sets of
control points with the same cardinality that, used together, build a
Bezier that is the same as the original one. Those sets of points are
obtained via subdivision algorithm (which is a clever implementation of
classic \emph{de Casteljau algorithm}) fixing parameter $\hat{t} =
\frac{1}{4}$: they are colored in green (the relative Bezier in magenta)
and in blue (the relative Bezier in cyan), respectively.
\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-four}
  \caption{Splitted curve}
  \label{fig:splitted-curve}
\end{figure}

\subsection{Repeating the same control point more times}
As required in Exercise 5, in
\autoref{fig:repeating-same-control-point} we report two Bezier curves: the
red one relative to control points $\{(2,4), (6,12), (10,1),
(12,12)\}$ the green one relative to control points $\{(2,4), (6,12),
(10,1), (10,1), (10,1), (10,1), (12,12)\}$, ie. with the point
$(10,1)$ repeated three more times. We see that curve relative to
the augmented control polygon goes down toward $(10,1)$ more than the
other curve: this can be explained from a probabilistic point of
view, since a Bezier curve can be thought as a \emph{mean} of the control
polygon, hence repeating point $(10,1)$ increases its weight.

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-five}
  \caption{Repeating the control point $(10,1)$ three more times}
  \label{fig:repeating-same-control-point}
\end{figure}

\subsection{Increasing degree}
As required in Exercise 6, we start from an original set of control
points, plotted in \autoref{fig:increasing-degree-original-curve}, and
we proceed by increasing degrees of successive Bezier curves three times.
We obtains three augmented set of control points, plotted in
\autoref{fig:some-increased-degrees}, respectively. It is possible to
check the slow convergence for the sequence of polygons to the Bezier
curve and, in \autoref{fig:increasing-degree-does-change-curve}, the
Bezier curves relative to each augmented polygons doesn't change in
shape, ie their set of points equal the original one (we simply plot
those Bezier curves in the same plot and each one is over the others).

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-six-original}
  \caption{Original curve before increasing degree}
  \label{fig:increasing-degree-original-curve}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-six-higher-degree-control-poly}
  \caption{Some control polygons, each one with one more degree}
  \label{fig:some-increased-degrees}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-six-one-more-degree-comparison}
  \caption{Increasing degree doesn't change the Bezier shape}
  \label{fig:increasing-degree-does-change-curve}
\end{figure}

\subsection{Joining curve requiring $\mathcal{C}^0, \mathcal{C}^1, \mathcal{C}^2$}

In this section we report plots about Bezier spline curves,
extending a fixed control net, requiring $\mathcal{C}^0,
\mathcal{C}^1, \mathcal{C}^2$ for each extension, both on the right
(see
\autoref{fig:bezier-spline-right-extension-continuity},
\autoref{fig:bezier-spline-right-extension-tangent},
\autoref{fig:bezier-spline-right-extension-obsculating},
respectively) both on the left (see
\autoref{fig:bezier-spline-left-extension-continuity},
\autoref{fig:bezier-spline-left-extension-tangent},
\autoref{fig:bezier-spline-left-extension-obsculating},
respectively).

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-continuity}
  \caption{Bezier spline curve, requiring $\mathcal{C}^0$, extending on the right}
  \label{fig:bezier-spline-right-extension-continuity}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-tangent}
  \caption{Bezier spline curve, requiring $\mathcal{C}^1$, extending on the right}
  \label{fig:bezier-spline-right-extension-tangent}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-obsculating}
  \caption{Bezier spline curve, requiring $\mathcal{C}^2$, extending on the right}
  \label{fig:bezier-spline-right-extension-obsculating}
\end{figure}

% \includegraphics{bezier-deCasteljau-curves/exercise-seven-a_succ_i}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-continuity-left}
  \caption{Bezier spline curve, requiring $\mathcal{C}^0$, extending on the left}
  \label{fig:bezier-spline-left-extension-continuity}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-tangent-left}
  \caption{Bezier spline curve, requiring $\mathcal{C}^1$, extending on the left}
  \label{fig:bezier-spline-left-extension-tangent}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-obsculating-left}
  \caption{Bezier spline curve, requiring $\mathcal{C}^2$, extending on the left}
  \label{fig:bezier-spline-left-extension-obsculating}
\end{figure}

% \includegraphics{bezier-deCasteljau-curves/exercise-seven-a_i-left}

\newpage

\subsection{First Polar of a Bezier curve}

In this section we elaborate an extra exercise relative to first derivative
of a Bezier curve and the first step of de Casteljau algorithm.

Let $\mathbf{p}_{1}(t)$ a Bezier curve built over a control net with $n$ points
$\mathbf{b}_{0}^{(1)},\ldots,\mathbf{b}_{n-1}^{(1)}$, respect a fixed parameter $\hat{t}$
after one step of de Casteljau algorithm. Just use the formal definition:
\begin{displaymath}
    \begin{split}
        \mathbf{p}_{1,\hat{t}}(t)  &= \sum_{i=0}^{n-1}{\mathbf{b}_{i}^{(1)}(\hat{t})B_{i}^{n-1}(t)} \\
            &=  \sum_{i=0}^{n-1}{\left( (1-\hat{t})\mathbf{b}_{i}^{(0)} +
                \hat{t}\mathbf{b}_{i+1}^{(0)}\right)B_{i}^{n-1}(t)} \\
            &=  \sum_{i=0}^{n-1}{\left( (1-\hat{t})\mathbf{b}_{i}^{(0)} +
                \hat{t}\mathbf{b}_{i+1}^{(0)} -\left( (1-t)\mathbf{b}_{i}^{(0)} +
                t\mathbf{b}_{i+1}^{(0)}\right)\right)B_{i}^{n-1}(t)} +
                \sum_{i=0}^{n-1}{\mathbf{b}_{i}^{(1)}(t)B_{i}^{n-1}(t)} \\
            &=  \sum_{i=0}^{n-1}{\left( (t-\hat{t})\mathbf{b}_{i}^{(0)} +
                (\hat{t}-t)\mathbf{b}_{i+1}^{(0)}\right)B_{i}^{n-1}(t)} +
                \sum_{i=0}^{n-1}{\mathbf{b}_{i}^{(1)}(t)B_{i}^{n-1}(t)} \\
            &=  (\hat{t}-t)\sum_{i=0}^{n-1}{\left(
                \mathbf{b}_{i+1}^{(0)} - \mathbf{b}_{i}^{(0)}\right)B_{i}^{n-1}(t)} +
                \sum_{i=0}^{n-1}{\mathbf{b}_{i}^{(1)}(t)B_{i}^{n-1}(t)} \\
    \end{split}
\end{displaymath}
where, with little abuse of notation, $\mathbf{b}_{i}^{(1)}(t) =
\left( (1-t)\mathbf{b}_{i}^{(0)} + t\mathbf{b}_{i+1}^{(0)}\right)$.
Recall that $n\sum_{i=0}^{n-1}{\left(
\mathbf{b}_{i+1}^{(0)} - \mathbf{b}_{i}^{(0)}\right)B_{i}^{n-1}(t)}$ is the
first derivative of a Bezier curve, hence:
\begin{displaymath}
    \begin{split}
        \mathbf{p}_{1, \hat{t}}(t)  &= \mathbf{b}_{1, t}(t) +
        \frac{\hat{t}-t}{n}\frac{\partial \mathbf{b}_{0, \hat{t}}(t) }{\partial t}
    \end{split}
\end{displaymath}
So, this is called the ``first polar form'' of Bezier curve $\mathbf{b}_{0, \hat{t}}(t)$
respect parameter $\hat{t}$. Geometrically, the term $\mathbf{b}_{1, t}(t)$ is an affine
combination of points $\mathbf{b}_{0}^{(0)},\ldots,\mathbf{b}_{n}^{(0)}$ respect parameter
$t$ (not $\hat{t}$), hence it is a point also; the term
        $\frac{\hat{t}-t}{n}\frac{\partial \mathbf{b}_{0, \hat{t}}(t) }{\partial t} $
is a vector, so $\mathbf{p}_{1, \hat{t}}(t)$ is a vector applied to a point, yielding a point
as required. It is quite interesting how the polar form combine a point produced using $t$
as parameter with the derivative vector produced using  parameter $\hat{t}$. This derivation
comes from \cite{Farin}, page 73, and we report the output of our implementation in
\autoref{fig:exercise-polar-form}.

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-polar}
  \caption{First Polar form of a Bezier curve}
  \label{fig:exercise-polar-form}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newpage

\section{B-Splines curves}

\subsection{Mushrooms from clumped, uniformed and closed partitions}

In this section we report three simple BSpline curves, using
three different knots partitions: \emph{clumped, uniform} and
\emph{cyclic} (see \autoref{fig:clumpled-mushroom},
    \autoref{fig:uniformed-mushroom} and \autoref{fig:closed-mushroom}
respectively). The control polygon doesn't change
  and aims to produce a mushroom in cartoon style, each knot
has multiplicity $1$ (so maximum continuity is required in each junction)
and no control point is repeated.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-zero-clumped.eps}
  \caption{Mushroom from clumped knots partition}
  \label{fig:clumpled-mushroom}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-zero-uniformed.eps}
  \caption{Mushroom from uniformed knots partition}
  \label{fig:uniformed-mushroom}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-zero-closed.eps}
  \caption{Mushroom from cyclic knots partition}
  \label{fig:closed-mushroom}
\end{figure}

\subsection{Increasing order $k$ while decreasing \emph{continuity} vector }
As Exercise 1 requires, in \autoref{fig:bspline-exercise-one-clumped} we
report five BSpline curves, each one of them drawn against the same control
net. We modify for each curve its knots partition, formally for curves
$\mathbf{c}_{1},\mathbf{c}_{2},\mathbf{c}_{3},\mathbf{c}_{4}$ and $\mathbf{c}_{5}$
the following extended partitions are used:
\begin{displaymath}
    \begin{split}
        \Delta_{1} &= \lbrace 0,0, \frac{1}{5}, \frac{2}{5},
            \frac{3}{5},\frac{4}{5},1,1 \rbrace \\
        \Delta_{2} &= \lbrace 0,0,0, \frac{1}{4},
            \frac{1}{2},\frac{3}{4},1,1,1 \rbrace \\
        \Delta_{3} &= \lbrace 0,0,0,0,
            \frac{1}{3},\frac{2}{3},1,1,1,1 \rbrace \\
        \Delta_{4} &= \lbrace 0,0,0,0,0,\frac{1}{2},
            1,1,1,1,1 \rbrace \\
        \Delta_{5} &= \lbrace 0,0,0,0,0,0,1,1,1,1,1,1 \rbrace \\
    \end{split}
\end{displaymath}
respectively, and each curve $\mathbf{c}_{i}$ has \emph{degree} $i$. As
we can see, each knots partition is clumped, so the first and last control points
are interpolated and the sequence of curves uses partitions with knots $0$ and $1$
repeated one more time in successive partitions. For lower degree curves, such as
$\mathbf{c}_{1}$, they are quite next to control net, in particular $\mathbf{c}_{1}$
is the control net itself, while curve $\mathbf{c}_{2}$ is tangent to control net
in two segments since it has \emph{order} $3$ and for $t \in [\frac{1}{4}, \frac{1}{2}) =
[t_4, t_5)$ curve $\mathbf{c}_{2}$ has support $[t_2,t_3,t_4]$ where knot $0$
has multiplicity $2$, hence $\frac{\partial \mathbf{c}_{2}(t)}{\partial t}$
lies on direction given by $\mathbf{V}_{3} - \mathbf{V}_{2}$.
The same reasoning can be done for knot $1$.
Finally, the very last partition allow to draw a \emph{Bezier} curve (the yellow one).

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-one-clumped.eps}
  \caption{Decreasing the \emph{continuity} vector collapsing in a Bezier }
  \label{fig:bspline-exercise-one-clumped}
\end{figure}

\subsection{Knots' multiplicities against the same control polygon}
In \autoref{fig:bspline-exercise-two} we report plots required by Exercise 2.
This composite picture contains 5 curves according the following specifications,
from top to bottom:
\begin{center}
    \begin{tabular}{ c c }
        order & knots partition \\
        \hline
        4 & $\lbrace 0^{4},1,2,3,4,5,6,7,8,9,10 \rbrace$  \\
        4 & $\lbrace 0^{4},1^{2},2^{2},3^{2},4^{4} \rbrace$  \\
        6 & $\lbrace 0^{6},1,2,3,4,5^{6} \rbrace$  \\
        6 & $\lbrace 0^{6},1,2,3^{2},4^{6} \rbrace$  \\
        8 & $\lbrace 0^{8},1,2,3^{8} \rbrace$  \\
    \end{tabular}
\end{center}
The curve relative to the first row interpolate the first control point
since $0$ has multiplicity $4$ which is the curve's order, while the last
control point isn't interpolated since knots partition ends uniformly.

The curve relative to the second row both interpolate the first control
point due to the multiplicity $k$ of $0$ and is tangent to control net
on the direction $\mathbf{V}_{4} - \mathbf{V}_{3}$ since for $t \in [t_{5}, t_{6}]$
the support is $[t_{2},t_{3},t_{4},t_{5}]$, where $0$ has multiplicity $3 = order -1$.

Curves relative to third and forth rows both interpolate first and last control
points because of clumped partition, while the last one reduce to only two
knots $1,2$, toward Bezier representation.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-two.eps}
  \caption{Knots's multiplicities increased against the same control polygon }
  \label{fig:bspline-exercise-two}
\end{figure}

\subsection{Linearity near doubled control point with a clumped partition}
In \autoref{fig:bspline-exercise-three} we plot curve required for Exercise 3:
a quadratic curve ($order = 3$) with a doubled control point $\mathbf{V}_{3} =
\mathbf{V}_{4} = (1,1)$ over an extended partition
$\Delta = \lbrace 0,0,0, \frac{1}{4},\frac{1}{2},\frac{3}{4},
1,1,1 \rbrace$, a clumped one with no repeated knots.

Lets consider the subcurve $\mathbf{c}_{5}(t)$ for $t \in [t_{5}, t_{6}]$:
\begin{displaymath}
    \mathbf{c}_{5}(t) = \sum_{i=3}^{5}{\mathbf{V}_{i}N_{i,3}(t)}
\end{displaymath}
Since $\mathbf{V}_{4} = \mathbf{V}_{3}$, for $t \in [t_{5}, t_{6}]$,
$\mathbf{c}_{5}(t)$ must lie on the convex hull $\mathbf{V}_{3}, \mathbf{V}_{4},
\mathbf{V}_{5}$ and,  for $t \in [t_{4}, t_{5}]$, $\mathbf{c}_{5}(t)$ must lie
on the convex hull $\mathbf{V}_{2}, \mathbf{V}_{3}, \mathbf{V}_{4}$, it does follow
that $\mathbf{c}_{5}(t_{5})$ has to lie in their intersection, therefore vertex
$\mathbf{V}_{4}$, interpolating it.

Moreover, $\mathbf{c}_{5}(t)$ has two linear segment between
$\mathbf{V}_{2}, \mathbf{V}_{3}$ and $\mathbf{V}_{4}, \mathbf{V}_{5}$,
both of them join in $t_{4}, t_{6}$ with continuity $\mathcal{C}^{1}$.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-three.eps}
  \caption{Linearity toward $(1,1)$ due to ``linearized'' convex hull}
  \label{fig:bspline-exercise-three}
\end{figure}

\subsection{Two clumped partitions, different \emph{continuity} vectors
    and same control polygon}
In \autoref{fig:bspline-exercise-four} we report two curves required by Exercise 4.
Both of them have $order = 4$ and a control net with a doubled point
$\mathbf{V}_{3} = \mathbf{V}_{4}$, while the former (colored green)
is over a knots partition
$\Delta_{1} = \lbrace 0,0,0,0,\frac{1}{4}, \frac{3}{4}, 1,1,1,1 \rbrace$
(having simple knots, no repeated one), the latter (colored blue) is over
a knots partition
$\Delta_{2} = \lbrace 0,0,0,0,\frac{1}{2}, \frac{1}{2}, 1,1,1,1 \rbrace$ (having
knot $\frac{1}{2}$ doubled). Let define the following subcurves of the former:
\begin{displaymath}
    \begin{split}
        \mathbf{c}_{4}(t) &= \sum_{i=1}^{4}{\mathbf{V}_{i}N_{i,4}(t)}, \quad
            t \in [t_{4}, t_{5}] \\
        \mathbf{c}_{5}(t) &= \sum_{i=2}^{5}{\mathbf{V}_{i}N_{i,4}(t)}, \quad
            t \in [t_{5}, t_{6}] \\
        \mathbf{c}_{6}(t) &= \sum_{i=3}^{6}{\mathbf{V}_{i}N_{i,4}(t)}, \quad
            t \in [t_{6}, t_{7}] \\
    \end{split}
\end{displaymath}
As we saw during classes \cite{Conti}, $\mathbf{c}_{5}^{\prime}(t_{5}) =
\mathbf{c}_{4}^{\prime}(t_{5})$ holds, so the
curve interpolate a point on the segment with extrema $\mathbf{V}_{2}, \mathbf{V}_{3}$
and it is tangent to the control net in $t_{5}$. On the other hand,
$\mathbf{c}_{6}^{\prime}(t_{6}) = \mathbf{c}_{5}^{\prime}(t_{6})$, so the
curve interpolate a point on the segment with extrema $\mathbf{V}_{4}, \mathbf{V}_{5}$
and it is tangent to the control net in $t_{6}$.
Finally, observe that the curve doesn't
interpolate the doubled control point $\mathbf{V}_{3} = \mathbf{V}_{4}$.

The previous argument doesn't apply to the second curve because its
knots partition is not simple, it contains a double knot.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-four.eps}
  \caption{Same control polygon with doubled $(0,6)$ againts two clumped partitions}
  \label{fig:bspline-exercise-four}
\end{figure}

\subsection{Increasing clumped partitions for increasing occurrences of a control point}
In Exercise 5 it is required to provide three extended knots partitions in
order to draw cubic curves reported in \autoref{fig:bspline-exercise-five}.
Each curve is defined against the ``same'' control net, having the middle vertex
of the top row repeated one, two, three times respectively. Our choice of knots
partitions, each clumped since interpolation of first and last control points
is requested:
\begin{displaymath}
    \begin{split}
        \Delta_{1} &= \lbrace 0,0,0,0,\frac{1}{2},1,1,1,1 \rbrace \\
        \Delta_{2} &= \lbrace 0,0,0,0,\frac{1}{3},\frac{2}{3},1,1,1,1 \rbrace \\
        \Delta_{3} &= \lbrace 0,0,0,0,\frac{1}{4},\frac{1}{2},\frac{3}{4},1,1,1,1 \rbrace \\
    \end{split}
\end{displaymath}

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-five.eps}
  \caption{Three B-Splines, each for $1,2,3$ occurrences of $(0,1)$ respectively }
  \label{fig:bspline-exercise-five}
\end{figure}

\subsection{Two B-Splines from two closed partitions}
In this last section we report two closed curves having $order = 4$ over
two cyclic knots partitions, in \autoref{fig:bspline-exercise-six-first}
and \autoref{fig:bspline-exercise-six-second}, respectively.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-six-first-closed.eps}
  \caption{A first B-Spline from a closed partition }
  \label{fig:bspline-exercise-six-first}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-six-second-closed.eps}
  \caption{A second B-Spline from a closed partition }
  \label{fig:bspline-exercise-six-second}
\end{figure}

\newpage

\begin{thebibliography}{}

\bibitem{Julia} Open Source Project,
  \emph{Julia language}, \url{http://julialang.org/}

\bibitem{Farin} Gerald Farin,
  \textit{Curves and surfaces for CAGD, Fifth Edition}

\bibitem{Conti} Costanza Conti,
    \emph{Lecture notes}, distributed during classes

\end{thebibliography}

\end{document}


\input{../common/common.tex}

\title{Math notes - Cat vs dog}
\author{Uwe Hoffmann}
\hypersetup{colorlinks, pdftitle={Math notes - Cat vs dog}}

\begin{document}

\setcounter{chapter}{1}
\section*{Cat vs dog}

\newthought{Bipartite graphs}, network flows, matchings and vertex covers are the topics of the problem \footnote{\bibentry{spotify_interview}} in this note.\index{bipartite graph}\index{network flow}\index{bipartite matching}\index{vertex cover}

\begin{fullwidth}

\vspace{10 mm}
\begin{problem}
The latest reality show has hit the TV: “Cat vs. Dog”. In this show, a bunch of cats and dogs compete for the very prestigious Best Pet Ever title. In each episode, the cats and dogs get to show themselves off, after which the viewers vote on which pets should stay and which should be forced to leave the show.
\\

Each viewer gets to cast a vote on two things: one pet which should be kept on the show, and one pet which should be thrown out. Also, based on the universal fact that everyone is either a cat lover (i.e. a dog hater) or a dog lover (i.e. a cat hater), it has been decided that each vote must name exactly one cat and exactly one dog.
\\

Ingenious as they are, the producers have decided to use an advancement procedure which guarantees that as many viewers as possible will continue watching the show: the pets that get to stay will be chosen so as to maximize the number of viewers who get both their opinions satisfied. Calculate this maximum number of satisfied viewers.
\end{problem}

\end{fullwidth}

At first glance this looks similar to a SAT problem \footnote{Boolean satisfiability problem \url{http://en.wikipedia.org/wiki/Boolean_satisfiability_problem}}, something like $(c_1 \land \lnot d_3),\  (c_3 \land \lnot d_1),\  (d_2 \land \lnot c_2),\ \dots$ where $c_i$ are the cats and $d_j$ are the dogs. The goal would be to pick the biggest subset of boolean expressions (votes) that are satisfied.

\begin{marginfigure}[0.01in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            node distance = 3cm, % distance between nodes
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (v1) at (0,0) {$c_1 \land \lnot d_3$};
        \node[state] (v2) at (0,2) {$c_2 \land \lnot d_2$};
        \node[state] (v3) at (0,4) {$c_4 \land \lnot d_1$};
        \node[state] (v4) at (0,6) {$c_3 \land \lnot d_1$};
        \node[state] (v5) at (4,1) {$\lnot c_2 \land d_1$};
        \node[state] (v6) at (4,3) {$\lnot c_4 \land d_3$};
        \node[state] (v7) at (4,5) {$\lnot c_1 \land d_2$};

        \node [draw=blue,fit=(v1) (v4),label=above:Cat lovers] {};
        \node [draw=green,fit=(v5) (v7),label=above:Dog lovers] {};

        \path (v1) edge (v6);
        \path (v1) edge (v7);
        \path (v2) edge (v5);
        \path (v2) edge (v7);
        \path (v3) edge (v5);
        \path (v3) edge (v6);
        \path (v4) edge (v5);

    \end{tikzpicture}
    \caption{Votes form a bipartite graph. A graph is \textbf{bipartite} if the vertex set is partitioned into two subsets (blue and green in this case) such that no vertices in a subset are adjacent.}
	\label{votes_conflict}
\end{marginfigure}

But SAT is about one boolean expression and about assigning values to boolean variables to satisfy it. Seems like SAT is fundamentally different and not a good approach in solving this problem. What if we want to visualize the boolean expressions and see the relationships between them, i.e. which ones are in conflict. Conflict between two boolean expressions means one expression has $c_i$ and the other expression has $\lnot c_i$ or one has $d_j$ and the other $\lnot d_j$. A good way to do that is with a graph as in Figure \ref{votes_conflict}. The nodes in the graph are the boolean expressions and edges connect  boolean expressions that are in conflict.

It becomes apparent that the graph is bipartite with cat lovers on one side and dog lovers on the other. That is good because a lot of graph algorithms are much simpler and work faster if the graphs are bipartite. But what algorithm should we use? We need to find the biggest subset of nodes in the graph that are not in conflict.

Sometimes it's easier to compute the complement of what we want: the smallest subset of nodes that are involved in conflicts. Removing these nodes and the edges they touch should leave us with a graph with only nodes and no edges, i.e. only votes without conflicts. Because we strive to remove the smallest subset of conflicting nodes we are left with the biggest subset of votes without conflicts.

The subset of nodes that are involved in conflicts is a vertex cover \footnote{A \textbf{vertex cover} is a subset of nodes such that each edge in a graph is incident to at least one vertex in the subset.} for our bipartite graph.

We need to compute a minimum vertex cover. This will be a good excuse to learn about network flows in graphs, maximum flows and minimum cuts. This delightful detour will eventually bring us to maximum matchings\footnote{A \textbf{matching} is a subset of edges such that no two edges in the subset share a vertex.} and then finally to minimum vertex covers.\newline

We begin with \textbf{network flows} in graphs. We work with a directed graph $G=(V, E)$ that has two special vertices $s$ and $t$ called \textbf{source} and \textbf{target}.  No edge goes into \emph{source} and no edge comes out of \emph{target}. We also have a function $c:E \to \mathbb{R}_{\geq 0}$ that assigns a non-negative capacity to each edge. The graph $G$ together with source $s$ and target $t$ and capacity function $c$ form a \textbf{network} $(G=(V, E), s \in V, t \in V, c)$.

\begin{marginfigure}
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            node distance = 3cm, % distance between nodes
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (u) at (1,0) {$u$};
        \node[state] (v) at (3,0) {$v$};

        \path[->] (u) edge node {$10/20$} (v);

    \end{tikzpicture}
    \caption{In figures we annotate an edge with flow and capacity as shown here. In this case $f(u \rightarrow v) = 10$ and $c(u \rightarrow v) = 20$. If only one number is annotating the edge then it's the capacity.}
	\label{edge_flow_cap}
\end{marginfigure}

\begin{defn}\label{stflow}
A function $f:E \to \mathbb{R}_{\geq 0}$ is a \textbf{flow} through network $(G, s, t, c)$ if $f$ satisfies the following constraints:

\begin{itemize}
  \item \emph{capacity constraint}: flow along an edge cannot exceed the capacity of the edge
  $$
  \forall e \in E: f(e) \leq c(e)
  $$
  \item \emph{conservation constraint}: incoming flow into a vertex (except for source and target) equals outgoing flow from the vertex
  $$
  \forall v \in V \setminus \{s,t\}: \sum_u f(u \rightarrow v) = \sum_w f(v \rightarrow w)
  $$
\end{itemize}
\end{defn}

\begin{marginfigure}
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (s) at (0,1) {$s$};
        \node[state] (v) at (2,2) {$v$};
        \node[state] (u) at (2,0) {$u$};
        \node[state] (t) at (4,1) {$t$};

        \path[->, line width=1.5pt] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$0/10$} (v);
        \path[->, line width=1.5pt] (u) edge node {$20/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$0/10$} (t);
		\path[->, line width=1.5pt] (v) edge node {$20/20$} (t);

    \end{tikzpicture}
    \caption{Example network flow. Here $|f| = 20$ and the whole flow is pumped along the path $s \rightarrow u \rightarrow v \rightarrow t$. In this exampe $f$ \textbf{saturates} $s \rightarrow u$ and $v \rightarrow t$ and \textbf{avoids} $s \rightarrow v$ and $u \rightarrow t$.}
	\label{ex_flow}
\end{marginfigure}

\marginnote[0.5in]{For notational simplicity we assume functions $f$ and $c$ are defined on $V \times V$ and $f(u \rightarrow v) = c(u \rightarrow v) = 0$ if $u \rightarrow v$ is not an edge in $G=(V, E)$.}

Source $s$ generates flow and target $t$ consumes flow. The \textbf{value} of flow $f$, denoted $|f|$, is defined as

$$
|f| = \sum_w f(s \rightarrow w) = \sum_v f(v \rightarrow t)
$$

\begin{marginfigure}[0.3in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (s) at (0,1) {$s$};
        \node[state] (v) at (2,2) {$v$};
        \node[state] (u) at (2,0) {$u$};
        \node[state] (t) at (4,1) {$t$};

        \path[->] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$10/10$} (v);
        \path[->] (u) edge node {$10/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$10/10$} (t);
		\path[->] (v) edge node {$20/20$} (t);

    \end{tikzpicture}
    \caption{Same example network with a flow of value $|f| = 30$.}
	\label{ex_flow_max}
\end{marginfigure}

Given a network $(G, s, t, c)$ what is the maximum flow value that can be pumped through it? Figure \ref{ex_flow} shows a flow of value $20$ through an example network. It saturates the flow along one particular path and avoids the other edges. Is $20$ the maximum flow value that can be achieved for this example network? Figure \ref{ex_flow_max} shows the same network but now with a flow of value $30$. Can we do better than $30$? The answer is no, because that would exceed the outgoing capacity of source $s$ or the incoming capacity of $t$.

Our goal is to device an algorithm that constructs a flow with maximum value through a given network. To gauge the progress of our algorithm we need an upper bound for the maximum flow value. As said before the maximum value clearly cannot exceed the outgoing capacity of source $s$ or the incoming capacity of $t$. But more generally if we sever the ties between source and target along some subset of edges such that there are no more paths from source to target then the maximum flow value cannot exceed the capacity of the cut. This seems like a useful concept to formalize.

\begin{marginfigure}
\includegraphics[scale=0.15]{soviettrainnetwork.jpeg}
\end{marginfigure}
\marginnote{\bibentry{Schrijver02onthe}:\\
 Network flows and minimum cuts played a role in the Cold War. The figure is a schematic diagram of the railway network of the Western Soviet Union and Eastern European countries, with a maximum flow of value 163,000 tons from Russia to Eastern Europe, and a cut of capacity 163,000 tons indicated as “The bottleneck”.}

\begin{defn}\label{networkcut}
In a network $(G, s, t, c)$ a \textbf{cut} is a partition of the vertex set $V$ into two subsets $S$ and $T$, such that $V = S \cup T$, $S \cap T = \emptyset$ and $s \in S, t \in T$. The \textbf{capacity} of the cut $(S, T)$, denoted $\|S,T\|$, is defined as

$$
\|S, T\| = \sum_{v \in S} \sum_{w \in T} c(v \rightarrow w)
$$
\end{defn}

\begin{thm}\label{flowcut}
With network $(G, s, t, c)$, for any flow $f$ and any cut $(S, T)$ we have
$$
  |f| \leq \|S, T\|
$$
Furthermore equality holds if and only if $f$ saturates every edge from $S$ to $T$ and avoids every edge from $T$ to $S$.
\end{thm}

\begin{marginfigure}[1in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (s) at (0,1) {$s$};
        \node[state] (v) at (2,2) {$v$};
        \node[state] (u) at (2,0) {$u$};
        \node[state] (t) at (4,1) {$t$};

        \path[->, line width=1.5pt] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$0/10$} (v);
        \path[->, line width=1.5pt] (u) edge node {$20/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$0/10$} (t);
		\path[->, line width=1.5pt] (v) edge node {$20/20$} (t);

		\path[->, dashed] (s) edge[bend left=60] node {$10$} (v);
		\path[->, dashed] (v) edge[bend left=30] node {$20$} (u);
		\path[->, dashed] (u) edge[bend right=70] node[right] {$10$} (t);

    \end{tikzpicture}
    \caption{Dashed edges show how the greedy $s-t$ path flow can be augmented and reversed in order to increase overall flow.}
	\label{ex_flow_dashed}
\end{marginfigure}

\begin{proof}

\begin{align*}
|f| &= \sum_w f(s \rightarrow w) \tag*{\tiny{(by definition)}}\\
    &= \sum_w f(s \rightarrow w) - \sum_v f(v \rightarrow s) \tag*{\tiny{(second sum terms are all zero)}}\\
    &= \sum_{u \in S} (\sum_w f(u \rightarrow w) - \sum_v f(v \rightarrow u)) \tag*{\tiny{(flow conservation constraint)}}\\
    &= \sum_{u \in S} (\sum_{w \in T} f(u \rightarrow w) - \sum_{v \in T} f(v \rightarrow u)) \tag*{\tiny{(edges in $S$ cancel each other out)}}\\
    &\leq \sum_{u \in S} \sum_{w \in T} f(u \rightarrow w) \tag*{\tiny{(because $f(v \rightarrow u) \geq 0$)}}\\
    &\leq \sum_{u \in S} \sum_{w \in T} c(u \rightarrow w) \tag*{\tiny{(flow capacity constraint)}}\\
    &= \|S, T\| \tag*{\tiny{(by definition)}}
\end{align*}

\end{proof}

Theorem \ref{flowcut} tells us that if we keep increasing a flow and/or decreasing a cut we should eventually meet at a maximum flow that equals a minimum cut. But given a network how do we start? A first valid flow is $\forall e \in E: f(e) = 0$. We could then try a greedy strategy. Starting with source $s$ find the path to $t$ with the biggest capacity\footnote{The capacity of a path is the minimum over the capacities of the edges forming the path.} and pump as much flow as we can through it as illustrated in Figure \ref{ex_flow}. Unfortunately we are stuck at that point. We cannot pump more flow out of $s$ on $s \rightarrow v$ because that would violate flow conservation at $v$ (we are at the maximum outgoing flow at $v$). The dashed edges in Figure \ref{ex_flow_dashed} show some of our options. On edges where the current flow leaves residual capacity we can pump more and on edges where there is existing flow we can reverse it. Again, this concept seems worth formalizing.

\begin{marginfigure}
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node at (0,2.7) {(a)};
        \node at (0,0) {(b)};

        \node[state] (sf) at (0,1) {$s$};
        \node[state] (vf) at (2,2) {$v$};
        \node[state] (uf) at (2,0) {$u$};
        \node[state] (tf) at (4,1) {$t$};

        \node[state] (s) at (0,4) {$s$};
        \node[state] (v) at (2,5) {$v$};
        \node[state] (u) at (2,3) {$u$};
        \node[state] (t) at (4,4) {$t$};

        \path[->] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$0/10$} (v);
        \path[->] (u) edge node {$20/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$0/10$} (t);
		\path[->] (v) edge node {$20/20$} (t);

        \path[<-, blue] (sf) edge node[below left=.05cm] {$20$} (uf);
        \path[->, blue] (sf) edge node {$10$} (vf);
        \path[->, blue] (uf) edge[bend left=30] node {$80$} (vf);
        \path[<-, blue] (uf) edge[bend right=30] node {$20$} (vf);
		\path[->, blue] (uf) edge node[below right=.05cm] {$10$} (tf);
		\path[<-, blue] (vf) edge node {$20$} (tf);

    \end{tikzpicture}
    \caption{\\
             (a) Example network with flow from Figure \ref{ex_flow}. \\
             (b) Residual network (in blue) with edges annotated with their residual capacity.}
	\label{ex_residual}
\end{marginfigure}

\begin{defn}\label{residualgraph}
A flow $f$ in a network $(G, s, t, c)$ induces a \textbf{residual network} $(G_f, s, t, c_f)$ with \textbf{residual graph} $G_f$ and \textbf{residual capacity} $c_f$ in the following way:

\begin{itemize}
	\item all vertices from $G$ are vertices in $G_f$, also source $s$ and target $t$ are the same in $G$ and $G_f$
	\item if $f(u \rightarrow v) > 0$ then $G_f$ has an edge $(v \rightarrow u)$ with capacity
$$
c_f(v \rightarrow u) = f(u \rightarrow v)
$$
    \item if $f(u \rightarrow v) < c(u \rightarrow v)$ then $G_f$ has an edge $(u \rightarrow v)$ with capacity
$$
c_f(u \rightarrow v) = c(u \rightarrow v) - f(u \rightarrow v)
$$
\end{itemize}

\end{defn}

Figure \ref{ex_residual} shows the residual network of our example network and flow. We observe that there is a simple path\footnote{A simple path is a path where every vertex on the path is visited only once.} $s \rightarrow v \rightarrow u \rightarrow t$ with capacity $10$ from source $s$ to target $t$ in the residual graph. This path shows that there still is unused capacity for flow to be pushed from $s$ to $t$. A simple path from $s$ to $t$ in $G_f$ is called an \textbf{augmenting path}.

\begin{marginfigure}[1in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node at (0,2.7) {(a)};
        \node at (0,0) {(b)};

        \node[state] (sf) at (0,1) {$s$};
        \node[state] (vf) at (2,2) {$v$};
        \node[state] (uf) at (2,0) {$u$};
        \node[state] (tf) at (4,1) {$t$};

        \node[state] (s) at (0,4) {$s$};
        \node[state] (v) at (2,5) {$v$};
        \node[state] (u) at (2,3) {$u$};
        \node[state] (t) at (4,4) {$t$};

        \path[->] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$0/10$} (v);
        \path[->] (u) edge node {$20/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$0/10$} (t);
		\path[->] (v) edge node {$20/20$} (t);

		\path[->] (sf) edge node[below left=.05cm] {$20/20$} (uf);
        \path[->] (sf) edge node {$\color{red}10\color{black}/10$} (vf);
        \path[->] (uf) edge node {$\color{red}10\color{black}/100$} (vf);
		\path[->] (uf) edge node[below right=.05cm] {$\color{red}10\color{black}/10$} (tf);
		\path[->] (vf) edge node {$20/20$} (tf);

    \end{tikzpicture}
    \caption{\\
             (a) Example network with flow from Figure \ref{ex_flow}. \\
             (b) Augmented flow (changed values in red) from augmenting path $s \rightarrow v \rightarrow u \rightarrow t$.}
	\label{augmented_flow}
\end{marginfigure}

\begin{thm}\label{augmenting}
Given is a flow $f$ in network $(G, s, t, c)$. If there is an augmenting path in $G_f$ with capacity $F$ then the function $f':V \times V \to \mathbb{R}_{\geq 0}$ defined as:

$$
f'(u \rightarrow v) =
\begin{cases}
f(u \rightarrow v) + F, & \text{if } u\rightarrow v \text{ is on the augmenting path} \\
f(u \rightarrow v) - F, & \text{if } v\rightarrow u \text{ is on the augmenting path} \\
f(u \rightarrow v), & \text{otherwise}
\end{cases}
$$

is a valid flow in network $(G, s, t, c)$ with $|f'| = |f| + F$.
\end{thm}

\begin{proof}
We need to check the capacity constraint and the conservation constraint.

Let's start with the capacity constraint. The definition of $f'$ has three cases, so we check all three:

\begin{itemize}

	\item Edge $u \rightarrow v$ is on the augmenting path:

\begin{align*}
f'(u \rightarrow v) &= f(u \rightarrow v) + F \tag*{\tiny{(by definition)}}\\
    &\leq f(u \rightarrow v) + c_f(u \rightarrow v) \tag*{\tiny{(by definition of $F$)}}\\
    &=  f(u \rightarrow v) + c(u \rightarrow v) - f(u \rightarrow v) \tag*{\tiny{(by definition of $c_f$)}}\\
    &=  c(u \rightarrow v)
\end{align*}

	\item Edge $v \rightarrow u$ is on the augmenting path:

\begin{align*}
f'(u \rightarrow v) &= f(u \rightarrow v) - F \tag*{\tiny{(by definition)}}\\
    &\geq f(u \rightarrow v) - c_f(u \rightarrow v) \tag*{\tiny{(by definition of $F$)}}\\
    &=  f(u \rightarrow v) - f(u \rightarrow v) \tag*{\tiny{(by definition of $c_f$)}}\\
    &=  0
\end{align*}

	\item Otherwise: In this case the flow of the edge hasn't changed so capacity constraint is satisfied.

\end{itemize}

Next is the conservation constraint. For vertices not on the augmenting path flow in and out of them hasn't changed, so conservation constraint is satisfied there. For a vertex $v$ on the augmenting path we have four cases (since the augmenting path is simple and $v \neq s, v \neq t$):

\begin{itemize}

	\item $u \rightarrow v$ on augmenting path and $v \rightarrow w$ on augmenting path: in this case one incoming edge into $v$ changed by $F$ and one outgoing changed by $F$, so conservation constraint holds for $v$

	\item $u \rightarrow v$ on augmenting path and $w \rightarrow v$ on augmenting path: in this case two incoming edges into $v$ changed, one by $F$ and the other by $-F$, so conservation constraint holds for $v$

	\item $v \rightarrow u$ on augmenting path and $w \rightarrow v$ on augmenting path: in this case one incoming edge into $v$ changed by $-F$ and one outgoing changed by $-F$, so conservation constraint holds for $v$

	\item $v \rightarrow u$ on augmenting path and $v \rightarrow w$ on augmenting path: in this case two outgoing edges from $v$ changed, one by $-F$ and the other by $F$, so conservation constraint holds for $v$

\end{itemize}

\end{proof}

\begin{marginfigure}
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node at (0,2.7) {(a)};
        \node at (0,0) {(b)};

        \node[state] (sf) at (0,1) {$s$};
        \node[state] (vf) at (2,2) {$v$};
        \node[state] (uf) at (2,0) {$u$};
        \node[state] (tf) at (4,1) {$t$};

        \node[state] (s) at (0,4) {$s$};
        \node[state] (v) at (2,5) {$v$};
        \node[state] (u) at (2,3) {$u$};
        \node[state] (t) at (4,4) {$t$};

        \path[->] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$10/10$} (v);
        \path[->] (u) edge node {$10/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$10/10$} (t);
		\path[->] (v) edge node {$20/20$} (t);

        \path[<-, blue] (sf) edge node[below left=.05cm] {$20$} (uf);
        \path[<-, blue] (sf) edge node {$10$} (vf);
        \path[->, blue] (uf) edge[bend left=30] node {$90$} (vf);
        \path[<-, blue] (uf) edge[bend right=30] node {$10$} (vf);
		\path[<-, blue] (uf) edge node[below right=.05cm] {$10$} (tf);
		\path[<-, blue] (vf) edge node {$20$} (tf);

    \end{tikzpicture}
    \caption{\\
             (a) Example network with flow from Figure \ref{ex_flow_max}. \\
             (b) Residual network (in blue) with edges annotated with their residual capacity.}
	\label{ex_residual_max}
\end{marginfigure}

What happens when there is no augmenting path in $G_f$? As the Figure \ref{ex_residual_max} hints we then have a maximum flow (in our example $|f| = 30$). The next theorem proves it.

\begin{thm}\label{no_augmenting}
Given is a flow $f$ in network $(G, s, t, c)$. If there is no augmenting path in $G_f$ then $f$ is a flow with maximum value.
\end{thm}

\begin{proof}
We define two subsets of $V$. The set $S$ holds all the vertices of $V$ that are reachable from $s$ in $G_f$. Since there is no augmenting path in $G_f$ we have $t \notin S$. We also define $T = V \setminus S$. Clearly $(S, T)$ is a cut of our network. Also there is no $G_f$ edge $u \rightarrow v$ with $u \in S$ and $v \in T$ because otherwise $v$ would be reachable from somewhere in $S$ but $v \notin S$, contradicting the definition of $S$. This means (by definition of $G_f$) that $f$ saturates every edge from $S$ to $T$ and avoids every edge from $T$ to $S$. According to Theorem \ref{flowcut} we then have $|f| = \|S, T\|$ which means we have a maximum flow and minimum cut.

\end{proof}

\begin{marginfigure}[0.0in]
\includegraphics[scale=0.8]{fulkerson.png}
\end{marginfigure}
\marginnote{Delbert Ray Fulkerson was an American mathematician who co-developed the Ford–Fulkerson algorithm. \url{https://en.wikipedia.org/wiki/D._R._Fulkerson}}

We can now piece together the following algorithm known as the \textbf{Ford-Fulkerson algorithm}:

\begin{lstlisting}[basicstyle=\small, label={lst:fordfulkerson}, frame=trBL, caption={Ford-Fulkerson algorithm}]
f = zero flow;
Gf = residual graph of f in G;

while (exists augmenting path in Gf):
   pa = choose any augmenting path;
   f = augment f with pa;
   Gf = residual graph of f in G;

return f
\end{lstlisting}

\begin{thm}\label{fordfulkerson}
If the network has capacities in $\mathbb{N}_{\geq 0}$ then the Ford-Fulkerson algorithm terminates and returns the maximum flow in the network.
\end{thm}

\begin{proof}
We prove by induction that $f: V \times V \to \mathbb{N}_{\geq 0}$: The base case is the zero flow which is in $\mathbb{N}_{\geq 0}$. Assume the current flow values are in $\mathbb{N}_{\geq 0}$. The augmenting operation adds or subtracts a positive integer value from the current flow values and conforms to capacity constraints, so it keeps the augmented flow values in $\mathbb{N}_{\geq 0}$ which completes the induction.

The augmented flow $f'$ modifies one outgoing edge from $s$ by $F > 0$, so by the definition of the value of a flow we have $|f'| = |f| + F$. This means that augmenting strictly increases the value of the flow. We also know that flow values have an upper bound (by Theorem \ref{flowcut} any cut capacity is an upper bound). This means the algorithm has to eventually reach the maximum flow and terminate.
\end{proof}

This concludes our detour into network flows\footnote{We have just scratched the surface of the topic on network flows and algorithms computing maximum flows (an area of active research). For example by making smart choices when choosing the augmenting path we can improve the runtime of the algorithm (also we haven't analyzed the runtime). What happens when the capacities are not in $\mathbb{N}_{\geq 0}$. For details on all this and more see:
\\
\bibentry{kleinberg2005ad}
\\
\bibentry{erickson2015a}.}.
\\
We should bring it back to our problem and the associated bipartite graph of conflicting votes. We want a minimal vertex cover and we would like to use the just derived Ford-Fulkerson algorithm to compute it. So we first have to transform our undirected bipartite graph into a network.

We have an undirected bipartite graph $G(V = X \cup Y, E)$ with $X \cap Y = \emptyset$ and $E \subseteq X \times Y$ ($X$ could be the votes of cat lovers and $Y$ the votes of dog lovers in our problem or vice versa). We add a source $s$ and a target $t$ and construct a network $(G', s, t, c)$ in the following way:

\begin{itemize}
	\item vertex set of $G'$ is $X \cup Y \cup \{s, t\}$
	\item $\forall u \in X$ add a directed edge $s \rightarrow u$ into edge set of $G'$
	\item $\forall v \in Y$ add a directed edge $v \rightarrow t$ into edge set of $G'$
	\item $\forall \{u, v\}$ undirected edge in $G$ with $u \in X$ and $v \in Y$ add a directed edge $u \rightarrow v$  into edge set of $G'$
	\item unit capacity\footnote{Unit capacity has the advantage that a flow either saturates the edge or avoids it.}: $\forall (u \rightarrow v) \in \text{ edge set of } G': c(u \rightarrow v) = 1$
\end{itemize}

With a network $(G', s, t, c)$ constructed from a bipartite graph $G$ as described above (an example is shown in Figure \ref{bipartite_network}) we have an equivalence between a matching in the bipartite graph and a flow in the network. The next theorem states this.

\begin{marginfigure}[1.0in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node at (0,4.5) {(a)};
        \node at (0,0) {(b)};

        \node[state] (s) at (0,3) {$s$};
        \node[state] (t) at (5,3) {$t$};

        \node[state] (x1) at (1.5,0.5) {$x_1$};
        \node[state] (x2) at (1.5,2) {$x_2$};
        \node[state] (x3) at (1.5,3) {$x_3$};
        \node[state] (x4) at (1.5,4) {$x_4$};

        \node[state] (y1) at (3,0.5) {$y_1$};
        \node[state] (y2) at (3,2) {$y_2$};
        \node[state] (y3) at (3,3) {$y_3$};

        \path[->] (x1) edge node {$1$} (y1);
        \path[->] (x2) edge node {$1$} (y1);
        \path[->] (x2) edge node {$1$} (y2);
        \path[->] (x2) edge node {$1$} (y3);
        \path[->] (x3) edge node {$1$} (y3);
        \path[->] (x4) edge node {$1$} (y3);

        \path[->] (s) edge node {$1$} (x1);
        \path[->] (s) edge node {$1$} (x2);
        \path[->] (s) edge node {$1$} (x3);
        \path[->] (s) edge node {$1$} (x4);

        \path[->] (y1) edge node {$1$} (t);
        \path[->] (y2) edge node {$1$} (t);
        \path[->] (y3) edge node {$1$} (t);

        \node[state] (bx1) at (1.5,5.5) {$x_1$};
        \node[state] (bx2) at (1.5,7) {$x_2$};
        \node[state] (bx3) at (1.5,8) {$x_3$};
        \node[state] (bx4) at (1.5,9) {$x_4$};

        \node[state] (by1) at (3,5.5) {$y_1$};
        \node[state] (by2) at (3,7) {$y_2$};
        \node[state] (by3) at (3,8) {$y_3$};

        \path[-] (bx1) edge node {} (by1);
        \path[-] (bx2) edge node {} (by1);
        \path[-] (bx2) edge node {} (by2);
        \path[-] (bx2) edge node {} (by3);
        \path[-] (bx3) edge node {} (by3);
        \path[-] (bx4) edge node {} (by3);

    \end{tikzpicture}
    \caption{\\
             (a) Bipartite graph \\
             (b) Network constructed from it.}
	\label{bipartite_network}
\end{marginfigure}

\begin{thm}\label{matchingflow}
A matching $M$ in $G$ induces a flow $f$ in $G'$ such that $|f| = |M|$. Conversely a flow in $G'$ induces a matching $M$ in $G$ such that $|M| = |f|$.
\end{thm}

\begin{proof}
\noindent$(\Rightarrow)$ We have a matching $M$ in $G$, i.e. a subset of edges that don't share a vertex. From the construction of $G'$ it follows that each of the edges in $M$ can be extended to paths from $s$ to $t$ which will only meet in $s$ and $t$. We define a function $f$ that gives unit values to the edges along these paths and zero value to all other edges. We claim that $f$ is a valid flow. It only assigns zero or unit values so it does satisfy the capacity constraint in $G'$. The paths don't intersect except in $s$ and $t$ (because $M$ is a matching), so for any vertex along the path there is exactly one incoming edge with unit value and one outgoing edge with unit value. The rest of the edges have value zero so don't play a role in conservation. This then means that the conservation constraint is satisfied also and $f$ is a flow.
Each edge in $M$ corresponds to one of the paths, so there are $|M|$ edges outgoing from $s$ that have unit value. Hence $|f| = |M|$.

\noindent$(\Leftarrow)$ We have a flow $f$ in $G'$. The flow either saturates or avoids an edge. We define the subset $M$ of edges that are saturated by $f$ and are between $X$ and $Y$. We claim that $M$ is a matching in $G$. Suppose it is not a matching. Then there exists a vertex that is shared by two edges in $M$. If this vertex is in $X$ then it means it has two outgoing edges of unit value but only one incoming edge of unit value (from $s$). If this vertex is in $Y$ then it means it has two incoming edges of unit value and only one outgoing edge of unit value (to $t$). In either case this is a contradiction to the conservation constraint of $f$. So $M$ has to be a matching. The size of $M$ is by its definition equal to the number of saturated edges from $X$ to $Y$. But this number has to be equal to the number of edges of unit value going out of $s$ (conservation constraint). Hence $|M| = |f|$.
\end{proof}

Theorem \ref{matchingflow} let's us use the Ford-Fulkerson algorithm to compute a maximum matching in our bipartite graph $G$. Once we have a maximum matching we get the size of a minimum vertex cover with the following theorem, known as \textbf{K\"onig's theorem}\footnote{For a short and elegant proof see:\\\bibentry{rizzi00}}:

\begin{thm}\label{koenig}
In a bipartite graph $G$ the size of a minimum vertex cover $C$ equals the size of a maximum matching $M$.
\end{thm}

\begin{marginfigure}[0.3in]
\includegraphics[scale=0.5]{koenig.pdf}
\caption{\\
             (a) A bipartite graph $G(X \cup Y, E)$. $X$ are green vertices, $Y$ are red vertices.\\
             (b) Maximum flow $f$ (thicker arrows) and minimum cut $(S, T)$ in the corresponding network $G'$. Thicker arrows between green and red vertices form the maximum matching.\\
             (c) Same cut displayed with the corresponding residual graph $G'_f$.}
	\label{koenig_example}
\end{marginfigure}

\begin{proof}

$C$ is a vertex cover, so it covers all edges, which means it certainly covers a subset $M$ of all edges. But $M$ is a matching, so no two edges share a vertex. It follows that $|C| \geq |M|$.

From the maximum matching $M$ we get the associated maximum flow $f$ as described in Theorem \ref{matchingflow}. The residual graph $G'_f$ of the associated network cannot have any augmenting paths.

We consider the minimum cut $(S, T)$ associated with the maximum flow $f$. We define the following sets:

\begin{itemize}
	\item $X_S = X \cap S, X_T = X \cap T$
	\item $Y_S = Y \cap S, Y_T = Y \cap T$
	\item $H = \{(u, v) \text{ edge in } G: u \in S, v \in T\}$
	\item $B = \{v \in Y_T: \exists u \in X_S \text{ with } (u, v) \text{ edge in } G\}$
	\item $D = X_T \cup Y_S \cup B$
\end{itemize}

\noindent $D$ is a vertex cover: $X_T \subseteq D$ and $Y_S \subseteq D$, so $D$ covers all edges that have endpoints in $X_T$ or $Y_S$. The set $B$ provides cover for $H$.

A vertex $u \in X_T$ is not reachable from $s$ in $G'_f$. It means that $f$ saturates $s \rightarrow u$ in $G'$, so the saturated edge $s \rightarrow u$ crosses the $(S, T)$ cut and counts towards $\|S, T\|$.

A vertex $v \in Y_S$ is reachable from $s$ in $G'_f$. It means that $f$ saturates $v \rightarrow t$ in $G'$ (otherwise some vertex from $T$ would be reachable from $v$ and also from $s$ in $G'_f$ which is a contradiction). The saturated edge $v \rightarrow t$ crosses the $(S, T)$ cut and counts towards $\|S, T\|$.

$f$ is a maximum flow so any edge from $X_S$ to $Y_T$ is saturated and counts towards $\|S, T\|$.

$$
\|S, T\| = |X_T| + |Y_S| + |H|
$$

\noindent Figure \ref{koenig_example} shows this. In (b) there are three saturated (thick) arrows crossing the cut. The first two (counting from left to right) are due to $X_T$ and the last one due to $Y_S$. In this example $H$ is the empty set.

We have
$$
|M| = |f| = \|S, T\| = |X_T| + |Y_S| + |H| \geq |X_T| + |Y_S| + |B| \geq |D|
$$
\noindent $D$ is a vertex cover and $C$ is a minimum vertex cover, so $|D| \geq |C|$. It follows that $|C| \geq |M| \geq |D| \geq |C|$ which means $|C| = |M|$.
\end{proof}

This solves the problem in this note. The number of satisfied viewers is $|V| - |M|$, where $V$ is the set of vertices in the bipartite graph $G$ of votes with their conflicts as edges and $M$ is a maximum matching in $G$ computed with the Ford-Fulkerson algorithm taking advantage of the min-max duality shown in Figure \ref{minmax_duality}.

\begin{marginfigure}
    \includegraphics[scale=0.5]{flowcutminmax.pdf}
    \caption{Min-max duality in bipartite graphs and corresponding networks.}
	\label{minmax_duality}
\end{marginfigure}{}

\bibliographystyle{plainnat}
\bibliography{../common/math}

\end{document}


%!TEX program = xelatex
\documentclass[aspectratio=169]{beamer}

\usepackage{blindtext}
\usepackage{url}
\usepackage{bm}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{xcolor}
\usepackage{tikzsymbols}
\usepackage{cleveref}
\usepackage{hyperref}
\usepackage{MyMnSymbol}
\hypersetup{
  colorlinks = true,
}
\usepackage[linesnumbered,ruled,vlined]{algorithm2e}
\usefonttheme[onlymath]{serif}
\usetheme{Execushares}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\title{Alternating Direction Method of Multipliers}
\subtitle{\large{ELEC5470/IEDA6100A - Convex Optimization}}
\author{\normalsize{\textbf{Vin\'icius and Prof. Daniel Palomar}}}
\date{December, 2020}

\setcounter{showSlideNumbers}{1}
\addtobeamertemplate{footnote}{\vspace{-2pt}\advance\hsize-0.5cm}{\vspace{6pt}}

\begin{document}
  \setcounter{showProgressBar}{0}
  \setcounter{showSlideNumbers}{0}

  \frame{\titlepage}

  \begin{frame}
    \frametitle{Contents}
    \begin{enumerate}
    \item Introduction \\ \textcolor{ExecusharesGrey}{\footnotesize\hspace{1em} Optimization algorithms, motivation}
    \item Alternating Direction Method of Multipliers \\ \textcolor{ExecusharesGrey}{\footnotesize\hspace{1em} The basics}
    \item Practical Examples \\ \textcolor{ExecusharesGrey}{\footnotesize\hspace{1em} Robust PCA and Graphical Lasso}
    \end{enumerate}
  \end{frame}

  \setcounter{framenumber}{0}
  \setcounter{showProgressBar}{1}
  \setcounter{showSlideNumbers}{1}

  \section*{Why use optimization algorithms?}
        \begin{frame}{Motivations}
          {\large methods for}
          \begin{itemize}
            \item large-scale optimization
              \begin{itemize}
                \item machine learning/statistics with huge datasets
                \item computer vision
              \end{itemize}
            \item descentralized optimization
              \begin{itemize}
                \item entities/agents/threads coordinate to solve a large problem by passing small messages
              \end{itemize}
          \end{itemize}
        \end{frame}

        \begin{frame}{Optimization Algorithms}
          \begin{itemize}
            \item Gradient Descent
            \item Newton
            \item Interior Point Methods (IPM)
            \item Block Coordinate Descent (BCD)
            \item Majorization-Minimization (MM)
            \item Block Majorization-Minimization (BMM)
            \item Successive Convex Approximation (SCA)
            \pause
            \item ...
            \pause
            \item \bf{Alternating Direction Method of Multipliers (ADMM)}
          \end{itemize}
        \end{frame}
  \begin{frame}
    \frametitle{Reference}
          \begin{itemize}
            \item Boyd \textit{et al.} \textbf{Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers}.
              \textit{Foundations and Trends in Machine Learning}. 2010.
            \item available online for \textbf{free}: \url{https://web.stanford.edu/\~boyd/papers/pdf/admm\_distr\_stats.pdf}
            \item citations: 13519\footnote{as of Nov. 24th 2020}
            \item Yuxin Chen's Princeton lecture notes ELE 522: Large-Scale Optimization for Data Science
          \end{itemize}
  \end{frame}

        \begin{frame}{Dual Problem}
          \begin{itemize}
            \item convex equality constrained optimization problem
             \begin{equation*}
                \begin{array}{ll}
                  \underset{\bm x}{\textsf{minimize}} & f(\bm x) \\
                  \textsf{subject to} & \bm A\bm x  = \bm b
                 \end{array}
            \end{equation*}
          \item Lagrangian: $L(\bm x, \bm y) = f(\bm x) + \bm y^\top(\bm A \bm x - \bm b)$
          \item dual function: $g(\bm y) = \underset{\bm x}{\textsf{inf}} ~~ L(\bm x, \bm y)$
          \item dual problem: $\underset{\bm y}{\textsf{maximize}} ~~ g(\bm y)$
          \item recover: $\bm x^\star = \underset{\bm x}{\textsf{argmin}}~~ L(\bm x, \bm y^\star)$
          \end{itemize}
        \end{frame}

        \section*{Dual Ascent}

        \begin{frame}{Dual Ascent}
          \begin{itemize}
            \item gradient method for dual problem: $\bm y^{k+1} = \bm y^k + \rho^k \nabla g(\bm y^k)$
            \item $\nabla g(\bm y^k) = \bm A \bm x^{k+1} - \bm b$, where $\bm x^{k+1} = \underset{\bm x}{\textsf{arg~min}}~~L(\bm x, \bm y^k)$
            \item dual ascent method is
            \begin{align*}
              \bm x^{k+1} &:= \underset{\bm x}{\textsf{arg~min}}~~L(\bm x, \bm y^k) \\
              \bm y^{k+1} &:= \bm y^k + \rho^k\left(\bm A \bm x^{k+1} - \bm b\right)
            \end{align*}
            \pause
            \item why?
          \end{itemize}
        \end{frame}

        \section*{Dual Decomposition}
        \begin{frame}{Dual Decomposition}
          \begin{itemize}
            \item suppose $f$ is separable:
            \begin{equation*}
              f(\bm x) = f_{1}(x_1) + \dots + f_n(x_n), \bm x = (x_1, \dots, x_n)
            \end{equation*}
            \item then the Lagrangian is separable in $\bm x$:
            \begin{equation*}
              L_i(x_i, \bm y) = f_i(x_i) + \bm y^\top\bm a_{*,i} x_i
            \end{equation*}
            \item $\bm x$-minimization splits into $n$ separate minimizations
            \begin{equation*}
              x^{k+1}_i := \underset{x_i}{\textsf{arg~min}}~ L_i(x_i, \bm y^k), i = 1, ..., n
            \end{equation*}
            which can be done in parallel and
              $\bm y^{k+1} = \bm y^k + \alpha^k\left(\sum_{i=1}^n \bm a_{*,i}x^{k+1}_i - \bm b\right)$
          \end{itemize}
        \end{frame}

    \begin{frame}
      \frametitle{Optimization Problem}
                  \vspace{1cm}
                    {\huge
                  \begin{equation}
                     \begin{array}{ll}
                       \underset{\bm x,~ \bm z}{\textsf{minimize}} & f(\bm x) + g(\bm z) \\
                       \textsf{subject to} & \bm A\bm x + \bm B \bm z = \bm c
                      \end{array}
                 \end{equation}
                    }
                 \begin{itemize}
                 \item variables: $\bm x \in \mathbb{R}^{n}$ and $\bm z \in \mathbb{R}^{m}$
                 \item parameters: $\bm A \in \mathbb{R}^{p\times n}$, $\bm B \in \mathbb{R}^{p\times m}$, and $\bm c \in \mathbb{R}^p$
                 \item optimal value: $p^\star = \underset{\bm x, \bm z}{\textsf{inf}}\left\{f(\bm x) + g(\bm z): \bm A\bm x + \bm B\bm z = \bm c\right\}$
                 \end{itemize}
    \end{frame}

    \section*{Augmented Lagrangian Method}

    \begin{frame}
        \frametitle{Augmented Lagrangian Method}
        \begin{itemize}
          \item Augmented Lagrangian:
            \begin{equation*}
              L_{\rho}(\bm x, \bm z, \bm y) = \underbrace{f(\bm x) + g(\bm z) + \langle \bm y, \bm A\bm x + \bm B \bm z - \bm c \rangle}_{\text{Lagrangian}}
              + \dfrac{\rho}{2}\norm{\bm A\bm x + \bm B \bm z - \bm c}^2_{\mathrm{F}}
            \end{equation*}
          \item ALM consists of the iterations:
            \begin{align*}
              \bm x^{k+1}, \bm z^{k+1} & := \underset{\bm x, \bm z}{\textsf{arg min}}~~ L_{\rho}(\bm x, \bm z, \bm y^k) ~{\color{gray} \text{// primal update}}\\
              \bm y^{k+1} & := \bm y^k + \rho\left(\bm A\bm x^{k+1} + \bm B \bm z^{k+1} - \bm c\right) ~{\color{gray} \text{// dual update}}
            \end{align*}
          \item $\rho > 0$ is a penalty hyperparameter
        \end{itemize}
    \end{frame}

    \begin{frame}{Issues with Augmented Lagrangian Method}
      \begin{itemize}
        \item the primal step is often expensive to solve -- as expensive as solving the original problem
        \item minimization of $\bm x$ and $\bm z$ has to be done jointly
      \end{itemize}
    \end{frame}

    \section*{Alternating Direction Method of Multipliers}

                \begin{frame}
                  \frametitle{Alternating Direction Method of Multipliers}
                  \begin{itemize}
                    \item Augmented Lagrangian:
                      \begin{equation*}
                        L_{\rho}(\bm x, \bm z, \bm y) = \underbrace{f(\bm x) + g(\bm z) + \langle \bm y, \bm A\bm x + \bm B \bm z - \bm c \rangle}_{\text{Lagrangian}}
                        + \dfrac{\rho}{2}\norm{\bm A\bm x + \bm B \bm z - \bm c}^2_{\mathrm{F}}
                      \end{equation*}
                    \item ADMM consists of the iterations:
                      \begin{align*}
                        \bm x^{k+1} & := \underset{\bm x}{\textsf{arg min}}~~ L_{\rho}(\bm x, \bm z^k, \bm y^k) \\
                        \bm z^{k+1} & := \underset{\bm z}{\textsf{arg min}}~~ L_{\rho}(\bm x^{k+1}, \bm z, \bm y^k) \\
                        \bm y^{k+1} & := \bm y^k + \rho\left(\bm A\bm x^{k+1} + \bm B \bm z^{k+1} - \bm c\right)
                      \end{align*}
                    \item $\rho > 0$ is a penalty hyperparameter
                  \end{itemize}
                \end{frame}

                \begin{frame}
                  \frametitle{Convergence and Stopping Criteria}
                  \begin{itemize}
                    \item assume (very little!)
                      \begin{itemize}
                        \item $f$, $g$ are convex, closed, proper % closed: sublevels sets are closed sets
                                                                  % proper: effective domain is non-empty and it never attains -infty
                        \item $L_{0}$ has a saddle point % saddle point theorem
                      \end{itemize}
                    \item then ADMM converges:
                      \begin{itemize}
                        \item iterates approach feasibility: $\bm A \bm x^k + \bm B \bm z^k - \bm c \rightarrow \bm 0$
                        \item objective approaches optimal value: $f(\bm x^k) + g(\bm z^k) \rightarrow p^\star$
                      \end{itemize}
                    \item false (in general) statements: $\bm x$ converges, $\bm z$ converges
                    \item true statement: $\bm y$ converges
                    \item what matters: residual is small and near optimality in objective value
                  \end{itemize}
                \end{frame}

                \begin{frame}{Convergence of ADMM in Practice}
                  \begin{itemize}
                    \item ADMM is often slow to converge to high accuracy
                    \item ADMM often converges to moderate accuracy within a few dozens of iterations, which is often sufficient
                    for most practical purposes
                  \end{itemize}
                \end{frame}

                \section*{Practical Examples}
                \begin{frame}{Robust PCA (Candes et al. '08)}
                  \begin{itemize}
                    \item We would like to model a data matrix $\bm M$ as low-rank plus sparse components:
                  \end{itemize}
                    {\huge
                    \begin{equation*}
                       \begin{array}{ll}
                         \underset{\bm L,~ \bm S}{\textsf{minimize}} & \norm{\bm L}_{*} + \lambda \norm{\bm S}_{1} \\
                         \textsf{subject to} & \bm L + \bm S = \bm M
                        \end{array}
                   \end{equation*}
                      }
                  \begin{itemize}
                    \item where $\norm{\bm L}_{*} := \sum_{i=1}^{n}\sigma_{i}(\bm{L})$ is the nuclear norm
                    \item and $\norm{\bm S}_1 := \sum_{i,j}\vert S_{ij}\vert$ is the entrywise $\ell_1$-norm
                  \end{itemize}
                \end{frame}
                \begin{frame}{Robust PCA via ADMM}
                  ADMM for solving robust PCA:
                  \begin{align*}
                    \bm L^{k+1} &= \underset{\bm L}{\textsf{arg~min}} ~~ \norm{\bm L}_{*} + \mathrm{tr}\left(\bm Y^{k\top} \bm L \right) + \dfrac{\rho}{2}\norm{\bm L + \bm S^k - \bm M}^2_{\textrm{F}}\\
                    \bm S^{k+1} &= \underset{\bm S}{\textsf{arg~min}} ~~ \lambda\norm{\bm S}_{1} + \mathrm{tr}\left(\bm Y^{k\top} \bm S \right) + \dfrac{\rho}{2}\norm{\bm L^{k+1} + \bm S - \bm M}^2_{\textrm{F}}\\
                    \bm Y^{k+1} &= \bm Y^k + \rho \left(\bm L^{k+1} + \bm S^{k+1} - \bm M\right)
                  \end{align*}
                \end{frame}
                \begin{frame}{Robust PCA via ADMM}
                  \begin{align*}
                    \bm L^{k+1} &= \textsf{SVT}_{\rho^{-1}}\left(\bm M - \bm S^k - \frac{1}{\rho}\bm Y^k\right)\\
                    \bm S^{k+1} &= \textsf{ST}_{\lambda \rho^{-1}}\left(\bm M - \bm L^{k+1} - \frac{1}{\rho}\bm Y^k\right) \\
                    \bm Y^{k+1} &= \bm Y^k + \rho \left(\bm L^{k+1} + \bm S^{k+1} - \bm M\right),
                  \end{align*}
                  where for any $\bm X$ with \textsf{SVD} $\bm X = \bm U \boldsymbol \Sigma \bm V^\top$, $\boldsymbol \Sigma = \textsf{diag}\left(\left\{\sigma_i\right\}\right)$, we have
                  \begin{equation*}
                    \textsf{SVT}_{\tau}\left(\bm X\right) = \bm U \textsf{diag}\left(\left\{(\sigma_i - \tau)^{+}\right\}\right)\bm V ^\top
                  \end{equation*}
                  and
                  \begin{equation*}
                     \left(\textsf{ST}_{\tau}\left(\bm X\right)\right)_{ij} =
                    \begin{cases}
                      X_{ij} - \tau, & ~\text{if}~ X_{ij} > \tau,\\
                      0, & ~\text{if}~ \vert X_{ij}\vert \leq \tau, \\
                      X_{ij} + \tau, & ~\text{if}~ X_{ij} < -\tau
                    \end{cases}
                  \end{equation*}
                \end{frame}

                \begin{frame}{Graphical Lasso}
                  \textbf{Precision matrix estimation} from Gaussian samples:
                  {\large
                  \begin{equation*}
                     \begin{array}{ll}
                       \underset{\boldsymbol \Theta}{\textsf{minimize}} & \underbrace{- \textrm{log det} ~ \boldsymbol \Theta + \langle \boldsymbol \Theta, \bm S\rangle}_{\text{neg. log likelihood}} + \lambda \norm{\boldsymbol \Theta}_{1}\\
                       \textsf{subject to} & \boldsymbol \Theta \succ \mathbf{0}
                      \end{array}
                 \end{equation*}
                    }
                  Or equivalently, using a slack variable $\boldsymbol \Psi = \boldsymbol \Theta$
                    {\large
                    \begin{equation*}
                       \begin{array}{ll}
                         \underset{\boldsymbol \Theta, \boldsymbol \Psi}{\textsf{minimize}} & \underbrace{- \textrm{log det} ~ \boldsymbol \Theta + \langle \boldsymbol \Theta, \bm S\rangle}_{\text{neg. log likelihood}} + \lambda \norm{\boldsymbol \Psi}_{1}\\
                         \textsf{subject to} & \boldsymbol \Theta \succ \mathbf{0}, \boldsymbol \Theta = \boldsymbol \Psi
                        \end{array}
                   \end{equation*}
                      }
                \end{frame}

                \begin{frame}{Graphical Lasso via ADMM}
                  \begin{align*}
                    \boldsymbol \Theta^{k+1} & = \underset{\boldsymbol \Theta \succ \mathbf{0}}{\textsf{arg min}} ~~ - \textrm{log det} ~ \boldsymbol \Theta + \langle \boldsymbol \Theta, \bm S + \bm Y^k\rangle + \dfrac{\rho}{2}\norm{\boldsymbol \Theta - \boldsymbol \Psi^k}^2_{\textrm{F}}\\
                    \boldsymbol \Psi^{k+1} & = \underset{\boldsymbol \Psi}{\textsf{arg min}} ~~ \lambda \norm{\boldsymbol \Psi}_{1} - \langle \boldsymbol \Psi, \bm Y^k\rangle + \dfrac{\rho}{2}\norm{\boldsymbol \Theta^k - \boldsymbol \Psi}^2_{\textrm{F}}\\
                    \bm Y^{k+1} & = \bm Y^k + \rho\left(\boldsymbol \Theta^{k+1} - \boldsymbol \Psi^{k+1}\right)
                  \end{align*}
                \end{frame}

                \begin{frame}{Graphical Lasso via ADMM}
                  \begin{align*}
                    \boldsymbol \Theta^{k+1} & = \mathcal{F}_\rho\left(\boldsymbol \Psi^k - \dfrac{1}{\rho}\left(\bm Y^k + \bm S\right)\right) \\
                    \boldsymbol \Psi^{k+1} & = \textsf{ST}_{\lambda\rho^{-1}}\left(\boldsymbol \Theta^{k+1} + \dfrac{1}{\rho}\bm Y^{k}\right) \\
                    \bm Y^{k+1} & = \bm Y^k + \rho\left(\boldsymbol \Theta^{k+1} - \boldsymbol \Psi^{k+1}\right)
                  \end{align*}
                  \begin{itemize}
                    \item where $\mathcal{F}_{\rho}(\bm X) := \frac{1}{2} \bm U \textsf{diag}\left(\left\{\lambda_i + \sqrt{\lambda^2_i + \frac{4}{\rho}}\right\}\right)\bm U^\top$, for $\bm X = \bm U \boldsymbol \Lambda \bm U^\top$.
                  \end{itemize}
                \end{frame}

                \begin{frame}{Network of stocks via Graphical Lasso}
                  \begin{figure}[!htb]
                    \centering
                    \includegraphics[scale=0.5]{images/stock-network-glasso.eps}
                  \end{figure}
                \end{frame}

                \begin{frame}{Network of stocks via Graphical Lasso}
                  \begin{figure}[!htb]
                    \centering
                    \includegraphics[scale=0.5]{images/augmented-lagrangian.pdf}
                    ~
                    \includegraphics[scale=0.5]{images/primal-residual.pdf}
                  \end{figure}
                \end{frame}

                \begin{frame}{Conclusion}
                  \begin{itemize}
                    \item ADMM is a versatile/flexible optimization framework
                    \item may not be the best for a specific case, but often performs well in practice
                    \item convergence often needs to be proved in a case-by-case scenario
                  \end{itemize}
                \end{frame}

                \section*{Questions?}
\end{document}


\section{Relative FM-index}

The \emph{relative FM-index} (\RFM) \cite{Belazzougui2014} is a compressed
suffix array of a sequence relative to the \CSA{} of another sequence.
The index is based on approximating the
\emph{longest common subsequence} (\LCS) of $\mBWT_{R}$ and $\mBWT_{S}$,
where $R$ is the reference sequence and $S$ is the target sequence, and
storing several structures based on the common subsequence. Given a
representation of $\mBWT_{R}$ supporting \rank{} and \select{}, we can use the
relative index $\mRFM_{S \mid R}$ to simulate \rank{} and \select{} on
$\mBWT_{S}$.

In this section, we describe the relative FM-index using the notation and the
terminology of this paper. We also give an explicit description of the
\locate{} and \extract{} functionality, which was not included in the original
paper. Finally, we describe a more space-efficient variant of the algorithm
for building a relative FM-index with full functionality.

\subsection{Basic index}

Assume that we have found a long common subsequence of sequences $X$ and $Y$.
We call positions $X[i]$ and $Y[j]$ \emph{lcs-positions}, if they are in the
common subsequence. If $B_{X}$ and $B_{Y}$ are the binary sequences marking
the common subsequence ($X[\select_1(B_{X},i)] = Y[\select_1(B_{Y},i)]$), we
can move between lcs-positions in the two sequences with \rank{} and \select{}
operations. If $X[i]$ is an lcs-position, the corresponding position in
sequence $Y$ is $Y[\mselect_{1}(B_{Y}, \mrank_{1}(B_{X}, i))]$. We denote this
pair of \emph{lcs-bitvectors} $\mAli(X,Y) = \langle B_X,B_Y \rangle$.

In its most basic form, the relative FM-index $\mRFM_{S \mid R}$ only supports
\find{} queries by simulating \rank{} queries on $\mBWT_{S}$. It does this by
storing $\mAli(\BWT_{R},\BWT_{S})$ and the complements (subsequences of
non-aligned characters) $\mCS(\mBWT_{R})$ and $\mCS(\mBWT_{S})$. The
lcs-bitvectors are compressed using \emph{entropy-based compression}
\cite{Raman2007}, while the complements are stored in structures similar to
the reference $\mBWT_{R}$.

To compute $\mrank_{c}(\mBWT_{S}, i)$, we first determine the number of
lcs-positions in $\mBWT_{S}$ up to position $S[i]$ with $k =
\mrank_{1}(B_{\mBWT_{S}}, i)$. Then we find the lcs-position $k$ in $\mBWT_{R}$
with $j = \mselect_{1}(B_{\mBWT_{R}}, k)$. With these positions, we can compute
\begin{eqnarray*}
\mrank_{c}(\mBWT_{S}, i) &=& \mrank_{c}(\mBWT_{R}, j) \\
   && - \mrank_{c}(\mCS(\mBWT_{R}),j-k) \\
   && + \mrank_{c}(\mCS(\mBWT_{S}), i-k).
\end{eqnarray*}

\subsection{Relative select}

We can implement the entire functionality of a compressed suffix array with
\rank{} queries on the \BWT. However, if we use the \CSA{} in a compressed
suffix tree, we also need \select{} queries to support \emph{forward
searching} with $\mPsi$ and $\mChild$ queries. We can always implement
\select{} queries by binary searching with \rank{} queries, but the result
will be much slower than the \rank{} queries.

A faster alternative to support \select{} queries in the relative FM-index
is to build a \emph{relative select} structure \rselect{}~\cite{Boucher2015}.
Let $\mF_{X}$ be a sequence consisting of the characters of sequence $X$ in
sorted order. Alternatively, $\mF_{X}$ is a sequence such that $\mF_{X}[i] =
\mBWT_{X}[\mPsi_X(i)]$. The relative select structure consists of bitvectors
$\mAli(\mF_{R}, \mF_{S})$, where $B_{\mF_{R}}[i] = B_{\mBWT_{R}}[\mPsi_R(i)]$
and $B_{\mF_{S}}[i] = B_{\mBWT_{S}}[\mPsi_S(i)]$, as well as the \C{} array
$\mC_{\mLCS}$ for the common subsequence.

To compute $\mselect_{c}(\mBWT_{S}, i)$, we first determine how many of
the first $i$ occurrences of character $c$ are lcs-positions with $k =
\mrank_{1}(B_{\mF_{S}}, \mC_{\mBWT_{S}}[c] + i) - \mC_{\mLCS}[c]$. Then we check
from bit $B_{\mF_{S}}[\mC_{\mBWT_{S}}[c] + i]$ whether the occurrence we are
looking for is an lcs-position or not. If it is,
we find the position in $\mBWT_{R}$ as $j = \mselect_{c}(\mBWT_{R},
\mselect_{1}(B_{\mF_{R}}, \mC_{\mLCS}[c] + k)- \mC_{R}[c])$, and then map $j$ to
$\mselect_{c}(\mBWT_{S}, i)$ by using $\mAli(\mBWT_{R}, \mBWT_{S})$. Otherwise we
find the occurrence in $\mCS(\mBWT_{S})$ with $j = \mselect_{c}(\mCS(\mBWT_{S}),
i-k)$, and return $\mselect_{c}(\mBWT_{S}, i) = \mselect_{0}(B_{\mBWT_{S}}, j)$.

\subsection{Full functionality}

If we want the relative FM-index to support \locate{} and \extract{} queries,
we cannot build it from any common subsequence of $\mBWT_{R}$ and $\mBWT_{S}$.
We need a \emph{bwt-invariant subsequence} \cite{Belazzougui2014}, where the
alignment of the \BWT{}s is also an alignment of the original sequences.

\begin{definition}\label{def:bwt-invariant}
Let $X$ be a common subsequence of $\mBWT_{R}$ and $\mBWT_{S}$, and let
$\mBWT_{R}[i_{R}]$ and $\mBWT_{S}[i_{S}]$ be the lcs-positions corresponding to
$X[i]$. Subsequence X is bwt-invariant if
$$
\mSA_{R}[i_{R}] < \mSA_{R}[j_{R}] \iff \mSA_{S}[i_{S}] < \mSA_{S}[j_{S}]
$$
for all positions $i, j \in \set{1, \dotsc, \abs{X}}$.
\end{definition}

In addition to the structures already mentioned, the full relative FM-index
has another pair of lcs-bitvectors, $\mAli(R,S)$, which marks the
bwt-invariant subsequence in the original sequences. If $\mBWT_{R}[i_{R}]$ and
$\mBWT_{S}[i_{S}]$ are lcs-positions, we set $B_{R}[\mSA_{R}[i_{R}]-1] = 1$ and
$B_{S}[\mSA_{S}[i_{S}]-1] = 1$.\footnote{For simplicity, we assume that the
endmarker is not a part of the bwt-invariant subsequence. Hence $\mSA[i] > 1$
for all lcs-positions $\mBWT[i]$.}

To compute the answer to a $\mlocate(i)$ query, we start by iterating
$\mBWT_{S}$ backwards with \LF{} queries, until we find an lcs-position
$\mBWT_{S}[i']$ after $k$ steps. Then we map position $i'$ to the corresponding
position $j'$ in $\mBWT_{R}$ by using $\mAli(\mBWT_{R},\mBWT_{S})$. Finally we
determine $\mSA_{R}[j']$ with a \locate{} query in the reference index, and map
the result to $\mSA_{S}[i']$ by using $\mAli(R,S)$.\footnote{If $\mBWT_{S}[i']$
and $\mBWT_{R}[j']$ are lcs-positions, the corresponding lcs-positions in the
original sequences are $S[\mSA_{S}[i']-1]$ and $R[\mSA_{R}[j']-1]$.} The result
of the $\mlocate(i)$ query is $\mSA_{S}[i']+k$.

The $\mISA_{S}[i]$ access required for \extract{} queries is supported in a
similar way. We find the lcs-position $S[i+k]$ for the smallest $k \ge 0$, and
map it to the corresponding position $R[j]$ by using $\mAli(R,S)$. Then we
determine $\mISA_{R}[j+1]$ by using the reference index, and map it back to
$\mISA_{S}[i+k+1]$ with $\mAli(\mBWT_{R},\mBWT_{S})$. Finally we iterate
$\mBWT_{S}$ $k+1$ steps backward with \LF{} queries to find $\mISA_{S}[i]$.

If the target sequence contains long
insertions not present in the reference, we may also want to include
some \SA{} and \ISA{} samples for querying those regions.

\subsection{Finding a bwt-invariant subsequence}

With the basic relative FM-index, we approximate the longest common
subsequence of $\mBWT_{R}$ and $\mBWT_{S}$ by partitioning the \BWT{}s according
to lexicographic contexts, finding the longest common subsequence for each
pair of substrings in the partitioning, and concatenating the results. The
algorithm is fast, easy to parallelize, and quite space-efficient. As such,
\RFM{} construction is practical, having been tested with datasets of hundreds
of gigabytes in size.

In the following, we describe a more space-efficient variant of the original
algorithm \cite{Belazzougui2014} for finding a bwt-invariant subsequence. We
\begin{itemize}
\item save space by simulating the \emph{mutual suffix array} $\mSA_{RS}$ with
$\mCSA_{R}$ and $\mCSA_{S}$;
\item \emph{match} suffixes of $R$ and $S$ only if they are adjacent in
$\mSA_{RS}$; and
\item run-length encode the match arrays to save space.
\end{itemize}

\begin{definition}
Let $R$ and $S$ be two sequences, and let $\mSA = \mSA_{RS}$ and $\mISA =
\mISA_{RS}$. The \emph{left match} of suffix $R[i,\abs{R}]$ is the suffix
$S[\mSA[\mISA[i]-1] - \abs{R}, \abs{S}]$, if $\mISA[i] > 1$ and
$\mSA[\mISA[i]-1]$ points to a suffix of $S$ ($\mSA[\mISA[i]-1] > \abs{R}$).
The \emph{right match} of suffix $R[i,\abs{R}]$ is the suffix
$S[\mSA[\mISA[i]+1] - \abs{R}, \abs{S}]$, if $\mISA[i] < \abs{RS}$ and
$\mSA[\mISA[i]+1]$ points to a suffix of $S$.
\end{definition}

We simulate the mutual suffix array $\mSA_{RS}$ with $\mCSA_{R}$, $\mCSA_{S}$,
and the \emph{merging bitvector} $B_{R,S}$ of length $\abs{RS}$. We set
$B_{R,S}[i] = 1$, if $\mSA_{RS}[i]$ points to a suffix of $S$. The merging
bitvector can be built in $\Oh(\abs{S} \cdot t_{\mLF})$ time, where $t_{\mLF}$ is
the time required for an \LF{} query, by extracting $S$ from $\mCSA_{S}$ and
backward searching for it in $\mCSA_{R}$ \cite{Siren2009}. Suffix
$R[i,\abs{R}]$ has a left (right) match, if $B_{R,S}[\mselect_{0}(B_{R,S},
\mISA_{R}[i])-1] = 1$ ($B_{R,S}[\mselect_{0}(B_{R,S}, \mISA_{R}[i])+1] = 1)$).

Our next step is building the \emph{match arrays} $\mleft$ and $\mright$,
which correspond to the arrays $A[\cdot][2]$ and $A[\cdot][1]$ in the original
algorithm. This is done by traversing $\mCSA_{R}$ backwards from
$\mISA_{R}[\abs{R}] = 1$ with \LF{} queries and following the left and the
right matches of the current suffix. During the traversal, we maintain
the invariant $j = \mSA_{R}[i]$ with $(i,j) \leftarrow (\mLF_{R}(i), j-1)$. If
suffix $R[j,\abs{R}]$ has a left (right) match, we use the shorthand $l(j) =
\mrank_{1}(B_{R,S}, \mselect_{0}(B_{R,S}, i)-1)$ ($r(j) = \mrank_{1}(B_{R,S},
\mselect_{0}(B_{R,S}, i)+1)$) to refer to its position in $\mCSA_{S}$.

We say that suffixes $R[j,\abs{R}]$ and $R[j+1,\abs{R}]$ have the same left
match if $l(j) = \mLF_{S}(l(j+1))$. Let $R[j,\abs{R}]$ to $R[j+\ell,\abs{R}]$
be a maximal run of suffixes having the same left match, with suffixes
$R[j,\abs{R}]$ to $R[j+\ell-1,\abs{R}]$ starting with the same characters as
their left matches.\footnote{The first character of a suffix can be determined
by using the $\mC$ array.} We find the left match of suffix $R[j,\abs{R}]$ as
$j' = \mSA_{S}[l(j)]$ by using $\mCSA_{S}$, and set $\mleft[j,j+\ell-1] =
[j',j'+\ell-1]$. The right match array $\mright$ is built in a similar way.

The match arrays require $2\abs{R} \log \abs{S}$ bits of space. If sequences
$R$ and $S$ are similar, the runs in the arrays tend to be long. Hence we can
run-length encode the match arrays to save space. The traversal takes
$\Oh(\abs{R} \cdot (t_{\mLF} + t_{\mrank} + t_{\mselect}) + rd \cdot
t_{\mLF})$ time, where $t_{\mrank}$ and $t_{\mselect}$ denote the time
required by \rank{} and \select{} operations, $r$ is the number of runs in the
two arrays, and $d$ is the suffix array sample interval in
$\mCSA_{S}$.\footnote{The time bound assumes text-order sampling.}

The final step is determining the bwt-invariant subsequence. We find a
binary sequence $B_{R}[1,\abs{R}]$, which marks the common subsequence in $R$,
and a strictly increasing integer sequence $Y$, which contains the positions
of the common subsequence in $S$. This can be done by finding the longest
increasing subsequence over $R$, where we consider both $\mleft[i]$ and
$\mright[i]$ as candidates for the value at position $i$, and using the found
subsequence as $Y$. If $Y[j]$ comes from $\mleft[i]$ ($\mright[i]$), we set
$B_{R}[i] = 1$, and align suffix $R[i, \abs{R}]$ with its
left (right) match $S[Y[j], \abs{S}]$ in the bwt-invariant subsequence. We can
find $B_{R}$ and $Y$ in $\Oh(\abs{R} \log \abs{R})$ time with
$\Oh(\abs{R} \log \abs{R})$ bits of additional working space with a
straightforward modification of the dynamic programming algorithm for finding
the longest increasing subsequence. The dynamic programming tables can be
run-length encoded, but we found that this did not yield good time/space
trade-offs.

As sequence $Y$ is strictly increasing, we can convert it into binary sequence
$B_{S}[1,\abs{S}]$, marking $B_{S}[Y[j]] = 1$ for all $j$.
Afterwards, we consider the binary sequences $B_{R}$ and $B_{S}$ as the
lcs-bitvectors $\mAli(R,S)$. Because every suffix of $R$ starts with the same
character as its matches stored in the $\mleft$ and $\mright$ arrays,
subsequences $R[B_{R}]$ and $S[B_{S}]$ are identical.

For any $i$, let $i_{R} = \mselect_{1}(B_{R}, i)$ and $i_{S} =
\mselect_{1}(B_{S}, i)$ be the lcs-positions of rank $i$. As suffixes
$R[i_{R}, \abs{R}]$ and $S[i_{S}, \abs{S}]$ are aligned in the bwt-invariant
subsequence, they are also adjacent in the mutual suffix array $\mSA_{RS}$.
Hence
$$
\mISA_{R}[i_{R}] < \mISA_{R}[j_{R}] \iff \mISA_{S}[i_{S}] < \mISA_{S}[j_{S}]
$$
for $1 \le i,j \le \abs{Y}$, which is equivalent to the condition in
Definition~\ref{def:bwt-invariant}. We can convert $\mAli(R,S)$ to
$\mAli(\mBWT_{R},\mBWT_{S})$ in $\Oh((\abs{R}+\abs{S}) \cdot t_{\mLF})$ time by
traversing $\mCSA_{R}$ and $\mCSA_{S}$ backwards. The resulting subsequence of
$\mBWT_{R}$ and $\mBWT_{S}$ is bwt-invariant.

Note that the full relative FM-index is more limited than the basic index,
because it does not handle \emph{substring moves} very well. Let $R = xy$ and
$S = yx$, for two random sequences $x$ and $y$ of length $n/2$ each. Because
$\mBWT_{R}$ and $\mBWT_{S}$ are very similar, we can expect to find a common
subsequence of length almost $n$. On the other hand, the length of the longest
bwt-invariant subsequence is around $n/2$, because we can either match the
suffixes of $x$ or the suffixes of $y$ in $R$ and $S$, but not both.


\section{Theories}
\label{theories}

The result of a session with the \HOL{} system is an object called a
{\it theory\/}.  This object is closely related to what a logician
would call a theory, but there are some differences arising from the
needs of mechanical proof.  A \HOL{} theory, like a logician's theory,
contains sets of types, constants, definitions and axioms.  In
addition, however, a \HOL{} theory contains an explicit list of
theorems that have been proved from the axioms and definitions.
Logicians normally do not need to distinguish theorems that have
actually been proved from those that could be proved, hence they do
not normally consider sets of proven theorems as part of a theory;
rather, they take the theorems of a theory to be the (often infinite)
set of all consequences of the axioms and definitions.  Another
difference between logicians' theories and \HOL{} theories is that, for
logicians, theories are relatively static objects, but in \HOL{} they
can be thought of as potentially extendable. For example, the \HOL\
system provides tools for adding to theories and combining theories.
A typical interaction with \HOL{} consists in combining some existing
theories, making some definitions, proving some theorems and then
saving the resulting new theory.

The purpose of the \HOL{} system is to provide tools to enable
well-formed theories to be constructed.  All the theorems of such
theories are logical consequences of the definitions and axioms of the
theory.  The \HOL{} system ensures that only well-formed theories can
be constructed by allowing theorems to be created by {\it formal
  proof\/} only.

A theory is represented in the \HOL{} system as a collection of SML
object files, called theory files.  Each file has a name of the form
$name$\ml{Theory.}$ext$, where $name$ is a string supplied by the
user, and $ext$ is one of \ml{sig}, \ml{sml}, \ml{ui}, \ml{uo}.

Theory files are structured hierarchically to represent sequences of
extensions of an initial theory called \ml{scratch}.  Each theory file
making up a theory records some types, constants, axioms and theorems,
together with pointers to other theory files called its {\it
  parents\/}.  This collection of reachable files is called the {\it
  ancestry\/} of the theory file. Axioms, definitions and theorems are
named in the \HOL{} system by two strings: the name of the theory file
where they are stored, together with a name within that file supplied
by the user.  Specifically, axioms, definitions and theorems are named
by a pair of strings $\langle thy,name\rangle$ where $thy$ is the name
of the theory current when the item was declared and $name$ is a
specific name supplied by the user (see the functions \ml{new\_axiom},
\ml{new\_definition} \etc\ below).

A typical piece of work with the \HOL{} system consists in a number of
sessions.  A theorem-proving session consists of interactions with the
system through its ``command-line interface''.  At the same time, it's
important that the commands the user enters are saved in some sort of
file (presumably through an editor) so that these commands can be
replayed later.  For the first sessions of theory development, each
time \HOL{} is started, the saved commands from previous sessions will
need to be re-entered into the system.  This can be done with
cut-and-paste functionality, or the \ml{use} command (which takes a
file-name and reads it for input).

Eventually (possibly even after the very first session; this is a
matter of taste), the theory under development will be in a
sufficiently polished state that the user will want to save it to a
``script'' file and compile it.  This process is described further in
the examples below, but produces a theory file that can be loaded in
the same way as the system's built-in theories.

There is always a {\it current theory\/}, whose name is given by the
function \ml{current\_theory}.  This function maps the unit value,
written `\ml{()}' in \ML, to a string giving the name of the current
theory.  Thus the \ML{} expression \ml{current\_theory ()} evaluates to
a string giving the name of the current theory.  Initially \HOL{} has
its current theory called \ml{scratch}.  In the examples so far, there
hasn't been a call to \ml{new\_theory}, so the current theory is still
``scratch''.

\begin{session}\begin{verbatim}
- current_theory();
> val it = "scratch" : string
-
\end{verbatim}\end{session}

    Executing \ml{new\_theory`$thy$`} creates a new theory called
    $thy$.  The remaining example sessions will demonstrate the
    construction of a theory of prime numbers.

\begin{session}\begin{verbatim}
- new_theory "primes";
> val it = () : unit
\end{verbatim}\end{session}

\noindent
This starts a theory called \ml{primes}, which is to be made into a
theory containing definitions and theorems about the prime numbers.
This will be developed further in chapter~\ref{euclid} to include
Euclid's proof that there are an infinite number of primes.

Because we loaded \ml{arithmeticTheory} and \ml{pairTheory} to begin,
both of these theories are part of our new theory's ancestry.  In
fact, there are a number of other ancestor theories that these two
depend on in turn.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "tutorial"
%%% End:


\chapter{Conclusion}

We have introduced a new method to perform causal inference in the bivariate additive
noise model. The method is simple, and is different from ANM methods in that it
exploits the residual directly rather than testing for independence between
the residual and input. This has the advantage of avoiding independence testing
which is known to be hard. The drawback is that we assume that the noise is independent
between each sample. The method is also theoretically sound as we were able to prove that
it is consistent asymptotically. While it is not as good as state of the art, it is
competitive: this shows that exploiting noise alone is also a viable
line of attack to infer the causal direction in the ANM setting.

% TODODODODODODODODO
% TODODODODODODODODO
% TODODODODODODODODO
% TODODODODODODODODO
% TODODODODODODODODO
% TODODODODODODODODO
% TODODODODODODODODO
% TODODODODODODODODO
% TODODODODODODODODO
% TODODODODODODODODO

% \subsection{TODO stuff}

% TODO (will remove this later, just as a reminder)
% + unsupervised
% + fast
% + no independence test needed
% + theoretically sound

% - iid on Z assumption
% - acc

% TODO:

% IGCI, at least mention a bit

% Some observations:

% While it could be $F(X, Z)$, often it might be $F(X, Z) + Z_2$ since there is often measurement error
% so noise additivity is a good assumption.

% Standardize residual notation! $e_x$...

% Briefly discuss AIC / model selection
% intution about using poly reg since it's local
% aprox
% https://stats.stackexchange.com/questions/9171/
% aic-or-p-value-which-one-to-choose-for-model-selection