\documentclass{article}
\usepackage{amsmath}
\begin{document}

\part {Sampled stabilization}

\section{Definitions}

Let $I_1, \dots, I_K$ be the $K$ inputs, and $B_k$ the set of points that are in input $I_k$.

We denote the geometric transformation between input and panorama space by $T_{I_k \rightarrow P} = T_{S \rightarrow P} \circ T_{I_k \rightarrow S}$,
where $S$ is the intermediate spheric space. Note that $T_{S \rightarrow P}$ does not depend on $k$, which we use in the following.

Let $r_k$ be the input sensor's inverse response for input $k$. Typical responses are:
\begin{itemize}
  \item Linear: $r_k(r, g, b) = (r, g, b)$
  \item Gamma: $r_k(r, g, b) = (r^\gamma, g^\gamma, b^\gamma)$
  \item (Inverse) Emor: $r_k(r, g, b) = (f(r), f(g), f(b))$ where $f(x) = f_0 + \sum_{i=1}^{5}{\mu_i h_i}$ and the $\mu_i$ are called \em{emor coefficients}.
\end{itemize}

Similary, the panorama output has a inverse response $R(r, g, b)$.

In addition, photometric correction happens in between $r_k$ and $R$.
This is linear transformation of the colors weighted by a vignetting function $v$ that depends on the distance from $p$ to the vignetting center $c_k$ of the input.
$d_k(p,r,g,b) = v_k(p) M_k (r,g,b)^\top$ . In our implementation $M_k$ is diagonal, and $v(p)$ is an inverse polynomial of degree $6$ with zero even coefficients.

\begin{equation}
  M_k = diag({m^r}_k 2^{e_k - e_P}, {m^g}_k 2^{e_k - e_P}, {m^b}_k 2^{e_k - e_P})
\end{equation}

\begin{equation}
  v_k(p) = \frac{1}{\nu_3 \rho_k^3 + \nu_2 \rho_k^2 + \nu_1 \rho_k + \nu_0}, \rho_k(p) = {\| p - c_k\|}^2
\end{equation}

Where the $e_k$ and $e_P$ are the exposure values of the inputs and the panorama respectively, and $\mathbf{m^r}$, $\mathbf{m^g}$ and $\mathbf{m^b}$ and the color correction coefficients.

If a point $p$ in input $k$ has a color $C_k(p) = (r(p), g(p), b(p))$, then the color of its image in the panorama just before blending
is
\begin{equation}
 C_P(T_{I_k \rightarrow P}(p)) = C_P(T_{I_k \rightarrow S}(p)) = R^{-1} \circ d_k(p) \circ r_k (C_k(p))
\end{equation}

Note that all $r_k$ and $R$ are monotonic of same direction along all components.

\section{Sampling}

For each input $k$, we draw points $p_{1,k}, p_{i,k}, \dots$ with a probability distribution $P$.

$P$ should be chosen such that there are more points around the image borders (where images have most chanches to overlap).

Let $p_{i,l}$ the projection of $p_{i,k}$ into input $l$. This is computable implicitly without doing explicit panorama
transformations $T_{I_k \rightarrow P}$ and $T_{I_l \rightarrow P}$ by applying only the input
space to spheric space transformation $T_{I_k \rightarrow S}$ followed by the spheric space to input space transformation $T^{-1}_{I_l \rightarrow S}$.

For each drawn point $p_{i,k}$, we compute $p_{i,l} = T^{-1}_{I_l \rightarrow S} \circ T_{I_k \rightarrow S} (p_{i,k})$ for all $l \in {1,\dots,K}$.
Note that by definition:

\begin{equation}
  T_{I_k \rightarrow P}(p_{i,k}) = T_{I_l \rightarrow P}(p_{i,l})
\end{equation}

Doing the computations implicitly in the input space instead of explicitly in the output space has a lot of advantages:

\begin{enumerate}
  \item {\bf Less computation:} No need to apply the pano transform transform, we only need to sample from the inputs.
  \item {\bf Smaller memory footprint:} No need to allocate output and mapping buffers, only the reader buffers are needed.
  \item {\bf Independant on the output projection:} Zero cases to handle instead of {\em equirect}, {\em stereographic},...
  \item {\bf Independant on the output transformation:} we don't care about any global tranformation since the formulation is independant of the output. Therefore we need to sample input points {\bf only once for the whole duration of the video}. This is the most important part.
\end{enumerate}

The {\em multipoint} $p_i$ is the set of images of $p_{i,1}$ in all inputs:

\begin{equation}
  p_i = \{p_{i,k} \mid  p_{i,k} \in B_k\}
\end{equation}

We use as sample points for computing exposure correction the points $p_i$ that link at least two images together:

\begin{equation}
  S = \{p_i \mid \#p_i >= 2\}
\end{equation}

To simplify notations, we reorder the $p_i$ such that $\forall 1 <= i <= N$, $p_i \in S$.

\section{Spacial exposure correction}

We're trying to minimize the spacial photometric error for each frame at time $t$:

\begin{align}
  E_S(\mathbf{e}_t) &= \sum_{i=1}^{N}{ \sum_{(p_{i,k}, p_{i,l}) \in p_i} { {\| C_P(T_{I_l \rightarrow S}(p_{i,k})) - C_P(T_{I_l \rightarrow S}(p_{i,l})) \|}^2 } } \\
                  &= \sum_{i=1}^{N}{ \sum_{(p_{i,k}, p_{i,l}) \in p_i} { {\| R^{-1} \circ d_k(p_{i,k}) \circ r_k (C_k(p_{i,k})) - R^{-1} \circ d_l(p_{i,l}) \circ r_l (C_l(p_{i,l})) \|}^2 } }
\end{align}

Where $C_P$ stands for the color in panorama space (e.g. the $(r, b, g)$ triple).

\section{Penalization}

There are more degrees of freedom that this minimization can impose, so we add a penalization term to that the sum of exposure values does not drift too far from their base state.

The classical penalization term:
\begin{equation}
  E_P(\mathbf{e}_t) = \sum_k{e_k^2}
\end{equation}
does not work here since the exposure values will just go down (less light, less error) until the penalization term above stops them at an arbitrary equilibrium value.

The following term is a lot better:
\begin{equation}
  E_P(\mathbf{e}_t) = {\frac{\sum_k{e_k}}{K}}^2
\end{equation}

The idea is to make sure that the average exposure value does not float away from zero, but individual exposures can.

\part {Histogram stabilization}

The idea is to compute histograms for each of the inputs of overlapping zones, and trying to make sure that histograms match.
The interesting property is that it's possible to compute these histograms with a specific merger.

The relationship between the color parameters and the histograms is non-linear (gamma or emor), so we need to compute the histogram in linear space and then,
for each optimization iteration, use the emor lookup table to convert the linear histogram into the final histogram.

\section{Definitions}

We call $\mathbf{{h^r}_{k,l}}$, $\mathbf{{h^g}_{k,l}}$, $\mathbf{{h^b}_{k,l}}$ the histogram of pixels in input $k$ that
overlap with pixels in output $l$ for color components $r$, $g$, and $b$ respectively:

\begin{equation}
  {h^r}_{k,l,i} = \sum_{(p, q) \in (I_k, I_l), T_{I_k \rightarrow P}(p) = T_{I_l \rightarrow P}(q)}{C_P(T_{I_k \rightarrow P}(p)) == i)}
\end{equation}

We're trying to find the exposure correction parameter values that minimize the discrepancies between histograms of overlapping inputs:

\begin{equation}
  E(\mathbf{e}_t) = \sum_{k,l}{\sum_c{d(\mathbf{{h^c}_{k,l}}, \mathbf{{h^c}_{l,k}})}}
\end{equation}

where $d$ is a histogram distance metric.

Note that since $R^{-1}$ is non-linear in the photometric transform $C_P$, the histogram is not simply translated by changes in the components of $\mathbf{e}_t$,
since each bin moves at a different rate. Therefore me must compute the histogram with $R^{-1}(c) = c$ and then integrate the histogram transformation in $d$.
$d$ then becomes a function of a pair of (histogram, transform) $(\mathbf{{h^r}_{k,l}}, {R_k}^{-1})$ and $(\mathbf{{h^r}_{l,k}}, {R_l}^{-1})$ instead of simply a pair of histograms.

\end{document}


\section{Practical implementation}
\label{sec:practical_impl}
In Section~\ref{sec:oracles}, we discuss estimators for learning rate and momentum tuning in \tuner. In our experiment practice, we have identified a few practical implementation details which are important for improving estimators. Zero-debias is proposed by~\citet{kingma2014adam}, which accelerates the process where exponential average adapts to the level of original quantity in the beginning. We applied zero-debias to all the exponential average quantities involved in our estimators. In some LSTM models, we observe that our estimated curvature may decrease quickly along the optimization process. In order to better estimate extremal curvature $h_{\max}$ and $h_{\min}$ with fast decreasing trend, we apply zero-debias exponential average on the logarithmic of $h_{\max, t}$ and $h_{\min, t}$, instead of directly on $h_{\max, t}$ and $h_{\min, t}$. Except from the above two techniques, we also implemented the slow start heuristic proposed by~\citep{schaul2013no}. More specifically, we use $\alpha = \min\{\alpha_t, t \cdot \alpha_t / (10 \cdot w) \}$ as our learning rate with $w$ as the size of our sliding window in $h_{\max}$ and $h_{\min}$ estimation. It discount the learning rate in the first $10 \cdot w$ steps and helps to keep the learning rate small in the beginning when the exponential averaged quantities are not accurate enough.


% Section 2
% 2021-08-19
% Alessandro Zanatta

\section{Tools description}
\label{section:foundations}

In this section we are going to see a brief overview on the syntax and foundations of Tamarin, Proverif and Verifpal - the three tools we are going to compare.

\subsection{Tamarin prover}
Let us start with Tamarin prover. For a more
in-depth description and further information, see \cite{TamarinFoundations, TamarinFoundationsExtended, TamarinManual}.

The security property model of Tamarin is based on \textbf{labelled multiset rewriting rules} to specify protocols and adversary capabilities, a \textbf{guarded fragment\footnote{The guarded fragment used by Tamarin is basically a subset of formulas from the first order logic with additional constraints on the arguments. See \cite{FragmentFirstOrderLogicPaper} for a definition from a mathematical point of view.} of first order logic} to specify security properties and functions and \textbf{equational theories} to model the algebraic properties of cryptographic protocols \cite{TamarinFoundations}. Additionally, every event in the security properties is annotated with a timepoint $t \in \Q$ and basic comparison of timepoints can be used.

Tamarin then applies a constraint-solving algorithm based on \textbf{backward-search and heuristics} which tries to validate or falsify security properties.

\subsubsection{Multiset rewriting system and state transitions}

\paragraph{Multiset rewriting ingredients} The ingredients of Tamarin multiset rewriting system are the following:

\lstset{language=tamarin}
\begin{itemize}
    \item{\textbf{Terms} - which can be thought of as messages;}
    \item{\textbf{Facts} - which model information in the protocol and are composed by terms. Two built-in facts are used to model network I/O: \lstinline{In(x)} and \lstinline{Out(x)}. \lstinline{K(x)} is used to model attacker knowledge. Finally, \lstinline{Fr(x)} is used to model a fresh value \lstinline{x};}
    \item{\textbf{State of the system} - which is represented using a \textit{multiset} of facts;}
    \item{\textbf{Transition rules} - which defines the possible transitions from one state to another. Multiset rewriting rules are written as $\msr{L}{A}{R}$, where $L$, $A$ and $R$ are multisets of facts, respectively called premises, actions and conclusions;}
    \item{\textbf{Trace} - a sequence $\left<A_1, \dots, A_n\right>$ of sets of ground facts (i.e. facts which do not contain any variable) denoting the sequence of \textit{actions} that happened during a protocol execution.}
\end{itemize}

\paragraph{State transitions}
Consider a ground multiset rewriting rule $\msr{l}{a}{r}$. If we call the state of the system $S$, the trace $T$ and assuming $l \msrsubseteq S$, the new state of the system $S'$ can be defined as $S' = S \msrsetminus l \msrcup r$. Additionally, we append $a$ to the end of the current trace $T$. \textsuperscript{\#} denotes multisets' set operations.

\subsection{Proverif}
\begin{figure}[t]
    \includegraphics[scale=.8]{proverif-verification-method}
    \centering
    \caption{Proverif verification method.\\Inspired by a representation from Bruno Blanchet \cite{SymbolicComputationalBlanchet}.}
    \label{fig:proverif-verification-method}
\end{figure}
Proverif protocols and security properties are based on an extended version of the \pic (the \textbf{applied \picnospace}). The tool also allows the user to define \textbf{constructors, destructors and equations}\footnote{Destructors are basically used to de-construct some previously constructed term (e.g. decryption of an encrypted ciphertext), while equations represent term equality of some sort (e.g. commutativity of multiplication).}, which form the cryptographic primitives. The protocol is then automatically translated to a \textbf{set of Horn-clauses}. Using this abstract representation of the protocol (based on Horn-clauses), the Proverif verifier uses a \textbf{resolution algorithm} on such clauses that allows for verification of security properties \cite{SymbolicComputationalBlanchet}.
A graphical representation of the entire process is given in \cref{fig:proverif-verification-method}.

A brief definition of the grammar of processes of the applied \pic is given in \cref{lst:apic-processes}, slightly modified to match the one actually used by Proverif.

\lstset{language=proverif}
\begin{lstlisting}[caption={Applied \pic grammar of processes.}, label={lst:apic-processes}]
0                    (* null process *)
out(N, M); P         (* output to channel N the message M *)
in(N, M: T); P       (* input from channel N of message M of sort T *)
P | Q                (* parallel composition *)
!P                   (* infinite replication *)
new a: T; P          (* fresh value of sort T *)
if M then P else Q   (* conditional*)
\end{lstlisting}

\lstset{language=proverif}
The null process \lstinline{0} does nothing;
\lstinline{out(N, M); P} (\lstinline{in(N, M: T); P}) outputs (gets) the message \lstinline{M} (of sort \lstinline{T}) into (from) channel \lstinline{N} and then continues with process \lstinline{P};
\lstinline{P | Q} is the parallel composition of \lstinline{P} and \lstinline{Q};
The process \lstinline{!P} effectively behaves as an infinite number of copies of \lstinline{P} running in parallel (\textit{unbounded} replication);
\lstinline{new a: T; P} creates a new fresh value of sort \lstinline{T}, before proceeding with process \lstinline{P};
\lstinline{if M then P else Q} is a standard conditional.

\lstset{language=proverif}
There are many additions to this grammar, such as:
\begin{itemize}
    \item{\lstinline{event EventName(...);} - allows to define a trace of events on which security properties can be defined;}
    \item{\lstinline{query event(EventName(...)).} - queries are used to define security properties. The reserved word \lstinline{attacker(x)} allows to ask Proverif if the attacker knows the term \lstinline{x};}
    \item{\lstinline{phase t;} - allows to execute a process only after processes of phases \lstinline{< t} have been executed. Intuitively, \lstinline{t} can be thought of as a global clock and a process in phase \lstinline{t}  is active only during phase \lstinline{t}. Processes without an explicit phase belong to phase 0;}
    \item{\lstinline{let MacroName = P.} - allows to create a process macro, which is simply substituted when using the name \lstinline{MacroName}.}
\end{itemize}

\begin{figure}[t]
    \makebox[\textwidth][c]{\includegraphics[scale=.55]{verifpal-internals}}
    \centering
    \caption{Verification process of Verifpal.\\ All credits to Nadim Kobeissi \cite{VerifpalManual}.}
    \label{fig:verifpal-verification}
\end{figure}

\newpage
\subsection{Verifpal}

Let us start with Verifpal syntax by explaining the minimal working example of \cref{lst:verifpal-minimal}.

\lstset{language=verifpal}
\begin{lstlisting}[numbers=left,caption={Verifpal's minimal working example}, label={lst:verifpal-minimal}]
attacker [active]

principal Alice []
principal Bob []

queries []
\end{lstlisting}

The \lstinline{attacker} can either be \lstinline{active} or \lstinline{passive}.
Principals are the honest parties of the protocol. Many constructs can be used between square brackets, such as \lstinline{generates}, \lstinline{knows}, assignments and cryptographic primitives.
Finally, in the \lstinline{queries} block we express security properties we want to check. Queries in Verifpal are limited to pre-defined ones: confidentiality, authentication, freshness, unlinkability and equivalence.
Verifpal also supports phases, like Proverif.

Verifpal does not support user-defined cryptographic primitives. Instead, it supports most of the conventional equational theories, such as Diffie-Hellman, (a)symmetric encryption, authenticated encryption, signatures, blinding and many more.

\subsubsection{Verifpal solving algorithm}

\Cref{fig:verifpal-verification} shows the process of protocol verification used by Verifpal. Let us briefly describe the 5 main steps:

\begin{enumerate}
    \item{\textbf{Gather values} - first of all, the attacker passively observes a protocol execution and gathers all the values shared on the public channel between parties;}
    \item{\textbf{Populate attacker state} - gathered values are inserted into the attacker's state;}
    \item{\textbf{Apply transformations} - the attacker applies the four transformations on all of the obtained value (see bottom-right squares);}
    \item{\textbf{Prepare next mutations} - if the attacker has learnt new values, it creates a combinatorial table of all possible substitutions and uses it to derive a set of possible value substitutions;}
    \item{\textbf{Mutate protocol executions} - finally, the attacker proceeds to execute the protocol, each time applying a mutation from the previous step. As long as the attacker learns new values, it returns to step 1.}
\end{enumerate}

Verifpal, after each stage, checks if any defined security property has been falsified (e.g. the attacker state contains a certain term).
Notice that many other techniques and heuristics are used to avoid state space explosion (e.g. a space state too large to explore in reasonable time).


\section{Geometry}
\subsection{2D}
\subsubsection{Point}
\lstinputlisting{"./geometry/2d_point.cc"}
\subsubsection{Convex hull}
\lstinputlisting{"./geometry/2d_convex.cc"}
\subsubsection{Intersect Area}
\lstinputlisting{"./geometry/2d_area.cc"}
\subsubsection{Universe}
\lstinputlisting{"./geometry/2d_universe.cc"}


\documentclass{article}
\usepackage[utf8x]{inputenc} % codifica scrittura
\usepackage[nochapters]{classicthesis} % nochapters

\usepackage[T1]{fontenc}
\usepackage[square,numbers]{natbib}
\usepackage{amsmath, amsthm, amssymb, amsfonts, tikz}
\usetikzlibrary{snakes}
\usepackage{titletoc}
\usepackage{verbatim}
\usepackage{hyperref}
\usepackage{boxedminipage}

\titlecontents{section}[3em]{}{\contentslabel{1em}}{}{\titlerule*[1.5pc]{.}\contentspage}
\titlecontents{subsection}[6em]{}{\contentslabel{2em}}{}{\titlerule*[1.5pc]{.}\contentspage}

\begin{document}

\title{\rmfamily\normalfont\spacedallcaps{Computer Aided Graphic
    Design course exercises}}

\author{\spacedlowsmallcaps{Massimo Nocentini}}
\date{\today}

\maketitle

\begin{abstract}
  This document contains some exercises and collects my work done during
  the CAGD course given by Prof. Alessandra Sestini and Prof. Costanza
  Conti at University of Florence.

  In particular, here we collect exercises requested by Prof.
  Costanza Conti about Bezier and BSplines curves. We implement numerical
  methods using Julia  language \cite{Julia} and everything (code for
  solving exercises and \TeX sources of this document) is under
  version control, available as open source Git repository
  \footnote{Hosted on \url{http://github.com/massimo-nocentini/cagd}},
  under \emph{MIT License}.
\end{abstract}

\tableofcontents

\newpage

\section{Bezier curves}

\subsection{Curve from simple set of control points}
In \autoref{fig:first-closed-curve} we report the very first Bezier
curve obtained using our implementation. The curve is obtained using control
points $(1,1), (3,4), (5,6),(7,8),(10,2),(1,1)$ in the given
order. This plot was the first test for our implementation of code
reported in Exercise 1 and requested in Exercise 2: it contains a
segmented curve in green which is the control polygon, and a Bezier
curve in red built using the given control points.
\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-one}
  \caption{Curve from simple set of control points}
  \label{fig:first-closed-curve}
\end{figure}

\subsection{Curve from parametric specification}
As required in Exercise 3, in \autoref{fig:curve-from-parametric-spec}
we report a Bezier curve matching the following parametric
specification:
\begin{displaymath}
  \left [  \begin{array}{c}
      x(u) \\
      y(u)
    \end{array} \right ] = \left [
    \begin{array}{c}
      1 + u + u^2 \\
      u^3
    \end{array} \right ]
\end{displaymath}
with $u\in[0,1]$. In order to find the control polygon we do simple
reductions with \emph{Maxima}:

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
a:v0*(1-t)^3 + v1*3*t*(1-t)^2 + v2*3*(t^2)*(1-t) + v3*t^3;
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o1) }
{t}^{3}\,v3+3\,\left( 1−t\right) \,{t}^{2}\,v2+3\,{\left( 1−t\right) }^{2}\,t\,v1+{\left( 1−t\right) }^{3}\,v0
\end{math}

\noindent
%%%%%%%%%%%%%%%
% %%% INPUT:
% \begin{minipage}[t]{8ex}{\color{red}\bf
% \begin{verbatim}
% (%i2)
% \end{verbatim}}
% \end{minipage}
% \begin{minipage}[t]{\textwidth}{\color{blue}
\begin{verbatim}
b:ratsimp(a,t);
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o2) }
{t}^{3}\,\left( v3−3\,v2+3\,v1−v0\right) +{t}^{2}\,\left( 3\,v2−6\,v1+3\,v0\right) +t\,\left( 3\,v1−3\,v0\right) +v0
\end{math}

\noindent
%%% INPUT:
\begin{verbatim}
b = 1 + t + t^2 + 0*t^3;
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o3) }
{t}^{3}\,\left( v3−3\,v2+3\,v1−v0\right) +{t}^{2}\,\left( 3\,v2−6\,v1+3\,v0\right) +t\,\left( 3\,v1−3\,v0\right) +v0={t}^{2}+t+1
\end{math}

\noindent
%%% INPUT:
\begin{verbatim}
solve([coeff(b, t,3) = 0, coeff(b, t,2) = 1, coeff(b, t,1) = 1,
        coeff(b, t,0) = 1], [v0,v1,v2,v3]);
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o4) }
[[v0=1,v1=\frac{4}{3},v2=2,v3=3]]
\end{math}
%%%%%%%%%%%%%%%

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
a:y0*(1-t)^3 + y1*3*t*(1-t)^2 + y2*3*(t^2)*(1-t) + y3*t^3;
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o5) }
{t}^{3}\,y3+3\,\left( 1−t\right) \,{t}^{2}\,y2+3\,{\left( 1−t\right) }^{2}\,t\,y1+{\left( 1−t\right) }^{3}\,y0
\end{math}
%%%%%%%%%%%%%%%

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
b:ratsimp(a,t);
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o6) }
{t}^{3}\,\left( y3−3\,y2+3\,y1−y0\right) +{t}^{2}\,\left( 3\,y2−6\,y1+3\,y0\right) +t\,\left( 3\,y1−3\,y0\right) +y0
\end{math}
%%%%%%%%%%%%%%%

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
b = t^3;
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}
\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o7) }
{t}^{3}\,\left( y3−3\,y2+3\,y1−y0\right) +{t}^{2}\,\left( 3\,y2−6\,y1+3\,y0\right) +t\,\left( 3\,y1−3\,y0\right) +y0={t}^{3}
\end{math}
%%%%%%%%%%%%%%%

\noindent
%%%%%%%%%%%%%%%
%%% INPUT:
\begin{verbatim}
solve([coeff(b, t,3) = 1, coeff(b, t,2) = 0, coeff(b, t,1) = 0,
        coeff(b, t,0) = 0], [y0,y1,y2,y3]);
\end{verbatim}}
%%% OUTPUT:
\definecolor{labelcolor}{RGB}{100,0,0}

\begin{math}\displaystyle
\parbox{8ex}{\color{labelcolor}(\%o8) }
[[y0=0,y1=0,y2=0,y3=1]]
\end{math}
%%%%%%%%%%%%%%%

Hence four control points are $\{(1,0), (\frac{3}{4},0), (2,0),
(3,1)\}$.
\begin{figure}
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-two}
  \caption{Curve from parametric specification}
  \label{fig:curve-from-parametric-spec}
\end{figure}

\subsection{Splitted curve on a given parameter $\hat{t}$}
As required in Exercise 4, in \autoref{fig:splitted-curve} we report
the control polygon for the original curve in red, and two sets of
control points with the same cardinality that, used together, build a
Bezier that is the same as the original one. Those sets of points are
obtained via subdivision algorithm (which is a clever implementation of
classic \emph{de Casteljau algorithm}) fixing parameter $\hat{t} =
\frac{1}{4}$: they are colored in green (the relative Bezier in magenta)
and in blue (the relative Bezier in cyan), respectively.
\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-four}
  \caption{Splitted curve}
  \label{fig:splitted-curve}
\end{figure}

\subsection{Repeating the same control point more times}
As required in Exercise 5, in
\autoref{fig:repeating-same-control-point} we report two Bezier curves: the
red one relative to control points $\{(2,4), (6,12), (10,1),
(12,12)\}$ the green one relative to control points $\{(2,4), (6,12),
(10,1), (10,1), (10,1), (10,1), (12,12)\}$, ie. with the point
$(10,1)$ repeated three more times. We see that curve relative to
the augmented control polygon goes down toward $(10,1)$ more than the
other curve: this can be explained from a probabilistic point of
view, since a Bezier curve can be thought as a \emph{mean} of the control
polygon, hence repeating point $(10,1)$ increases its weight.

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-five}
  \caption{Repeating the control point $(10,1)$ three more times}
  \label{fig:repeating-same-control-point}
\end{figure}

\subsection{Increasing degree}
As required in Exercise 6, we start from an original set of control
points, plotted in \autoref{fig:increasing-degree-original-curve}, and
we proceed by increasing degrees of successive Bezier curves three times.
We obtains three augmented set of control points, plotted in
\autoref{fig:some-increased-degrees}, respectively. It is possible to
check the slow convergence for the sequence of polygons to the Bezier
curve and, in \autoref{fig:increasing-degree-does-change-curve}, the
Bezier curves relative to each augmented polygons doesn't change in
shape, ie their set of points equal the original one (we simply plot
those Bezier curves in the same plot and each one is over the others).

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-six-original}
  \caption{Original curve before increasing degree}
  \label{fig:increasing-degree-original-curve}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-six-higher-degree-control-poly}
  \caption{Some control polygons, each one with one more degree}
  \label{fig:some-increased-degrees}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-six-one-more-degree-comparison}
  \caption{Increasing degree doesn't change the Bezier shape}
  \label{fig:increasing-degree-does-change-curve}
\end{figure}

\subsection{Joining curve requiring $\mathcal{C}^0, \mathcal{C}^1, \mathcal{C}^2$}

In this section we report plots about Bezier spline curves,
extending a fixed control net, requiring $\mathcal{C}^0,
\mathcal{C}^1, \mathcal{C}^2$ for each extension, both on the right
(see
\autoref{fig:bezier-spline-right-extension-continuity},
\autoref{fig:bezier-spline-right-extension-tangent},
\autoref{fig:bezier-spline-right-extension-obsculating},
respectively) both on the left (see
\autoref{fig:bezier-spline-left-extension-continuity},
\autoref{fig:bezier-spline-left-extension-tangent},
\autoref{fig:bezier-spline-left-extension-obsculating},
respectively).

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-continuity}
  \caption{Bezier spline curve, requiring $\mathcal{C}^0$, extending on the right}
  \label{fig:bezier-spline-right-extension-continuity}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-tangent}
  \caption{Bezier spline curve, requiring $\mathcal{C}^1$, extending on the right}
  \label{fig:bezier-spline-right-extension-tangent}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-obsculating}
  \caption{Bezier spline curve, requiring $\mathcal{C}^2$, extending on the right}
  \label{fig:bezier-spline-right-extension-obsculating}
\end{figure}

% \includegraphics{bezier-deCasteljau-curves/exercise-seven-a_succ_i}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-continuity-left}
  \caption{Bezier spline curve, requiring $\mathcal{C}^0$, extending on the left}
  \label{fig:bezier-spline-left-extension-continuity}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-tangent-left}
  \caption{Bezier spline curve, requiring $\mathcal{C}^1$, extending on the left}
  \label{fig:bezier-spline-left-extension-tangent}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-seven-obsculating-left}
  \caption{Bezier spline curve, requiring $\mathcal{C}^2$, extending on the left}
  \label{fig:bezier-spline-left-extension-obsculating}
\end{figure}

% \includegraphics{bezier-deCasteljau-curves/exercise-seven-a_i-left}

\newpage

\subsection{First Polar of a Bezier curve}

In this section we elaborate an extra exercise relative to first derivative
of a Bezier curve and the first step of de Casteljau algorithm.

Let $\mathbf{p}_{1}(t)$ a Bezier curve built over a control net with $n$ points
$\mathbf{b}_{0}^{(1)},\ldots,\mathbf{b}_{n-1}^{(1)}$, respect a fixed parameter $\hat{t}$
after one step of de Casteljau algorithm. Just use the formal definition:
\begin{displaymath}
    \begin{split}
        \mathbf{p}_{1,\hat{t}}(t)  &= \sum_{i=0}^{n-1}{\mathbf{b}_{i}^{(1)}(\hat{t})B_{i}^{n-1}(t)} \\
            &=  \sum_{i=0}^{n-1}{\left( (1-\hat{t})\mathbf{b}_{i}^{(0)} +
                \hat{t}\mathbf{b}_{i+1}^{(0)}\right)B_{i}^{n-1}(t)} \\
            &=  \sum_{i=0}^{n-1}{\left( (1-\hat{t})\mathbf{b}_{i}^{(0)} +
                \hat{t}\mathbf{b}_{i+1}^{(0)} -\left( (1-t)\mathbf{b}_{i}^{(0)} +
                t\mathbf{b}_{i+1}^{(0)}\right)\right)B_{i}^{n-1}(t)} +
                \sum_{i=0}^{n-1}{\mathbf{b}_{i}^{(1)}(t)B_{i}^{n-1}(t)} \\
            &=  \sum_{i=0}^{n-1}{\left( (t-\hat{t})\mathbf{b}_{i}^{(0)} +
                (\hat{t}-t)\mathbf{b}_{i+1}^{(0)}\right)B_{i}^{n-1}(t)} +
                \sum_{i=0}^{n-1}{\mathbf{b}_{i}^{(1)}(t)B_{i}^{n-1}(t)} \\
            &=  (\hat{t}-t)\sum_{i=0}^{n-1}{\left(
                \mathbf{b}_{i+1}^{(0)} - \mathbf{b}_{i}^{(0)}\right)B_{i}^{n-1}(t)} +
                \sum_{i=0}^{n-1}{\mathbf{b}_{i}^{(1)}(t)B_{i}^{n-1}(t)} \\
    \end{split}
\end{displaymath}
where, with little abuse of notation, $\mathbf{b}_{i}^{(1)}(t) =
\left( (1-t)\mathbf{b}_{i}^{(0)} + t\mathbf{b}_{i+1}^{(0)}\right)$.
Recall that $n\sum_{i=0}^{n-1}{\left(
\mathbf{b}_{i+1}^{(0)} - \mathbf{b}_{i}^{(0)}\right)B_{i}^{n-1}(t)}$ is the
first derivative of a Bezier curve, hence:
\begin{displaymath}
    \begin{split}
        \mathbf{p}_{1, \hat{t}}(t)  &= \mathbf{b}_{1, t}(t) +
        \frac{\hat{t}-t}{n}\frac{\partial \mathbf{b}_{0, \hat{t}}(t) }{\partial t}
    \end{split}
\end{displaymath}
So, this is called the ``first polar form'' of Bezier curve $\mathbf{b}_{0, \hat{t}}(t)$
respect parameter $\hat{t}$. Geometrically, the term $\mathbf{b}_{1, t}(t)$ is an affine
combination of points $\mathbf{b}_{0}^{(0)},\ldots,\mathbf{b}_{n}^{(0)}$ respect parameter
$t$ (not $\hat{t}$), hence it is a point also; the term
        $\frac{\hat{t}-t}{n}\frac{\partial \mathbf{b}_{0, \hat{t}}(t) }{\partial t} $
is a vector, so $\mathbf{p}_{1, \hat{t}}(t)$ is a vector applied to a point, yielding a point
as required. It is quite interesting how the polar form combine a point produced using $t$
as parameter with the derivative vector produced using  parameter $\hat{t}$. This derivation
comes from \cite{Farin}, page 73, and we report the output of our implementation in
\autoref{fig:exercise-polar-form}.

\begin{figure}[h!]
  \centering
  \includegraphics{bezier-deCasteljau-curves/exercise-polar}
  \caption{First Polar form of a Bezier curve}
  \label{fig:exercise-polar-form}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newpage

\section{B-Splines curves}

\subsection{Mushrooms from clumped, uniformed and closed partitions}

In this section we report three simple BSpline curves, using
three different knots partitions: \emph{clumped, uniform} and
\emph{cyclic} (see \autoref{fig:clumpled-mushroom},
    \autoref{fig:uniformed-mushroom} and \autoref{fig:closed-mushroom}
respectively). The control polygon doesn't change
  and aims to produce a mushroom in cartoon style, each knot
has multiplicity $1$ (so maximum continuity is required in each junction)
and no control point is repeated.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-zero-clumped.eps}
  \caption{Mushroom from clumped knots partition}
  \label{fig:clumpled-mushroom}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-zero-uniformed.eps}
  \caption{Mushroom from uniformed knots partition}
  \label{fig:uniformed-mushroom}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-zero-closed.eps}
  \caption{Mushroom from cyclic knots partition}
  \label{fig:closed-mushroom}
\end{figure}

\subsection{Increasing order $k$ while decreasing \emph{continuity} vector }
As Exercise 1 requires, in \autoref{fig:bspline-exercise-one-clumped} we
report five BSpline curves, each one of them drawn against the same control
net. We modify for each curve its knots partition, formally for curves
$\mathbf{c}_{1},\mathbf{c}_{2},\mathbf{c}_{3},\mathbf{c}_{4}$ and $\mathbf{c}_{5}$
the following extended partitions are used:
\begin{displaymath}
    \begin{split}
        \Delta_{1} &= \lbrace 0,0, \frac{1}{5}, \frac{2}{5},
            \frac{3}{5},\frac{4}{5},1,1 \rbrace \\
        \Delta_{2} &= \lbrace 0,0,0, \frac{1}{4},
            \frac{1}{2},\frac{3}{4},1,1,1 \rbrace \\
        \Delta_{3} &= \lbrace 0,0,0,0,
            \frac{1}{3},\frac{2}{3},1,1,1,1 \rbrace \\
        \Delta_{4} &= \lbrace 0,0,0,0,0,\frac{1}{2},
            1,1,1,1,1 \rbrace \\
        \Delta_{5} &= \lbrace 0,0,0,0,0,0,1,1,1,1,1,1 \rbrace \\
    \end{split}
\end{displaymath}
respectively, and each curve $\mathbf{c}_{i}$ has \emph{degree} $i$. As
we can see, each knots partition is clumped, so the first and last control points
are interpolated and the sequence of curves uses partitions with knots $0$ and $1$
repeated one more time in successive partitions. For lower degree curves, such as
$\mathbf{c}_{1}$, they are quite next to control net, in particular $\mathbf{c}_{1}$
is the control net itself, while curve $\mathbf{c}_{2}$ is tangent to control net
in two segments since it has \emph{order} $3$ and for $t \in [\frac{1}{4}, \frac{1}{2}) =
[t_4, t_5)$ curve $\mathbf{c}_{2}$ has support $[t_2,t_3,t_4]$ where knot $0$
has multiplicity $2$, hence $\frac{\partial \mathbf{c}_{2}(t)}{\partial t}$
lies on direction given by $\mathbf{V}_{3} - \mathbf{V}_{2}$.
The same reasoning can be done for knot $1$.
Finally, the very last partition allow to draw a \emph{Bezier} curve (the yellow one).

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-one-clumped.eps}
  \caption{Decreasing the \emph{continuity} vector collapsing in a Bezier }
  \label{fig:bspline-exercise-one-clumped}
\end{figure}

\subsection{Knots' multiplicities against the same control polygon}
In \autoref{fig:bspline-exercise-two} we report plots required by Exercise 2.
This composite picture contains 5 curves according the following specifications,
from top to bottom:
\begin{center}
    \begin{tabular}{ c c }
        order & knots partition \\
        \hline
        4 & $\lbrace 0^{4},1,2,3,4,5,6,7,8,9,10 \rbrace$  \\
        4 & $\lbrace 0^{4},1^{2},2^{2},3^{2},4^{4} \rbrace$  \\
        6 & $\lbrace 0^{6},1,2,3,4,5^{6} \rbrace$  \\
        6 & $\lbrace 0^{6},1,2,3^{2},4^{6} \rbrace$  \\
        8 & $\lbrace 0^{8},1,2,3^{8} \rbrace$  \\
    \end{tabular}
\end{center}
The curve relative to the first row interpolate the first control point
since $0$ has multiplicity $4$ which is the curve's order, while the last
control point isn't interpolated since knots partition ends uniformly.

The curve relative to the second row both interpolate the first control
point due to the multiplicity $k$ of $0$ and is tangent to control net
on the direction $\mathbf{V}_{4} - \mathbf{V}_{3}$ since for $t \in [t_{5}, t_{6}]$
the support is $[t_{2},t_{3},t_{4},t_{5}]$, where $0$ has multiplicity $3 = order -1$.

Curves relative to third and forth rows both interpolate first and last control
points because of clumped partition, while the last one reduce to only two
knots $1,2$, toward Bezier representation.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-two.eps}
  \caption{Knots's multiplicities increased against the same control polygon }
  \label{fig:bspline-exercise-two}
\end{figure}

\subsection{Linearity near doubled control point with a clumped partition}
In \autoref{fig:bspline-exercise-three} we plot curve required for Exercise 3:
a quadratic curve ($order = 3$) with a doubled control point $\mathbf{V}_{3} =
\mathbf{V}_{4} = (1,1)$ over an extended partition
$\Delta = \lbrace 0,0,0, \frac{1}{4},\frac{1}{2},\frac{3}{4},
1,1,1 \rbrace$, a clumped one with no repeated knots.

Lets consider the subcurve $\mathbf{c}_{5}(t)$ for $t \in [t_{5}, t_{6}]$:
\begin{displaymath}
    \mathbf{c}_{5}(t) = \sum_{i=3}^{5}{\mathbf{V}_{i}N_{i,3}(t)}
\end{displaymath}
Since $\mathbf{V}_{4} = \mathbf{V}_{3}$, for $t \in [t_{5}, t_{6}]$,
$\mathbf{c}_{5}(t)$ must lie on the convex hull $\mathbf{V}_{3}, \mathbf{V}_{4},
\mathbf{V}_{5}$ and,  for $t \in [t_{4}, t_{5}]$, $\mathbf{c}_{5}(t)$ must lie
on the convex hull $\mathbf{V}_{2}, \mathbf{V}_{3}, \mathbf{V}_{4}$, it does follow
that $\mathbf{c}_{5}(t_{5})$ has to lie in their intersection, therefore vertex
$\mathbf{V}_{4}$, interpolating it.

Moreover, $\mathbf{c}_{5}(t)$ has two linear segment between
$\mathbf{V}_{2}, \mathbf{V}_{3}$ and $\mathbf{V}_{4}, \mathbf{V}_{5}$,
both of them join in $t_{4}, t_{6}$ with continuity $\mathcal{C}^{1}$.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-three.eps}
  \caption{Linearity toward $(1,1)$ due to ``linearized'' convex hull}
  \label{fig:bspline-exercise-three}
\end{figure}

\subsection{Two clumped partitions, different \emph{continuity} vectors
    and same control polygon}
In \autoref{fig:bspline-exercise-four} we report two curves required by Exercise 4.
Both of them have $order = 4$ and a control net with a doubled point
$\mathbf{V}_{3} = \mathbf{V}_{4}$, while the former (colored green)
is over a knots partition
$\Delta_{1} = \lbrace 0,0,0,0,\frac{1}{4}, \frac{3}{4}, 1,1,1,1 \rbrace$
(having simple knots, no repeated one), the latter (colored blue) is over
a knots partition
$\Delta_{2} = \lbrace 0,0,0,0,\frac{1}{2}, \frac{1}{2}, 1,1,1,1 \rbrace$ (having
knot $\frac{1}{2}$ doubled). Let define the following subcurves of the former:
\begin{displaymath}
    \begin{split}
        \mathbf{c}_{4}(t) &= \sum_{i=1}^{4}{\mathbf{V}_{i}N_{i,4}(t)}, \quad
            t \in [t_{4}, t_{5}] \\
        \mathbf{c}_{5}(t) &= \sum_{i=2}^{5}{\mathbf{V}_{i}N_{i,4}(t)}, \quad
            t \in [t_{5}, t_{6}] \\
        \mathbf{c}_{6}(t) &= \sum_{i=3}^{6}{\mathbf{V}_{i}N_{i,4}(t)}, \quad
            t \in [t_{6}, t_{7}] \\
    \end{split}
\end{displaymath}
As we saw during classes \cite{Conti}, $\mathbf{c}_{5}^{\prime}(t_{5}) =
\mathbf{c}_{4}^{\prime}(t_{5})$ holds, so the
curve interpolate a point on the segment with extrema $\mathbf{V}_{2}, \mathbf{V}_{3}$
and it is tangent to the control net in $t_{5}$. On the other hand,
$\mathbf{c}_{6}^{\prime}(t_{6}) = \mathbf{c}_{5}^{\prime}(t_{6})$, so the
curve interpolate a point on the segment with extrema $\mathbf{V}_{4}, \mathbf{V}_{5}$
and it is tangent to the control net in $t_{6}$.
Finally, observe that the curve doesn't
interpolate the doubled control point $\mathbf{V}_{3} = \mathbf{V}_{4}$.

The previous argument doesn't apply to the second curve because its
knots partition is not simple, it contains a double knot.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-four.eps}
  \caption{Same control polygon with doubled $(0,6)$ againts two clumped partitions}
  \label{fig:bspline-exercise-four}
\end{figure}

\subsection{Increasing clumped partitions for increasing occurrences of a control point}
In Exercise 5 it is required to provide three extended knots partitions in
order to draw cubic curves reported in \autoref{fig:bspline-exercise-five}.
Each curve is defined against the ``same'' control net, having the middle vertex
of the top row repeated one, two, three times respectively. Our choice of knots
partitions, each clumped since interpolation of first and last control points
is requested:
\begin{displaymath}
    \begin{split}
        \Delta_{1} &= \lbrace 0,0,0,0,\frac{1}{2},1,1,1,1 \rbrace \\
        \Delta_{2} &= \lbrace 0,0,0,0,\frac{1}{3},\frac{2}{3},1,1,1,1 \rbrace \\
        \Delta_{3} &= \lbrace 0,0,0,0,\frac{1}{4},\frac{1}{2},\frac{3}{4},1,1,1,1 \rbrace \\
    \end{split}
\end{displaymath}

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-five.eps}
  \caption{Three B-Splines, each for $1,2,3$ occurrences of $(0,1)$ respectively }
  \label{fig:bspline-exercise-five}
\end{figure}

\subsection{Two B-Splines from two closed partitions}
In this last section we report two closed curves having $order = 4$ over
two cyclic knots partitions, in \autoref{fig:bspline-exercise-six-first}
and \autoref{fig:bspline-exercise-six-second}, respectively.

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-six-first-closed.eps}
  \caption{A first B-Spline from a closed partition }
  \label{fig:bspline-exercise-six-first}
\end{figure}

\begin{figure}[h!]
  \centering
  \includegraphics{b-splines/exercise-six-second-closed.eps}
  \caption{A second B-Spline from a closed partition }
  \label{fig:bspline-exercise-six-second}
\end{figure}

\newpage

\begin{thebibliography}{}

\bibitem{Julia} Open Source Project,
  \emph{Julia language}, \url{http://julialang.org/}

\bibitem{Farin} Gerald Farin,
  \textit{Curves and surfaces for CAGD, Fifth Edition}

\bibitem{Conti} Costanza Conti,
    \emph{Lecture notes}, distributed during classes

\end{thebibliography}

\end{document}


\input{../common/common.tex}

\title{Math notes - Cat vs dog}
\author{Uwe Hoffmann}
\hypersetup{colorlinks, pdftitle={Math notes - Cat vs dog}}

\begin{document}

\setcounter{chapter}{1}
\section*{Cat vs dog}

\newthought{Bipartite graphs}, network flows, matchings and vertex covers are the topics of the problem \footnote{\bibentry{spotify_interview}} in this note.\index{bipartite graph}\index{network flow}\index{bipartite matching}\index{vertex cover}

\begin{fullwidth}

\vspace{10 mm}
\begin{problem}
The latest reality show has hit the TV: “Cat vs. Dog”. In this show, a bunch of cats and dogs compete for the very prestigious Best Pet Ever title. In each episode, the cats and dogs get to show themselves off, after which the viewers vote on which pets should stay and which should be forced to leave the show.
\\

Each viewer gets to cast a vote on two things: one pet which should be kept on the show, and one pet which should be thrown out. Also, based on the universal fact that everyone is either a cat lover (i.e. a dog hater) or a dog lover (i.e. a cat hater), it has been decided that each vote must name exactly one cat and exactly one dog.
\\

Ingenious as they are, the producers have decided to use an advancement procedure which guarantees that as many viewers as possible will continue watching the show: the pets that get to stay will be chosen so as to maximize the number of viewers who get both their opinions satisfied. Calculate this maximum number of satisfied viewers.
\end{problem}

\end{fullwidth}

At first glance this looks similar to a SAT problem \footnote{Boolean satisfiability problem \url{http://en.wikipedia.org/wiki/Boolean_satisfiability_problem}}, something like $(c_1 \land \lnot d_3),\  (c_3 \land \lnot d_1),\  (d_2 \land \lnot c_2),\ \dots$ where $c_i$ are the cats and $d_j$ are the dogs. The goal would be to pick the biggest subset of boolean expressions (votes) that are satisfied.

\begin{marginfigure}[0.01in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            node distance = 3cm, % distance between nodes
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (v1) at (0,0) {$c_1 \land \lnot d_3$};
        \node[state] (v2) at (0,2) {$c_2 \land \lnot d_2$};
        \node[state] (v3) at (0,4) {$c_4 \land \lnot d_1$};
        \node[state] (v4) at (0,6) {$c_3 \land \lnot d_1$};
        \node[state] (v5) at (4,1) {$\lnot c_2 \land d_1$};
        \node[state] (v6) at (4,3) {$\lnot c_4 \land d_3$};
        \node[state] (v7) at (4,5) {$\lnot c_1 \land d_2$};

        \node [draw=blue,fit=(v1) (v4),label=above:Cat lovers] {};
        \node [draw=green,fit=(v5) (v7),label=above:Dog lovers] {};

        \path (v1) edge (v6);
        \path (v1) edge (v7);
        \path (v2) edge (v5);
        \path (v2) edge (v7);
        \path (v3) edge (v5);
        \path (v3) edge (v6);
        \path (v4) edge (v5);

    \end{tikzpicture}
    \caption{Votes form a bipartite graph. A graph is \textbf{bipartite} if the vertex set is partitioned into two subsets (blue and green in this case) such that no vertices in a subset are adjacent.}
	\label{votes_conflict}
\end{marginfigure}

But SAT is about one boolean expression and about assigning values to boolean variables to satisfy it. Seems like SAT is fundamentally different and not a good approach in solving this problem. What if we want to visualize the boolean expressions and see the relationships between them, i.e. which ones are in conflict. Conflict between two boolean expressions means one expression has $c_i$ and the other expression has $\lnot c_i$ or one has $d_j$ and the other $\lnot d_j$. A good way to do that is with a graph as in Figure \ref{votes_conflict}. The nodes in the graph are the boolean expressions and edges connect  boolean expressions that are in conflict.

It becomes apparent that the graph is bipartite with cat lovers on one side and dog lovers on the other. That is good because a lot of graph algorithms are much simpler and work faster if the graphs are bipartite. But what algorithm should we use? We need to find the biggest subset of nodes in the graph that are not in conflict.

Sometimes it's easier to compute the complement of what we want: the smallest subset of nodes that are involved in conflicts. Removing these nodes and the edges they touch should leave us with a graph with only nodes and no edges, i.e. only votes without conflicts. Because we strive to remove the smallest subset of conflicting nodes we are left with the biggest subset of votes without conflicts.

The subset of nodes that are involved in conflicts is a vertex cover \footnote{A \textbf{vertex cover} is a subset of nodes such that each edge in a graph is incident to at least one vertex in the subset.} for our bipartite graph.

We need to compute a minimum vertex cover. This will be a good excuse to learn about network flows in graphs, maximum flows and minimum cuts. This delightful detour will eventually bring us to maximum matchings\footnote{A \textbf{matching} is a subset of edges such that no two edges in the subset share a vertex.} and then finally to minimum vertex covers.\newline

We begin with \textbf{network flows} in graphs. We work with a directed graph $G=(V, E)$ that has two special vertices $s$ and $t$ called \textbf{source} and \textbf{target}.  No edge goes into \emph{source} and no edge comes out of \emph{target}. We also have a function $c:E \to \mathbb{R}_{\geq 0}$ that assigns a non-negative capacity to each edge. The graph $G$ together with source $s$ and target $t$ and capacity function $c$ form a \textbf{network} $(G=(V, E), s \in V, t \in V, c)$.

\begin{marginfigure}
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            node distance = 3cm, % distance between nodes
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (u) at (1,0) {$u$};
        \node[state] (v) at (3,0) {$v$};

        \path[->] (u) edge node {$10/20$} (v);

    \end{tikzpicture}
    \caption{In figures we annotate an edge with flow and capacity as shown here. In this case $f(u \rightarrow v) = 10$ and $c(u \rightarrow v) = 20$. If only one number is annotating the edge then it's the capacity.}
	\label{edge_flow_cap}
\end{marginfigure}

\begin{defn}\label{stflow}
A function $f:E \to \mathbb{R}_{\geq 0}$ is a \textbf{flow} through network $(G, s, t, c)$ if $f$ satisfies the following constraints:

\begin{itemize}
  \item \emph{capacity constraint}: flow along an edge cannot exceed the capacity of the edge
  $$
  \forall e \in E: f(e) \leq c(e)
  $$
  \item \emph{conservation constraint}: incoming flow into a vertex (except for source and target) equals outgoing flow from the vertex
  $$
  \forall v \in V \setminus \{s,t\}: \sum_u f(u \rightarrow v) = \sum_w f(v \rightarrow w)
  $$
\end{itemize}
\end{defn}

\begin{marginfigure}
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (s) at (0,1) {$s$};
        \node[state] (v) at (2,2) {$v$};
        \node[state] (u) at (2,0) {$u$};
        \node[state] (t) at (4,1) {$t$};

        \path[->, line width=1.5pt] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$0/10$} (v);
        \path[->, line width=1.5pt] (u) edge node {$20/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$0/10$} (t);
		\path[->, line width=1.5pt] (v) edge node {$20/20$} (t);

    \end{tikzpicture}
    \caption{Example network flow. Here $|f| = 20$ and the whole flow is pumped along the path $s \rightarrow u \rightarrow v \rightarrow t$. In this exampe $f$ \textbf{saturates} $s \rightarrow u$ and $v \rightarrow t$ and \textbf{avoids} $s \rightarrow v$ and $u \rightarrow t$.}
	\label{ex_flow}
\end{marginfigure}

\marginnote[0.5in]{For notational simplicity we assume functions $f$ and $c$ are defined on $V \times V$ and $f(u \rightarrow v) = c(u \rightarrow v) = 0$ if $u \rightarrow v$ is not an edge in $G=(V, E)$.}

Source $s$ generates flow and target $t$ consumes flow. The \textbf{value} of flow $f$, denoted $|f|$, is defined as

$$
|f| = \sum_w f(s \rightarrow w) = \sum_v f(v \rightarrow t)
$$

\begin{marginfigure}[0.3in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (s) at (0,1) {$s$};
        \node[state] (v) at (2,2) {$v$};
        \node[state] (u) at (2,0) {$u$};
        \node[state] (t) at (4,1) {$t$};

        \path[->] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$10/10$} (v);
        \path[->] (u) edge node {$10/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$10/10$} (t);
		\path[->] (v) edge node {$20/20$} (t);

    \end{tikzpicture}
    \caption{Same example network with a flow of value $|f| = 30$.}
	\label{ex_flow_max}
\end{marginfigure}

Given a network $(G, s, t, c)$ what is the maximum flow value that can be pumped through it? Figure \ref{ex_flow} shows a flow of value $20$ through an example network. It saturates the flow along one particular path and avoids the other edges. Is $20$ the maximum flow value that can be achieved for this example network? Figure \ref{ex_flow_max} shows the same network but now with a flow of value $30$. Can we do better than $30$? The answer is no, because that would exceed the outgoing capacity of source $s$ or the incoming capacity of $t$.

Our goal is to device an algorithm that constructs a flow with maximum value through a given network. To gauge the progress of our algorithm we need an upper bound for the maximum flow value. As said before the maximum value clearly cannot exceed the outgoing capacity of source $s$ or the incoming capacity of $t$. But more generally if we sever the ties between source and target along some subset of edges such that there are no more paths from source to target then the maximum flow value cannot exceed the capacity of the cut. This seems like a useful concept to formalize.

\begin{marginfigure}
\includegraphics[scale=0.15]{soviettrainnetwork.jpeg}
\end{marginfigure}
\marginnote{\bibentry{Schrijver02onthe}:\\
 Network flows and minimum cuts played a role in the Cold War. The figure is a schematic diagram of the railway network of the Western Soviet Union and Eastern European countries, with a maximum flow of value 163,000 tons from Russia to Eastern Europe, and a cut of capacity 163,000 tons indicated as “The bottleneck”.}

\begin{defn}\label{networkcut}
In a network $(G, s, t, c)$ a \textbf{cut} is a partition of the vertex set $V$ into two subsets $S$ and $T$, such that $V = S \cup T$, $S \cap T = \emptyset$ and $s \in S, t \in T$. The \textbf{capacity} of the cut $(S, T)$, denoted $\|S,T\|$, is defined as

$$
\|S, T\| = \sum_{v \in S} \sum_{w \in T} c(v \rightarrow w)
$$
\end{defn}

\begin{thm}\label{flowcut}
With network $(G, s, t, c)$, for any flow $f$ and any cut $(S, T)$ we have
$$
  |f| \leq \|S, T\|
$$
Furthermore equality holds if and only if $f$ saturates every edge from $S$ to $T$ and avoids every edge from $T$ to $S$.
\end{thm}

\begin{marginfigure}[1in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node[state] (s) at (0,1) {$s$};
        \node[state] (v) at (2,2) {$v$};
        \node[state] (u) at (2,0) {$u$};
        \node[state] (t) at (4,1) {$t$};

        \path[->, line width=1.5pt] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$0/10$} (v);
        \path[->, line width=1.5pt] (u) edge node {$20/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$0/10$} (t);
		\path[->, line width=1.5pt] (v) edge node {$20/20$} (t);

		\path[->, dashed] (s) edge[bend left=60] node {$10$} (v);
		\path[->, dashed] (v) edge[bend left=30] node {$20$} (u);
		\path[->, dashed] (u) edge[bend right=70] node[right] {$10$} (t);

    \end{tikzpicture}
    \caption{Dashed edges show how the greedy $s-t$ path flow can be augmented and reversed in order to increase overall flow.}
	\label{ex_flow_dashed}
\end{marginfigure}

\begin{proof}

\begin{align*}
|f| &= \sum_w f(s \rightarrow w) \tag*{\tiny{(by definition)}}\\
    &= \sum_w f(s \rightarrow w) - \sum_v f(v \rightarrow s) \tag*{\tiny{(second sum terms are all zero)}}\\
    &= \sum_{u \in S} (\sum_w f(u \rightarrow w) - \sum_v f(v \rightarrow u)) \tag*{\tiny{(flow conservation constraint)}}\\
    &= \sum_{u \in S} (\sum_{w \in T} f(u \rightarrow w) - \sum_{v \in T} f(v \rightarrow u)) \tag*{\tiny{(edges in $S$ cancel each other out)}}\\
    &\leq \sum_{u \in S} \sum_{w \in T} f(u \rightarrow w) \tag*{\tiny{(because $f(v \rightarrow u) \geq 0$)}}\\
    &\leq \sum_{u \in S} \sum_{w \in T} c(u \rightarrow w) \tag*{\tiny{(flow capacity constraint)}}\\
    &= \|S, T\| \tag*{\tiny{(by definition)}}
\end{align*}

\end{proof}

Theorem \ref{flowcut} tells us that if we keep increasing a flow and/or decreasing a cut we should eventually meet at a maximum flow that equals a minimum cut. But given a network how do we start? A first valid flow is $\forall e \in E: f(e) = 0$. We could then try a greedy strategy. Starting with source $s$ find the path to $t$ with the biggest capacity\footnote{The capacity of a path is the minimum over the capacities of the edges forming the path.} and pump as much flow as we can through it as illustrated in Figure \ref{ex_flow}. Unfortunately we are stuck at that point. We cannot pump more flow out of $s$ on $s \rightarrow v$ because that would violate flow conservation at $v$ (we are at the maximum outgoing flow at $v$). The dashed edges in Figure \ref{ex_flow_dashed} show some of our options. On edges where the current flow leaves residual capacity we can pump more and on edges where there is existing flow we can reverse it. Again, this concept seems worth formalizing.

\begin{marginfigure}
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node at (0,2.7) {(a)};
        \node at (0,0) {(b)};

        \node[state] (sf) at (0,1) {$s$};
        \node[state] (vf) at (2,2) {$v$};
        \node[state] (uf) at (2,0) {$u$};
        \node[state] (tf) at (4,1) {$t$};

        \node[state] (s) at (0,4) {$s$};
        \node[state] (v) at (2,5) {$v$};
        \node[state] (u) at (2,3) {$u$};
        \node[state] (t) at (4,4) {$t$};

        \path[->] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$0/10$} (v);
        \path[->] (u) edge node {$20/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$0/10$} (t);
		\path[->] (v) edge node {$20/20$} (t);

        \path[<-, blue] (sf) edge node[below left=.05cm] {$20$} (uf);
        \path[->, blue] (sf) edge node {$10$} (vf);
        \path[->, blue] (uf) edge[bend left=30] node {$80$} (vf);
        \path[<-, blue] (uf) edge[bend right=30] node {$20$} (vf);
		\path[->, blue] (uf) edge node[below right=.05cm] {$10$} (tf);
		\path[<-, blue] (vf) edge node {$20$} (tf);

    \end{tikzpicture}
    \caption{\\
             (a) Example network with flow from Figure \ref{ex_flow}. \\
             (b) Residual network (in blue) with edges annotated with their residual capacity.}
	\label{ex_residual}
\end{marginfigure}

\begin{defn}\label{residualgraph}
A flow $f$ in a network $(G, s, t, c)$ induces a \textbf{residual network} $(G_f, s, t, c_f)$ with \textbf{residual graph} $G_f$ and \textbf{residual capacity} $c_f$ in the following way:

\begin{itemize}
	\item all vertices from $G$ are vertices in $G_f$, also source $s$ and target $t$ are the same in $G$ and $G_f$
	\item if $f(u \rightarrow v) > 0$ then $G_f$ has an edge $(v \rightarrow u)$ with capacity
$$
c_f(v \rightarrow u) = f(u \rightarrow v)
$$
    \item if $f(u \rightarrow v) < c(u \rightarrow v)$ then $G_f$ has an edge $(u \rightarrow v)$ with capacity
$$
c_f(u \rightarrow v) = c(u \rightarrow v) - f(u \rightarrow v)
$$
\end{itemize}

\end{defn}

Figure \ref{ex_residual} shows the residual network of our example network and flow. We observe that there is a simple path\footnote{A simple path is a path where every vertex on the path is visited only once.} $s \rightarrow v \rightarrow u \rightarrow t$ with capacity $10$ from source $s$ to target $t$ in the residual graph. This path shows that there still is unused capacity for flow to be pushed from $s$ to $t$. A simple path from $s$ to $t$ in $G_f$ is called an \textbf{augmenting path}.

\begin{marginfigure}[1in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node at (0,2.7) {(a)};
        \node at (0,0) {(b)};

        \node[state] (sf) at (0,1) {$s$};
        \node[state] (vf) at (2,2) {$v$};
        \node[state] (uf) at (2,0) {$u$};
        \node[state] (tf) at (4,1) {$t$};

        \node[state] (s) at (0,4) {$s$};
        \node[state] (v) at (2,5) {$v$};
        \node[state] (u) at (2,3) {$u$};
        \node[state] (t) at (4,4) {$t$};

        \path[->] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$0/10$} (v);
        \path[->] (u) edge node {$20/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$0/10$} (t);
		\path[->] (v) edge node {$20/20$} (t);

		\path[->] (sf) edge node[below left=.05cm] {$20/20$} (uf);
        \path[->] (sf) edge node {$\color{red}10\color{black}/10$} (vf);
        \path[->] (uf) edge node {$\color{red}10\color{black}/100$} (vf);
		\path[->] (uf) edge node[below right=.05cm] {$\color{red}10\color{black}/10$} (tf);
		\path[->] (vf) edge node {$20/20$} (tf);

    \end{tikzpicture}
    \caption{\\
             (a) Example network with flow from Figure \ref{ex_flow}. \\
             (b) Augmented flow (changed values in red) from augmenting path $s \rightarrow v \rightarrow u \rightarrow t$.}
	\label{augmented_flow}
\end{marginfigure}

\begin{thm}\label{augmenting}
Given is a flow $f$ in network $(G, s, t, c)$. If there is an augmenting path in $G_f$ with capacity $F$ then the function $f':V \times V \to \mathbb{R}_{\geq 0}$ defined as:

$$
f'(u \rightarrow v) =
\begin{cases}
f(u \rightarrow v) + F, & \text{if } u\rightarrow v \text{ is on the augmenting path} \\
f(u \rightarrow v) - F, & \text{if } v\rightarrow u \text{ is on the augmenting path} \\
f(u \rightarrow v), & \text{otherwise}
\end{cases}
$$

is a valid flow in network $(G, s, t, c)$ with $|f'| = |f| + F$.
\end{thm}

\begin{proof}
We need to check the capacity constraint and the conservation constraint.

Let's start with the capacity constraint. The definition of $f'$ has three cases, so we check all three:

\begin{itemize}

	\item Edge $u \rightarrow v$ is on the augmenting path:

\begin{align*}
f'(u \rightarrow v) &= f(u \rightarrow v) + F \tag*{\tiny{(by definition)}}\\
    &\leq f(u \rightarrow v) + c_f(u \rightarrow v) \tag*{\tiny{(by definition of $F$)}}\\
    &=  f(u \rightarrow v) + c(u \rightarrow v) - f(u \rightarrow v) \tag*{\tiny{(by definition of $c_f$)}}\\
    &=  c(u \rightarrow v)
\end{align*}

	\item Edge $v \rightarrow u$ is on the augmenting path:

\begin{align*}
f'(u \rightarrow v) &= f(u \rightarrow v) - F \tag*{\tiny{(by definition)}}\\
    &\geq f(u \rightarrow v) - c_f(u \rightarrow v) \tag*{\tiny{(by definition of $F$)}}\\
    &=  f(u \rightarrow v) - f(u \rightarrow v) \tag*{\tiny{(by definition of $c_f$)}}\\
    &=  0
\end{align*}

	\item Otherwise: In this case the flow of the edge hasn't changed so capacity constraint is satisfied.

\end{itemize}

Next is the conservation constraint. For vertices not on the augmenting path flow in and out of them hasn't changed, so conservation constraint is satisfied there. For a vertex $v$ on the augmenting path we have four cases (since the augmenting path is simple and $v \neq s, v \neq t$):

\begin{itemize}

	\item $u \rightarrow v$ on augmenting path and $v \rightarrow w$ on augmenting path: in this case one incoming edge into $v$ changed by $F$ and one outgoing changed by $F$, so conservation constraint holds for $v$

	\item $u \rightarrow v$ on augmenting path and $w \rightarrow v$ on augmenting path: in this case two incoming edges into $v$ changed, one by $F$ and the other by $-F$, so conservation constraint holds for $v$

	\item $v \rightarrow u$ on augmenting path and $w \rightarrow v$ on augmenting path: in this case one incoming edge into $v$ changed by $-F$ and one outgoing changed by $-F$, so conservation constraint holds for $v$

	\item $v \rightarrow u$ on augmenting path and $v \rightarrow w$ on augmenting path: in this case two outgoing edges from $v$ changed, one by $-F$ and the other by $F$, so conservation constraint holds for $v$

\end{itemize}

\end{proof}

\begin{marginfigure}
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node at (0,2.7) {(a)};
        \node at (0,0) {(b)};

        \node[state] (sf) at (0,1) {$s$};
        \node[state] (vf) at (2,2) {$v$};
        \node[state] (uf) at (2,0) {$u$};
        \node[state] (tf) at (4,1) {$t$};

        \node[state] (s) at (0,4) {$s$};
        \node[state] (v) at (2,5) {$v$};
        \node[state] (u) at (2,3) {$u$};
        \node[state] (t) at (4,4) {$t$};

        \path[->] (s) edge node[below left=.05cm] {$20/20$} (u);
        \path[->] (s) edge node {$10/10$} (v);
        \path[->] (u) edge node {$10/100$} (v);
		\path[->] (u) edge node[below right=.05cm] {$10/10$} (t);
		\path[->] (v) edge node {$20/20$} (t);

        \path[<-, blue] (sf) edge node[below left=.05cm] {$20$} (uf);
        \path[<-, blue] (sf) edge node {$10$} (vf);
        \path[->, blue] (uf) edge[bend left=30] node {$90$} (vf);
        \path[<-, blue] (uf) edge[bend right=30] node {$10$} (vf);
		\path[<-, blue] (uf) edge node[below right=.05cm] {$10$} (tf);
		\path[<-, blue] (vf) edge node {$20$} (tf);

    \end{tikzpicture}
    \caption{\\
             (a) Example network with flow from Figure \ref{ex_flow_max}. \\
             (b) Residual network (in blue) with edges annotated with their residual capacity.}
	\label{ex_residual_max}
\end{marginfigure}

What happens when there is no augmenting path in $G_f$? As the Figure \ref{ex_residual_max} hints we then have a maximum flow (in our example $|f| = 30$). The next theorem proves it.

\begin{thm}\label{no_augmenting}
Given is a flow $f$ in network $(G, s, t, c)$. If there is no augmenting path in $G_f$ then $f$ is a flow with maximum value.
\end{thm}

\begin{proof}
We define two subsets of $V$. The set $S$ holds all the vertices of $V$ that are reachable from $s$ in $G_f$. Since there is no augmenting path in $G_f$ we have $t \notin S$. We also define $T = V \setminus S$. Clearly $(S, T)$ is a cut of our network. Also there is no $G_f$ edge $u \rightarrow v$ with $u \in S$ and $v \in T$ because otherwise $v$ would be reachable from somewhere in $S$ but $v \notin S$, contradicting the definition of $S$. This means (by definition of $G_f$) that $f$ saturates every edge from $S$ to $T$ and avoids every edge from $T$ to $S$. According to Theorem \ref{flowcut} we then have $|f| = \|S, T\|$ which means we have a maximum flow and minimum cut.

\end{proof}

\begin{marginfigure}[0.0in]
\includegraphics[scale=0.8]{fulkerson.png}
\end{marginfigure}
\marginnote{Delbert Ray Fulkerson was an American mathematician who co-developed the Ford–Fulkerson algorithm. \url{https://en.wikipedia.org/wiki/D._R._Fulkerson}}

We can now piece together the following algorithm known as the \textbf{Ford-Fulkerson algorithm}:

\begin{lstlisting}[basicstyle=\small, label={lst:fordfulkerson}, frame=trBL, caption={Ford-Fulkerson algorithm}]
f = zero flow;
Gf = residual graph of f in G;

while (exists augmenting path in Gf):
   pa = choose any augmenting path;
   f = augment f with pa;
   Gf = residual graph of f in G;

return f
\end{lstlisting}

\begin{thm}\label{fordfulkerson}
If the network has capacities in $\mathbb{N}_{\geq 0}$ then the Ford-Fulkerson algorithm terminates and returns the maximum flow in the network.
\end{thm}

\begin{proof}
We prove by induction that $f: V \times V \to \mathbb{N}_{\geq 0}$: The base case is the zero flow which is in $\mathbb{N}_{\geq 0}$. Assume the current flow values are in $\mathbb{N}_{\geq 0}$. The augmenting operation adds or subtracts a positive integer value from the current flow values and conforms to capacity constraints, so it keeps the augmented flow values in $\mathbb{N}_{\geq 0}$ which completes the induction.

The augmented flow $f'$ modifies one outgoing edge from $s$ by $F > 0$, so by the definition of the value of a flow we have $|f'| = |f| + F$. This means that augmenting strictly increases the value of the flow. We also know that flow values have an upper bound (by Theorem \ref{flowcut} any cut capacity is an upper bound). This means the algorithm has to eventually reach the maximum flow and terminate.
\end{proof}

This concludes our detour into network flows\footnote{We have just scratched the surface of the topic on network flows and algorithms computing maximum flows (an area of active research). For example by making smart choices when choosing the augmenting path we can improve the runtime of the algorithm (also we haven't analyzed the runtime). What happens when the capacities are not in $\mathbb{N}_{\geq 0}$. For details on all this and more see:
\\
\bibentry{kleinberg2005ad}
\\
\bibentry{erickson2015a}.}.
\\
We should bring it back to our problem and the associated bipartite graph of conflicting votes. We want a minimal vertex cover and we would like to use the just derived Ford-Fulkerson algorithm to compute it. So we first have to transform our undirected bipartite graph into a network.

We have an undirected bipartite graph $G(V = X \cup Y, E)$ with $X \cap Y = \emptyset$ and $E \subseteq X \times Y$ ($X$ could be the votes of cat lovers and $Y$ the votes of dog lovers in our problem or vice versa). We add a source $s$ and a target $t$ and construct a network $(G', s, t, c)$ in the following way:

\begin{itemize}
	\item vertex set of $G'$ is $X \cup Y \cup \{s, t\}$
	\item $\forall u \in X$ add a directed edge $s \rightarrow u$ into edge set of $G'$
	\item $\forall v \in Y$ add a directed edge $v \rightarrow t$ into edge set of $G'$
	\item $\forall \{u, v\}$ undirected edge in $G$ with $u \in X$ and $v \in Y$ add a directed edge $u \rightarrow v$  into edge set of $G'$
	\item unit capacity\footnote{Unit capacity has the advantage that a flow either saturates the edge or avoids it.}: $\forall (u \rightarrow v) \in \text{ edge set of } G': c(u \rightarrow v) = 1$
\end{itemize}

With a network $(G', s, t, c)$ constructed from a bipartite graph $G$ as described above (an example is shown in Figure \ref{bipartite_network}) we have an equivalence between a matching in the bipartite graph and a flow in the network. The next theorem states this.

\begin{marginfigure}[1.0in]
\begin{tikzpicture}[
            shorten > = 1pt, % don't touch arrow head to node
            auto,
            semithick % line style
        ]

        \tikzstyle{every state}=[
            draw = black,
            thick,
            fill = white,
            minimum size = 4mm
        ]

        \node at (0,4.5) {(a)};
        \node at (0,0) {(b)};

        \node[state] (s) at (0,3) {$s$};
        \node[state] (t) at (5,3) {$t$};

        \node[state] (x1) at (1.5,0.5) {$x_1$};
        \node[state] (x2) at (1.5,2) {$x_2$};
        \node[state] (x3) at (1.5,3) {$x_3$};
        \node[state] (x4) at (1.5,4) {$x_4$};

        \node[state] (y1) at (3,0.5) {$y_1$};
        \node[state] (y2) at (3,2) {$y_2$};
        \node[state] (y3) at (3,3) {$y_3$};

        \path[->] (x1) edge node {$1$} (y1);
        \path[->] (x2) edge node {$1$} (y1);
        \path[->] (x2) edge node {$1$} (y2);
        \path[->] (x2) edge node {$1$} (y3);
        \path[->] (x3) edge node {$1$} (y3);
        \path[->] (x4) edge node {$1$} (y3);

        \path[->] (s) edge node {$1$} (x1);
        \path[->] (s) edge node {$1$} (x2);
        \path[->] (s) edge node {$1$} (x3);
        \path[->] (s) edge node {$1$} (x4);

        \path[->] (y1) edge node {$1$} (t);
        \path[->] (y2) edge node {$1$} (t);
        \path[->] (y3) edge node {$1$} (t);

        \node[state] (bx1) at (1.5,5.5) {$x_1$};
        \node[state] (bx2) at (1.5,7) {$x_2$};
        \node[state] (bx3) at (1.5,8) {$x_3$};
        \node[state] (bx4) at (1.5,9) {$x_4$};

        \node[state] (by1) at (3,5.5) {$y_1$};
        \node[state] (by2) at (3,7) {$y_2$};
        \node[state] (by3) at (3,8) {$y_3$};

        \path[-] (bx1) edge node {} (by1);
        \path[-] (bx2) edge node {} (by1);
        \path[-] (bx2) edge node {} (by2);
        \path[-] (bx2) edge node {} (by3);
        \path[-] (bx3) edge node {} (by3);
        \path[-] (bx4) edge node {} (by3);

    \end{tikzpicture}
    \caption{\\
             (a) Bipartite graph \\
             (b) Network constructed from it.}
	\label{bipartite_network}
\end{marginfigure}

\begin{thm}\label{matchingflow}
A matching $M$ in $G$ induces a flow $f$ in $G'$ such that $|f| = |M|$. Conversely a flow in $G'$ induces a matching $M$ in $G$ such that $|M| = |f|$.
\end{thm}

\begin{proof}
\noindent$(\Rightarrow)$ We have a matching $M$ in $G$, i.e. a subset of edges that don't share a vertex. From the construction of $G'$ it follows that each of the edges in $M$ can be extended to paths from $s$ to $t$ which will only meet in $s$ and $t$. We define a function $f$ that gives unit values to the edges along these paths and zero value to all other edges. We claim that $f$ is a valid flow. It only assigns zero or unit values so it does satisfy the capacity constraint in $G'$. The paths don't intersect except in $s$ and $t$ (because $M$ is a matching), so for any vertex along the path there is exactly one incoming edge with unit value and one outgoing edge with unit value. The rest of the edges have value zero so don't play a role in conservation. This then means that the conservation constraint is satisfied also and $f$ is a flow.
Each edge in $M$ corresponds to one of the paths, so there are $|M|$ edges outgoing from $s$ that have unit value. Hence $|f| = |M|$.

\noindent$(\Leftarrow)$ We have a flow $f$ in $G'$. The flow either saturates or avoids an edge. We define the subset $M$ of edges that are saturated by $f$ and are between $X$ and $Y$. We claim that $M$ is a matching in $G$. Suppose it is not a matching. Then there exists a vertex that is shared by two edges in $M$. If this vertex is in $X$ then it means it has two outgoing edges of unit value but only one incoming edge of unit value (from $s$). If this vertex is in $Y$ then it means it has two incoming edges of unit value and only one outgoing edge of unit value (to $t$). In either case this is a contradiction to the conservation constraint of $f$. So $M$ has to be a matching. The size of $M$ is by its definition equal to the number of saturated edges from $X$ to $Y$. But this number has to be equal to the number of edges of unit value going out of $s$ (conservation constraint). Hence $|M| = |f|$.
\end{proof}

Theorem \ref{matchingflow} let's us use the Ford-Fulkerson algorithm to compute a maximum matching in our bipartite graph $G$. Once we have a maximum matching we get the size of a minimum vertex cover with the following theorem, known as \textbf{K\"onig's theorem}\footnote{For a short and elegant proof see:\\\bibentry{rizzi00}}:

\begin{thm}\label{koenig}
In a bipartite graph $G$ the size of a minimum vertex cover $C$ equals the size of a maximum matching $M$.
\end{thm}

\begin{marginfigure}[0.3in]
\includegraphics[scale=0.5]{koenig.pdf}
\caption{\\
             (a) A bipartite graph $G(X \cup Y, E)$. $X$ are green vertices, $Y$ are red vertices.\\
             (b) Maximum flow $f$ (thicker arrows) and minimum cut $(S, T)$ in the corresponding network $G'$. Thicker arrows between green and red vertices form the maximum matching.\\
             (c) Same cut displayed with the corresponding residual graph $G'_f$.}
	\label{koenig_example}
\end{marginfigure}

\begin{proof}

$C$ is a vertex cover, so it covers all edges, which means it certainly covers a subset $M$ of all edges. But $M$ is a matching, so no two edges share a vertex. It follows that $|C| \geq |M|$.

From the maximum matching $M$ we get the associated maximum flow $f$ as described in Theorem \ref{matchingflow}. The residual graph $G'_f$ of the associated network cannot have any augmenting paths.

We consider the minimum cut $(S, T)$ associated with the maximum flow $f$. We define the following sets:

\begin{itemize}
	\item $X_S = X \cap S, X_T = X \cap T$
	\item $Y_S = Y \cap S, Y_T = Y \cap T$
	\item $H = \{(u, v) \text{ edge in } G: u \in S, v \in T\}$
	\item $B = \{v \in Y_T: \exists u \in X_S \text{ with } (u, v) \text{ edge in } G\}$
	\item $D = X_T \cup Y_S \cup B$
\end{itemize}

\noindent $D$ is a vertex cover: $X_T \subseteq D$ and $Y_S \subseteq D$, so $D$ covers all edges that have endpoints in $X_T$ or $Y_S$. The set $B$ provides cover for $H$.

A vertex $u \in X_T$ is not reachable from $s$ in $G'_f$. It means that $f$ saturates $s \rightarrow u$ in $G'$, so the saturated edge $s \rightarrow u$ crosses the $(S, T)$ cut and counts towards $\|S, T\|$.

A vertex $v \in Y_S$ is reachable from $s$ in $G'_f$. It means that $f$ saturates $v \rightarrow t$ in $G'$ (otherwise some vertex from $T$ would be reachable from $v$ and also from $s$ in $G'_f$ which is a contradiction). The saturated edge $v \rightarrow t$ crosses the $(S, T)$ cut and counts towards $\|S, T\|$.

$f$ is a maximum flow so any edge from $X_S$ to $Y_T$ is saturated and counts towards $\|S, T\|$.

$$
\|S, T\| = |X_T| + |Y_S| + |H|
$$

\noindent Figure \ref{koenig_example} shows this. In (b) there are three saturated (thick) arrows crossing the cut. The first two (counting from left to right) are due to $X_T$ and the last one due to $Y_S$. In this example $H$ is the empty set.

We have
$$
|M| = |f| = \|S, T\| = |X_T| + |Y_S| + |H| \geq |X_T| + |Y_S| + |B| \geq |D|
$$
\noindent $D$ is a vertex cover and $C$ is a minimum vertex cover, so $|D| \geq |C|$. It follows that $|C| \geq |M| \geq |D| \geq |C|$ which means $|C| = |M|$.
\end{proof}

This solves the problem in this note. The number of satisfied viewers is $|V| - |M|$, where $V$ is the set of vertices in the bipartite graph $G$ of votes with their conflicts as edges and $M$ is a maximum matching in $G$ computed with the Ford-Fulkerson algorithm taking advantage of the min-max duality shown in Figure \ref{minmax_duality}.

\begin{marginfigure}
    \includegraphics[scale=0.5]{flowcutminmax.pdf}
    \caption{Min-max duality in bipartite graphs and corresponding networks.}
	\label{minmax_duality}
\end{marginfigure}{}

\bibliographystyle{plainnat}
\bibliography{../common/math}

\end{document}


%!TEX program = xelatex
\documentclass[aspectratio=169]{beamer}

\usepackage{blindtext}
\usepackage{url}
\usepackage{bm}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{xcolor}
\usepackage{tikzsymbols}
\usepackage{cleveref}
\usepackage{hyperref}
\usepackage{MyMnSymbol}
\hypersetup{
  colorlinks = true,
}
\usepackage[linesnumbered,ruled,vlined]{algorithm2e}
\usefonttheme[onlymath]{serif}
\usetheme{Execushares}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}

\title{Alternating Direction Method of Multipliers}
\subtitle{\large{ELEC5470/IEDA6100A - Convex Optimization}}
\author{\normalsize{\textbf{Vin\'icius and Prof. Daniel Palomar}}}
\date{December, 2020}

\setcounter{showSlideNumbers}{1}
\addtobeamertemplate{footnote}{\vspace{-2pt}\advance\hsize-0.5cm}{\vspace{6pt}}

\begin{document}
  \setcounter{showProgressBar}{0}
  \setcounter{showSlideNumbers}{0}

  \frame{\titlepage}

  \begin{frame}
    \frametitle{Contents}
    \begin{enumerate}
    \item Introduction \\ \textcolor{ExecusharesGrey}{\footnotesize\hspace{1em} Optimization algorithms, motivation}
    \item Alternating Direction Method of Multipliers \\ \textcolor{ExecusharesGrey}{\footnotesize\hspace{1em} The basics}
    \item Practical Examples \\ \textcolor{ExecusharesGrey}{\footnotesize\hspace{1em} Robust PCA and Graphical Lasso}
    \end{enumerate}
  \end{frame}

  \setcounter{framenumber}{0}
  \setcounter{showProgressBar}{1}
  \setcounter{showSlideNumbers}{1}

  \section*{Why use optimization algorithms?}
        \begin{frame}{Motivations}
          {\large methods for}
          \begin{itemize}
            \item large-scale optimization
              \begin{itemize}
                \item machine learning/statistics with huge datasets
                \item computer vision
              \end{itemize}
            \item descentralized optimization
              \begin{itemize}
                \item entities/agents/threads coordinate to solve a large problem by passing small messages
              \end{itemize}
          \end{itemize}
        \end{frame}

        \begin{frame}{Optimization Algorithms}
          \begin{itemize}
            \item Gradient Descent
            \item Newton
            \item Interior Point Methods (IPM)
            \item Block Coordinate Descent (BCD)
            \item Majorization-Minimization (MM)
            \item Block Majorization-Minimization (BMM)
            \item Successive Convex Approximation (SCA)
            \pause
            \item ...
            \pause
            \item \bf{Alternating Direction Method of Multipliers (ADMM)}
          \end{itemize}
        \end{frame}
  \begin{frame}
    \frametitle{Reference}
          \begin{itemize}
            \item Boyd \textit{et al.} \textbf{Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers}.
              \textit{Foundations and Trends in Machine Learning}. 2010.
            \item available online for \textbf{free}: \url{https://web.stanford.edu/\~boyd/papers/pdf/admm\_distr\_stats.pdf}
            \item citations: 13519\footnote{as of Nov. 24th 2020}
            \item Yuxin Chen's Princeton lecture notes ELE 522: Large-Scale Optimization for Data Science
          \end{itemize}
  \end{frame}

        \begin{frame}{Dual Problem}
          \begin{itemize}
            \item convex equality constrained optimization problem
             \begin{equation*}
                \begin{array}{ll}
                  \underset{\bm x}{\textsf{minimize}} & f(\bm x) \\
                  \textsf{subject to} & \bm A\bm x  = \bm b
                 \end{array}
            \end{equation*}
          \item Lagrangian: $L(\bm x, \bm y) = f(\bm x) + \bm y^\top(\bm A \bm x - \bm b)$
          \item dual function: $g(\bm y) = \underset{\bm x}{\textsf{inf}} ~~ L(\bm x, \bm y)$
          \item dual problem: $\underset{\bm y}{\textsf{maximize}} ~~ g(\bm y)$
          \item recover: $\bm x^\star = \underset{\bm x}{\textsf{argmin}}~~ L(\bm x, \bm y^\star)$
          \end{itemize}
        \end{frame}

        \section*{Dual Ascent}

        \begin{frame}{Dual Ascent}
          \begin{itemize}
            \item gradient method for dual problem: $\bm y^{k+1} = \bm y^k + \rho^k \nabla g(\bm y^k)$
            \item $\nabla g(\bm y^k) = \bm A \bm x^{k+1} - \bm b$, where $\bm x^{k+1} = \underset{\bm x}{\textsf{arg~min}}~~L(\bm x, \bm y^k)$
            \item dual ascent method is
            \begin{align*}
              \bm x^{k+1} &:= \underset{\bm x}{\textsf{arg~min}}~~L(\bm x, \bm y^k) \\
              \bm y^{k+1} &:= \bm y^k + \rho^k\left(\bm A \bm x^{k+1} - \bm b\right)
            \end{align*}
            \pause
            \item why?
          \end{itemize}
        \end{frame}

        \section*{Dual Decomposition}
        \begin{frame}{Dual Decomposition}
          \begin{itemize}
            \item suppose $f$ is separable:
            \begin{equation*}
              f(\bm x) = f_{1}(x_1) + \dots + f_n(x_n), \bm x = (x_1, \dots, x_n)
            \end{equation*}
            \item then the Lagrangian is separable in $\bm x$:
            \begin{equation*}
              L_i(x_i, \bm y) = f_i(x_i) + \bm y^\top\bm a_{*,i} x_i
            \end{equation*}
            \item $\bm x$-minimization splits into $n$ separate minimizations
            \begin{equation*}
              x^{k+1}_i := \underset{x_i}{\textsf{arg~min}}~ L_i(x_i, \bm y^k), i = 1, ..., n
            \end{equation*}
            which can be done in parallel and
              $\bm y^{k+1} = \bm y^k + \alpha^k\left(\sum_{i=1}^n \bm a_{*,i}x^{k+1}_i - \bm b\right)$
          \end{itemize}
        \end{frame}

    \begin{frame}
      \frametitle{Optimization Problem}
                  \vspace{1cm}
                    {\huge
                  \begin{equation}
                     \begin{array}{ll}
                       \underset{\bm x,~ \bm z}{\textsf{minimize}} & f(\bm x) + g(\bm z) \\
                       \textsf{subject to} & \bm A\bm x + \bm B \bm z = \bm c
                      \end{array}
                 \end{equation}
                    }
                 \begin{itemize}
                 \item variables: $\bm x \in \mathbb{R}^{n}$ and $\bm z \in \mathbb{R}^{m}$
                 \item parameters: $\bm A \in \mathbb{R}^{p\times n}$, $\bm B \in \mathbb{R}^{p\times m}$, and $\bm c \in \mathbb{R}^p$
                 \item optimal value: $p^\star = \underset{\bm x, \bm z}{\textsf{inf}}\left\{f(\bm x) + g(\bm z): \bm A\bm x + \bm B\bm z = \bm c\right\}$
                 \end{itemize}
    \end{frame}

    \section*{Augmented Lagrangian Method}

    \begin{frame}
        \frametitle{Augmented Lagrangian Method}
        \begin{itemize}
          \item Augmented Lagrangian:
            \begin{equation*}
              L_{\rho}(\bm x, \bm z, \bm y) = \underbrace{f(\bm x) + g(\bm z) + \langle \bm y, \bm A\bm x + \bm B \bm z - \bm c \rangle}_{\text{Lagrangian}}
              + \dfrac{\rho}{2}\norm{\bm A\bm x + \bm B \bm z - \bm c}^2_{\mathrm{F}}
            \end{equation*}
          \item ALM consists of the iterations:
            \begin{align*}
              \bm x^{k+1}, \bm z^{k+1} & := \underset{\bm x, \bm z}{\textsf{arg min}}~~ L_{\rho}(\bm x, \bm z, \bm y^k) ~{\color{gray} \text{// primal update}}\\
              \bm y^{k+1} & := \bm y^k + \rho\left(\bm A\bm x^{k+1} + \bm B \bm z^{k+1} - \bm c\right) ~{\color{gray} \text{// dual update}}
            \end{align*}
          \item $\rho > 0$ is a penalty hyperparameter
        \end{itemize}
    \end{frame}

    \begin{frame}{Issues with Augmented Lagrangian Method}
      \begin{itemize}
        \item the primal step is often expensive to solve -- as expensive as solving the original problem
        \item minimization of $\bm x$ and $\bm z$ has to be done jointly
      \end{itemize}
    \end{frame}

    \section*{Alternating Direction Method of Multipliers}

                \begin{frame}
                  \frametitle{Alternating Direction Method of Multipliers}
                  \begin{itemize}
                    \item Augmented Lagrangian:
                      \begin{equation*}
                        L_{\rho}(\bm x, \bm z, \bm y) = \underbrace{f(\bm x) + g(\bm z) + \langle \bm y, \bm A\bm x + \bm B \bm z - \bm c \rangle}_{\text{Lagrangian}}
                        + \dfrac{\rho}{2}\norm{\bm A\bm x + \bm B \bm z - \bm c}^2_{\mathrm{F}}
                      \end{equation*}
                    \item ADMM consists of the iterations:
                      \begin{align*}
                        \bm x^{k+1} & := \underset{\bm x}{\textsf{arg min}}~~ L_{\rho}(\bm x, \bm z^k, \bm y^k) \\
                        \bm z^{k+1} & := \underset{\bm z}{\textsf{arg min}}~~ L_{\rho}(\bm x^{k+1}, \bm z, \bm y^k) \\
                        \bm y^{k+1} & := \bm y^k + \rho\left(\bm A\bm x^{k+1} + \bm B \bm z^{k+1} - \bm c\right)
                      \end{align*}
                    \item $\rho > 0$ is a penalty hyperparameter
                  \end{itemize}
                \end{frame}

                \begin{frame}
                  \frametitle{Convergence and Stopping Criteria}
                  \begin{itemize}
                    \item assume (very little!)
                      \begin{itemize}
                        \item $f$, $g$ are convex, closed, proper % closed: sublevels sets are closed sets
                                                                  % proper: effective domain is non-empty and it never attains -infty
                        \item $L_{0}$ has a saddle point % saddle point theorem
                      \end{itemize}
                    \item then ADMM converges:
                      \begin{itemize}
                        \item iterates approach feasibility: $\bm A \bm x^k + \bm B \bm z^k - \bm c \rightarrow \bm 0$
                        \item objective approaches optimal value: $f(\bm x^k) + g(\bm z^k) \rightarrow p^\star$
                      \end{itemize}
                    \item false (in general) statements: $\bm x$ converges, $\bm z$ converges
                    \item true statement: $\bm y$ converges
                    \item what matters: residual is small and near optimality in objective value
                  \end{itemize}
                \end{frame}

                \begin{frame}{Convergence of ADMM in Practice}
                  \begin{itemize}
                    \item ADMM is often slow to converge to high accuracy
                    \item ADMM often converges to moderate accuracy within a few dozens of iterations, which is often sufficient
                    for most practical purposes
                  \end{itemize}
                \end{frame}

                \section*{Practical Examples}
                \begin{frame}{Robust PCA (Candes et al. '08)}
                  \begin{itemize}
                    \item We would like to model a data matrix $\bm M$ as low-rank plus sparse components:
                  \end{itemize}
                    {\huge
                    \begin{equation*}
                       \begin{array}{ll}
                         \underset{\bm L,~ \bm S}{\textsf{minimize}} & \norm{\bm L}_{*} + \lambda \norm{\bm S}_{1} \\
                         \textsf{subject to} & \bm L + \bm S = \bm M
                        \end{array}
                   \end{equation*}
                      }
                  \begin{itemize}
                    \item where $\norm{\bm L}_{*} := \sum_{i=1}^{n}\sigma_{i}(\bm{L})$ is the nuclear norm
                    \item and $\norm{\bm S}_1 := \sum_{i,j}\vert S_{ij}\vert$ is the entrywise $\ell_1$-norm
                  \end{itemize}
                \end{frame}
                \begin{frame}{Robust PCA via ADMM}
                  ADMM for solving robust PCA:
                  \begin{align*}
                    \bm L^{k+1} &= \underset{\bm L}{\textsf{arg~min}} ~~ \norm{\bm L}_{*} + \mathrm{tr}\left(\bm Y^{k\top} \bm L \right) + \dfrac{\rho}{2}\norm{\bm L + \bm S^k - \bm M}^2_{\textrm{F}}\\
                    \bm S^{k+1} &= \underset{\bm S}{\textsf{arg~min}} ~~ \lambda\norm{\bm S}_{1} + \mathrm{tr}\left(\bm Y^{k\top} \bm S \right) + \dfrac{\rho}{2}\norm{\bm L^{k+1} + \bm S - \bm M}^2_{\textrm{F}}\\
                    \bm Y^{k+1} &= \bm Y^k + \rho \left(\bm L^{k+1} + \bm S^{k+1} - \bm M\right)
                  \end{align*}
                \end{frame}
                \begin{frame}{Robust PCA via ADMM}
                  \begin{align*}
                    \bm L^{k+1} &= \textsf{SVT}_{\rho^{-1}}\left(\bm M - \bm S^k - \frac{1}{\rho}\bm Y^k\right)\\
                    \bm S^{k+1} &= \textsf{ST}_{\lambda \rho^{-1}}\left(\bm M - \bm L^{k+1} - \frac{1}{\rho}\bm Y^k\right) \\
                    \bm Y^{k+1} &= \bm Y^k + \rho \left(\bm L^{k+1} + \bm S^{k+1} - \bm M\right),
                  \end{align*}
                  where for any $\bm X$ with \textsf{SVD} $\bm X = \bm U \boldsymbol \Sigma \bm V^\top$, $\boldsymbol \Sigma = \textsf{diag}\left(\left\{\sigma_i\right\}\right)$, we have
                  \begin{equation*}
                    \textsf{SVT}_{\tau}\left(\bm X\right) = \bm U \textsf{diag}\left(\left\{(\sigma_i - \tau)^{+}\right\}\right)\bm V ^\top
                  \end{equation*}
                  and
                  \begin{equation*}
                     \left(\textsf{ST}_{\tau}\left(\bm X\right)\right)_{ij} =
                    \begin{cases}
                      X_{ij} - \tau, & ~\text{if}~ X_{ij} > \tau,\\
                      0, & ~\text{if}~ \vert X_{ij}\vert \leq \tau, \\
                      X_{ij} + \tau, & ~\text{if}~ X_{ij} < -\tau
                    \end{cases}
                  \end{equation*}
                \end{frame}

                \begin{frame}{Graphical Lasso}
                  \textbf{Precision matrix estimation} from Gaussian samples:
                  {\large
                  \begin{equation*}
                     \begin{array}{ll}
                       \underset{\boldsymbol \Theta}{\textsf{minimize}} & \underbrace{- \textrm{log det} ~ \boldsymbol \Theta + \langle \boldsymbol \Theta, \bm S\rangle}_{\text{neg. log likelihood}} + \lambda \norm{\boldsymbol \Theta}_{1}\\
                       \textsf{subject to} & \boldsymbol \Theta \succ \mathbf{0}
                      \end{array}
                 \end{equation*}
                    }
                  Or equivalently, using a slack variable $\boldsymbol \Psi = \boldsymbol \Theta$
                    {\large
                    \begin{equation*}
                       \begin{array}{ll}
                         \underset{\boldsymbol \Theta, \boldsymbol \Psi}{\textsf{minimize}} & \underbrace{- \textrm{log det} ~ \boldsymbol \Theta + \langle \boldsymbol \Theta, \bm S\rangle}_{\text{neg. log likelihood}} + \lambda \norm{\boldsymbol \Psi}_{1}\\
                         \textsf{subject to} & \boldsymbol \Theta \succ \mathbf{0}, \boldsymbol \Theta = \boldsymbol \Psi
                        \end{array}
                   \end{equation*}
                      }
                \end{frame}

                \begin{frame}{Graphical Lasso via ADMM}
                  \begin{align*}
                    \boldsymbol \Theta^{k+1} & = \underset{\boldsymbol \Theta \succ \mathbf{0}}{\textsf{arg min}} ~~ - \textrm{log det} ~ \boldsymbol \Theta + \langle \boldsymbol \Theta, \bm S + \bm Y^k\rangle + \dfrac{\rho}{2}\norm{\boldsymbol \Theta - \boldsymbol \Psi^k}^2_{\textrm{F}}\\
                    \boldsymbol \Psi^{k+1} & = \underset{\boldsymbol \Psi}{\textsf{arg min}} ~~ \lambda \norm{\boldsymbol \Psi}_{1} - \langle \boldsymbol \Psi, \bm Y^k\rangle + \dfrac{\rho}{2}\norm{\boldsymbol \Theta^k - \boldsymbol \Psi}^2_{\textrm{F}}\\
                    \bm Y^{k+1} & = \bm Y^k + \rho\left(\boldsymbol \Theta^{k+1} - \boldsymbol \Psi^{k+1}\right)
                  \end{align*}
                \end{frame}

                \begin{frame}{Graphical Lasso via ADMM}
                  \begin{align*}
                    \boldsymbol \Theta^{k+1} & = \mathcal{F}_\rho\left(\boldsymbol \Psi^k - \dfrac{1}{\rho}\left(\bm Y^k + \bm S\right)\right) \\
                    \boldsymbol \Psi^{k+1} & = \textsf{ST}_{\lambda\rho^{-1}}\left(\boldsymbol \Theta^{k+1} + \dfrac{1}{\rho}\bm Y^{k}\right) \\
                    \bm Y^{k+1} & = \bm Y^k + \rho\left(\boldsymbol \Theta^{k+1} - \boldsymbol \Psi^{k+1}\right)
                  \end{align*}
                  \begin{itemize}
                    \item where $\mathcal{F}_{\rho}(\bm X) := \frac{1}{2} \bm U \textsf{diag}\left(\left\{\lambda_i + \sqrt{\lambda^2_i + \frac{4}{\rho}}\right\}\right)\bm U^\top$, for $\bm X = \bm U \boldsymbol \Lambda \bm U^\top$.
                  \end{itemize}
                \end{frame}

                \begin{frame}{Network of stocks via Graphical Lasso}
                  \begin{figure}[!htb]
                    \centering
                    \includegraphics[scale=0.5]{images/stock-network-glasso.eps}
                  \end{figure}
                \end{frame}

                \begin{frame}{Network of stocks via Graphical Lasso}
                  \begin{figure}[!htb]
                    \centering
                    \includegraphics[scale=0.5]{images/augmented-lagrangian.pdf}
                    ~
                    \includegraphics[scale=0.5]{images/primal-residual.pdf}
                  \end{figure}
                \end{frame}

                \begin{frame}{Conclusion}
                  \begin{itemize}
                    \item ADMM is a versatile/flexible optimization framework
                    \item may not be the best for a specific case, but often performs well in practice
                    \item convergence often needs to be proved in a case-by-case scenario
                  \end{itemize}
                \end{frame}

                \section*{Questions?}
\end{document}


\subsection{Direct Optimization}
\begin{itemize}
	\newpage
	\item Steepest Descent
	\begin{itemize}
		\item Quadratic Interpolation
		\begin{figure}[H]
			\caption{Steepest Descent and Quadratic Interpolation}
			\centering
			\includegraphics[width=11.5cm]{../Figure/Q4/part a Steepest Descent + Quadratic Interpolation.png}
		\end{figure}
		\item Golden Section
		\begin{figure}[H]
			\caption{Steepest Descent and Golden Section}
			\centering
			\includegraphics[width=11.5cm]{../Figure/Q4/part a Steepest Descent + Golden Section.png}
		\end{figure}
	\end{itemize}
\newpage
	\item BFGS
	\begin{itemize}
		\item Quadratic Interpolation
		\begin{figure}[H]
			\caption{BFGS and Quadratic Interpolation}
			\centering
			\includegraphics[width=11.5cm]{../Figure/Q4/part a BFGS + Quadratic Interpolation.png}
		\end{figure}
		\item Golden Section
		\begin{figure}[H]
			\caption{BFGS and Golden Section}
			\centering
			\includegraphics[width=11.5cm]{../Figure/Q4/part a BFGS + Golden Section.png}
		\end{figure}
	\end{itemize}
\end{itemize}