% \documentclass[draft,11pt]{article}
\chapter{Solving Laplacian
  Linear Equations}

%\allowdisplaybreaks

%%% For this lecture
%\newcommand\symset{S}
%\newcommand\psdset{S_+}
%\newcommand\pdset{S_{++}}

%\newcommand\symsetn{\symset^n}
%\newcommand\psdsetn{\psdset^n}
%\newcommand\pdsetn{\pdset^n}

%\newcommand{\vcliq}[2]{\textsc{Clique}\!\left({#1,#2}\right)}
%\newcommand{\vstar}[2]{\textsc{Star}\!\left({#1,#2}\right)}

%%% added by Hongjie
%\newcommand{\Hongjie}[1]{{\color{red} Hongjie: #1}}
%\newcommand{\vcliqsp}[2]{\textsc{CliqueSample}\!\left({#1,#2}\right)}

%%% layout and code
%\usepackage[vlined, ruled]{algorithm2e}
%\usepackage{float}

%\begin{document}
\sloppy
%\lecture{9 --- Wednesday, April 22nd}
%{Spring 2020}{Rasmus Kyng, Scribe: Hongjie Chen}{Solving Laplacian
%  Linear Equations}

% \todo{introduce stopped martingale}

% \todo{remarks, Tropp and Matrix Freedman and things to know}

% \paragraph{outline}
% \begin{itemize}
% \item objective
% \item precond
% \item algo
% \item martingale
% \item mat conc
% \end{itemize}

\section{Solving Linear Equations Approximately}

Given a Laplacian $\LL$ of a connected graph and a demand vector
$\dd \perp \vecone$, we want to find $\xx^*$ solving the linear equation
$\LL \xx^* = \dd$.
We are going to focus on fast algorithms for finding approximate (but
highly accurate) solutions.

This means we need a notion of an approximate solution.
Since our definition is not special to Laplacians, we state it more
generally for positive semi-definite matrices.
\begin{definition}
  Given PSD matrix $\MM$ and $\dd \in \ker(\MM)^{\perp}$, let
  $\MM \xx^* = \dd$.
  We say that $\xxtil$ is an $\epsilon$-approximate solution to the
  linear equation $\MM \xx = \dd$ if
  \[
    \norm{\xxtil - \xx^*}_{\MM}^2 \leq \epsilon \norm{\xx^*}_{\MM}^2.
  \]
\end{definition}

\begin{remark}
  The requirement $\dd \in \ker(\MM)^{\perp}$ can be removed,
   but this is not important for us.
 \end{remark}

\begin{theorem}[Spielman and Teng (2004) \cite{st04}]
Given a Laplacian $\LL$ of a weighted undirected graph $G = (V,E,\ww)$
with $\abs{E}=m$ and $\abs{V} = n$ and a demand vector $\dd \in \R^V$,
we can find $\xxtil$ that is an $\epsilon$-approximate solution to
$\LL \xx = \dd$, using an algorithm that takes time
$O(m \log^c n \log(1/\epsilon))$ for some fixed constant $c$ and succeeds with probability $1 - 1/n^{10}$.
\end{theorem}
In the original algorithm of Spielman and Teng, the exponent on the
log in the running time was $c \approx 70$.

Today, we are going to see a simpler algorithm. But first, we'll look
at one of the key tools behind all algorithms for solving Laplacian
linear equations quickly.

\section{Preconditioning and
Approximate Gaussian Elimination}
Recall our definition of two positive semi-definite matrices being
approximately equal.
\begin{definition}[Spectral approximation]
  Given $\AA, \BB \in \psdsetn$, we say that
  \[
    \AA \approx_{K} \BB
    \text{ if and only if }
    \frac{1}{1+ K} \AA \preceq \BB \preceq (1+ K) \AA.
    \]
\end{definition}
Suppose we have a positive definite matrix $\MM \in \pdsetn$ and want to
solve a linear equation $\MM \xx =\dd$.
We can do this using gradient descent or accelerated gradient
descent, as we covered in Graded Homework 1.
But if we have access to an easy-to-invert matrix that happens to also
be a good spectral approximation of $\MM$, then we can use this to
speed up the (accelerated) gradient descent algorithm.
An example of this would be that we have a factorization
$\matlow\matlow^{\trp} \approx_{K} \MM$, where $\matlow$ is
lower triangular and sparse, which means we can invert it quickly.

The following lemma, which you will prove in Problem Set 6, makes this
preconditioning precise.
\begin{lemma}
  \label{lem:cholprecond}
  Given a matrix $\MM \in \pdsetn$, a vector $\dd$ and a decomposition $\MM
  \approx_{K} \matlow\matlow^{\trp}$,
  we can find $\xxtil$ that $\epsilon$-approximately solves $\MM \xx =
  \dd$, using
$O( (1+ K)\log(K/\epsilon)(T_{\text{matvec}} + T_{\text{sol}} + n) ) $ time.
\begin{itemize}
\item $T_{\text{matvec}}$ denotes the time required to compute $\MM
  \zz$ given a vector $\zz$, i.e.  a ``matrix-vector multiplication''.
\item  $T_{\text{sol}}$ denotes the time required to compute
  $\matlow^{-1}\zz$ or $(\matlow^{\trp})^{-1}\zz$ given a vector
  $\zz$.
\end{itemize}
\end{lemma}
\paragraph{Dealing with pseudo-inverses.}
When our matrices have a null space, preconditioning becomes slightly
more complicated, but as long as it is easy to project to the
complement of the null space, there's no real issue. The following
describes precisely what we need (but you can ignore the null-space
issue when first reading these notes without losing anything
significant).
\begin{lemma}
  \label{lem:cholprecondpinv}
  Given a matrix $\MM \in \psdsetn$, a vector $\dd \in \ker(\MM)^{\perp}$ and a decomposition $\MM
  \approx_{K} \matlow\calDD\matlow^{\trp}$, where $\matlow$ is invertible,
  we can find $\xxtil$ that $\epsilon$-approximately solves $\MM \xx =
  \dd$, using
${O( (1+ K)\log(K/\epsilon)(T_{\text{matvec}} + T_{\text{sol}} +
T_{\text{proj}} + n) ) }$ time.
\begin{itemize}
\item $T_{\text{matvec}}$ denotes the time required to compute $\MM
  \zz$ given a vector $\zz$, i.e.  a ``matrix-vector multiplication''.
\item  $T_{\text{sol}}$ denotes the time required to compute
  $\matlow^{-1}\zz$ and $(\matlow^{\trp})^{-1}\zz$ and $\calDD^{+}\zz$ given a vector
  $\zz$.
\item $T_{\text{proj}}$ denotes the time required to compute
  $\proj_{\MM} \zz$ given a vector $\zz$.
\end{itemize}
\end{lemma}

\begin{theorem}[Kyng and Sachdeva (2015) \cite{ks16}]
  \label{thm:apxgauss}
Given a Laplacian $\LL$ of a weighted undirected graph $G = (V,E,\ww)$
with $\abs{E}=M$ and $\abs{V} = n$,
we can find a decomposition
$\matlow\matlow^{\trp} \approx_{0.5} \LL $, such that
$\matlow$ has number of non-zeroes $\nnz(\matlow) = O(m \log^3 n)$,
with probability at least $1 - 3/n^{5}$.
in time  $O(m \log^3 n)$.
\end{theorem}
We can combine Theorem~\ref{thm:apxgauss} with
Lemma~\ref{lem:cholprecondpinv} to get a fast algorithm for solving
Laplacian linear equations.
% ...
% \begin{corollary}
%   Given a decomposition $\matlow\matlow^{\trp} \approx_{0.5} \LL $,
% \end{corollary}
\begin{corollary}
  \label{cor:precondsolver}
  Given a Laplacian $\LL$ of a weighted undirected graph $G = (V,E,\ww)$
with $\abs{E}=m$ and $\abs{V} = n$ and a demand vector $\dd \in \R^V$,
we can find $\xxtil$ that is an $\epsilon$-approximate solution to
$\LL \xx = \dd$, using an algorithm that takes time $O(m \log^3 n
\log(1/\epsilon))$ and succeeds with probability $1 - 1/n^{10}$.
\end{corollary}
\begin{proof}[Proof sketch]

  First we need to get a factorization that confirms to
  Lemma~\ref{lem:cholprecondpinv}.
  The decomposition $\matlow\matlow^{\trp}$ provided by
  Theorem~\ref{thm:apxgauss} can be rewritten as
  $\matlow\matlow^{\trp} = \matlowtil \calDD (\matlowtil)^{\trp}$
  where
$\matlowtil$ is equal to $\matlow$ except $\matlow(n,n) = 1$
and we let $\calDD$ be the identity matrix, except $\calDD (n,n)
= 0$.
This ensures $\calDD^{\pinv} = \calDD$ and that $\matlowtil$ is
invertible and lower triagular with  $O(m \log^3 n)$ non-zeros.
We note that the inverse of an invertible lower or upper triangular
  matrix with $N$ non-zeros can be applied in time $O(N)$ given an
  adjacency list representation of the matrix.
  Finally, as $\ker(\matlow\matlow^{\trp}) = \Span\setof{\vecone}$,
we have $\proj_{ \matlowtil \calDD (\matlowtil)^{\trp}}
= \II -\frac{1}{n} \vecone \vecone^{\trp}$, and this projection matrix can be
applied in $O(n)$ time.
Altogether, this means that $T_{\text{matvec}} + T_{\text{sol}} +
T_{\text{proj}} = O(n)$, which suffices to complete the proof.
\end{proof}

\section{Approximate Gaussian Elimination Algorithm}
Recall \emph{Gaussian Elimination / Cholesky decomposition} of a graph Laplacian $\LL$.
We will use $\AA(:,i)$ to denote the the $i$th
column of a matrix $\AA$.
We can write the algorithm as
% \begin{algorithm}[h]
%   \begin{algorithmic}
%  \For{$i = 1$ to $i = n-1$}
%   \State $\ll_i = \frac{1}{\sqrt{\SS_{i-1}(i,i)}} \SS_{i-1}(:,i)$
%   \State  $\SS_{i} = \SS_{i-1} - \ll_i \ll_i^{\trp}.$
%   \EndFor
%   \State $\ll_n = \veczero_{n \times 1}$
%   \State $\matlow=\begin{bmatrix} \ll_1 \cdots \ll_n \end{bmatrix}$
% \end{algorithmic}
% \end{algorithm}

\begin{algorithm}[H]
\label{alg:ge}
\caption{Gaussian Elimination / Cholesky Decomposition}
\KwIn{Graph Laplacian $\LL$}
\KwOut{Lower triangular $\matlow$ s.t. $\matlow\matlow^{\trp} = \LL$}
Let $\SS_0 = \LL$ \;
 \For{$i = 1$ to $i = n-1$}{
$\ll_i = \frac{1}{\sqrt{\SS_{i-1}(i,i)}} \SS_{i-1}(:,i)$\;
$\SS_{i} = \SS_{i-1} - \ll_i \ll_i^{\trp}.$\
}
$\ll_n = \veczero_{n \times 1}$\;
\Return{$\matlow=\begin{bmatrix} \ll_1 \cdots \ll_n \end{bmatrix}$}\;
\end{algorithm}
% Now, for $i = 1$ to $i = n-1$ we define
% \begin{align*}
%   \ll_i &= \frac{1}{\sqrt{\SS_{i-1}(i,i)}} \SS_{i-1}(:,i), \\
%   \SS_{i} &= \SS_{i-1} - \ll_i \ll_i^{\trp}.
% \end{align*}
% Finally, we let $\ll_n = \veczero_{n \times 1}$. It follows that
% $\matlow=\begin{bmatrix} \ll_1 \cdots \ll_n \end{bmatrix}$ is lower
% triangular and $\LL=\matlow\matlow^\trp$.
We want to introduce some notation that will help us describe and
analyze a faster version of Gaussian elimination -- one that uses sampling
to create a sparse approximation of the decomposition.

Consider a Laplacian $\SS$ of a graph $H$ and a vertex $v$ of $H$.
We define $\vstar{v}{\SS}$ to be the Laplacian of the subgraph of $H$
consisting of edges incident on $v$.
We define
\[
\vcliq{v}{\SS} = \vstar{v}{\SS} - \frac{1}{\SS(v,v)} \SS(:,v)  \SS(:,v)^{\trp}
\]
For example, suppose
\[
  \LL =
\left(
\begin{array}{ccc}
W & -\aa^\trp \\
-\aa& \diag(\aa) + \LL_{-1}
\end{array} \right)
\]
Then
\[
\vstar{1}{\LL}
=
\left(
\begin{array}{ccc}
W & -\aa^\trp \\
-\aa& \diag(\aa)
\end{array} \right)
\text{ and }
\vcliq{1}{\LL}
=
\left(
\begin{array}{ccc}
0 &  \veczero \\
\veczero & \diag(\aa) - \frac{1}{W} \aa \aa^\trp
\end{array} \right)
\]
which is illustrated in Figure \ref{fig:schurclique}.
%
\begin{figure}[H]
  \centering
  \includegraphics[width=1
  \textwidth]{fig/lecture7_schur-clique.jpeg}
  \caption{Gaussian Elimination:
    $\vcliq{1}{\LL} = \vstar{1}{\LL} - \frac{1}{\LL(1,1)} \LL(:,1)
    \LL(:,1)^{\trp}$.}
      \label{fig:schurclique}
    \end{figure}
In Chapter~\ref{cha:ge}, we proved that $\vcliq{v}{\SS}$ is a graph Laplacian --
it follows from the proof of Claim~\ref{clm:optimgaussclosed} in that chapter.
Thus we have that following.
 \begin{claim}
\label{clm:optimgaussclosedagain}
If $\SS$ is the Laplacian of a connected graph, then
$\vcliq{v}{\SS}$ is a graph Laplacian.
% \begin{enumerate}
% \item $\vcliq{v}{\SS}$ is a graph Laplacian.
% \item
% \end{enumerate}
\end{claim}

Note that in Algorithm~\ref{alg:ge}, we have  $\ll_i\ll_i^\trp = \vstar{v_i}{\SS_{i-1}}-\vcliq{v_i}{\SS_{i-1}}$. The update rule can be rewritten as
\[ \SS_{i} = \SS_{i-1} - \vstar{v_i}{\SS_{i-1}} +
  \vcliq{v_i}{\SS_{i-1}}, \]

This also provides way to understand why
Gaussian Elimination is slow in some cases.
At each step, one vertex is eliminated, but a clique is added to the
subgraph on the remaining vertices, making the graph denser.
And at the $i$th step, computing $\vstar{v_i}{\SS_{i-1}}$
takes around $\deg(v_i)$ time, but computing $\vcliq{v_i}{\SS_{i-1}}$
requires around $\deg(v_i)^2$ time.
In order to speed up Gaussian
Elimination, the algorithmic idea of \cite{ks16}
%\cite{?}
 is to plug in a sparser appproximate of the intended clique instead
 of the entire one.

The following procedure $\vcliqsp{v}{\SS}$ produces a sparse
approximation of $\textsc{clique}(v,\SS)$.
Let $V$ be the vertex set of the graph associated with $\SS$ and $E$
the edge set.
We define $\bb_{i,j} \in \R^V$ to be the vector with
\[
  \bb_{i,j}(i) = 1
  \text{ and }
  \bb_{i,j}(j) = -1
  \text{ and }
  \bb_{i,j}(k) = 0
  \text{ for }
  k \neq i,j.
\]
Given weights $\ww \in \R^E$ and a vertex $v \in V$,
we let
\[
  \ww_v = \sum_{(u,v) \in E} \ww(u,v)
  .
  \]
\begin{algorithm}[H]
\label{alg:cliquesamp}
\caption{$\vcliqsp{v}{\SS}$}
\KwIn{Graph Laplacian $\SS \in \R^{V \times V}$, of a graph with edge
  weights $\ww$, and vertex $v \in V$}
\KwOut{$\YY_v \in \R^{V \times V}$ sparse approximation of
  $\textsc{clique}(v,\SS)$  }
 $\YY_v   \gets  \matzero_{n \times n} $\;
 \ForEach{Multiedge $e
   = (v,i)$ from $v$ to a neighbor $i$}{
    Randomly pick a neighbor $j$ of $v$ with probability $\frac{\ww(j,v)}{\ww_v}$\;
    If $i \neq j$, let $\YY_v   \gets \YY_v + \frac{\ww(i,v)\ww(j,v)}{\ww(i,v)+\ww(j,v)} \bb_{i,j}\bb_{i,j}^\trp$\;
}
\Return{$\YY_v$}\;
\end{algorithm}
% Let $\YY_v = \matzero_{n \times n}$. For each neighbor $i$ of $v$, randomly pick a neighbor $j$ of $v$ with probability $\frac{\ww(j,v)}{\ww_v}$ where $\ww_v = \sum_{u \sim v} \ww(u,v)$ is the sum of weights of edges incident to $v$. If $i\neq j$, then draw an edge with weight $\ww(i,v)\ww(j,v)/(\ww(i,v)+\ww(j,v))$ between $i$ and $j$, i.e. update $\YY_v$ by
% \[ \YY_v \gets \YY_v + \frac{\ww(i,v)\ww(j,v)}{\ww(i,v)+\ww(j,v)} \bb_{i,j}\bb_{i,j}^\trp. \]
%
\begin{remark}
  We can implement each sampling of a neighbor $j$ in $O(1)$ time
  using a classical algorithm known as Walker's method (also known as the Alias
  method or Vose's method).
  This algorithm requires an additional $O(\deg_{\SS}(v))$ time to
  initialize a data structure used for sampling.
  Overall, this means the total time for $O(\deg_{\SS}(v))$ samples is still $O(\deg_{\SS}(v))$.
\end{remark}

% %TODO this paragraph sometimes causes horizontal "badness" failure
% In what sense does $\vcliqsp{v}{\SS}$ produce a sparse approximation
% of the clique $\vcliq{v}{\SS}$ created by Gaussian elimination?
% This is a crucial question, and the next lemma gives part of the
% answer: Namely that the output in expectation equals the clique.
\begin{lemma}\label{lem:cliquesample_expectation}
  $\E{\YY_v} = \vcliq{v}{\SS}$.
\end{lemma}
\begin{proof}
  Let $\CC=\vcliq{v}{\SS}$. Observe that both $\E{\YY_v}$ and $\CC$ are Laplacians. Thus it suffices to verify $\Ex{\YY_v(i,j)}=\CC(i,j)$ for $i\neq j$.
  \[ \CC(i,j) = -\frac{\ww(i,v)\ww(j,v)}{\ww_v}, \]
  \[ \Ex{\YY_v(i,j)} = -\frac{\ww(i,v)\ww(j,v)}{\ww(i,v)+\ww(j,v)} \left( \frac{\ww(j,v)}{\ww_v} + \frac{\ww(i,v)}{\ww_v} \right) = -\frac{\ww(i,v)\ww(j,v)}{\ww_v} = \CC(i,j). \]
\end{proof}
\begin{remark}
  Lemma \ref{lem:cliquesample_expectation} shows that $\vcliqsp{v}{\LL}$ produces the original $\vcliq{v}{\LL}$ in expectation.
\end{remark}

Now, we define \emph{Approximate Gaussian Elimination}.
% Let $\SS_0 = \LL$. Generate a radom permutation $\pi$ on $[n]$. For $i = 1$ to $i = n-1$ we define
% \begin{align*}
%   \ll_i &= \frac{1}{\sqrt{\SS_{i-1}(\pi(i),\pi(i))}} \SS_{i-1}(:,\pi(i)), \\
%   \SS_{i} &= \SS_{i-1} - \vstar{\pi(i)}{\SS_{i-1}} + \vcliqsp{\pi(i)}{\SS_{i-1}}.
              %   \end{align*}

\begin{algorithm}[H]
\label{alg:apxge}
\caption{Approximate Gaussian Elimination / Cholesky Decomposition}
\KwIn{Graph Laplacian $\LL$}
\KwOut{Lower triangular\footnote{$\matlow$ is not actually lower
    triangular. However, if we let $\PP_\pi$ be the permutation matrix
    corresponding to $\pi$, then $\PP_\pi \matlow$ is lower
    triangular. Knowing the ordering that achieves this is enough to
    let us implement forward and backward substitution for solving
    linear equations in $\matlow$ and $\matlow^\trp$.}
  $\matlow$ as
  given in Theorem~\ref{thm:apxgauss}}
Let $\SS_0 = \LL$\;
Generate a random permutation $\pi$ on $[n]$\;
\For{$i = 1$ to $i = n-1$}{
$\ll_i = \frac{1}{\sqrt{\SS_{i-1}(\pi(i),\pi(i))}} \SS_{i-1}(:,\pi(i)) $\;
$\SS_{i} = \SS_{i-1} - \vstar{\pi(i)}{\SS_{i-1}} + \vcliqsp{\pi(i)}{\SS_{i-1}}$\
}
$\ll_n = \veczero_{n \times 1}$\;
\Return{$\matlow=\begin{bmatrix} \ll_1 \cdots \ll_n \end{bmatrix}$ and
$\pi$}\;
\end{algorithm}
% $\PP_\pi$ be the permutation matrix corresponding to $\pi$. It follows that $\matlow =
% \PP_\pi \begin{bmatrix} \ll_1 \cdots \ll_n \end{bmatrix}$ is lower
% triangular and $\LL=\matlow\matlow^\trp$.
Note that if we replace
$\vcliqsp{\pi(i)}{\SS_{i-1}}$ by $\vcliq{\pi(i)}{\SS_{i-1}}$ at each
step, then we can recover Gaussian Elimination, but with a random
elimination order.

\section{Analyzing Approximate Gaussian Elimination}

In this Section, we're going to analyze Approximate Gaussian
Elimination, and see why it works.

Ultimately, the main challenge in proving Theorem~\ref{thm:apxgauss}
will be to prove for the output $\matlow$
of Algorithm~\ref{alg:apxge} that with high probability
\begin{equation}
  \label{eq:relerrgoal}
0.5 \LL
\preceq
\matlow \matlow^{\trp}
\preceq 1.5 \LL
.
\end{equation}
We can reduce this to proving that with high probability
\begin{equation}
\label{eq:spectralnormgoal}
 \norm{\LL^{+/2} (\matlow \matlow^{\trp} -  \LL) \LL^{+/2}} \leq 0.5
\end{equation}
Ultimately, the proof is going to have a lot in common with our proof
of Matrix Bernstein in Chapter~\ref{cha:randmat}.
Overall, the lesson there was that when we have a sum of independent,
zero-mean random matrices, we can show that the sum is likely to have
small spectral norm if the spectral norm of each random matrix is
small, and the matrix-valued variance is also small.

Thus, to replicate the proof, we need control over
\begin{enumerate}
\item The \emph{sample norms}.
\item The \emph{sample variance}.

\end{enumerate}

But, there is seemlingly another major obstacle: We are trying to
analyze a process where the samples are far from independent.
Each time we sample edges, we add new edges to the remaining graph,
which we will the later sample again. This creates a lot of
dependencies between the samples, which we have to handle.

However, it turns out that independence is more than what is needed to
prove concentration. Instead, it suffices to have a sequence of random
variables such that each is mean-zero in expectation, conditional on
the previous ones. This is called a martingale difference sequence.
We'll now learn about those.

\subsection{Normalization, a.k.a. Isotropic Position}

Since our analysis requires frequently measuring matrices after right
and left-multiplication by $\LL^{\pinv/2}$, we reintroduce the
``normalizing map'' $\Phi : \R^{n\times n} \to
\R^{n\times n} $
defined by
\[
  \Phi(\AA) = \LL^{\pinv/2}\AA\LL^{\pinv/2}.
  \]
We previously saw this in Chapter~\ref{cha:randmat}.

\subsection{Martingales}

A scalar martingale is a sequence of random variables $Z_0, \ldots,
Z_k$, such that
\begin{align}
\label{eq:martingale}
  \E{Z_i \mid Z_0, \ldots, Z_{i-1}} = Z_{i-1}
.
\end{align}
That is, conditional on the outcome of all the previous random
variables, the expectation of $Z_{i}$ equals $Z_{i-1}$.
If we unravel the sequence of conditional expectations, we
get that \emph{without conditioning}
$\E{Z_k}=\E{Z_0}$.

Typically, we use martingales to show a statement along like  ``$Z_k$
is concentrated around $\E{Z_k}$''.

We can also think of a martingale in terms of the sequence of changes
in the $Z_i$ variables.
Let $X_i = Z_i - Z_{i-1}$.
The sequence of $X_i$s is called a martingale difference sequence.
We can now state the martingale condition as
\[
  \E{X_i \mid Z_0, \ldots, Z_{i-1}} = 0
.
\]
And because $Z_0$ and $X_{1}, \ldots, X_{i-1}$ completely determine
$Z_1,  \ldots, Z_{i-1}$, we could also write
the martingale condition equivalently as
\[
  \E{X_i \mid Z_0, X_1, \ldots, X_{i-1}} = 0
  .
\]
Crucially, we can write
\[
Z_k = Z_0 + \sum_{i=1}^k Z_i - Z_{i-1} = Z_0 + \sum_{i=1}^k X_i
\]
and when we are trying to prove concentration, the martingale difference property
of the $X_i$'s is often ``as good as'' independence, meaning that
$\sum_{i=1}^k X_i$ concentrates similarly to a sum of independent
random variables.

\paragraph{Matrix-valued martingales.}
We can also define matrix-valued martingales.
In this case, we replace the martingalue condition of
Equation~\eqref{eq:martingale}, with the condition that the whole matrix
stays the same in expectation.
For example, we could have a sequence of random matrices $\ZZ_0, \ldots,
\ZZ_k \in \R^{n \times n}$, such that
\begin{align}
\label{eq:matmartingale}
  \E{\ZZ_i \mid \ZZ_0, \ldots, \ZZ_{i-1}} = \ZZ_{i-1}
.
\end{align}
\begin{lemma}\label{lem:ApxGE_martingale}
  Let $\LL_i = \SS_i + \sum_{j=1}^i \ll_j\ll_j^\trp$ for $i=1,...,n$ and $\LL_0=\SS_0=\LL$. Then
  \[ \E{\LL_i | \text{all random variables before } \vcliqsp{\pi(i)}{\SS_{i-1}} } = \LL_{i-1}. \]
\end{lemma}
\begin{proof}
  Let's only consider $i=1$ here as other cases are similar.
  \[ \LL_0 = \LL = \ll_1\ll_1^\trp + \vcliq{v}{\LL} + \LL_{-1} \]
  \begin{align*}
    \LL_1 &= \ll_1\ll_1^\trp + \vcliqsp{v}{\LL} + \LL_{-1} \\
    \E{\LL_1 | \pi(1)}
    &= \ll_1\ll_1^\trp + \E{\vcliqsp{v}{\LL}|\pi(1)} + \LL_{-1} \\
    &= \ll_1\ll_1^\trp + \vcliq{v}{\LL} + \LL_{-1} \\
    &= \LL_0
  \end{align*}
  where we used Lemma~\ref{lem:cliquesample_expectation} to get
  $\E{\vcliqsp{v}{\LL}|\pi(1)}  = \vcliq{v}{\LL} $.
\end{proof}
\begin{remark}
  $\sum_{j=1}^i \ll_j\ll_j^\trp$ can be treated as what has already been eliminated by (Approximate) Gaussian Elimination, while $\SS_i$ is what still left or going to be eliminated.
  In Approximate Gaussian Elimination, $\LL_n=\sum_{i=1}^n\ll_i\ll_i^\trp$ and our goal is to show that $\LL_n\approx_K\LL$.
  Note that $\LL_i$ is always equal to the original Laplacian $\LL$ for all $i$ in Gaussian Elimination.
  Lemma \ref{lem:ApxGE_martingale} demonstrates that $\LL_0,\LL_1,...,\LL_n$ forms a matrix martingale.
\end{remark}

Ultimately, our plan is to use this matrix martingale structure to show
that ``$\LL_n$ is concentrated around $\LL$'' in some appropriate
sense.
More precisely, the spectral approximation we would like to show can
be established by showing that  ``$\Phi(\LL_n)$ is concentrated around $\Phi(\LL)$''

\subsection{Martingale Difference Sequence as Edge-Samples}
We start by taking a slightly different view of the observations we
used to prove Lemma~\ref{lem:ApxGE_martingale}.
Recall that $\LL_i = \SS_i + \sum_{j=1}^i \ll_j\ll_j^\trp$,
and  $\LL_{i-1} = \SS_{i-1} + \sum_{j=1}^{i-1} \ll_j\ll_j^\trp$
and
\[
  \SS_{i} = \SS_{i-1} - \vstar{\pi(i)}{\SS_{i-1}} +
  \vcliqsp{\pi(i)}{\SS_{i-1}}
 . \]
Putting these together, we get
\begin{align}
  \LL_i - \LL_{i-1}
  &=
  \ll_i\ll_i^\trp
  +
  \vcliqsp{\pi(i)}{\SS_{i-1}}
  -
    \vstar{\pi(i)}{\SS_{i-1}}
     \nonumber
  \\
    \label{eq:apxgaussdiffseq}
  &=
  \vcliqsp{\pi(i)}{\SS_{i-1}}
  -
    \vcliq{\pi(i)}{\SS_{i-1}}
\\
  &=
  \vcliqsp{\pi(i)}{\SS_{i-1}}
  -
    \E{\vcliqsp{\pi(i)}{\SS_{i-1}} \mid \text{preceding samples}}
 \tag*{by Lemma~\ref{lem:cliquesample_expectation}.}
\end{align}
In particular, recall that by Lemma~\ref{lem:cliquesample_expectation},
conditional on the randomness before
the call to $\vcliqsp{\pi(i)}{\SS_{i-1}}$, we
have
\[
\E{\vcliqsp{\pi(i)}{\SS_{i-1}} \mid \text{preceding samples}}= \vcliq{\pi(i)}{\SS_{i-1}}
\]
Adopting the notation of Lemma~\ref{lem:cliquesample_expectation}
we write
\[
  \YY_{\pi(i)} = \vcliqsp{\pi(i)}{\SS_{i-1}}
\]
and we further introduce notation each multi-edge sample for
$e \in \vstar{\pi(i)}{\SS_{i-1}}$, as $\YY_{\pi(i),e}$, denoting the random
edge Laplacian sampled when the algorithm is processing multi-edge
$e$.
Thus, conditional on preceding samples, we have

\begin{equation}
  \label{eq:cliquesamplesum}
  \YY_{\pi(i)} = \sum_{e \in
    \vstar{\pi(i)}{\SS_{i-1}}}\YY_{\pi(i),e}
\end{equation}
Note that even the number of multi-edges in
$\vstar{\pi(i)}{\SS_{i-1}}$ depends on the preceding samples.
We also want to associate zero-mean variables with each edge.
Conditional on preceding samples, we also define
\[
  \XX_{i,e} = \Phi\left( \YY_{\pi(i),e} - \E{\YY_{\pi(i),e}} \right)
  \text{ and }
  \XX_{i} = \sum_{e \in
    \vstar{\pi(i)}{\SS_{i-1}}}\XX_{i,e}
\]
and combining this with Equations~\eqref{eq:apxgaussdiffseq} and \eqref{eq:cliquesamplesum}
\[
  \XX_{i}
  = \Phi(\YY_{\pi(i)} - \E{\YY_{\pi(i)}})
  = \Phi(\LL_i - \LL_{i-1})
\]
Altogether, we can write
\[
  \Phi\left(  \LL_n -\LL \right)
  =
    \sum_{i=1}^n \Phi(\LL_i - \LL_{i-1})
  =
  \sum_{i=1}^n \XX_i
  =
  \sum_{i=1}^n \sum_{e \in
    \vstar{\pi(i)}{\SS_{i-1}}}\XX_{i,e}
\]
Note that the $\XX_{i,e}$ variables form a martingale difference
sequence, because the linearity of $\Phi$ ensures they are zero-mean
conditional on preceding randomness.

\subsection{Stopped Martingales}
Unfortunately, directly analyzing the concentration properties of
the $\LL_i$ martingale that we just introduced turns out to be
difficult.
The reason is that we're trying to prove some very delicate
multiplicative error guarantees.
And, if we analyze $\LL_i$, we find that the multiplicative error is
not easy to control, \emph{after it's already gotten big}.
But that's not really what we care about anyway: We want to say it
never gets big in the first place, with high probability.
So we need to introduce another martingale, that lets us ignore the
bad case when the error has already gotten too big.
At the same time, we also need to make sure that statements about our
new martingale can help us prove guarantees about  $\LL_i$.
Fortunately, we can achieve both at once.
The technique we use is related to the much broader topic of
martingale \emph{stopping times}, which we only scratch the surface of
here. We're also going to be quite informal about it, in the interest
of brevity.
Lecture notes by Tropp \cite{tropp19} give a more formal introduction for
those who are interested.

We define the stopped martingale sequence $\LLtil_i$ by
\begin{equation}
  \label{eq:stoppedmartingale}
  \LLtil_i
  =
  \begin{cases}
    \LL_i & \text{ if for all } j < i \text{ we have }
       \LL_i \preceq 1.5 \LL
       \\
    \LL_{j^*} & \text{ for } j^* \text{ being the least $j$ such that }  \LL_{j} \not\preceq 1.5 \LL
  \end{cases}
\end{equation}

Figure~\ref{fig:stopmart} shows the $\LLtil_i$ martingale getting
stuck at the first time $\LL_{j^*} \not\preceq 1.5 \LL$.
\begin{figure}[H]
  \centering
  \includegraphics[width=1
  \textwidth]{fig/lec9_stoppedmartingale.png}
  \caption{Gaussian Elimination
    :
    $\vcliq{1}{\LL} = \vstar{1}{\LL} - \frac{1}{\LL(1,1)} \LL(:,1)
    \LL(:,1)^{\trp}$.}
      \label{fig:stopmart}
    \end{figure}

We state the following without proof:
\begin{claim}
\noindent
  \begin{enumerate}
  \item The sequence $\setof{\LLtil_i}$ for $i = 0, \ldots, n$ is a
    martingale.
  \item $\norm{\LL^{+/2} (\LLtil_i -  \LL) \LL^{+/2}} \leq 0.5$
    implies $\norm{\LL^{+/2} (\LL_i -  \LL) \LL^{+/2}} \leq 0.5$
  \end{enumerate}
\end{claim}
The martingale property also implies that the unconditional
expectation satisfies $\E{\LLtil_n} = \LL$.
The proof of the claim is easy to sketch: For Part 1, each difference is
zero-mean if the condition has not been violated, and is identically
zero (and hence zero-mean) if it has been violated.
For Part 2, if the martingale $\setof{\LLtil_i}$ has stopped,
then $\norm{\LL^{+/2}
  (\LLtil_i -  \LL) \LL^{+/2}} \leq 0.5$ is false, and the implication
is vacuosly true.
If, on the other hand, the martingale has not stopped, the
quantities are equal, because $\LLtil_i  = \LL_i$, and again it's easy
to see the implication holds.

Thus, ultimately, our strategy is goin to be to show that
$\norm{\LL^{+/2} (\LLtil_i -  \LL) \LL^{+/2}} \leq 0.5$
with high probability.
Expressed using the normalizing map $\Phi(\cdot)$,
our goal is to show that with high probability
\[
  \norm{\Phi(\LLtil_n -  \LL)} \leq 0.5.
\]

\paragraph{Stopped martingale difference sequence.}
In order to prove the spectral norm bound, we want to express the
$\setof{\LLtil_i}$ martingale in terms of a sequence of martingale
differences.
To this end, we define
$\XXtil_i = \Phi(\LLtil_i -  \LLtil_{i-1})$.
This ensures that
\begin{equation}
  \label{eq:stoppedmartingale}
  \XXtil_i
  =
  \begin{cases}
    \XX_i& \text{ if for all } j < i \text{ we have }
       \LL_i \preceq 1.5 \LL
       \\
    \matzero & \text{ otherwise }
  \end{cases}
\end{equation}
Whenever the modified martingale $\XXtil_i$ has not yet stopped, we
also introduce individual modified edge samples $\XXtil_{i,e} =
\XX_{i,e}$.
If the martingale \emph{has} stopped, i.e. $\XXtil_i = \matzero$,
then we can take these edge samples $\XXtil_{i,e}$
to be zero.
We can now write
\[
  \Phi\left(  \LLtil_n -\LL \right)
  =
    \sum_{i=1}^n \Phi(\LLtil_i - \LLtil_{i-1})
  =
  \sum_{i=1}^n \XXtil_i
  =
  \sum_{i=1}^n \sum_{e \in
    \vstar{\pi(i)}{\SS_{i-1}}}\XXtil_{i,e}
  .
\]
Thus, we can see that Equation~\eqref{eq:spectralnormgoal} is implied by
\begin{equation}
  \label{eq:zeromeanspectralnormgoal}
  \norm{\sum_{i=1}^n \XXtil_i} \leq 0.5.
\end{equation}

\subsection{Sample Norm Control}
In this Subsection, we're going to see that the norms of each
multi-edge sample is controlled throughout the algorithm.
\begin{lemma}\label{lem:cliquesample_bounded_norm}
  Given two Laplacians $\LL$ and $\SS$ on the same vertex
  set.\footnote{$\LL$ can be regarded as the original Laplacian we
    care about, while $\SS$ can be regarded as some intermediate
    Laplacian appearing during Approximate Gaussian Elimination.} If
  each multiedge $e$ of $\vstar{v}{\SS}$ has bounded norm in the following sense,
  \[ \norm{\LL^{\pinv/2}\ww_{\SS}(e)\bb_e\bb_e^{\trp}\LL^{\pinv/2}} \leq R, \]
  then each possible sampled multiedge $e'$ of $\vcliqsp{v}{\SS}$ also satisfies
  \[ \norm{\LL^{\pinv/2}\ww_{\mathrm{new}}(e')\bb_{e'}\bb_{e'}^{\trp}\LL^{\pinv/2}} \leq R. \]
\end{lemma}
\begin{proof}
  Let $\ww=\ww_{\SS}$ for simplicity. Consider a sampled edge between $i$ and $j$ with weight $\ww_{\mathrm{new}}(i,j) = \ww(i,v)\ww(j,v)/(\ww(i,v)+\ww(j,v))$.
  \begin{align*}
    \norm{\LL^{\pinv/2} \ww_{\mathrm{new}}(i,j) \bb_{ij}\bb_{ij}^{\trp}\LL^{\pinv/2}}
    &= \ww_{\mathrm{new}}(i,j) \norm{\LL^{\pinv/2} \bb_{ij}\bb_{ij}^{\trp}\LL^{\pinv/2}} \\
    &= \ww_{\mathrm{new}}(i,j) \norm{ \LL^{\pinv/2}\bb_{ij} }^2 \\
    &\leq \ww_{\mathrm{new}}(i,j) \left( \norm{\LL^{\pinv/2}\bb_{iv}}^2 + \norm{\LL^{\pinv/2}\bb_{jv}}^2 \right) \\
    &= \frac{\ww(j,v)}{\ww(i,v)+\ww(j,v)} \norm{\LL^{\pinv/2} \ww(i,v)\bb_{iv}\bb_{iv}^{\trp}\LL^{\pinv/2}} + \\
    & ~\quad \frac{\ww(i,v)}{\ww(i,v)+\ww(j,v)} \norm{\LL^{\pinv/2} \ww(j,v)\bb_{jv}\bb_{jv}^{\trp}\LL^{\pinv/2}} \\
    &\leq \frac{\ww(j,v)}{\ww(i,v)+\ww(j,v)} R + \frac{\ww(i,v)}{\ww(i,v)+\ww(j,v)} R \\
    &= R
  \end{align*}
  The first inequality uses the triangle inequality of effective
  resistance in $\LL$, in that effective resistance is a distance as
  we proved in Chapter~\ref{cha:pinver}. The second inequality just uses the conditions of this lemma.
\end{proof}
\begin{remark}
  Lemma \ref{lem:cliquesample_bounded_norm} only requires that each single multiedge has small norm instead of that the sum of all edges between a pair of vertices have small norm. And this lemma tells us, after sampling, each multiedge in the new graph still satisfies the bounded norm condition.
\end{remark}
From the Lemma, we can conclude that each edge sample $\YY_{\pi(i),e}$
satisfies
$\norm{\Phi(\YY_{\pi(i),e})} \leq R$
provided the assumptions of the
Lemma hold. Let's record this observation as a Lemma.
\begin{lemma}
  If for all $e \in \vstar{v}{\SS_i}$,
  \[
    \norm{\Phi(\ww_{\SS_i }(e)\bb_{e} \bb_{e}^{\trp})}
    \leq
    R
    .
  \]
  then all $e \in \vstar{\pi(i)}{\SS_i}$,
  \[
    \norm{\Phi(\YY_{\pi(i),e})} \leq R
 .
  \]
\end{lemma}

\paragraph{Preprocessing by multi-edge splitting.}
In the original graph of Laplacian $\LL$ of graph $G = (V,E,\ww)$, we
have for each edge $\hat{e}$ that
\[
  \ww(\hat{e}) \bb_{\hat{e}}\bb_{\hat{e}}^{\trp} \preceq \sum_e \ww(e) \bb_{e}\bb_{e}^{\trp}= \LL
\]
This also implies that
\[
  \norm{\LL^{\pinv/2}  \ww(\hat{e})  \bb_{\hat{e}}\bb_{\hat{e}}^{\trp} \LL^{\pinv/2} }\leq 1.
\]
Now, that means that if we split every original edge $e$ of the graph
into $K$ multi-edges $e_1, \ldots e_K$, with a fraction $1/K$ of the
weight, we get a new graph $G' = (V, E', \ww')$ such that
\begin{claim}
  \label{clm:edgesplit}
  \noindent
  \begin{enumerate}
  \item $G'$ and $G$ have the same graph Laplacian.
  \item $\abs{E'} = K\abs{E}$
  \item For every multi-edge in $G'$
    \[
      \norm{\LL^{\pinv/2} \ww'(e) \bb_{e}\bb_{e}^{\trp} \LL^{\pinv/2}
      }\leq 1/K.
    \]
  \end{enumerate}
\end{claim}
Before we run Approximate Gaussian Elimination, we are going to do
this multi-edge splitting to ensure we have control over multi-edge
sample norms.
%
Combined with Lemma~\ref{lem:cliquesample_bounded_norm} immediately
establishes the next lemma, because we start off with all multi-edges
having bounded norm and only produce multi-edges with bounded norm.
\begin{lemma}
  \label{lem:edgesampnorm}
  When Algorithm~\ref{alg:apxge} is run on the (multi-edge) Laplacian of $G'$,
  arising from splitting edges of $G$ into $K$ multi-edges, the every
  edge sample $\YY_{\pi(i),e}$ satisfies
  \[
    \norm{\Phi(\YY_{\pi(i),e})} \leq 1/K
 .
  \]
\end{lemma}
As we will see later $K = 200 \log^2 n$ suffices.

\subsection{Random Matrix Concentration from Trace Exponentials}
Let us recall how matrix-valued variances come into the picture when
proving concentration following the strategy from Matrix Bernstein in
Chapter~\ref{cha:randmat}.

For some matrix-valued random variable $\XX \in \symsetn$,
we'd like to show $\Pr[ \norm{\XX} \leq 0.5 ]$.
Using Markov's inequality, and some observations about matrix
exponentials and traces, we saw that for all $\theta > 0$,
\begin{equation}
  \label{eq:masterprob}
  \Pr[ \norm{\XX} \geq 0.5 ]
  \leq
  \exp(-0.5\theta ) \left(\E{\trace{ \exp\left(\theta \XX\right)}}
  +
  \E{\trace{ \exp\left(-\theta \XX\right)}}
  \right)
  .
\end{equation}
We then want to bound $\E{\trace{ \exp\left(\theta \XX\right)}}$
using Lieb's theorem.
We can handle $\E{\trace{ \exp\left(-\theta \XX\right)}}$ similarly.
\begin{theorem}[Lieb]
  \label{thm:Lieb}
  Let $f:S^n_{++}\to\R$ be a matrix function given by
  \[ f(\AA) = \trace{\exp\left(\HH+\log(\AA)\right)} \]
  for some $\HH\in S^n$. Then $-f$ is convex (i.e. $f$ is concave).
\end{theorem}
As observed by Tropp, this is useful for proving matrix concentration statements.
Combined with Jensen's inequality, it gives that for a random matrix
$\XX \in \symsetn $ and a fixed $\HH \in \symsetn$
\[
  \E{\trace{ \exp\left(\HH+\XX\right)}}
    \leq
    \trace{ \exp\left(\HH+\log(\E{\exp(\XX)})\right)}
  .
\]
The next crucial step was to show that it suffices to obtain an upper
bound on the matrix $\E{\exp(\XX)}$ w.r.t the Loewner order.
Using the following three lemmas, this conclusion is an immediate
corollary.
\begin{lemma}\label{lem:trexpmono}
  If $\AA\preceq\BB$, then $\trace{\exp(\AA)}\leq\trace{\exp(\BB)}$.
  % i.e. $\ZZ \mapsto \trace{\exp(\ZZ)}$ is monotone increasing.
\end{lemma}
\begin{lemma}\label{lem:logmono}
  If $0\prec\AA\preceq\BB$, then $\log(\AA)\preceq\log(\BB)$.
\end{lemma}
\begin{lemma}\label{lem:ineq_log}
  $\log(\II+\AA) \preceq \AA$ for $\AA\succ-\II$.
\end{lemma}
%
\begin{corollary}
  \label{cor:trexpub}
  For a random matrix
  $\XX \in \symsetn $ and a fixed $\HH \in \symsetn$,
  if $\E{\exp(\XX)} \preceq \II + \UU$ where $\UU \succ -\II$, then
\[
  \E{\trace{ \exp\left(\HH+\XX\right)}}
    \leq
    \trace{ \exp\left(\HH+\UU\right)}
  .
\]
\end{corollary}

% \begin{corollary}\label{cor:trexpmono}
%   If $\AA\preceq\BB$, then $\trace{\exp(\AA)}\leq\trace{\exp(\BB)}$,
%   i.e. $\XX \mapsto \trace{\exp(\XX)}$ is monotone increasing.
% \end{corollary}
% \begin{lemma}\label{lem:logmono}
%   If $0\prec\AA\preceq\BB$, then $\log(\AA)\preceq\log(\BB)$.
% \end{lemma}

\subsection{Mean-Exponential Bounds from Variance Bounds}
To use Corollary~\ref{cor:trexpub}, we need to construct useful upper
bounds on $\E{\exp(\XX)}$.
This can be done, starting from the following lemma.
\begin{lemma}\label{lem:ineq_exp}
  $\exp(\AA) \preceq \II + \AA + \AA^2$ for $\|\AA\|\leq1$.
\end{lemma}
If $\XX$ is zero-mean and $\norm{\XX} \leq 1$,
this means that $\E{\exp(\XX)} \preceq \II + \E{\XX^2}$,
which is how we end up wanting to bound the matrix-valued variance $\E{\XX^2}$.
In the rest of this Subsection, we're going to see the matrix-valued variance of the stopped
martingale is bounded throughout the algorithm.

Firstly, we note that for a single edge sample $\XXtil_{i,e}$,
by Lemma~\ref{lem:edgesampnorm}, we have that
\[
  \norm{\XXtil_{i,e}} \leq
  \norm{\Phi\left( \YY_{\pi(i),e} - \E{\YY_{\pi(i),e}} \right)}
    \leq 1/K,
\]
using that $\norm{\AA-\BB} \leq \max( \norm{\AA}, \norm{\BB} )$, for
$\AA, \BB \succeq \matzero$, and $\norm{\E{\AA}} \leq \E{\norm{\AA}}$
by Jensen's inequality.

Thus, if $0 < \theta \leq K$, we have that
\begin{align}
  \label{eq:edgelogexp}
  \E{ \exp(\theta \XXtil_{i,e}) \mid \text{preceding samples} }
  &\preceq
      \II +
    \E{(\theta \XXtil_{i,e})^2 \mid \text{preceding samples} }
  \\ \nonumber   &\preceq
        \II +
    \frac{1}{K}\theta^2\cdot
    \E{\Phi(\YY_{\pi(i),e}) \mid \text{preceding samples} }
\end{align}
% \begin{lemma}
%   \label{lem:edgelogexp}
%   \[
%     \log \Ex{\XX_{v_i,d_i-1}}\exp(\theta\XX_{v_i,d_i}) \preceq
%     \theta^2 R \cdot
%     \Ex{\XX_{v_i,d_i-1}}\Phi(\YY_{v_i,d_i})
%   \]
% \end{lemma}

\subsection{The Overall Mean-Trace-Exponential Bound}

We will use $\Ex{(<i)} $ to denote expectation over variables preceding the
$i$th elimination step.
We are going to refrain from explicitly writing out conditioning in
our expectations, but any \emph{inner} expectation that appears inside another
\emph{outer} expectation should be taken as conditional on the outer
expectation.
We are going to use $d_i$ to denote the multi-edge degree of vertex
$\pi(i)$ in $\SS_{i-1}$. This is exactly the number of edge samples in
the $i$th elimination.
Note that there is no elimination at step $n$ (the algorithm is
already finished). As a notational convenience, let's write $\hat{n} =
n-1$.
With all that in mind, we bound the mean-trace-exponential for some
parameter
$0 <\theta \leq 0.5/\sqrt{K}$
\begin{align}
    \label{eq:fulltrexpub1}
&  \Ex{}\trace{\exp(\theta\sum_{i = 1}^{\hat{n}}\XXtil_{i} )}
  \\ \nonumber
&  = \Ex{(<\hat{n})} \Ex{\pi(\hat{n})} \Ex{\XXtil_{\hat{n},1}} \cdots \Ex{\XXtil_{\hat{n},d_{\hat{n}}-1}}  \Ex{\XXtil_{\hat{n},d_{\hat{n}}}}
   \tr\exp\left( \underbrace{\sum_{i = 1}^{\hat{n}-1}\theta\XXtil_{i}
     + \sum_{e =1}^{d_{\hat{n}}-1}\theta\XXtil_{\hat{n},e}}_{\HH}
    +\theta \XXtil_{\hat{n},d_{\hat{n}}} \right)
   \tag*{$\XXtil_{\hat{n},1} \ldots, \XXtil_{\hat{n},d_{\hat{n}}}$  are independent
    conditional on $(<\hat{n}), \pi(\hat{n})$
  }
\\  \nonumber
&  \leq \Ex{(<\hat{n})} \Ex{\pi(\hat{n})} \Ex{\XXtil_{\hat{n},1}} \cdots \Ex{\XXtil_{\hat{n},d_{\hat{n}}-1}}
   \tr\exp\left(\sum_{i = 1}^{\hat{n}-1}\theta\XXtil_{i}
     +  \sum_{e =1}^{d_{\hat{n}}-1}\theta\XXtil_{\hat{n},e}
    +  \frac{1}{K}\theta^2\cdot
    \Ex{\XXtil_{\hat{n},d_{\hat{n}}}}{\Phi(\YY_{\pi(\hat{n}), d_{\hat{n}}})}
  \right)
  \tag*{By Equation~\eqref{eq:edgelogexp} and Corollary~\ref{cor:trexpub}
  .
  }
  \\ \nonumber
&  \vdots
 \tag*{ Repeat for each multi-edge sample $\XXtil_{\hat{n},1} \ldots,
       \XXtil_{\hat{n},d_{\hat{n}}-1}$ }
\\  \nonumber
&  \leq  \Ex{(<\hat{n})} \Ex{\pi(\hat{n})}
   \tr\exp\left(\sum_{i = 1}^{\hat{n}-1}\theta\XXtil_{i}
     +  \sum_{e =1}^{d_{\hat{n}} }
     \frac{1}{K}\theta^2\cdot
    \Ex{\XXtil_{\hat{n},e}}{\Phi(\YY_{\pi(\hat{n}), e})}
     \right)
 \\  \nonumber
&  = \Ex{(<\hat{n})} \Ex{\pi(\hat{n})}
   \tr\exp\left(\sum_{i = 1}^{\hat{n}-1}\theta\XXtil_{i}
     +
     \frac{1}{K}\theta^2\Phi(\vcliq{\pi(\hat{n})}{\SS_{\hat{n}-1}})
  \right)
\end{align}
To further bound the this quantity, we now need to deal with the
random choice of $\pi(\hat{n})$.
We'll be able to use this to bound the trace-exponential in a very
strong way.
From a random matrix perspective, it's the following few
steps that give the analysis it's surprising strength.

We can treat
$\frac{1}{K}\theta^2\Phi(\vcliq{\pi(\hat{n})}{\SS_{\hat{n}-1}})$ as a random
matrix.
It is not zero-mean, but we can still bound the trace-exponential
using Corollary~\ref{cor:trexpub}.

We can also bound the expected matrix exponential in
that case, using a simple corollary of Lemma~\ref{lem:ineq_exp}.
\begin{corollary}\label{cor:ineq_exp_cher}
  $\exp(\AA) \preceq \II + (1+R)\AA $
  for $\matzero \preceq \AA$ with $\norm{\AA} \leq R \leq 1$.
\end{corollary}
\begin{proof}
  The conclusion follows after observing that for $\matzero \preceq \AA$ with $\norm{\AA} \leq R$,
  we have $\AA^2 \preceq R \AA$.
  We can see this by considering the spectral decomposition of $\AA$
  and dealing with each eigenvalue separately.
\end{proof}

Next, we need a simple structural observation about the cliques
created by elimination:
\begin{claim}
  \label{clm:cliquevsstar}
  \[
    \vcliq{\pi(i)}{\SS_i} \preceq \vstar{\pi(i)}{\SS_i} \preceq \SS_i
  \]
\end{claim}
\begin{proof}
  The first inequality is immediate from
  $
     \vcliq{\pi(i)}{\SS_i} \preceq   \vcliq{\pi(i)}{\SS_i} + \ll_i\ll_i^\trp
     =
     \vstar{\pi(i)}{\SS_i}
   $
   .
   The latter inequality $\vstar{\pi(i)}{\SS_i} \preceq \SS_i$
  follows from the star being a subgraph of the whole Laplacian $\SS_i$.
\end{proof}

Next we make use of the fact that $\XXtil_{i}$ is from the
difference sequence of the \emph{stopped} martingale.
This means we can assume
\[
  \SS_{i} \preceq 1.5 \LL,
\]
since
otherwise $\XXtil_{i} = \matzero$ and we get an even better
bound on the trace-exponential.
To make this formal, in Equation~\eqref{eq:fulltrexpub1},
we ought to do a case analysis that also includes
the case $\XXtil_{i} = \matzero$ when the martingale has
stopped, but we omit this.

Thus we can conclude by Claim~\ref{clm:cliquevsstar} that
\[
  \norm{\Phi(\vcliq{\pi(i)}{\SS_{i}})} \leq 1.5.
\]

By our assumption $0 <\theta \leq 0.5/\sqrt{K}$, we have
$\norm{\frac{1}{K}\theta^2\Phi(\vcliq{\pi(i)}{\SS_{i-1}})}
\leq 1$, so that by Corollary~\ref{cor:ineq_exp_cher},
\begin{align}
  \Ex{\pi(i)}
  \exp\left(\
     \frac{1}{K}\theta^2\Phi(\vcliq{\pi(i)}{\SS_{i-1}})
  \right)
  &\preceq
  \II
  +
  \frac{2}{K} \theta^2\Ex{\pi(i)}
    \Phi(\vcliq{\pi(i)}{\SS_{i-1}})
  \\
  &\preceq
  \II
  +
  \frac{2}{K} \theta^2\Ex{\pi(i)}
  \Phi(\vstar{\pi(i)}{\SS_{i-1}})
  \tag*{by Claim~\ref{clm:cliquevsstar}.}
\end{align}
Next we observe that, because every multi-edge appears in exactly two
stars, and $\pi(i)$ is chosen uniformly at random among the $n+1-i$
vertices that $\SS_{i-1}$ is supported on, we have
\[
  \Ex{\pi(i)}
  \vstar{\pi(i)}{\SS_{i-1}}
  =
  2\frac{1}{n+1-i}\SS_{i-1}
  .
\]
And, since we assume $\SS_{i} \preceq 1.5 \LL$, we further get
\[
  \Ex{\pi(i)}
  \exp\left(\
     \frac{1}{K}\theta^2\Phi(\vcliq{\pi(i)}{\SS_{i-1}})
  \right)
  \preceq
  \II
  +
  \frac{6\theta^2}{K(n+1-i)}\II
  .
\]
We can combine this with Equation~\eqref{eq:fulltrexpub1} and
Corollary~\ref{cor:trexpub} to get
\begin{align*}
  &  \Ex{}\trace{\exp(\theta\sum_{i = 1}^{\hat{n}}\XXtil_{i} )}
     \\*  \nonumber
  &  \leq
    \Ex{(<\hat{n})} \Ex{\pi(\hat{n})}
   \tr\exp\left(\sum_{i = 1}^{\hat{n}-1}\theta\XXtil_{i}
     +
     \frac{1}{K}\theta^2\Phi(\vcliq{\pi(\hat{n})}{\SS_{\hat{n}-1}})
    \right)
\\  \nonumber
  &  \leq
    \Ex{(<\hat{n})}
   \tr\exp\left(\sum_{i = 1}^{\hat{n}-1}\theta\XXtil_{i}
     +
    \frac{6\theta^2}{K(n+1-i)}\II
  \right)
\end{align*}
And by repeating this analysis for each term $\XXtil_{i}$, we get
\begin{align*}
 \Ex{}\trace{\exp(\theta\sum_{i = 1}^{\hat{n}}\XXtil_{i} )}
&\leq
   \tr\exp\left(\sum_{i = 1}^{\hat{n}}
    \frac{6\theta^2}{K(n+1-i)}\II
    \right)
\\
  &\leq
\tr\exp\left(
    \frac{7\theta^2\log(n)}{K}\II
    \right)
\\
  &=
    n\exp\left(
    \frac{7\theta^2\log(n)}{K}
    \right)
\end{align*}
Then, by choosing $K = 200 \log^2 n$ and $\theta = 0.5 \sqrt{K}$,
we get
\[
\exp(-0.5 \theta ) \Ex{}\trace{\exp(\theta\sum_{i =
    1}^{\hat{n}}\XXtil_{i} )}
\leq
\exp(-0.5 \theta )
 n\exp\left(
    \frac{7\theta^2\log(n)}{K}
  \right)
  \leq
  1/n^5.
\]
$\Ex{}\trace{\exp(-\theta\sum_{i =
    1}^{\hat{n}}\XXtil_{i} )}$ can be bounded by an identical
argument, so that Equation~\eqref{eq:masterprob} gives
\[
  \Pr\left[ \norm{
\sum_{i =
    1}^{\hat{n}}\XXtil_{i}
  } \geq 0.5 \right] \leq 2/n^5.
  \]
Thus we have established $\norm{
\sum_{i =
    1}^{\hat{n}}\XXtil_{i}
  } \leq 0.5 $ with high probability
  (Equation~\eqref{eq:zeromeanspectralnormgoal}), and this in turn
  implies Equation~\eqref{eq:spectralnormgoal}, and finally
  Equation~\eqref{eq:relerrgoal}:
  \[
0.5 \LL
\preceq
\matlow \matlow^{\trp}
\preceq 1.5 \LL
.
\]

Now, all that's left to note is that the running time is linear in the
multi-edge degree of the vertex being eliminated in each iteration (and
this also bounds the number of non-zero entries being created in
$\matlow$).
The total number of multi-edges left in the remaining graph stays
constant at $K m = O(m\log^2 n)$.
Thus the expected degree in the $i$th elimination is $K m/(n+i-1)$,
because the remaining number of vertices is $n+i-1$.
Hence the total running time and total number of non-zero entries
created can both be bounded as
\[
  Km\sum_i 1/(n+i-1) = O(m\log^3 n)
  .
  \]
We can further prove that the bound $O(m\log^3 n)$ on running time and
number of non-zeros in $\LL$ holds with high probability
(e.g. $1-1/n^5$).
To show this, we essentially need a scalar Chernoff bound, in except
the degrees are in fact not independent, and so we need a scalar
martingale concentration result, e.g. Azuma's Inequality.
This way, we complete the proof of Theorem~\ref{thm:apxgauss}.
\FloatBarrier

% \todo{reenable lecture}
%\bibliographystyle{alpha}
%\bibliography{refs}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "agao21_script"
%%% TeX-engine: luatex
%%% End:


\documentclass[11pt]{article}

% preamble
\usepackage[margin=1in]{geometry}
\usepackage{palatino}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathpazo}
\usepackage{setspace}
\usepackage{float}
\usepackage{enumitem}  % control the indentation value of an enumerate environment
\usepackage{hyperref}
\usepackage{siunitx}
\sisetup{
  round-mode = places,
  round-precision = 2,
}

\usepackage[backend=bibtex,style=ieee]{biblatex}
\bibliography{advanced_algorithms}

\usepackage[type={CC}, modifier={by-nc-sa}, version={3.0},]{doclicense}  % license

\usepackage{algpseudocode}  % pseudocode block
\usepackage{algorithm}  % pseudocode block

\usepackage{pgf,tikz,pgfplots}  % draw mathematics graphs
\pgfplotsset{compat=1.15}
\usepackage{mathrsfs}
\usetikzlibrary{arrows}
% \pagestyle{empty}  % suppress subsequent page numbers
\renewcommand{\figurename}{\hspace{-12mm}Example}  % override default figure caption title

\title{Exercises for Advanced Algorithms}
\author{Solution by: \href{mailto:neo-mashiro@hotmail.com}{Wentao Lu}}
\date{}

% document
\begin{document}
  \newcommand{\solution}{\noindent \textbf{\textsc{Solution:}} \dotfill \vspace{2mm}}  % solution title
  \definecolor{ffqqqq}{rgb}{1,0,0}  % node color in hypercube
  \definecolor{qqqqff}{rgb}{0,0,1}  % node color in hypercube

  \maketitle
  \pagenumbering{arabic}

  % \rule[raise]{width}{height}
  \noindent \hspace{\fill} Models of computation \rule[0mm]{20mm}{2mm} (20\%)\\
  \hspace*{\fill} NP-complete \rule[0mm]{55mm}{2mm} (55\%)\\
  \hspace*{\fill} Approximation algorithms \rule[0mm]{40mm}{2mm} (40\%)\\
  \hspace*{\fill} Linear programming \rule[0mm]{30mm}{2mm} (30\%)\\
  \hspace*{\fill} Computational geometry \rule[0mm]{30mm}{2mm} (30\%)\\
  \hspace*{\fill} Parallel models and algorithms \rule[0mm]{40mm}{2mm} (40\%)

  \section{NP-complete}
    Provide a proof of NP-completeness for the two-machine scheduling problem by formulating a reduction from the knapsack problem and state its corresponding optimization problem.
    \begin{itemize}[leftmargin=*]
      \item \textsc{Two-machine scheduling:} Given the execution time $a_1, a_2 \dots, a_n$ of $n$ independent tasks and a deadline $D$, is it possible to schedule the $n$ tasks on two identical machines in a non-preemptive fashion (meaning that once a task starts executing it must run to completion) such that all the task complete by the deadline $D$.
      \item \textsc{Knapsack:} Given $n$ objects with volumes $v_i$ and prices $s_i$, \(1 \le i \le n\), a knapsack of volume $V$, and a constant $S > 0$, is there a subset $N' \subseteq \{1, \dots, n\}$ such that $\Sigma_{i \in N'} v_i < V$ (the objects in $N'$ can be placed in the knapsack) and $\Sigma_{i \in N'} s_i \geq S$ (their price exceeds the given limit $S$)?
    \end{itemize}

    \solution\\
    The complete proof consists of two steps. Given any candidate solution (purported certificate) of this problem which divides the $n$ independent tasks into two bins, we can easily compute the total execution time of tasks in each bin, and then check if both of them $\leq D$. Since the solution can be verified in linear time, by definition\autocite[1049]{Cormen09} it is an NP problem.\\

    After Stephen Cook proved 3-SAT to be the first known NP-complete problem in 1971\autocite{Cook71}, the next year Richard Karp further proved that another 21 common computational problems are all NP-complete\autocite{Karp72}, one of which is the knapsack problem. To prove NP-hardness, we will show that the knapsack problem is reducible to the two-machine scheduling problem.\\

    For every instance of the knapsack problem, construct $n + 2$ independent tasks with execution time $\{a_1, \dots, a_{n+2}\}$ and a deadline $D = \Sigma_{i=1}^{n} a_i + 2$. Let
    \begin{align*}
      a_i     &= v_i + s_i \,,\, 1 \le i \le n\\
      a_{n+1} &= \Sigma_{i \in P} v_i + \Sigma_{i \in P} s_i + 2\\
      a_{n+2} &= \Sigma_{i=1}^{n} a_i - \Sigma_{i \in P} v_i - \Sigma_{i \in P} s_i + 2
    \end{align*}

    where $P$ denotes the set of items placed in the knapsack. It is evident that this mapping takes polynomial time in the size of input. Now let's prove that this translation is indeed a reduction.\\

    $\Rightarrow$ First, suppose that the knapsack problem has a true instance, which means there is indeed a subset of items $P \subseteq \{1, \dots, n\}$ such that
    \begin{align*}
      \Sigma_{i \in P} v_i &< V\\
      \Sigma_{i \in P} s_i &\geq S
    \end{align*}

    Let
    \begin{align*}
      M_1 &= \{a_i \mid i \in P\} \cup \{a_{n+2}\}\\
      M_2 &= \{a_i \mid i \notin P\} \cup \{a_{n+1}\}
    \end{align*}

    Then we have
    \begin{align}
      \text{the sum of } M_1 &= \Sigma_{i \in P} v_i + \Sigma_{i \in P} s_i + \Sigma_{i=1}^{n} a_i - \Sigma_{i \in P} v_i - \Sigma_{i \in P} s_i + 2 = \Sigma_{i=1}^{n} a_i + 2\\
      \text{the sum of } M_2 &= \Sigma_{i=1}^{n} a_i - \Sigma_{i \in P} v_i - \Sigma_{i \in P} s_i + \Sigma_{i \in P} v_i + \Sigma_{i \in P} s_i + 2 = \Sigma_{i=1}^{n} a_i + 2
    \end{align}

    From (1) and (2), it's clear that we have $M_1=M_2=D$, also notice that $M_1 \cup M_2 $ is the set of all tasks, therefore we have found a feasible schedule for two identical machines.\\

    $\Leftarrow$ Conversely, suppose that the translated two-machine scheduling problem has a true instance, so the set of $n + 2$ independent tasks can be divided into two subsets, which can be scheduled on two machines $M_1$ and $M_2$, respectively, by the deadline $D = \Sigma_{i=1}^{n} a_i + 2$.\\

    Since execution time must be nonnegative, and notice that $a_{n+1} + a_{n+2} = \Sigma_{i=1}^{n} a_i + 4>D$, this means $a_{n+1}$ and $a_{n+2}$ must not be executed on the same machine. Without loss of generality, assume that $a_{n+2}$ is scheduled on $M_2$, then the execution time of the rest of the tasks scheduled on $M_2$ must sum up to
    \begin{align*}
      D - a_{n+2} = \Sigma_{i=1}^{n} a_i + 2 - a_{n+2} = \Sigma_{i \in P} v_i + \Sigma_{i \in P} s_i
    \end{align*}

    As per our reduction assumption, it imples that this subset $P \subseteq \{1, \dots, n\}$ has the property
    \begin{align*}
      \Sigma_{i \in P} v_i &< V\\
      \Sigma_{i \in P} s_i &\geq S
    \end{align*}

    Thus, it makes a true instance of the knapsack problem as well.\\

    Now we have \textsc{Knapsack} $\leq_p$ \textsc{Two-machine scheduling}, which implies that the two-machine scheduling problem is at least as hard as the knapsack. This completes our proof that two-machine scheduling is also NP-complete.\\

    The optimization variant of this is the so-called \textit{makespan} problem, in our case with two machines denoted by $P2||C_{max}$\autocite{Schuurman00}, which can be stated as: Given the execution time $a_1, a_2 \dots, a_n$ of $n$ independent tasks, find a way to schedule the $n$ tasks on two identical parallel machines in a non-preemptive fashion, so as to minimize the maximum task completion time. In other words, we would like to do our best to equally assign the $n$ tasks to both machines, such that the total execution time on the machine that finishes last is minimized. This variant is NP-hard.\autocite{Karp72}

  \section{Approximate Maximum Clique}
    Let $G = (V,E)$ be an undirected graph. For integer $k \geq 1$ define $G^{(k)} = (V^{(k)}, E^{(k)})$ such that $V^{(k)} = \{(v_1, v_2, \dots, v_k) | v_i \in V, 1 \leq i \leq k\}$ and $((v_1, v_2, \dots, v_k), (w_1 ,w_2 ,\dots ,w_k)) \in E^{(k)}$ iff either $(v_i, w_i) \in E$ or $v_i = w_i$ for all $1 \leq i \leq k$.
    \begin{enumerate}[leftmargin=*]
      \item Prove $|C^{(k)}| = |C|^k$, where $C^{(k)}$ and $C$ are the maximum cliques of $G^{(k)}$ and $G$, respectively.
      \item Argue that the existence of an approximation algorithm for finding the maximum clique with a constant approximation ratio implies the existence of a polynomial-time approximation scheme for finding the maximum clique.
    \end{enumerate}

    \solution
    \begin{enumerate}[leftmargin=*, topsep=0pt]
      \item Let $V' = \{(v_1, v_2, \dots, v_k) | v_i \in C, 1 \leq i \leq k\} \subseteq V^{(k)}$, where $C$ is the maximum clique of $G$, so every pair of vertices $(v_i, v_j)$ is adjacent, clearly $V'$ is a clique of $G^{(k)}$. Since each element $v_i$ in the $k$-tuple $(v_1, v_2, \dots, v_k)$ comes from $C$, the size of $V'$ is $|C|^k$. Therefore, the size of the maximum clique of $G^{(k)}$ is at least $|C|^k$. We will prove by induction that this is indeed the size of the maximum clique.

      In the base case, when $k = 1$, $G^{(1)} = G$, we have $|C^{(1)}| = |C|^1$. Now suppose that $|C^{(k)}| = |C|^k$ is true for $1 \leq i \leq k$. Let $C^{(k+1)} = \{(p_i, v_1, v_2, \dots, v_k)\}$ where $p_i$ is some vertex in $G$.

      Since the adjacency condition of $(v_1, v_2, \dots, v_k)$ and $(w_1, w_2, \dots, w_k)$ only requires the adjacency of $v_i$ and $w_i$, the choice of vertices at index $i$ is independent of vertices at any other index. Hence, if we remove the prefix vertex $p_i$ from every tuple in $C^{(k+1)}$, the remaining $k$-tuples must form a maximum clique of $G^{(k)}$, otherwise we can always do better. By induction hypothesis, it has size $|C|^k$.

      In order to make $C^{(k+1)}$ a clique of $G^{(k+1)}$, we must choose a set of prefixes $\{p_i\} \subseteq V$ such that every pair of vertices $(p_i, p_j)$ is connected by an edge. Moreover, to make it a maximum clique, we also want to maximize the number of prefixes we choose. In other words, we are trying to find a maximum set of mutually adjacent vertices from $G$ and prepend them to every $k$-tuple in $C^{(k)}$, so we should choose the set of prefix vertices $\{p_i\} = C$. Therefore, the size of $C^{(k+1)}$ is $|C|^k \times |C| = |C|^{k+1}$.

      \item Given a $c$-approximation algorithm for finding the maximum clique, we can apply it on $G^{(k)}$ and obtain a clique of size $n$, while the true maximum clique of $G^{(k)}$ has size $|C|^k$.

      By definition of an approximation ratio, $|C|^k/n \leq c$, so we have $|C|/n^{1/k} \leq c^{1/k}$. For any $\varepsilon > 0$, we can choose a positive integer $k > \log_{1+\varepsilon}{c}$, then $|C|/n \leq |C|/n^{1/k} \leq c^{1/k} < 1 + \varepsilon$. Since it is polynomial in the size of input, so we have a polynomial-time approximation scheme for finding the maximum clique.
    \end{enumerate}

  \section{Linear Inequality Feasibility}
    Given a set of $m$ linear inequalities over $n$ variables, the linear inequality feasibility problem asks whether there exists a set of the instances that satisfies all the linear inequalities simultaneously.
    \begin{enumerate}[leftmargin=*]
      \item Prove that we can use an algorithm for linear programming to solve linear inequality feasibility problems. The number of variables and constraints used in the linear programming problem must be polynomial in $n$ and $m$.
      \item Prove that we can use an algorithm for the linear inequality feasibility problem to solve linear programming problems. The number of variables and linear inequalities must be polynomial in the number of variables and constraints of the linear program.
    \end{enumerate}

    \solution
    \begin{enumerate}[leftmargin=*, topsep=0pt]
      \item Given a set of linear inequalities, we can simply formulate the equality feasibility problem as a linear program. First, we can set the objective function to be a constant such as 0 because there's nothing to maximize or minimize in this case. Next, we need to transform those linear inequalities into their slack form, and then apply the simplex algorithm. If the linear program returns a feasible solution, it implies that all the linear inequalities can be satisfied at the same time. Otherwise, those linear inequalities are not feasible. The variables and constraints in the linear program are exactly the same as in the equality feasibility problem so they are polynomial in $n$ and $m$.

      \item Given a linear program in standard form, we want to maximize the objective function $\Sigma_{j=1}^{n} c_jx_j$ with $n$ variables, subject to $m$ constraints $\Sigma_{j=1}^{n} a_{ij}x_j \leq b_i, 1 \leq i \leq m$. Suppose we have an algorithm for the linear inequality feasibility problem, then we can apply it on the $m$ constraints of the linear program. If the algorithm fails to find a solution for the linear inequality feasibility problem, it means that those inequalities can not be satisfied, so the linear program should return infeasible. If the algorithm does find an assignment of the $n$ variables that satisfies all the linear inequalities, then the linear program is also feasible.

      In order to solve the linear program and maximize the objective function, in each step we first calculate the value of our objective function using the assignment values returned by that algorithm, then we update the linear program by introducing a new constraint. For example, if the algorithm finds an assignment $\overline{x}$ which gives an objective function value of $\Sigma_{j=1}^{n} c_jx_j = k$, we will add another constraint $\Sigma_{j=1}^{n} c_jx_j > k$ to the original linear program. As long as this updated linear program is feasible, it must be equivalent to the original linear program because both the objective function and all other constraints are not affected. If we iteratively run this step, we are getting closer to the optimal solution. We will repeat this step until the algorithm detects that the updated constraints are not feasible anymore, in which case we stop and return the last calculated objective function value as the optimal solution.

      Whenever the algorithm returns a feasible assignment so that all constraints are satisfied, it represents a point that's somewhere in the simplex. The basic idea is to update that point until it reaches a local maximum, which is also a global maximum because the simplex is convex. Since we are only adding 1 more constraint in each step, we can optimistically assume that the number of linear inequalities is polynomial in the number of constraints. If that's not true, we can modify the constraint to be $\Sigma_{j=1}^{n} c_jx_j > k + \alpha$, where $\alpha$ is a reasonable step size to choose in different trials. Unless the original linear program has an unbounded solution, we will eventually converge to an optimal solution.
    \end{enumerate}

  \section{Ghostbusters and Ghosts}
    A group of $n$ ghostbusters is battling $n$ ghosts. Each buster carries a proton pack, which shoots a stream at a ghost, eradicating it. A stream goes in a straight line and terminates when it hits the ghost. The ghostbusters decide upon the following strategy: They will pair off with the ghosts, forming $n$ ghostbuster-ghost pairs, and then simultaneously each ghostbuster will shoot a stream at his chosen ghost. As we all know, it is \textit{very} dangerous to let streams cross, and so the busters must choose pairings for which no streams will cross. Assume that the position of each buster and each ghost is a fixed point in the plane and that no three positions are colinear.
    \begin{enumerate}[leftmargin=*]
      \item Argue that there exists a line passing through one Ghostbuster and one ghost such that the number of Ghostbusters on one side of the line equals the number of ghosts on the same side. Describe how to find such a line in $O(n \log n)$ time.
      \item Give an $O(n^2 \log n)$-time algorithm to pair Ghostbusters with ghosts such that no streams cross.
    \end{enumerate}

    \solution
    \begin{figure}[H]
      \centering
      \begin{tikzpicture}
        \draw[black, dotted, thick]
        (-0.5,0) -- (5.5,0)
        (0,-0.5) -- (0,3.5);
        \draw[->](0,0) -- (5,1);
        \draw[->](0,0) -- (4,2);
        \draw[->](0,0) -- (5,2);
        \draw[->](0,0) -- (3,3);
        \draw[->](0,0) -- (3,2);
        \draw[->](0,0) -- (1,2);
        \draw[color=black] (-0.25,-0.25) node {$P_0$};
        \draw (5,1) node;
        \draw (4,2) node;
        \draw (5,2) node;
        \draw (3,3) node;
        \draw (3,2) node;
        \draw (1,2) node;
        \draw [fill=black] (5,1) circle (1.0pt);
        \draw [fill=black] (4,2) circle (1.0pt);
        \draw [fill=black] (5,2) circle (1.0pt);
        \draw [fill=black] (3,3) circle (1.0pt);
        \draw [fill=black] (3,2) circle (1.0pt);
        \draw [fill=black] (1,2) circle (1.0pt);
        \draw [fill=ffqqqq] (0,0) circle (1.5pt);
      \end{tikzpicture}
      \caption{Graham Scan}
    \end{figure}

    \begin{enumerate}[leftmargin=*, topsep=0pt]
      \item First we pick the leftmost bottom point $P_0$ as the origin. For the remaining points, we can easily sort them by polar angle in counterclockwise order. Concretely, the cross product $(P_1 - P_0) \times (P_2 - P_0)$ gives us a comparison relationship between any two points $P_1$ and $P_2$, which can be applied to any comparison-based sorting algorithm such as merge sort.

      Given the sorted points by polar angle with respect to $P_0$, we then visit them one by one in this order, and count the number of ghostbusters and ghosts along the way. We would stop at a point $P_k$ when the following two conditions are met:

      \begin{itemize}[leftmargin=*, topsep=0pt]
          \item Either $P_0$ is a ghostbuster and $P_k$ is a ghost, or $P_0$ is a ghost and $P_k$ is a ghostbuster.
          \item The number of ghostbusters and ghosts visited so far are equal (including $P_0$ and $P_k$).
      \end{itemize}

      It follows that the line passing through $P_0$ and $P_k$ is the solution. Since the number of ghostbusters and ghosts are equal, we are guaranteed to find such a line. It takes $O(n\log n)$ to sort and $O(n)$ to scan, so the running time is $O(n\log n)$ in total.

      \item Now we have paired one ghostbuster with one ghost in $O(n\log n)$ time, applying this algorithm recursively on both sides of this line will pair all the ghostbusters with ghosts.

      That is, for the subset of points on each side, we choose the origin with the minimum coordinate, sort the remaining points on this side and then scan through them until we find a new line. During each recursion step, work done on one side of the line is independent from the other, so that no lines will ever cross. We need to find $n$ lines, so this divide-and-conquer approach runs in $O(n^2\log n)$ time.
    \end{enumerate}

  \section{Divide and Conquer Prefix Computations}
    We use again divide and conquer method to perform prefix computation (with $\circ$ as binary operator) on a sequence $S$ of size $n$. This time the running time will be $O(\log \log n)$ time using $O(n/\log \log n)$ processors. This algorithm would be optimal, faster than the one mentioned in the book, but also using more processors. To obtain such an algorithm we proceed as follows:

    \begin{enumerate}[leftmargin=*]
      \item[a] Divide the sequence $S$ into $n^{1/2}$ subsequences of size $n^{1/2}$ each. Perform prefix computation on each sequence recursively using $n^{1/2}$ processors. Let the result of the computation on the $i$-th sequence be $s(i,1), s(i,2), \dots, s(i,n^{1/2})$.
      \item[b] Perform a prefix computation on the sequence $s(1,n^{1/2}), s(2,n^{1/2}), \dots, s(n^{1/2}-1,n^{1/2})$ using $n$ processors. Let the result of this computation be $s^\prime(1,n^{1/2}), s^\prime(2,n^{1/2}), \dots, s^\prime(n^{1/2}-1,n^{1/2})$.
      \item[c] For all $1 \leq i < n^{1/2}$ use the binary operator $\circ$ to combine $s^\prime(i,n^{1/2})$ with all the elements of $s(i+1,1),s(i+1,2),\dots,s(i+1,n^{1/2})$ using $n^{1/2}$ processors per subsequence.
    \end{enumerate}

    \begin{enumerate}[leftmargin=*]
      \item Show that the algorithm described above uses $n$ processors to run in $O(\log \log n)$ time.
      \item \label{l} Show how the number of processors can be reduced to $O(n/\log \log n)$ while maintaining the $O(\log \log n)$ running time, thus achieving the optimal cost $O(n)$.
      \item Discuss the implications of the result obtained in Question \ref{l} on related computations such as array packing.
    \end{enumerate}

    \solution
    \begin{enumerate}[leftmargin=*, topsep=0pt]
      \item In step 1 we recursively perform prefix computation on each subsequence of size $n^{1/2}$ using $n^{1/2}$ processors. Suppose our recursive algorithm on the entire sequence $S$ has running time $t(n)$, then the recursion call on each subsequence takes $t(n^{1/2})$. Since this is done in parallel, this step takes $t(n^{1/2})$.

      In step 2, the sequence $s(1,n^{1/2}), s(2,n^{1/2}), \dots, s(n^{1/2}-1,n^{1/2})$ contains every last element of a subsequence except the first one, so there are only $n^{1/2}-1$ elements, but we have $n$ processors available. We can take advantage of all of them by letting $n^{1/2}$ processors work on each element simultaneously, that is, for some element $s(i,n^{1/2})$, we use $n^{1/2}$ processors in parallel to perform binary operations on $s(i,n^{1/2})$ with all other elements $s(j,n^{1/2})$, where $j \neq i, 1 \leq j \leq n^{1/2}-1$. Since we have $n$ processors so that all elements can update in parallel, this step takes $O(1)$. However, this does not work for all binary operations, in fact, the binary operation must be both associative and commutative, such as addition, multiplication and union, but not concatenation because concurrent writes would mess up the order. Besides, this speed up is possible only with the Combining CRCW PRAM model such that concurrent reads and writes can be effectively reconciled.

      In step 3, for each subsequence we use $n^{1/2}$ processors to combine its elements with the last element in the preceding subsequence. Each element can be mapped to a processor, so this single operation is done in parallel, this step takes $O(1)$ time.

      Now combining these results we have $t(n) = t(n^{1/2}) + O(1)$. To derive the running time from this equation, we substitute $n$ with $e^m$, so $t(e^m) = t(e^{m/2}) + O(1)$. Let $s(m) = t(e^m)$, then $s(m) = s(m/2) + O(1)$. According to the Master's Theorem, this implies that
      \begin{equation*}
        s(m) = O(\log m) \hspace{3mm} \Rightarrow \hspace{3mm} t(n) = t(e^m) = s(m) = O(\log m) = O(\log \log n)
      \end{equation*}

      \item If we want to reduce the number of processors to $O(n/\log \log n)$, the previous partition will not work, we need a new partition. Besides, we want to follow the previous three steps, but need to make some modifications.

      In concrete, we will divide $S$ into $n/\log \log n$ subsequences or groups, each group has size $\log \log n$. Since the number of groups is equal to the number of processors, each group can only have one processor, so in step 1, prefix computation must be executed sequentially within each group, while all groups do this in parallel. Hence, this step takes $O(\log \log n)$.

      In step 2, the sequence will be $s(1,\log \log n), s(2,\log \log n), \dots, s(n/\log \log n-1,\log \log n)$ because we only have $n/\log \log n$ groups, processors and last elements. To perform prefix computation on this sequence, we will adopt the algorithm in Question 1. As we have shown in Question 1, that algorithm runs in $O(\log \log n)$ time on $n$ elements using $n$ processors, so here with $n/\log \log n$ elements and processors, it will take $O(\log \log(n/\log \log n))$ time to run. By removing the trivial terms, we have
      \begin{equation*}
        O(\log \log(n/\log \log n)) = O(\log(\log n-\log(\log \log n))) = O(\log \log n)
      \end{equation*}

      In step 3, for the same reason as in step 1, the combining operation has to be executed sequentially for each group due to the lack of processors. As a result, each group takes $O(\log \log n)$ to run, while different groups run in parallel with each other. This step takes $O(\log \log n)$.

      In summation, the total running time is $3 \times O(\log \log n) = O(\log \log n)$ which is identical to the previous one, but the cost is optimal as we are now using less than $n$ processors.

      \item The result obtained in Question 2 has several implications.

      We have seen that it is possible to devise an optimal parallel algorithm that takes a suboptimal algorithm as an intermediate step. The divide and conquer approach allows us to divide a problem into many subproblems which can be solved in parallel, and the way we partition the data can sometimes make a difference. In addition, different PRAM models may lead to different optimal algorithms to solve the same problem, we are free to choose among them depending on the scenario. This gives us a lot more flexibility from a practical perspective since the number of processors and hardware is often fixed in real world. For example, compared to the other optimal algorithm discussed in the class notes, this one is also optimal but much faster, so it can be very useful if we are strict on the running time and have sufficient processors that support CRCW PRAM models. It has also proved the possibility of applications being highly scalable based on commutative binary operations such as prefix sum. For example, the array packing problem uses addition as the underlying binary operation, so that as long as the hardware requirement is satisfied, it can easily scale up to boost performance, especially when the input array size is huge.
    \end{enumerate}

  \section{Descending in a Hypercube}
    Let $N = 2^g$ data be stored in the processors of a hypercube, one value per processor (so that the value $x_i$ is stored in $P_i$, $0 \leq i < N$). A technique with wide applicability to hypercube algorithms is called \textsc{Descend} and consists of $g$ iterations. During the $j$-th iteration a basic binary operation \textsc{Operation}$(P_i, P_l)$ is performed on data in cores whose indices differ by $2^j$. Here is the pseudocode for this process, with $i_{g-1}i_{g-2}\dots i_{1}i_{0}$ the binary representation of index $i$:

    \newpage

    \begin{algorithm}
      % \renewcommand\thealgorithm{}  % suppress the caption number
      \caption{\textsc{Descend}}
      \label{ad}
      \begin{algorithmic}[1]
        \For{$j \leftarrow g - 1$ \textbf{to} $0$} \Comment{iterate over each bit}
          \For{$i \leftarrow 0$ \textbf{to} $2^{g}-1$} \Comment{iterate over each core}
            \If{$i_j = 0$}  \textsc{Operation}$(P_i, P_{i+2^j})$
              \hspace{2mm}\raisebox{.5\baselineskip}[0pt][0pt]{$\left.\rule{0pt}{1.0\baselineskip}\right\}\ \mbox{in parallel}$} \Comment{operate on adjacent cores}
      \end{algorithmic}
    \end{algorithm}

    \noindent If \textsc{Operation} requires constant time then \textsc{Descend} runs in $O(\log N)$ time. The dual to \textsc{Descend} is \textsc{Ascend}, in which $j$ in the outer loop varies from $0$ to $g-1$.

    \begin{enumerate}[leftmargin=*]
      \item Show how \textsc{Descend} can be used to obtain an algorithm that computes the sum of all the values $x_i$ and places the result in $P_0$.
      \item Use either \textsc{Ascend} or \textsc{Descend} to broadcast the datum held by some processor $P_k$ to all the other processors.
      \item Suggest other problems that can be solved using these paradigms.
    \end{enumerate}

    \solution\\
    If two integers differ by $2^k$, then in their binary representation, all bits must be the same except the $k$-th bit where they differ by 1. That is, in the $j$-th iteration of the \textsc{Descend} algorithm, the binary operation is applied on any pair of processors whose indices differ only at the $j$-th bit. A bit can be either 0 or 1, only 2 possible values. Therefore, in each iteration, all processors are divided into halves by some bit, one half will exchange data with the other half via the binary operation. After $g$ iterations, we would have divided the set of processors bit by bit in every possible way, so that every processor has talked to all the other processors exactly once, either directly or indirectly.

    \begin{enumerate}[leftmargin=*]
      \item[1.] Since the addition operation is commutative, it doesn't matter if we loop from the highest bit or the lowest bit, so we don't mind whether the \textsc{Descend} or \textsc{Ascend} algorithm is being used. In order to place the final result in $P_0$ whose index is a bunch of zeros in binary format, during the $j$-th step we must store the sum of $P_i$ and $P_{i+2^j}$ into $P_i$ because $i_j=0$. Then after $g$ iterations, the value in each processor will be added exactly once to $P_0$. So we define
      \begin{enumerate}
          \item[] \textsc{Operation}($P_i$, $P_{i+2^j}$) to be \fbox{$P_i$ = $P_i$ + $P_{i+2^j}$}
      \end{enumerate}
      As an example, we can graphically illustrate how it works in a 3 dimensional hypercube. In the figure below, processors $P_0,P_1,\dots,P_7$ correspond to the 8 corner vertices of the cube, which are represented in their binary format as $(0,0,0),(0,0,1),\dots,(1,1,1)$. For simplicity, suppose that initially each processor has value $x_i=1$, so we expect $P_0$ to be 8 in the end.

      In the first iteration the algorithm looks at the highest bit, which is 0 for vertices on the left side and 1 for vertices on the right side. So as to perform the binary operation, values on the right side are added to those on the left. Vertices whose values have been updated are colored blue. Likewise, the second round looks at the middle bit, so values on the upper side are added to those on the lower side, and so forth. When the algorithm finishes, $P_0(0,0,0)$ has been involved in all iterations and colored blue for three times, hence it has the sum of all values and we color it red. In the end, we managed to compute the sum of 8 numbers in merely 3 steps.
    \end{enumerate}

    % tex code generated by GeoGebra
    \begin{figure}[H]
      \centering
      \begin{tikzpicture}[line cap=round,line join=round,>=triangle 45,x=1cm,y=1cm]
      \clip(-1,-8.5) rectangle (16,6.6);
      \draw [line width=0.4pt] (3.5,-7.5)-- (5.5,-5.5);
      \draw [line width=0.4pt] (7.5,-7.5)-- (9.5,-5.5);
      \draw [line width=0.4pt] (3.5,-3.5)-- (5.5,-1.5);
      \draw [line width=0.4pt] (7.5,-3.5)-- (9.5,-1.5);
      \draw [line width=0.4pt] (0,0)-- (2,2);
      \draw [line width=0.4pt] (4,0)-- (6,2);
      \draw [line width=0.4pt] (0,4)-- (2,6);
      \draw [line width=0.4pt] (4,4)-- (6,6);
      \draw [line width=0.4pt] (8,0)-- (10,2);
      \draw [line width=0.4pt] (12,0)-- (14,2);
      \draw [line width=0.4pt] (8,4)-- (10,6);
      \draw [line width=0.4pt] (12,4)-- (14,6);
      \draw (0.7150220913107512,-0.25439503619441556) node[anchor=north west] {(a) right to left};
      \draw (8.676730486008838,-0.216132368148914) node[anchor=north west] {(b) top to bottom};
      \draw (4.332842415316643,-7.715615305067217) node[anchor=north west] {(c) back to front};
      \draw [line width=0.4pt] (0,4)-- (4,4);
      \draw [line width=0.4pt] (2,6)-- (6,6);
      \draw [line width=0.4pt] (0,0)-- (4,0);
      \draw [line width=0.4pt] (2,2)-- (6,2);
      \draw [line width=0.4pt] (0,4)-- (0,0);
      \draw [line width=0.4pt] (2,2)-- (2,6);
      \draw [line width=0.4pt] (6,2)-- (6,6);
      \draw [line width=0.4pt] (4,4)-- (4,0);
      \draw [line width=0.4pt] (14,6)-- (14,2);
      \draw [line width=0.4pt] (8,4)-- (12,4);
      \draw [line width=0.4pt] (10,6)-- (14,6);
      \draw [line width=0.4pt] (8,0)-- (12,0);
      \draw [line width=0.4pt] (10,2)-- (14,2);
      \draw [line width=0.4pt] (8,4)-- (8,0);
      \draw [line width=0.4pt] (12,0)-- (12,4);
      \draw [line width=0.4pt] (10,6)-- (10,2);
      \draw [line width=0.4pt] (5.5,-1.5)-- (9.5,-1.5);
      \draw [line width=0.4pt] (3.5,-3.5)-- (7.5,-3.5);
      \draw [line width=0.4pt] (9.5,-5.5)-- (9.5,-1.5);
      \draw [line width=0.4pt] (7.5,-7.5)-- (3.5,-7.5);
      \draw [line width=0.4pt] (5.5,-5.5)-- (5.5,-1.5);
      \draw [line width=0.4pt] (3.5,-3.5)-- (3.5,-7.5);
      \draw [line width=0.4pt] (9.5,-5.5)-- (5.5,-5.5);
      \draw [line width=0.4pt] (7.5,-3.5)-- (7.5,-7.5);
      \draw [->,line width=0.4pt,dash pattern=on 1pt off 1pt] (3.5,2.9) -- (0.5,2.9);
      \draw [->,line width=0.4pt,dash pattern=on 1pt off 1pt] (9.622754360465116,3.5725472383720938) -- (9.622754360465116,0.5725472383720931);
      \draw [->,line width=0.4pt,dash pattern=on 1pt off 1pt] (7,-3) -- (5,-5);
      \begin{scriptsize}
      \draw [fill=ffqqqq] (3.5,-7.5) circle (2.5pt);
      \draw[color=ffqqqq] (4.59499263622975,-7.193846949327817) node {(0,0,0) = 8};
      \draw [fill=qqqqff] (7.5,-7.5) circle (2.5pt);
      \draw[color=qqqqff] (8.100883652430046+0.5,-7.093846949327817-0.1) node {(1,0,0) = 4};
      \draw [fill=qqqqff] (7.5,-3.5) circle (2.5pt);
      \draw[color=qqqqff] (8.100883652430046+0.5,-3.0953981385729055-0.1) node {(1,1,0) = 2};
      \draw [fill=qqqqff] (3.5,-3.5) circle (2.5pt);
      \draw[color=qqqqff] (4.09499263622975+0.5,-3.0953981385729055-0.1) node {(0,1,0) = 4};
      \draw [fill=black] (5.5,-5.5) circle (2.5pt);
      \draw[color=black] (6.0979381443298974+0.2,-5.085056876938985-0.1) node {(0,0,1) = 4};
      \draw [fill=black] (9.5,-5.5) circle (2.5pt);
      \draw[color=black] (10.103829160530191+0.2,-5.085056876938985-0.1) node {(1,0,1) = 2};
      \draw [fill=black] (9.5,-1.5) circle (2.5pt);
      \draw[color=black] (10.103829160530191+0.2,-1.0866080661840742-0.1) node {(1,1,1) = 1};
      \draw [fill=black] (5.5,-1.5) circle (2.5pt);
      \draw[color=black] (6.0979381443298974+0.2,-1.0866080661840742-0.1) node {(0,1,1) = 2};
      \draw [fill=qqqqff] (0,0) circle (2.5pt);
      \draw[color=qqqqff] (1.1023564064801179,0.30563598759048614) node {(0,0,0) = 2};
      \draw [fill=black] (4,0) circle (2.5pt);
      \draw[color=black] (4.595729013254787+0.5,0.40563598759048614-0.1) node {(1,0,0) = 1};
      \draw [fill=black] (4,4) circle (2.5pt);
      \draw[color=black] (4.595729013254787+0.5,4.404084798345398-0.1) node {(1,1,0) = 1};
      \draw [fill=qqqqff] (0,4) circle (2.5pt);
      \draw[color=qqqqff] (0.6023564064801179+0.5,4.404084798345398-0.1) node {(0,1,0) = 2};
      \draw [fill=qqqqff] (2,2) circle (2.5pt);
      \draw[color=qqqqff] (2.6053019145802656+0.2,2.4144260599793173-0.1) node {(0,0,1) = 2};
      \draw [fill=black] (6,2) circle (2.5pt);
      \draw[color=black] (6.598674521354934+0.2,2.4144260599793173-0.1) node {(1,0,1) = 1};
      \draw [fill=black] (6,6) circle (2.5pt);
      \draw[color=black] (6.598674521354934+0.2,6.412874870734228-0.1) node {(1,1,1) = 1};
      \draw [fill=qqqqff] (2,6) circle (2.5pt);
      \draw[color=qqqqff] (2.6053019145802656+0.2,6.412874870734228-0.1) node {(0,1,1) = 2};
      \draw [fill=qqqqff] (8,0) circle (2.5pt);
      \draw[color=qqqqff] (9.10162002945508,0.30563598759048614) node {(0,0,0) = 4};
      \draw [fill=qqqqff] (12,0) circle (2.5pt);
      \draw[color=qqqqff] (12.59499263622975+0.5,0.40563598759048614-0.1) node {(1,0,0) = 2};
      \draw [fill=black] (12,4) circle (2.5pt);
      \draw[color=black] (12.59499263622975+0.5,4.404084798345398-0.1) node {(1,1,0) = 1};
      \draw [fill=black] (8,4) circle (2.5pt);
      \draw[color=black] (8.60162002945508+0.5,4.404084798345398-0.1) node {(0,1,0) = 2};
      \draw [fill=qqqqff] (10,2) circle (2.5pt);
      \draw[color=qqqqff] (10.604565537555228+0.2,2.4144260599793173-0.1) node {(0,0,1) = 4};
      \draw [fill=qqqqff] (14,2) circle (2.5pt);
      \draw[color=qqqqff] (14.597938144329897+0.2,2.4144260599793173-0.1) node {(1,0,1) = 2};
      \draw [fill=black] (14,6) circle (2.5pt);
      \draw[color=black] (14.597938144329897+0.2,6.412874870734228-0.1) node {(1,1,1) = 1};
      \draw [fill=black] (10,6) circle (2.5pt);
      \draw[color=black] (10.604565537555228+0.2,6.412874870734228-0.1) node {(0,1,1) = 2};
      \end{scriptsize}
      \end{tikzpicture}
      \vspace{-9mm}
      \caption{Parallel sum in a 3-hypercube}
    \end{figure}

    \begin{enumerate}[leftmargin=*]
      \item[2.] No matter which bit we start from, the \textsc{Descend} or \textsc{Ascend} algorithm always divides the processors by a different dimension in each round, then the binary operation is performed on pairs of processors, one from each group. Suppose that we use the \textsc{Ascend} algorithm that loops from the $0$-th bit, further assume that $P_k$ has the datum set $\{x\}$ while other processors have no data ($\emptyset$) initially. In order to broadcast, we can simply define
      \begin{enumerate}
          \item[] \textsc{Operation}($P_i$, $P_{i+2^j}$) to be \fbox{$P_i \bigcup P_{i+2^j}$}
      \end{enumerate}

      In the first round, $P_k$ sends $\{x\}$ to its counterpart $\bar{P_{k}}$ in the other group via the binary operation, this implies that $P_k$ and $\bar{P_{k}}$ can only differ by 1 at the $0$-th bit. On the other hand, the rest of the processors are empty, so the binary operation has no effect on them. When this round finishes, now 2 processors know about $\{x\}$. In the second round, we have a new partition of processors. Since $P_k$ and $\bar{P_{k}}$ differs only at the $0$-th bit but now we have divided the processors by another bit, they cannot be counterparts this time or in any future rounds. Hence, they will both have a new counterpart who can receive the datum $\{x\}$. When this round finishes, now 4 processors know about $\{x\}$, and so forth...

      Given that any two processors can be counterparts in only one round, we will always have new pairs of counterparts in each round. As a result, after the $j$-th iteration, $2^j$ processors will know about $\{x\}$, and finally the datum will be propagated across the entire interconnection network.

      \item[3.] The \textsc{Descend} and \textsc{Ascend} paradigm can be a powerful tool in many scenarios.

      If the binary operation is set intersection and each processor has a set, we can find the common values and remove duplicates. If the binary operation is set union, we can also find all distinct values or aggregate the data. For other binary operations such as the max()/min() function or multiplication, we can compute the maximum, minimum and product of numbers in an array or other data structures. It is also possible to use logical gates in parallel, where each processor handles a condition that is either true or false. For instance, we can check if the constraints of a linear program are violated, or if an interpretation of the 3-SAT formula is satisfiable.
    \end{enumerate}

  \vspace{8mm}
  \printbibliography

  \vfill
  \doclicenseThis

\end{document}