\section{Traceability}~\label{sec:trace}
 To ensure that the requirements and safety analyses performed early in
systems development are reflected throughout the lifecycle, many
guidelines for safety-critical software, such as DO-178C, require
documentation of traceability from requirements to object
code. Consequently, to promote the acceptance of high-assurance RV, the
monitor generation frameworks should produce documentation that
supports traceability from specification to monitor code.

 \paragraph{Challenge:} \emph{Support  traceability from the requirements  and system level analysis to the actual monitor code.}

 \paragraph{Copilot Approach:} Using SBV to generate C monitors may
 create many small files and it can be quite difficult to relate this to
 the specification. The code generation module has recently been
 revised to generate documentation that improves traceability. The
 user can insert labels in their specifications that flow down to the
 documentation. The translation process creates C header files with
 documentation formatted to be processed by the plain text graph
 description language processor DOT~\cite{dot}. Each C function has
 accompanying auto-generated graphical documentation.

 In the case of the following example:
\begin{lstlisting}[frame=single]
hor_rr sx sy  = (label "?hor_rr_dividend" $ sqrt $
                (normsq2dim sx sy) - (minHorSep * minHorSep))
                / (label "?hor_rr_divisor" $ minHorSep)
\end{lstlisting}
the SBV translation breaks this relatively simple expression into
numerous small C functions and function parameters get instantiated
with the variables being monitored.  The auto-generated documentation for one
of these files  appears similar to   Figure~\ref{fig:trace2}, where the labels have
the names of the program variables being monitored.

%\begin{figure}[ht!]
%        \centering
%        \resizebox {0.5\textwidth} {!} {
%\begin{tikzpicture}[->, node distance=1.5cm, auto, shorten >=1pt, bend angle=45 ]
%\tikzstyle{every state}=[rectangle]
%\node[state] (fil1) {ext\_ident\_double\_10\_arg0};
%\node[state](la)[below of = fil1]{label:hor\_rr\_dividend};
%\node[state](sqrt) [below of = la]{ext\_sqrt9};
%        \tikzstyle{every node}=[]
%        \path
%        (fil1) edge [->, anchor=east] node {} (la)
%        (la) edge [->, anchor=east] node  {}(sqrt);
%        \end{tikzpicture}
%}
%        \caption{Autogenerated }
%       \label{fig:trace}%        \end{figure}

\begin{figure}[ht!]
        \centering
        \resizebox {0.5\textwidth} {!} {
\begin{tikzpicture}[->, node distance=2.3cm, auto, shorten >=1pt, bend angle=45 ]
\tikzstyle{every state}=[rectangle]
\node[state] (fil2) {ext\_sqrt};
\node[state](opmin)[right of = fil2]{OP2:-};
\node[state](v1)[below left of = opmin]{ext\_ident\_double\_8};
\node[state](opmul)[below right of = opmin]{OP2:*};
\node[state](mv1)[below left of =opmul]{ext\_min\_hor\_sep};
\node[state](mv2)[below right of =opmul]{ext\_min\_hor\_sep};

        \tikzstyle{every node}=[]
        \path
        (fil2) edge [->, anchor=east] node {} (opmin)
        (opmin) edge [->, anchor=east] node  {}(v1)
        (opmin) edge [->, anchor=west] node  {}(opmul)
        (opmul) edge [->, anchor=east] node  {}(mv1)
        (opmul) edge [->, anchor=west] node  {}(mv2);
        \end{tikzpicture}
}
        \caption{Autogenerated Documentation }
        \label{fig:trace2}
        \end{figure}

%\begin{figure}[ht]
%\begin{center}
%\includegraphics[width=\textwidth,height=10cm]{rrfig2.pdf}
% \label{fig:label}
%\end{center}
%\end{figure}

%\begin{itemize}
%\item Many conventional processes based  guidance
% documents require ``traceability'' from the requirements to the
%  executable.
%\item In assured RV we should be able to trace from requirements to
%  monitor spec to monitor.
%\item We have developed a feature for adding labels to Copilot specs
 %  facilitating traceability.
%\item  We leveraged the GNU dot tool to help visualize what is going
%  on.

%\end{itemize}


\subsection{Representational sparsity}

This is where the values in nodes are often \(0\), as opposed to just the parameters.


\documentclass[11pt,a4paper]{article}
\usepackage{a4wide}
\usepackage{enumerate}
\usepackage{enumitem}
\usepackage{pcptex}
\usepackage{xspace}
\usepackage[toc,page]{appendix}

\graphicspath{{figures/}}
\usepackage{subcaption}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}

\usepackage[font=small,labelfont=bf,tableposition=top]{caption}

% ref packages
\usepackage{nameref}
% folowing  must be in this order
\usepackage{varioref}
\usepackage{hyperref}
\usepackage{cleveref}

%\usepackage{wrapfig}
\usepackage[utf8]{inputenc}
\usepackage{pgfplots}
%\usepgfplotslibrary{plotmarks}
\newlength{\figureheight}
\newlength{\figurewidth}

\makeatletter
\def\BState{\State\hskip-\ALG@thistlm}
\makeatother

\begin{document}

\pptitle{A High Performance Implementation of the Escape Time Algorithm to generate the Mandelbrot Set}

\vspace{-1.1cm}
\ppintro{Concetto Emanuele Bugliarello}{EPFL}{/}{Only PI}{\today}{June 4, 2017}{Deneb}{MNDLBRT}

\vspace{-1.3cm}
\ppabstract{
The Mandelbrot set is at the heart of fractal-like structures and many people have generated breathtaking drawings by sampling the complex numbers in it.
In this project, we present a parallel implementation of the \textquotedblleft escape time\textquotedblright~algorithm, one of the algorithms used to produce such drawings.
Our application is coded in C and makes use of the MPI standard and the CUDA platform.
The source code is available in the c4science repository at: \url{https://c4science.ch/diffusion/3860/mndlbrt.git}
}

\section{Scientific Background}

The Mandelbrot set is the set of complex numbers $c$ for which the function $f_{c}(z)=z^{2}+c$ does not diverge when iterated from $z=0$~\cite{wiki:mandelbrot}.
That is, if we denote the Mandelbrot set by $M$, then by repeatedly applying the quadratic map
\[
\left\{
\begin{array}{ll}
z_0 = 0\\
z_{n+1} = z_n + c,
\end{array}
\right.
\]
for any complex number $c$, we have that $c\in M\iff\limsup _{n\to \infty }|z_{n+1}|\leq 2$.\\

To generate Mandelbrot set images, we can sample the complex numbers and iterate this function for each point $c$ that has been sampled.
If the function goes to infinity, then the point belongs to the set.
Each sampled point can then be represented on a 2D plane by treating its real and imaginary parts as image coordinates $(x+yi)$ and coloring the corresponding pixel according to how quickly the sequence $z_{n}^2 + c$ diverges.\\
Two examples of such representation are shown in Figure \ref{fig:both-mandelbrot}.
In particular, points belonging to the set are colored in black, while points not in the set are colored according to how many iterations are required for the absolute values of the corresponding sequences to become greater than a cutoff value (2).\\
\begin{figure}[t]
	\vspace{-0.5cm}
	\centering
	\begin{subfigure}{.35\textwidth}
		\centering
		\includegraphics[width=.95\linewidth, clip, trim={3cm 0cm 2cm 0cm}]{red.pdf}
		\caption{Red-shaded drawing.}
		\label{fig:mandelbrot-red}
	\end{subfigure}%
	\begin{subfigure}{.35\textwidth}
		\centering
		\includegraphics[width=.95\linewidth, clip, trim={3cm 0cm 2cm 0cm}]{blue.pdf}
		\caption{Smooth, blue-shaded drawing.}
		\label{fig:mandelbrot-blue}
	\end{subfigure}
	\vspace{-0.3cm}
	\caption{Mandelbrot set drawings generated with our palettes.}
	\label{fig:both-mandelbrot}
\end{figure}

There exist many algorithms to draw the Mandelbrot set.
Since our goal is to produce a parallel implementation in CUDA~\cite{cuda} and MPI~\cite{mpi}, we use one of the simplest algorithms to generate a pictorial representation of the Mandelbrot set: the \textquotedblleft escape time\textquotedblright~algorithm.
In fact, this algorithm results in an embarrassingly parallel problem: each pixel does not depend on any other, allowing each thread in the GPU to run its computations independently and avoiding exchanging ghost cells when MPI is used.
The pseudocode of this algorithm follows.\\
\setlength{\textfloatsep}{6pt} % Space after algorithm
\begin{algorithm}
	\caption{Escape Time algorithm}\label{euclid}
	\begin{algorithmic}[1]
%		\Procedure{}{}
		\For {each pixel (Px, Py) on the screen}
		\State $x_0 \gets$ scaled x coordinate of pixel
		\State $y_0 \gets$ scaled y coordinate of pixel
		\State $x \gets 0.0$
		\State $y \gets 0.0$
		\State $iteration \gets 0$
		\While {$(x\times x + y\times y < 2\times 2$  AND  $iteration < max\_iteration)$}
			\State $x_{temp} \gets x\times x - y\times y + x_0$
			\State $y \gets 2\times x\times y + y_0$
			\State $x \gets x_{temp}$
			\State $iteration \gets iteration + 1$
		\EndWhile
		\State $color \gets palette[iteration]$
		\State $plot(Px, Py, color)$
		\EndFor
%		\EndProcedure
	\end{algorithmic}
\end{algorithm}
\noindent
Where $z = x + iy$, $c = x_0 + iy_0$, $x = Re(z^2+c)$ and $y=Im(z^2+c)$.\\
Note that we keep iterating up to a fixed number of steps.
Hence, we decide that a sampled point is \textquotedblleft probably\textquotedblright~in the Mandelbrot set after $max\_iteration$ iterations.

\vspace{-0.3cm}
\section{Implementations}

The application is coded in C~\cite{ritchie} and three main classes of implementations are developed:
\begin{itemize}
\vspace{-0.2cm}
\itemsep0em

\item Serial versions.
\item GPU versions using the CUDA parallel computing platform.
\item Hybrid versions using both CUDA and MPI (including MPI-IO).
\end{itemize}
\vspace{-0.1cm}
The code has been fully debugged using the \texttt{gdb} debugger.
The \texttt{Valgrind} tool has been used to make sure that all heap blocks are freed and hence no memory leaks are possible.

\vspace{-0.3cm}
\subsection{Serial implementations}

We start by implementing a naive version of the described algorithm and two plotting functions: a simple one with a red hue (Figure \ref{fig:mandelbrot-red}), and a smoother version in blue\footnote{It uses an approximation of the Normalized Iteration Count method presented in \cite{smooth}.} (Figure \ref{fig:mandelbrot-blue}).
While the latter is prettier, it needs the real and imaginary values reached by each pixel in the quadratic map, thus requiring, in a first implementation, the allocation of two additional matrices of doubles (one for the real and one for the imaginary parts) of the same size as the total number of pixels.
Throughout this project, we produce smooth, blue drawings as in Figure \ref{fig:mandelbrot-blue}.\\[-7pt]
\begin{table}[t]
	\vspace{-0.5cm}
	\begin{minipage}[b]{0.4\linewidth}
		\centering
		\frame{\includegraphics[width=0.565\textwidth]{bulbs.png}}
		\vspace{0.2cm}
		\captionof{figure}{Periods of hyperbolic components~\cite{wiki:mandelbrot}.} \label{fig:bulbs}
	\end{minipage}\hfill
	\begin{minipage}[b]{0.56\linewidth}
		\centering
		\begin{tabular}{|l|l|}
			\hline
			\textbf{Optimization level} & \textbf{Execution time [s]} \\ [0.2ex]
			\hline\hline
			\texttt{-O0} & 422.91 \\
			\texttt{-O1} & 230.99 \\
			\texttt{-O2} & 208.96 \\
			\texttt{-O3} & 208.49 \\
			\texttt{-O3 -ftree-vectorize} & 205.41 \\ [0.2ex]
			\hline
		\end{tabular}
		\vspace{0.2cm}
		\caption{Execution times of \texttt{opti3} for a $25600\times14400$-pixel image and $max\_iteration=10,000$ as a function of the optimization level.}
		\label{tab:opti-levels}
		\vspace{-0.2cm}
	\end{minipage}
\end{table}

\noindent
Beside the standard optimizations (such as vectorization), we exploit the characteristics of the problem at hand in order to further improve the performance.\\[-7pt]

Firstly, as we can see from the pictures above, the resulting figure is symmetric with respect to the $x$ axis.
Then, we only compute the values for half of the specified height and fill the other half of the image during the plotting phase.
We refer to this first optimization as \texttt{opti1}.\\[-7pt]

Secondly, it is possible to skip the calculations for the points lying within the cardioid or in the period-2 bulb (marked with 1 and 2, respectively, in Figure \ref{fig:bulbs}).

%\begin{figure}
%	\vspace{-0.2cm}
%	\centering
%	\frame{\includegraphics[width=0.2\textwidth]{bulbs.png}}
%	\caption{Periods of hyperbolic components~\cite{wiki:mandelbrot}.} \label{fig:bulbs}
%	\vspace{-0.1cm}
%\end{figure}
\noindent
To do so, before passing a point to the escape time algorithm, we first check whether one of the following equations hold:
\[
\left\{
\begin{array}{ll}
q \left(q + \left(x - \frac{1}{4}\right)\right) < \frac{1}{4}y^2\\
\\
\left(x+1\right)^2 + y+^2 < \frac{1}{16},
\end{array}
\right.
\]
where $q = \left(x - \frac{1}{4}\right)^2 + y^2$, and $x$ and $y$ represent the real and the imaginary parts of the point.\\
The first equation in the system determines if the point is within the cardioid, while the second one determines if it is within the period-2 bulb.
The two shapes cover $\approx34\%$ of the total area.\\
3rd- and higher-order bulbs do not have equivalent tests because they are not perfectly circular.\\
We refer to the implementation using this optimization and the previous one as \texttt{opti2}.\\[-7pt]

Finally, we apply some finer-grained optimizations.
In the above pseudocode, $y$ is computed with two multiplications and one addition.
Since multiplications are more expensive than additions, we would like to replace them by additions.
Here, we can remove one multiplication in lieu of an addition: we firstly assign $x\times y$ to $y$, then we add $y$ to itself (hence removing the multiplication by 2), and finally add $y_0$.
Moreover, since we need $x^2$ and $y^2$ both in the \texttt{while} condition and in $x_{temp}$, we add two variables to store them so as to avoid recomputing two multiplications.
By doing so, we only have three multiplications per loop, which is the minimum that can be achieved in this algorithm.
%In addition, we optimize the plotting function by using fast indexing into the image array.
%Since each pixel is defined by three colors (RGB) and they are adjacent in the image data structure, we introduce a variable called \texttt{offset} and use \texttt{offset++} to fill it.
%This is faster than either computing the right offset each time or computing \texttt{offset+1}, \texttt{offset+2} and \texttt{offset+3}.
We also employ fast array indexing while plotting the image.
The implementation using all the described optimizations is denoted \texttt{opti3}.\\[-7pt]

The performances of the presented optimizations are compared in Figure \ref{fig:serial-optis}.
\begin{figure}[t]
	\vspace{-0.7cm}
	\centering
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{serial-optis-px.pdf}
		\caption{}
		\label{fig:serial-optis-px}
	\end{subfigure}%
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{serial-optis-maxiter.pdf}
		\caption{}
		\label{fig:serial-optis-maxiter}
	\end{subfigure}
	\vspace{-0.4cm}
	\caption{Time to solution for the different serial implementations with respect to (a) the number of pixels (max\_iteration=10,000) and (b) max\_iteration (368,640,000 pixels).}
	\label{fig:serial-optis}
%	\vspace{-0.2cm}
\end{figure}
%\begin{figure}
%	\centering
%	\begin{subfigure}{.5\textwidth}
%		\centering
%		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{serial-plotnoplot-px.pdf}
%		\caption{}
%	\end{subfigure}%
%	\begin{subfigure}{.5\textwidth}
%		\centering
%		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{serial-plotnoplot-maxiter.pdf}
%		\caption{}
%	\end{subfigure}
%	\vspace{-0.4cm}
%	\caption{Time to solution of the \texttt{opti3} version when the image is plotted or not with respect to (a) the number of pixels (max\_iteration=10,000) and (b) max\_iteration (368,640,000 pixels).}
%	\label{fig:plot-noplot}
%	\vspace{-0.28cm}
%\end{figure}
\noindent
As we can see, each of these optimizations reduces the time to solution.
In particular, we halve the time to solution when we evaluate the escape time only on half image, and a drastic reduction is observed when we avoid evaluating the escape time for the sampled numbers lying within the cardioid or in the period-2 bulb as each of them would reach $max\_iteration$ otherwise.

Each of serial implementations used in generating this figure were compiled using \texttt{gcc} and setting the compiler's optimization level to \texttt{-O3} with the loop vectorizer (\texttt{-ftree-vectorize}).
In fact, this gives the best performance for each of them.
As an example, Table \ref{tab:opti-levels} illustrates the execution times for \texttt{opti3} as it is compiled with different optimization levels.
%\begin{table}[h!]
%	\centering
%	\vspace{-0.1cm}
%	\begin{tabular}{|l|l|}
%		\hline
%		\textbf{Optimization level} & \textbf{Execution time [s]} \\ [0.2ex]
%		\hline\hline
%		\texttt{-O0} & 422.91 \\
%		\texttt{-O1} & 230.99 \\
%		\texttt{-O2} & 208.96 \\
%		\texttt{-O3} & 208.49 \\
%		\texttt{-O3 -ftree-vectorize} & 205.41 \\ [0.2ex]
%		\hline
%	\end{tabular}
%	\vspace{-0.1cm}
%	\caption{Execution times of \texttt{opti3} for a $25600\times14400$-pixel image and $max\_iteration=10,000$ as a function of the optimization level.}
%	\label{tab:opti-levels}
%	\vspace{-0.2cm}
%\end{table}

%Finally, Figure \ref{fig:plot-noplot} compares the times to solution of \texttt{opti3} both when the resulting image is drawn and when the execution stops just after evaluating the color of each pixel.

We also compare the times to solution of \texttt{opti3} both when the resulting image is drawn and when the execution stops just after evaluating the color of each pixel: the two curves basically overlap suggesting that the writing time is negligible.

%\subsection{Optimizations}
%
%Beside the standard sequential optimizations (loop reordering, vectorization, etc..), we put a special effort to tackle the main bottleneck of the beta-versions of the MPI and hybrid implementations : the I/O part. The current production version implements a BMP-file writer in parallel using MPI-IO.

\vspace{-0.2cm}
\section{Prior results}

We run our implementations on the Deneb cluster at EPFL, whose specifics are~\cite{scitas}:
\begin{itemize}
\vspace{-0.2cm}
\itemsep0em

\item 376 compute nodes, each with (i) 2 Ivy Bridge processors running at 2.6 GHz, with 8 cores each, and (ii) 64 GB of DDR3 RAM.
\item 144 compute nodes, each with (i) 2 Haswell processors running at 2.5 GHz, with 12 cores each, and (ii) 64 GB of DDR4 RAM.
\item 16 GPU accelerated nodes, each with 4 K40 NVIDIA cards.
\item Infiniband QDR 2:1 connectivity.
\item GPFS filesystem.
\item Total peak performance: 293 TFLOPs (211 in CPUs, 92 in GPUs).
\end{itemize}

\vspace{-0.5cm}
\subsection{Theoretical computational cost}

In this section, we try to estimate the computational cost of the escape time algorithm.
In particular, we consider the \texttt{opti3} implementation of the algorithm.\\
For the sake of simplicity, let's consider generating square figures, where width and height have $N$ pixels.
Due to the previous symmetry optimization, the algorithm is called $N\times\frac{N}{2}$ times.\\[-8pt]

An approximation of the theoretical cost can be obtained by considering a worst-case scenario where each sampled complex number that is neither in the cardioid nor in the period-2 bulb iterates $max\_iteration$ times.\\[-7pt]

%However, we can provide a slightly better approximation by considering that the average number of iterations is independent of the number of pixels and of the maximum number of iterations, as shown in Figure \ref{fig:const}.
%\begin{figure}
%	\centering
%	\begin{subfigure}{.5\textwidth}
%		\centering
%		\includegraphics[width=.95\linewidth]{itersVSsize.png}
%		\caption{}
%		\label{fig:const-size}
%	\end{subfigure}%
%	%	\begin{subfigure}{.42\textwidth}
%	%		\centering
%	%		\setlength{\figureheight}{6cm}
%	%		\setlength{\figurewidth}{6cm}
%	%		\input{figures/itersVSsize.tikz}
%	%		\caption{Box plots for the different performance metrics.}
%	%		\label{fig:box-plots2}
%	%	\end{subfigure}
%	\begin{subfigure}{.5\textwidth}
%		\centering
%		\includegraphics[width=.95\linewidth]{itersVSmaxiter.png}
%		\caption{}
%		\label{fig:const-max}
%	\end{subfigure}
%	\caption{Evolution of the average number of iterations per pixel with respect to (a) number of pixels and (b) max\_iteration.}
%	\label{fig:const}
%\end{figure}
%Nevertheless, the execution time increases both when the number of pixels grows and when the maximum number of iterations is larger.
%This is what we expected by looking at the pictures above: most of the pixels are usually mapped to similar regions independently of the image size, and there are few pixels that iterates many times (pixels in higher-order bulbs and close to the boundaries).
%In order to obtain an optimal parallelization, we will need to properly deal with this skewness in the number of iterations (load balancing).\\
\noindent
In evaluating the computational cost, we assume the following costs per operation~\cite{costs}:
\begin{itemize}
	\vspace{-0.2cm}
	\itemsep0em

	\item addition, subtraction, comparison (1);
	\item multiplication, division by 2 (4).
	\vspace{-0.15cm}
\end{itemize}
In our implementation of the escape time algorithm, we have:
\begin{itemize}
	\vspace{-0.2cm}
	\itemsep0em

	\item 5 sums, 4 multiplications and 2 divisions by 2 to initialize the variables of each pixel;
	\item 5 sums, 1 subtraction, 2 comparisons and 5 multiplications to check if the pixel is within the cardioid or in the period-2 bulb;
	\item 5 sums, 1 subtraction, 2 comparisons and 3 multiplications in each iteration.
\end{itemize}
\vspace*{-1.1cm}
Note that these are simple approximations and the real cost of each operation actually depends on the computer architecture.
Furthermore, we are ignoring the cost of conditional statements which lead to expensive branch mispredictions.\\
That said, an estimation of the computational cost for an $N\times N$-pixel image is then given by:
\vspace{-0.15cm}
$$\approx (c + 0.76\times max\_iteration) \times \dfrac{N^2}{2}$$
\vspace{-0.1cm}
%\begin{multline*}
%\frac{N^2}{2}\times[\left(5\times1 + 4\times4 + 2\times4\right) + \\
%\left(5\times1 + 1\times1 + 2\times1 + 5\times4\right) + \\
%\left(1-0.34\right)\times221\times\left(5\times1 + 1\times1 + 2\times1 + 3\times4\right)]
%\end{multline*}
%\begin{multline*}
%\approx c\times N^2~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%\end{multline*}
\noindent
Where $0.76$ is the percentage of pixels that are neither in the cardioid nor in the period-2 bulb and $c\approx 57$.
Hence, the algorithm's computational cost is $O(N^2)$ and so linear in the number of pixels (as we expected from its formulation).\\[-7pt]

Note that this cost does not take into account the expensive operation to determine the smooth color for each pixel and the additive time needed to store the matrix of iterations as an image, which are also $O(N^2)$.
Memory requirements are in the order of $O(N^2)$ too.\\[-7pt]

Finally, if we fix the size of the image, then the computational time increases linearly with the value of $max\_iteration$.
All these relationships are confirmed by the curves in Figure \ref{fig:serial-optis}.

\vspace{-0.1cm}
\subsection{Strong scaling}
Strong scaling means using parallel computing to run a problem faster than on a single core.\\
Strong scaling is usually equated with Amdahl's Law, which specifies the maximum speedup that can be expected by parallelizing portions of a serial program:
\vspace{-0.1cm}
$$S_p = \frac{1}{\alpha + \frac{1 - \alpha}{p}} = \frac{T_1}{T_p}$$
Where $\alpha$ is the fraction of non-parallelizable code, $p$ is the number of processors, and $T_1$ and $T_p$ are the execution times on one and p processors, respectively.\\[-7pt]

Hence, we need to estimate the fraction of non-parallelizable code $\alpha$.
The core of the algorithm is intrinsically parallel because of the independence of the computations between the pixels.
However, serial sequences are present as well to (i) allocate and deallocate the variables, (ii) set the corresponding number of iterations in the other half of the image and (iii) store the resulting image onto disk.
The impact of the serial part will be more and more evident as the size of the image (number of pixels) increases.\\
We use \texttt{gprof} to obtain an estimation of the value of $\alpha$.
In particular, we do this when drawing both a 4K image ($8,294,400$ pixels) and a $25600\times14400$ image ($368,640,000$ pixels).
In both cases we set $max\_iteration$ to $10,000$.
The value of $\alpha$ is estimated to be $0$ in the former case, while equal to $1.91$ in the latter; hence suggesting a perfect speedup for relatively small images.\\
Figure \ref{fig:exp-speedup} shows the expected speedup in these two scenarios.
\begin{figure}[H]
	\centering
	\includegraphics[width = 0.5\textwidth, clip, trim={1.5cm 1cm 2cm 2cm}]{expected-speedup.pdf}
	\caption{Expected speedup for small ($\alpha=0\%$) and large ($\alpha=1.91\%$) images from Amdahl's law.}
	\label{fig:exp-speedup}
\end{figure}

%\begin{figure}[ht]
%	\centering
%	\setlength\figureheight{10cm}
%	\setlength\figurewidth{16cm}
%	\input{figures/expected-speedup.tikz}
%	\caption{Box plots for the different performance metrics.}
%	\label{fig:exp-speedup}
%\end{figure}

\subsection{Weak scaling}
Weak scaling is what we refer to when we are talking about using parallel computing to run a larger problem in the same amount of time as a smaller one on a single core.
In other words, it is a measure of how the time to solution changes as more processors are added to a system with a fixed problem size per processor.\\[-7pt]
%Weak scaling is often equated with Gustafson's Law, which states that the problem size scales with the number of processors:
%$$S_p = p - \beta(n)\left(p - 1\right) = \frac{T_1}{T_p}$$
%Where $\beta$ is the fraction of non-parallelizable code and $n$ is the problem size.\\

We expect a significant speedup also in this case, but we are somehow limited by the size of the images we can create.
So, we can instead increase the complexity of the problem by choosing larger values of $max\_iteration$.
Knowing that the peak number of floating-point operations per second in one TESLA K40 GPU card is $1.43~TFLOPs$ for double-precision and $4.29~TFLOPs$ for single-precision~\cite{gpus}, while having a peak of $\frac{211}{376\times2\times8 + 144\times2\times12}\approx22.3~GFLOPs$ in a single core of CPU in the Deneb cluster, we can predict solving problems that are $\approx66$ (double-precision) or $\approx197$ (single-precision) times larger on the GPU in the same time as a reference problem on a single CPU core.
Despite this wide gap, we should remember that these are theoretical predictions and several issues will actually limit the parallel performance, such as memory communication overheads and load balancing.

\vspace{-0.2cm}
\section{Going parallel}

After having debugged, profiled and optimized our serial code, we can proceed with the parallelization phase.
In fact, since each element in the array is independent of all the others, our algorithm leads to an embarrassingly parallel formulation.

\vspace{-0.2cm}
\subsection{CUDA implementations}

We start parallelizing our code with CUDA; once this version is optimized, we will add MPI.\\[-7pt]

Firstly, we just port \texttt{opti3} to CUDA.
We do so by parallelizing the core of the algorithm: that is, evaluating the escape times for each of the points in the bottom half of the figure.
This version is still far from optimal: most of the time spent by the serial code is in computing the color associated to a given iteration value and this part is still done serially.
Moreover, we still use three matrices: one for the escape times, one for the real parts reached by the quadratic mappings and one for the corresponding imaginary parts.
Sending three matrices on the PCIe bus surely does not lead to high performance computing, but this version serves as a first working CUDA implementation of the algorithm.\\[-7pt]

After debugging this version, we proceed implementing a new one: \texttt{opti4}.
In this version, all the workload is carried out by the GPU.
The only values exchanged by the between CPU and the GPU are the chars representing the RGB values for each pixel in the bottom half of the image.
The code is split into serial part (\texttt{.c}) and parallel part (\texttt{.cu}), compiled with \texttt{gcc} and \texttt{nvcc}, respectively, and then linked together into one executable.
To do so, we need a C wrapper for the kernel that allows the CPU to share parameters with the CUDA part.
Moreover, all the constant values are copied to the Constant memory: a read-only, quick cache that can broadcast to all active threads.\\
When we profile this application with \texttt{nvprof}, we notice that it is compute-bound: that is, its arithmetic intensity is underneath the peak performance ceiling in the Roofline model.\\[-7pt]

Finally, we move from a double-precision to a single-precision representation with \texttt{opti5}.
Strictly speaking, the resulting drawings are different from the previous ones but no human being would be able to tell the difference between two images produced using double-precision and single-precision operands.
We make this shift to single precision in order to maximize instruction throughput (as we described above and it is suggested in \cite{guide}).\\
The resulting optimization is still compute-bound.\\[-7pt]

\noindent
The performances of the presented CUDA optimizations are compared in Figure \ref{fig:cuda-optis}.\\[-7pt]
\begin{figure}[t]
	\vspace{-0.7cm}
	\centering
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{cuda-optis-px.pdf}
		\caption{}
		\label{fig:cuda-optis-px}
	\end{subfigure}%
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{cuda-optis-maxiter.pdf}
		\caption{}
		\label{fig:cuda-optis-maxiter}
	\end{subfigure}
	\vspace{-0.4cm}
	\caption{Time to solution for the CUDA implementations with respect to (a) the number of pixels (max\_iteration=10,000) and (b) max\_iteration (368,640,000 pixels) with block size $512\times1$.}
	\label{fig:cuda-optis}
\vspace{0.3cm}
\end{figure}

We complete our study on CUDA by tuning the \textit{block size} used by \texttt{opti5}.
This is a crucial hyperparameter that leads to performance improvements and it is determined by both the problem at hand and the specifics of the used GPU.
A properly set block size leads to increased occupancy; i.e., the ratio between the number of active warps per multiprocessor and the maximum number of active warps.
Table \ref{tab:block-size} presents the achieved occupancy as a function of the block size for two image sizes when launched on Deneb.
The winning size is \texttt{256$\times$1}.
We make use of \texttt{nvprof} to evaluate the occupancy.
Even though larger image sizes and higher $max\_iteration$ would lead to a more accurate evaluation of the achieved occupancy, we had to set $max\_iteration$ to only $10,000$ in order for \texttt{nvprof} to evaluate this metrics.
Nevertheless, the highest achieved occupancy is close to $66\%$, which is enough to usually saturate the bandwidth~\cite{occupancy}.
Further, one should look at increasing occupancy only if the the kernel is bandwidth-bound, which is not our case.
\begin{table}[H]
	\begin{subtable}{.5\linewidth}
		\centering
		\begin{tabular}{|l|l|l|}
			\hline
			\textbf{Block size} & \multicolumn{2}{|c|}{\textbf{Occupancy [$\%$]}} \\
			\cline{2-3}
			 & \texttt{2560x1440} & \texttt{25600x14400} \\ [0.2ex]
			\hline\hline
			\texttt{32x1} & 18.05 & 18.81 \\
			\texttt{64x1} & 25.41 & 32.65 \\
			\texttt{128x1} & 31.82 & 51.52 \\
			\texttt{192x1} & 34.85 & 56.01 \\
			\texttt{256x1} & \textbf{36.15} & \textbf{58.80} \\
			\texttt{384x1} & 35.62 & 57.30 \\
			\texttt{512x1} & 28.76 & 52.37 \\
%			\texttt{768x1} & 13.79 & 37.59 \\
			\texttt{1024x1} & 8.39 & 25.50 \\
			\texttt{8x8} & 19.04 & 20.12 \\
			\texttt{16x16} & 26.91 & 25.93 \\
			\texttt{32x32} & 5.91 & 7.74 \\ [0.2ex]
			\hline
		\end{tabular}
		\caption{}
	\end{subtable}%
	\begin{subtable}{.5\linewidth}
		\centering
		\begin{tabular}{|l|l|l|}
			\hline
			\textbf{Block size} & \multicolumn{2}{|c|}{\textbf{Execution time [s]}} \\
			\cline{2-3}
			& \texttt{2560x1440} & \texttt{25600x14400} \\ [0.2ex]
			\hline\hline
			\texttt{32x1} & 2.53 & 79.61 \\
			\texttt{64x1} & 2.29 & 50.32 \\
			\texttt{128x1} & 2.10 & 37.52 \\
			\texttt{192x1} & 2.05 & 36.56 \\
			\texttt{256x1} & \textbf{2.04} & \textbf{35.91} \\
			\texttt{384x1} & 2.11 & 36.60 \\
			\texttt{512x1} & 2.17 & 37.69 \\
%			\texttt{768x1} & 2.38 & 44.73 \\
			\texttt{1024x1} & 5.65 & 90.60 \\
			\texttt{8x8} & 4.09 & 237.72 \\
			\texttt{16x16} & 2.45 & 97.12 \\
			\texttt{32x32} & 7.48 & 270.24 \\ [0.2ex]
			\hline
		\end{tabular}
		\caption{}
	\end{subtable}
	\caption{(a) Occupancy and (b) execution time for two image sizes as the block size varies.}
	\label{tab:block-size}
\end{table}

\vspace*{-0.27cm}
\subsection{Hybrid implementations}

After having optimized on a single GPU, we introduce an additional layer of parallelization: we distribute the computations over different CPUs and GPUs with MPI.
Specifically, we split the bottom half of the image to all the reserved GPUs by rows, and so each GPU receives the same number of pixels~\footnote{If dividing the number of rows by the number of GPUs does not return an integer, the remaining rows are assigned as uniformly as possible among all the GPUs.}.\\
Each GPU node on Deneb has two processors and four GPUs.
We then distribute \texttt{opti5} so as to make use of the resources available on a single node; we do so with two versions.\\[-7pt]

Firstly, we distribute the image to the two processors and each of them uses one GPU to evaluate the escape times and the corresponding colors.
We provide two implementations: a first implementation using non-blocking, point-to-point communications where the processor with rank $0$ receives the image parts in an out-of-order fashion.
In particular, no \texttt{MPI\_Barrier()} is used.
The second implementation, instead, makes use of parallel I/O to increase the writing performance on the GPFS parallel file system.\\[-7pt]

Secondly, we distribute the image to the two processors and each of them uses two GPUs.
That is, each processor is in charge of half of the bottom half image, which is then further split to the two GPUs.
In this scenario, to achieve the highest performance, all the CUDA functions are asynchronous (with no barrier or thread synchronization used).
Also this version comes with both the centralized, non-blocking implementation and the one using parallel I/O.\\[-7pt]

The performances of the hybrid implementations are shown in Figure \ref{fig:hybrid-optis}, where we can see that the benefits of using multiple GPUs is manifested only for massive loads.
\begin{figure}
	\vspace{-0.7cm}
	\centering
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{hybrid-optis-px.pdf}
		\caption{}
		\label{fig:hybrid-optis-px}
	\end{subfigure}%
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{hybrid-optis-maxiter.pdf}
		\caption{}
		\label{fig:hybrid-optis-maxiter}
	\end{subfigure}
	\vspace{-0.4cm}
	\caption{Time to solution for the hybrid implementations with respect to (a) the number of pixels (max\_iteration=10,000) and (b) max\_iteration (368,640,000 pixels) with block size $256\times1$.}
	\label{fig:hybrid-optis}

\end{figure}

\vspace{-0.2cm}
\section{Final results}

In this section, we present the results we obtained for scaling when CUDA and MPI are used.\\
The results using multiple GPUs versions are obtained from the parallel I/O implementations.\\[-7pt]

Figure \ref{fig:strong} presents the results for strong scaling: specifically, for a $25600\times14000$-pixel image and $max\_iteration=10,000,000$.\\
Figure \ref{fig:strong-mpi} shows how increasing the number of GPUs affects the speedup: here, we can see that the speedup obtained using four GPUs is higher than the ideal one.
This can be explained by the fact that all the CUDA functions are called asynchronously in this version and both the multi-GPU versions use two processors.\\
Figure \ref{fig:strong-tts} illustrates how the time to solution varies from the serial version to the hybrid version.\\[-7pt]
\begin{figure}[t]
	\vspace{-0.5cm}
	\centering
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{strong-mpi.pdf}
		\caption{}
		\label{fig:strong-mpi}
	\end{subfigure}%
	\begin{subfigure}{.5\textwidth}
		\centering
		\includegraphics[width=.99\linewidth, clip, trim={1.5cm 1cm 2.5cm 2cm}]{strong-tts.pdf}
		\caption{}
		\label{fig:strong-tts}
	\end{subfigure}
	\vspace{-0.4cm}
	\caption{Strong scaling as (a) the speedup for an increasing number of GPUs (via MPI) and (b) the reduced time to solution of the same problem ($25600\times14000$-pixel image, $max\_iteration=10,000,000$) for the different implementations.}
	\label{fig:strong}
\vspace{0.3cm}
\end{figure}

Finally, to evaluate the effect of weak scaling, we fix the image size to $368,640,000$ pixels and determine the value of $max\_iteration$ that allows each version to execute in around $76$ seconds.
The results are reported in the figure below.
\begin{figure}[H]
	\centering
	\includegraphics[width = 0.5\textwidth, clip, trim={1.5cm 1cm 2cm 2cm}]{weak.pdf}
	\caption{Weak scaling as a function of the problem size that can be solved in the same amount of time by the different implementations.}
	\label{fig:weak}
\end{figure}

\section{Resources budget}
\vspace{-0.1cm}
In order to fulfill the requirements of our project, we present hereafter the resource budget.

\vspace{-0.2cm}
\subsection{Computing Power}
\vspace{-0.1cm}
The parallel implementations are compute-bound; hence, powerful GPUs are required.

\vspace{-0.2cm}
\subsection{Raw storage}
\vspace{-0.1cm}
The main requirement in terms of raw storage is due to the output images.
The largest image that we generate is $2.1~GB$.

\vspace*{-1cm}
\subsection{Grand Total}

\begin{center}
\begin{tabular}{| l | l |}
	\hline
	Total number of requested GPUs & 4\\
%	\hline
%	Minimum total memory & \textbf{\texttt{!! FIXME !!}} \\
%	\hline
%	Maximum total memory & \textbf{\texttt{!! FIXME !!}} \\
	\hline
	Temporary disk space for a single run & 3 GB \\
	\hline
	Permanent disk space for the entire project & 50 GB \\
	\hline
	Communications & MPI \\
	\hline
	License & MIT \\
	\hline
	Code publicly available ? & Yes \\
	\hline
	Library requirements & CUDA $\&$ MPI \\
	\hline
	Architectures where code ran & Deneb  \\
	\hline
\end{tabular}
\end{center}

\vspace{-0.3cm}
\section{Scientific outcome}
\vspace{-0.1cm}
The aim of this project is to gain experience and expertise on CUDA and MPI while developing applications in a supercomputer.
Throughout the development process, we have acquired familiarity with different tools for profiling and optimizing serial and parallel applications, as well as different technologies (such as non-blocking communications and parallel I/O in MPI).
All the skills that we have mastered in this context will constitute a fundamental basis for real submissions in larger high performance computing centers.

\vspace{-0.2cm}
\begin{thebibliography}{1}

\bibitem{wiki:mandelbrot} Wikipedia, {\em Mandelbrot set}, Online - accessed 09-May-2017, \url{https://en.wikipedia.org/wiki/Mandelbrot_set}\\[-17pt]

\bibitem{cuda} NVIDIA, {\em CUDA: Parallel Programming and Computing Platform}, Online - accessed 09-May-2017, \url{http://www.nvidia.com/object/cuda_home_new.html}\\[-17pt]

\bibitem{mpi} The MPI Forum, {\em MPI: A Message-Passing Interface Standard}, Technical Report, 1994\\[-17pt]

\bibitem{ritchie} Kernighan, Brian W. and Ritchie, Dennis M., {\em The C Programming Language Second Edition}, Prentice-Hall, Inc.,1988\\[-17pt]

\bibitem{smooth} Stack Overflow, {\em Improvement to my Mandelbrot set code}, Online - accessed 09-May-2017, \url{http://stackoverflow.com/questions/16124127/improvement-to-my-mandelbrot-set-code}\\[-17pt]

\bibitem{scitas} Scientific IT and Application Support, {\em Deneb and its extension, Eltanin}, Online - accessed 14-May-2017, \url{http://scitas.epfl.ch/hardware/deneb-and-eltanin}\\[-17pt]

\bibitem{costs} Vincent Hindriksen, {\em How expensive is an operation on a CPU?}, Online - accessed 14-May-2017, \url{https://streamcomputing.eu/blog/2012-07-16/how-expensive-is-an-operation-on-a-cpu/}\\[-17pt]

%\bibitem{if} Stack Overflow, {\em Is “IF” expensive?}, Online - accessed 14-May-2017,
%\url{http://stackoverflow.com/questions/315306/is-if-expensive}\\[-17pt]

\bibitem{gpus} NVIDIA, {\em NVIDIA TESLA GPU ACCELERATORS}, Online - accessed 1-June-2017,
\url{http://www.nvidia.com/content/tesla/pdf/nvidia-tesla-kepler-family-datasheet.pdf}\\[-17pt]

\bibitem{guide} NVIDIA, {\em CUDA C Programming Guide}, Online - accessed 1-June-2017,
\url{http://docs.nvidia.com/cuda/cuda-c-programming-guide/#axzz4j1NpsHeT}\\[-17pt]

\bibitem{occupancy} NVIDIA, {\em CUDA Warps and Occupancy}, Online - accessed 1-June-2017,
\url{http://on-demand.gputechconf.com/gtc-express/2011/presentations/cuda_webinars_WarpsAndOccupancy.pdf}

\end{thebibliography}

\end{document}


\section{Poisson equation with PETSc}
  \label{section:applications:petsc}

\chapterDescription
  {
    60 minutes.
  }
  {
    Chapter \ref{chapter:quickstart} and a working PETSc installation.
  }

I wrote Peano with matrix-free methods for computational fluid dynamics in mind.
It turns out that it is not too complicated to use it for other application
areas or to established linear algebra packages.
We assume that you have PETSc plus BLAS, Lapack and MPI (if required) ready
installed on your system.
I have tested all the present concepts with PETSc 3.7.2 but everything discussed
here is pretty basic, so it should work with older versions as well.
Please note that this is a feasibility study and we do not optimise at all.

\subsection{Preparation of Peano}

If we use explicit assembly, we need an enumeration of our grid.
So we do start with a grid
\begin{code}
> java -jar <mypath>/pdt.jar --create-project petsc petsc
\end{code}

\noindent
and add each vertex an index.
We do restrict to a discretisation with one degree of freedom per vertex for the
time being:

\begin{code}
Packed-Type: short int;

class petsc::records::Vertex {
  parallelise persistent int index;
};
\end{code}

\noindent
For the present exercise, we will have four different things to do:
create the grid, create and assemble the matrix, solve the problem and plot the
result.
The solve is a pure PETSc operation.
For each of the remaining three phases, we need a mapping/adapter in Peano and
we also want access to \texttt{index}.
For the latter, we could write setters and getters and stick to an
object-oriented paradigm.
But this way, it is easier (though not as beautiful):

\begin{code}
component: MyFancyPETScProject

namespace: ::petsc

vertex:
  dastgen-file: Vertex.def
  read scalar(int): Index   // mind the uppercase
  write scalar(int): Index

cell:
  dastgen-file: Cell.def

state:
  dastgen-file: State.def

event-mapping:
  name: CreateGrid

event-mapping:
  name: Enumerate

event-mapping:
  name: Assemble

adapter:
  name: CreateGrid
  merge-with-user-defined-mapping: CreateGrid

adapter:
  name: Enumerate
  merge-with-user-defined-mapping: Enumerate

adapter:
  name: Assemble
  merge-with-user-defined-mapping: Assemble

adapter:
  name: Plot
  merge-with-predefined-mapping: VTKPlotVertexValue(result,getU,u)
\end{code}

We generate all glue code
\begin{code}
> java -jar <mypath>/pdt.jar --generate-gluecode \
  petsc/project.peano-specification petsc <mypath>/pdt/usrtemplates
> make -f petsc/makefile
\end{code}

\noindent
We recognise that this code does not compile yet as we have not yet realised \texttt{double Vertex::getU() const}.
For the time being, we make the routine return 0.
Finally, we change into the mapping the creational events yield a regular grid:

\begin{code}
void petsc::mappings::CreateGrid::createInnerVertex(...) {
  logTraceInWith6Arguments( "createInnerVertex(...)", ... );

  if (coarseGridVerticesEnumerator.getLevel()<3) {
	fineGridVertex.refine();
  }

  logTraceOutWith1Argument( "createInnerVertex(...)", fineGridVertex );
}

void petsc::mappings::CreateGrid::createBoundaryVertex(...) {
  logTraceInWith6Arguments( "createBoundaryVertex(...)", ... );

  if (coarseGridVerticesEnumerator.getLevel()<3) {
	fineGridVertex.refine();
  }

  logTraceOutWith1Argument( "createBoundaryVertex(...)", fineGridVertex );
}
\end{code}

\noindent
Compile it with \texttt{make -f petsc/makefile}, run it, visualise the output.

\subsection{Connecting to PETSc}

As PETSc's philosophy is straightforward and modest (they do, e.g.,
not hijack the main), the initialisation and integration are straightforward.
Please note that we stick to serial jobs in the first part of this
section.
Nevertheless, I typically find it easier to use
the MPI compiler wrapper to translate the code as I faced
issues when I tried to use PETSc without MPI\footnote{Please consult
\url{http://www.mcs.anl.gov/petsc/petsc-current/docs/installation.html\#i-dont-want-to-use-mpi}
for the usage with OpenMPI as the \texttt{LD\_LIBRARY\_PATH} has to be set
properly.}.
Luckily, most MPI implementations even are fine if you translate your code with
\texttt{mpiCC} but then skip a run through \texttt{mpirun} and instead invoke
the executable directly.

\begin{enumerate}
  \item Before we start to code with PETSc, we have to make the makefile know
  where PETSc is installed. If you use your own build environment, adopt all
  pathes accordingly. If you use the pre-generated makefile, open it and add
  your PETSc include directory to the search path. Also switch to
  \texttt{mpiCC} as discussed before.
  \begin{code}
    PROJECT_CFLAGS = -DParallel -I/opt/petsc/include
    PROJECT_LFLAGS = -L/opt/petsc/lib -lpetsc
    [...]
    CC=mpiCC
  \end{code}
  Depending on your MPI version, you
  might run into issues with the compiler's error management if you stick to
  the variants set by Peano's PDT.
  I recommend to
  remove \texttt{-Werror -pedantic-errors} from your makefile in this case.
%   \item As the present PETSc solutions use MPI (through they run serially), we
%   have to make a few modifications in the runner. These modifications are
%   detailed in Chapter \ref{section:parallelisation:mpi}, so we simply show
%   what has to be changed in \texttt{runners/Runner.cpp}. This solution should
%   work out-of-the-box.
%   For a description of its semantics, please see the aforementioned chapter.
%   \begin{code}
% #include "tarch/parallel/FCFSNodePoolStrategy.h"
% #include "peano/parallel/loadbalancing/OracleForOnePhaseWithGreedyPartitioning.h"
%
% [...]
%
% int petsc::runners::Runner::run() {
%  [...]
%  petsc::repositories::Repository* repository =
%   petsc::repositories::RepositoryFactory::getInstance().createWithSTDStackImplementation(
%    geometry,
%    tarch::la::Vector<DIMENSIONS,double>(1.0),   // domainSize,
%    tarch::la::Vector<DIMENSIONS,double>(0.0)    // computationalDomainOffset
%   );
%
%
%  // This is new because of PETSc:
%  if (tarch::parallel::Node::getInstance().isGlobalMaster()) {
%   tarch::parallel::NodePool::getInstance().setStrategy(
%    new tarch::parallel::FCFSNodePoolStrategy()
%   );
%  }
%  tarch::parallel::NodePool::getInstance().restart();
%  tarch::parallel::NodePool::getInstance().waitForAllNodesToBecomeIdle();
%  peano::parallel::loadbalancing::Oracle::getInstance().setOracle(
%   new peano::parallel::loadbalancing::OracleForOnePhaseWithGreedyPartitioning(false)
%  );
%
%  [...]
% }
%   \end{code}
  \item We have to initialise and shut down PETSc properly in
  \texttt{main.cpp}:
  \begin{code}
#include "petscsys.h"

[...]

int main(int argc, char** argv) {
  [...]
  PetscInitialize(&argc,&argv,(char*)0,(char*)0);

  int programExitCode = 0;

  if (programExitCode==0) {
    [...]
    petsc::runners::Runner runner;
    programExitCode = runner.run();
  }

  PetscFinalize();
  [...]
}
  \end{code}
  \item Finally, we should be able to run an empty project once we have
  augmented the \texttt{LD\_LIBRARY\_PATH} by PETSc.
\end{enumerate}

\begin{remark}
   According to PETSc's documentation, we could use PETSc's initialisation to
   set up MPI instead of the Peano initialisation. However, Peano's
   initialisation does not check whether MPI has been set up before.
   Thus, it makes sense to initialise Peano's MPI through
   \texttt{peano::initParallelEnvironment(\&argc,\&argv)} before you initialise
   PETSc. In return, you should invoke \texttt{PetscFinalize()} before you call
   \texttt{peano::shutdownParallelEnvironment()}.
\end{remark}

\subsubsection*{Some reiteration/remarks on the MPI side}

Even if you don't invoke your code through \texttt{mpirun}, you have to
configure MPI properly if you translate your code with \texttt{-DParallel}.
How to do so is subject of the whole Chapter \ref{section:parallelisation:mpi},
so I give only snippets here that allow users to quick-n-dirty get something
with PETSc up and running:

\begin{enumerate}
  \item Include four more headers in your \texttt{Runner.cpp}:
  \begin{code}
#include "tarch/parallel/Node.h"
#include "tarch/parallel/NodePool.h"
#include "tarch/parallel/FCFSNodePoolStrategy.h"
#include "peano/parallel/loadbalancing/OracleForOnePhaseWithGreedyPartitioning.h"
  \end{code}
  \item Add a few lines just after you have created your repository:
\begin{code}
  ...::repositories::Repository* repository =
    ...::repositories::RepositoryFactory::getInstance().create...

  // This is new because of PETSc:
  if (tarch::parallel::Node::getInstance().isGlobalMaster()) {
    tarch::parallel::NodePool::getInstance().setStrategy(
      new tarch::parallel::FCFSNodePoolStrategy()
    );
  }
  tarch::parallel::NodePool::getInstance().restart();
  tarch::parallel::NodePool::getInstance().waitForAllNodesToBecomeIdle();
  peano::parallel::loadbalancing::Oracle::getInstance().setOracle(
    new peano::parallel::loadbalancing::OracleForOnePhaseWithGreedyPartitioning(false)
  );
\end{code}
\end{enumerate}

\subsection{The algorithmic and realisation blueprint}

Once we have created the grid (and perhaps visualised it), we can start to set
up the PETSc data structures corresponding to the grid, we can call its solver,
and we can finally free everything afterwards.
We stick to the regular case for the time being.
Again, our example solves the problem
\[
-\Delta u = f
\]
on the unit square with a Finite Element 9-point stencil.

Every fine grid vertex in the grid has a right-hand side and a solution entry.
Different to previous sections, we do not store the $u$ and $f$ values within
the vertex.
Instead, we will make PETSc hold two vectors $u$ and $f$ and
make each vertex hold an index within these vectors, i.e.~each vertex knows
which entry in the two vectors is associated with the vertex.

Here is a high level sketch of what we have to do:
\begin{enumerate}
  \item Create the grid.
  \item Assign each vertex/dof a unique number.
  \item Ensure that the global vectors $u$ and $f$ do exist and also ensure that
  we have a proper matrix data structure for $A$ available. Initialise both of
  them and free them when the program terminates.
  \item Once $A$ and $f$ contain valid values, solve the equation system $Au=f$
  (note that we reuse the symbols $u$ and $f$ from the PDE though they
  represent continuous values there while they are nodal values in the
  implementation).
  \item Finally, we have to ensure that $A$ and $f$ hold the right values,
  i.e.~we have to assemble them.
\end{enumerate}

\noindent
We first ensure that the assembly process (and also the plotter, runner, \ldots)
know the respective data structures.
For convenience, I recommend to move the three important data variables
into \texttt{Vertex.h}:

\begin{code}
#include "petscvec.h"
#include "petscmat.h"

namespace petsc {
  class Vertex;

  // Forward declaration
  class VertexOperations;

  // These are the global vectors that we use to make the adapter communicate
  // with PETSc:
  extern Vec  u;
  extern Vec  f;
  extern Mat  A;
}
\end{code}

\noindent
We also have to add their definition
to \texttt{Vertex.cpp}:

\begin{code}
Vec  petsc::u;
Vec  petsc::f;
Mat  petsc::A;
\end{code}

\subsection{Enumerating the vertices and initialising vectors and the matrix}

If we tie the entries of a vector/matrix to a vertex, we have to assign the
vertices indices.
We have to couple the vertices from our grid to the PETSc unknowns.
For this, we add a field

\begin{code}
int  _unknownCounter;
\end{code}

\noindent
to a class/mapping \texttt{EnumerateUnknowns} and make each fine grid vertex
hold a unique index which is basically the corresponding entry in the $u$ and $f$ vector.

\begin{code}
#include "petsc/VertexOperations.h"

void petsc::mappings::EnumerateUnknowns::beginIteration(
  petsc::State&  solverState
) {
  logTraceInWith1Argument( "beginIteration(State)", solverState );

  _unknownCounter = 0;

  logTraceOutWith1Argument( "beginIteration(State)", solverState);
}

void petsc::mappings::EnumerateUnknowns::touchVertexFirstTime(
  ...
) {
  logTraceInWith6Arguments( "touchVertexFirstTime(...)", ... );

  if (
    fineGridVertex.getRefinementControl()==Vertex::Records::Unrefined
    &&
    fineGridVertex.isInside() // it is not a boundary point
  ) {
    VertexOperations::writeIndex(fineGridVertex,_unknownCounter);
    _unknownCounter++;
  }
  else {
    VertexOperations::writeIndex(fineGridVertex,-1);
  }

  logTraceOutWith1Argument( "touchVertexFirstTime(...)", fineGridVertex );
}
\end{code}

\noindent
Once we know how many unknowns we are handling, we can create our two required
vectors and we can initialise our matrix properly:

\begin{code}
#include "petsc/VertexOperations.h"

[...]

void petsc::mappings::EnumerateUnknowns::endIteration(...) {
 logTraceInWith1Argument( "endIteration(State)", solverState );

 const PetscInt numberOfUnknowns = _unknownCounter;

 PetscErrorCode errorU = VecCreate(tarch::parallel::Node::getInstance().getCommunicator(), &u);
 PetscErrorCode errorF = VecCreate(tarch::parallel::Node::getInstance().getCommunicator(), &f);

 if (errorU!=0) {
   PetscError( tarch::parallel::Node::getInstance().getCommunicator(),
     __LINE__,  __FUNCT__,  __FILE__, errorU,  PETSC_ERROR_INITIAL,
     "creating global solution vector failed" );
 }
 if (errorF!=0) {
   PetscError( tarch::parallel::Node::getInstance().getCommunicator(),
     __LINE__,  __FUNCT__,  __FILE__, errorF,  PETSC_ERROR_INITIAL,
     "creating global rhs vector failed" );
 }

 VecSetSizes(u, PETSC_DECIDE,numberOfUnknowns);
 VecSetSizes(f, PETSC_DECIDE,numberOfUnknowns);
 VecSetType(u, VECMPI);
 VecSetType(f, VECMPI);
 PetscObjectSetName((PetscObject)u, "Solution");
 PetscObjectSetName((PetscObject)f, "Right-hand side");

 PetscErrorCode errorA = MatCreateSeqAIJ(
   tarch::parallel::Node::getInstance().getCommunicator(),
   numberOfUnknowns,numberOfUnknowns,
   THREE_POWER_D, NULL,
   &A
 );
 if (errorA!=0) {
   PetscError( tarch::parallel::Node::getInstance().getCommunicator(),
     __LINE__,  __FUNCT__,  __FILE__, errorA,  PETSC_ERROR_INITIAL,
     "creating system matrix failed" );
 }

 logTraceOutWith1Argument( "endIteration(State)", solverState);
}
\end{code}

\begin{remark}
 The above snippet where we predefine the number of nonzeros to be $3^d$ per
 row does work only on a regular grid. Adaptive grids might require
 modifications.
\end{remark}

\begin{remark}
 One might be tempted to use the repository's \texttt{getNumberOfInnerVertices}
 routines or similar routines to initialise PETSc.
 This might work, but there are pitfalls: In a distributed memory environment
 these counters might count replicated vertices along domain boundaries twice,
 and the AMR framework cannot know which boundary vertices are real unknowns
 (Neumann, e.g.) and which do not carry unknowns.
\end{remark}

\subsection{Free the data}

Before we continue, we make the runner
destroy all data structures properly:

\begin{code}
int petsc::runners::Runner::runAsMaster(petsc::repositories::Repository& repository) {
  ...

  // Clean up PETSc data structures
  VecDestroy(&u);
  VecDestroy(&f);
  MatDestroy(&A);

  repository.logIterationStatistics( true );
  repository.terminate();

  return 0;
}
\end{code}

\subsection{Solve the equation system}

We finally have to solve the actual equation system in the runner. For this, we
plug into the runner snippet above and furthermore include
\texttt{\#include "petscksp.h"}. We use PETSc's default which is a GMRES
and add a Jacobi preconditioner.

\begin{code}
repository.switchToCreateGrid(); repository.iterate();
repository.switchToEnumerate(); repository.iterate();
repository.switchToAssemble(); repository.iterate();

KSP krylovSolver;
PC  preconditioner;

KSPCreate(tarch::parallel::Node::getInstance().getCommunicator(),&krylovSolver);
KSPSetOperators(krylovSolver,A,A); // use matrix from Vector.h

// connect/get preconditioner
KSPGetPC(krylovSolver,&preconditioner);

// set Jacobi as preconditioner
PCSetType(preconditioner,PCBJACOBI);

KSPSolve(krylovSolver,f,u);
KSPDestroy(&krylovSolver);

repository.switchToPlot(); repository.iterate();
\end{code}

\noindent
This is not a particular sophisticated code, but it does the job for the time
being.

\begin{remark}
If your application requires $k$ command line arguments, it makes sense to check
that there are at least $k$ arguments and then to let PETSc select a solver
through the command line:
 \begin{code}
KSPCreate(tarch::parallel::Node::getInstance().getCommunicator(),&krylovSolver);
KSPSetFromOptions(krylovSolver);
KSPType krylovSolverType;
KSPGetType(krylovSolver,&krylovSolverType);
logInfo( "run(...)", "use solver " << std::string(krylovSolverType) );
KSPSetOperators(krylovSolver,A,A); // use matrix from Vector.h
 \end{code}
PETSc Krylov solvers are discussed in the PETSc manual in Chapter 4
(\url{http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf#chapter.4}).
I furthermore typically add a
\begin{code}
int iterations;
KSPGetIterationNumber(krylovSolver, &iterations);
logInfo( "run(...)", "solver terminated after " << iterations << " iteration(s)" );
\end{code}
statements to track the convergence behaviour.
\end{remark}

\subsection{Explicit assembly 2/2: Assembling the right-hand side and the
matrix}

We finally assemble our systems. First, we have to tell
PETSc when we start and when we have finished the assembly.
This matches Peano's idea of a \texttt{beginIteration} and
\texttt{endIteration} event:

\begin{code}
void petsc::mappings::Assemble::beginIteration(petsc::State&  solverState) {
  VecAssemblyBegin( u );
  VecAssemblyBegin( f );
  MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
}

void petsc::mappings::Assemble::endIteration(petsc::State&  solverState) {
  VecAssemblyEnd( u );
  VecAssemblyEnd( f );
  MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
}
\end{code}

\noindent
While the binning of the right-hand side into the matrix is straightforward
(note how we grab the index from the vertex object and then use this index to
write the vector data into the right place)
\begin{code}
void petsc::mappings::Assemble::touchVertexFirstTime(...) {
  if (fineGridVertex.isUnknown()) {
    fineGridVertex.setRhs(
      ... // take 1.0 for the time being. In the present Finite Differences mindset, this means the right-hand
          // side us 1 everywhere. Obviously, you might later want to compute more complicated things.
    );
  }
}

bool petsc::Vertex::isUnknown() const {
  return _vertexData.getIndex()>=0;
}

void petsc::Vertex::setRhs(double rhs) {
  PetscInt     indices[] = {_vertexData.getIndex()};
  PetscScalar  values[]  = {rhs};
  VecSetValues(f,1,indices,values, INSERT_VALUES);
}
\end{code}

\noindent
we have to break down the assembly of the matrix element-wisely:

\begin{code}
#include "tarch/la/Matrix.h"

void petsc::mappings::Assemble::enterCell(...) {
 tarch::la::Matrix<4,4,double> localA;
 localA =  1.0,  -0.5, -0.5,  0.0,
          -0.5,   1.0,  0.0, -0.5,
          -0.5,   0.0,  1.0, -0.5,
           0.0,  -0.5, -0.5,  1.0;

 for(int ii=0; ii<4; ii++)
 for(int jj=0; jj<4; jj++) {
  if (
   fineGridVertices[fineGridVerticesEnumerator(ii)].isUnknown()
   and
   fineGridVertices[fineGridVerticesEnumerator(jj)].isUnknown()
  ) {
   PetscInt row = VertexOperations::readIndex(
     fineGridVertices[fineGridVerticesEnumerator(ii)] );
   PetscInt col = VertexOperations::readIndex(
     fineGridVertices[fineGridVerticesEnumerator(jj)] );
   MatSetValue(A,row,col,localA(ii,jj),ADD_VALUES);
  }
 }
}
\end{code}

\noindent
Once you have filled the matrix with entries, you might wanna visualise the
content in the runner with
\begin{code}
  MatView(A,PETSC_VIEWER_STDOUT_WORLD);
  MatView(A,PETSC_VIEWER_DRAW_WORLD);
  std::cin.get();
\end{code}

\pagebreak

\begin{center}
  \includegraphics[width=0.4\textwidth]{43_dg/matrix-pattern.png}
  \\
  {
  \footnotesize
  It is important to remember that Peano uses a space-filling curve to traverse
  the grid but can switch to regular subgrids if it deems to be suitable. As a
  consequence, the sparsity pattern even for regular grids might not equal the
  expected $k$-diagonal matrix pattern.
  }
\end{center}

\begin{remark}
  If you know the stencils associated to your vertices, you might prefer to use
  \texttt{matrixfree::stencil::getElementWiseAssemblyMatrix} from Peano's
  \texttt{matrixfree} toolbox which extracts the element-wise stiffness matrix
  from the stencils.
\end{remark}

\subsection{Retrieve properties from the PETSc data structures}

There are many technical ways to couple the vertex to PETSc and to retrieve
data from the explicit assembly.
We do it via the vertices:

\begin{code}
double petsc::Vertex::getU() const {
  double result = 0.0;
  if (_vertexData.getIndex()>=0) {
    PetscInt     indices[] = {_vertexData.getIndex()};
    PetscScalar  values[]  = {0.0};

    VecGetValues(u,1,indices,values);

    result = values[0];
  }
  return result;
}
\end{code}

%
% \noindent
% Needless to say, this is a 2d example. 3d requires more indices. You might want
% to have a look into the Peano utilities for dimension-generic routines. And it
% is also obvious that this is not a very elegant and fast solution as we write
% into the matrix entry by entry rather than in batches.
%
% Once the matrix is assembled, it makes sense to add statements alike
% \begin{code}
%   KSPCreate(tarch::parallel::Node::getInstance().getCommunicator(),&krylovSolver);
%   KSPSetOperators(krylovSolver,A,A); // use matrix from Vector.h
% \end{code}
%
% \noindent
% into the runner to see the matrix pattern, e.g.
%
%
%
% \subsection{Some remarks on discontinuous Galerkin}
%
%
% \begin{figure}
%   \begin{center}
%     \includegraphics[width=0.6\textwidth]{43_dg/dg-data-layout.pdf}
%   \end{center}
%   \caption{One possible data layout for DG with PETSc. All indices are stored
%   within vertices.}
%   \label{figure:43_petsc:data-layout}
% \end{figure}
%
% There are various ways to implement discontinuous Galerkin in Peano.
% We sketch a vertex-based variant  at
% hands of bi-quadratic shape functions.
% For such an approach, we have to store nine unknown per cell.
% However, we propose not to store indices within a cell but within vertices.
% This is advantageous as we have, without additional effort, only vertices within
% a cell at hand.
% We do not know neighbour cells.
%
% So we propose to hold nine integer indices within each individual vertex.
% Basically, every unknown is stored within its closest neighbouring vertex
% (Figure \ref{figure:43_petsc:data-layout}).
% For those where multiple options do exist, we store it to the left bottom.
% The corresponding vertex data structure reads as follows:
% \begin{code}
% Packed-Type: short int;
%
% class petsc::records::Vertex {
%   parallelise persistent int index[9];
% };
% \end{code}
%
%
%
% \subsection{Work in progress \ldots}
%
% I am planning to start some programming with PETSc plus Peano plus MPI soon and
% I also plan to dive into PETSc's matrix-free routines. If anybody reads this
% comment and has suggestions, feel free to contact me.
%
% \ldots to be continued
%
%
%
% \subsection*{Further reading}
%
% \begin{itemize}
%   \item Weinzierl, Marion and Weinzierl, Tobias: {\em Quasi-matrix-free hybrid
%   multigrid on dynamically adaptive Cartesian grids}, arXiv:1607.00648
% \end{itemize}


\documentclass[11pt, a4paper]{report}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
% \usepackage{polski}
\usepackage{fancyhdr}
\usepackage{lastpage}
\usepackage{relsize}
\usepackage{listings}
\usepackage{mathtools}
\usepackage{array}
\usepackage{multirow}
\usepackage{tabularx}
\usepackage{placeins}
\usepackage{paralist}
\usepackage{xcolor}
\usepackage{amsthm}
\usepackage{float}
\usepackage{hyperref}
\usepackage[normalem]{ulem}
\usepackage{blindtext}

\usepackage[
    bindingoffset=-15pt,
	voffset=0pt,
	hoffset=10pt,
	textwidth=500pt,
    textheight=650pt,
	marginparwidth=20pt,
	footskip=60pt
]{geometry}
\pagestyle{fancy}

\headheight=28pt

\renewcommand{\footrulewidth}{0.4pt}

\begin{document}

\lhead{Mateusz Pach}
\chead{\textbf{King Phase Protocol}}
\rhead{\today{}}
\cfoot{Page \thepage/\pageref{LastPage}}

\theoremstyle{plain}
\newtheorem{thm}{Theorem} % reset theorem numbering for each chapter
\newtheorem{lem}{Lemma} % definition numbers are dependent on theorem numbers
\newtheorem{exmp}{Example}% same for example numbers
\theoremstyle{definition}
\newtheorem{defn}{Definition} % definition numbers are dependent on theorem numbers

\subsection*{Introduction}
Let us remind the consensus problem.
\begin{defn}[The consensus problem]
We are given processors of which $t$ may be faulty. The processors are given initial values and after exchanging messages all correct ones must agree on a single value choosen from among the initial values. Formally, three conditions must be met.
\begin{itemize}
    \item \textbf{Agreement} All correct processes must agree on the same (single) value.
    \item \textbf{Validity} If all the correct processes have the same initial value, then the agreed upon value by all the correct processes must be that same value.
    \item \textbf{Termination} Each correct process must eventually decide on a value.
\end{itemize}
\end{defn}

One can measure the quality of protocols solving the consensus problem using the following parameters: total number of processors $n$, number of rounds of message exchange $r$ and maximal message length $m$. Their optima are known to be respectively $3t+1, t+1$ and $1$.

The King Phase protocol, proposed by Berman, Garay and Perry, is one of protocols solving the consensus problem. It's also asymptotically optimal in all the three parameters mentioned earlier. Namely, their values for the protocol are respectively $3t+1$, $3t+3$ and $2$.

\subsection*{Protocol}
The idea of the protocol is as follows. It runs $t+1$ phases, each consisting of three exchange rounds. Each processor has a local variable $V$ which can obtain values from $\{0,1,2\}$ and two arrays $C$ and $D$ counting messages with different values received from the other processors in the first and the second exchange round respectively. Variables $V$ are set to the initial values at the beginning. In each phase a unique processor becomes a king of the phase (the phase number determines the id of the king). Let us proof the following lemmas.
\begin{lem}
If all correct processors have the same values V equal to 0 or 1 before the phase, then it holds also after the phase.
\end{lem}
\begin{proof}
If all correct processors have the same values $V$ equal to $v\in\{0,1\}$ before the phase, then they all broadcast $v$ in the first exchange round and for each of them $C(v) \geq n-t$ holds. For each $k\neq v$ we know that $C(k) < n-t$, as otherwise $n\geq (n-t) + (n-t) - t = 2n-3t$, but we know it's impossible because $n>3t$. It means the processors have the same values $V$ equal to $v$ before the second exchange round. It implies they all broadcast $v$ in the second exchange round and for each of them $D(k)>t$ holds only for $k=v$, so their values $V$ remain the same before the third exchange round. The third exchange round does not alter values $V$, because $D(v)\geq n-t$ just like $C(v)\geq n-t$. It proves none of exchange rounds change values $V$ in correct processors, hence the claim.
\end{proof}
\begin{lem}
If a king is a correct processor, then after its round all correct processors have the same values $V$ equal to 0 or 1.
\end{lem}
\begin{proof}
The first exchange round set all the correct processors' $V$ variables to merely zeroes and twos or to merely ones and twos, because if there was a pair of correct processors having $V=1$ and $V=2$, then $n\ge2(n-t)-t$, but $n\geq 3t+1$, which would mean $3t\ge n \ge 3t+1$, which is a contradiction. If the king is correct, then he broadcast his $V$. Two cases may occur now. Either all the correct processors set $V$ to the same value from $\{0,1\}$ in the third round or there is some for which $V\neq2$ and $D(V)\ge n-t$. The later case implies that for each of the correct processor having $V\neq2$ follows $D(V)>t$ (because there are $t$ faulty processors and $n\geq 3t+1$), meaning that the correct processors have $V$ already set to the same value from $\{0,1\}$ after the second phase, which completes the proof.
\end{proof}
Lemma 1. does not only assure the validity, but also implies it's enough to reach the agreement once, in any phase. At least one of $t+1$ kings must be correct, so due to Lemma 2. and the previous sentence the correct processors must reach agreement. Finite number of operations assures the termination. Thus, we derive the valid protocol for the consensus problem.

Let's discuss a complexity of the protocol. Minimal number of processors is $3t+1$, as it is a sufficient bound needed for correctness and we know from the previous works that it cannot be lower. There are in total $3t+3$ message exchange rounds, 3 per each of $t+1$ phases. Finally, there are only 3 types of messages which can be encoded using 2 bits.

\subsection*{References}
\begin{itemize}
    \item P. Berman, J. A. Garay and K. J. Perry, \emph{Towards optimal distributed consensus,} 30th Annual Symposium on Foundations of Computer Science, 1989, pp. 410-415, doi: 10.1109/SFCS.1989.63511.
    \item A. D. Kshemkalyani and M. Singhal, \emph{Distributed Computing Principles, Algorithms, and Systems,} 2011, pp. 510-514, ISBN: 9780521189842.
\end{itemize}

\end{document}


% \documentclass[draft,11pt]{article}
%\documentclass[11pt]{article}
%\input{../defs}
%\input{../layout-defs}
%\usepackage{amsmath}
%\usepackage{amssymb}
%\usepackage{amsthm}
%\usepackage{graphicx}
%\usepackage{float}
%\usepackage[ruled,vlined]{algorithm2e}
%\SetKwBlock{Repeat}{repeat}{}
%%% for this lecture
%\newcommand{\gap}{\text{gap}}
% \newtheorem{theorem}{Definition}
%\interfootnotelinepenalty=10000
% \newtheorem{lemma}{Lemma}

%\usepackage{tikz}
%\usetikzlibrary{arrows}

\chapter{Classical
  Algorithms for Maximum Flow II}
\label{cha:maxflow2}

%\begin{document}

\sloppy
%\lecture{11 --- Wednesday, May 6th}
%{Spring 2020}{Rasmus Kyng, Scribe: Timon Knigge}{Classical
  %Algorithms for Maximum Flow II}

\section{Overview}
In this chapter we continue looking at classical max flow algorithms. We derive Dinic's algorithm which
(unlike Ford-Fulkerson) converges in a polynomial number of iterations. We will also show faster
convergence on unit capacity graphs.
The setting is the same as last chapter: we have a directed graph $G = (V, E, \cc)$ with
positive capacities
on the edges. Our goal is to solve the maximum flow problem on this graph for a given source $s$ and sink
$t \neq s$:

\begin{align}
  \begin{aligned}
    \max_{F>0} \;F\qquad\text{s.t.}\quad&\BB\ff = F\bb_{s,t} \\
    &\veczero\leq \ff\leq \cc
  \end{aligned}
\end{align}

\section{Blocking Flows}
Let $\ff$ be any feasible flow in $G$ and let $G_{\ff}$ be its residual graph.
Observe that we can partition the
vertices of $G_{\ff}$ into `layers': first $s$ itself, then all vertices reachable from $s$ in one hop, then all
vertices reachable in two hops, etc. For each vertex $v\in V$ define its \textbf{level} in $G_{\ff}$ as the
length of the shortest path in $G_{\ff}$ from $s$ to $v$, denoted by $\ell_{G_{\ff}}(v)$ (or just $\ell(v)$ if the
graph is clear from context). An edge $(u, v) \in E$ can only take you up `one level': if
$\ell(v) \geq 2+\ell(u)$ this would imply we can find a shorter $s$-$v$ path by appending $(u, v)$ to the
shortest $s$-$u$ path. However, edges can `drop down' multiple levels (or be contained in the same level).

A key strategy in Dinic's algorithm will be to focus on `short' augmenting paths. We can use the levels we
defined above to isolate a subgraph of $G_{\ff}$ containing all information we need to find shortest paths.
\begin{definition}
  Call an edge $(u, v)$ \textbf{admissible} if $\ell(u)+1 = \ell(v)$. Let $L$ be the set of
  admissible edges in $G_{\ff}$ and define the \textbf{level graph} of $G_{\ff}$ to be the subgraph
  induced by these edges.
\end{definition}

\begin{definition}
  Define a \textbf{blocking flow} in a residual graph $G_{\ff}$ to be a flow $\hat{\ff}$ feasible in $G_{\ff}$,
  such that $\hat{\ff}$ only uses admissible edges. Furthermore we require that for any $s$-$t$ path in
  the level graph of $G_{\ff}$, $\hat{\ff}$ saturates at least one edge on this path.
\end{definition}

The last condition makes a blocking flow `blocking': it blocks any shortest augmenting path in $G_{\ff}$ by
saturating one of its edges. Note however that a blocking flow is not necessarily a maximum flow in the
level graph:

\begin{figure}
  \centering
  \begin{tikzpicture}
    \tikzset{vertex/.style = {shape=circle,draw,minimum size=1.5em}}
    \tikzset{edge/.style = {->,> = latex'}}

    \node[vertex] (a) at (0, 0) {$s$};
    \node[vertex] (b) at (2, -1) {};
    \node[vertex] (c) at (2, 1) {};
    \node[vertex] (d) at (4, 0) {};
    \node[vertex] (e) at (4, 2) {};
    \node[vertex] (f) at (6, 1) {$t$};

    \draw[edge] (a) -- (b) node [midway, below] {$0/1$};
    \draw[edge] (b) -- (d) node [midway, below] {$0/1$};
    \draw[edge] (c) -- (e) node [midway, above] {$0/1$};
    \draw[edge] (e) -- (f) node [midway, above] {$0/1$};

    \draw[edge, line width=2pt] (a) -- (c) node [midway, above] {$1/1$};
    \draw[edge, line width=2pt] (c) -- (d) node [midway, above] {$1/1$};
    \draw[edge, line width=2pt] (d) -- (f) node [midway, below] {$1/1$};

  \end{tikzpicture}
  \caption{A blocking flow is not a maximum flow.}
\end{figure}

\section{Dinic's Algorithm}
We can now formulate Dinic\footnote{Sometimes also transliterated as Dinitz.}'s algorithm: start with
$\ff = \veczero$, and then repeatedly add to $\ff$ a blocking flow $\hat{\ff}$ in $G_{\ff}$, until no more $s$-$t$ paths
exist in $G_{\ff}$.

Note that by doing a path decomposition on $\hat{\ff}$ and adding these paths to $\ff$ one by one, we see that
our algorithm is a `special case' of Ford-Fulkerson (with a particular augmenting path strategy) and hence
inherits all the behaviors/bounds we proved in the last chapter.

\begin{lemma}
  Let $\ff$ be a feasible flow, $\hat{\ff}$ a blocking flow in $G_{\ff}$ and define $\ff' = \ff+\hat{\ff}$ (think
  of $\ff$ and $\ff'$ to be the flows at some steps $k$ and $k+1$ in the algorithm). Then
  $\ell_{G_{\ff'}}(t) \geq \ell_{G_{\ff}}(t) + 1$.
\end{lemma}
\begin{proof}
  Let $L, L'$ be the edge sets of the level graphs of $G_{\ff}, G_{\ff'}$
  respectively.
  % \todo{I didn't really
  %   get the proof in the class for this part very well so I wrote my own. Feel
  %   free to change, I hope it's okay though.}
  We would now like to show that $d_{L'}(s, t) \geq 1 + d_L(s, t)$. Let's first assume that in fact
  $d_{L'}(s, t) < d_L(s, t)$. Now take a shortest $s$-$t$ path in the level graph of
  $G_{\ff'}$, say $s=v_0, v_1, \dots, v_{d_{L'}(s, t)} =  t$.
  Let $v_j$ be the first vertex along the path such that
  $d_{L'}(s, v_j) < d_L(s, v_j)$.
  As $d_{L'}(s, t) < d_L(s, t)$, such a $v_j$ must exist.

  We'd like to understand the edges in the level
  graph $L'$.
  Let $E(G_{\ff})$ denote the edges of the $G_{\ff}$,
  and let $\operatorname{rev}(L)$ denote the set of
  reversed edges of $L$.
  The level graph edges of $G_{\ff'}$ must satisfy
  $L' \subseteq L \union
  \operatorname{rev}(L) \union (E(G_{\ff}) \setminus
  L)$.

  In our $s$-$t$ path in $L'$, the vertex $v_j$ is reached by an edge from $v_{j-1}$,
  and the level of $v_{j-1}$ in $G_{\ff}$ and $G_{\ff'}$
  are the same, so the edge $(v_{j-1},v_j) \in L$ skips
  at least one level forward  in $G_{\ff}$.
  But, edges in $L$ do not skip a level in
  $G_{\ff}$, and edges in
  $\operatorname{rev}(L)$ or
  $E(G_{\ff}) \setminus L$ do not move from a lower level to higher
  level in $G_{\ff}$.
  So this edge cannot exist and we have reached a contradiction.
  % Write down the levels of the vertices in
  % $G_{\ff}$ along the path,
  % then
  % there are more levels in $G_{\ff}$ than
  % there are vertices in the path, so there must a be
  % level in $G$
  % so by the pigeonhole principle there must be a level in $G_{\ff}$ that
  % the path jumps over, i.e. there is some edge that jumps up more than one
  % level in $G_{\ff}$.
  % However, we claim this is impossible. Such an edge cannot exist in $G_{\ff}$ by definition of the level
  % graph. The only other edges in $G_{\ff'}$ are reversals of edges used in the blocking flow $\hat{\ff}$,
  % but by construction this flow only uses admissible edges in $G_{\ff}$ (i.e. edges that `step up' one
  % level). Hence the edge $(v_i, v_{i+1})$ cannot exist and we have arrived at a contradiction.

  Now suppose that $d_{L'}(s, t) = d_L(s, t)$. This means there is an $s$-$t$ path in $L'$ using
  edges in $L$ -- but such a path must contain an edge saturated by
  the blocking flow $\hat{\ff}$.

  (Note: the reason this path must use only edges in $L$ is similar to
  the previous case: the other possible types of edges do not move
  to higher levels in $G_{\ff}$ at all, making the path too long if we use any of them.)

  So we must have $d_{L'}(s, t) \geq 1 + d_L(s, t)$.
\end{proof}
An immediate corollary of this lemma is the convergence of Dinic's algorithm:
\begin{theorem}
  Dinic's algorithm terminates in $O(n)$ iterations.
\end{theorem}
\begin{proof}
  By the previous lemma, the level of $t$ increases by at least one each iteration. A shortest path
  can only contain each vertex once (otherwise it would contain a cycle) so the level of any vertex
  is never more than $n$.
\end{proof}

For graphs with unit capacities ($c = 1$) we can prove even better bounds on the number of iterations.
\begin{theorem}
  On unit capacity graphs, Dinic's algorithm terminates in
  $$O\left(\min\{m^{1/2}, n^{2/3}\}\right)$$ iterations.
\end{theorem}
\begin{proof}
  We prove the two bounds separately.
  \begin{enumerate}
  \item Suppose we run Dinic's algorithm for $k$ iterations, obtaining a flow $\ff$ that is
    feasible but not necessarily yet optimal. Let $\tilde{\ff}$ be an optimal flow in
    $G_{\ff}$ (i.e. $\ff+\tilde{\ff}$ is a maximum flow for the whole graph),
    and consider a path decomposition of $\tilde{\ff}$. Since after $k$ iterations
    any $s$-$t$ path has length $k$ or more, we use up a total capacity of at least
    $\text{val}(\tilde{\ff})k$ across all edges. But the edges in $G_{\ff}$ are either edges
    from the original graph $G$ or their reversals (but never both) meaning the total
    capacity of $G_{\ff}$ is at most $m$, hence $\text{val}(\tilde{\ff}) \leq m/k$.

    Recalling our earlier observation that our algorithm is a special case of
    Ford-Fulkerson, this implies that our algorithm will terminate after at most
    another $m/k$ iterations. Hence the number of iterations is bounded by $k+m/k$ for
    any $k>0$. Substituting $k=\sqrt{m}$ gives the first desired bound.
  \item Suppose again that we run Dinic's algorithm for $k$ iterations obtaining a flow $\ff$.
    The level graph of $G_{\ff}$ partitions the vertices into sets
    $D_i = \{u\mid\ell(u) = i\}$ for $i\geq 0$.
    As shown before, the sink $t$ must be at a level of at
    least $k$, meaning we have at least this many non-empty levels starting from
    $D_0 = \{s\}$. To simplify, discard all vertices and levels beyond $D_{\ell(t)}$.

    Now, consider choosing a level $I$ uniformly at random from
    $\setof{1,\ldots,k}$.
    Since there are at most $n-1$ vertices in total across these levels,
    $\E{\abs{D_I}} \leq (n-1)/k$, and by Markov's inequality
    \[
      \Pr[ \abs{D_I} \geq 2n/k ] \leq
      \frac{(n-1)/k}{2n/k} < \frac{1}{2}
      .
    \]
    % \todo{I found this a bit easier to follow than Markov, feel free to change (again).}
    % divide the levels into two groups: those with $|D_i|\leq 2n/k$ and those with
    % $|D_i| > 2n/k$. If the latter group contained at least $k/2$ levels then together
    % they would contain strictly more than $(2n/k)\cdot(k/2) = n$ vertices, which is a
    % contradiction (note the $D_i$ are disjoint), so more than $k-k/2 = k/2$ levels
    % satisfy the first condition,

    Thus strictly more than $k/2$ of the levels $i \in \setof{1,\ldots,k}$
    have $\abs{D_i} < 2n/k$ and so there must be two adjacent levels $j$, $j+1$
    for which this upper bound on the size holds.
    There can be at most
    $|D_j|\cdot|D_{j+1}| \leq 4n^2/k^2$ edges between these levels, and by the min-cost
    max-flow theorem we saw in the previous chapter, this is an upper bound on the flow
    in $G_{\ff}$, and hence on the number of iterations still needed for our algorithm to
    terminate.

    This means the number of iterations is bounded by $k + 4n^2/k^2$ for any $k>0$,
    which is $O(n^{2/3})$ at $k=2n^{2/3}$.
  \end{enumerate}
\end{proof}

\section{Finding Blocking Flows}
 \label{sec:findBlockingFlows}
What has been missing from our discussion so far is the process of actually finding a blocking flow.
We can achieve this using repeated depth-first search. We repeatedly do a search in the level graph (so
only using edges $L$) for $s$-$t$ paths and augment these. We erase edges whose subtrees have been
exhausted and do not contain any augmenting paths to $t$. Pseudocode for the algorithm is given below.

\begin{algorithm}[H]
  \SetAlgoLined
  $\ff \leftarrow \veczero$\;
  $H \leftarrow L$\;
  \Repeat{
    $P\leftarrow\text{\textsc{Dfs}}(s, H, t)$\;
    \eIf{$P\neq\emptyset$}{
      Let $\hat{\ff}$ be a flow that saturates path $P$.\\
      $\ff\leftarrow \ff+\hat{\ff}$\;
      Remove from $H$ all edges saturated by $\hat{\ff}$.
    }{
      \Return{\ff}\;
    }
  }
  \caption{\textsc{FindBlockingFlow}}
\end{algorithm}

\begin{algorithm}[H]
  \SetAlgoLined
  \If{$u=t$}{\Return{\textnormal{the path $P$ on the dfs-stack.}}}
  \For{$(u,v)\in H$}{
    $P\leftarrow\text{\textsc{Dfs}}(v, H, t)$\;
    \eIf{$P\neq\emptyset$}{
      \Return{$P$}\;
    }{
      Erase $(u, v)$ from $H$.
    }
  }
  \Return{$\emptyset$}\;
  \caption{\textsc{Dfs}(u, H, t)}
\end{algorithm}

\begin{lemma}
  For general graphs, \textsc{FindBlockingFlow} returns a blocking flow in $O(nm)$ time.
  Hence Dinic's algorithm runs in $O(n^2 m)$ time on general capacity graphs.
\end{lemma}
\begin{proof}
  % We sketch an approach to amortized analysis of the running time.
  First, consider the amount of work spent pushing edges onto the
  stack which eventually results in augmentation along an $s$-$t$ path
  consisting of those edges (i.e.\ adding flow along that path).
  Since each augmenting path saturates at least one edge,  we do at most $m$ augmentations. Each
  path has length at most $n$.
  Thus the total amount of work pushing these edges to the stack, and removing them
  from the stack upon augmentation, and deleting saturated edges, can
  be bounded by $O(mn)$.
  Now consider the work spent pushing edges onto the stack which are
  later deleted because we ``retreat'' after not finding an $s$-$t$.
  An edge can only be pushed onto the stack once in this way, since it
  is then deleted. So the total amount spent pushing edges to the
  stack and deleting them this way is $O(m)$.
   % First, consider the amount of work spent pushing edges onto the
  % Since each augmenting path saturates at least one edge, there are at most $m$ augmenting paths. Each
  % path has length at most $n$.
  % Hence the calls to \textsc{Dfs} that result in augmenting paths account for
  % $O(mn)$ work jointly. The calls to \textsc{Dfs} that do not result in an augmenting path together account
  % for $O(m)$ work because each such call erases an edge, meaning across the whole algorithm run there are at
  % most $m$ such calls.
\end{proof}
\begin{lemma}
  For unit capacity graphs, \textsc{FindBlockingFlow} returns a blocking flow in $O(m)$ time. Hence Dinic's
  algorithm runs in $O\left(\min\{m^{3/2}, mn^{2/3}\}\right)$ time on unit capacity graphs.
\end{lemma}
\begin{proof}
  When our depth-first search traverses some edge $(u, v)$, one of two things will happen: either we find no
  augmenting path in the subtree of $v$, leading to the erasure of this edge, or we find an augmenting path
  which will necessarily saturate $(u, v)$, again leading to its erasure. This means each edge will be
  traversed at most once by the depth-first search.
\end{proof}

Another interesting bound on the runtime, which we will not prove here, is that Dinic's algorithm will run in
$O\left(\abs{E}\sqrt{\abs{V}}\right)$ time on bipartite matching graphs.

\section{Minimum Cut as a Linear Program}
Finally we show that minimum cut may be formulated as a linear program. For a subset $S \subseteq V$ write
$c_G(S)$ for the sum of $\cc(e)$ over all edges $e = (u, v)$ that cross the cut, i.e.\ $u \in S, v\not\in S$.
Note that `reverse' edges are not counted in this cut. The minimum cut problem asks us to find some $S\subseteq V$
such that $s\in S$ and $t\not\in S$ such that $c_G(S)$ is minimal.
\begin{align}
  \begin{aligned}
    \min_{S\subseteq V} \quad &c_G(S) \\
    s.t.\quad & s \in S \\
    &t\not\in S\\
  \end{aligned}
  \label{prog0}
\end{align}

We claim this is equivalent to the following
minimization problem:
\begin{align}
  \begin{aligned}
    \min_{\xx \in \R^{V}} \quad &\sum_{e\in E} \cc(e) \max\left\{\bb_e^\trp \xx, 0\right\} \\
    s.t. \quad & \xx(s) = 0 \\
    & \xx(t) = 1 \\
    &\veczero\leq \xx\leq \vecone \\
  \end{aligned}
  \label{prog1}
\end{align}
Recall $\bb_e^\trp \xx = \xx(v) - \xx(u)$ for $e = (u, v)$.
We can rewrite this as a proper linear program by
introducing extra variables for the maximum:
\begin{align}
  \begin{aligned}
    \min_{\xx \in \R^{V}, \uu \in \R^{E}} \quad & \cc^\trp \uu \\
    s.t. \quad & \bb_{s,t}^\trp\xx = 1 \\
    &\veczero\leq \xx\leq \vecone \\
    &\uu \geq \veczero \\
    &\uu \geq \BB^\trp \xx \\
  \end{aligned}
  \label{prog2}
\end{align}
And, in fact, we'll also see that we don't need the constraint
$\veczero\leq \xx\leq \vecone$, leading to the following simpler
program:
\begin{align}
  \begin{aligned}
    \min_{\xx \in \R^{V}, \uu \in \R^{E}} \quad & \cc^\trp \uu \\
    s.t. \quad & \bb_{s,t}^\trp\xx = 1 \\
    &\uu \geq \veczero \\
    &\uu \geq \BB^\trp \xx \\
  \end{aligned}
  \label{prog3}
\end{align}

\begin{lemma}
  Programs \eqref{prog0}, \eqref{prog2}, and \eqref{prog3} have equal optimal values.
\end{lemma}
\begin{proof}
  We start by considering equality between optimal values of Program
  \eqref{prog0} and Program \eqref{prog2}
  Let $S$ be an optimal solution to Program~\eqref{prog0} and take $\xx = \vecone_{V\setminus S}$. Then $\xx$ is a
  feasible solution to Program~\eqref{prog2} with cost equal to $c_G(S)$.

  Conversely, suppose $\xx$ is an optimal solution to Program~\eqref{prog2}. Let $\alpha$ be uniform in $[0,1]$ and define
  $S = \{v\in V\mid\xx(v) \leq \alpha\}$. This is called a threshold cut.
  We can verify that
  \[ \Pr\left[\text{$e$ is cut by $S$}\right] = \max\{\bb_e^\trp \xx, 0\}. \]
  and hence $\mathbb{E}_t [c_G(S)]$ is exactly the optimization function of \ref{prog1} (and hence
  \ref{prog2}).
  Since at least one outcome must do as well as the average, there is a subset $S\subseteq V$ achieving this value
  (or less).

  We can use the same threshold cut to show that we can round a
  solution to Program~\eqref{prog3} to an equal or smaller value cut
  feasible for Program~\eqref{prog0}. The only difference is that in
  this case, we get
  \[
    \Pr\left[\text{$e$ is cut by $S$}\right] \leq \max\{\bb_e^\trp \xx, 0\}
  \]
  which is still sufficient.
\end{proof}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "agao21_script"
%%% End:

%\end{document}


\section{Race}
A character's \textbf{race} determines quite a lot of things about a character \textemdash their expected appearence, the number of sexes they can \textit{"choose from"}, the maximums and minimums for the various attributes, the number of body parts they have \textit{(each body part has its own counter of health points)}, etc. Some backgrounds are restricted to certain races.\newline
Characters may also have a secondary race, which is typically something they assumed during their life, while their primarily race is what they were born as. For example, for a vampire who was born a human, their primary race would be Human, with their secondary race being Vampire.\newline
Races also have a variable called an \textbf{ageing factor}, which determines how long is the typical lifespan of a race compared to humans. When calculating a character's biological age in days \textemdash so long as the character is older than 6574.5 days \textemdash we must take their chronological age, reduce it by 6574.5, divide by the race's ageing factor, then increment by 6574.5. In other words: \[\frac{chronologicalAge-6574.5}{ageingFactor}+6574.5\]
It is recommended for programmers to implement secondary races dynamically generating them from primary races and having a pointer variable to the parent race.