\chapter{Specifying and running a computation}

\section{Normalizations}
\label{sec:normalizations}

{\setlength{\parindent}{0cm}
Dimensional quantities in \sfincs~are normalized to ``reference'' values that are denoted by a bar:\\
$\bar{B}$ = reference magnetic field, typically 1 Tesla.\\
$\bar{R}$ = reference length, typically 1 meter.\\
$\bar{n}$ = reference density, typically $10^{20}$ m$^{-3}$, $10^{19}$ m$^{-3}$, or something similar.\\
$\bar{m}$ = reference mass, typically either the mass of hydrogen or deuterium.\\
$\bar{T}$ = reference temperature in energy units, typically 1 eV or 1 keV.\\
$\bar{v} = \sqrt{2 \bar{T} / \bar{m}}$ = thermal speed at the reference temperature and mass\\
$\bar{\Phi}$ = reference electrostatic potential, typically 1 V or 1 kV.\\
}

You can choose any reference parameters you like, not just the values
suggested here. However, if you use a {\ttfamily vmec} or {\ttfamily .bc} magnetic equilibrium
by choosing \parlink{geometryScheme} = 5, 11, or 12, then you MUST use $\bar{B}$ = 1 Tesla and $\bar{R}$ = 1 meter.
The code ``knows'' about the reference values only through
the 3 combinations \parlink{Delta}, \parlink{alpha}, and \underscoreparlink{nu\_n}{nu_n}
in the {\ttfamily \hyperref[sec:physicsParameters]{physicsParameters}} namelist.

Normalized quantities are denoted by a ``hat''.  Taking the magnetic field as an example,
$\hat{B}=B/\bar{B}$, where $\hat{B}$ is called {\ttfamily BHat} in the fortran code and \HDF~output file.

\section{Radial coordinates}
\label{sec:radialCoordinates}

A variety of flux-surface label coordinates are used in other codes and in the literature.
One common choice (used in \vmec) is $\psi_N$, the toroidal flux normalized to its
value at the last closed flux surface.  Another common choice is an ``effective normalized minor radius''
$r_N$, defined by $r_N=\sqrt{\psi_N}$.  For gradients of density, temperature, and electrostatic potential (i.e. the radial
electric field), it is useful to use a dimensional local minor radius $r = r_N a$, where $a$ is some
measure of the plasma effective outer minor radius.  Finally, one could also use $\psi$ directly.
For maximum flexibility, \sfincs~permits any of these four radial coordinates to be used, and different radial
coordinates can be used in different aspects of a given computation.  Output quantities which depend
on the radial coordinate, such as radial fluxes, are often given with respect to all radial coordinates.
In \sfincs, the four radial coordinates are named as follows:\\

{\setlength{\parindent}{0cm}

{\ttfamily \hypertarget{psiHat}{psiHat}} = $\hat\psi$ is the toroidal flux (divided by $2\pi$), normalized by $\bar{B}\bar{R}^2$.\\

{\ttfamily \hypertarget{psiN}{psiN}} = $\psi_N$ is the toroidal flux normalized by its value at the last closed flux surface.\\

{\ttfamily \hypertarget{rHat}{rHat}} = $\hat{r}$ is defined as {\ttfamily aHat}$\sqrt{\mbox{\ttfamily psiN}}$, where {\ttfamily aHat} is an effective minor radius of the last closed flux surface normalized by $\bar{R}$.\\

{\ttfamily \hypertarget{rN}{rN}} = $r_N$ is defined as $\sqrt{\mbox{\ttfamily psiN}}$.\\

}

These four radial coordinates are identified by the numbers 0, 1, 2, and 3 respectively.

When setting up a run, you can make several independent choices for radial coordinates.  One parameter
you select is \parlink{inputRadialCoordinateForGradients} in the
{\ttfamily \hyperref[sec:geometryParameters]{geometryParameters}} namelist.
This parameter controls which coordinate is used to specify the gradients. The possible values of \parlink{inputRadialCoordinateForGradients} are:\\

{\setlength{\parindent}{0cm}

0: Use derivatives with respect to {\ttfamily psiHat}: Density gradients are specified using \parlink{dnHatdpsiHats},
temperature gradients are specified using \parlink{dTHatdpsiHats},  a single $E_r$ is specified using
\parlink{dPhiHatdpsiHat}, and a range of $E_r$ for a scan is specified using \parlink{dPhiHatdpsiHatMin}-\parlink{dPhiHatdpsiHatMax}.
\\

1: Use derivatives with respect to {\ttfamily psiN}: Density gradients are specified using \parlink{dnHatdpsiNs},
temperature gradients are specified using \parlink{dTHatdpsiNs},  a single $E_r$ is specified using
\parlink{dPhiHatdpsiN}, and a range of $E_r$ for a scan is specified using \parlink{dPhiHatdpsiNMin}-\parlink{dPhiHatdpsiNMax}.
\\

2: Use derivatives with respect to {\ttfamily rHat}: Density gradients are specified using \parlink{dnHatdrHats},
temperature gradients are specified using \parlink{dTHatdrHats},  a single $E_r$ is specified using
\parlink{dPhiHatdrHat}, and a range of $E_r$ for a scan is specified using \parlink{dPhiHatdrHatMin}-\parlink{dPhiHatdrHatMax}.
\\

3: Use derivatives with respect to {\ttfamily rN}: Density gradients are specified using \parlink{dnHatdrNs},
temperature gradients are specified using \parlink{dTHatdrNs},  a single $E_r$ is specified using
\parlink{dPhiHatdrN}, and a range of $E_r$ for a scan is specified using \parlink{dPhiHatdrNMin}-\parlink{dPhiHatdrNMax}.
\\

4: Same as option 2, except the radial electric field is specified using \parlink{Er}. Thus, derivatives with respect to {\ttfamily rHat} will be used:
Density gradients are specified using \parlink{dnHatdrHats},
temperature gradients are specified using \parlink{dTHatdrHats},  a single $E_r$ is specified using
\parlink{Er}, and a range of $E_r$ for a scan is specified using \parlink{ErMin}-\parlink{ErMax}.
\\
}

The most common choice is the default, 4.  The quantity \parlink{Er} in both the input and output files is defined as
$- d\hat{\Phi} / d \hat{r} = -(\bar{R} / \bar{\Phi}) d \Phi / d r$.

Another choice involving radial coordinates is how to specify the flux surface for the computation.
This choice is made using the parameter \parlink{inputRadialCoordinate} in the {\ttfamily \hyperref[sec:geometryParameters]{geometryParameters}}
namelist, which is again an integer from 0 to 3, and this parameter need not be the same as \parlink{inputRadialCoordinateForGradients}.
An extra complication with specifying the flux surface is that the magnetic equilibrium file will contain data on a finite number of surfaces,
and you may wish to use one of these surfaces.  For this reason, the parameters for specifying the flux surface have {\ttfamily \_wish}
appended to the name. In other words, the allowed values for \parlink{inputRadialCoordinate} are:\\

{\setlength{\parindent}{0cm}

0: Specify the flux surface using \underscoreparlink{psiHat\_wish}{psiHat_wish}.\\

1: Specify the flux surface using \underscoreparlink{psiN\_wish}{psiN_wish}.\\

2: Specify the flux surface using \underscoreparlink{rHat\_wish}{rHat_wish}.\\

3: Specify the flux surface using \underscoreparlink{rN\_wish}{rN_wish}.\\

}

When using \parlink{geometryScheme} == 11 or 12, \sfincs~will always shift the ``wish'' value so it matches an available surface in the magnetic equilibrium file.
For \parlink{geometryScheme} == 5, the \parlink{VMECRadialOption} parameter lets you can choose whether to shift to the nearest surface in the magnetic equilibrium file,
or to interpolate the \vmec~data onto the exact value of radius you specify.

If you perform a radial scan, then there is a third choice you can make: which radial coordinate to use in the {\ttfamily profiles} file.
This choice is made with an integer 0, 1, 2, or 3 in the first non-comment line of the {\ttfamily profiles} file.
The radial coordinate used in the {\ttfamily profiles} file need not be the same as either
\parlink{inputRadialCoordinate} or \parlink{inputRadialCoordinateForGradients}.
Note however that the maximum and minimum radial electric field specified in the {\ttfamily profiles}
file must be given in terms of the electric field variable selected by \parlink{inputRadialCoordinateForGradients}.

For more details about the behavior of \parlink{inputRadialCoordinate}, \parlink{inputRadialCoordinateForGradients}, and \parlink{VMECRadialOption},
see section \ref{sec:geometryParameters}.

\section{Trajectory models}
\label{sec:trajectoryModels}

As discussed in \cite{sfincsPaper},
one of the capabilities of \sfincs~is to compare various models for the terms in the kinetic equation involving $E_r$.
These variations of the kinetic equation are called ``trajectory models'' in \cite{sfincsPaper}.
The relevant terms in the kinetic equation can be turned off and on by certain Boolean parameters in the {\ttfamily \hyperref[sec:physicsParameters]{physicsParameters}} namelist.
The models described in \cite{sfincsPaper} are selected as follows:\\

{\setlength{\parindent}{0cm}

\underline{Full trajectories:}\\
{\ttfamily
\parlink{includeXDotTerm} = .true.\\
\parlink{includeElectricFieldTermInXiDot} = .true.\\
\parlink{useDKESExBDrift} = .false.\\
}

\underline{Partial trajectories:}\\
{\ttfamily
\parlink{includeXDotTerm} = .false.\\
\parlink{includeElectricFieldTermInXiDot} = .false.\\
\parlink{useDKESExBDrift} = .false.\\
}

\underline{DKES trajectories:}\\
{\ttfamily
\parlink{includeXDotTerm} = .false.\\
\parlink{includeElectricFieldTermInXiDot} = .false.\\
\parlink{useDKESExBDrift} = .true.\\
}
}

There is not a significant difference in computational cost between these models.

%\section{Quasineutrality and variation of the electrostatic potential on the flux surface}
\section{Calculations with $\Phi_1(\theta,\zeta)$, includePhi1 = {\ttfamily .true.}}
\label{sec:qn}

One choice you should consider in setting up a computation is whether or not
to include variation on the flux surface of the electrostatic potential, $\Phi_1(\theta,\zeta)$.
Such variation does occur to some degree in a real plasma, but it has traditionally been
neglected in analytical theory and in many codes such as \dkes~(however, nowadays analytical modelling exists, see e.g. \cite{Buller2018} and references therein).  It can be proved
that including $\Phi_1$ has no effect on the particle or heat fluxes, parallel flows, or bootstrap current
when $E_r=0$, but generally there can be some difference when $E_r \ne 0$.
(There is a subtlety in showing that the heat flux is the same with and without $\Phi_1$,
discussed in the notes 20150325-01 in \path{sfincs/doc/}.)

In \sfincs, you can choose whether or not to
include $\Phi_1$ using the parameter \parlink{includePhi1} in the
{\ttfamily \hyperref[sec:physicsParameters]{physicsParameters}}
namelist.
Furthermore, the parameters \parlink{includePhi1InKineticEquation}, \parlink{includePhi1InCollisionOperator} and \parlink{quasineutralityOption} can be used to control how $\Phi_1$ is included in the system of equations.
\sfincs~also allows to read $\Phi_1$ as an input by using \parlink{readExternalPhi1}. This is useful for looking at the effect on the transport by a $\Phi_1$ generated by some mechanism not treated by \sfincs, or for using $\Phi_1$ calculated by another code. Moreover, with $\Phi_1$ as an input the runs are typically cheaper than if \sfincs~calculates $\Phi_1$ itself, so if the potential variation is not expected to change in a parameter scan this setting could be useful to reuse the same $\Phi_1$ and decrease the computational costs.
If and only if $\Phi_1$ is included as an unknown (i.e. not if $\Phi_1$ is an input with \parlink{readExternalPhi1}), a quasineutrality equation is solved
at each point on the flux surface.  Due to these extra unknowns ($\Phi_1$) and extra equations
(quasineutrality), the system matrix is slightly larger when \parlink{includePhi1} is \true.
Specifically, the number of rows and columns are each increased by \Ntheta$\times$\Nzeta$+1$.  This increase is miniscule compared
to the number of rows and columns associated with the kinetic equation, which depends not only on real space
but also on velocity space and species.
%Thus, there is very little extra computational cost associated
%with \parlink{includePhi1}.
Thus, there is typically not a substantial increase in memory requirements associated with \parlink{includePhi1} (however, setting the preconditioner option \parlink{reusePreconditioner} = \false~can lead to a significant increase in memory requirements).
The time requirements will normally increase when \parlink{includePhi1} is \true~because the system of equations is then nonlinear (except if \parlink{readExternalPhi1} is \true).
The nonlinear equations are solved with a Newton method using the Scalable Nonlinear Equations Solvers (SNES) in \PETSc. SNES internally employs KSP for the solution of its linear systems.
Note that for a nonlinear calculation the \sfincs~output file \parlink{outputFileName} will contain the results of each nonlinear iteration in the relevant output parameters. The final result of the calculation is stored in the last position of the arrays.
The resolution parameters required for numerical convergence in nonlinear calculations are often similar to the parameters in the corresponding linear calculations (based on experience).
A sanity check to control that the nonlinear calculation is not incorrect (however it is not a guarantee for numerical convergence), is to check that the total flux-surface averaged particle flux for the lowest order distribution function $\displaystyle \left\langle \int d^3 v \, f_{s0} \, \bm{v}_{\mathrm{d} s} \cdot \nabla \psi \right\rangle = \left\langle \int d^3 v \, f_{s0} \left(\bm{v}_{\mathrm{m} s} + \bm{v}_{E}\right) \cdot \nabla \psi \right\rangle$ of each plasma species $s$ comes out as 0. This can e.g. be done by comparing \underscoreparlink{particleFlux\_vm0\_psiHat}{particleFlux_vm0_psiHat} to \underscoreparlink{particleFlux\_vE0\_psiHat}{particleFlux_vE0_psiHat} in the \sfincs~output file \parlink{outputFileName} and see that they are equal but with opposite sign.

You may wish to set \parlink{includePhi1} = \true~when
using \sfincs~to model an experiment, and set \parlink{includePhi1} = \false~when
comparing \sfincs~with analytic theory or with another code that does not include $\Phi_1$.
In Ref~\cite{Mollen2018} there are examples of \sfincs~calculations with \parlink{includePhi1} = \true~for experimental data.

\subsection{\PETSc~commands}
In this section we summarize experiences with the Scalable Nonlinear Equations Solvers (SNES) in \PETSc,
and list some of the command-line flags associated with \PETSc~which can be useful for nonlinear calculations.
(See section~\ref{sec:PETScCommands} for general useful options in \sfincs.)

If the nonlinear solver fails to converge with the default settings, a first useful thing to try is setting \parlink{reusePreconditioner} = \false~in the {\ttfamily \hyperref[sec:preconditionerOptions]{preconditionerOptions}} namelist (note that memory requirements might increase).
Secondly, it can be useful to change the SNES linesearch type (see -snes\_linesearch\_type flag below) and to increase the absolute convergence tolerance (see -ksp\_atol flag below) for the KSP iterations and the relative convergence tolerance (see -snes\_rtol flag below) for the SNES iterations.
Moreover, note that in each nonlinear iteration the nested linear iterations will start with the current residual. If the calculation progresses towards nonlinear convergence this implies that the starting residual norm in each nonlinear iteration is smaller and smaller. Therefore in a calculation with \parlink{includePhi1} = \true~it is sometimes useful to use a less strict requirement on \parlink{solverTolerance} in the {\ttfamily \hyperref[sec:resolutionParameters]{resolutionParameters}} namelist, since in the last nonlinear iterations the residual will otherwise end up being unnecessarily small causing long run times (this problem is also avoided by appropriate -ksp\_atol and -snes\_rtol flags).\\
Some useful \PETSc~flags taken from Ref~\cite{PETSc2017}:\\

\subsubsection{KSP}

KSP is intended for solving nonsingular linear systems of the form $A x = b$.
The residual at iteration $k$ is $r_k \equiv b - A x_k$ and convergence is detected if
\[
\left\|r_k\right\|_2 < \mathrm{max} \left(\text{\ttfamily rtol} \cdot \left\|b\right\|_2, \text{\ttfamily atol}\right).
\]
Divergence is detected if
\[
\left\|r_k\right\|_2 > \text{\ttfamily dtol} \cdot \left\|b\right\|_2.
\]
\\

\myhrule

\PETScParam{-ksp\_atol {\normalfont \ttfamily$<$atol$>$}}
{The absolute convergence tolerance absolute size of the (possibly preconditioned) residual norm in the KSP iterations.
The default is {\ttfamily 1e-50}, which in practice implies that absolute convergence will never be fulfilled.
Setting this to a value of the order {\ttfamily 1e-10} can be useful in a nonlinear calculation.}

\myhrule

\PETScParam{-ksp\_rtol {\normalfont \ttfamily$<$rtol$>$}}
{The relative convergence tolerance, relative decrease in the (possibly preconditioned) residual norm in the KSP iterations.
The default in \PETSc~is {\ttfamily 1e-5},
but the value in \sfincs~is set by \parlink{solverTolerance} in the {\ttfamily \hyperref[sec:resolutionParameters]{resolutionParameters}} namelist. This can be overwritten by using the flag.}

\myhrule

\PETScParam{-ksp\_divtol {\normalfont \ttfamily$<$dtol$>$}}
{The divergence tolerance, amount (possibly preconditioned) residual norm can increase before KSP concludes that the method is diverging.
The default is {\ttfamily 1e5}, which is usually a good setting.}

\myhrule

\PETScParam{-ksp\_max\_it {\normalfont \ttfamily$<$maxits$>$}}
{Maximum number of iterations to use in KSP.
The default is {\ttfamily 1e4}, which is usually a good setting.}

\subsubsection{SNES}

The SNES class includes methods for solving systems of nonlinear equations of the form $\bm{F} \left(\bm{x}\right) = 0$,
where $\bm{F}~:~\Re^n \rightarrow \Re^n$.
\PETSc's default method for solving the nonlinear equation is Newton's method. The general
form of the $n$-dimensional Newton's method for solving the system is
\[
\bm{x}_{k + 1} = \bm{x}_{k} - \bm{J} \left(\bm{x}_{k}\right)^{-1} \bm{F} \left(\bm{x}_{k}\right),~~k = 0, 1, \ldots
\]
where $\bm{x}_{0}$ is an initial approximation to the solution and $\bm{J} \left(\bm{x}_{k}\right) = \bm{F}^\prime \left(\bm{x}\right)$, the Jacobian, is nonsingular at each iteration. In practice, the Newton iteration is implemented by the following two
steps: (Approximately) solve $\bm{J} \left(\bm{x}_{k}\right) \Delta \bm{x}_{k} = - \bm{F} \left(\bm{x}_{k}\right)$,  update $\bm{x}_{k + 1} = \bm{x}_{k} + \Delta \bm{x}_{k}$.
In the output it is useful to pay attention to the reason why the SNES iterations have converged/diverged.
SNES iterations typically only finish successfully due to one of the following reasons (see SNES flags): {\ttfamily SNES\_CONVERGED\_FNORM\_ABS}, {\ttfamily SNES\_CONVERGED\_FNORM\_RELATIVE},\\ {\ttfamily SNES\_CONVERGED\_SNORM\_RELATIVE}.
In the \sfincs~output file \parlink{outputFileName} there is also a flag {\ttfamily didNonlinearCalculationConverge} which specifies if the nonlinear calculation converged.\\

\myhrule

\PETScParam{-snes\_type  {\normalfont \ttfamily$<$type$>$}}
{SNES includes several Newton-like nonlinear solvers based on line search techniques and trust region methods. A table of \PETSc~nonlinear solvers can be found in Ref~\cite{PETSc2017}. In \sfincs~the (default) method SNESNEWTONLS ({\ttfamily \bfseries -snes\_type} {\normalfont \ttfamily newtonls}) seems to work the best and we have not yet had any success with the other solvers.
}

\myhrule

\PETScParam{-snes\_linesearch\_type  {\normalfont \ttfamily$<$type$>$}}
{This flag is only used when {\ttfamily \bfseries -snes\_type} {\normalfont \ttfamily newtonls}. The different options are {\normalfont \ttfamily basic}, {\normalfont \ttfamily bt}, {\normalfont \ttfamily l2}, {\normalfont \ttfamily cp}, {\normalfont \ttfamily nleqerr}, {\normalfont \ttfamily shell}.
The default is {\normalfont \ttfamily bt}, but in \sfincs~the experience is that {\normalfont \ttfamily l2} or {\normalfont \ttfamily cp} can often work better.
Depending on the chosen method there are other flags that can be used to control how the search is performed: {\ttfamily \bfseries -snes\_linesearch\_alpha}, {\ttfamily \bfseries -snes\_linesearch\_damping}, {\ttfamily \bfseries -snes\_linesearch\_maxstep}, {\ttfamily \bfseries -snes\_linesearch\_max\_it}, {\ttfamily \bfseries -snes\_linesearch\_minlambda}, {\ttfamily \bfseries -snes\_linesearch\_order}. We have not yet had any success with other values than the default for all these flags.
}

\myhrule

\PETScParam{-snes\_atol {\normalfont \ttfamily$<$atol$>$}}
{The absolute convergence tolerance absolute size of the residual norm in the SNES iterations.
The default is {\ttfamily ?}, but usually this flag can be ignored.
If $\left\|\bm{F} \left(\bm{x}_{k}\right)\right\|_2 \leq \text{\ttfamily atol}$ then the output flag {\ttfamily SNES\_CONVERGED\_FNORM\_ABS} is specified.}

\myhrule

\PETScParam{-snes\_rtol {\normalfont \ttfamily$<$rtol$>$}}
{The relative convergence tolerance, relative decrease in the residual norm in the SNES iterations.
The default in \PETSc~is {\ttfamily ?}, setting this to {\ttfamily 1e-6} can be a good option.
If $\left\|\bm{F} \left(\bm{x}_{k}\right)\right\|_2 \leq \text{\ttfamily rtol} \cdot \left\|\bm{F} \left(\bm{x}_{0}\right)\right\|_2$ then the output flag {\ttfamily SNES\_CONVERGED\_FNORM\_RELATIVE} is specified.}

\myhrule

\PETScParam{-snes\_stol {\normalfont \ttfamily$<$stol$>$}}
{The convergence tolerance in terms of the norm of the change in the solution between steps in the SNES iterations.
The default is {\ttfamily ?}, but usually this flag can be ignored.
If $\left\|\Delta \bm{x}\right\|_2 \leq \text{\ttfamily stol} \cdot \left\|\bm{x}\right\|_2$ then the output flag {\ttfamily SNES\_CONVERGED\_SNORM\_RELATIVE} is specified.}

\myhrule

\PETScParam{-snes\_max\_it {\normalfont \ttfamily$<$maxit$>$}}
{The maximum number of iterations before SNES stops.
The default is {\ttfamily 50}, which is usually a good setting. (Sometimes convergence is slow in SNES but after more than 50 iterations it is unlikely that convergence will be reached.)}

\myhrule

\PETScParam{-snes\_view}
{Dumps detailed information to stdout related to the nonlinear solver.}

\myhrule

\PETScParam{-snes\_linesearch\_monitor}
{Dumps detailed information to stdout at every iteration of the nonlinear solver to display the iteration's progress when a line search is performed.}

\section{Poloidal and toroidal magnetic drifts}
\label{sec:magneticDrifts}

You can choose to either include or not include the poloidal and toroidal magnetic drifts.
These drifts are turned off by default.  To turn them on, all you need to do is
set \parlink{magneticDriftScheme}~$>$~0 (with the default being \parlink{magneticDriftScheme}~=~1) in the {\ttfamily \hyperref[sec:physicsParameters]{physicsParameters}} namelist.
(Setting \parlink{magneticDriftScheme}~=~2 uses a slightly different parallel magnetic drift, which gives
indistinguishable results to setting 1 for all cases examined so far, and which is in fact exactly identical to setting 1
in the limit of vanishing plasma beta.)
If the poloidal/toroidal magnetic drifts are turned on, you must use VMEC
geometry (\parlink{geometryScheme}~=~5), since the magnetic drifts
depend on various derivatives of the components of the magnetic field which
are not available in the simplified geometry models.
(In fact, to include poloidal/toroidal magnetic drifts in a radially local code like \sfincs~is somewhat ambiguous since the approximation made when neglecting the radial derivatives of the unknowns is different in different coordinate system.)

The magnetic drift terms introduce nonzeros in the system matrix,
and therefore increase the memory and time required for factorization.
The change is small; a typical increase in both memory and time is 20-40\%.
These magnetic drift terms typically have a minor effect on the physics outputs except when the radial
electric field is near 0.

If the electrostatic potential is not constant on flux surfaces and poloidal/toroidal magnetic drifts
are included, certain cross-terms exist in the kinetic equation which have not yet been implemented in \sfincs.
Thus, at present it is not strictly correct to simultaneously set  \parlink{magneticDriftScheme}~$>$~0
and  \parlink{includePhi1} = \true, but it is allowed for testing purposes.

\section{Sparse direct solver packages}
\label{sec:solvers}

As discussed briefly in section \ref{sec:gmres}, the most computationally demanding step in
\sfincs~is the direct $LU$-factorization of a very large sparse nonsymmetric real matrix. The \PETSc~library
which \sfincs~uses has interfaces to a large number of other packages for direct factorization of such matrices,
making it possible to choose among the various solver packages with just a command-line flag ({\ttfamily -pc\_factor\_mat\_solver\_package}).  Some lists of the direct solvers
available in \PETSc~can be found \href{http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSolverPackage.html#MatSolverPackage}{here}
or in the ``direct solvers''-``LU'' section of \href{http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html}{this page}.
It is important for the $LU$-solver package to be one that is efficiently parallelized,
in order to be able to solve problems at the high resolutions required for experimentally relevant collisionality and magnetic geometry.
The recommended choice of $LU$ solver is \href{http://mumps-solver.org/}{\mumps} (which is the default), and another good option is \href{http://crd-legacy.lbl.gov/~xiaoye/SuperLU/}{\superludist}.
(The {\ttfamily PARDISO} library available in the Intel Math Kernel Library is probably suitable as well, though we have not investigated it yet.)
In side-to-side comparisons, we find \mumps~systematically uses substantially less memory and time than \superludist~for factorization.
In principle, other solver packages for asymmetric matrices that are interfaced to \PETSc~could be used as well, such as {\ttfamily UMFPACK}, {\ttfamily PASTIX}, etc.

There are two ways to choose between solver packages.
One method is the \sfincs~parameter \parlink{whichParallelSolverToFactorPreconditioner} in the
{\ttfamily \hyperref[sec:otherNumericalParameters]{otherNumericalParameters}} namelist.
This parameter only allows you to choose between \mumps~ and \superludist, not other solver packages interfaced to \PETSc.
Another way to choose between solver packages is the command-line flag {\ttfamily -pc\_factor\_mat\_solver\_package},
followed by one of the options in quotation marks \href{http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSolverPackage.html#MatSolverPackage}{here}.
The command-line flag overrides the namelist parameter.

The physics outputs of the code should be independent of the solver package used to several significant digits.
In principle, different solver packages solving the same linear system should find the identical solution.
However there will be small differences in the solutions associated with roundoff error.

Note that {\ttfamily superlu} and \superludist~are distinct libraries. The former is serial while the latter is parallelized.
Therefore there is no reason to use {\ttfamily superlu}; \superludist~is always preferable.

The \mumps~package has a large number of control parameters, which are documented in the \mumps~manual which can be downloaded \href{http://mumps-solver.org/}{here}.
You do not need to be aware of most of these control parameters.
However, several parameters which may be useful are discussed in section \ref{sec:mumpsControlParameters}.
It is also worth being aware of the section of the \mumps~manual on error messages.
This section is useful for interpreting the {\ttfamily INFO(1)} and {\ttfamily INFO(2)} error codes that
are reported if \sfincs~exits with an error associated with \mumps.

The \superludist~package has many fewer options than \mumps.
You can find of list of the options by running \sfincs~with the {\ttfamily -help} command-line flag when
using \superludist, and searching the output for lines containing {\ttfamily superlu\_dist}.
The \superludist~options are also documented in the package's manual, available \href{http://crd-legacy.lbl.gov/~xiaoye/SuperLU/#superlu_dist}{here}.
We have not found any advantage in adjusting
any of the \superludist~options.

The \PETSc~library includes a built-in sparse direct solver which works on only a single processor.
You can select this solver using the command-line flag \\
\centerline{\ttfamily -pc\_factor\_mat\_solver\_package petsc}\\
This solver could potentially be useful if you are running on a system that does not have \mumps~or \superludist~installed,
and you are only considering problems that require sufficiently little memory (e.g. tokamaks) that parallelization is not required.
However, this solver is less robust than \mumps~or \superludist, sometimes exiting with an error message that there is a zero pivot
even though  \mumps~and \superludist~can solve the same system with no problem. Therefore, even if you plan to use only a single processor,
we still recommend that you install  \mumps~or \superludist.

\section{Parallelization: Choosing the number of nodes \& processors}
\label{sec:parallelization}

Usually the limiting factor for \sfincs~is not time but memory.
(The time required depends on the resolution used, but jobs for experimentally relevant W7-X parameters
typically take under 10 minutes.)
Therefore, when considering how many nodes to request for a \sfincs~job,
the first issue to consider is ensuring you have requested sufficient total memory.
A good way to determine the memory required (assuming you are using the default solver \mumps)
is to first run \sfincs~on 1 node using the parameters of interest
and look for the following line in standard output:\\
\centerline{\ttfamily ** TOTAL     space in MBYTES for IC factorization         :       1072}\\
The number at the end will generally be different; it depends not only on the
resolution and number of species used, but also increases with the number of processors
as discussed below.  Make sure the number of nodes requested times the number of megabytes per node exceeds this number.
This line in standard output is generated by \mumps, so if you are using a different solver, this information is not printed,
and you may need to determine the number of nodes by trial-and-error.
Since some memory on each node is used by the operating system and by \sfincs~functions other than the solver,
you may need to use a slightly higher number of nodes than this estimate suggests.
For experimentally relevant W7-X and HSX parameters, we typically use 2-6 nodes with 64 GB each.
Problems with lower resolution requirements, such as tokamaks, often can be run on a single node.

The `IC' in the above line stands for `in core', meaning the $L$ and $U$ factors are stored
in memory rather than on disk.  In \mumps, one can also choose to do an `out of core' (OOC) solve,
in which case substantially less memory is typically required.  The price you pay for this memory savings is time, since
disk access is slow compared to memory access.  Due to this slowdown, we have not used the OOC capability much, but
you might find it useful in some circumstances.
To see how much memory would be required for an OOC solve,
look for the following line in standard output:\\
\centerline{\ttfamily ** TOTAL     space in MBYTES for OOC factorization        :       128}\\
(The number at the end will generally be different.)
To invoke out-of-core mode, you must take two steps. First, use the following command-line flag when calling \sfincs:\\
\centerline{\ttfamily -mat\_mumps\_icntl\_22 1}\\
Second, you must set the environment variable {\ttfamily MUMPS\_OOC\_TMPDIR} to some reasonable directory
before calling \sfincs. This environment variable could be set for example in the batch job file.
Temporary files containing the $L$ and $U$ factors will be stored in the directory indicated.

Another issue to consider is how many processors to use.  It is not always best to use the maximum number of processes
available on the number of nodes you have chosen. The reason is that as the preconditioner matrix is divided among
more and more processes, the $LU$ factorization becomes less efficient, requiring more memory and more communication.
If you examine the \mumps~IC and OOC memory requirements indicated above, you will find they increase somewhat
as the number of processes increase.
One needs to find a balance between speed (favoring many processes) and memory requirements (favoring few processes).
While it is almost always better overall to use 2 processes compared to 1 process, it is not always better to use 128 processes
compared to 64 processes.  The sweet spot is often in the range of 16-64 processes.
To determine how to request fewer processes than the maximum available on a given number of nodes,
see the documentation for your computing system.

If more memory is required than is available, the system will usually terminate your job with an out-of-memory (OOM) error.
When this occurs, you need to either increase the number of nodes requested or decrease the number of processors
requested.

Most of the time, \sfincs~is run via \sfincsScan~as some parameter is scanned, such as the radial electric field.
In this case, the scan is ``embarassingly parallel'' in the sense that each job in the scan is completely independent
of the other jobs.  Even if each individual job requires only 1 or a small number of nodes, it is still useful
to run \sfincs~on a computing system with many nodes so the scan can be carried out in parallel.

%\section{Issues with running on 1 processor}

\section{Monoenergetic transport coefficients}
\label{sec:monoenergetic}

By setting \parlink{RHSMode} = 3, \sfincs~can be run in a mode
where it solves the same kinetic equation (prior to discretization) as
\dkes~and other monoenergetic codes.
When \parlink{RHSMode} = 3, the values of \parlink{Zs}, \parlink{THats}, \parlink{nHats},
\parlink{mHats}, \underscoreparlink{nu\_n}{nu_n}, and {\ttfamily dPhiHatdXXX} are all ignored.
Instead, the collisionality is set by \parlink{nuPrime}, and the radial electric field is set
by \parlink{EStar}.  The first of these quantities is the dimensionless collisionality
\begin{equation}
\mbox{\parlink{nuPrime}} = \frac{(G+\iota I) \nu}{v B_0}
\end{equation}
where $G$ and $I$ are defined in (\ref{eq:covariant}), $\iota=1/q$ is the rotational transform,
$v$ is the speed at which the monoenergetic calculation is being performed, and $B_0$ is the (0,0) Fourier harmonic of $B$
with respect to the Boozer poloidal and toroidal angles. The collision
frequency $\nu$ is here the value of $\nu_\mathrm{ii}$ one would have if
$v$ were the thermal speed. That is, in SI units,
\begin{equation}
  \nu=\frac{4}{3\sqrt{\pi}}\frac{n Z^4e^4\ln \Lambda}{4\pi\epsilon_0^2m^2v^3}.
\end{equation}
%
The normalized radial electric field is
\begin{equation}
\mbox{\parlink{EStar}} = \frac{cG}{\iota v B_0} \frac{d\Phi}{d\psi}
\end{equation}
(Gaussian units).
When {\ttfamily RHSMode} == 1, {\ttfamily nuPrime} and {\ttfamily EStar} are ignored.
\todo{Should be change the behavior of RHSMode=2 so it uses nuPrime and EStar instead of nu\_n?}

The two parameters \parlink{nuPrime} and \parlink{EStar} are
related to the corresponding DKES parameters {\ttfamily CMUL} and
{\ttfamily EFIELD} by
\begin{eqnarray}
  \mbox{\ttfamily
    CMUL}&\equiv&\frac{\nu_\mathrm{D}}{v}=\frac{3\sqrt{\pi}}{4}\left(\mathrm{erf}(1)-\mathrm{Ch}(1)\right)
  \frac{B_0}{G+\iota I}\mbox{\ttfamily nuPrime},\\
  \mbox{\ttfamily EFIELD}&\equiv&-\left[\frac{d\Phi}{dr}\right]_\mathrm{DKES}\frac{1}{vB_0}=-\frac{\iota}{G}\left[\frac{d\Psi}{dr}\right]_\mathrm{DKES}\mbox{\ttfamily EStar},
\end{eqnarray}
where $\mathrm{Ch}$ is the Chandrasekhar function and $\nu_\mathrm{D}$
is the actual pitch-angle deflection frequency of the particle.
\todo{These expressions are in SI units. Should there be some
  factors of $c$ to adhere to the Gaussian standard used here?
  Probably not.}

\section{Poloidal and toroidal angles}

If you are interested in any of the output quantities that vary on a flux surface,
such as the density or electrostatic potential, then it is important to know
how the poloidal and toroidal angles ($\theta$ and $\zeta$) in \sfincs~are defined.
The definitions of the poloidal and toroidal angles in
\sfincs~depend on the input parameter \parlink{geometryScheme}. When a \vmec~equilibrium is imported by setting
\parlink{geometryScheme} = 5, then \sfincs~will use the same poloidal and toroidal angles
as \vmec.  The toroidal angle in this case is the normal cylindrical coordinate. Note that field lines
are not straight in these \vmec~coordinates.
For any other setting of \parlink{geometryScheme}, \sfincs~will use Boozer coordinates.


\chapter{$k$-nearest Neighbor}
\input{6DL/KNN}
\chapter{$k$-Means Clustering Problem and Algorithm}
\input{6DL/Kmeans}


%!TEX root = ../thesis.tex
%*******************************************************************************
%*********************************** First Chapter *****************************
%*******************************************************************************
\chapter{Background}
\label{chapter:background}
% \section{Notation and Basic Concepts}
% \paragraph{}
% It will be useful to clarify some of the notation throughout this work.
% \begin{itemize}
% \item Vectors will be denoted by boldface lowercase letters: $\vec{u}, \vec{x}, ...$
% \item Matrices will be denoted by uppercase letters: $A, M, ...$
% \item Probability mass functions will be denoted by uppercase letters: $P(x),
%   Q(z), ...$
% \item Probability density functions will be denoted by lowercase letters: $p(y),
%   q(u), ...$
% \item In general, exact/continuous values will be denoted by unannotated letters
%   (e.g. $z_i, \vec{w}$), their quantized counterparts denoted by a hat
%   ($\hat{z}_i, \hat{\vec{w}}$) and their approximate counterparts by a tilde
%   ($\tilde{z}_i, \vec{\tilde{w}}$).
% \item $\Exp_{p(x)}[f(x)]$ denotes the expected value of $f(x)$ with respect to
%   the mass / density $p(x)$, i.e.:
%   \[
%     \Exp_{p(x)}[f(x)] = \int_\Omega f(x) \d p(x),
%   \]
%   where $\Omega$ is the sample space. As $\Omega$ will usually denote $\Reals^n$
%   or will be understood from context, it will be omitted, and the integral will
%   be rewritten as
%   \[
%     \Exp_{p(x)}[f(x)] = \int f(x)p(x) \d x.
%   \]
% \item $H[X]$ denotes the Shannon entropy of the random variable $X$. If $X$ is
%   discrete, then it is defined as
%   \[
%     -\sum_{X=x}P(X=x)\log P(X=x).
%   \]
%   If it is continuous, then it will refer to the \textit{differential entropy}
%   of $X$, namely
%   \[
%     -\int_{\X}\log p(x) \d p(x),
%   \]
%   where $\X$ denotes the support of $X$.
%   \paragraph{Note:} we used the natural logarithm in our definition of entropy,
%   and hence its units are \textbf{nats}. If we used the base 2 logarithm
%   instead, the units would be \textbf{bits}.
% \item $\KL{q(x)}{p(x)}$ denotes the Kullback-Leibler divergence between two
%   distributions and is defined as
%   \[
%     \KL{q(x)}{p(x)} = \Exp_{q(x)}\left[\log\frac{q(x)}{p(x)}\right].
%   \]
% \item $I[X : Y]$ denotes the mutual information between random variables $X$ and
%   $Y$ and is defined as
%   \[
%     I[X : Y] = \KL{p(x, y)}{p(x)p(y)},
%   \]
%   where $(X, Y) \sim p(x, y)$ and $p(x)$ and $p(y)$ denote the marginals.
% \end{itemize}
\section{Image Compression}
\label{sec:intro_image_compression}
\par
The field of image compression is a vast topic that mainly spans over the fields
of computer science and signal processing, but incorporates methods and
knowledge from several
other disciplines, such as mathematics, neuroscience, psychology
and photography. In this section, we introduce the reader to the basics of the
topic, starting with source coding, then through lossy compression, we arrive at
the concepts of rate and distortion. Finally, we introduce transform coding, the
category in which our work falls as well.

\subsection{Source Coding}
From a theoretical point of view, given some source $S$, a sender and a
receiver, compression may be described as the aim of the sender communicating an
arbitrary sequence $X_1, X_2, \hdots, X_n$ taken from $S$ to the receiver in as few bits
as possible, such that the receiver may recover relevant information from the message.
If the receiver can always recover all the information from the message of the sender, we
call the algorithm \textbf{lossless}, otherwise, we call it \textbf{lossy}.
\par
At first, it might not seem intuitive to allow for lossy compression, and in some
domains, this is true, e.g. in text compression. However,
humans' audio-visual perception is neither completely aligned with the range of
what can be digitally represented, nor does it always scale the same way
(\cite{eskicioglu1994image}, \cite{psnr}, \cite{gupta2011modified}). Hence,
there is a great opportunity for compressing media in a lossy way by discarding
information with the change being imperceptible for a human observer while
making significant gains in size reduction.
\subsection{Lossy Compression}
As the medium of interest in lossy compression is generally assumed to be a
real-valued vector $\vec{x} \in \Reals^N$, such as RGB pixel intensities in an
image or frequency coefficients in an audio file, the usual pipeline consists of
an encoder $C \circ \Enc$, mapping a point $\vec{x} \in \Reals^N$ to a string of bits and a
decoder mapping from bitstrings to some reconstruction $\hat{\vec{x}}$. The
factor of the encoder $\Enc$  can be understood as a map from $\Reals^N$ to a
finite symbol set $\A$, called a \textbf{lossy encoder}, and $C$ can be
understood as a map from $\A$ to a
string of bits called a \textbf{lossless code} (\cite{goyal2001theoretical}).
We examine both $\Enc$ and $C$ in more detail in Section
\ref{sec:transform_coding}.
The decoder then can be thought of as inverting the code first and then using an
approximate inverse of
$\Enc$ to get the reconstruction $\hat{\vec{x}}$: $\Dec \circ C^{-1}$.
Given these, it is of paramount importance to quantify
\begin{itemize}
\item The \textbf{distortion} of the compressor. On average, how closely does
  $\hat{\vec{x}}$ resemble $\vec{x}$?
\item The \textbf{rate} of the compressor. On average, how many bits are
  required to communicate $\vec{x}$? We want this to be as low as possible of course.
\end{itemize}

\subsection{Distortion}
\label{sec:intro_distrotion}
In order to measure ``closeness'' in the space of interest $\ImSpace$,
a distance metric $d(\cdot, \cdot): \ImSpace
\times \ImSpace \rightarrow \Reals$ is introduced. Then, the distortion $D$ is
is defined as
\[
  D = \Exp{d(\vec{x}, \hat{\vec{x}})}{p(\hat{\vec{x}})}.
\]
A popular choice of $d$, across many domains of compression is the normalized $L_2$ metric
or Mean Squared Error (MSE), defined as
\[
  d(\vec{x}, \hat{\vec{x}}) = \frac{1}{N} \sum_{i}^N (x_i - \hat{x}_i)^2, \quad
  \ImSpace = \Reals^N.
\]
It is a popular metric as it is simple, easy to implement and has nice
interpretations in both the Bayesian (\cite{bishop2013pattern}) and the MDL
(\cite{hinton1993keeping}, to be introduced in Section \ref{sec:mdl}) settings.
In the image compression setting, however, the MSE is problematic, since
optimizing for it does not necessarily translate to obtaining pleasant-looking
reconstructions (\cite{zhao2015loss}). Hence, more appropriate, so-called
\textit{perceptual metrics} were developed. The two most common ones used today
are Peak Signal-to-Noise Ratio (PSNR) (\cite{psnr}, \cite{gupta2011modified}) and the
Structural Similarity Index (SSIM) (\cite{wang2004image}) and its multiscale
version (MS-SSIM) (\cite{msssim}). Crucially, these two metrics are also
differentiable, thus they lend themselves for gradient-based optimization.

\subsection{Rate}
We noted above that the code used after the lossy encoder is lossless. To
further elaborate, in virtually all cases it is an \textbf{entropy code}
(\cite{goyal2001theoretical}). This means that we assume that each symbol
in the representation $\vec{z} = \Enc(\vec{x})$ has some probability mass
$P(z_i)$. A fundamental result by Shannon states that $\vec{z}$ may not be
encoded losslessly in fewer than $H[\vec{z}]$ nats:
\begin{theorem}{(Proven by \cite{shannon1998mathematical}, presented as stated in
    \cite{mackay2003information})}
$N$ i.i.d. random variables each with entropy $H[X]$ can be compressed into more
than $N\cdot H[X]$ bits with negligible risk of information loss, as $N
\rightarrow \infty$; conversely if they are compressed into fewer than $NH[X]$ bits it is virtually certain that information will be lost.
\end{theorem}
Entropy codes, such as Huffman codes (\cite{huffman1952method}) or Arithmetic
Coding (\cite{rissanen1981universal}) can get very close to this lower bound.
We discuss coding methods further in Section \ref{sec:coded_sampling}.
\begin{framed}
  In particular, entropy codes can compress each symbol $z_i$ in $-\log P(z_i)$ nats.
\end{framed}
The rate (in nats) of the compression algorithm is defined
as the average number of nats required to code a single dimension of the input, i.e.
\[
  R = \frac{1}{N} H[\vec{z}].
\]
\subsection{Transform Coding}
\label{sec:transform_coding}
The issue with source coding is that coding $\vec{x}$ might have a lot of
dependencies across its dimensions. For images, this manifests on multiple
scales and semantic levels, e.g. a pixel being blue might indicate that most
pixels around it are blue as the scene is depicting the sky or a body of water;
a portrait of a face will also imply that eyes, a nose and mouth are probably
present, etc. Modelling and coding this dependence structure in very high
dimensions is challenging or intractable, and hence we need
to make simplifying assumptions about it to proceed.
\par
\textit{Transform coding} attempts to solve the above problem by decomposing the
encoder function $\Enc = Q \circ T$ into a so-called \textbf{analysis transform}
$T$ and a \textbf{quantizer} $Q$. The idea is to transform the input into a
domain, such that the dependencies between the dimensions are removed, and hence
they can be coded individually. The decoder inverts the steps of the encoder,
where the inverse operation of $T$ is called the \textbf{synthesis transform}
(\cite{gupta2011modified}).
\par
In \textit{linear transform coding}, $T$ is an invertible linear transformation,
such as a discrete cosine transformation (DCT), as it is in the case of JPEG
(\cite{wallace1992jpeg}), or discrete wavelet transforms in JPEG 2000
(\cite{rabbani2002overview}). While simple, fast and elegant, linear transform
coding has the key limitation that it can only at most remove correlations (i.e.
first-order dependencies), and this can severely limit its efficiency
(\cite{balle2016endtrans}). Instead, \cite{balle2016endtrans} propose a method for
\textit{non-linear transform coding}, where $T$ is replaced by a highly
non-linear transformation, and its inverse is now replaced by an approximate
inverse, which is a separate non-linear transformation. Both $T$ and its
approximate inverse are learnt, and the authors show that with a more
complicated transformation they can easily surpass the performance of the much
more fine-tuned JPEG codecs.
\par
Our work also falls into this line of research, although with significant
differences, which will be pointed out later.

\subsection{The Significance of Quantization in Lossy Compression}
\par
The reason why quantization is required in lossy compression algorithms is
that it allows to reducing information content of data. To study the
precise meaning of this, we put the problem in a formal setting.
Let us define the quantizer as a function $[\cdot]: R \rightarrow S$, where
$R$ is the original representation space (in transform coding this would be the
image $T(\Reals^N)$), and a quantized space $S$ (usually $\Ints^N$). $[\cdot]$
is always many-to-one mapping. Let $[s]^{-1} = \{x \in R \mid [x] = s \}$ be
the preimage of $s$. Then, we have the further requirement on $[\cdot]$
that the fibres of $S$ partition $R$, i.e.
\[
  \text{if } s \neq t \Rightarrow [s]^{-1} \cap [t]^{-1} = \emptyset.
\]
A popular option for the quantizer is the rounding function, mapping $[\cdot]: \Reals
\rightarrow \Ints$, where for each integer $z \in \Ints$ it is defined as $x
\in \left[z - \frac12, z + \frac12\right) \mapsto z$. Given some probability mass $P(x)$
for some data $x$, we have seen that using entropy coding $x$ can be encoded in
$-\log P(x)$ nats. The way quantization enables better compression, is that it
aggregates the probability mass of all elements in $[s]^{-1}$ into the mass of $s$.
Namely, for each $s$, the quantizer induces a new probability mass function
$\hat{Q}(s)$, such that
\[
  \hat{Q}(s) = \int_{[s]^{-1}} p(x) \d x,
\]
where the integral is replaced by summation for discrete $[s]^{-1}$. This will allow
us to code $x$ in potentially much fewer nats. To put it precisely, assume $[x] =
s$, then
\[
  -\log \hat{Q}(s) = -\log \int_{[s]^{-1}} p(x) \d x \leq -\log P(x).
\]
This is at the cost of introducing distortion (see Section
\ref{sec:intro_distrotion}), as we will not be able to reconstruct $x$ from $s$.
In particular, quantization is vital for continuous $x$, as the probability mass
of each $x$ is 0, and hence we would require $-\log P(x) = \infty$ nats to
encode them without quantization.

\section{Theoretical Foundations}
\label{sec:intro_theoretical_foundations}
\par
We now shift our focus from image compression to the foundations of
neural compression. We begin with the Minimum Description Length (MDL)
Principle (\cite{rissanen1986stochastic}) and the Bits-Back Argument
(\cite{hinton1993keeping}), the two core theoretical guiding
principles of this work. We then see how based on these, as well as on
more recent work (\cite{harsha2007communication}, \cite{havasi2018minimal})
we can develop a general ML-based compression framework that does not include
quantization in its pipeline, thus allowing gradient-based optimization methods
to be used in training our compression algorithms.

\subsection{The Minimum Description Length Principle}
\label{sec:mdl}
Our approach is based on the Minimum Description Length (MDL) Principle
(\cite{rissanen1986stochastic}). In essence, it is a formalization of Occam's
Razor, i.e. the simplest model that describes the data well is the best model of
the data (\cite{grünwald2007minimum}). Here, ``simple'' and ``well'' need to be
defined, and these definitions are precisely what the MDL principle gives us.
Informally, it asserts that given a class of hypotheses $\Hypos$ (e.g. a certain
statistical model and its parameters) and some data $\Data$, if a particular
hypothesis $H \in \Hypos$ can be described with at most $L(H)$ bits and the using the
hypothesis the data can be described with at most $L(\Data \mid H)$ bits, then the
minimum description length of the data is
\begin{equation}
\label{eq:min_desc_princ}
  L(\Data) = \min_{H \in \Hypos}\{ L(H) + L(\Data \mid H) \},
\end{equation}
and the best hypothesis is an $H$ that minimizes the above quantity.
\par
Crucially, the MDL principle can thus be interpreted as telling us that
\textbf{the best model of the data is the one that compresses it the most}.
This makes Eq \ref{eq:min_desc_princ} a very appealing learning objective for
optimization-based compression methods, ours included.
Below, we briefly review how this has been applied so far and how it translates
to our case.
\subsection{The Bits-back Argument}
Here we present the bits-back argument, introduced in
\cite{hinton1993keeping}. The main goal of their work was to develop a
regularisation technique for neural networks, and while they talk about the
compression of the model, the first method that realized bits-back
 efficiency came much later, developed by \cite{havasi2018minimal}.
Although the argument is essentially just the direct application of the MDL
principle, it can seem quite counter-intuitive at first. Hence, we begin this
section with an example to illustrate the goal of the argument,
and only then move on to formulate it in more generality.

\paragraph{Example}
Let us be given a simple regression problem on the dataset $\Data = (\X, \Y)$,
where $\X = (x_1, \hdots, x_n), \Y = (y_1, \hdots, y_n)$ are both one
dimensional input and target sets and $(x_i, y_i)$ are a corresponding training
pair. Assume we wish to fit a simple model:
\[
  \hat{y} = f(x) = \alpha x + \beta,
\]
where we wish to learn the parameters $\alpha$ and $\beta$.
Assuming a Gaussian likelihood with mean 0 and variance 1 on the residuals $\delta
= y - \hat{y}$ ,
\[
  p(\delta \mid x, \alpha, \beta) = \Norm{\delta \mid 0, 1},
\]
a popular way of fitting the model is using Maximum Likelihood Estimation (MLE),
i.e. maximizing $\prod_i p(\delta_i \mid x_i, \alpha, \beta)$ which is equivalent to
minimizing the negative logarithm of this quantity, $-\sum_i \log p(\delta_i \mid
x_i, \alpha, \beta)$. It can be easily seen that this works out to be equivalent
to minimizing the Mean Squared Error (MSE) between the predicted values and the
targets:
\[
  L(\Data \mid \alpha, \beta) = \frac{1}{n} \sum_i (y_i - f(x_i))^2.
\]
A usual issue with MLE algorithms is that they are heavily overparameterized for
the problem they are supposed to be solving and hence can easily overfit (this
is most likely not an issue with our toy model, but we shall pretend for the
sake of the argument). To solve this issue, a standard technique is to
introduce some regularisation term to the loss. Here we are interested in
applying the MDL principle directly.
\par
Before we discuss how it is applied, we must make precise the setting in which
it \textit{can} apply. In particular, the MDL principle assumes the form of a
communications problem. Assume two parties, Alice and Bob share $\X$, and some
other arbitrary pre-agreed information, but only Alice has access to $\Y$. Then,
the MDL principle asks for the minimal message that Alice needs to send to Bob,
such that he may recover $\Y$ completely. With this setup in mind, we can continue.
\par
To apply the MDL principle, we need to be able to calculate the
MDLs of the data given a hypothesis and the MDLs of our hypotheses.
Notice, that the former is already available
in the form of the MSE for a given hypothesis, and hence $L(\Data \mid \alpha,
\beta)$ is not an overload of notation. To code the hypothesis
(the pair ($\alpha, \beta$) in our
case), we need to define two distributions over our parameters: a prior $P_\theta$, that
gives us the regularizing effect and stays fixed, and a posterior $Q_\phi$, the
distribution that we learn and assume that our parameters come
from it. We use the $\theta$ and $\phi$ to denote the sufficient
statistics of the prior and posterior, respectively. Now, learning changes, as
we are no longer optimizing a single hypothesis ($\alpha, \beta$), but a whole
class of hypotheses $Q_\phi(\alpha, \beta)$, by finding the best fitting set
of sufficient statistics $\phi$ for our dataset. Thus, our initial data
description length now becomes an expectation over the possible hypotheses:
\[
  L(\Data \mid \phi) = \Exp{L(\Data \mid \alpha, \beta)}{Q_\phi}.
\]
Defining the regularizing term, however, turns out to be trickier than expected,
and lies at the core of the bits-back argument. We seek to find the minimum
description length of a hypothesis $(\alpha, \beta)$. Using a $Q_\phi$, we know
we can encode a concrete hypothesis in $-\log Q_\phi (\alpha, \beta)$ nats, and thus a
reasonable first guess for the MDL would be
\begin{equation}
\label{eq:hypothesis_entropy}
  \Exp{-\log Q_\phi(\alpha, \beta)}{Q_\phi},
\end{equation}
i.e. the Shannon entropy of $Q_\phi$.
This turns out to be wrong, however, for the reason that Bob should be able to
decode Alice's message, and since he does not have access to $Q_\phi$, he cannot
do this.
At this point, we note, that as $P_\theta$ is fixed, we may assume that Alice
and Bob share it a priori. This allows us to code a pair $(\alpha, \beta)$ in
$-\log P_\theta(\alpha, \beta)$ nats that Bob can decode, and hence a
reasonable second guess for the MDL could be
\begin{equation}
\label{eq:hypothesis_cross_entropy}
\Exp{-\log P_\theta(\alpha, \beta)}{Q_\phi},
\end{equation}
i.e. the cross-entropy between $Q_\phi$ and $P_\theta$, which is also the
expected length of the actual message that gets sent to communicate the
parameters. Note, that since the hypotheses are still drawn from $Q_\phi$,
the expectation needs to be taken over it. This also turns out to be wrong,
as the bits-back argument shows that \textit{not all} of the bits used for
the message are used to code $Q_\phi$. Once the parameters are sent, Alice also
sends every residual $\delta_i$, obtained by using the parameter set sent to Bob.
\par
Once Bob has decoded $(\alpha, \beta)$ and each $\delta_i$,
he can fully recover each $y_i$ by calculating $x_i + \delta_i$. Now, since he
has access to both $\X, \Y$ and $P_\theta$, he may also fit a $Q_\psi$ to the
data, using the same learning algorithm as Alice used to fit her $Q_\phi$. The
key observation in (\cite{hinton1993keeping}) is that so long as this learning
algorithm is \textit{deterministic} after sufficient training Bob can achieve
$\psi = \phi$, i.e. he recovers Alice's posterior distribution. This means that
Bob can sample the same $(\alpha, \beta)$ pair that was sent to him (e.g.
by also sharing a random seed with Alice either before their communication or
during, at at most an $\Oh(1)$ cost, which is negligible).
\begin{framed}
This must mean, that
Alice not only communicated $Q_\phi$ itself to Bob, but also the \textit{random
  bits} that were used in conjunction with $Q_\phi$ to draw the sample $(\alpha,
\beta)$.
\end{framed}
The fact that Alice has communicated both $Q_\phi$ and the random bits in a $-\log
P_\theta(\alpha, \beta)$ nat long message, means that to get the cost
of communicating $Q_\phi$ only, we simply need to subtract the length of the
random bits. But since $(\alpha, \beta)$ were drawn from $Q_\phi$, their length
is going to be precisely $-\log Q_\phi(\alpha, \beta)$. Hence, the expected
hypothesis description length is the expectation of this difference, namely
\[
  \Exp{-\log P_\theta (\alpha, \beta) - (-\log Q_\phi(\alpha, \beta))}{Q_\phi}
  =  \Exp{\log \frac{Q_\phi(\alpha, \beta)}{P_\theta (\alpha, \beta)}}{Q_\phi}
  = \KL{Q_\phi}{P_\theta}.
\]
Above the rightmost term is called the \textbf{Kullback-Leibler Divergence}
between $Q_\phi$ and $P_\phi$. It is defined as
\[
  \KL{Q}{P} = \sum_{x \in \Omega} Q(x)\log\frac{Q(x)}{P(x)}
\]
for probability mass functions $Q$ and $P$, where $\Omega$ denotes the sample
space, and
\[
  \KL{q}{p} = \int_\Omega q(x) \log\frac{q(x)}{p(x)} \d x
\]
for probability density functions $q$ and $p$.
\par
The fact that Bob can ``get the random bits back ''used in sampling the
hypothesis is the namesake of the argument.

\paragraph{The general argument}
We are now ready to state the general bits-back argument. Assume Alice has
trained a model for a regression problem, on a dataset $\Data = (\X, \Y)$, with
training pairs $(\vec{x}_i, \vec{y}_i)$, and shares $\X$ with Bob. Her model has
parameters $\vec{w}$, with prior $p_\theta(\vec{w})$, and uses the likelihood
function $p(\vec{y} \mid \vec{w}, \vec{x})$, both shared with Bob. Assume that
Alice has a learned posterior $q_\phi(\vec{w} \mid \Data)$ over the weights, and now
wishes to communicate the targets $\Y$ to Bob.
\par
Then, the bits-back argument states that if Alice acts according to the MDL
principle, then she can communicate $q_\phi$ to Bob in $\KL{q_\phi}{p_\theta}$
nats, as follows:
\begin{enumerate}
\item Alice draws a random sample $\hat{\vec{w}} \sim q_{\phi}(\vec{w})$. This
  represents a message of $- \log q_\phi(\hat{\vec{w}})$ nats.
\item $\hat{\vec{w}}$ is then used to calculate the residuals $\vec{r}_i$ between
  the model's output and the targets.
\item $\hat{\vec{w}}$ is coded
  using its prior $p_\theta$, and sent to Bob alongside the residuals
  $\vec{r}_i$.
  The total length of the message that contains
  the posterior information is hence $-\log p_{\theta}(\hat{\vec{w}})$.
\item Bob, decodes $\hat{\vec{w}}$ using the same prior $p_\theta$. He then
  recovers all targets $\Y$ by adding each $\vec{r}_i$ to his model's output with
  parameters set to $\hat{\vec{w}}$ upon input $\vec{x}_i$.
\item He then trains his model using the same deterministic algorithm as Alice
  did, to recover Alice's posterior $q_\phi$. Hence, the random bits that were
  used to communicate the sample must be deducted from the cost of
  communicating $q_\phi$. The cost of these bits is precisely $-\log
  q_\phi(\hat{\vec{w}})$. Taking the expectation of the difference w.r.t. $q_\phi$,
  the total cost of communicating $q_\phi$ is
  \[
    \Exp{\log q_\phi(\vec{w}) - \log p_\theta(\vec{w})}{q_\phi} = \KL{q_\phi}{p_\theta}.
  \]
\end{enumerate}

\paragraph{Caveats of the argument}
Note, that the original argument merely derives the minimum description length
for the weights $\vec{w}$, but clearly does not achieve it (as we have to send a
message whose expected length is $\Exp{-\log p_\theta(\vec{w})}{q_\phi}$). The
authors merely state that these bits can be ``recovered'', and propose that a
``free'' auxiliary message might be coded in them, but do not give any
propositions as to how sending these bits in the first place might be avoided.
Nonetheless, as the notion of bit-back efficiency has expanded in recent years,
it is customary to call any method \textit{bits-back efficient} that transmits
some information in $\KL{q}{p}$ nats, for some posterior $q$ and prior $p$ over
the information.

\section{Compression without Quantization}
\label{sec:compression_without_quantization}
\par
In this section, we present a general framework for lossy data compression, based on
the arguments presented above, as well as the works of
\cite{harsha2007communication} and \cite{havasi2018minimal}.

\par
As mentioned at the end of the previous section, the bits-back argument
postulates that communicating the distribution of the parameter set of a model
may be achieved in $K = \KL{q(\vec{w})}{p(\vec{w})}$ nats, where $q$ and $p$ are the
posterior and prior over the parameters, respectively. However, they do not give
a method for achieving this, rather they show that only $K$ nats are used to
communicate the posterior in a longer message. Furthermore, the original MDL setup
also requires to send the residuals from the model output.
\par
For compression, however, we are only interested the communicating a sample and
not its distribution, though still at bits-back efficiency.
The correct communication problem for this was formulated by
\cite{harsha2007communication}, and it is as
follows:
\begin{framed}
Let $X$ and $Y$ be two correlated random variables, with sample spaces
$\X$ and $\Y$ respectively, and with joint distribution $p(X, Y) = q(Y \mid
X)p(Y)$. Given a concrete $x \in \X$, what is the minimal
message Alice needs to send to Bob, such he can generate a sample according to
the distribution $q(Y \mid X = x)$?
\end{framed}
\par
We can interpret $\X$ as the set of all data that we might wish to compress
(e.g. the set of all RGB-coded natural images, the set of all MP3 coded audio
files, etc.), and $\Y$ as the set of latent codes of the data, from which we may
obtain our lossy reconstruction.
\par
The solution to the above problem requires essentially the same mild assumptions
the bits-back argument does, namely that Alice and Bob are allowed to
share a fixed prior $p(Y)$ on the latent codes, as well as the seed used for
their random generators. The significance of the latter assumption is that Alice
and Bob will be able to reconstruct the same sequence of random numbers. Given
these assumptions, \cite{harsha2007communication} propose a rejection sampling
algorithm to sample from $q(Y \mid X = x)$ using $p(Y)$, depicted in Algorithm
\ref{alg:harsha_rej_sampling} in the Appendix. Alice uses this algorithm to
sample $q$, but she also keeps track of the number of proposals made by the
algorithm. Once Alice's algorithm accepts a proposal from $p$, it is sufficient
for Alice to communicate the sample's index $K$ to Bob. Bob can then obtain the
desired sample from $q$, by simply drawing $K$ samples from $p$, and since he
can generate the same $K$ samples as Alice did, the $K$th sample he draws is
going to be an exact sample from $q$. The communication cost of $K$ is
$\log K$ nats. \cite{harsha2007communication} then also prove the following result.
\begin{theorem}{(\cite{harsha2007communication})}
  \label{thm:bits-back_efficiency}
Let $X$ and $Y$ be random variables as given above. And let the communication
problem be set as above. Let $T[X : Y]$ denote the MDL (in nats) of a sample
$Y=y \sim q(Y \mid X=x)$. Then,
\begin{equation}
\label{eq:harsha_upper_bound}
  I[X : Y] \leq T[X : Y] \leq I[X : Y] + 2\log \left[ I[X : Y] + 1 \right] + \Oh(1),
\end{equation}
where $I[ X : Y ]$ is called the mutual information between $X$ and $Y$, and is
defined as
\[
  I[ X : Y ] = \Exp{\KL{q(Y \mid X)}{p(Y)}}{p(X)}
\]
Furthermore, $\log K$, given by Algorithm \ref{alg:harsha_rej_sampling},
achieves the upper bound in Eq \ref{eq:harsha_upper_bound}.
\end{theorem}
\par
The above theorem tells us that while in the classical sense bits-back efficiency
is the best that we can do, it also tells us that we can get very close to it.
Hence, from now on, we shall refer to any algorithm that achieves this tight
upper bound as bits-back efficient as well.
\par
To translate this to a general ML-based compression framework, we shall switch
to notation more common in statistical modelling, concretely, we shall denote
our data by $\vec{x}$ and the latent code $\vec{z}$. Now, let us assume a
generative model over these variables, $p(\vec{x}, \vec{z}) = p(\vec{x} \mid
\vec{z})p_\theta(\vec{z})$, where $p(\vec{x} \mid \vec{z})$ is the data likelihood, and
$p_\theta(\vec{z})$ is the prior over the latent code, with sufficient
statistics $\theta$. Let us also assume an approximate posterior $q_\phi(\vec{z}
\mid \vec{x})$ over the latent code, with sufficient statistics $\phi$. Then our
framework is as follows:
\begin{enumerate}
\item Given some dataset $\Data = \{\vec{x}_1, \hdots, \vec{x}_n\}$ where the
  training examples are distributed according to $p(\vec{x})$, we fit our
  generative model to it, by fitting $\theta$ and $\phi$ using the (weighted) MDL
  objective:
  \begin{equation}
    \label{eq:mdl_elbo}
    \begin{gathered}
      L(\Data) = \Exp{L(\vec{x})}{p(\vec{x})}, \\
      \text{where } L(\vec{x}) = \Exp{L(\vec{x} \mid \vec{z}) + L(\vec{z})}{q_\phi}
      =  -\Exp{\log p(\vec{x} \mid \vec{z})}{q_\phi} + \beta\KL{q_\phi}{p_\theta}.
    \end{gathered}
  \end{equation}
  This training objective is well known in the neural generative modelling
  literature as the Evidence Lower Bound (ELBO) (\cite{kingma2013auto},
  \cite{higgins2017beta}). The expectation over $p(\vec{x})$ is usually taken over
  randomly drawn mini-batches from $\Data$ using Stochastic Gradient Descent (SGD).
  Here $\beta$ is a
  hyperparameter that can be set to trade off a smaller description length at
  the cost of worse reconstruction, or the other way around, thus allowing the
  user to reach different points on the rate-distortion curve. See Section
  \ref{sec:derive_weighted_elbo} for the derivation of Eq \ref{eq:mdl_elbo} and
  discussion on its validity.

  \item Once $\theta$ and $\phi$ have been learned, we fix them (equivalent to
    sharing them with Bob in the communication problem).

  \item Now, if we wish to compress some new data $\vec{x}'$, use a bits-back
    efficient sampling algorithm (such as Algorithm
    \ref{alg:harsha_rej_sampling}) to sample $q(\vec{z} \mid \vec{x}')$ using
    $p(\vec{z})$, and use the code output of the sampling algorithm as the
    compression code, along with the random seed that was used to obtain the sample.
    We shall refer to such algorithms as \textbf{coded sampling algorithms}.

  \item To decompress, since we always have access to the fixed prior
    $p_\theta$, and we have the random seed the compressing party used, we may
    run the coded sampling algorithm in ``decode'' mode to recover the sample $\vec{z}'$
    from $q_\phi$. Finally, we may run the reconstruction transformation of our
    generative model to recover a lossy reconstruction $\hat{\vec{x}}'$.
\end{enumerate}

\par
This framework is inspired by the work of \cite{havasi2018minimal}, where they
used a very similar framework to achieve state-of-the-art weight compression in
Bayesian Neural Networks.
\par
In this thesis, we use this framework to train $\beta$-VAEs as our choice of
generative models and demonstrate the efficiency of our method compared to the
state-of-the-art in neural compression. More details on this will be given in
Chapter \ref{chapter:method}.

\subsection{Relation of Quantization to Our Framework}
\par
We present a similar argument to the one given in \cite{havasi2018minimal}.
Recall the original representation space $R$ and quantized space $S$ of a
quantizer $[\cdot]$. Recall also the Kronecker delta function on $x$, defined as
\[
  \delta_{x}(y) =
  \begin{cases}
    1 & \text{if } y = x \\
    0 & \text{otherwise}.
  \end{cases}
\]
Given a particular $x \in R$, we have seen that
quantization allows us to code it in $-\log \hat{Q}([x])$ nats. If we manipulate
this term slightly, we get
\begin{align*}
  -\log \hat{Q}([x]) &= \sum_{s \in S} \left[ -\delta_{[x]}(s)\log \hat{q}([x]) + \underbrace{\delta_{[x]}(s) \log \delta_{[x]}(s)}_{= 0} \right] \\
                     &= \sum_{s \in S} \delta_{[x]}(s)\log\frac{\delta_{[x]}(s)}{\hat{q}([x])} \\
                     &= \KL{\delta_{[x]}}{\hat{Q}}.
\end{align*}
This shows that quantization of a deterministic parameter set is also
bits-back efficient, with the posterior distribution family restricted to point masses. Thus
the clear advantage of our framework comes from the fact that we allow much more
posteriors than point masses.

\subsection{Derivation of the Training Objective}
\label{sec:derive_weighted_elbo}
\par
In this section, we present the derivation of Eq \ref{eq:mdl_elbo}. Thus, let our
likelihood $p(\vec{x} \mid \vec{z})$, our latent prior $p_\theta(\vec{z})$ and
approximate posterior $q_\phi(\vec{z} \mid \vec{x})$ be given. Then, given a budget
of $C$ nats, we want to optimize the following constrained objective on the
description lengths:
\[
  \Exp{\Exp{-L(\vec{x} \mid \vec{z})}{q_\phi(\vec{z})}}{p(\vec{x})}
  \quad \text{subject to } \Exp{L(\vec{z})}{p(\vec{x})} < C.
\]
As we have seen in the sections above, these quantities can be replaced by
\begin{equation}
\label{eq:framework_hard_train_target}
\Exp{\Exp{\log p(\vec{x} \mid \vec{z})}{q_\phi(\vec{z})}}{p(\vec{x})}
\quad \text{subject to } \Exp{\KL{q_{\phi}(\vec{z} \mid
    \vec{x})}{p_{\theta}(\vec{z})}}{p(\vec{x})} < C.
\end{equation}

As we want to use gradient-based optimization of our models, we need to find a
continuous relaxation of Eq \ref{eq:framework_hard_train_target}.
To this end, we rewrite the terms inside the ``outer'' expectation as their
Lagranagian relaxation under the KKT conditions (\cite{karush2014minima},
\cite{kuhn2014nonlinear}, \cite{higgins2017beta}) and get:
\[
  \F(\theta, \phi, \beta, \vec{x}) =
  \Exp{\log p(\vec{x} \mid \vec{z})}{q_\phi(\vec{z})}
  - \beta (\KL{q_{\phi}(\vec{z} \mid \vec{x})}{p_{\theta}(\vec{z})} - C).
\]
By the KKT conditions if $C \geq 0$ then $\beta \geq 0$, hence discarding the last
term in the above equation will provide a lower bound for it:
\[
  \F(\theta, \phi, \beta, \vec{x}) \geq
  \L(\theta, \phi, \beta, \vec{x}) =
  \Exp{\log p(\vec{x} \mid \vec{z})}{q_\phi(\vec{z})} - \beta
  \KL{q_{\phi}(\vec{z} \mid \vec{x})}{p_{\theta}(\vec{z})}.
\]
Finally, taking the expectation over this again gives
\begin{equation}
\label{eq:framework_train_target}
\begin{aligned}
  \Exp{\L(\theta, \phi, \beta, \vec{x})}{p(\vec{x})} &=
  \Exp{\Exp{\log p(\vec{x} \mid \vec{z})}{q_\phi(\vec{z})}}{p(\vec{x})} - \beta
  \Exp{\KL{q_{\phi}(\vec{z} \mid \vec{x})}{p_{\theta}(\vec{z})}}{p(\vec{x})} \\
  &= \Exp{\Exp{\log p(\vec{x} \mid \vec{z})}{q_\phi(\vec{z})}}{p(\vec{x})} - \beta
  I[ \vec{x} : \vec{z} ] \\
  &= \Exp{L(\vec{x})}{p(\vec{x})}.
\end{aligned}
\end{equation}
This is the training objective of $\beta$-VAEs first derived in
\cite{higgins2017beta}, although we note that it is applicable any generative
model where the assumed conditions are present.
\par
An important caveat of the above formulation is that the samples from
$p(\vec{x})$ should comparable, in the sense the initial hard optimization objective
of setting an average nat budget is reasonable. In the case of image data, if
all images are the same size, this is fine, as our continuous relaxation will
allow for images with high information content to have slightly longer code
lengths than $C$ nats and ones with low information content will have shorter
lengths. However, if we used different sized images during training, it would be
less justified to set the same average code budget for, say, a $200 \times 300$ pixel
image and a $2000 \times 2000$ image, as the latter will naturally contain more
information than the former. Hence, in this case, it would be more reasonable
to make the budget a function of the number of pixels the image contains,
although this might make the formulation of the training objective much harder.
\par
An approach taken in all neural image compression methods we examined is instead
to train on random, but equal-sized patches extracted from each training image.
While other works do this to make training more computationally feasible. As
far as we are aware, we are the first ones to argue that this practice is not only
convenient but mandatory for the training procedure to be sound.

\nomenclature[z-VAE]{VAE}{Variational Auto-Encoder}
\nomenclature[z-MSE]{MSE}{Mean Squared Error}
\nomenclature[z-MAE]{MAE}{Mean Absolute Error}
\nomenclature[z-PLN]{PLN}{Probabilistic Ladder Network}