\documentclass[]{article}

% insert here the call for the packages your document requires
\usepackage[utf8]{inputenc}
% Use nameref to cite supporting information files (see Supporting Information section for more info)
\usepackage{nameref,hyperref}
% amsmath and amssymb packages, useful for mathematical formulas and symbols
\usepackage{amsmath,amssymb}
% useful for consistent display and control of units of measurement
\usepackage{siunitx}
% figures!
\usepackage{graphicx}
% for including TODO notes
\usepackage{todonotes}
% syntax highlighting for code
\usepackage{minted}
% Uncomment next two lines to add line numbers.
\usepackage{lineno}
\linenumbers

\begin{document}

\title{Supplementary Materials for Safety-Conscious Design of Terrain Park
Jumps: Ethical Issues and Online Software}

\author{
  Jason~K. Moore \and
  Bryn Cloud \and
  Mont Hubbard \and
  Christopher~A. Brown
}

\maketitle

\section{Example Software Library Use}
\label{sec:example}
%
The closed form equation
%
\begin{align}
  h = \left[\frac{x^2}{4(x\tan\theta_T - y)\cos^{2}\theta_T} - y\right]
    \sin^{2}
    \left[\tan^{-1}\left(\frac{2y}{x} - \tan\theta_T\right) -
    \tan^{-1}\frac{dy}{dx}\right]
  \label{eq:efh}
\end{align}

is useful for understanding the fundamental relationship of equivalent fall
height (EFH) to the landing surface shape. It will predict EFH for small jumps
but other factors may be useful to include in the model. For example, jumpers
are subject to aerodynamic drag and this is not negligible for larger jumps. If
drag is included there is no closed form solution for the EFH, but the EFH can
be computed through iterative simulation~\cite{Levy2015}. The jumper's flight
path is found by integrating the flight equations of motion at various takeoff
velocities and computing the misalignment of jumper landing and slope angles to
then compute the EFH. This more general simulation method is implemented in the
software described herein and the results reflect the inclusion of both
gravitational and drag forces. Even with drag incorporated, the calculating EFH
still only require measurements of the landing surface cross-sectional profile
coordinates $(x,y)$ relative to the takeoff point and a measurement of the
takeoff angle.  Listing~\ref{lis:example-efh-calc} demonstrates the new
software library features creating a surface from some measured data points and
then calculating the EFH at 0.2\si{\meter} increments.
%
\begin{listing*}
  \begin{minted}{pycon}
>>> import numpy as np
>>> from skijumpdesign import Surface, Skier, plot_efh
>>> takeoff_ang = 10  # degrees
>>> takeoff_point = (0, 0)  # (x, y) in meters
>>> x_ft = np.array([-232.3,-203.7,-175.0,-146.3,-117.0,-107.4,
...    -97.7,-88.0,-78.2,-68.5,-58.8,-49.1,-39.4,-34.5,-29.7,
...    ...
...    38.8,43.3,47.8,52.3,56.8,61.5,66.2,70.9,75.7,80.6,85.5,
...    88.4,88.4])
...
>>> y_ft = np.array([55.5,46.4,37.7,29.1,22.2,19.7,17.2,14.8,
...    12.5,10.2,7.7,5.2,2.9,1.8,0.7,-0.2,-1.0,-1.2,-1.4,-1.6,
...    ...
...    -16.2,-18.1,-19.8,-21.4,-22.9,-24.0,-25.0,-25.6,-25.6])
...
>>> x_mt = x_ft*0.3048 # convert to meters
>>> y_mt = y_ft*0.3048 # convert to meters
>>> # create a surface from the data
>>> measured_surf = Surface(x_mt, y_mt)
>>> # create a skier
>>> skier = Skier(mass=75.0, area=0.34, drag_coeff=0.821)
>>> # calculate the EFH
>>> x, efh, v = measured_surf.calculate_efh(
...     np.deg2rad(takeoff_ang), takeoff_point, skier, increment=0.2)
...
>>> x  # display the x coordinates
array([ 0. ,  0.2,  0.4,  0.6,  0.8,  1. ,  1.2,  1.4,  1.6,  1.8,  2. ,
        2.2,  2.4,  2.6,  2.8,  3. ,  3.2,  3.4,  3.6,  3.8,  4. ,  4.2,
       ...
       24.2, 24.4, 24.6, 24.8, 25. , 25.2, 25.4, 25.6, 25.8, 26. , 26.2,
       26.4, 26.6, 26.8])
>>> efh  # display the EFH for each x coordinate
array([0.        , 0.02541035, 0.03479384, 0.03264587, 0.05956476,
       0.09096091, 0.12358184, 0.13702364, 0.15202999, 0.17018343,
       ...
       3.93910556, 3.97387212, 4.00891899, 4.04424779, 4.07984952,
       4.11573359, 4.68049185, 5.53413479, 6.45253722, 7.42628019])

>>> v  # display takeoff speeds to reach x positions
array([0.07373847, 0.13081777, 0.1878382 , 0.2447865 , 0.30166299,
       0.35851949, 0.41537661, 0.47221055, 0.52897197, 0.58564902,
       ...
       6.71699974, 6.76760188, 6.81816819, 6.86869777, 6.9191902 ,
       6.96962124, 7.02001551, 7.07037288, 7.1206941 ])
>>> # calculate and plot the efh curve
>>> plot_efh(measured_surf, takeoff_ang, takeoff_point, increment=0.2)
  \end{minted}
  \caption{Python interpreter session illustrating how one could compute the
  EFH of a measured jump.}
  \label{lis:example-efh-calc}
\end{listing*}

\section{Jump Shape Measurement}
\label{sec:jump-shape-measurement}
%
Calculating EFH requires the Cartesian coordinates and slope of the landing
surface along the path of the jumper. There are a number of possible
measurement techniques for collecting data adequate for the EFH calculation but
the simplest method requires only a digital level \footnote{Smartphone digital
level measurement applications are likely sufficient and readily available.}, a
flexible tape measure, and less than an hour's time from one person per jump. A
tenth of a degree accuracy from the level and down to 25~\si{\centi\meter}
accuracy from the tape measure should be more than sufficient for typical
snowsport jumps.

To measure the jump, the takeoff point should be identified and the tape
measure should then be draped over the contour of the landing surface along the
projection of the expected flight path onto the landing surface. The origin of
the tape measure should be aligned with the takeoff point. Starting with the
takeoff point, the digital level should be used to record the absolute angle at
regular increments along the tape. The increment can be varied between
25~\si{\centi\meter} and 100~\si{\centi\meter}, with the former used for steep
slope changes and the later for less steep; 50~\si{\centi\meter} increments are
appropriate for average jump shapes. Positive angles should be recorded for
positive slope and negative angles for negative slope. The tabulated data
should include the distance along the surface from the takeoff point, $d_i$,
and the associated surface angle, $\theta_i$, at each distance measurement for
$n$ measurements. Assuming $\theta_i$ is in radians, the Cartesian coordinates
can be computed using the average angle to find the adjacent coordinates. The
following equations show the calculation of the Cartesian coordinates from
these two measures used in the software.
%
\begin{align}
  \frac{dy_i}{dx_i} & = \tan^{-1}{\theta_i} \quad \text{for } i=1\ldots n \\
  x_{i + 1} & =
  \begin{cases}
    0 & \text{for } i=0 \\
    x_i + (d_{i+1} - d_i)\cos{\frac{\theta_{i+1} + \theta_i}{2}} &  \text{for }
    i=1\ldots n-1
  \end{cases} \\
  y_{i + 1} & =
  \begin{cases}
    0 & \text{for } i=0 \\
    y_i + (d_{i+1} - d_i)\sin{\frac{\theta_{i+1} + \theta_i}{2}} &  \text{for }
    i=1\ldots n-1
  \end{cases}
\end{align}

Listing~\ref{lis:example-meas-calc} demonstrates calculating the landing
surface's Cartesian coordinates from measured distance and angle data collected
with the method described above.
%
\begin{listing*}
  \begin{minted}{pycon}
>>> import numpy as np
>>> from skijumpdesign import cartesian_from_measurements
>>> dis = np.array([14.5, 15.0, 15.5, 16.0, 16.5, 17.0])  # meters
>>> ang = np.deg2rad([4.6, -7.4, -16.5, -9.7, -11, -6.9])  # radians
>>> x, y, to_point, to_angle = cartesian_from_measurements(dis, ang)
>>> print(x)  # meters
[0.         0.49985074 0.98901508 1.47600306 1.96786738 2.46177962]
>>> print(y)  # meters
[ 0.         -0.01221609 -0.1157451  -0.22907075 -0.31890113 -0.39668737]
>>> print(to_point)  # takeoff point in meters
(0.0, 0.0)
>>> print(to_angle)  # takeoff angle in radians
0.08028514559173916
  \end{minted}
  \caption{Python interpreter session showing how one could compute the
  Cartesian coordinates from EFH of a measured jump.}
  \label{lis:example-meas-calc}
\end{listing*}

\bibliographystyle{ieeetr}
\bibliography{references}   % name your BibTeX data base

\end{document}


% !TEX root=/home/tavant/these/manuscript/src/manuscript.tex

\section{Sheath model with electron emission}
  \label{sec-sheath}

  The floating sheath is the plasma response to the presence of an absorbing boundary.
  The light electrons are quickly absorbed by the material.
  Hence, in order to balance the particles fluxes, a non-neutral region appears between the plasma and the boundary.

  The sheath model featuring SEE processes has been historically studied by \citet{hobbs1967}, but is still an active research topic nowadays \citep{ahedo2005}.
  The sheath is often considered to be collisionless and isothermal, while the plasma is composed of hot Maxwellian electrons and cold ions. A third population of electron-induced secondary electrons is also present in the sheath, and the re-emission rate $\rate$ is assumed to be constant.
  The sheath must ensure the charge balance, hence
  \begin{equation} \label{eq-sheath_SEE}
    \Gamma_i = ( 1 - \rate) \Gamma_e
  \end{equation}
  with $\Gamma_i$ and $\Gamma_e$ the ion and electron particle flux to the wall, respectively.
  \nomenclature[Q]{\ensuremath{ \Gamma_s}}{Particle flux of the species $s$}
  The SEE process modifies the potential drop in the sheath as \citep{hobbs1967}
  \begin{equation} \label{eq-sheathhobbs}
    \dphisheath = \Te \ln \lp (1 - \rate) \sqrt{\frac{m_i}{2 \pi m_e}}   \rp.
  \end{equation}
  Adding a pre-sheath drop of $\Te/2$ \citep{ahedo2002}, the total potential drop to the wall becomes

  \begin{equation} \label{eq-total_drop}
    \dphi = \dphisheath + \frac{\Te}{2} =  \Te \lp \frac{1}{2} + \ln \lb (1 - \rate) \sqrt{\frac{m_i}{2 \pi m_e}}   \rb  \rp
  \end{equation}

  We can see that \cref{eq-sheathhobbs} becomes negative for a critical value of the emission rate
  \begin{equation} \label{eq-ratecrone}
    \rate_{\rm max} = 1 - \sqrt{\frac{2 \pi m_e}{m_i}} \simeq 0.985 \text{ for xenon.}
  \end{equation}

  However, before that $\rate$ reaches $\rate_{\rm max}$, the model of \citet{hobbs1967} presents another behavior against the hypotheses of  \cref{eq-sheathhobbs}, as the sheath becomes \ac{SCL}.
  In the \ac{SCL} conditions, the electron emission is so large that the electric field at the wall becomes zero
  \begin{equation} \label{eq-scl_Er}
    \deriv{\phi}{r} \at{\rm wall}= 0
  \end{equation}
  In this case, the plasma potential drop to the wall for any ion mass is \citep{hobbs1967}
  \begin{equation} \label{eq-dphi_scl}
    \dphiscl \simeq 1.02 \Te \simeq \Te,
  \end{equation}
  and the limit emission rate is
  \begin{equation} \label{eq-ratecr}
    \ratecr \simeq 1 - 8.3 \sqrt{\frac{m_e}{m_i}}
  \end{equation}

  For xenon, \cref{eq-ratecr} gives $\ratecr = 0.983$ \citep{goebel2008}.
  \Cref{fig-potential_profile} illustrates the sheath model of \citet{hobbs1967} by showing the plasma potential profile in the sheath for different values of electron emission rate for a xenon plasma.
  \cref{fig-potential_profile}.{\bf a} gives the potential $\phi$ normalized by $\dphisheath$ from \cref{eq-sheathhobbs}, and \cref{fig-potential_profile}.{\bf b} gives $\phi$ normalized by $\Te$.
  We can see in \cref{fig-potential_profile}  that for $\rate < \ratecr$, the plasma potential reached $\dphisheath$.
  However, for $\rate > \ratecr$, the plasma potential does not reach $\dphisheath$, resulting in a non-zero current to the wall.
  Indeed, the sheath is not monotonic in this case, hence the hypotheses needed to develop \cref{eq-sheathhobbs} are not fulfilled.
  We see in \cref{fig-potential_profile}.{\bf a} that \cref{eq-sheathhobbs} is correct even for $\rate=0.984 > \ratecr$.
  This is because the value  $\ratecr = 0.983$ is obtained after several approximations, as \cref{eq-scl_Er}, some of which should not allow to use three significant digits.

  In \cref{fig-potential_profile}.{\bf b}, we can see that at $\rate \simeq \ratecr$, the plasma potential tends towards $\Te$, as mentioned by \cref{eq-dphi_scl}.
  \begin{figure}[hbt]
    \centering
    \begin{tabular}{@{} c c}
      \subfigure{plasma_profile_normed}{a}{25,70} &
      \subfigure{plasma_profile}{b}{25,20}
    \end{tabular}
    \caption{Evolution of the plasma potential in the sheath for different values of electron emission rate $\rate$ for a xenon plasma, ({\bf a}) normalized by the total potential drop $\dphisheath$ from \cref{eq-sheathhobbs}, and ({\bf b}) normalized by the electron temperature, but $\dphisheath/\Te$ is noted with the black dotted line.  }
    \label{fig-potential_profile}
  \end{figure}


\section{Preconditioner routines}
\label{sec:precs}

% \section{Preconditioners}
\label{sec:psprecs}
The base PSBLAS library  contains the implementation of two simple
preconditioning techniques:
\begin{itemize}
\item Diagonal Scaling
\item Block Jacobi with ILU(0) factorization
%% \item Additive Schwarz with the Restricted Additive Schwarz and
%%   Additive Schwarz with Harmonic extensions;
\end{itemize}
The supporting data type and subroutine interfaces are defined in the
module  \verb|psb_prec_mod|.
The old interfaces \verb|psb_precinit| and  \verb|psb_precbld| are still supported for
backward compatibility
%% We also provide a companion  package of multi-level Additive
%%   Schwarz preconditioners called MD2P4; this is actually a family of
%%   preconditioners since there is the possibility to choose between
%%   many variants, and is currently in an experimental stateIts
%%   documentation is planned to appear after stabilization of the
%%   package, which will characterize release  2.1 of our library.

\clearpage\subsection{init --- Initialize a  preconditioner}

\begin{verbatim}
call prec%init(icontxt,ptype, info)
\end{verbatim}

\begin{description}
\item[Type:] Asynchronous.
\item[\bf On Entry]
\item[icontxt] the communication context.\\
Scope:{\bf global}.\\
Type:{\bf required}.\\
Intent: {\bf in}.\\
Specified as: an integer value.
\item[ptype] the type of preconditioner.
Scope: {\bf global} \\
Type: {\bf required}\\
Intent: {\bf in}.\\
Specified as: a character string, see usage notes.
%% \item[iv] integer parameters for the precondtioner.
%% Scope: {\bf global} \\
%% Type: {\bf required}\\
%% Specified as: an integer array, see usage notes.
%% \item[rs]
%% Scope: {\bf global} \\
%% Type: {\bf optional}\\
%% Specified as: a long precision real number.
\item[\bf On Exit]

\item[prec]
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf inout}.\\
Specified as: a preconditioner data structure \precdata.
\item[info]
Scope: {\bf global} \\
Type: {\bf required}\\
Intent: {\bf out}.\\
Error code: if no error, 0 is returned.
\end{description}
{\par\noindent\large\bfseries Notes}
%% The PSBLAS 2.0 contains a number of preconditioners, ranging from a
%% simple diagonal scaling to 2-level domain decomposition. These
%% preconditioners may use the SuperLU or the UMFPACK software, if
%% installed; see~\cite{SUPERLU,UMFPACK}.
Legal inputs to this subroutine are interpreted depending on the
$ptype$ string as follows\footnote{The string is case-insensitive}:
\begin{description}
\item[NONE] No preconditioning, i.e. the preconditioner is just a copy
  operator.
\item[DIAG] Diagonal scaling; each entry of the input vector is
  multiplied by the reciprocal of the sum of the absolute values of
  the coefficients in the corresponding row of matrix  $A$;
\item[BJAC] Precondition by a  factorization of the
  block-diagonal of matrix $A$, where block boundaries are determined
  by the data allocation boundaries for each process; requires no
  communication. Only the incomplete factorization $ILU(0)$ is
  currently implemented.
\end{description}

\clearpage\subsection{build --- Builds a preconditioner}

\begin{verbatim}
call prec%build(a, desc_a, info[,amold,vmold,imold])
\end{verbatim}

\begin{description}
\item[Type:] Synchronous.
\item[\bf On Entry]
\item[a] the system sparse matrix.
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf in}, target.\\
Specified as: a sparse matrix data structure \spdata.
\item[prec] the preconditioner.\\
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf inout}.\\
Specified as: an already initialized precondtioner data structure \precdata\\
\item[desc\_a] the problem communication descriptor.
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf in}, target.\\
Specified as: a communication descriptor data structure \descdata.
%% \item[upd]
%% Scope: {\bf global} \\
%% Type: {\bf optional}\\
%% Intent: {\bf in}.\\
%% Specified as: a character.
\item[amold] The desired dynamic type for the internal matrix storage.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an object of a class derived from \spbasedata.
\item[vmold] The desired dynamic type for the internal vector storage.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an object of a class derived from \vbasedata.
\item[imold] The desired dynamic type for the internal integer vector storage.\\
Scope: {\bf local}.\\
Type: {\bf optional}.\\
Intent: {\bf in}.\\
Specified as: an object of a class derived from (integer) \vbasedata.
\end{description}

\begin{description}
\item[\bf On Return]
\item[prec] the preconditioner.\\
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf inout}.\\
Specified as: a precondtioner data structure \precdata\\
\item[info] Error code.\\
Scope: {\bf local} \\
Type: {\bf required} \\
Intent: {\bf out}.\\
An integer value; 0 means no error has been detected.
\end{description}
The \verb|amold|, \verb|vmold| and \verb|imold|  arguments may be
employed to interface with special devices, such as GPUs and other
accelerators.

\clearpage\subsection{apply --- Preconditioner application
  routine}

\begin{verbatim}
call prec%apply(x,y,desc_a,info,trans,work)
call prec%apply(x,desc_a,info,trans)
\end{verbatim}

\begin{description}
\item[Type:] Synchronous.
\item[\bf On Entry]
\item[prec] the preconditioner.
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf in}.\\
Specified as: a preconditioner data structure \precdata.
\item[x] the source vector.
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf inout}.\\
Specified as:  a rank one array or an object of type \vdata.
\item[desc\_a] the problem communication descriptor.
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf in}.\\
Specified as: a communication data structure \descdata.
\item[trans]
Scope: {\bf } \\
Type: {\bf optional}\\
Intent: {\bf in}.\\
Specified as: a character.
\item[work] an optional work space
Scope: {\bf local} \\
Type: {\bf optional}\\
Intent: {\bf inout}.\\
Specified as: a double precision array.
\end{description}

\begin{description}
\item[\bf On Return]
\item[y] the destination vector.
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf inout}.\\
Specified as:  a rank one array or an object of type \vdata.
\item[info] Error code.\\
Scope: {\bf local} \\
Type: {\bf required} \\
Intent: {\bf out}.\\
An integer value; 0 means no error has been detected.
\end{description}

\clearpage\subsection{descr --- Prints a description of current
  preconditioner}

\begin{verbatim}
call prec%descr(info)
call prec%descr(info,iout, root)
\end{verbatim}

\begin{description}
\item[Type:] Asynchronous.
\item[\bf On Entry]
\item[prec] the preconditioner.
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf in}.\\
Specified as: a preconditioner data structure \precdata.
\item[iout] output unit.
Scope: {\bf local} \\
Type: {\bf optional}\\
Intent: {\bf in}.\\
Specified as: an integer number. Default: default output unit.
\item[root] Process from which to print
Scope: {\bf local} \\
Type: {\bf optional}\\
Intent: {\bf in}.\\
Specified as: an integer number between 0 and $np-1$, in which case
the specified process will print the description, or $-1$, in which case
all processes will print. Default: 0.
\item[\bf On Return]
\item[info] Error code.\\
Scope: {\bf local} \\
Type: {\bf required} \\
Intent: {\bf out}.\\
An integer value; 0 means no error has been detected.
\end{description}

\clearpage\subsection{clone --- clone  current
  preconditioner}
\begin{verbatim}
call  prec%clone(precout,info)
\end{verbatim}

\begin{description}
\item[Type:] Asynchronous.
\item[\bf On Entry]
\item[prec] the preconditioner.\\
Scope: {\bf local}.\\
% Type: {\bf required}.\\
% Intent: {\bf in}.\\
% Specified as: a object of type \descdata.
\end{description}

\begin{description}
\item[\bf On Return]
\item[precout] A copy of the input object.
\item[info] Return code.
\end{description}

\clearpage\subsection{free --- Free a  preconditioner}

\begin{verbatim}
call prec%free(info)
\end{verbatim}

\begin{description}
\item[Type:] Asynchronous.
\item[\bf On Entry]
\item[prec] the preconditioner.\\
Scope: {\bf local}.\\
Type: {\bf required}\\
Intent: {\bf inout}.\\
Specified as: a preconditioner data structure \precdata.
%% \item[iv] integer parameters for the precondtioner.
%% Scope: {\bf global} \\
%% Type: {\bf required}\\
%% Specified as: an integer array, see usage notes.
%% \item[rs]
%% Scope: {\bf global} \\
%% Type: {\bf optional}\\
%% Specified as: a long precision real number.
\item[\bf On Exit]

\item[prec]
Scope: {\bf local} \\
Type: {\bf required}\\
Intent: {\bf inout}.\\
Specified as: a preconditioner data structure \precdata.
\item[info]
Scope: {\bf global} \\
Type: {\bf required}\\
Intent: {\bf out}.\\
Error code: if no error, 0 is returned.
\end{description}
{\par\noindent\large\bfseries Notes}
%% The PSBLAS 2.0 contains a number of preconditioners, ranging from a
%% simple diagonal scaling to 2-level domain decomposition. These
%% preconditioners may use the SuperLU or the UMFPACK software, if
%% installed; see~\cite{SUPERLU,UMFPACK}.
Releases all internal storage.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "userguide"
%%% End:


\documentclass[12pt]{amsart}
\usepackage{geometry} % see geometry.pdf on how to lay out the page. There's lots.
\usepackage{bsymb}
\usepackage{unitb}
\usepackage{calculational}
\usepackage{ulem}
\usepackage{hyperref}
\normalem
\geometry{a4paper} % or letter or a5paper or ... etc
% \geometry{landscape} % rotated page geometry

% See the ``Article customise'' template for some common customisations

\title{}
\author{}
\date{} % delete this line to display the current date

%%% BEGIN DOCUMENT
\setcounter{tocdepth}{4}
\begin{document}

\maketitle
\tableofcontents

\newcommand{\G}{\text{G}}
\renewcommand{\H}{\text{H}}

\section{Initial model}
\begin{machine}{m0}

% \with{functions}
\with{sets}
% \with{intervals}
\newset{\G}

\begin{align*}
\variable{p : \Int } \\
\end{align*}

\newevent{evt0}{event\_0}
\newevent{evt1}{event\_1}

\begin{align*}
\evassignment{evt0}{m0:act0}{ p' = p + 1 }
\\ \evassignment{evt1}{m0:act0}{ p' = p-1 }
\end{align*}

\dummy{ k : \Int }
\begin{align*}
% \safetyB{saf0}{evt0}{ p \le k }{ \false }
\\ \safety{saf1}{ p \le k }{ \false }
\end{align*}

\end{machine}

\begin{machine}{m1}

\with{functions}
\with{sets}
% \with{intervals}
\newset{\G}

\begin{align*}
\variable{ f : \Int \pfun \Int } \\
\end{align*}

\newevent{evt0}{event\_0}
\newevent{evt1}{event\_1}
\indices{evt0}{ i : \Int }
\indices{evt1}{ i : \Int }

\begin{align*}
\evassignment{evt0}{m0:act0}{ f' = f | i \fun f.i - 1 | i+1 \fun f.(i+1) +1 }
\\ \evassignment{evt1}{m0:act0}{ f' = f | i \fun f.i + 1 }
\end{align*}

\dummy{ i,k : \Int }
\begin{align*}
% \safetyB{saf0}{evt0}{ k \le f.i }{ \false }
\\ \safety{saf1}{ k \le f.i }{ \false }
\end{align*}

\end{machine}

\end{document}


\hypertarget{introduction}{%
\section{Introduction}\label{introduction}}

\hypertarget{motivation}{%
\subsection{Motivation}\label{motivation}}

Important infrastructure systems such as electrical transmission grids, potable water distribution, and roadway transportation have been modeled as networks for analysis and design \cite{levy1967monte} \cite{cotilla2012comparing}. As these networks grow in size and become more complex, such as the addition of distributed power generation and energy storage in electrical networks \cite{escalera2018survey}, better algorithms are needed to analyze these networks and guarantee their safety and reliability. The tensor network contraction (TNC) algorithm we propose in this work is a step in this direction.

\hypertarget{problem-definition}{%
\subsection{Problem Definition}\label{problem-definition}}

A network is a graph consisting of nodes that describe a discrete component of the network (such as a power plant or household) and edges that describe a connection between two nodes (such as power lines or water pipes). We will consider a model of a graph \(G = (N,E)\) where \(N\) is the set of labeled nodes \(n_1,...,n_{|N|}\) and \(E\) is the set of labeled edges \(e_1,...,e_{|E|}\). Each node \(n_i \in N\) has a list of attributes \([S,T]\). In particular, \(n_i.S\), the \(S\) attribute of variable \(n_i\), is a Boolean variable equal to \(True\) if \(n_i\) is a source node, and \(n_i.T\) is a Boolean variable equal to \(True\) if \(n_i\) is a terminal node. Each edge \(e_i \in E\) has a list of attributes \([n_p,n_s,bi,p]\). In particular, \(e_i.n_p\) is the predecessor node, \(e_i.n_s\) is the successor node, \(e_i.bi\) is a Boolean variable equal to \(True\) if the edge is bidirectional and \(False\) if the edge is directed, and \(e_i.p\) is the edge reliability, the probability the edge exists.

To measure graph reliability, we define a function \(C(G_r)\), where \(C(G_r)=1\) if there is a path from a source node to every terminal node for the particular graph realization \(G_r\), indicating a Connected graph, and \(C(G_r)=0\) otherwise. A graph realization is defined with a vector \(r\) of length \(|E|\) where \(r_i=1\) if \(e_i\) exists, and \(r_i=0\) otherwise. The probability of a given graph realization is defined below: \[P(G_r) = \prod_{i=1}^{|E|} 1-r_i-e_i.p+2*r_i*e_i.p\] The set of all possible realization vectors is \(R\). This set contains \(2^{|E|}\) elements.

Reliability is defined as \(Rel(G) = \sum_{r \in R} [C(G_r)*P(G_r)]\). \(Rel(G)\) will take on a value between 0 and 1, where a higher \(Rel(G)\) is desired in practice.

This model as defined allows us to consider two cases of network reliability that are important for managers of infrastructure systems. All Terminal Reliability is \(Rel_{ATR}(G)\) where all edges are bidirectional and all nodes are terminal nodes. Source-Terminal reliability is \(Rel_{S-T}(G)\) where all edges are directed, and in general the set of source and terminal nodes is much smaller than the set of nodes. For this work, only one source and one terminal node will be considered for \(Rel_{S-T}(G)\) (also known as 2-terminal reliability).

\hypertarget{justification-and-objectives}{%
\subsection{Justification and Objectives}\label{justification-and-objectives}}

As the time to compute the reliability of a probabilistic graph scales exponentially with the size of the graph \cite{valiant1979complexity}, a naïve brute force enumeration of all graph realizations is not feasible for the large graphs frequently encountered in infrastructure systems. Therefore, different approaches are needed to calculate values of \(Rel(G)\).

One approach is to develop an algorithm with parameterized complexity to solve \(Rel(G)\). While these algorithms may scale exponentially in the worst case, they may not scale as fast for the problems we are interested in. Another approach is to solve a superset problem of \(Rel(G)\). In this work, the edge cover problem will be shown to be a superset problem of \(Rel_{ATR}(G)\). A final approach is to give up on obtaining an exact answer and instead obtain an approximate answer for \(Rel(G)\) using Monte Carlo (MC) simulation.

The objective of this work is to efficiently calculate values of \(Rel(G)\) in a principled way. This work will formulate TNC algorithms for exactly solving \(Rel_{S-T}(G)\) when the graph is directed and acyclic, and exactly solving an upper bound for \(Rel_{ATR}(G)\) by solving the edge cover problem. These proposed TNC algorithms will be shown to have computational complexity parameterized by the treewidth of the graph. The performance of these algorithms will be tested on grid graphs, random cubic graphs, and a selection of real world transmission graphs.

\hypertarget{background}{%
\section{Background}\label{background}}

\hypertarget{exact-solvers}{%
\subsection{Exact Solvers}\label{exact-solvers}}

\hypertarget{binary-decision-diagram-methods}{%
\paragraph{\texorpdfstring{Binary Decision Diagram Methods\\
}{Binary Decision Diagram Methods }}\label{binary-decision-diagram-methods}}

The current state-of-the-art for exactly solving undirected K-Terminal reliability problems (which includes ATR and S-T) is using binary decision diagrams \cite{carlier1996decomposition}. These methods consider one edge at a time, factoring the graph into subgraphs, and pruning by identifying isomorphic graphs \cite{hardy2007k}.

However, the pruning is not very efficient. In the worst case for All Terminal Reliability problems, the number of subgraphs to be considered is proportional to \(BELL(F_{max})\), where \(BELL(k)\) is the \(k^{th}\) bell number and \(F_{max}\) is the linear-width, or pathwidth, of the graph. If the pathwidth of the graph is small, this algorithm is still useful, but \(BELL(k)>(\frac{k}{e\ln(k)})^k\) \cite{berend2010improved}, growing faster than \(2^n\), so binary decision diagram methods quickly become computationally infeasible.

\hypertarget{sat-solvers}{%
\paragraph{\texorpdfstring{\#SAT solvers\\
}{\#SAT solvers }}\label{sat-solvers}}

The edge cover problem \(Rel_{EC}(G)\) is a superset of \(Rel_{ATR}(G)\). We can define \[Rel_{EC}(G) = \sum_{r \in R} [EC(G_r)*P(G_r)]\] where \(EC(G_r)=1\) if every node has at least one existing edge, and \(EC(G_r)=0\) otherwise. \(Rel_{ATR}(G) \subset Rel_{EC}(G)\) because for every case of \(Rel_{ATR}(G)\) every node must have at least one existing edge to ensure connectivity, but \(Rel_{EC}(G) \not\subset Rel_{ATR}(G)\) because unconnected ``islands'' of nodes can satisfy \(EC(G_r)\) while not satisfying global connectivity for \(C(G_r)\).

Rewritten in conjunctive normal form (CNF) as a monotone \#SAT problem \cite{roth1996hardness} \cite{vaisman2015model}: \[EC(G_r) = \bigwedge_{n_i \in N} (\bigvee_{j \in n_i} e_j(r))\] where each node \(n\) has a set of associated edges \(j \in n_i\) if edge \(e_j\) has node \(n_i\) as a predecessor or successor. The function \(e_j(r)\) is equal to True if \(r_j=1\) for graph realization \(r\).

Our ability to write this problem in CNF form, a set of clauses which all must be true, and each clause is satisfied if at least one variable in the clause is true, allows it to be solved by powerful existing model counting solvers such as cachet \cite{sang2005heuristics}, miniC2D \cite{oztok2015top}, and d4 \cite{lagniez2017improved}. However, these solvers are considered to be ``black-box'' solvers, so there is no ability to estimate their computation time for an arbitrary graph \(G\).

\hypertarget{probabilistic-solvers}{%
\subsection{Probabilistic Solvers}\label{probabilistic-solvers}}

\hypertarget{monte-carlo-mc-methods}{%
\paragraph{\texorpdfstring{Monte Carlo (MC) Methods\\
}{Monte Carlo (MC) Methods }}\label{monte-carlo-mc-methods}}

If a set of \(K\) independent random realizations of a given graph are generated, \(G_{rand}\), then \(Rel(G)\) can be estimated from this random sample of graphs. \(C(G_{rand})\) can be considered to be a set of Bernoulli trials, a binomial experiment \(B(K,p)\) so MC methods can be used to estimate the \(p\) of the Bernoulli process and provide a confidence interval.

Given a specific simulation of \(G_{rand}\), the log-likelihood profile of \(p\) can be obtained. The most likely value of \(p\), \(p_{true}\), is obtained by finding the maximum of the profile. To obtain a confidence interval, we can use the profile likelihood method \cite{venzon1988method}. For a given desired \(1-\alpha\) and \(\epsilon=\frac{p_{true}}{p_{-(1-\alpha)\%}}\), the number of needed samples \(K\) can be calculated.

For a range of \(\epsilon\) values, Figure \ref{fig:MC Trial Graph} shows on a log-log scale how the number of needed MC samples decreases as \(p_{true}\) increases. Specifically, if \(p_{true}\) increases by a factor of 10, the number of trials needed decreases by a factor of 10 while \(p_{true}<0.1\).

\begin{figure}[t]
\caption{Needed MC trials for a given $\epsilon$ at a 95\% confidence interval shown on a log-log scale graph: As $p_{true}$ decreases, the number of needed MC trials for a given relative error increases.}
\label{fig:MC Trial Graph}
\includegraphics[width=0.5\textwidth]{../figures/MCTrials.png}
\end{figure}

The advantage of MC Methods is the number of samples required is only proportional to \(\frac{1}{p_{true}}\) and is not proportional to the problem size. The drawback is that \(p_{true}\) is not known ahead of time, so the number of samples required could be large if \(p_{true}\) is very small and cannot be known ahead of time, and stopping rules must be used, adding uncertainty.

\hypertarget{fully-polynomial-time-randomized-approximation-scheme-fpras}{%
\paragraph{\texorpdfstring{Fully Polynomial-time Randomized Approximation Scheme (FPRAS)\\
}{Fully Polynomial-time Randomized Approximation Scheme (FPRAS) }}\label{fully-polynomial-time-randomized-approximation-scheme-fpras}}

From the analysis above, MC methods become infeasible for estimating graph failure rates when \(Rel(G)\) is close to 1. In response to this drawback, Karger \cite{karger2001randomized} developed a Fully Polynomial-time Randomized Approximation Scheme (FPRAS) for estimating \((1-Rel_{ATR}(G))\) that runs in \(O(\frac{|E||N|^4}{\epsilon^3}\ln(|N|))\) time.

However, the FPRAS algorithm only works when \((1-Rel_{ATR}(G))<|N|^{-4}\). For most engineering applications, the failure chances we care about are small. For example, 1\% for 50 year earthquake hazards \cite{IBC_2017} or \(2.0*10^{-6}\) per year for nuclear power plants \cite{NRC_2010}. Therefore, highly reliable engineering networks with more than \((1-2.0*10^{-6})^{-\frac{1}{4}} \approx 27\) nodes cannot be practically solved using known FPRAS methods. In addition, this FPRAS algorithm is limited and only works for All-Terminal Reliability. While extensions exist for K-Terminal problems, such as the one developed by Paredes \cite{paredes2019principled} which works well in practical settings, these extensions rely on NP-oracles and therefore have exponential worst case behavior.

Therefore, we desire an approximation algorithm that is not dependent on \(Rel(G)\), and is instead parameterized by some other graph property that is small for engineering networks of interest. TNC algorithms fit this desire, being parameterized by the treewidth of a graph, which is usually small and constrained for the almost-planar engineering networks we care about.

\hypertarget{tensor-network-contraction-tnc}{%
\section{Tensor Network Contraction (TNC)}\label{tensor-network-contraction-tnc}}

\hypertarget{definitions}{%
\subsection{Definitions}\label{definitions}}

The goal of TNC is to write the underlying satisfiability problem as a series of tensor products. Similar techniques have been investigated in the physics community to solve specific quantum mechanics problems \cite{pan2020contracting} \cite{biamonte2019lectures}. Each clause in the satisfiability problem is represented as a tensor \(T^c_{x_1,x_2,...,x_k}\). If the variables \(x_1,x_2,...,x_k\) satisfy the underlying clause, then \(T_{x_1,x_2,...,x_k}=1\). For example, the \(T^c\) that encodes the Boolean clause \((x_1 \lor x_2 \lor x_3 \lor x_4)\) is: \begin{equation*}
T^c_{x_1,x_2,x_3,x_4}=
\begin{cases}
  0, & \text{if}\ x_1=x_2=x_3=x_4=0 \\
  1, & \text{otherwise}
\end{cases}
\end{equation*}

The number of solutions to the satisfiability problem can be calculated by applying the tensor product to every clause tensor \(T^c\). The tensor product is defined as \ten \[T^p_{x_1,...,x_k,z_1,...,z_k}=T^c_{x_1,...,x_k,y_1,...,y_k} \otimes T^c_{y_1,...,y_k,z_1,...,z_k}\] \normalsize where \(\otimes\) expands into \ten \[T^p_{x_1,...,x_k,z_1,...,z_k}=\sum_{y_i \in Y} \sum_{y_i=0}^{|y_i|} T^c_{x_1,...,x_k,y_1,...,y_k} * T^c_{y_1,...,y_k,z_1,...,z_k}\] \normalsize where \(Y\) is the set of all variables in common between the two tensors, and \(|y_i|\) is the number of states that variable \(y_i\) can take. While \(y_i\) can take an arbitrary number of states, the remainder of this work will only consider a two-state Boolean variable.

One complication is that a tensor product is only clearly defined if each variable appears exactly once or twice, while in many satisfiability problems a variable can appear more than twice. This complication can be addressed by assigning each \(T^c\) a unique set of variables, and then creating additional variable tensors \(T^v\) to apply constraints on the variables.

Two common constraints are defined below. To constrain a Boolean variable \(x_1\) to take the opposite value of \(x_2\) (as needed for a Boolean formula containing \(x_1\) and \(\neg x_1\)), the following tensor \(T^v_{x_1,x_2}\) is set as \begin{equation*}
T^v_{x_1,x_2}=
\begin{cases}
  1, & \text{if}\ x_1=0\ \text{and}\ x_2=1\\
  1, & \text{if}\ x_1=1\ \text{and}\ x_2=0\\
  0, & \text{otherwise}
\end{cases}
\end{equation*}

To constrain a Boolean variable \(x_1\) to take the same value of \(x_2\) and apply a probability \(p\) of both variables being true (as needed to define an unreliable network edge), the tensor \(T^v_{x_1,x_2}\) is set as \begin{equation*}
T^v_{x_1,x_2}=
\begin{cases}
    p, & \text{if}\ x_1=1\ \text{and}\ x_2=1 \\
    1-p, & \text{if}\ x_1=0\ \text{and}\ x_2=0 \\
    0, & \text{otherwise}
\end{cases}
\end{equation*}

\hypertarget{graph-representation}{%
\subsection{Graph Representation}\label{graph-representation}}

Tensor multiplications can be represented as a node and edge graph, \(G_T\), where each tensor is a node and each variable is an edge. An example of a tensor graph can be seen in Figure \ref{fig:Tensor Graph Example}a. A tensor product can be represented as an edge contraction on this graph. The contraction of edge \(X_1\) is visually shown in Figure \ref{fig:Tensor Graph Example}b. Once all edges are contacted, only a scalar value remains, counting the number of solutions to the Boolean problem.

\begin{figure}[t]
\caption{Example Tensor Graph [a] and example tensor contraction of edge $X_1$ [b]}
\label{fig:Tensor Graph Example}
\includegraphics[width=0.5\textwidth]{../figures/TensorGraphExample.png}
\end{figure}

\hypertarget{contraction-ordering}{%
\subsection{Contraction Ordering}\label{contraction-ordering}}

Care must be taken when choosing the order to perform the edge contractions. Assuming Boolean variables, the product \(T^c_{x_1,...,x_n,y_1,...,y_n} \otimes T^c_{y_1,...,y_n,z_1,...,z_n}\) requires \(2^{|x|+|y|+|z|}\) multiplications and additions, and \(2^{|x|+|z|}\) numbers need to be stored in memory for the resulting tensor. Markov, Igor L and Shi, Yaoyun show how to determine an optimal edge contraction ordering to minimize \(|x|\), \(|y|\), and \(|z|\), also known as elimination ordering, from an optimal tree decomposition of the line graph of \(G_T\), \(LG(G_T)\) \cite{markov2008simulating}. Dumitrescu et al.~\cite{dumitrescu2018benchmarking} demonstrate how algorithms from the PACE 2017 challenge \cite{dell2018pace} can be used to obtain better approximate tree decompositions for some tensor graphs representing quantum many body problems.

Harvey, Daniel J and Wood, David R provide a few different upper bounds for the treewidth of \(LG(G_T)\), \(tw(LG(G_T))\), bounding the size of the largest tensor \cite{harvey2018treewidth}: \begin{equation*}
tw(LG(G_T))<(tw(G_T)+1)*D_m(G_T)-1
\end{equation*} where \(D_m(G_T)\) is the maximum degree of graph \(G_T\). Dudek et al.~\cite{dudek2019efficient} also show how high-rank tensors can be factored into a tensor tree to further minimize memory and computational requirements of the TNC.

Overall, in the worst case for infrastructure networks with bounded max degree (due to physical limitations), the largest number of variables for a single tensor is linearly proportional to the treewidth of \(G_T\). Therefore, the computational complexity is at most \(2^{C*tw(G)}\), where C is a constant between 1 and \(D_m(G_T)\).

For the following formulations, if every variable tensor is contracted into an adjacent clause tensor, the resulting tensor graph is isomorphic to the underlying graph \(G\). Therefore, for these formulations, \(tw(G_T)\) is equal to \(tw(G)\).

\hypertarget{all-terminal-reliability-formulation}{%
\subsection{All Terminal Reliability Formulation}\label{all-terminal-reliability-formulation}}

For \(Rel_{ATR}(G)\) there is no known polynomial sized satisfiability equation, unless auxiliary variables are used \cite{paredes2019principled}. Therefore, a tensor graph for the edge cover problem, \(Rel_{EC}(G)\) will be formulated instead. The edge cover problem is satisfied if every node in the graph \(G\) has at least one existing edge. Therefore, the clause for a node \(n\) with connecting edges \(e \in E\) is \((e_1^n \lor e_2^n \lor ... \lor e_i^n)\).

Each edge \(e_i\) has a probability \(p\) of existing, and each variable \(e_i^n\) takes the same correlated state for every superscript \(n\). Contracting the tensor graph \(G_T\) of these tensors will yield the probability of a satisfying edge cover for the graph \(G\).

\hypertarget{s-t-reliability-formulation}{%
\subsection{S-T Reliability Formulation}\label{s-t-reliability-formulation}}

For \(Rel_{S-T}(G)\), the problem is satisfied if any inbound edge connected to the terminal node \(n_t\) is connected to a ``marked'' node. A node is marked if any of its inbound edges is connected to a ``marked'' node or a ``source'' node. For an acyclic directed network, a node \(n_b\) is ``marked'' if and only if there is a path from the source node to node \(n_b\) (This statement does not hold true for graphs with cycles). Even in this restricted case, \(Rel_{S-T}(G)\) is still a \#P-complete problem \cite{provan1986complexity}.

The clause for the terminal node \(n_t\) with inbound edges \(e \in E\) is \((e_1^{n_t} \lor e_2^{n_t} \lor ... \lor e_i^{n_t})\).

For a node between the source and terminal nodes \(n_b\) with inbound edges \(e \in E\), it must satisfy the following clause: \begin{equation*}
\begin{split}
& (m_{n_b} \land (e_1^{n_b} \lor ... \lor e_i^{n_b})) \lor \\
& (\neg m_{n_b} \land \neg (e_1^{n_b} \lor ... \lor e_i^{n_b}))
\end{split}
\end{equation*} where \(m_{n_b}\) is a variable indicating if node \(n_b\) is marked.

For each variable \(e_i\), it must be constrained to only be True with probability \(e_i.p\) when the tail is connected to a ``marked'' or source node, \(m_{n_b}=1\), and always False when the tail is not connected to a ``marked'' node. Therefore, the corresponding variable tensor for edge \(i\) outbound from node \(b\) and inbound to node \(y\) is: \begin{equation*}
\begin{split}
& T^{v_e}_{m_{n_{b,y}},e_i^{n_y}}= \\
& \begin{cases}
  e_i.p, & \text{if}\ m_{n_{b,y}}=1\ \text{and}\ e_i^{n_y}=1 \\
  1-e_i.p, & \text{if}\ m_{n_{b,y}}=1\ \text{and}\ e_i^{n_y}=0 \\
  1, & \text{if}\ m_{n_{b,y}}=0\ \text{and}\ e_i^{n_y}=0 \\
  0, & \text{if}\ m_{n_{b,y}}=0\ \text{and}\ e_i^{n_y}=1 \\
\end{cases}
\end{split}
\end{equation*}

In addition, the directed ``marked'' variable \(m_{n_{b,y}}\) must be constrained to the same value as \(m_{n_b}\). Contracting the tensor graph \(G_T\) of these tensors will yield the exact probability of a satisfying path from the source node to the terminal node for the graph \(G\).

\hypertarget{tensor-network-contraction-tnc-advantages}{%
\subsection{Tensor Network Contraction (TNC) Advantages}\label{tensor-network-contraction-tnc-advantages}}

TNC algorithms have many advantages over the previously described exact solvers and probabilistic solvers. The upper bound computational complexity of \(2^{C*tw(G)}\) is significantly better than the \(BELL(Pathwidth(G))\) of the binary decision diagram methods and the unknown upper bounds of the \#SAT methods. This bound is not dependent on \(Rel(G)\), so TNCs can solve some highly reliable networks faster than probabilistic solvers. The computational effort of a TNC can be known ahead of time (after the approximate tree decomposition), so reliability engineers can confidently choose the most efficient reliability solver algorithm. In addition, for infrastructure networks of interest, they are usually near-planar, which bounds treewidth to \(2*\sqrt{6*(k+1)*|N|})\) \cite{dujmovic2017structure} where k is the number of allowed crossings for each edge, and treewidth is frequently lower than this bound \cite{maniu2019experimental}.

TNCs only require vectorized multiplication and addition operations which are very efficient for CPUs and GPUs to compute, while binary decision diagram methods and \#SAT methods require many conditional if-then statements which are more difficult to optimize. While probabilistic solvers are perfectly parallel (each sample can be done on a separate computer), the individual tensor contractions can also be broken up and dispatched to multiple parallel computing units.

\hypertarget{results-and-discussion}{%
\section{Results and Discussion}\label{results-and-discussion}}

\hypertarget{benchmark-graphs}{%
\subsection{Benchmark Graphs}\label{benchmark-graphs}}

To evaluate the empirical performance of the proposed TNC algorithm, a few classes of graphs will be considered. The first considered class of graphs are grid graphs. As most infrastructure networks are usually near-planar, grid graphs can be considered as the ideal case of planar graphs.

Second, random connected cubic graphs will be considered. Using a set of reliability preserving transformations \cite{shooman1991exact}, and by splitting high degree nodes into a chain of degree 3 nodes connected by unfailing edges, all graphs can be converted to a cubic graph with equivalent \(Rel(G)\). A 1-Flipper Markov Chain Monte Carlo (MCMC) algorithm will be used to uniformly generate these random cubic graphs \cite{feder2006local}.

Third, a collection of 58 US power transmission networks \cite{li2016characterizing} will be considered. These graphs will be reduced using reliability preserving transformations before reliability calculations are performed.

All benchmarks are performed on a Intel Core i7-4810MQ CPU @ 2.90GHz, with 16 GB of RAM. All code is single threaded. The code used to generate these graphs can be seen at this link: \url{https://github.com/KyleAnthonyShepherd/SISRRA_tensor_contraction/tree/main/ICOSSAR_2021}

\hypertarget{grid-graphs}{%
\subsection{Grid Graphs}\label{grid-graphs}}

\hypertarget{computational-time}{%
\subsubsection{Computational Time}\label{computational-time}}

As seen in Figure \ref{fig:Grid Graph Computational Time}a, as the grid dimension increases, both the number of subgraphs for the binary decision diagram method and the number of floating point operations for the TNC increases exponentially. Figure \ref{fig:Grid Graph Computational Time}b shows the wall clock time taken for each method. In both cases, the slope of the TNC is significantly smaller than the binary decision diagram method, showing significant computational advantages for calculating \(EC(G)\) and \(Rel_{S-T}(G)\).

Further computational advantages can be seen if one dimension of the grid graph is fixed in size (8 nodes is large enough for a non-trivial treewidth size and small enough to be computed quickly by both methods). As seen in Figure \ref{fig:Grid Graph Computational Time}c, as \(n\) increases for the \(8xn\) grids, both the number of subgraphs for the binary decision diagram method and the number of floating point operations for the TNC increases linearly. However, as seen in Figure \ref{fig:Grid Graph Computational Time}d, the wall clock time taken for the binary decision diagram method increases quadratically (each subgraph needs an \(O(|E|)\) connectivity check) while the TNC time only increases linearly. For graphs of bounded treewidth, TNCs show significant computational improvement.

\begin{figure}[t]
\caption{Computational complexity and wall clock time for $nxn$ and $8xn$ grid graphs. Subgraphs and $REL_{ATR}(G)$ are BDD calculations, and floating point operations and $EC(G)$ are TNC calculations.}
\label{fig:Grid Graph Computational Time}
\includegraphics[width=0.5\textwidth]{../figures/GridGraphComputeTime8.png}
\end{figure}

\hypertarget{monte-carlo-mc-comparison}{%
\subsubsection{Monte Carlo (MC) Comparison}\label{monte-carlo-mc-comparison}}

As the TNC only bounds \(Rel_{ATR}(G)\) by calculating \(EC(G)\), we can evaluate the quality of this estimate by determining the number of MC trials needed to obtain bounds of \(Rel_{ATR}(G)\) better than \(EC(G)\). Using a 95\% confidence interval, we can calculate the number of MC trials needed to create a confidence interval that excludes \(EC(G)\).

For an edge failure rate of 0.01, both \(Rel_{ATR}(G)\) and \(EC(G)\) are approximately constant at 0.9996. Approximately 24 million MC trials are needed to rule out \(EC(G)\), and this count is insensitive to the size of the grid. Therefore, for reliable grid graphs, \(EC(G)\) is a good bound.

\hypertarget{random-connected-cubic-graphs}{%
\subsection{Random Connected Cubic Graphs}\label{random-connected-cubic-graphs}}

\hypertarget{computational-time-1}{%
\subsubsection{Computational Time}\label{computational-time-1}}

A random selection of 10,000 random connected cubic graphs \(G_{rc}\) from node count \(|N|=20\) to \(|N|=400\) were generated. The treewidth of \(G_{rc}\) and the treewidth of \(LG(G_{rc})\) were computed using an approximate treewidth solver \cite{dell2018pace} for 6 seconds. The pathwidth of \(G_{rc}\) was estimated from the tree decomposition of \(G_{rc}\). As seen in Figure \ref{fig:Random Cubic Treewidth}, there is a linear increase in approximated treewidth as graph size increases.

\begin{figure}[t]
\caption{Approximated treewidth, pathwidth, and treewidth of the line graph of $G_{rc}$ for 10,000 randomly generated connected cubic graphs $G_{rc}$, and the ratios between these widths.}
\label{fig:Random Cubic Treewidth}
\includegraphics[width=0.5\textwidth]{../figures/RcubicWidth_short.png}
\end{figure}

\hypertarget{monte-carlo-mc-comparison-1}{%
\subsubsection{Monte Carlo (MC) Comparison}\label{monte-carlo-mc-comparison-1}}

A random selection of 10,000 random connected cubic graphs \(G_{rc}\) from node count \(|N|=20\) to \(|N|=50\) were generated and solved for \(EC(G)\) and \(Rel_{ATR}(G)\). Each edge had a failure rate of 0.01 to represent a network with high reliability. \(EC(G)\) is very tightly constrained, with a standard deviation of \(1.1*10^{-12}\) for all \textbar N\textbar, and can be estimated as \(EC(G) \approx (1-|N|*0.01^3)\). \(Rel_{ATR}(G)\) had greater variance, with a range of {[}0.92265,0.99998{]}.

For each graph, the number of MC trials needed to rule out the \(EC(G)\) bound at the 95\% confidence interval was calculated. Figure \ref{fig:Random Cubic MC} shows the proportion of cubic graphs that need less than \(X\) MC trials to rule out the \(EC(G)\) bound. The observed segmented stair-step pattern is unusual, and is likely caused by the graphs in each segment sharing some topological feature such as a bridge.

As graph size increases, the empirical cumulative distribution function pushes up and to the left, indicating more graphs need fewer MC trials to rule out the \(EC(G)\) bound. This means as highly reliable cubic graphs become larger, \(EC(G)\) becomes a worse bounding value of \(Rel_{ATR}(G)\).

\begin{figure}[t]
\caption{The empirical cumulative distribution function of randomly generated connected cubic graphs of size |N| that need less than $X$ MC trials of $Rel_{ATR}(G)$ to rule out the bound from $EC(G)$ at the 95\% confidence level.}
\label{fig:Random Cubic MC}
\includegraphics[width=0.5\textwidth]{../figures/RcubicTrials.png}
\end{figure}

Overall, despite the computational advantages of TNCs, the treewidth of these graphs scales linearly with size, resulting in computational complexity growing exponentially with the size of the graph. In addition, \(EC(G)\) as measured by TNCs is a poor bounding value for many cubic graphs, only a few MC trials are needed to achieve a better bounding value. This bounding value becomes worse as the graph size increases.

\hypertarget{power-transmission-grids}{%
\subsection{Power Transmission Grids}\label{power-transmission-grids}}

The \(Rel_{ATR}(G)\) and \(EC(G)\) of 58 transmission power grids \cite{li2016characterizing} were calculated. Table \ref{tab:Power Transmission Grid} shows the node and edge count of these graphs after reliability preserving reductions, and the result of the \(Rel_{ATR}(G)\) and \(EC(G)\) calculations at edge failure rate 0.5, 0.1 and 0.01. Some graphs were omitted due to trivial structure or inability to compute \(Rel_{ATR}(G)\).

Figure \ref{fig:Width Comparison} shows the treewidth and \(tw(LG(G))\) of the power grids in relation to the previously analyzed graphs. In general, the treewidth of the power grids is smaller than equal sized cubic graphs, making them very computationally efficient to solve. However, \(tw(LG(G))\) of the power grids are significantly greater than their treewidth, comparable to \(tw(LG(G_{rc}))\) of equal sized cubic graphs, due to the presence of high degree nodes in the power grids. The tensor factoring techniques in \cite{dudek2019efficient} may reduce these large values of \(tw(LG(G))\). Despite this, TNCs still quickly solve \(EC(G)\) and \(Rel_{S-T}(G)\) of these graphs in comparison to the binary decision diagram techniques.

\begin{figure}[t]
\caption{Treewidth and $tw(LG(G))$ comparison between grid graphs, random cubic graphs, and power transmission grids.}
\label{fig:Width Comparison}
\includegraphics[width=0.5\textwidth]{../figures/WidthCompare.png}
\end{figure}

When each edge only has a 1\% chance of failure, \(EC(G)\) is a good approximation for 37 of the power grids as seen in the tall green bars in Figure \ref{fig:Power Grid MC}. It would take more than 1,000,000 MC trials to rule out the \(EC(G)\) approximation for these graphs. For the largest power grids, only 1,000 MC trials are needed to rule out \(EC(G)\).

\begin{figure}[t]
\caption{Number of MC Trials needed to rule out $EC(G)$ at the 95\% confidence level for each power grid at different edge failure rates $p$.}
\label{fig:Power Grid MC}
\includegraphics[width=0.5\textwidth]{../figures/PowerGridMC.png}
\end{figure}

\hypertarget{conclusion}{%
\section{Conclusion}\label{conclusion}}

\hypertarget{results-summary}{%
\subsection{Results Summary}\label{results-summary}}

Overall, TNCs for solving \(Rel_{S-T}(G)\) and estimating \(Rel_{ATR}(G)\) demonstrate many computational advantages on many practical networks. These methods are parameterized by the treewidth of the network, so graphs with low treewidth such as grids and the 58 power transmission networks can be quickly solved. In the general case as represented as random cubic graphs, TNCs are not as computationally efficient due to the linear relationship between treewidth and random cubic graph size. In addition, the presence of high degree nodes in the power transmission networks introduces a large constant factor between treewidth and the computationally relevant treewidth of the line graph. Despite these limitations, the tensor methods are still 10 to 100 times faster than the state-of-the-art binary decision diagram methods as measured by wall clock time.

When estimating \(Rel_{ATR}(G)\) by calculating \(EC(G)\), TNCs show excellent results on grid graphs. As edge failure rate decreases, as is the case for highly reliable networks, \(EC(G)\) becomes a better estimator. This increase in estimation accuracy is likely due to the fact that as edge failures become less likely, multiple edge failures needed to disconnect the graph become exponentially less likely, which heavily discounts occurrences of disconnected ``islands'' of nodes that satisfy \(EC(G)\) and do not satisfy \(Rel_{ATR}(G)\).

However, \(EC(G)\) is only a good estimate for \(Rel_{ATR}(G)\) for a small subset of cubic graphs and 37 out of 55 of the power grids. This work did not investigate why \(EC(G)\) was a good estimate for these graphs, although it is likely due to topological bottlenecking effects.

\hypertarget{future-work}{%
\subsection{Future Work}\label{future-work}}

In relation to algorithm design, the primary bottleneck to TNCs is the memory requirement. Techniques such as sparse arrays, online matrix compression, or tensor factoring can be used to reduce the memory footprint of large tensors. Additionally, many quantum computer algorithms can be described as tensor contractions \cite{biamonte2017tensor} \cite{duenas2018quantum}, so these TNCs may be able to exploit quantum computers to achieve a quantum speedup over traditional algorithms.

As \(EC(G)\) is not always a good approximation for \(Rel_{ATR}(G)\), it would be beneficial to classify the graphs where \(EC(G)\) is a good approximation of \(Rel_{ATR}(G)\). If an algorithm can quickly identify these classes of graphs, then tensor methods can quickly and confidently estimate \(Rel_{ATR}(G)\) using \(EC(G)\). In addition, determining how to incorporate TNCs with other \#SAT solvers into a virtual best solver will greatly expand the classes of graphs where \(EC(G)\) can be quickly solved.

While \(Rel_{S-T}(G)\) can be exactly solved by tensor methods for directed acyclic graphs, the introduction of cycles causes drastic multiplicative overcounting of solutions if the given \(Rel_{S-T}(G)\) formulation is used. Determining a better tensor graph for solving \(Rel_{S-T}(G)\) or determining how to compensate for the multiplicative overcounting can expand the number of graphs that can be exactly solved by TNCs.

\begin{table*}[ht]
\centering
\caption{Power Transmission Network Attributes}
\label{tab:Power Transmission Grid}
\newcolumntype{a}{X}
\newcolumntype{b}{>{\hsize=.33\hsize}X}
\renewcommand{\arraystretch}{.75}
\begin{tabularx}{\textwidth}{bbbbbbaaaaaaaa}
\toprule
ID & |V| & |E| & TW & PW & LGW & $Rel_{time}$ & $Rel_{0.5}$ & $Rel_{0.1}$ & $Rel_{0.01}$ & $EC_{time}$ & $EC_{0.5}$ & $EC_{0.1}$ & $EC_{0.01}$ \tabularnewline
\midrule
\input{../tables/PowerGridsRELTable.tex}
\bottomrule
\end{tabularx}
\end{table*}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{parskip}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{blindtext}

\title{Information Theory for Neural Network Regularization}
\author{Jakub Bartczuk}
\date{June 2020}

\newcommand{\loss}{\mathcal{L}}
\begin{document}
\maketitle

\section{Introduction}

Learning neural networks for supervised learning is commonly formalized as optimizing log loss

$$\loss = E_{(x,y) \sim p}[log\, p_\theta(y|x)] \approx \frac{1}{N} \sum_i log\, p_\theta(y_i|x_i) $$

When model, $p_\theta$ is a complicated function there is risk of overfitting - the model might learn useless features that do not generalize from training set.

To address that problem, most neural network use some form of \textit{regularization}. Regularization is performed by applying some procedure to neural network outputs or by optimizing $\loss' = \loss + P$ where $P$ is some penalty.

Usual regularization methods are based on injecting noise - be it in explicit form of dropout (zeroing in activations of intermediate layers) or implicit in the form of weight decay (it is a known fact that for linear models L2 weight decay is equivalent to assuming a prior normal distribution on weights which is in turn equivalent to adding gaussian noise).

\section{Information theory and supervised learning}

Log loss has natural information theoretic interpretation: $\loss = - H(p) - KL(p_{\theta}|p)$, so optimizing w.r.t $p_{\theta}$ means minimizing KL divergence between model and true probability.

It is an empirical fact that when overfitting often manifests in model being overly confident of its predictions. This is because $L$ only measures error for true class $y_i$ and discards information about other classes.

In practice the problem of model being overly confident is commonly solved by using \textit{label smoothing} which turns one-hot encoded vector $y_i$ that represents indicator function for appropriate class into vector which has some nonzero probabilities for other classes.

\subsection{Regularizing Neural Networks by Penalizing Confident Output Distributions}

Less known method that is based on information-theoretic interpretation is penalizing confident distributions from \cite{penalizing}. Distribution confidence is measured using entropy.In this method becomes $\loss'$

$$\loss' = E_{(x,y) \sim p}[log\, p_\theta(y|x)] + \lambda H(p_\theta(y|x))$$

\section{Information Bottleneck}

\begin{thebibliography}{9}
\bibitem{penalizing}
  Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, Geoffrey Hinton,
  Leslie Lamport,
  \textit{Regularizing Neural Networks by Penalizing Confident Output Distributions},
  2017.

\end{thebibliography}

\end{document}


\section{SQL}
SQL is a declarative language (orders to execute a query without specifying how) to interact with databases and retrieve data. Optimization is not only on the query level, but also on storage and data types: for instance, strings can be managed adding a pointer to their size (if they are long).

Various SQL dialects have slight differences between semantics, but there are general standards starting from older versions. Different databases might also have different type management, operations and speed.

Another common problem with databases is handling null values while evaluating boolean expressions: NULL and FALSE results in FALSE, therefore this may give an inaccurate representation of information. On the other hand, NOT NULL columns are beneficial for optimization (NULL columns have a reserved memory space for this possibility).

Some PostgreSQL data types:
\begin{itemize}
	\item Numeric, slow arbitrary precision numbers with unbounded size;
	\item Float, mobile floating point (4 bytes) or double precision (8 bytes);
	\item Strings of variable length (pre-allocated or dynamic);
	\item Other common types such as \texttt{bytea}, \texttt{timestamp} and \texttt{interval}.
\end{itemize}

\subsection{SQL commands}
Standard SQL commands are for instance SELECT, FROM, WHERE, GROUP BY, ORDER BY. Those can be found in normal queries or subqueries, which allow to temporarily subset data without saving it to memory.

The \texttt{\textbackslash copy} statement allows to import data from a text file, but can be slow with large datasets since it requires a full scan.

Pattern matching can be done either by comparison (=) or regular expression (LIKE or $\sim$).

Random samples can be extracted with various methods, of which the most accurate in randomness (Bernoulli) is also the slowest.

Set operations are performed thanks to the following instructions:
\begin{enumerate}
	\item UNION, which vertically merges two tables with the same attributes;
	\item EXCEPT, which performs the set difference \texttt{A - B};
	\item INTERSECT, to only keep rows in common between two tables.
\end{enumerate}
The default option implies removing duplicates; to keep them, specifying ALL is necessary.

\subsection{Views and CTEs}
A view is an abstract database object of a stored query, accessed as a virtual table but physically not containing data. It simplifies complexity of interrogations because it wraps commonly used information and makes it reusable.

Views are faster, but can be viewed globally, causing potential issues with naming.

A better way to optimize through a temporary table is \texttt{WITH}, a statement which defines common table expressions (CTE).

CTEs are temporary result sets that can be referenced with another SQL instruction to simplify and speed up queries. Results get materialized and then scanned. The main difference from views is that CTEs have a scope only within the query, and get destroyed as soon as the execution is finished.

CTEs overcome SELECT statement limitations such as referencing themselves, which is the basis of recursion.

\subsection{Regular expression matching}
There are different algorithms which establish whether a given regex matches a string:
\begin{enumerate}
	\item Deterministic finite automata, built from a non-deterministic one, which allows to match in linear time yet is constructed in exponential state (to the size);
	\item NFA with caching to simulate a DFA, avoiding the construction cost but raising the computational to almost quadratic;
	\item Pattern matching by backtracking, forcing an exponential number of sub-cases.
\end{enumerate}
Variations of the Boyer-Moore algorithm (grep) are commonly used, optimized with automata.

Postgres uses Harry Spencer's \texttt{regex} library for matching, an hybrid DFA/NFA implementation (thank you Tom Lane for making clear documentation) which initially generates a NFA representation, then optimizes it and executes a DFA materializing states only when needed and keeping them in cache.

The parser also includes an NFA mode to implement features such as capturing parentheses and back-references. It constructs a tree of sub-expressions which is recursively scanned similarly to tradition engines, therefore quite slow.

To make this faster, each node has an associated DFA representing what it could potentially match as a regular expression. Before searching for sub-matches, the string goes through the DFA so that if it does not match the following stages can be avoided.

\subsection{String comparison}
String comparison is a task which can be computationally expensive in case of very large output.

There are several methods to implement this, such as suffix trees and edit distance, which also allow to search for errors in text through string similarity.

The metric represents the number of operations that need to be applied to one of the inputs to make it equal to the other one, through dynamic programming (construction of a matrix).

\subsection{Indexing}
Postgres provides several index types, of which the most common is the B-tree, a self-balancing data structure allowing logarithmic-time operations (the most important is searching).

It is useful because nodes can have more than two children: the number of sub-trees can in fact be a multiple of the file system block size. The structure is optimized to have few accesses to disk avoiding loading data within reads.

Further improvements can be made creating auxiliary indexes containing the first record in each disk block, narrowing the search in the main database.

\subsection{Counting}
Counting, unless an index is present, is not optimizable: to have an exact result, the DBMS must perform a linear scan of the table, to avoid eventual transaction conflicts.

Counting distinct values or grouping before joining, on the other hand, is faster since it involves an approximation.

Number of total rows can be approximated as well, thanks to sketches: probabilistic data structures that contain frequency tables of events in a stream of data.

They use hash tables to map events to frequencies, and at any time the sketch can be queried for the frequency of a particular element.

To reduce variance and have a more accurate estimation, many independent copies can be stored in parallel, and their outputs is combined.

\subsection{Count-distinct problem}
The count-distinct problem finds how many unique elements are there in a dataset with duplicates.

The naive implementation consists in initializing an efficient data structure (hash table or tree) in which insertion and membership can be performed quickly, but this solution does not scale well.

Streaming algorithms use randomization to produce a close approximation of the number, hashing every element and store either the maximum (likelihood estimator) or the $m$ minimal values.

\subsubsection{HyperLogLog}
HyperLogLog is one of the most widely used algorithms to count distinct elements avoiding calculating the cardinality of a set.

It is a probabilistic algorithm, able to estimate cardinalities of $10^{9}$ with a typical standard error around 2\% using very little memory.

The process derives from Flajolet-Martin algorithm: given a hash function which uniformly distributes output, it is taken the least-significant set bit position from every binary representation.

Since data is uniformly distributed, the probability of having an output ending with $2^k$ (one followed by $k$ zeros) is $2^{-(k+1)}$. The cardinality is then estimated using the maximum number of leading zeros. With $n$ zeros, the number of distinct elements is circa $2^n$.

HyperLogLog also allows to add new elements, count and merge to obtain the union of two sets.

\subsection{Sampling}
Sampling is usually performed through the hypergeometrical distribution, a formula composed by 3 binomial coefficients. Computing factorials, approximated with exponentiation, can be a heavy operation.

Factorials can be estimated using Stirling approximation: $\log(n!) \approx n\log n - n + \frac{1}{2}(2\pi n)$, which is cheaper and works for sufficiently big $n$, but still not efficient for very large numbers.

\section{Query decorrelation}
Often queries are easier to formulate using subqueries, yet this can take a toll on performance: since SQL is a declarative language, the best evaluation strategy is up to the DBMS.

A subquery is correlated if it refers to tuples from the outer query. The execution of a correlated query is poor, since the inner one gets computed for every tuple of the outer one, causing a quadratic running time.

Correlation can be recognized observing fields that do not have a bound with the attributes of the outer query. In SQL, they usually are in WHERE or EXISTS statements.

Example (from TCP-H dataset):
\begin{lstlisting}[language=SQL]
select avg(l_extendedprice)
from lineitem l1
where l_extendedprice =
	(select min(l_extendedprice)
	from lineitem l2
	where l1.l_orderkey = l2.l_orderkey;)
\end{lstlisting}

This query can be rewritten to avoid correlation:
\begin{lstlisting}[language=SQL]
select avg(l_extendedprice)
from lineitem l1,
	(select min(l_extendedprice) m, l_orderkey
	from lineitem
	group by l_orderkey) l2
where l1.l_orderkey = l2.l_orderkey
and l_extendedprice = m;
\end{lstlisting}

The new query is much more efficient, yet not as intuitive. Compilers sometimes manage to automatically decorrelate, but correlations can be complex (inequalities, disjunctions).

Instead of running a function every time, the subset is pre-computated trying to evaluate all the possible choices to then just use a lookup.

A join dependency allows to recreate a table by joining other tables which have subsets of the attributes (relationships are independent of each other). In other words, a table subject to join dependency can be decomposed in other tables and created again joining them.

Dependent joins rely on nested loop evaluations, which are very expensive: the goal of unnesting is to eliminate them all.

The easiest practice consists in pulling predicates up and creating temporary tables inside FROM instead of WHERE, evaluating the subquery for all possible bindings simultaneously.

The subquery in some cases is a set without duplicates which allows equivalence between dependent join and bindings of free variables, therefore dependency can be removed.

Sometimes attributes of the set can be inferred, computing a superset of the values and eliminating them with the final join. This avoids computing the subquery, but causes a potential loss of pruning the evaluation tree.

The approach is overall always working with relational algebra, yet not necessarily on the SQL level: it can add memory overhead which cannot always be removed, despite the ideal linear computational time.

To decorrelate subqueries, a general method can be used which is effective even with more complicated interrogations containing inequalities and several conditions.

The procedure can be executed several times, always starting from the highest level of nesting:
\begin{enumerate}
	\item Moving the correlated subquery to FROM;
	\item Since there might be multiple keys with the same value, putting DISTINCT helps reducing cardinality;
	\item Joining is performed in the outer query, and if there is more than one value, an OR condition is applied so that either can be possible;
	\item Values are grouped to maintain the same domain, according to the elements in DISTINCT and JOIN;
	\item Other conditions taken from outside the subquery are applied after pre-aggregation.
\end{enumerate}

The inner table should always have an unique keys and some groups, which then can be selected in the outer one.

Example:
\begin{lstlisting}
select o_orderpriority ,count (*) as order_count
from orders
where o_orderdate >= date '1993-07-01'
and o_orderdate < date '1993-10-01'
and exists (
	select *
	from lineitem
	where l_orderkey = o_orderkey
	and l_commitdate < l_receiptdate)
group by o_orderpriority
order by o_orderpriority;
\end{lstlisting}

Decorrelated query:
\begin{lstlisting}[language=SQL]
select o_orderpriority, count(*) as order_count
from orders, (
	select o2.o_orderkey
	from lineitem, (
		select distinct o_orderkey
		from orders) o2
	where l_orderkey = o2.o_orderkey
	and l_commitdate < l_receiptdate
	group by o2.o_orderkey) preagg
where o_orderdate >= date '1993 -07 -01'
and o_orderdate < date '1993 -10 -01'
and orders.o_orderkey = preagg.o_orderkey
group by  o_orderpriority
order by  o_orderpriority;
\end{lstlisting}

Foundations of Data Engineering/lectures/lecture3.tex