\chapter{Fitting the Spectra}

Fitting of the spectra involves selecting a spectral line of interest (e.g. \ion{Fe}{12} 195.12\,\AA) from the spectral windows of the data and determining a guess on the fit parameters. The next ingredient for a fit is the selection of an optimization method\sidenote{Here we use a Python implementation of the well-known IDL method mpfit which solves the non-linear least squares problem using the Levenberg-Marquardt algorithm. The Python implementation mpfit.py is found on GitHub (https://github.com/segasai/astrolibpy/) and included in our analysis software.}.

For this we've created a set of fit templates for different spectral lines. An \verb+h5dump+ on the file shows that it contains a \verb+/template+ group for the initial guess on the fit parameters and a \verb+/parinfo+ group containing constraints on the parameters for \verb+mpfit.py+.

\begin{lstlisting}
h5dump -n fe_12_195_119.2c.template.h5
HDF5 "fe_12_195_119.2c.template.h5" {
FILE_CONTENTS {
 group      /
 group      /parinfo
 dataset    /parinfo/fixed
 dataset    /parinfo/limited
 dataset    /parinfo/limits
 dataset    /parinfo/tied
 dataset    /parinfo/value
 group      /template
 dataset    /template/component
 dataset    /template/data_e
 dataset    /template/data_x
 dataset    /template/data_y
 dataset    /template/fit
 dataset    /template/fit_back
 dataset    /template/fit_gauss
 dataset    /template/line_ids
 dataset    /template/n_gauss
 dataset    /template/n_poly
 dataset    /template/order
 dataset    /template/wmax
 dataset    /template/wmin
 }
\end{lstlisting}

 The object \verb+eis_read_template.py+ can be used to read a template file and examine the contents.

\begin{lstlisting}
from eis_read_template import eis_read_template
filename = 'fe_12_195_119.2c.template.h5'
template = eis_read_template(filename)
\end{lstlisting}

This produces the output below, showing the \verb+/parinfo+ group that contains  parameters (peak, centroid, width, background) for a double Gaussian fit along with the parameter constraints. Note that this is specific to using the \verb+mpfit+ method (see the GitHub page for more info).
\begin{lstlisting}
+ template file = fe_12_195_119.2c.template.h5
*PARAMETER CONSTRAINTS*
*              Value      Fixed            Limited                 Limits               Tied
 p[0]     57514.6647          0          1          0       0.0000       0.0000
 p[1]       195.1179          0          1          1     195.0778     195.1581
 p[2]         0.0289          0          1          1       0.0191       0.0510
 p[3]      8013.4013          0          1          0       0.0000       0.0000
 p[4]       195.1779          0          1          1     195.1378     195.2181          p[1]+0.06
 p[5]         0.0289          0          1          1       0.0191       0.0510          p[2]
 p[6]       664.3349          0          0          0       0.0000       0.0000
 \end{lstlisting}

 Next you'll want to prep the data for fitting. Once you've read in a template file, you can use the central wavelength to find the desired spectral window in the data using \verb+eis_read_raster+ as shown in the previous chapter.

\begin{lstlisting}
from eis_read_raster import eis_read_raster
from eis_read_template import eis_read_template

# input data and template files
file_data     = 'eis_20190404_131513.data.h5'
file_template = 'fe_12_195_119.2c.template.h5'

# read fit template
template = eis_read_template(file_template)

# get central wavelength
wmin = template.template['wmin']
wmax = template.template['wmax']
wave = wmin + (wmax-wmin)*0.5

# read raster
raster = eis_read_raster(file_data, wave)
ints   = raster.data['data']
wave   = raster.data['wave']
corr   = raster.data['wave_corr']
\end{lstlisting}

Prepping of the data can be handled at various levels of sophistication at the user's discretion, however, at a minimum it should include handling bad values\sidenote{Negative values are a result of the background subtraction.} in the raster, correcting for the wavelength offsets\sidenote{From thermal drift over the  orbit.}, and computing the errors on the intensities\sidenote{The square root of the counts is a good first-order approximation.}.

\begin{lstlisting}
# get dimensions
ndata = ints.shape
nx    = ndata[0]
ny    = ndata[1]
nz    = ndata[2]

# bad data correction
bad = np.where(ints<0)
ints[bad] = 0.0

# compute error on counts
errs = np.sqrt(ints)

# wavelength correction
newwave = np.zeros(ndata)
for i in range(nx):
    for j in range(ny):
        newwave[i,j,::] = wave-corr[i,j]
wave = newwave
\end{lstlisting}

Now on to the fitting! Now that you have a fit template and the data elements, you can perform a fit of the entire raster by calling \verb+eis_fit_raster.py+\sidenote{Here's what's happening under the hood. The object eis-fit-raster calls eis-scale-guess to scale the initial parameter guess to the data, then calls eis-mpfit to implement the Levenberg-Marquardt fitting. The module eis-fit-deviates contains the callable function that returns the fit deviates computed from a model function for eis-mpfit.}. The fit results can be saved and read back using \verb+eis_save_fit.py+ and \verb+eis_read_fit.py+.

\begin{lstlisting}
from eis_fit_raster import eis_fit_raster
from eis_save_fit import save_fit
from eis_read_fit import read_fit

# fit profile
parinfo  = template.parinfo
template = template.template
fit = eis_fit_raster(wave, ints, errs, template, parinfo)

# save fit output
fit = fit.fit
file_fit = save_fit(fit, file_data)

# read fit output back from file
fit = read_fit(file_fit[0])
\end{lstlisting}

The output fit parameters are stored to a dictionary.

\begin{lstlisting}
background   float64      (512, 87, 1)
centroid     float64      (512, 87, 2)
chi2         float64      (512, 87)
component    int64        1
e_background float64      (512, 87, 1)
e_centroid   float64      (512, 87, 2)
e_int        float64      (512, 87, 2)
e_peak       float64      (512, 87, 2)
e_width      float64      (512, 87, 2)
int          float64      (512, 87, 2)
line_ids     object       (2,)
n_gauss      int16        1
n_poly       int16        1
params       float64      (512, 87, 7)
peak         float64      (512, 87, 2)
perror       float64      (512, 87, 7)
status       float64      (512, 87)
wavelength   float64      (512, 87, 24)
width        float64      (512, 87, 2)
\end{lstlisting}

The above steps are illustrated in the example routine \verb+eis_fit_example.py+, which produces a plot like the one shown below.

\begin{figure}[t]
  \centerline{\includegraphics[clip,width=0.8\linewidth,bb=0 0 750 737]{figures/fit_example.pdf}}
  \centerline{\includegraphics[clip,width=0.8\linewidth,bb=750 0 1500 737]{figures/fit_example.pdf}}
  \caption{Example line profile fits. The top two panels show the raster formed by summing over the
    profile (left) and fitting each profile (right). The bottom panels show fits to four
    \ion{Fe}{12} 195.119\,\AA\ line profiles.}
  \label{fig:fit_example}
\end{figure}

As a final note about the fitting routines, there's also a parallelized version for fitting the raster that uses the Python Multiprocessing module \sidenote{This uses a pool of processes and parallelizes over the rasters so that each sub-process gets a raster position.}. When using the parallel version, an extra argument can be passed to the function to specify the number of processes to use (default=4) as in the below example. You can also check out the example in \verb+eis_fit_example_parallel.py+.

\begin{lstlisting}
fit = eis_fit_raster_parallel(wave, ints, errs, template, parinfo, ncpu=8)
\end{lstlisting}


\subsection{Overview of Examples}
We establish some fundamental properties of solutions to the SIP under different QoI maps in terms of the skewness of the $Q$'s being compared.
We seek an estimated probability measure $\hat{\PP}_\pspace$ on the parameter space to converge (with respect to the metric $d_\text{TV}$) to some reference measure $\PP_\pspace$ as more samples (i.e., model evaluations), $\nsamps$ are used.
Such a reference measure could be either some known distribution taken as truth, or another approximation deemed to be sufficiently resolved for the given application or computational budget (i.e. higher-fidelity model, mesh, or Monte Carlo sample-size).\footnote{However, we could also choose to interrogate the push-forward measures given by propagating the $\hat{\PP}_\pspace$ and $\PP_\pspace$ forward to a data space by a QoI map and taking the distance on the resulting output space.
This would measure the ability of the maps to reconstruct the output probability measure.}

In Figure~\ref{fig:voronoi_sols}, we illustrate the solution to the problem of comparing measures defined on two different (implicitly-defined) $\sa$s shown in Figure~\ref{fig:voronoi_issues}.
By introducing a third set against which both sample sets of size $\nsamps=50$ are compared, we can leverage theoretical results from Lemma~\ref{lem:measuresets} to compare solutions to the SIP under different QoI maps.

\begin{figure}[ht]
\centering
	\begin{minipage}{.275\textwidth}
		\includegraphics[width=\linewidth]{./images/voronoi_diagrams/voronoi_diagram_N25_r0_no_label}
	\end{minipage}
	\begin{minipage}{.4\textwidth}
		\includegraphics[width=\linewidth]{./images/voronoi_diagrams/voronoi_diagram_N500_r50}
	\end{minipage}
		\begin{minipage}{.275\textwidth}
		\includegraphics[width=\linewidth]{./images/voronoi_diagrams/voronoi_diagram_N25_r10_no_label}
	\end{minipage}
\caption{
(Left/Right): The two partitions from Figure~\ref{fig:voronoi_issues} will be projected onto a third reference partition (center), in order to compare them on a common $\sa$.
Center: A possibly over-resolved reference sample set, generated using $\nsamps = 500$ uniform i.i.d.~random samples.
}
\label{fig:voronoi_sols}
\end{figure}

To isolate the effect of skewness on the ability to approximate sets with finite sampling, we choose the maps so that they preserve the sizes of sets between $\pspace$ and $\dspace$ under the push-forward measure given in Eq.~\eqref{eq:dataspace_pushforward_measure}.
The sizes of these inverse sets correspond to the average precision of maps $Q$, so we fix the maps to all be equally informative from this perspective; for linear maps, this means they all have the same determinant.

All of our experiments follow the same structure, where $\qspace$ denotes a set of QoI maps under consideration in each example:
\begin{itemize}
\item[[0-a]] Select $\qoi\in\qspace$ and define $\PP_{\dspace_\qoi}$ as a uniform distribution centered on a reference QoI value $Q(\paramref)$ for $\paramref$ taken as the midpoint of $\pspace$.
Note that $\PP_{\dspace_\qoi}$ is exactly discretized with $M=1$ sample, so that
\[
P_{\pspace, 1} = P_\pspace.
\]
\item[[0-b]] Create a regular grid of samples in $\pspace=[0,1]^n$ using $N_{\text{ref},i}$ equispaced points in each dimension.
Set $\bar{N} := \prod N_{\text{ref},i}$.
Since $n$ is small in the numerical examples shown here, we select $N_{\text{ref},i} = 200 \; \forall \; i$ in each example.
\item[[0-c]] Use Algorithm~\ref{alg:inv_density} to construct a reference solution $\PP_{\pspace,\bar{N}}\approx \PP_\pspace$.
\item[[1]] Generate $\set{S_k^{(n)}}_{n=1}^{50}$ sets of uniform i.i.d.~random samples where $N_k = 25, 50, 100, 200, \hdots, 6400$, and $n$ represents the number of repeated trials of a sample size $N_k$.
%, constructing $\set{\set{\VVV_k^{(j)}}_{k=1}^{50}}_{j=1}^{N}$ so that when we compute Total Variation distances on the approximate measures defined on each $\set{\VVV_k^{(j)}}{j=1}{N}$, we can reduce the variance in our expected Total Variation distance values for each instance of $N$. Note that we experimented with using more trials and found the variance in expected Total Variation distances was sufficiently low with as few as twenty trials for the maps under consideration herein.
%\item[[3]] For every trial $T$ and $N$ value (including $\bar{N}$), the reference parameter $\lambda = (\lambda_1, \lambda_2) = (0.5, 0.5)$ is mapped by $Q$ to $\dspace_\qoi = Q(\pspace)$.
%\item[3] A uniform distribution with support $[Q(\lambda_1) - 0.05, Q(\lambda_1) + 0.05] \times [Q(\lambda_2) - 0.05, Q(\lambda_2) + 0.05]$ is defined on $\dspace_\qoi$, representing equal uncertainty in each component of our measured functional values.
\item[[2]] Solve the SIPs using Algorithm~\ref{alg:inv_density} to construct $\set{\PP_{\pspace,M,N}^{(n)}}_{n=1}^{50}$.
\item[[3]] Use $1E5$ i.i.d.~random samples to estimate $\set{d_H^2( \PP_{\pspace,M,N}^{(n)}, \PP_{\pspace,\bar{N}})}_{n=1}^{50}$.
\item[[4]] Average over all trials $n$ for each $N$ to estimate the {\em expected} Total Variation distance for $N$ samples and analyze convergence to $\PP_{\pspace,\bar{N}}$.
\item[[5]] Repeat steps [0-a]--[4] for each $\qoi\in\qspace$ under consideration.
\end{itemize}

\FloatBarrier


\documentclass{article}
\newtheorem{theorem}{Theorem}
\newtheorem{lemma}{Lemma}
% necessary packages
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bm}
\usepackage{oldstyle}
\usepackage{color}
\usepackage{soul}
% disabled b/c we're using tcolorbox
\usepackage[usenames,dvipsnames,svgnames,table]{xcolor}

% math stuff
\providecommand{\inv}[1]{{#1}^{\ensuremath{\mathsf{-1}}}} % inverse ...
\providecommand{\tr}[1]{{#1}^{\ensuremath{\mathsf{T}}}} % transpose ...
\providecommand{\itr}[1]{{#1}^{\ensuremath{\mathsf{-T}}}} % transpose ...
\providecommand{\strans}[1]{{#1}^{\ensuremath{\mathsf{S}}}} % transpose ...
\providecommand{\abs}[1]{\lvert#1\rvert}
\providecommand{\norm}[1]{\left\lVert#1\right\rVert}
\providecommand{\svect}[1]{\hat{\bm{#1}}}
\providecommand{\vect}[1]{\bm#1}
\providecommand{\mat}[1]{\mathbf#1}
\providecommand{\smat}[1]{\hat{\mathbf{#1}}}
\providecommand{\bigTheta}{\mathit{\Theta}}
\providecommand{\bigO}{\mathit{O}}
\providecommand{\sgn}[1]{\textrm{sgn}(#1)}
\providecommand{\xform}[3]{\hspace{0mm}_#1\mat{#2}_#3}
\newcommand{\ud}{\,\mathrm{d}}

% lists
\newcommand{\ia}{(\textos{a})\!}
\newcommand{\ib}{(\textos{b})\!}
\newcommand{\ic}{(\textos{c})\!}
\newcommand{\id}{(\textos{d})\!}
\newcommand{\ie}{(\textos{e})\!}
\newcommand{\1}{(\textos{1})\!}
\newcommand{\2}{(\textos{2})\!}
\newcommand{\3}{(\textos{3})\!}
\newcommand{\4}{(\textos{4})\!}
\newcommand{\5}{(\textos{5})\!}
\newcommand{\6}{(\textos{6})\!}
\newcommand{\7}{(\textos{7})\!}
\newcommand{\8}{(\textos{8})\!}

\DeclareMathOperator*{\minimize}{minimize}

% indicate necessary changes
\providecommand{\captiontxt}[1]{\caption{\emph{#1}}}
\providecommand{\note}[1]{\textcolor{orange}{#1}}
\providecommand{\changeme}[1]{\textcolor{red}{#1}}
\providecommand{\TODO}[1]{
\sethlcolor{yellow}
\noindent
$\cdot$\ \hl{#1}
\\
}

\title{Computing the principal pivoting transform for solving Linear Complementarity Problems with Lemke's Algorithm}
\date{}
\author{Hongkai Dai and Evan Drumwright\\Toyota Research Institute}

\begin{document}
\maketitle
\section{Problem statement} Given tuples $w = \{ w_1, \hdots, w_n \}$ and $z = \{z_1, \hdots, z_n, z_{n+1}\}$, which can be arranged into vectors as follows:
\begin{align*}
	\vect{w} =& \tr{\begin{bmatrix}w_1 & \hdots & w_n \end{bmatrix}}\in\mathbb{R}^n \\
		\vect{z} =& \tr{\begin{bmatrix}z_1 & \hdots & z_n & z_{n+1} \end{bmatrix}}\in\mathbb{R}^{n+1},
\end{align*}
we seek to permute the relationship:
\begin{align*}
	\vect{w} = \vect{q} + \mat{M}\vect{z}
\end{align*}
where $\vect{q}\in\mathbb{R}^n, \mat{M}\in\mathbb{R}^{n\times (n+1)}$ are a given vector/matrix (note that boldface is used to help distinguish vectors and matrices from sets). $z_{n+1}$ will hereforth be referred to as the ``artificial variable'' by swapping some entries in the ``dependent variable tuple'' $w$ with some entries in the ``independent variable'' tuple $z$, yielding two new sets $z'$ and $w'$. Similarly to the matrix/vector relationship above, vectors $\vect{z}' \in \mathbb{R}^{n+1}$ and $\vect{w}' \in \mathbb{R}^n$ can be defined that correspond to variable indices of $z$ and $w$. \emph{The problem that this document seeks to solve is the vector} $\vect{q}'$ \emph{and the matrix} $\mat{M}'$ (actually, just a single column of it) \emph{such that the relationship}:
\begin{align*}
\vect{w}' = \vect{q}' + \mat{M}'\vect{z}'
\end{align*}
follows from the ordering in the vectors corresponding to the positions of the
individual $w$ and $z$ variables in the tuples $w'$ and $z'$. Recall that
pivoting takes the value $\vect{z'} = \vect{0}$, which allows $\vect{w'}$ to be
determined simply be setting it equal to $\vect{q}'$, via the equation above.
This rearrangement is known as a \emph{principal pivoting transform}.

One of the variables in the independent variable tuple $z'$ will be identified as the \textit{driving variable}. We will denote this variable as $z'_{\textrm{driving}}$. Our goal is to find the vector $\vect{q}'$, together with the column in $\mat{M}'$ that will be multiplied (i.e., via the inner product operation) with $z'_{\textrm{driving}}$. We will denote this column as $\mat{M}'_{\textrm{driving}}$.

\subsubsection{Running example}
We consider the LCP from Example 4.4.7 in \cite{Cottle:1992}:
\begin{align*}
	\begin{bmatrix} w_1 \\ w_2 \\ w_3\end{bmatrix} = \underbrace{\begin{bmatrix}-3 \\ 6 \\ -1\end{bmatrix}}_{q} + \underbrace{\begin{bmatrix}  0 & -1 & 2 & 1\\ 2 & 0 & -2 & 1 \\ -1 & 1 & 0 & 1\end{bmatrix}}_{M}\begin{bmatrix}z_1 \\ z_2 \\ z_3 \\ z_4\end{bmatrix}
\end{align*}
After some number of pivoting operations, assume that the ``dependent'' tuple $w'$ and ``independent'' tuple $z'$ consist of:
\begin{align}
	w' = \{ z_4, w_2, z_3 \} ,
	z' = \{ w_1, w_3, z_2, z_1 \}
    \tag{1} % NOTE: Do not change equation number without changing
            % unrevised_lemke_solver_test.cc
\end{align}

\subsection{Terminology} \label{subsec:terminology}
We now introduce some terminology:
\begin{itemize}
	\item \textsc{independent w}: the tuple of variables from $w$ that (partially) comprise $z'$ in the order in which they are found in $w$  (i.e., the elements of \textsc{independent w} appear in ascending order, e.g., $\{ w_1, w_2 \}$). In the running example, \textsc{independent w} is $\{ w_1, w_3 \}$.
		\begin{itemize}
			\item  $\alpha$: the positions in $w$ of the elements from \textsc{independent w}. In the running example, $\alpha = \{ 1, 3 \}$. From the definition of \textsc{independent w}, these $w$ elements of $\alpha$ appear in ascending order.
			\item $\alpha'$: the respective indices in $z'$ of the variables from \textsc{independent w}. In the running example, $\alpha' = \{ 1, 2\}$. These $w$ elements of $\alpha'$ \emph{elements will not necessarily be present in ascending order.}
		\end{itemize}
		Note that the elements in the ``vanilla'' tuple correspond to variable indices, while elements in the primed tuple correspond to tuple indices. Also, \textbf{observe} the invariant $w_{\alpha_i} = z'_{\alpha'_i}$.
	\item \textsc{dependent z}: the tuple of variables from $z$ that (partially) comprise $w'$ in the order in which they are found in $z$ (i.e., the elements of \textsc{dependent z} appear in ascending order, e.g., $\{ z_3, z_4 \}$). In the running example, \textsc{dependent z} is $\{ z_3, z_4 \}$.
		\begin{itemize}
			\item $\beta$: the positions in $z$ of the elements from \textsc{dependent z}. In the running example, $\beta = \{3, 4\}$ (corresponding to $z_3$ and $z_4$). From the definition of \textsc{dependent z}, these $z$ elements of $\beta$ appear in ascending order.
			\item $\beta'$: the respective indices in $w'$ of the variables from \textsc{dependent z}. In the running example, $\beta' = \{3, 1\}$. These $z$ elements of $\alpha'$ \emph{elements will not necessarily be present in ascending order.}
		\end{itemize}
		Again, note that the elements in the ``vanilla'' tuple correspond to variable indices, while elements in the primed tuple correspond to tuple indices. Also, \textbf{observe} the invariant $z_{\beta_i} = w'_{\beta'_i}$.
	\item \textsc{dependent w}: the tuple of variables from $w$ that (partially) comprise $w'$ in the order in which they are found in $w$ (i.e., the elements of \textsc{dependent w} appear in ascending order, e.g., $\{ w_1, w_2 \}$). In the running example, \textsc{dependent w} is $\{ w_2 \}$.
		\begin{itemize}
			\item $\bar{\alpha}$: the positions in $w$ of the elements from \textsc{dependent w}. In the running example, $\bar{\alpha} = \{ 2 \}$. From the definition of \textsc{dependent w}, these $w$ elements of $\bar{\alpha}$ appear in ascending order.
			\item $\bar{\alpha}'$: the respective indices in $w'$ of the variables from \textsc{dependent w}. In the running example, $\bar{\alpha}' = \{ 2 \}$. These $w$ elements of $\bar{\alpha}$'s \emph{elements are not necessarily present in ascending order}.
		\end{itemize}
		Again, note that the elements in the ``vanilla'' tuple correspond to variable indices, while elements in the primed tuple correspond to tuple indices. Also, \textbf{observe} the invariant $w_{\bar{\alpha}_i} = w'_{\bar{\alpha}'_i}$.
	\item \textsc{independent z}: the tuple of variables from $z$ that (partially) comprise $z'$ in the order in which they are found in $z$ (i.e., the elements of \textsc{independent z} appear in ascending order, e.g., $\{ z_1, z_2 \}$). In the running example, \textsc{independent z} is $\{ z_1, z_2 \}$. The elements in \textsc{independent z} are present in ascending order (e.g., $\{ z_1, z_2 \}$).
		\begin{itemize}
			\item $\bar{\beta}$: one greater, respectively, than the positions in $z$ of the elements from \textsc{independent z}. In the running examples, $\bar{\beta} = \{1, 2 \}$. From the definition of \textsc{independent z}, the elements of $\bar{\beta}$'s elements appear in ascending order.
			\item $\bar{\beta}'$: the respective indices in $z'$ of the variables from \textsc{independent z}. In the running example, $\bar{\beta}' = \{4, 3 \}$. These $z$ elements of $\bar{\beta}'$ \emph{elements are not necessarily present in ascending order}.
		\end{itemize}
		Once more, note that the elements in the ``vanilla'' tuple correspond to variable indices, while elements in the primed tuple correspond to tuple indices. Also, \textbf{observe} the invariant $z_{\bar{\beta}_i} = z'_{\bar{\beta}'_i}$.
\end{itemize}

\section{Approach}
\begin{lemma}
	The length of \textsc{independent w} is the same as that of \textsc{dependent z}.
\end{lemma}
We will denote the length of \textsc{independent w} as $m$. In the running example, $m = 2$.
\subsection{Computing $q'$}
We first define a \textit{square} matrix $\mat{M}^{\alpha\beta} \in\mathbb{R}^{m \times m}$ as being comprised of entries $i \in 1,\ldots,m$ and $j \in 1, \ldots, m$:
\begin{align*}
	\mat{M}^{\alpha\beta}_{ij} = \mat{M}_{\alpha_i\beta_j}
\end{align*}
and a rectangular matrix $\mat{M}^{\bar{\alpha}\beta} \in\mathbb{R}^{(n-m)\times m}$ as being comprised of entries $i \in 1,\ldots,n-m$ and $j \in 1, \ldots, m$:
\begin{align*}
	\mat{M}^{\bar{\alpha}\beta}_{ij} = \mat{M}_{\bar{\alpha}_i\beta_j}
\end{align*}

Likewise, we define vectors $\vect{q}^{\alpha} \in \mathbb{R}^m$ and $\vect{q}^{\bar{\alpha}} \in \mathbb{R}^{(n-m)}$ as being comprised of entries $i \in 1,\ldots,m$ and $j \in 1, \ldots, n-m$:
\begin{align*}
	q^{\alpha}_i & = q_{\alpha_i} \\
	q^{\bar{\alpha}}_j & = q_{\bar{\alpha}_j}
\end{align*}
In the running example---recall that $\alpha = \{1, 3\}, \beta = \{3, 4\},
\bar{\alpha} = \{2 \}$---we highlight the parts of $\vect{q}$ and $\mat{M}$ that correspond to $\textcolor{red}{q^{\alpha}}$, $\textcolor{red}{M^{\alpha\beta}}$ in red, and $\textcolor{blue}{q^{\bar{\alpha}}}$, $\textcolor{blue}{M^{\bar{\alpha}\beta}}$ in blue. Now putting the example into ``tableaux form'':
\begin{center}
	\begin{tabular}{|c|c|c c c c|}
		\hline
		      &  & $z_1$ & $z_2$ & $\textcolor{red}{z_3}$ & $\textcolor{red}{z_4}$\\
		\hline
		$\textcolor{red}{w_1}$ &\textcolor{red}{-3} & 0 & -1 & \textcolor{red}{2} & \textcolor{red}{1}\\
		$\textcolor{blue}{w_2}$ & \textcolor{blue}{6} & 2 & 0 & \textcolor{blue}{-2} & \textcolor{blue}{1}\\
		$\textcolor{red}{w_3}$ & \textcolor{red}{-1} & -1 & -1 & \textcolor{red}{0} & \textcolor{red}{1}\\
		\hline
	\end{tabular}
\end{center}
thereby yielding the following vectors and matrices:
\begin{align*}
		\textcolor{red}{q^{\alpha} = \begin{bmatrix} -3 \\ -1\end{bmatrix}},
		\textcolor{blue}{q^{\bar{\alpha}} = \begin{bmatrix} 6\end{bmatrix}},
		\textcolor{red}{M^{\alpha\beta} = \begin{bmatrix} 2 & 1 \\ 0 & 1\end{bmatrix}},
		\textcolor{blue}{M^{\bar{\alpha}\beta} = \begin{bmatrix} -2 & 1 \end{bmatrix}}
\end{align*}
With the index sets $\alpha', \bar{\alpha}'$ defined in Subsection
\ref{subsec:terminology}, and Equation 10 from Page 71 of~\cite{Cottle:1992}, we
can write $\vect{q}'$ as:
\begin{align}
	\vect{q}'^{\beta'} = -(\mat{M}^{\alpha\beta})^{-1}\vect{q}^{\alpha}
 \tag{2} \\
  % NOTE: Do not change equation number without changing
  % unrevised_lemke_solver.cc
	\vect{q}'^{\bar{\alpha}'} = \vect{q}^{\bar{\alpha}} +
    \mat{M}^{\bar{\alpha}\beta}\vect{q}'^{\beta'}
 \tag{3}
  % NOTE: Do not change equation number without changing
  % unrevised_lemke_solver.cc
\end{align}
where $\vect{q}'^{\alpha'}, \vect{q}'^{\bar{\alpha}'}$ are ``views'' of the
vector $q'$, as $\vect{q}'^{\alpha'}_i = \vect{q}'_{\alpha'_i},
\vect{q}'^{\bar{\alpha}'}_i = \vect{q}'_{\bar{\alpha}'_i}$. In the running
example $\vect{q}'^{\alpha'} = \tr{\begin{bmatrix}1 & 1\end{bmatrix}},
\vect{q}'^{\bar{\alpha}'} = [5]$. \textbf{$\vect{q}'$ is not composed by
stacking $\vect{q}'^{\alpha'}, \vect{q}'^{\bar{\alpha}'}$ as is done in similar procedures in~\cite{Cottle:1992}.}

\subsection{Computing $\mat{M}'_{\textrm{driving}}$}
This computation requires considering two cases for the driving variable, which is an entry in the new independent variable $z'$. The driving variable can be either:
\begin{enumerate}
		\item a dependent variable from $w$.
		\item an independent variable $z$.
\end{enumerate}
in the running example, if the driving variable is $w_1$ or $w_3$, then it belongs to the first case; if the driving variable is $z_2$ or $z_1$, then it belongs to the second case. Let's discuss the two cases separately.

\subsubsection{Driving variable is a dependent variable from $w$}
If the driving variable $z'_{\textrm{driving}}$ is a dependent variable from $w$, then it also belongs to the vector \textsc{independent w}. We denote the index of $z'_{\textrm{driving}}$ in \textsc{independent w} as $\gamma$, namely:
\begin{align}
	w^{\alpha}_\gamma = w_{\alpha_\gamma} = z'_{\textrm{driving}} \tag{4}
  % NOTE: Do not change equation number without changing
  % unrevised_lemke_solver.cc
\end{align}
where $w^{\alpha}$ is the vector \textsc{independent w}. In the running example,
\textsc{independent w} is $\{ w_1, w_3 \}$, thus $\alpha = \{ 1, 3 \}$. If
$z'_{\textrm{driving}} \equiv w_1$, then the index $\gamma \equiv 1$. If
$z'_{\textrm{driving}} \equiv w_3$, then the index $\gamma \equiv 2$. To compute
the column vector $\mat{M}'_{\textrm{driving}}$, we need to first compute the
$\gamma^{\textrm{th}}$ column of the matrix $(\mat{M}^{\alpha\beta})^{-1}$. To
this end, we define a unit vector $\vect{e} \in\mathbb{R}^m$ as:
\begin{align*}
	&e_{\gamma} = 1\\
	&e_i = 0 \quad \text{if } i \neq \gamma
\end{align*}
and we can compute $M'_{\textrm{driving}}$ as:
\begin{align}
	&\mat{M}'^{\beta'}_{\textrm{driving}} =
    (\mat{M}^{\alpha\beta})^{-1}\vect{e}\label{eq:b_prime_alpha_prime}
    \tag{5}\\
  % NOTE: Do not change equation number without changing
  % unrevised_lemke_solver.cc
	&\mat{M}'^{\bar{\alpha}'}_{\textrm{driving}} =
    \mat{M}^{\bar{\alpha}\beta}\mat{M}'_{\textrm{driving}_{\beta'}} \tag{6}
  % NOTE: Do not change equation number without changing
  % unrevised_lemke_solver.cc
\end{align}
Notice that $\mat{M}'_{\textrm{driving}_{\alpha'}}$ is the
$\gamma^{\textrm{th}}$ column of $\mat{M}_{\alpha\beta}^{-1}$ according to Equation~\eqref{eq:b_prime_alpha_prime}.

\subsubsection{Driving variable is an independent variable from $z$}
If the driving variable $z'_{\textrm{driving}}$ is an independent variable from $z$, we denote the position in $z$ of $z'_{\textrm{driving}}$ as $\zeta$. In the running example, if the driving variable is $z_2$, then $\zeta \equiv 2$; if the driving variable is $z_1$, then $\zeta \equiv 1$.

To compute the column vector $\mat{M}'_{\textrm{driving}}$ in $\mat{M}'$, we
need the $\zeta^{\textrm{th}}$ column of $\mat{M}$, denoted $\vect{g}$, which
will be decomposed into two sub-vectors. One sub-vector  $\vect{g}^{\alpha}$
contains the entries with row indices in $\alpha$ while the other,  $\vect{g}^{\bar{\alpha}}$, contains the entries with row indices from $\bar{\alpha}$. Namely:
\begin{align*}
	&g^{\alpha}_i = M_{\alpha_i\zeta}\\
	&g^{\bar{\alpha}}_i = M_{\bar{\alpha}_i\zeta}
\end{align*}

In the running example, $\textcolor{red}{w_1, w_3}$ will be pivoted to $z'$, and
$\textcolor{blue}{w_2}$ will remain in $w'$. If the driving variable is $z_2$,
we highlight the vector $\textcolor{red}{g^{\alpha}},
\textcolor{blue}{g^{\bar{\alpha}}}$ in $\mat{M}$ as:
\begin{center}
\begin{tabular}{|c|cccc|}
	\hline
		& $z_1$ & $z_2$ & $z_3$ & $z_0$\\
		\hline
		$\textcolor{red}{w_1}$ & 0 & \textcolor{red}{-1} & 2 & 1\\
		$\textcolor{blue}{w_2}$ & 2 & \textcolor{blue}{0} & -2 & 1\\
		$\textcolor{red}{w_3}$ & -1 & \textcolor{red}{1} & 0 & 1\\
		\hline
\end{tabular}
\end{center}
so $\textcolor{red}{g^{\alpha}} \equiv \tr{\begin{bmatrix}-1 & 1\end{bmatrix}}, \textcolor{blue}{g^{\bar{\alpha}}} = [0]$.

with the vector $\vect{g}^{\alpha}, \vect{g}^{\bar{\alpha}}$, we can then
compute $\mat{M}'_{\textrm{driving}}$ as
\begin{align}
	&\mat{M}'^{\beta'}_{\textrm{driving}} =
    -(\mat{M}^{\alpha\beta})^{-1}\vect{g}^{\alpha} \tag{7} \\
  % NOTE: Do not change equation number without changing
  % unrevised_lemke_solver.cc
	&\mat{M}'^{\bar{\alpha}'}_{\textrm{driving}} = \vect{g}^{\bar{\alpha}} +
    \mat{M}^{\bar{\alpha}\beta}\mat{M}'^{\beta'}_{\textrm{driving}} \tag{8}
  % NOTE: Do not change equation number without changing
  % unrevised_lemke_solver.cc
\end{align}

\bibliographystyle{abbrv}
\begin{thebibliography}{56}
\bibitem{Cottle:1992}
Richard W. Cottle, Jong-Shi Pang, and R.E. Stone. \emph{The Linear
Complementarity Problem}, Academic Press, 1992.
\end{thebibliography}

\end{document}


\chapter{Experiment Setup And Execution}
\section{Goals}
As already mentioned this study is investigating the coverage of the parts which already can be automatically optimized by Polly, the various reasons for these parts not being even larger and the impact to the coverage and the speedup if these reasons could be eliminated.

\section{Measurement Methology}
For measuring, analysis and evaluation of the results the following definitions are introduced.
\subsection{Definition Coverage}
Let \(T_i\in\mathbb{N}\) be the execution time of all instructions within all \scops of a project and \(T\in\mathbb{N}\) the overall execution time of that project.
Then the coverage of parallelizable regions \(DynCov\in\mathbb{Q}\) is defined as:
\[DynCov := \frac{T_i}{T}\]

\subsection[Amdahl's Law]{Amdahl's Law \cite{AmdahlsLaw}}
Let \(N\in\mathbb{N}\) be the number of processors and \(DynCov\in\mathbb{Q}\) be the coverage of parallelizable regions.
Then the speedup \(S\in\mathbb{Q}\) is defined as:
\[S := \frac{N}{(1-DynCov)*N+DynCov}\]
\subsection{Reasons for SCoPs being invalid}
These are the criteria implemented by Polly for rejecting a parent of a \scop being also a \scop.
Some of these criteria are referencing to a \llvm specific value called \undefv \cite{llvmUndef}.
\begin{itemize}
    \item Parent is top level (\autoref{lst:parentIsToplevel})\\
        This is simply when the parent of a \scop is a top level region.
        Per default a top level region can not be a \scop in Polly.
    \item Unsupported terminator instruction\\
        This is occurring when a flow breaking instruction like \texttt{return}, \texttt{throw} or \texttt{goto} is called within a loop.
    \item Unreachable in exit block (\autoref{lst:unreachableExitBlock})\\
        This is appearing when \texttt{unreachable()} is called within an exit block \cite{llvmUnreachable}.
    \item Irreducible regions (\autoref{lst:irreducibleLoops})\\
        \Eg a region is interfering with another region using labels.
        So that the loop can not be changed in general.
    \item Undefined branch condition\\
        It is occurring whenever at least a part of a branch condition is based on the value \undefv.
    \item Non-integer branch condition\\
        Appears if the branch condition is neither constant nor using an integer comparison.
    \item Undefined operands in comparison\\
        Arises when the value \undefv is used in a comparison.
    \item Non-affine branch condition\\
        This is the case if the branch is not affine like it contains a function call.
    \item No base pointer\\
        Occurs if a base pointer is missing.
    \item Undefined base pointer\\
        It is raised when a base pointer holds the value \undefv.
    \item Variant base pointer (\autoref{lst:variantBasePointer})\\
        Occurs when a base pointer in a region is not invariant.
    \item Non-affine memory accesses (\autoref{lst:nonAffineMemoryAccesses})\\
        This is appearing when accessing memory in a non-affine way like using \(i^2\) as access steps.
        More precisely: Appears if the array subscript of an access is not affine.
    \item Accesses with differing sizes\\
        Arises if an array is accessed using data types of differing sizes.
    \item Uncomputable loop bounds (\autoref{lst:uncomputeableLoopBounds})\\
        Appears if Polly fails to derive an affine loop bound \eg the loop bounds could not be computed because they are depending on an input parameter.
    \item Loop without exit (\autoref{lst:loopWithoutExit})\\
        Occurs if the loop has no exit or if not all latches are part of the loop region.
    \item Function call with side effects (\autoref{lst:functionCallSideEffects})\\
        This is raised when a function is called which is not guaranteed to have no side effects or at least it is not known whether it has side effects.
    \item Complicated access semantics (volatile or atomic)\\
        Arises if memory is accessed which is not \enquote{simple}.
        This means the keywords volatile is used or an atomic variable is accessed.
    \item Base address aliasing (\autoref{lst:baseAddressAliasing})\\
        Occurs if Polly can not determine whether two pointer are accessing the same memory or -- when talking about arrays -- whether these arrays overlap.
    \item Integer to pointer conversion (\autoref{lst:integerToPointer})\\
        This occurs when a regular int is used as pointer somewhere into the memory.
    \item Stack allocations\\
        Stack allocations like using \texttt{alloca} can not be handled.
    \item Unknown instructions\\
        Arises if an instruction is used where Polly knows no definition.
    \item Contains entry block\\
        Polly is yet not able to handle regions which contain the entry of a function.
    \item Assumed to be unprofitable (\autoref{lst:assumedUnprofitable})\\
        When the region is very small or it is not likely that an optimization improves the execution it is left out.
        This means at least Polly is not able to find a profitable polyhedral optimization for the given region.
        As already mentioned in this study also these unprofitable regions are discussed.
    \item Polly does not return a reason\\
        When requesting the reason for a region not being a \scop Polly may return an empty string.
        This occurs if the Log object used internally has errors or if it does not exist at all.
\end{itemize}

\begin{comment}
    Following is copy\&pasted from pollys ScopDetectionDiagnostic.cpp

SCOP_STAT(CFG, ""),
SCOP_STAT(InvalidTerminator, "Unsupported terminator instruction"),
SCOP_STAT(UnreachableInExit, "Unreachable in exit block"),
SCOP_STAT(IrreducibleRegion, "Irreducible loops"),
SCOP_STAT(LastCFG, ""),
SCOP_STAT(AffFunc, ""),
SCOP_STAT(UndefCond, "Undefined branch condition"),
SCOP_STAT(InvalidCond, "Non-integer branch condition"),
SCOP_STAT(UndefOperand, "Undefined operands in comparison"),
SCOP_STAT(NonAffBranch, "Non-affine branch condition"),
SCOP_STAT(NoBasePtr, "No base pointer"),
SCOP_STAT(UndefBasePtr, "Undefined base pointer"),
SCOP_STAT(VariantBasePtr, "Variant base pointer"),
SCOP_STAT(NonAffineAccess, "Non-affine memory accesses"),
SCOP_STAT(DifferentElementSize, "Accesses with differing sizes"),
SCOP_STAT(LastAffFunc, ""),
SCOP_STAT(LoopBound, "Uncomputable loop bounds"),
SCOP_STAT(LoopHasNoExit, "Loop without exit"),
SCOP_STAT(FuncCall, "Function call with side effects"),
SCOP_STAT(NonSimpleMemoryAccess,
          "Compilated access semantics (volatile or atomic)"),
SCOP_STAT(Alias, "Base address aliasing"),
SCOP_STAT(Other, ""),
SCOP_STAT(IntToPtr, "Integer to pointer conversions"),
SCOP_STAT(Alloca, "Stack allocations"),
SCOP_STAT(UnknownInst, "Unknown Instructions"),
SCOP_STAT(Entry, "Contains entry block"),
SCOP_STAT(Unprofitable, "Assumed to be unprofitable"),
SCOP_STAT(LastOther, "")
\end{comment}

\section{Experiment Variables}
The experiment variables are listed in \autoref{tab:experimentVariables}.\\
The classification \(C\) describes two types of regions handled within the study.
The one is \enquote{\scop} like explained in \autoref{subsec:definitionScop}.
The other is \enquote{parent} which is the next bigger region surrounding a \scop.\\
The variables \dyncovs and \dyncovp are depending on \(TI\) because the selection of tests to run influences the regions executed and how often they are executed.
So \(TI\) has to be controlled.
This is done by using on the one hand integrated benchmarks brought by the tested programs and on the other hand using benchmarks which are accepted by the community like SPEC2006. (see \autoref{tab:subjectPrograms})\\
Also the speedups \(S_s\) and \(S_p\) are dependent because they are calculated out of \dyncovs and \dyncovp.
\begin{table}[H]
    \myfloatalign
    \small
    \begin{tabularx}{\textwidth}{Xccccc} \toprule
        \tableheadline{Name}              & \tableheadline{Abbr.} & \tableheadline{Type} & \tableheadline{Scale Type} & \tableheadline{Unit}                          & \tableheadline{Range} \\ \midrule
        Classification                    & \(C\)                 & Indep.               & Nominal                    & \makecell{\{Parent,\\\scop\}}                 & Text\\
        Test inputs                       & \(TI\)                & Con.                 & Nominal                    & \makecell{see\\\autoref{tab:subjectPrograms}} & Text\\
        \midrule
        Coverage of \scops                & \dyncovs              & Dep.                 & Ratio                      & \%                                            & \([0; 1[ \in \mathbb{Q}\)\\
        Coverage of MaxRegions            & \dyncovp              & Dep.                 & Ratio                      & \%                                            & \([0; 1[ \in \mathbb{Q}\)\\
        Theoretical speedup of \scops     & \(S_s\)               & Dep.                 & Ratio                      & \%                                            & \([0; 1[ \in \mathbb{Q}\)\\
        Theoretical speedup of MaxRegions & \(S_p\)               & Dep.                 & Ratio                      & \%                                            & \([0; 1[ \in \mathbb{Q}\)\\
        \bottomrule
    \end{tabularx}
    \caption[Experiment Variables]{
        Experiment variables (Dep.=Dependent; Indep.=Independent; Con.=Controlled).
        The term \enquote{MaxRegions} contains all regions classified as parent and the \scops which have only a parent which is a top level region.
    }
    \label{tab:experimentVariables}
\end{table}
Besides the experiment variables also the term \enquote{MaxRegions} has to be introduced.
MaxRegions contains all regions classified as parent and the \scops which have only a parent which is a top level regions.

\section{Hypotheses}
\(H_1\): The main reason for rejecting a \scop should be that its parent already is a top level region because also all \scops classified as unprofitable by Polly are processed.
So a lot of very small functions may be processed.
Whenever such a function has a \scop it is likely that its parent is already a top level region.\\
\(H_2\): The coverage \dyncovs is expected to be about \hTwoAbout, on average.
Even if there are a lot of \scops which are processed the -- according to the execution time -- big \scops may not be able to be optimized, on average.\\
\(H_3\): It is expected that \dyncovp \(\gg\) \dyncovs, on average.
This may be a result of the fact that \dyncovp ignores whether the reasons of parents being rejected as \scop can be eliminated or not.\\

\section{Subject Programs}
The intention is to measure programs which are well known and often used by the community to get a more realistic picture.
The selection of programs (\autoref{tab:subjectPrograms}) is limited due the list of programs available using benchbuild\footnote{Benchbuild is a tool for automating the process of downloading, building, applying an experiment at compile time and executing a project on a SLURM \cite{slurm} Cluster.} \cite{benchbuild}.
\LTXtable{\textwidth}{tables/subjectPrograms.tex}

\section{Tasks}
\begin{sloppypar}
    We use benchmarks and scenarios integrated into the projects or test-suites that are respected by the corresponding development communities to get as close to a real-world scenario as possible.
    Furthermore, \lnt provides a whole range of benchmarks used by \llvm.
    Projects annotated with \enquote{\(program_{lnt}\)} belong to \lnt.
\end{sloppypar}

\section{Design}
Based on the experiment variables (see \autoref{tab:experimentVariables}) three experiments are performed to validate the hypotheses.\\
The first experiment takes the most common reason for rejecting a parent as valid \scop and checks whether it is \enquote{parent is top level region}.\\
In the second experiment the average value of \dyncovs is calculated and checked whether it is near \hTwoAbout.\\
The third experiment \dyncovp and \dyncovs are compared to check whether \dyncovp \(\gg\) \dyncovs.

\section{Experiment Setting}\label{sec:experimentSettings}
\begin{sloppypar}
    This study uses instrumentation to retrieve precise timing information instead of using frameworks like OProfile \cite{oprofile} or GProf \cite{gprof}.
    The measurement is realized using the \papi library \cite{papi}.\\
    To take these measurement the following steps are performed:\\
    The first step is implementing a new pass for Polly for instrumenting regions.
    Afterwards a so called experiment for benchbuild \cite{benchbuild} has to be created.
    This experiment is calling the actual pass of Polly by specifying the appropriate clang options using \texttt{-Xclang}, collects the generated data and persists it.\\
    The pass itself is a FunctionPass.
    Thus it is processing every function of the program.
    In this case the pass is called twice.
    Once for instrumenting the actual \scops within, which also includes regions classified by Polly as \enquote{unprofitable}, and once for instrumenting their parents.
    Even the \enquote{unprofitable} \scops are included because they may get profitable to optimize when they are extended to the size of their parents.
    To include these \scops the option \texttt{-polly-process-unprofitable} has to be specified using the \texttt{-mllvm} option of clang.
    While doing so the information about why these parents were rejected as \scops are collected.\\
    The instrumentation is designed to work for any region (\autoref{subsec:definitionRegion}).
    For a given region every incoming edge in the \cfg (\autoref{subsec:cfg}) is bent to a new node splitting the edges and pointing to the entry of the region.
    Within this new node an instruction is placed starting the measurement of the execution time.
    This instruction has the same for regions unique id and remembers the system time at the point where it was called.
    The same way every outgoing edges in the \cfg of this region is also split introducing a further new node in which an instruction is placed for stopping the measurement having the same id.
    When it is called the current system time is requested again and the difference to the starting call is calculated providing the duration of the measured region.
    The instrumented version of \autoref{lst:matmulcpp} is shown in \autoref{fig:afterInstrument} (The nodes having names ending with \enquote{.profile.exit.split} are the newly introduced nodes).
    \begin{figure}[!h]
        \caption[Example of instrumented method]{
            This figure shows \autoref{lst:matmulcpp} after it is instrumented.
            The nodes introduced through the instrumentation are the nodes having names ending with \enquote{.profile.exit.split}.
            In this case Polly warns about \enquote{Call instruction: tail call @enter\_region(...) [...]} which are exactly these entry calls.
        }
        \includegraphics[width=\textwidth]{gfx/matmulScopsAfterInstrumentation.png}
        \label{fig:afterInstrument}
    \end{figure}
    The exact versions of the used software components are listed in \autoref{tab:usedSoftware}.\\
    All measurements are executed on an Intel Xeon E5-2650 v2 machine (2 Sockets with each 8 physical cores each 2.6GHz) with 128GB RAM.
    The precise specifications are listed in \autoref{tab:lscpu} and \autoref{tab:meminfo}.
\end{sloppypar}
\begin{table}[!h]
    \myfloatalign
    \begin{tabularx}{.5\textwidth}{Xc}
        \tableheadline{Name} & \tableheadline{Version}\\ \toprule
        benchbuild           & (2bd0060)\\
        LLVM                 & (04facbb8)\\
        clang                & (89be37b)\\
        polly                & (0fd4fea)\\
        polli                & (697351d)\\
        \bottomrule
    \end{tabularx}
    \caption[Software used for measurements]{
        This table lists the software used for the measurements.
        The components LLVM, clang, polly and polli belong to the compiler infrastructure \llvm itself.
        benchbuild is a program for applying a certain \enquote{experiment} (see \autoref{sec:experimentSettings}) on a selection of programs, distributing them on a SLURM \cite{slurm} cluster and run them.
        (Versions in parenthesis represent short git hashes)
    }
    \label{tab:usedSoftware}
\end{table}

\section{Deviations}\label{sec:deviations}
Only about \usefulRatio of the measured execution times can be used due to on the one hand bugs in the implementation and one the other hand the projects measured which could not be fixed because of time constraints.
The bug in the implementation is occurring in certain constellations of the basic blocks when instrumenting a parent like the following one which is found in the project bzip2 within the file bzlib.c:
\begin{code}
    \caption[Example of a non working parent]{
        This lists a method within the file bzlib.c of the project bzip2.
        The current instrumentation is not working when instrumenting regions within this method.
    }
    \inputminted{c}{c/notWorkingParent.c}
    \label{lst:notWorkingParent}
\end{code}
To cope with it measurements of regions which have a bigger duration than the hole execution time of the project are excluded.
Also measurements of coverages are excluded where the coverage is greater than 100\%.
These occur because the filtering of the obviously bad measurements is not enough to exclude all bad ones.\\
Furthermore for minimizing the influences of the underlying OS multithreading is deactivated, the cluster node is used exclusively, the number of tasks is limited to one and the number of CPUs used per task is limited to 10.


\section{Accounting for the variability in gene copy number during the cell
cycle} \label{supp_multi_gene}

(Note: The Python code used for the calculations presented in this section can
be found in the
\href{https://www.rpgroup.caltech.edu/chann_cap/src/theory/html/moment_dynamics_cell_division.html}{following
link} as an annotated Jupyter notebook)

When growing in rich media, bacteria can double every $\approx$ 20 minutes.
With two replication forks each traveling at $\approx$ 1000 bp per second, and
a genome of $\approx$ 5 Mbp for {\it E. coli} \cite{Moran2010}, a cell would
need $\approx$ 40 minutes to replicate its genome. The apparent paradox  of
growth rates faster than one division per 40 minutes is solved by the fact that
cells have multiple replisomes, i.e. molecular machines that replicate the
genome running in parallel. Cells can have up to 8 copies of the genome being
replicated simultaneously depending on the growth rate \cite{Bremer1996}.

This observation implies that during the cell cycle gene copy number varies.
This variation depends on the growth rate and the relative position of the gene
with respect to the replication origin, having genes close to the replication
origin spending more time with multiple copies compare to genes closer to the
replication termination site. This change in gene dosage has a direct effect on
the cell-to-cell variability in gene expression \cite{Jones2014a,
Peterson2015}.

\subsection{Numerical integration of moment equations}

(Note: The Python code used for the calculations presented in this section can
be found in the
\href{https://www.rpgroup.caltech.edu//chann_cap/software/moment_dynamics_cell_division.html}{following
link} as an annotated Jupyter notebook)

For our specific locus ({\it galK}) and a doubling time of $\approx$ 60 min for
our experimental conditions, cells have on average 1.66 copies of the reporter
gene during the cell cycle \cite{Jones2014a}. What this means is that cells
spend 60\% of the time having one copy of the gene and 40\% of the time with
two copies. To account for this variability in gene copy number across the cell
cycle we numerically integrate the moment equations derived in
\siref{supp_moments} for a time $t = [0, t_s]$ with an mRNA production rate
$r_m$, where $t_s$ is the time point at which the replication fork reaches our
specific locus. For the remaining time before the cell division $t = [t_s,
t_d]$ that the cell spends with two promoters, we assume that the only
parameter that changes is the mRNA production rate from $r_m$ to $2 r_m$. This
simplifying assumption ignores potential changes in protein translation rate
$r_p$ or changes in the repressor copy number that would be reflected in
changes on the repressor on rate $\kron$.

\subsubsection{Computing distribution moments after cell division}

(Note: The Python code used for the calculations presented in this section can
be found in the
\href{https://www.rpgroup.caltech.edu//chann_cap/software/binomial_moments.html}{following
link} as an annotated Jupyter notebook)

We have already solved a general form for the dynamics of the moments of the
distribution, i.e. we wrote differential equations for the moments ${d\ee{m^x
p^y}\over dt}$. Given that we know all parameters for our model we can simply
integrate these equations numerically to compute how the moments of the
distribution evolve as cells progress through their cell cycle. Once the cell
reaches a time $t_d$ when is going to divide the mRNA and proteins that we are
interested in undergo a binomial partitioning between the two daughter cells.
In other words, each molecule flips a coin and decides whether to go to either
daughter. The question then becomes given that we have a value for the moment
$\ee{m^x p^y}_{t_d}$ at a time before the cell division, what would the value
of this moment be after the cell division takes place $\ee{m^x p^y}_{t_o}$?

The probability distribution of mRNA and protein after the cell division
$P_{t_o}(m, p)$ must satisfy
\begin{equation}
  P_{t_o}(m, p) = \sum_{m'=m}^\infty \sum_{p'=p}^\infty
                  P(m, p \mid m', p') P_{t_d}(m', p'),
\label{eq_dist_post_div}
\end{equation}
where we are summing over all the possibilities of having $m'$ mRNA and $p'$
proteins before cell division. Note that the sums start at $m$ and $p$; this is
because for a cell to have these copy numbers before cell division it is a
requirement that the mother cell had at least such copy number since we are not
assuming that there is any production at the instantaneous cell division time.
Since we assume that the partition of mRNA is independent from the partition of
protein, the conditional probability $P(m, p \mid m', p')$ is simply given by a
product of two binomial distributions, one for the mRNA and one for the
protein, i.e.
\begin{equation}
P(m, p \mid m', p') = {m' \choose m} \left( {1 \over 2} \right)^{m'} \cdot
                      {p' \choose p} \left( {1 \over 2} \right)^{p'}.
\label{eq_binom_prod}
\end{equation}
Because of these product of binomial probabilities are allowed to extend the
sum from
\eref{eq_dist_post_div} to start at $m'=0$ and $p'=0$ as
\begin{equation}
  P_{t_o}(m, p) = \sum_{m'=0}^\infty \sum_{p'=0}^\infty
                  P(m, p \mid m', p') P_{t_d}(m', p'),
\end{equation}
since the product of the binomial distributions in \eref{eq_binom_prod} is zero
for all $m' < m$ and/or $p' < 0$. So from now on in this section we will assume
that a sum of the form $\sum_x \equiv \sum_{x=0}^\infty$ to simplify notation.

We can then compute the distribution moments after the cell division $\ee{m^x
p^y}_{t_o}$ as
\begin{equation}
\ee{m^x p^y}_{t_o} = \sum_m \sum_p m^x p^y P_{t_o}(m, p),
\end{equation}
for all $x, y \in \mathbb{N}$. Substituting \eref{eq_dist_post_div} results in
\begin{equation}
\ee{m^x p^y}_{t_o} = \sum_m \sum_p m^x p^y
\sum_{m'} \sum_{p'} P(m, p \mid m', p') P_{t_d}(m', p').
\end{equation}
We can rearrange the sums to be
\begin{equation}
\ee{m^x p^y}_{t_o} = \sum_{m'} \sum_{p'} P_{t_d}(m', p')
                     \sum_m \sum_p m^x p^y P(m, p \mid m', p').
\end{equation}
The fact that \eref{eq_binom_prod} is the product of two independent events
allows us to rewrite the joint probability $P(m, p \mid m', p')$ as
\begin{equation}
P(m, p \mid m', p') = P(m \mid m') \cdot P(p \mid p').
\end{equation}
With this we can then write the moment $\ee{m^x p^y}_{t_o}$ as
\begin{equation}
\ee{m^x p^y}_{t_o} = \sum_{m'} \sum_{p'} P_{t_d}(m', p')
                     \sum_m  m^x  P(m \mid m')
                     \sum_p p^y P(p \mid p').
\end{equation}
Notice that both terms summing over $m$ and over $p$ are the conditional
expected values, i.e.
\begin{equation}
\sum_z  z^x  P(z \mid z') \equiv \ee{z^x \mid z'}, \;
{\text{ for } z\in \{m, p \}}.
\end{equation}
These conditional expected values are the expected values of a binomial random
variable $z \sim \text{Bin}(z', 1/2)$, which can be easily computed as we will
show later in this section. We then rewrite the expected values after the cell
division in terms of these moments of a binomial distribution
\begin{equation}
\ee{m^x p^y}_{t_o} = \sum_{m'} \sum_{p'} \ee{m^x \mid m'} \ee{p^y \mid p'}
                     P_{t_d}(m', p').
  \label{eq_general_binom_mom}
\end{equation}

To see how this general formula for the moments after the cell division works
let's compute the mean protein per cell after the cell division $\ee{p}_{t_o}$.
That is setting $x = 0$, and $y = 1$. This results in
\begin{equation}
\ee{p}_{t_o} = \sum_{m'} \sum_{p'} \ee{m^0 \mid m'} \ee{p \mid p'}
               P_{t_d}(m', p').
\end{equation}
The zeroth moment $\ee{m^0 \mid m'}$ by definition must be one since we have
\begin{equation}
\ee{m^0 \mid m'} = \sum_m m^0 P(m \mid m') = \sum_m P(m \mid m') = 1,
\end{equation}
since the probability distribution must be normalized. This leaves us then with
\begin{equation}
\ee{p}_{t_o} = \sum_{m'} \sum_{p'} P_{t_d}(m', p') \ee{p \mid p'}.
\end{equation}
If we take the sum over $m'$ we simply compute the marginal probability
distribution $\sum_{m'} P_{t_d}(m', p') = P_{t_d}(p')$, then we have
\begin{equation}
\ee{p}_{t_o} = \sum_{p'} \ee{p \mid p'} P_{t_d}(p').
\end{equation}
For the particular case of the first moment of the binomial distribution with
parameters $p'$ and $1/2$ we know that
\begin{equation}
\ee{p \mid p'} = {p' \over 2}.
\end{equation}
Therefore the moment after division is equal to
\begin{equation}
\ee{p}_{t_o} = \sum_{p'} {p' \over 2} P_{t_d}(p')
             = {1 \over 2} \sum_{p'} p' P_{t_d}(p').
\end{equation}
Notice that this is just 1/2 of the expected value of $p'$ averaging over the
distribution prior to cell division, i.e.
\begin{equation}
\ee{p}_{t_o} = {\ee{p'}_{t_d} \over 2},
\end{equation}
where $\ee{\cdot}_{t_d}$ highlights that is the moment of the distribution
prior to the cell division. This result makes perfect sense. What this is
saying is that the mean protein copy number right after the cell divides is
half of the mean protein copy number just before the cell division. That is
exactly we would expect. So in principle to know the first moment of either the
mRNA distribution $\ee{m}_{t_o}$ or the protein distribution $\ee{m}_{t_o}$
right after cell division it suffices to multiply the moments before the cell
division $\ee{m}_{t_d}$ or $\ee{p}_{t_d}$ by 1/2. Let's now explore how this
generalizes to any other moment $\ee{m^x p^y}_{t_o}$.

\subsubsection{Computing the moments of a binomial distribution}

The result from last section was dependent on us knowing the functional form of
the first moment of the binomial distribution. For higher moments we need some
systematic way to compute such moments. Luckily for us we can do so by using
the so-called moment generating function (MGF). The MGF of a random variable
$X$ is defined as
\begin{equation}
M_X(t) = \ee{e^{tX}},
\end{equation}
where $t$ is a dummy variable. Once we know the MGF we can obtain any moment of
the distribution by simply computing
\begin{equation}
  \ee{X^n} = \left. {d^n \over dt^n} M_X(t) \right\vert_{t=0},
  \label{eq_mgf_def}
\end{equation}
i.e. taking the $n$-th derivative of the MGF returns the $n$-th moment of the
distribution. For the particular case of the binomial distribution $X \sim
\text{Bin}(N, q)$ it can be shown that the MGF is of the form
\begin{equation}
M_X(t) = \left[ (1 - q) + qe^{t} \right]^N.
\end{equation}
As an example let's compute the first moment of this binomially distributed
variable. For this, the first derivative of the MGF results in
\begin{equation}
  {d M_X(t) \over dt} = N [(1 - q) + qe^t]^{N - 1} q e^t.
\end{equation}
We just need to follow \eref{eq_mgf_def} and set $t = 0$ to obtain the first
moment
\begin{equation}
  \left. {d M_X(t) \over dt} \right\vert_{t=0} = N q,
  \label{eq_mgf_mean}
\end{equation}
which is exactly the expected value of a binomially distributed random
variable.

So according to \eref{eq_general_binom_mom} to compute any moment $\ee{m^x
p^y}$ after cell division we can just take the $x$-th derivative and the $y$-th
derivative of the binomial MGF to obtain $\ee{m^x \mid m'}$ and $\ee{p^y \mid
p'}$, respectively, and take the expected value of the result. Let's follow on
detail the specific case for the moment $\ee{m p}$. When computing the moment
after cell division $\ee{mp}_{t_o}$ which is of
the form
\begin{equation}
\ee{mp}_{t_o} = \sum_{m'} \sum{p'} \ee{m \mid m'} \ee{p \mid p'}
                P_{t_d}(m', p'),
\end{equation}
the product $\ee{m \mid m'} \ee{p \mid p'}$ is then
\begin{equation}
\ee{m \mid m'} \ee{p \mid p'} = {m' \over 2} \cdot {p' \over 2},
\end{equation}
where we used the result in \eref{eq_mgf_mean}, substituting $m$ and $p$ for
$X$, respectively, and $q$ for 1/2. Substituting this result into the moment
gives
\begin{equation}
\ee{mp}_{t_o} = \sum_{m'} \sum_{p'} {m' p' \over 4} P_{t_d}(m', p')
              = {\ee{m' p'}_{t_d} \over 4}.
\end{equation}
Therefore to compute the moment after cell division $\ee{mp}_{t_o}$ we simply
have to divide by 4 the corresponding equivalent moment before the cell
division.

Not all moments after cell division depend only on the equivalent moment before
cell division. For example if we compute the third moment of the protein
distribution $\ee{p^3}_{t_o}$, we find
\begin{equation}
  \ee{p^3}_{t_o} = {\ee{p^3}_{t_d} \over 8} + {3 \ee{p^2}_{t_d} \over 8}.
\end{equation}
So for this particular case the third moment of the protein distribution
depends on the third moment and the second moment before the cell division. In
general all moments after cell division $\ee{m^x p^y}_{t_o}$ linearly depend on
moments before cell division. Furthermore, there is ``moment closure'' for this
specific case in the sense that all moments after cell division depend on lower
moments before cell division. To generalize these results to all the moments
computed in this work let us then define a vector to collect all moments before
the cell division up the $\ee{m^x p^y}_{t_d}$ moment, i.e.
\begin{equation}
\bb{\ee{m^x p^y}}_{t_d} = \left(
\ee{m^0 p^0}_{t_d}, \ee{m^1}_{t_d}, \ldots , \ee{m^x p^y}_{t_d}
\right).
\end{equation}
Then any moment after cell division $\ee{m^{x'} p^{y'}}_{t_o}$ for $x' \leq x$ and $y' \leq y$ can be computed as
$$
\ee{m^{x'} p^{y'}}_{t_o} = \bb{z}_{x'y'} \cdot \bb{\ee{m^x p^y}}_{t_d},
$$
where we define the vector $\bb{z}_{x'y'}$ as the vector containing all the
coefficients that we obtain with the product of the two binomial distributions.
For example for the case of the third protein moment $\ee{p^3}_{t_o}$ the
vector $\bb{z}_{x'y'}$ would have zeros for all entries except for the
corresponding entry for $\ee{p^2}_{t_d}$ and for $\ee{p^3}_{t_d}$, where it
would have $3/8$ and $1/8$ accordingly.

If we want then to compute all the moments after the cell division up to
$\ee{m^x p^y}_{t_o}$ let us define an equivalent vector
\begin{equation}
\bb{\ee{m^x p^y}}_{t_o} = \left(
\ee{m^0 p^0}_{t_o}, \ee{m^1}_{t_o}, \ldots , \ee{m^x p^y}_{t_o}
\right).
\end{equation}
Then we need to build a square matrix $\bb{Z}$ such that each row of the matrix
contains the corresponding vector $\bb{z}_{x' y'}$ for each of the moments.
Having this matrix we would simply compute the moments after the cell division
as
\begin{equation}
\bb{\ee{m^x p^x}}_{t_o} = \bb{Z} \cdot \bb{\ee{m^x p^x}}_{t_d}.
\end{equation}
In other words, matrix $\bb{Z}$ will contain all the coefficients that we need
to multiply by the moments before the cell division in order to obtain the
moments after cell division. Matrix $\bb{Z}$ was then generated automatically
using Python's analytical math library sympy \cite{sympy}.

\fref{sfig_first_mom_cycles} (adapted from \fref{fig3_cell_cycle}(B)) shows how
the first moment of both mRNA and protein changes over several cell cycles. The
mRNA quickly relaxes to the steady state corresponding to the parameters for
both a single and two promoter copies. This is expected since the parameters
for the mRNA production were determined in the first place under this
assumption (See \siref{supp_model}). We note that there is no apparent delay
before reaching steady state of the mean mRNA count after the cell divides.
This is because the mean mRNA count for the two promoters copies  state is
exactly twice the expected mRNA count for the single promoter state (See
\siref{supp_model}). Therefore once the mean mRNA count is halved after the
cell division, it is already at the steady state value for the single promoter
case. On the other hand, given that the relaxation time to steady state is
determined by the degradation rate, the mean protein count does not reach its
corresponding steady state value for either promoter copy number state.
Interestingly once a couple of cell cycles have passed the first moment has a
repetitive trajectory over cell cycles. We have observed this experimentally by
tracking cells as they grow under the microscope. Comparing cells at the
beginning of the cell cycle with the daughter cells that appear after cell
division shown that on average all cells have the same amount of protein at the
beginning of the cell cycle (See Fig. 18 of \cite{Phillips2019}), suggesting
that these dynamical steady state takes place \textit{in vivo}.

\begin{figure}[h!]
	\centering \includegraphics
  {../fig/si/figS08.pdf}
	\caption{\textbf{First and second moment dynamics over cell the cell cycle.}
	Mean $\pm$ standard deviation mRNA (upper panel) and mean $\pm$ standard
	deviation protein copy number (lower panel) as the cell cycle progresses. The
	dark shaded region delimits the fraction of the cell cycle that cells spend
	with  a single copy of the promoter. The light shaded region delimits the
	fraction of the cell cycle that cells spend with two copies of the promoter.
	For a 100 min doubling time at the {\it galK} locus cells spend 60\% of the
	time with one copy of the promoter and the rest with two copies.}
  \label{sfig_first_mom_cycles}
\end{figure}

In principle when measuring gene expression levels experimentally from an
asynchronous culture, cells are sampled from any time point across their
individual cell cycles. This means that the moments determined experimentally
correspond to an average over the cell cycle. In the following section we
discuss how to account for the fact that cells are not uniformly distributed
across the cell cycle in order to compute these averages.

\subsection{Exponentially distributed ages}

As mentioned in \siref{supp_param_inference}, cells in exponential growth have
exponentially distributed ages across the cell cycle, having more young cells
compared to old ones. Specifically the probability of a cell being at any time
point in the cell cycle is given by \cite{Powell1956}
\begin{equation}
  P(a) = (\ln 2) \cdot 2^{1 - a},
  \label{seq_age_prob}
\end{equation}
where $a \in [0, 1]$ is the stage of the cell cycle, with $a = 0$ being the
start of the cycle and $a = 1$ being the cell division. In
\siref{supp_cell_age_dist} we reproduce this derivation. It is a surprising
result, but can be intuitively thought as follows: If the culture is growing
exponentially, that means that all the time there is an increasing number of
cells. That means for example that if in a time interval $\Delta t$ $N$ ``old''
cells divided, these produced $2N$ ``young'' cells. So at any point there is
always more younger than older cells.

Our numerical integration of the moment equations gave us a time evolution of
the moments as cells progress through the cell cycle. Since experimentally we
sample asynchronous cells that follow \eref{seq_age_prob}, each time point
along the moment dynamic must be weighted by the probability of having sampled
a cell at such specific time point of the cell cycle. Without loss of
generality let's focus on the first mRNA moment $\ee{m(t)}$ (the same can be
applied to all other moments). As mentioned before, in order to calculate the
first moment across the entire cell cycle we must weigh each time point by the
corresponding probability that a cell is found in such point of its cell cycle.
This translates to computing the integral
\begin{equation}
  \ee{m}_c = \int_{\text{beginning cell cycle}}^{\text{end cell cycle}}
                       \ee{m(t)} P(t) dt,
\end{equation}
where $\ee{m}_c$ is the mean mRNA copy number averaged over the entire cell
cycle trajectory, and $P(t)$ is the probability of a cell being at a time $t$ of
its cell cycle.

If we set the time in units of the cell cycle length we can use
\eref{seq_age_prob} and compute instead
\begin{equation}
  \ee{m} = \int_0^1 \ee{m(a)} P(a) da,
  \label{seq_moment_avg}
\end{equation}
where $P(a)$ is given by \eref{seq_age_prob}.

What \eref{seq_moment_avg} implies is that in order to compute the first moment
(or any moment of the distribution) we must weigh each point in the moment
dynamics by the corresponding probability of a cell being at that point along
its cell cycle. That is why when computing a moment we take the time trajectory
of a single cell cycle as the ones shown in \fref{sfig_first_mom_cycles} and
compute the average using \eref{seq_age_prob} to weigh each time point. We
perform this integral numerically for all moments using Simpson's rule.

\subsection{Reproducing the equilibrium picture}

Given the large variability of the first moments depicted in
\fref{sfig_first_mom_cycles} it is worth considering why a simplistic
equilibrium picture has shown to be very successful in predicting the mean
expression level under diverse conditions \cite{Garcia2011c, Brewster2014,
Barnes2019, Razo-Mejia2018}. In this section we compare the simple repression
thermodynamic model with this dynamical picture of the cell cycle. But before
diving into this comparison, it is worth recapping the assumptions that go into
the equilibrium model.

\subsubsection{Steady state under the thermodynamic model}

Given the construction of the thermodynamic model of gene regulation for which
the probability of the promoter microstates rather than the probability of mRNA
or protein counts is accounted for,  we are only allowed to describe the
dynamics of the first moment using this theoretical framework
\cite{Phillips2015}. Again let's only focus on the mRNA first moment $\ee{m}$.
The same principles apply if we consider the protein first moment. We can write
a dynamical system of the form
\begin{equation}
  \dt{\ee{m}} = r_m \cdot \pbound - \gm \ee{m},
\end{equation}
where as before $r_m$ and $\gm$ are the mRNA production and degradation rates
respectively, and $\pbound$ is the probability of finding the RNAP bound to the
promoter \cite{Bintu2005a}. This dynamical system is predicted to have a single
stable fixed point that we can find by computing the steady state. When we
solve for the mean mRNA copy number at steady state $\ee{m}_{ss}$ we find
\begin{equation}
  \ee{m}_{ss} = {r_m \over \gm} \pbound.
\end{equation}

Since we assume that the only effect that the repressor has over the regulation
of the promoter is exclusion of the RNAP from binding to the promoter, we
assume that only $\pbound$ depends on the repressor copy number $R$. Therefore
when computing the fold-change in gene expression we  are left with
\begin{equation}
  \foldchange = {\ee{m (R \neq 0)}_{ss} \over \ee{m (R = 0)}_{ss}}
              = {\pbound (R \neq 0) \over \pbound (R = 0)}.
\end{equation}
As derived in \cite{Garcia2011c} this can be written in the language of
equilibrium statistical mechanics as
\begin{equation}
  \foldchange = \left(1 + {R \over \Nns}e^{-\beta \eR}  \right)^{-1},
  \label{seq_fold_change_thermo}
\end{equation}
where $\beta \equiv (k_BT)^{-1}$, $\eR$ is the repressor-DNA binding energy,
and $\Nns$ is the number of non-specific binding sites where the repressor can
bind.

To arrive at \eref{seq_fold_change_thermo} we ignore the physiological changes
that occur during the cell cycle; one of the most important being the
variability in gene copy number that we are exploring in this section. It is
therefore worth thinking about whether or not the dynamical picture exemplified
in \fref{sfig_first_mom_cycles} can be reconciled with the predictions made by
\eref{seq_fold_change_thermo} both at the mRNA and protein level.

\fref{sfig_lacI_titration} compares the predictions of both theoretical
frameworks for varying repressor copy numbers and repressor-DNA affinities. The
solid lines are directly computed from \eref{seq_fold_change_thermo}. The
hollow triangles and the solid circles, represent the fold-change in mRNA and
protein respectively as computed from the moment dynamics. To compute the
fold-change from the kinetic picture we first numerically integrate the moment
dynamics for both the two- and the three-state promoter (See
\fref{sfig_first_mom_cycles} for the unregulated case) and then average the
time series accounting for the probability of cells being sampled at each stage
of the cell cycle as defined in \eref{seq_moment_avg}. The small systematic
deviations between both models come partly from the simplifying assumption that
the repressor copy number, and therefore the repressor on rate $\kron$ remains
constant during the cell cycle. In principle the gene producing the repressor
protein itself is also subjected to the same duplication during the cell cycle,
changing therefore the mean repressor copy number for both stages.

\begin{figure}[h!]
	\centering \includegraphics
  {../fig/si/figS09.pdf}
	\caption{\textbf{Comparison of the equilibrium and kinetic reressor titration
	predictions.} The equilibrium model (solid lines) and the kinetic model with
	variation over the cell cycle (solid circles and white triangles) predictions
	are compared for varying repressor copy numbers and operator binding energy.
	The equilibrium model is directly computed from \eref{seq_fold_change_thermo}
	while the kinetic model is computed by numerically integrating the moment
	equations over several cell cycles, and then averaging over the extent of the
	cell cycle as defined in \eref{seq_moment_avg}.}
  \label{sfig_lacI_titration}
\end{figure}

For completeness \fref{sfig_IPTG_titration} compares the kinetic and
equilibrium models for the extended model of \cite{Razo-Mejia2018} in which the
inducer concentration enters into the equation. The solid line is directly
computed from Eq. 5 of \cite{Razo-Mejia2018}. The hollow triangles and solid
points follow the same procedure as for \fref{sfig_lacI_titration}, where the
only effect that the inducer is assume to have in the kinetics is an effective
change in the number of active repressors, affecting therefore $\kron$.

\begin{figure}[h!]
	\centering \includegraphics
  {../fig/si/figS10.pdf}
	\caption{\textbf{Comparison of the equilibrium and kinetic inducer titration
	predictions.} The equilibrium model (solid lines) and the kinetic model with
	variation over the cell cycle (solid circles and white triangles)
	predictions are compared for varying repressor copy numbers and inducer
	concentrations. The equilibrium model is directly computed as Eq. 5 of
	reference \cite{Razo-Mejia2018} with repressor-DNA binding energy $\eR =
	-13.5 \; k_BT$ while the kinetic model is computed by numerically
	integrating the moment dynamics over several cell cycles, and then averaging
	over the extent of a single cell cycle as defined in \eref{seq_moment_avg}.}
  \label{sfig_IPTG_titration}
\end{figure}

\subsection{Comparison between single- and multi-promoter kinetic model}

After these calculations it is worth questioning whether the inclusion of this
change in gene dosage is drastically different with respect to
the simpler picture of a kinetic model that ignores the gene copy number
variability during the cell cycle. To this end we systematically computed the
average moments for varying repressor copy number and repressor-DNA affinities.
We then compare these results with the moments obtained from a single-promoter
model and their corresponding parameters. The derivation of the steady-state
moments of the distribution for the single-promoter model are detailed in
\siref{supp_moments}.

\fref{sfig_lacI_titration} and \fref{sfig_IPTG_titration} both suggest that
since the dynamic multi-promoter model can reproduce the results of the
equilibrium model at the first moment level it must then also be able to
reproduce the results of the single-promoter model at this level (See
\siref{supp_param_inference}). The interesting comparison comes with higher
moments. A useful metric to consider for gene expression variability is the
noise in gene expression \cite{Shahrezaei2008}. This quantity, defined as the
standard deviation divided by the mean, is a dimensionless metric of how much
variability there is with respect to the mean of a distribution. As we will
show below this quantity differs from the also commonly used metric known as
the Fano factor (variance / mean) in the sense that for experimentally
determined expression levels in fluorescent arbitrary units, the noise is a
dimensionless quantity while the Fano factor is not.

\fref{sfig_noise_comparison} shows the comparison of the predicted protein
noise between  the single- (dashed lines) and the multi-promoter model (solid
lines) for different operators and repressor copy numbers. A striking
difference between both is that the single-promoter model predicts that as the
inducer concentration increases, the standard deviation grows much slower than
the mean, giving a very small noise. In comparison the multi-promoter model has
a much higher floor for the lowest value of the noise, reflecting the expected
result that the variability in gene copy number across the cell cycle should
increase the cell-to-cell variability in gene expression \cite{Peterson2015,
Jones2014a}

\begin{figure}[h!]
	\centering \includegraphics
  {../fig/si/figS11.pdf}
	\caption{\textbf{Comparison of the predicted protein noise between a single-
	and a multi-promoter kinetic model.} Comparison of the noise
	(standard deviation/mean) between a kinetic model that considers a single
	promoter at all times (dashed line) and the multi-promoter model developed
	in this section (solid line) for different repressor operators. (A) Operator
	O1,  $\eR = -15.3 \; k_BT$, (B) O2, $\eR = -13.9 \; k_BT$, (C) O3, $\eR =
	-9.7 \; k_BT$}
  \label{sfig_noise_comparison}
\end{figure}

\subsection{Comparison with experimental data}\label{supp_theory_vs_data_mom}

Having shown that the kinetic model presented in this section can not only
reproduce the results from the equilibrium picture at the mean level (See
\fref{sfig_lacI_titration} and \fref{sfig_IPTG_titration}), but make predictions
for the cell-to-cell variability as quantified by the noise (See
\fref{sfig_noise_comparison}), we can assess whether or not this model is able
to predict experimental measurements of the noise. For this we take the single
cell intensity measurements (See Methods) to compute the noise at the protein
level.

As mentioned before this metric differs from the Fano factor since for
fluorescent arbitrary units the noise is a dimensionless quantity. To see why
consider that the noise is defined as
\begin{equation}
\text{noise} \equiv \frac{\sqrt{\left\langle p^2 \right\rangle -
                        \left\langle p \right\rangle^2}}
                        {\left\langle p \right\rangle}.
    \label{seq_noise_protein}
\end{equation}
We assume that the intensity level of a cell $I$ is linearly proportional to
the absolute protein count, i.e.
\begin{equation}
I = \alpha p,
\label{seq_calibration_factor}
\end{equation}
where $\alpha$ is the proportionality constant between arbitrary units and
protein absolute number $p$. Substituting this definition on
\eref{seq_noise_protein} gives
\begin{equation}
  \text{noise} = {\sqrt{\ee{(\alpha I)^2} - \ee{\alpha I}^2} \over
                \ee{\alpha I}}.
\end{equation}

Since $\alpha$ is a constant it can be taken out of the average operator
$\ee{\cdot}$, obtaining
\begin{equation}
  \text{noise} = {\sqrt{\alpha^2 \left(\ee{I^2} -
                \ee{I}^2 \right)} \over
                \alpha \ee{I}}
       = {\sqrt{\left(\ee{I^2} - \ee{I}^2 \right)} \over
                \ee{I}}.
\end{equation}

Notice that in \eref{seq_calibration_factor} the linear proportionality between
intensity and protein count has no intercept. This ignores the autofluorescence
that cells without reporter would generate. To account for this, in practice we
compute
\begin{equation}
\text{noise} = {\sqrt{\left(\ee{(I - \ee{I_\text{auto}})^2} -
                    \ee{I - \ee{I_\text{auto}}}^2 \right)} \over
                \ee{I - \ee{I_\text{auto}}}}.
\end{equation}
where $I$ is the intensity of the strain of interest and $\ee{I_\text{auto}}$
is the mean autofluorescence intensity, obtained from a strain that does not
carry the fluorescent reporter gene.

\fref{sfig_noise_delta} shows the comparison between theoretical predictions
and experimental measurements for the unregulated promoter. The reason we split
the data by operator despite the fact that since these are unregulated
promoters, they should in principle have identical expression profiles is to
precisely make sure that this is the case. We have found in the past that
sequences downstream of the RNAP binding site can affect the expression level
of constitutively expressed genes. We can see that both models, the
single-promoter (gray dotted line) and the multi-promoter (black dashed line)
underestimate the experimental noise to different degrees. The single-promoter
model does a worse job at predicting the experimental data since it doesn't
account for the differences in gene dosage during the cell cycle. But still we
can see that accounting for this variability takes us to within a factor of two
of the experimentally determined noise for these unregulated strains.

\begin{figure}[h!]
	\centering \includegraphics
  {../fig/si/figS12.pdf}
	\caption{\textbf{Protein noise of the unregulated promoter.} Comparison of
	the experimental noise for different operators with the theoretical
	predictions for the single-promoter (gray dotted line) and the multi-promoter
	model (black dashed line). Each datum represents a single date measurement of
	the corresponding $\Delta lacI$ strain with $\geq 300$ cells. The points
	correspond to the median, and the error bars correspond to the 95\%
	confidence interval as determined by 10,000 bootstrap samples.}
  \label{sfig_noise_delta}
\end{figure}

To further test the model predictive power we compare the predictions for the
three-state regulated promoter. \fref{sfig_noise_reg} shows the theoretical
predictions for the single- and multi-promoter model for varying repressor
copy numbers and repressor-DNA binding affinities as a function of the inducer
concentration. We can see again that our zero-parameter fits systematically
underestimates the noise for all strains and all inducer concentrations. We
highlight that the $y$-axis is shown in a log-scale to emphasize more this
deviation; but, as we will show in the next section, our predictions still fall
within a factor of two from the experimental data.

\begin{figure}[h!]
	\centering \includegraphics
  {../fig/si/figS13_v2.pdf}
	\caption{\textbf{Protein noise of the regulated promoter.} Comparison of the
	experimental noise for different operators ((A) O1,  $\eR = -15.3 \; k_BT$,
	(B) O2, $\eR = -13.9 \; k_BT$, (C) O3, $\eR = -9.7 \; k_BT$) with the
	theoretical predictions for the single-promoter (dashed lines) and the
	multi-promoter model (solid lines). Points represent the experimental noise
	as computed from single-cell fluorescence measurements of different {\it E.
	coli} strains under 12 different inducer concentrations. Dotted line
	indicates plot in linear rather than logarithmic scale. Each datum represents
	a single date measurement of the corresponding strain and IPTG concentration
	with $\geq 300$ cells. The points correspond to the median, and the error
	bars correspond to the 95\% confidence interval as determined by 10,000
	bootstrap samples. White-filled dots are plot at a different scale for better
	visualization.}
  \label{sfig_noise_reg}
\end{figure}

\subsubsection{Systematic deviation of the noise in gene expression}

\fref{sfig_noise_delta} and \fref{sfig_noise_reg} highlight that our model
underestimates the cell-to-cell variability as measured by the noise. To
further explore this systematic deviation \fref{sfig_noise_pred_vs_data} shows
the theoretical vs. experimental noise both in linear and log scale. As we can
see the data is systematically above the identity line. The data is colored by
their corresponding experimental fold-change values. The data that has the
largest deviations from the identity line also corresponds to the data with the
largest error bars and the smallest fold-change. This is because measurements
with very small fold-changes correspond to intensities very close to the
autofluorescence background. Therefore minimal changes when computing the noise
are amplified given the ratio of std/mean. In \siref{supp_empirical} we will
explore empirical ways to improve the agreement between our minimal model and
the experimental data to guide future efforts to improve the minimal.

\begin{figure}[h!]
	\centering \includegraphics
  {../fig/si/figS14.pdf}
	\caption{\textbf{Systematic comparison of theoretical vs experimental noise
	in gene expression.} Theoretical vs. experimental noise both in linear
	(left) and log (right) scale. The dashed line shows the identity line of
	slope 1 and intercept zero. All data are colored by the corresponding value
	of the experimental fold-change in gene expression as indicated by the color
	bar. Each datum represents a single date measurement of the corresponding
	strain and IPTG concentration with $\geq 300$ cells. The points correspond
	to the median, and the error bars correspond to the 95\% confidence interval
	as determined by 10,000 bootstrap samples.}
  \label{sfig_noise_pred_vs_data}
\end{figure}