\section{GPUE: GPU Gross--Pitaevskii equation solver}\label{sec:GPUE}

Given the effectiveness of GPU computing in the simulation of the linear Sch\"odinger equation system for SAP, we next applied the newly-developed techniques to simulating Bose--Einstein condensates, which formed the bulk of work during my thesis. The body of software developed for this project has been released as the tool ``GPUE'', available at \url{https://github.com/mlxd/gpue}~\cite{MLXD_GPUE}. Performance testing of this code was carried out by Peter Wittek, ICFO, Barcelona~\cite{Wittek:2016}. A comparison was performed between GPUE, the Trotter--Suzuki (TS) package developed by Wittek \textit{et} al. \cite{NUM:Wittek_cpc_2013}, and the mature GPELab software suite for MATLAB \cite{NUM:GPElab_1,NUM:GPElab_2}. The sample results taken for time evolution are given in Fig.~\ref{fig:gpuevsts}. GPUE and GPU-enabled TS clearly beat MATLAB, and CPU performance by a significant margin. Although TS is a more generalised suite for computing, as far as we are currently aware the GPU computation does not yet allow for Gross--Pitaevskii solutions with angular momentum. GPUE is currently the optimal choice for rotating condensate systems out of the examined software suites.

\begin{figure}[htb]
    \centering
    \includegraphics[width=0.5\textwidth,]{ch3_numerics/GPUEvsTS.png}
    \caption{Performance benchmark of GPUE and other simulation packages for the evolution of a harmonically trapped atom in a superposition state between ground and first excited states. Lower numbers are better and give results in faster times. Data adapted from \cite{Wittek:2016}. }
    \label{fig:gpuevsts}
\end{figure}

Figures ~\ref{fig:profile_ev} and \ref{fig:profile_im} demonstrate some of the resulting calls to different segments of the code, with timings given in Tables \ref{tbl:gpue_ev} and \ref{tbl:gpue_im}. The important data of the figures is both the kernel percentage utilisation, and that the operations are mostly saturating the available number of GPU cores. The data shows the average time spent in each individual kernel during both real and imaginary time evolution for 1010 steps at $2^{10}\times 2^{10}$ resolution with (real) and without (imaginary) angular momentum operators. As can be seen, the inclusion of angular momentum operators lead to a performance hit, compared with an imaginary time evolution for a static condensate. While further optimisations can almost always be provided for such simulations, the performance of the software as a whole is defined by its slowest component. In this case the routines are equally met in performance by the Fourier transforms, which are already fully optimised as an external library. As such, improving performance much beyond this with the other kernels will be wasteful in time and resources.

%\iffalse
\begin{figure}
    \centering
    \includegraphics[width=0.98\textwidth,]{ch3_numerics/CUDA/Profiler_ev_1k_1024.png}
    \caption{Nvidia Nsight performance analysis of GPUE for real time evolution simulation for 1010 steps at $2^{10}\times 2^{10}$ resolution. The respective kernel calls and total utilisation are listed on the left.}
    \label{fig:profile_ev}
\end{figure}

\begin{table}
    \scriptsize
    \centering
\begin{tabular}{c|c|c|c|c}
\textbf{Kernel}  & \textbf{Info} & \textbf{Avg. runtime} & \textbf{\# Runs} & \textbf{Total time} \\
\hline
Mem. copy [H2D] & Memory copy from host to GPU & $2.312$ ms & 11 & $25.432$ ms\\
Mem. copy [D2H] & Memory copy from GPU to host & $2.18$ ms & 9 & $19.62$ ms\\
cMult & Complex mult. in time ev. & $0.342$ ms & 3030 & $1.0363$ s\\
cMultDensity & Complex mult. in time ev. for nonlinear op. & $0.456$ ms & 2020 & $0.921$ s \\
scalarDiv & Renorm. of $\Psi$ following FFT & $0.2216$ ms & 8080 & $1.791$ s\\
dpRadix0032B & Internal CUFFT operation & $0.237$ ms & 8080 & $1.915$ s\\
dpVector1024D & Internal CUFFT operation & $0.252$ ms & 8080 & $2.0362$ s\\
\end{tabular}
\caption{Kernel usage and timings for 1010 steps of real time evolution with angular momentum.}\label{tbl:gpue_ev}
\end{table}

\begin{figure}
    \centering
    \includegraphics[width=0.98\textwidth,]{ch3_numerics/CUDA/Profiler_im_1k_1024.png}
    \caption{Nvidia Nsight performance analysis of GPUE for imaginary time simulation for 1010 steps at $2^{10}\times 2^{10}$ resolution with angular momentum. The respective kernel calls and total utilisation are listed on the left.}
    \label{fig:profile_im}
\end{figure}

\begin{table}
    \scriptsize
    \centering
\begin{tabular}{c|c|c|c|c}
\textbf{Kernel}  & \textbf{Info} & \textbf{Avg. runtime} & \textbf{\# Runs} & \textbf{Total time} \\
\hline
Mem. copy [H2D] & Memory copy from host to GPU & $2.311$ ms & 3 & $6.933$ ms\\
Mem. copy [D2H] & Memory copy from GPU to host & $1.85$ ms & 5 & $9.25$ ms\\
cMult & Complex mult. in time ev. & $0.342$ ms & 1010 & $0.345$ s\\
cMultDensity & Complex mult. in time ev. for nonlinear op. & $0.346$ ms & 2020 & $0.698$ s \\
multipass & Optimised parallel summation & $0.125$ ms & 3030 & $0.378$ s \\
scalarDiv & Renorm. of $\Psi$ following FFT & $0.221$ ms & 2020 & $0.446$ s \\
scalarDiv_wfcNorm & Normalisation of wavefunction during ev. & $0.226$ ms & 1010 & $0.228$ s\\
dpRadix0032B & Internal CUFFT operation & $0.237$ ms & 4040 & $0.957$ s \\
dpVector1024D & Internal CUFFT operation & $0.251$ ms & 2020 & $0.507$ s \\
\end{tabular}
\caption{Kernel usage and timings for 1010 steps of imaginary time evolution for a non-rotating condensate.}\label{tbl:gpue_im}
\end{table}

A simplified sequence and state diagram combination is given in Figs.~\ref{fig:gpue_seq1} and~\ref{fig:gpue_seq2} which describes the operating process for GPUE. A document listing all aspects of component dependencies and intercommunication is available at~\cite{MLXD_GPUE}. For brevity, we will refer the reader to this location for more information.\footnote{The documentation is built using ~\href{http://www.stack.nl/~dimitri/doxygen/}{Doxygen} with the command ``doxygen ./docs/gpue_doxy.conf'', and requires the \textit{dot} package for figure generation. This may change during future releases.}.

\begin{figure}[]
    \centering
        \includegraphics[height=\textwidth,angle=270]{ch3_numerics/GPUE_Seq1}
    \caption{Simplified combined sequence and state diagram for GPUE operation (1 of 2). The operation procedure of GPUE is outlined in sequence from top-to-bottom. While much of the setup and analysis takes place on the host (CPU), the device (GPU) is used to offload all the time-evolution calculations. After setup, the wavefunction and all required operators are sent to the GPU.}
    \label{fig:gpue_seq1}
\end{figure}
\begin{figure}[]
    \centering
        \includegraphics[height=\textwidth,angle=270]{ch3_numerics/GPUE_Seq2}
    \caption{Simplified combined sequence and state diagram for GPUE operation (2 of 2). Following the completion of the time evolution after a predetermined number of steps, the wavefunction is unloaded from the GPU and returned to the CPU for output. Minimising this transfer allows for optimal performance from the device. Further details of dependencies and data flow are given by \cite[docs/gpue.pdf]{MLXD_GPUE}.}
    \label{fig:gpue_seq2}
\end{figure}

\subsection{Angular momentum operators using Fourier split-operator  method}\label{ss:ang_mom_fso}
As discussed earlier, in the presence of large values of angular momentum, the condensate wavefunction will accommodate many vortices. To ensure a well ordered lattice, more consideration is required than to just directly numerically solve the GPE at the required rotation rate. Assuming an initial Gaussian guess, and using the imaginary time evolution algorithm to find the ground state, a large number of vortices will enter the condensate from the edge and compete for lattice sites to form the expected Abrikosov pattern. Due to the highly dense spectrum of the condensate close to the ground state in this regime, only minimal energy shifts will be given for deviations from the perfect Abriksov geometry. As a result, it can take a significantly long time to reach an ordered state for rotation frequencies close to the transverse trapping frequency \cite{Vtx:Mueller_prl_2002}. To overcome this issue, one can choose to follow the ground state of the condensate with a ramp of the rotation rate. This essentially mimics adiabatic evolution, and allows for the determination of the vortex lattice ground state for all rotation frequencies.

The Fourier split-operator algorithm described earlier works well in handling cases where the individual operators live in position or momentum space respectively. However, the angular momentum operators are a combination of both spaces. Taking the angular momentum operator along the $z$-axis, $L_z = xp_y - yp_x$, and applying it to the wavefunction requires each basis element to be in a different space in the different directions. For applying this operator we must therefore Fourier transform along a single dimension, multiply by the respective $\mathbf{k}$-space component, take the inverse, multiply by the respective $\mathbf{r}$-space component in the other direction, and then perform this operation along the other dimensions, summing the results.

This accrues an error which is not encountered using methods that are solely in position or momentum space. Following the process given in Sec.~\ref{sec:fso} the error can be determined by checking the commutativity of the respective components of the angular momentum operator as

 \begin{align}
 	\alpha_1 = [x p_y,-y p_x] &= [x p_y,-y] p_x  -  y[x p_y,p_x] = -[-y,x p_y] p_x + y [p_x, x p_y] \nonumber \\
 		&= -\left( {\cancelto{0}{[-y,x]}} p_y + x [-y,p_y] \right) p_x + y \left( [p_x,x] p_y + x {\cancelto{0}{[p_x,p_y]}} \right) \nonumber \\
 		&= -x {\cancelto{-\textrm{i}\hbar}{[-y, p_y]}} p_x + y {\cancelto{-\textrm{i}\hbar}{[p_x,x]}} p_y \nonumber \\
        &= \textrm{i}\hbar \left(x p_x - y p_y \right).
 \end{align}

 The complex error term can be seen as, in the case of the above implemented evolution, allowing the angular momentum operator to change from imaginary time to real-time, and vice-versa in each respective case. To overcome this, we simply swap the application order of the operator components, between odd and even steps during the evolution. Starting with the alternate order we obtain a value of $\alpha_2 = [-y p_x, x p_y] = \textrm{i}\hbar \left(-x p_x + y p_y \right)$. Since we are applying these operators to the condensate we can overcome the error of one term by the application of the other, as
 \begin{equation}
 \exp{\textrm{i} \alpha_1}\exp{\textrm{i} \alpha_2} = 1.
 \end{equation}

 Although alternating will provide a cancellation of this error, it can be assumed that for large timesteps the error will have a significant contribution to the overall dynamics, as the wavefunction evolves during each timestep. For greater accuracy of this method one can perform a decomposition following Eq.~\eqref{eqn:3} for a third-order error, or using the above splitting for second-order.

An example of the density of the ground state and the associated wavefunction phase at a rotation frequency of $\Omega = 0.995\omega_x$ is given in Fig.~\ref{fig:showingoff} at a resolution of $2^{11}\times 2^{11}$ ($2048\times 2048$). Although aliasing may be apparent in the phase, this is due to the limited resolution of the computer monitor (printer). The presence of a well ordered Abrikosov lattice is clearly visible.

 \begin{figure}
     \centering
     \includegraphics[width=0.45\textwidth,]{ch3_numerics/Rho_995}
     \includegraphics[width=0.45\textwidth,]{ch3_numerics/phi_995}
     \caption{Condensate density (left) and phase (right) at a rotation rate of $\Omega=0.995\omega_x$ for a $2^{11}\times 2^{11}$ grid showing approximately 600 vortices in the visible density regions. For both images the box size is $700~\mu\textrm{m} ~\times ~700~\mu\textrm{m}$.}
     \label{fig:showingoff}
 \end{figure}

 \subsection{Vortex tracking}\label{sec:vortrack}
 To efficiently follow the dynamics of individual vortices, a robust algorithm is needed to track their positions. One could track regions where the density drops to zero. However, this gives very little information on the topological excitation, and may miss many vortices, as the numerical wavefunction may never truly approach this value. A more effective way is to locate the $\pm 2\pi$ charge in the wavefunction phase, which is a signature of quantum vortices. For this we examine each $2\times 2$ subgrid of the underlying lattice and check if the phase rotates from $-\pi$ to $+\pi$ (or vice versa). After an initial pass to identify vortex locations closest the nearest grid element, a least-squares fit is performed to more accurately determine the vortex core position \cite{c42f}. Linear least squares is used generally for an overdetermined linear system $\mathbf{A}\mathbf{r} = \mathbf{b}$, where unique solutions are unlikely to exist. Thus, for a solution, we seek the best fit plane that minimises the error, of the form

 \begin{equation}
 S(\mathbf{r}) = \displaystyle\sum |b_i - \displaystyle\sum A_{ij} r_j |^2
 \end{equation}
 where $S$ is the objective function to be minimised, following $\mathbf{b} = \argmin S(\mathbf{r})$. The solution of this minimisation problem is given by
 \begin{subequations}
\begin{align}
    \mathbf{A} ^{T}\mathbf{A} \mathbf{r} &= \mathbf{A} ^{T}\mathbf{b}, \\
    \mathbf{r} &= (\mathbf{A}^{T}\mathbf{A})^{-1}\mathbf{A}^{T}\mathbf{b}.
\end{align}
\end{subequations}
The best-fit plane is sought of the form
$a_0 c + \displaystyle\sum\limits_{i}^{m} a_i r_i = f(\mathbf{r})$
which for a two-dimensional system, $\mathbf{r} = (x,y)$, is given by the matrix,
\begin{equation}
    \mathbf{A} = \left(
    \begin{array}{ccc}
        0 & 0 & 1 \\
        0 & 1 & 1 \\
        1 & 0 & 1 \\
        1 & 1 & 1
    \end{array}\right).
\end{equation}
The above matrix is composed of all possible planes that can fit over a square $2\times 2$ grid plaquette,
and
\begin{equation}
    \mathbf{b} = \left(
    \begin{array}{cccc}
        \Psi(x_0,y_0) & \Psi(x_0,y_1) & \Psi(x_1,y_0) & \Psi(x_1,y_1)
    \end{array} \right)^{T},
\end{equation}
are the wavefunction values around the sampled $2\times 2$ grid.
Upon evaluating the vector $\mathbf{r}$ above, one can obtain the best fit plane  solution as
\begin{equation}\left(
    \begin{array}{c}
        x \\
        y \\
        c
    \end{array}\right)
    = \left(
    \begin{array}{c}
        {\left( -\Psi(x_0,y_0) + \Psi(x_0,y_1) - \Psi(x_1,y_0) + \Psi(x_1,y_1) \right)}{/2} \\
        {\left( -\Psi(x_0,y_0) - \Psi(x_0,y_1) + \Psi(x_1,y_0) + \Psi(x_1,y_1) \right)}{/2} \\
        3\Psi(x_0,y_0) + \Psi(x_0,y_1) - \Psi(x_1,y_0) - \Psi(x_1,y_1)
    \end{array}\right).
\end{equation}

The goal is to find where both the real and imaginary components cross through zero, and thus we seek a solution of the form $x + y = -c$. Rearranging the above equations as
\begin{equation}\left(
    \begin{array}{cc}
        \Re(x) & \Re(y) \\
        \Im(x) & \Im(y) \\
    \end{array}\right)
    \left(
    \begin{array}{c}
        \delta x \\
        \delta y
    \end{array}\right)
    = -
    \left(
    \begin{array}{c}
        \Re(c) \\
        \Im(c)
    \end{array}\right),
\end{equation}
and again solving the linear system by inverting the left-hand matrix and multiplying across allows one to seek the corrections to the vortex position, $\delta \mathbf{r} = (\delta x, \delta y )$.

 With this, we can accurately determine the position of the vortices with high precision. To track their motion during the evolution, we create an initial list of positions and give each vortex a unique identifier (UID). Assuming the vortex cores can travel a limited distance (some multiple of the grid resolution) between time steps, we can say at subsequent times which vortex has moved to the newly found positions.

 This process is performed by representing the vortices as a graph, each with an assigned unique identifier, associated location, phase winding and on/off flag. Edges are created between vortices that are separated by at most root-two the average of the inter-vortex spacings. A finite boundary is chosen to examine only vortices in areas of significant condensate density, since vortices can easily appear and disappear close to the condensate boundary. Any vortex which appears without association to an initial vortex, or any tracked vortex that crosses the boundary, is switched off and remains so for all analysis. A graph for an initial ($t=0$) vortex lattice shown in Fig.~\ref{fig:graphit}, with the identifiers indicated on each node, and the neighbouring distances indicated by the edge weights. %The almost perfect triangular structure is clearly confirmed. %%This is a graph, not a location plot. Positions are chosen at mathematicas discretion

 \begin{figure}
     \centering
     \includegraphics[width=0.98\textwidth,]{ch3_numerics/Graph_full}
     \caption{Graph of vortex lattice positions indicating the vortex identifier and the intervortex distances in units of grid-spacing $\Delta= \Delta x = \Delta y$. A hard-walled boundary is chosen such that the vortex distances remain almost uniform, and are shown here with a mean value of $\bar{r} = 30.2451$ and variance of $\sigma^2 = 0.16118$.}
     \label{fig:graphit}
 \end{figure}


% !TEX root = ../00_thesis.tex

\section{Concrete Realization of \DRP}
\label{sec:concrete_realization}

\squarepar{%}
  This section discusses how to concretely implement \DRP's concepts. In particular, one needs to define
  (i)~the fixed flushing interval $T_f^s$ of the \CPs~(\cref{subsec:CP_schedule}), and
  (ii)~how to dynamically compute the network deadline \ndeadlinei of a flow \flowi and the flushing interval $T_f^d$ of each \ap (\cref{subsec:D_Tfd_computation}).%
}

Then, a worst-case buffer analysis (\cref{subsec:WC_buffer}) will allow to formulate admission tests (\cref{subsec:admission}), one for {\ap}s and one for {\cp}s.
The success of all admission tests guarantees that both contracts \textbf{Source}~$\boldsymbol{\leftrightarrow}$~\textbf{Blink} and \textbf{Blink}~$\boldsymbol{\leftrightarrow}$~\textbf{Destination} can be satisfied by \DRP.

\subsection{Setting \CPs' Flushing Interval}\label{subsec:CP_schedule}

\squarepar{%}
  To guarantee that all \CPs fulfill their share of the contracts (\ie prevent buffer overflows), we conceive a time-triggered approach to schedule all tasks of \CPs. It consists of (\emph{i}) setting the flushing interval $T_f^s$ of all \CPs to the same constant value, and (\emph{ii}) letting the round interval $T_{net}$ be a multiple of $T_f^s$.
  As discussed in \cref{sec:designDetailed}, $T_f^s$ should not to constrain the achievable deadline: thus, we aim to set it as short as possible.
  \CPs have three tasks to perform%
}

\begin{itemize}
 \item flushing \bolt before each communication round,
 \item participating in the communication during the rounds,
 \item writing all received messages into \bolt after the rounds.
\end{itemize}

Performing those tasks altogether takes $C_{CP} + C_{net}$ time units, where $C_{CP} = C_f + \nslotsmax * C_w$, and \nslotsmax denotes the number of time slots in one round.
Hence, $C_{CP} + C_{net}$ is the smallest admissible round interval (otherwise \CPs' task set is not schedulable). Thus we set for all \CPs in the system,
\begin{align}
\label{eq:design_flush_period_source}
	& T_f^s = C_{CP} + C_{net} \\
\intertext{and we let the round interval be a multiple of $T_f^s$. That is, for $k \in \mathbb{N}$, $k > 0$,}
\label{eq:design_round_period}
	& T_{net} = k * T_f^s
\end{align}
For a given $C_{net}$, a larger $k$ entails less available bandwidth but also lower energy consumption.
\blink is designed to dynamically adjust $k$ to match the bandwidth requirements and save energy~\cite{zimmerling2017Blink}.

\subsection{\mbox{Computing Network Deadlines \& \APs ' Flushing Interval}}
\label{subsec:D_Tfd_computation}

Having fixed \CPs' flushing interval, we now turn to the problem of dynamically computing the network deadline \ndeadlinei of flow \flowi and the flushing interval $T_f^d$ of \flowi's destination \apdst, such that the end-to-end deadline \deadlinei is met.
To this end, we need to define expressions for the functions $f$ and $g$ (introduced in \cref{sec:designDetailed}), and derive values for \ndeadlinei and $T_f^d$ such that equations \eqref{eq:function_f} and \eqref{eq:function_g} are satisfied.

\begin{theorem}
  \label{thm:delta}
  For any flow \flowi = (\flowsrci,\flowdsti,\periodi,\jitteri,\deadlinei), and given the duration of communication rounds $C_{net}$, functions $f$ and $g$ are upper-bounded as follows
  \begin{align}
  f(T_f^s ,  \ndeadlinei)
  	& \; \leq \;
  	T_i + D_i + \overline{\jitteri}  + \delta_f^{const} \label{eq:expression_f}\\
  g(T_f^d)
  	& \; \leq  \;
  	T_f^d(\flowdsti) + \delta_g^{const} \label{eq:expression_g}
  \end{align}
  where $ \delta_f^{const}$ and $\delta_f^{const}$ are constant delays that depend on the WCETs of the \bolt API functions, on the maximum number of messages \nslotsmax that can be %inside a \bolt queue at the end of a round
  served by \blink in one round, and on the fixed flushing interval $T_f^s$ of \CPs,
  \begin{align}
  \delta_f^{const} &\;= \;\; C_w + C_f + T_f^s \\
  \delta_g^{const} &\;= \;\;\nslotsmax*C_w - (\nslotsmax-1)* C_r + C_f\\
  \label{eq:jitter_bar}
  \overline{\jitteri}  &\;= \;\; \floor*{(\jitteri + C_f - C_r)/{T_f^s}}\cdot T_f^s
  \end{align}
\end{theorem}

\begin{proof}%
Function $f$ is the time between when a message is written into \bolt by the source \apsrc and when the communication round in which the message is sent by \blink ends (\ie when the message is available at the destination \cpdst).
This is the sum of two delays: $\delta_{source}$, the time until the message is available for communication at the source \cpsrc; and $\delta_{network}$, the time until the message is shipped over the network to \cpdst.

Similarly, function $g$ is the time between when a packet is available at the destination \cpdst and the end of the \opflush operation that reads the message out of \bolt at the destination \apdst (\ie when the message can be processed by the destination application).
We refer to this delay as $\delta_{dest}$.

Hence, the expressions for functions $f$ and $g$ in \eqref{eq:expression_f} and \eqref{eq:expression_g} directly follow from the delays expression derived in Lemmas~\ref{lem:delta_source}, \ref{lem:delta_network}, and \ref{lem:delta_destination}~(\cref{append:drp_WCanalysis}).
\
\end{proof}

We use Theorem~\ref{thm:delta} to express conditions on \ndeadlinei and $T_f^d$ such that \eqref{eq:function_f} and \eqref{eq:function_g} are satisfied.
In particular, it is sufficient that for any flow $\flowi = (\flowsrci,\flowdsti,\periodi,\jitteri,\deadlinei) \in \flowset$
\begin{align}
\nonumber
 T_i + D_i + \overline{\jitteri} \;
	& \leq  \;\; r * \deadlinei - \delta_f^{const} \\
\label{eq:ndeadline_constraint_latency}
\Rightarrow \qquad T_i \;
	& \leq  \;\; r * \deadlinei - \delta_f^{const} - D_i - \overline{\jitteri} \\
\intertext{and}
\label{eq:APflush_constraint_base}
T_f^d(\flowdsti) \;
	& \leq \, (1-r)*\deadlinei -  \delta_g^{const}
\end{align}

As low-power wireless networks typically feature limited bandwidth,%
\footnote{Low-power radio bit rates are often limited to 256\kbps; the latest version of Bluetooth (Bluetooth~5) supports up to 2\Mbps.}
%
it makes sense to choose the network deadline \ndeadlinei as large as possible in order to increase the schedulability of flows in the network.
However, \blink only supports constrained deadlines ($\ndeadlineany \leq \periodany$) and deadlines must be multiples of the round length ($\ndeadlineany = 0 \mod \rperiod $).
Furthermore, a network deadline cannot be smaller than \rperiodmin (see \cref{subsec:details_blink}).
%
Hence, for any flow \flowi, it must hold that
\begin{align}
  \label{eq:ndeadline_constraint_period}
  \rperiodmin \; &\leq \; \ndeadlinei \; \leq \; \periodi \\
  \ndeadlinei \; &= \; 0 \mod \rperiod
\end{align}

Finally, to satisfy all contracts in the system, \eqref{eq:ndeadline_constraint_latency}, \eqref{eq:APflush_constraint_base} and \eqref{eq:ndeadline_constraint_period} must hold for all flows $\flowi \in \flowset$.
Hence, the values for \ndeadlinei and $T_f^d$ computed dynamically at runtime must satisfy for any flow $\flowi = (\flowsrci,\flowdsti,\periodi,\jitteri,\deadlinei) \in \flowset$ and any $n \in \mathcal{N}$
\begin{align}
\label{eq:design_ndeadline}
\ndeadlinei \;
	& = \; \min \left(\; \periodi \;\, , \,  r*\deadlinei - \delta_f^{const} - \periodi - \overline{\jitteri} \right) - (\ndeadlinei \mod \rperiod) \\
\label{eq:design_delay_destination}
T_f^d(n) \;
	& \leq \; \min_{F_j\in \mathcal{F}, n = \flowdstj}
	\left( (1-r)*\deadlinej - \delta_g^{const} \right)
\end{align}
If using \eqref{eq:design_ndeadline} leads to a violation of the constraint in \eqref{eq:ndeadline_constraint_period} or if $T_f^d$ results in a load that \ap at node $n$ cannot handle, \DRP rejects the flow since the two contracts cannot be guaranteed.

\subsection{Worst-case Buffer Analysis}
\label{subsec:WC_buffer}

Satisfying all contracts entails preventing overflows of message buffers in the system.
Specifically, as shown in \cref{fig:design_overview},
\begin{itemize}
	\item \APs are responsible for ensuring that the incoming \bolt queues do not overflow, and
	\item \CPs are responsible for ensuring that their local message buffers and the outgoing \bolt queues do not overflow.
\end{itemize}

\squarepar{%}
  To formulate the admission tests for {\ap}s and {\cp}s, we first need the worst-case buffer sizes (\ie maximum number of messages in a buffer) induced by a given flow set \flowset.
  For ease of exposition, we make the following hypothesis.%
}
\begin{hypothesis}\label{hyp:f_max}
For a given flow set \flowset, an \ap (resp. \cp) never writes more messages into \bolt than can be flushed by \cp (resp. \ap) in one \emph{\opflush} operation in the time span between two \emph{\opflush}.
\end{hypothesis}
This hypothesis implies that the \bolt queues are always empty at the end of a \opflush operation.
We prove at the end of this section that our admission tests effectively guarantee that Hypothesis~\ref{hyp:f_max} is always verified.

\begin{lemma} \label{lem:buffer_bolt_out}
Given a flow set \flowset, the buffer size of the outgoing \bolt queue of node $n \in \nodeset$, $B_{\bolt,out}(n)$, is upper-bounded,
\begin{equation}
\label{eq:bolt_out_buffer_stress}
	B_{\bolt, out}(n) \; \leq \;
	\sum_{\substack{F_i\in \mathcal{F},\, n = \flowsrci}} \ceil*{\frac{T_f^s + C_w + C_r + \jitteri}{T_i}}
\end{equation}
\end{lemma}

\begin{proof}%
According to the \textbf{Source} $\boldsymbol{\leftrightarrow}$ \textbf{Blink} contract, \apsrc at node $n$ does not write more than one message every \periodi with jitter \jitteri into the outgoing \bolt queue.
Based on Hypothesis~\ref{hyp:f_max}, the buffer size is bounded by the number of messages that can be written by \apsrc during the maximum time a message can stay inside the queue, which is $\Delta = T_f^s + C_w + C_r$ (see \cref{fig:delta_source_time_graph}).
The maximum number of messages that can be written by \apsrc within any time interval $\Delta$ is $\ceil*{(\Delta + J_i)/{T_i}}$ for each flow \flowi sourced by~$n$. \
\end{proof}

The worst-case buffer size of a \cp depends on (i) the maximum time a message can stay in \cp's local memory awaiting to be served by \blink, and (ii) the number of messages that can be sent within one round to a node.

\begin{lemma} \label{lem:buffer_CP}
Given a flow set \flowset, the buffer size of {\cp}'s internal memory of node $n \in \nodeset$, $B_{CP}(n)$, is upper-bounded,
\begin{equation}
\label{eq:CP_memory_stress}
	B_{CP}(n) \leq
	\sum_{\substack{F_i\in \mathcal{F},\\ n = \flowsrci}}
	1 + \ceil*{\frac{\ndeadlinei + \overline{\jitteri} + C_f}{T_i}} +
	\sum_{\substack{F_i\in \mathcal{F},\\ n = \flowdsti}} 1
\end{equation}
\end{lemma}

\begin{proof}%
On the source side, we make the conservative assumption that all messages read out during a \emph{\opflush} occupy memory in \cpsrc from the beginning of the \emph{\opflush}. Hence, the maximum waiting time in \cpsrc for a message until it is served by \blink is $\delta_{network} + C_f$ (see Lemma~\ref{lem:delta_network} -- \cref{append:drp_WCanalysis}). The number of messages in \cpsrc due to the source is upper-bounded by the maximum number of messages \apsrc can write during this time interval, given by
$\ceil*{\delta_{network} + C_f)/T_i}$.
Using Lemma~\ref{lem:delta_network}, this is at most
$1 + \ceil*{( \ndeadlinei + \overline{\jitteri} + C_f)/\periodi }$  per outgoing flow.

One the destination side, during a round, \cpdst may receive several messages, which it immediately writes into \bolt after the round.
However, \blink expects one packet every \periodi from each flow, which it serves within \ndeadlinei. As $\ndeadlinei \leq \periodi$, \blink never schedules more than one packet per round for each flow.
Thus, the maximum number of messages in \cpdst due to the destination is 1 packet per incoming flow.
\
\end{proof}

\begin{lemma} \label{lem:buffer_bolt_in}
Given a flow set \flowset, the buffer size of the incoming \bolt queue of node $n \in \nodeset$, $B_{\bolt,in}(n)$, is upper-bounded,
\begin{equation}
\label{eq:bolt_in_buffer_stress}
	B_{\bolt, in}(n)\;  \leq \;
	\sum_{\substack{F_i\in \mathcal{F},\\ n = \flowdsti}} \ceil*{\frac{T_f^d(n) + C_w + C_r + D_i}{T_i}}
\end{equation}
\end{lemma}

\begin{proof}%
As specified in the \textbf{Source} $\boldsymbol{\leftrightarrow}$ \textbf{Blink} contract, \blink delivers packets from any flow \flowi before the network deadline \ndeadlinei~(\cref{sec:designDetailed}). Therefore, \blink delivers at most one packet every \periodi time units, with a jitter equal to \ndeadlinei, which are written into \bolt immediately after the round.

Based on Hypothesis~\ref{hyp:f_max},
the buffer constraint of the incoming \bolt queue is bounded by the number of packets that can be written by \cpdst during the maximum elapsed time before a packet is read out by \apdst. As in the proof of Lemma~\ref{lem:buffer_bolt_out}, there are at most $\ceil*{(T_f^d(n) + C_w + C_r + D_i)/{T_i}}$ such messages from each flow \flowi that has node $n$ as destination.
\end{proof}

\subsection{Admission Tests}
\label{subsec:admission}

We now combine the above results and formulate the admission tests for \CPs and \APs, which form the cornerstone of \DRP's registration mechanism described in \cref{sec:designDetailed}.
We further show that the computation complexity of the admission tests is not only small but \emph{constant}, and hence supports the requirements of \feature{Adaptability} and \feature{Efficiency}.

\squarepar{%}
  Let $F_j$ be the flow for which a request has been issued, and $\mathcal{F}_{new} = \mathcal{F} \cup \{F_j\}$.
  The \cp of node $n$ is responsible for preventing overflows of its local memory (of size $S_{CP}$) and of the outgoing \bolt queue of node $n$ (of size $S_{\bolt}$).%
}

\begin{theorem}[{Admission Test of \cp}]\label{thm:CP}
If
\begin{align*}
S_{\bolt} \;
	 & \geq \sum_{\substack{\flowi \in \mathcal{F}_{new}, \\ n = \flowsrci}}
	 	\ceil*{\frac{T_f^s + C_w + C_r + \jitteri}{T_i}} \\
\textup{and}\qquad
S_{CP} \;
 	& \geq
	\sum_{\substack{\flowi \in \mathcal{F}_{new}, \\ n = \flowsrci}}
		1 + \ceil*{\frac{\ndeadlinei + \overline{\jitteri} + C_f}{T_i}}
	+ \sum_{\substack{\flowi \in \mathcal{F}_{new}, \\ n = \flowdsti}}
		1
\end{align*}
then the requested flow \flowj can be safely admitted by $\cp$.
\end{theorem}

\begin{proof}Immediate from Lemmas~\ref{lem:buffer_bolt_out} and \ref{lem:buffer_CP}. \
\end{proof}

The \ap of node $n$ is responsible for preventing overflows of the incoming \bolt queue (of size $S_{\bolt}$) and for guaranteeing its share of the end-to-end deadline.

\begin{theorem}[{Admission Test of \ap}]
\label{thm:AP}
  If there exists $T_f^d(n)$ such that
  \begin{align*}
    T_f^d(n) \;
    & \leq \, \min_{\substack{F_i\in \mathcal{F}_{new}, \\  n = \flowdsti}}
    \left( (1-r)*\deadlinei - \delta_g^{const} \right)\\
\textup{and}\qquad
    S_{\bolt} \;
    & \geq \sum_{\substack{\flowi \in \mathcal{F}_{new}, \\ n = \flowdsti}} \ceil*{\frac{T_f^d(n) + C_w + C_r + D_i}{T_i}}
  \end{align*}
  then the requested flow \flowj can be safely admitted by $\ap$.
\end{theorem}
\begin{proof}
Immediate from Lemma~\ref{lem:buffer_bolt_in} and equation \eqref{eq:design_delay_destination}.
\end{proof}

Finally, we verify that Hypothesis~\ref{hyp:f_max} holds, showing the validity of our buffer analysis. From \eqref{eq:f_max=M} we have $f_{max} = S_{\bolt}$. Thus, by performing the admission tests at runtime, it follows from Theorems~\ref{thm:CP} and \ref{thm:AP} and Lemmas~\ref{lem:buffer_bolt_out} and \ref{lem:buffer_bolt_in} that $f_{max}$ is always bigger than the filling level of any \bolt queue, which entails Hypothesis~\ref{hyp:f_max} is true.


\title{\bf Spectra}

\section{Basics \& Nomenclature}

Spectra of objects can be determined in a number of ways, through
refraction, diffraction, or through energy-sensitive devices (most
commonly in the X-rays). Here we concentrate on the issues most
relevant to diffraction grating spectra, though many of these issues
are also relevant in other contexts.

Typically, spectra are obtained by putting a dispersive element in the
collimated beam. Although prisms can and have been used at the
objective pupil, more typically the dispersing element is at a pupil
beyond the focus. In addition to providing a smaller area for the
dispersing element to cover, this allows the focal plane to be used to
remove light not associated with the object or object of
interest. Slit spectrographs put a slit (or many slits) in this focal
plane, recollimate the diverging beam, disperse it, and then refocus
it on a new focal plane. Fiber spectrographs use optical fibers in the
focal plane and then align the fiber outputs in a convenient manner
with respect to the detectors.

Diffraction gratings consist of optical elements ruled in some manner
with a set of closely spaced apertures. At any given wavelength, light
coming through the grating will be diffracted. The majority of energy
passes straight through in the 0th order mode, but diffraction causes
additional modes at angles perpendicular to the ruling. The location
of the 1st and greater order modes depends on wavelength, meaning the
higher-order modes yield spectra, with larger dispersion but a smaller
fraction of the incoming energy at higher order modes. Traditional
diffraction gratings were usually ruled metal or mirrors, often mated
to a prism (a configuration known as a {\it grism}) to reorient the
desired mode along a convenient path. Modern gratings are usually
volume-phase holographic gratings, created within a substrate through
holographic techniques. With greater control over the grating
geometry, these gratings typically result in optics with higher
throughput by avoiding occlusion along the geometric optics light
path.

Spectra can be described as having a characteristic resolution. The
raw spectrum is usually detected on in a two-dimensional image, and
the spectrum appears along a somewhat curved line, known as the {\it
trace}, traversing the image along the dispersion direction. The
resolution parallel to the dispersion direction is known as the {\it
line spread function}, whereas the resolution perpendicular to the
dispersion direction is known as the {\it point spread function}, in
analogy to an image.

For slit spectra, the point spread function is determined largely by
the seeing and telescope PSF; the actual profile of the spectrum
perpendicular to the trace is the image of the object convolved with
the PSF. For fiber spectra, the light is usually sufficiently
scrambled that the width of the PSF is determined by the size of the
fiber, as reimaged by the camera.

The line spread function is charactered by the resolution $R= \lambda
/ \Delta \lambda$, where $\Delta\lambda$ is usually the FWHM of the
LSF (but not always). It is determined by the size of the fiber and
the dispersion power of the grating. For exposures more than a few
minutes, any flexure of the spectrograph over the course of the
exposure may also cause blurring in the LSF (or PSF).

The raw spectra need to be extracted and calibrated. The {\it
extraction} involves inferring a signal as a function of position on
the trace, to create a one-dimensional spectrum. The position in
pixels needs to be converted to wavelength; the relationship between
the two is usually inferred by injecting a signal into the
spectrograph, often sourced by an {\it arc lamp} which emits light at
discrete lines associated with ionized inert gases (He, Ne, Ar,
etc).

The signal needs to be converted to flux units. If we assume the
detector is bias-subtracted as if it were an image, the signal in the
trace can be modeled as:
\begin{equation}
{\rm DN}(x) = \left[f(\lambda(x)) + f_{\rm
sky}(\lambda(x))\right] \times F(x) \times T(\lambda)
\end{equation}
The quantity $F(x)$, the {\it flat}, characterizes the effects of the
detector response and is typically determined by injecting light from
a flat-field lamp, which has a broad spectrum; in fact, only
small-scale features are usually retained from the flat-field for
reasons that will soon be clear.

The quantity $T(\lambda)$ characterizes the throughput as a function
of wavelength, and is affected by the atmospheric transmission, how
much of the light enters the spectroscopic aperture at any given
wavelength, and the throughput of the optics, as well as any remaining
dependence of efficiency on wavelength on scales larger than those
probed by the flat. It must be determined in part by observing
standard stars whose true spectra are assumed, either through
additional slits or fibers at the same time as the objects of
interest, or through the same slits or fibers at slightly different
times. Either approach requires accounting for the resulting slight
differences in the observations.

The sky signal $f_{\rm sky}(\lambda)$ also needs to be removed. In the
case of slit observations, the sky can be estimated from the outer
parts of the slit, which contain little object flux. In the case of
fiber observations, the sky needs to be estimated from other nearby
fibers. Because the fiber varies with position and time on the sky,
this can be uncertain.

\section{Commentary}

In ground-based spectroscopy it is extremely important to pay
attention to the chromatic atmospheric refraction. This effect can
lead to different amounts of light getting through fiber or slit as a
function of wavelength, because the image in the telescope focal plane
is a function of wavelength. These differences lead to both a
variation in throughput and potential loss of signal-to-noise, and
also may make the spectrum hard to calibrate (if the standards used
are not under the same conditions). Slit spectra are often arranged so
that the slit is parallel to the parallactic direction to minimize
this effect, but there is no such mitigation for fibers. Atmospheric
dispersion correctors consisting of crossed prisms can be used to
reduce this effect, but they need to act in a pupil so are large and
expensive.

In practice, the difficulty in sky subtraction is almost entirely due
to the LSF modeling. Typically the issue arises because there are
strong lines that need to be subtracted, and so the LSF model needs to
be very good to be accurate enough. In addition, the sky can have a
different LSF than the object, since there are slit-filling
issues or due to imperfections in the extraction method.

\section{Key References}

\begin{itemize}
  \item
    {\it Design and Construction of Large Telescopes},
      \citet{bely03a}
  \item
    {\it Astrophysical Techniques}, \citet{kitchin09a}
\end{itemize}

\section{Order-of-magnitude Exercises}

\begin{enumerate}
\item Assuming the centroid accuracy is related to $R$ and $S/N$ ratio
    in a similar way for optical spectra of single emission lines as
    for point sources in an image, for an unresolved line with
    $S/N \sim 10$ in its total flux, how high $R$ do you have to be to
    determine a Doppler velocity to 10 km s$^{-1}$ precision?  How
    would a redshift determination using a full optical spectrum
    (i.e. using many lines) change the precision of the determination
    qualitatively? How does dependence of the velocity precision on
    $R$ differ qualitatively for absorption lines?

\begin{answer}[Author: Matthew Daunt]
    Due to the Doppler shift, we can predict the radial velocity of an
    object using the observed wavelength and observed wavelength,
    via
\begin{equation}
    v_r = \frac{\lambda_{obs} -\lambda_0}{\lambda_0} c
\end{equation}
    Resolution is defined as
    \begin{equation}
        R = \frac{\lambda}{\Delta \lambda}
    \end{equation}
    where $\Delta \lambda$ is the full width half max (FWHM). We
    can express the resolution in velocity units therefore as
    \begin{equation}
    \Delta v = \frac{\lambda}{R } \frac{c}{\lambda} = \frac{c}{R}
\end{equation}
    The precision in the radial velocity should be better when the FWHM is
    smaller and the $S/N$ is higher. The scaling is the same as for
    positions in an image, such that
    \begin{equation}
    \sigma \sim {\rm FWHM~}/ (S/N) = \frac{c}{R} \frac{1}{S/N}
    \end{equation}
    For $S/N = 10$, this equation is therefore:
    \begin{equation}
    \sigma \sim \frac{30,000 {\rm ~km} {\rm ~s}^{-1}}{R}
    \end{equation}
    which therefore requires $R\sim 3,000$.  If you have more than one
    emission line, you can use the $S/N$ in each of the, and the
    combined precision will be better than just using one. The case of
    absorption lines will differ because the noise in the continuum
    will contribute, and a larger width of the line will degrade the
    precision further than for an emission line.
\end{answer}
\end{enumerate}

% \section{Analytic Exercises}

% \begin{enumerate}
% \item Assuming that the
% \end{enumerate}

\section{Numerics and Data Exercises}

\begin{enumerate}
\item Find an SDSS spectrum of a galaxy and sky fiber from the same
    plate of the BOSS survey. Plot the sky and the object spectrum.
\item Find an arc spectrum from the BOSS survey, and a corresponding sky
spectrum taken during the same calibration-exposure sequence. You will
need to use
the \href{https://data.sdss.org/datamodel/files/SPECTRO_REDUX/RUN2D/PLATE4/spPlan2d.html}{\tt
spPlan2d} files to figure out which
\href{https://data.sdss.org/datamodel/files/BOSS_SPECTRO_REDUX/RUN2D/PLATE4/spCFrame.html}{\tt
spCframe} files to use. Find at least one pair of sky and arc lines,
each of which is bright and isolated from other lines. Estimate the
FWHM of the LSF from each and compare them.
\end{enumerate}

\bibliographystyle{apj}
\bibliography{exex}


\part{part}
part text
part text
\chapter{chapter long title}
 chapter text
 chapter text
 \[
	 f(x)=x^2
 \]
 \section[for the toc]{section}
  section text
  section text
 \section[for the toc]{section}
  section text
  section text
  \subsection[for the toc]{subsection}
  subsection text
  subsection text
  \subsection[for the toc]{subsection}
  subsection text
  subsection text
 \section[for the toc]{section}
  section text
  section text
\chapter{chapter}
 chapter text
 chapter text
 \part{part}
 part text
 part text
\chapter[toc]{chapter title}
 chapter text
 chapter text
 \section[for the toc]{section}
  section text
  section text
  \subsubsection[for the toc]{subsubsection}
  subsubsection text
  subsubsection text
  \paragraph{paragraph}
  paragraph text
  paragraph text
  \subparagraph{subparagraph}
  subparagraph text
  subparagraph text
 \section[for the toc]{section}
  section text
  section text
  \subsubsection[for the toc]{subsubsection}
  subsubsection text
  subsubsection text
  \paragraph{paragraph}
  paragraph text
  paragraph text
  \subparagraph{subparagraph}
  subparagraph text
  subparagraph text
\chapter[somethingelse]{goes here}


\chapter{Fitting in the light of LHC data}
\label{ch:LHClight}
The addition of the first LHC datasets into an NNPDF fit allowed for important gains to be made in the precision of the resulting sets. However the potential dataset available for PDF determination from the LHC is increasing at a considerable rate, and datasets are being rapidly updated with more precise measurements. There is therefore still much more potential in the LHC to provide PDF constraint, especially in collider only fits.

With the ever enlarging dataset comes an important question: whether the fitting methodology applied to the pre-LHC dataset is still the best procedure for the extraction of precise parton densities in the LHC era. In order to accommodate the growing LHC dataset and to be able to efficiently explore methodological options, the toolchain used by the NNPDF collaboration had to be updated. The need for an updated fitting apparatus was recognised near the end of development of the NNPDF2.3 PDF set. Previous NNPDF sets were generated by a {\tt FORTRAN} codebase which grew out of the earliest NNPDF determinations. Consequently the codebase suffered from a great deal of inflexibility with regard to the treatment of data. In particular, performing varying cuts and fits to reduced or special datasets were complicated procedures. Additionally as the fits to the pre-LHC dataset were considerably less computationally intensive the core fitting apparatus was not designed with computational efficiency as the first priority, meaning that fits with the LHC dataset were rather sluggish. Beyond being a mere technicality, such slow fits actually meant that detailed studies of the methodology applied to an LHC dataset were prohibitively expensive in computer time.

With these issues in mind, the {\tt nnpdf++} project was initiated, whereby the full NNPDF toolchain has been implemented from scratch in {\tt C++}. The core of the project was built around the efficient $\tt FK$ method described previously, allowing for a much clearer separation in code between theoretical predictions and experimental data along with a much greater efficiency in the convolution. The {\tt FK} products themselves are accelerated via explicit use of {\tt SIMD} vectorisation, and {\tt OpenMP}~\cite{openmp08} provides multiprocessor options.

The framework was designed to be as modular as possible, to allow for the simple and safe modification of sections of the NNPDF methodology without requiring major modifications to the remaining codebase. The re-implementation of the whole NNPDF toolchain also provided an extremely thorough cross-check of the two implementations, and allowed for the step-by-step evaluation of several methodological elements. The results of this re-evaluation and investigation of alternative procedures shall be described in this chapter along with the consequences for future determinations.

\section{Closure testing}
\label{sec:closuretest}
The central element in the methodological review conducted with the {\tt nnpdf++} code after NNPDF2.3 is the closure testing procedure.

In a closure test, a PDF fitter takes their tools and applies them to a set of pseudo-experimental data generated from a known prior parton distribution set. Provided that the theory used to generate the pseudodata is identical to that used in the fitting procedure, the results of the fit should reproduce the generating function to within the estimation of PDF error. The test is an extremely sensitive check of a fitting procedure, in that it tests the ability of a methodology to resolve the underlying law when said law is known exactly. The method can also be used to study the effect of data inconsistencies by artificially modifying data uncertainties as is examined in Ref.~\cite{Watt:2012tq}, however here we shall restrict ourselves to examining the quality of reproduction of the underlying law.

Closure tests in the NNPDF methodology can be performed in a number of ways. One possible method is a direct fit to theoretical predictions generated from a known distribution, in this way the pseudo-dataset is free from the statistical noise that would be present in experimental data. $N_{\text{rep}}$ PDF replicas are then fitted to the theory predictions, without performing the generation of a Monte Carlo artificial data sample. In this type of fit one aims to reproduce as well as possible the generating function at the end of the fit. As no statistical noise is inserted at any point the final fit quality should approach $\chi^2=0$, we shall therefore denote such a fit a \emph{level zero} closure test.

Alternatively one may perform a fit where statistical noise is introduced to the pseudo-dataset according to the experimental uncertainty present in the real dataset. This can be done in two ways; either the noise is introduced directly to the pseudo-data itself whereby all Monte Carlo replicas fit to the same noisy sample, or noise is introduced on a replica-by-replica basis as in the normal Monte Carlo procedure. These types of fit we denote \emph{level one} closure tests.

Finally one can introduce two levels of noise to the data. The first; applied directly to the pseudo-data, simulates the experimental noise in the distributions. The second level is introduced through the normal Monte Carlo generation of artificial data replicas. This is denoted a \emph{level two} closure test and is the closest to a full fledged PDF fit. The main exception here being the lack of any inconsistency between datasets, as they have all been generated from the same initial distribution. In the case of a level two fit the PDF fitter wishes to reproduce the underlying law to within their quoted PDF uncertainties, the exact reproduction available at level zero is now unavailable due to the introduced pseudo-experimental noise. The level two fit is therefore the most stringent test of a fitting procedure in that it tests the central claim of a fitting group; that the underlying law should lie within the quoted PDF uncertainty band at the quoted confidence level. The settings used in the different closure tests are summarised in Table~\ref{tab:closurelevel}. As a direct comparison of some example pseudodata, Figure~\ref{fig:closurepseudodata} shows example data at closure test levels zero, one and two.

The new structure present in the {\tt nnpdf++} code, particularly the modular treatment of experimental data and theoretical predictions, allows for the straightforward use of predictions in the place of experimental data while keeping the experimental covariance matrices intact. The closure testing method has therefore been extensively applied to the development of the NNPDF methodology, with the procedure used for the NNPDF3.0 determination being guided largely by results from closure testing. Here we shall outline some general results, before demonstrating the application of the procedure to methodological development in the subsequent sections.

\begin{table}[h!]
\label{tab:closurelevel}
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
C.~Level & Exp.~Noise & Art.~Data\\ \hline
0  &  \text{\sffamily X} & \text{\sffamily X} \\
1a  & \checkmark & \text{\sffamily X} \\
1b  & \text{\sffamily X} & \checkmark \\
2  & \checkmark & \checkmark \\
\hline
\end{tabular}
\caption[Levels available in a closure test fit]{Levels available in a closure test fit (C.~Level), Exp.~Noise corresponds to simulating experimental noise in the pseudodata sample. Art.~Data refers to the generation of artificial data replicas in the Monte Carlo uncertainty procedure.}
\end{center}
\end{table}%

\begin{figure}[hp]
\centering
\includegraphics[width=0.9\textwidth]{7-PostLHC/figs/Closuretest_levels/closuretest_levels.pdf}
\caption[Closure test pseudodata example]{Examples of pseudodata used in a closure test for all three levels. The black circles show the level zero pseudodata, and the experimental error bars. The blue squares show the pseudodata after experimental noise has been simulated (level one) and the red diamonds after both statistical noise simulation and Monte Carlo replica generation (level two). All points are normalised to the generating PDF set (NNPDF2.3).}
\label{fig:closurepseudodata}
\end{figure}
\clearpage

\subsubsection{Early closure tests}
The earliest NNPDF closure tests were conducted to assess the usefulness of the procedure, and performed with the full NNPDF2.3 procedure. As an initial test, a fit was performed to the toy PDF parametrisation as used in the Les Houches evolution benchmarks~\cite{Giele:2002hx}, a parametrisation based upon the CTEQ5M determination~\cite{Lai:1999wy}. In this set, the initial state distributions are given as

\begin{eqnarray}
\label{gsav-eq9}
  xu_v(x,\mu_{\rm f,0}^2)       &\! =\! & 5.107200\: x^{0.8}\: (1-x)^3,
    \nonumber \\
  xd_v(x,\mu_{\rm f,0}^2)       &\! =\! & 3.064320\: x^{0.8}\: (1-x)^4,
    \nonumber \\
  xg\,(x,\mu_{\rm f,0}^2)       &\! =\! & 1.700000\, x^{-0.1} (1-x)^5,
    \nonumber \\
  x\bar{d}\,(x,\mu_{\rm f,0}^2) &\! =\! & .1939875\, x^{-0.1} (1-x)^6,
    \nonumber\\
  x\bar{u}\,(x,\mu_{\rm f,0}^2) &\! =\! & (1-x)\: x\bar{d}\,(x,\mu_{\rm f,0}^2),
    \nonumber\\
  xs\,(x,\mu_{\rm f,0}^2)       &\! =\! & x\bar{s}\,(x,\mu_{\rm f,0}^2)
    \: = \: 0.2\, x(\bar{u}+\bar{d}\,)(x,\mu_{\rm f,0}^2),
\end{eqnarray}
where $u_v$ and $d_v$ refer to the up and down valence distributions respectively. Predictions for the NNPDF2.3 dataset were made according to these distributions, and used in the place of experimental data. Experimental noise was simulated in the pseudodata by application of the same procedure used to provide artificial data replicas. The full NNPDF2.3 procedure including Monte Carlo artificial replicas was then applied to the dataset, the resulting PDF set therefore being a level two type closure test where the generating PDF set should be recovered by the fit within the estimated uncertainties.

Figure~\ref{fig:LHtoyclosure1} displays the results of the level two closure test fit with the Les Houches toy PDFs used as a generating function. The result demonstrates impressive agreement, with the NNPDF2.3 methodology able to accommodate the predictions of the Les Houches toy generating function despite it deviating significantly from the standard NNPDF2.3 result. For all four PDF combinations shown, the results of the closure test maintain distances of less than one standard deviation to the generating function across a wide kinematic range. Of additional interest are the strange distributions, relatively poorly constrainted by the data included in the pseudo-dataset. The strange valence in particular is set to zero in the Les Houches toy. Figure~\ref{fig:LHtoyclosure2} shows the results from the closure test for both the total strangeness and strange valence distributions, the NNPDF methodology is able to clearly reproduce the underlying law within uncertainties in both cases, and is able to comfortably resolve a zero strange valence contribution.

The results are particularly impressive considering that this is a test of a methodology that has not been previously verified by closure test. The example case of a pseudo-dataset generated according to the Les Houches toy PDF is however a rather simplified case, and methodological refinements can be made by examining closure tests with greater structure in the generating function.

A good level of agreement can also be found at the level of the $\chi^2$ to both the pseudodata sample, and the real experimental data. In Figure~\ref{fig:CPPclosurechi2} we compare the fit quality of a closure test and its generating PDF dataset by dataset by presenting the $\chi^2$ to each measurement from both the closure test result and the generating PDF. In this case the generating function has considerably greater complexity, being an early {\tt nnpdf++} test fit with most of the NNPDF methodology in place. While agreement is generally very good, especially on the level of total $\chi^2$; we begin to see some elements of discrepancy in datasets sensitive to flavour separation and strangeness such as the NuTeV dataset and electroweak vector boson production data. Such discrepancies can help in pinpointing areas where further development is needed.

\clearpage
\begin{figure}[h!]
\centering
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/gluon.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/singlet.pdf}\\
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/valence.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/triplet.pdf}
\caption[PDFs obtained through a Closure test fit with toy PDFs as a generating function]{PDFs obtained through a closure test fit with Les Houches toy PDFs as a generating function, displayed as a ratio to the generating function. Shown are the distributions for the gluon, singlet, valence and triplet PDFs. In green are the results obtained through the closure test, and the red curves show the standard NNPDF2.3 result.}
\label{fig:LHtoyclosure1}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/strangesea.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/c1_n64.pdf}
\caption[Strange PDFs obtained through a Closure test fit with toy PDFs as a generating function]{Strange sea (left) and valence (right) PDFs obtained through a closure test fit with Les Houches toy PDFs as a generating function. The strange sea is presented as a ratio to the LH toy PDF, and the strange valence is presented directly as the PDF, with the (zero) LH toy line shown.}
\label{fig:LHtoyclosure2}
\end{figure}
\clearpage

\begin{figure}[!]
\centering
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/chi2_histo_nnpdf1.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/chi2_histo_nnpdf2.pdf}
\caption[$\chi^2$ values to the pseudo- and experimental-datasets of a closure fit and the generating PDF]{Example $\chi^2$ values to the pseudo- (left) and experimental- (right) datasets of a closure fit and the generating PDF from early {\tt nnpdf++} test fits. The red bars show the fit quality of the generating PDF set while the green bars demonstrate the $\chi^2$ for the closure test set. The horizontal lines indicate the average and $1\sigma$ of the fit qualities in their associated colours.}
\label{fig:CPPclosurechi2}
\end{figure}

\section{Preprocessing}

Early closure tests performed with the NNPDF2.3 methodology showed generally very good agreement between the produced PDFs and the underlying functions used to generate the pseudo-dataset. However some PDF combinations demonstrated rather poorer agreement than others, particularly distributions sensitive to flavour separation. Such disagreements became more apparent when considering closure tests to underlying functions with more structure than available in the Les Houches toy set. The disagreements were found to originate in the choice of the preprocessing exponents used in the definition of the NNPDF parametrisation. Recalling Eqn.~\ref{eq:NNPDF23param}, the structure of the basic NNPDF parametrisation follows
\be f(x) \propto x^{-\alpha} (1-x)^{\beta} \text{NN}(x),\ee
where NN represents the neural network itself, and $\alpha$ and $\beta$ are the preprocessing exponents randomised on a replica-by-replica basis at the start of a fit. The range in which the exponents were randomised has been fixed in the fits up to and including NNPDF2.3, set to a span large enough such that the dependence of the results upon the choice of range was minimised. In such a way the preprocessing was considered to provide a backbone for the neural-network fit and, aside from improving fitting efficiency, to have a minimal impact upon the results.

To study the effect of different preprocessing ranges we can look at estimators for the \emph{effective} asymptotic exponents,
\be \alpha_{\text{eff}} = -\frac{\log{(|f(x)|)}}{\log(x)},\quad\quad \beta_{\text{eff}} = \frac{\log{(|f(x)|)}}{\log(1-x)}, \ee
such that in the limits of $x\to0,1$ the exponents $\alpha,\beta$ are recovered. By examining these effective exponents in the high- and low-$x$ regions, we can ascertain if there is a data preference for a different preprocessing range than was used in a fit, and if the preprocessing range used was too restrictive.

In Figure~\ref{fig:preproc1} an example preprocessing analysis is shown for a closure test based upon an MSTW08 underlying law at NLO. The sea asymmetry $\bar{u} -\bar{d}$ is shown for two choices of preprocessing range, the NNPDF2.3 standard and a range modified to better accommodate the data preference visible in the effective exponents. From the figure we can see that the choice of exponent randomisation range has a significant effect on the resulting distributions, and that the effective exponents can show a clear data preference for a different range. In Figure~\ref{fig:preproc2} we can see the same analysis applied to the triplet PDF where similar conclusions may be drawn.

These analyses demonstrate that the sensitivity to the preprocessing exponent randomisation ranges is somewhat larger than suspected previously, and needs to be studied in detail in order to avoid minimisation difficulties in a fit where the preprocessing ranges are ill-suited to the dataset. Furthermore, the uncertainty bulges visible in both the triplet and sea asymmetry distributions in Figures~\ref{fig:preproc1} and~\ref{fig:preproc2} are generated by the preprocessing suppressing genuine data uncertainty in the asymptotic regions. These problems may be alleviated by lifting the requirement that such distributions should be preprocessed to zero at low-$x$, and implementing a procedure for the iterative and data-driven determination of preprocessing exponents.

To improve the minimisation performance, hampered by ill-suited preprocessing, NNPDF fits have now adopted the following iterative procedure for the determination of both high and low-$x$ randomisation ranges:

\begin{itemize}
\item \textbf{Singlet and gluon PDFs}\\
Exponent randomisation ranges are set to be twice the $1\sigma$ interval of the previous iteration's effective exponents
at the asymptotic points.
\item \textbf{Nonsinglet PDF combinations}\\
The low-$x$ randomisation interval is set to be the maximal extent of two effective exponent ranges; twice the $1\sigma$ interval at the asymptotic point and twice the $1\sigma$ interval at the point $x=1\times 10^{-3}$. The high-$x$ interval is set identically as with the singlet and gluon.
\end{itemize}

In such a way convergence of the randomisation interval can be established typically in two or three fit iterations, and the preprocessing exponents are obtained from the preference of the experimental dataset. As an example of a fit generated from such an iterative procedure consider Figure~\ref{fig:preproc3} which demonstrates the preprocessing analysis for the $\Delta_s$ and Triplet distributions resulting from the new procedure. In comparison to Figure~\ref{fig:preproc1} where the old settings are used, the low-$x$ preprocessing ranges have relaxed considerably and are no longer constrained by the chosen exponent range but driven by the experimental data. Furthermore the agreement with the underlying law is noticeably improved over the previous result shown in Figure~\ref{fig:preproc1}.

\begin{figure}[hp!]
\centering
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/Preproc1/pdf_xDs_log_others.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/Preproc2/pdf_xDs_log_others.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/Preproc2/alphapreproc_4.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/Preproc2/betapreproc_4.pdf}
\caption[Demonstration of the impact made by changes in preprocessing to the sea asymmetry PDF in a closure test fit]{Demonstration of the impact made by changes in preprocessing to the sea asymmetry PDF in a closure test fit to an MSTW08 NLO underlying law. The top two figures demonstrate the results for the $\Delta_s$ distribution for two choices of preprocessing ranges, with the left figure using NNPDF2.3 standard preprocessing. In both cases, the red curve shows the underlying law used in the Closure test. The right figure demonstrates slightly improved agreement, particularly at low-$x$. The lower figures show the low and high $x$ effective exponent plots for the two ranges. The solid horizontal lines delineate the regions in which the preprocessing exponents were initialised, and the bands show the $1\sigma$ contours of the effective exponents.}
\label{fig:preproc1}
\end{figure}

\clearpage
\begin{figure}[!]
\centering
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/Preproc1/pdf_xT3_log_others.pdf}
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/Preproc2/pdf_xT3_log_others.pdf}
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/Preproc2/alphapreproc_3.pdf}
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/Preproc2/betapreproc_3.pdf}
\caption[Demonstration of the impact made by changes in preprocessing to the triplet PDF in a closure test fit]{A further preprocessing analysis as in Figure~\ref{fig:preproc1}, performed upon the Triplet PDF combination for the same two closure test fits.}
\label{fig:preproc2}
\end{figure}

\begin{figure}[!]
\centering
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xDs_log_others.pdf}
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xDs_others.pdf}
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xT3_log_others.pdf}
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xT3_others.pdf}
\caption[Impact of improved preprocessing range selection in the sea asymmetry and triplet PDFs]{Impact of improved preprocessing range selection in the sea asymmetry and triplet PDFs. The top figures demonstrate the $\Delta_s$ PDF obtained via a closure test to MSTW08 using the improved preprocessing procedure in green, with the underlying law shown in red. The figures below show the equivalent plots for the triplet PDF with the improved preprocessing ranges.}
\label{fig:preproc3}
\end{figure}
\clearpage

% Old figures
%\begin{figure}[!]
%\centering
%\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xDs_log_others.pdf}
%\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xDs_others.pdf}
%\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/alphapreproc_4.pdf}
%\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/betapreproc_4.pdf}
%\caption[Impact of improved preprocessing range selection in the sea asymmetry PDF]{Impact of improved preprocessing range selection in the sea asymmetry PDF. The top figures demonstrate the $\Delta_s$ PDF obtained via a closure test to MSTW08 using the improved preprocessing procedure in green, with the underlying law shown in red. The figures below show the effective exponents for this PDF, and the effect of the improved range selection.}
%\label{fig:preproc3}
%\end{figure}
%\clearpage
%
%
%\begin{figure}[!]
%\centering
%\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xT3_log_others.pdf}
%\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xT3_others.pdf}
%\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/alphapreproc_3.pdf}
%\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/betapreproc_3.pdf}
%\caption[Impact of improved preprocessing range selection in the triplet PDF]{Impact of improved preprocessing range selection in the triplet PDF. Displayed as in Figure~\ref{fig:preproc3}. In the case of the high $x$ effective exponent, the data's preferred range lies well within the set randomisation range of the exponents. For the small-$x$ exponent, more iteration is needed here to settle on a larger range. The general features of the extended preprocessing ranges can be clearly seen in the top figures.}
%\label{fig:preproc4}
%\end{figure}

\subsection{Strange valence preprocessing}
A special case when considering the preprocessing of the neural networks is that of the strange valence distribution. As specified in Equation~\ref{eq:NNPDF23param}, the strange valence PDF in the NNPDF2.3 determination had an auxiliary term to encourage the PDF to perform its required sign change in the valence region. Such an additional term has been previously needed due to the lack of specific data constraints upon the strange valence distribution before the LHC, introducing a bias, albeit a physically motivated one. Additionally the auxiliary term provides a mechanism by which the strange valence sum rule may be imposed. In the NNPDF3.0 determination and beyond this auxiliary term has been removed given the enlarged dataset and it's improved sensitivity to the strange PDF.

Figure~\ref{fig:preproc5} demonstrates the effect of the removal of the strange auxiliary term upon a closure test fit to the MSTW08 set. While the NNPDF2.3 methodology closure fit struggles to accommodate the MSTW08 strange valence distribution, the updated methodology is able to reproduce the underlying law well, within enlarged uncertainties.

\begin{figure}[h!]
\centering
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/Preproc1/pdf_xsminus_log_others.pdf}
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/Preproc1/pdf_xsminus_others.pdf}
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xsminus_log_others.pdf}
\includegraphics[width=0.42\textwidth]{7-PostLHC/figs/PreprocFixed/pdf_xsminus_others.pdf}
\caption[Impact of more flexible treatment of strange valence PDF in fits post NNPDF2.3]{Impact of more flexible treatment of strange valence PDF in fits post NNPDF2.3. The top two figures show a comparison of a closure test performed with the NNPDF2.3 preprocessing, and the figures below show the results using the more flexible parametrisation.}
\label{fig:preproc5}
\end{figure}

\clearpage

\section{PDF parametrisation}
The choice of PDF parameterisation and basis used in the fitting procedure has been reassessed with the help of the closure test procedure. In particular, a modification to the choice of fitting basis has been made necessary by the removal of the strange valence sum rule enforcing auxiliary term in the strange valence parametrisation. The most direct choice of fitting basis is the same basis as is used in PDF evolution, and therefore the basis required for PDFs in the {\tt FK} product. In this basis, the required quantum number sum rules may be applied as normalisations to the total valence, $V_3$ and $V_8$ distributions,
\ba V(x,Q^2_0) &=& N_V \left( u^- + d^-+ s^- \right)(x,Q^2_0),\nonumber \\
V_3(x,Q^2_0) &=& N_{V3} \left( u^- - d^- \right)(x,Q^2_0), \nonumber \\
V_8(x,Q^2_0) &=& N_{V8} \left( u^- + d^- - 2s^-\right)(x,Q^2_0),
\ea
where the normalisations $N$ are set such that
\ba \int_0^1 dx\, V(x,Q^2_0) &=& 3,\\
 \int_0^1 dx\, V_3(x,Q^2_0) &=& 1,\\
 \int_0^1 dx\, V_8(x,Q^2_0) &=& 3.\ea

In such a way, the total valence quantum number is fixed, along with the up, down and strange valence quantum numbers. The evolution basis also has the advantage of being
particularly efficient, not requiring any transformation before combination with {\tt FK} tables to calculate physical observables. We have shown, based upon closure test results, that the fit results
show a good degree of stability under such a change in parametrisation basis. While the previous strategy was designed to construct PDF combinations with specific data constraints, the flexibility of the fit
means that the results do not suffer when moving away from such a basis. Figure~\ref{fig:EVOLvs23BASIS} shows the statistical distance between a fit with the full evolution basis with a fit based upon the standard NNPDF2.3 parametrisation basis.
As expected, any differences are isolated to those PDFs whose parametrisation (and therefore preprocessing) has substantially changed e.g $\Delta_S$ and the strange PDFs. Even in these PDFs the differences are typically
less than half a standard deviation.

\begin{figure}[!]
\centering
\includegraphics[width=0.9\textwidth]{7-PostLHC/figs/EVOLvs23BASIS/distances_evol.pdf}
\caption[Distance comparison of two closure test fits with differing parametrisation bases]{Distance comparison of two closure test fits with differing parametrisation bases. Distances are defined through the measure in Appendix~\ref{app:distances}, whereby a distance of 10 corresponds to $1\sigma$.}
\label{fig:EVOLvs23BASIS}
\end{figure}

\section{Minimisation and stopping}
In addition to examining areas where the choice of parametrisation may lead to some degree of bias, the closure test procedure is particularly useful for assessing the efficacy of a fitting methodology. Furthermore, the substantial gains in computational efficiency made in the transition to the {\tt nnpdf++} code mean that far more aggressive genetic minimisation strategies may be implemented.

The entirety of the NNPDF minimisation procedure has therefore been re-examined to ensure that it is the most effective methodology in the light of additional constraints coming from the LHC. Here we shall summarise some of the major modifications made since the NNPDF2.3 determination.

\subsection{Target weighted training}
Target Weighted Training (TWT) was a central feature of previous NNPDF determinations. TWT was developed in early NNPDF fits as a method of obtaining a balanced training across datasets, solving a problem with early neural network fits whereby some smaller datasets
were largely ignored by the minimisation in favour of larger, more constraining sets. This typically led to a very uneven fit quality profile over the complete experimental dataset. The TWT procedure solved this problem by introducing a training epoch at the beginning of a fit where each dataset had a target $\chi^2$. In the event where a fit iteration reached a $\chi^2$ value higher than the target, a large weight in fit quality was applied to that dataset in order to bring its fit quality down.

While ensuring a relatively even training profile, the TWT procedure had a number of difficulties. The most important being the restriction of the early fit to a $\chi^2$ fit quality measure applied on a dataset-by-dataset basis, ignoring experimental uncertainty cross-correlations such as luminosity uncertainties, between datasets. Furthermore the TWT procedure introduced a considerable amount of complexity in the fitting procedure. With this in mind, real data fits with target weighted training were compared to fits without in the {\tt nnpdf++} framework with the large experimental dataset of NNPDF2.3 and updated genetic algorithm parameters. Figure~\ref{fig:twtvsnotwt} compares the dataset-by-dataset fit quality of two such example fits. With these fits we can see clearly that with a larger dataset and more efficient GA procedure, no large training imbalance can be seen in the fits even without the TWT procedure applied. Future NNPDF fits will therefore be performed without target weights, allowing for the consistent application of the experimental correlations across datasets throughout the fitting procedure.

\begin{figure}[!]
\centering
\includegraphics[width=1\textwidth]{7-PostLHC/figs/TWTvsnoTWT/chi2_histo.pdf}
\caption[Comparison of $\chi^2$ by dataset between real data fits with and without Target Weighted Training]{Comparison of $\chi^2$ by dataset between real data fits with (green bars) and without (red bars) Target Weighted Training.}
\label{fig:twtvsnotwt}
\end{figure}

\subsection{Genetic algorithm}
A number of changes have been made to the GA procedure used in NNPDF fits in order to improve fitting efficiency and provide more precise PDF determinations. In the analysis of the efficacy of a GA, level zero closure tests are particularly helpful in that they
directly test the ability of a minimisation procedure to reproduce a given function precisely. In these fits the closure test fit should be able to effectively draw a line between datapoints, leading to an ideal $\chi^2$ of zero to the pseudo-data.
%
%
%140430-r1753-001-sc.ini - DynStop
%140512-r1765-003-cd.ini  - LookBack 30k
%140512-r1765-004-cd.ini  - LookBack 60k
%
%
A number of modifications to the procedure have been tested, the most effective of which is the implementation of \emph{Nodal} mutations in the GA~\cite{Montana:1989:TFN:1623755.1623876}. In previous versions of the NNPDF GA, mutations were performed upon individual parameters of each neural network with no consideration as to their position in the network.

The concept of nodal mutations introduces the strategy of mutating all parameters associated with a particular neural network \emph{node} at once. In this procedure a node of the network is chosen at random, then all of its associated weights connected to the earlier layer are mutated along with its threshold parameter. Doing so yields a much more effective genetic algorithm as demonstrated in the comparison in Figure~\ref{fig:nodalvsnonnodal}, where a standard GA is compared to a nodal mutation GA in their reproduction of the MSTW underlying law. The nodal GA is able to better resolve the underlying law, and to a greater precision. The comparison in Figure~\ref{fig:nodalvsnonnodal} is corroborated by the $\chi^2$ values of the two fits to the perfect pseudo-data in the level zero fit. The standard GA fit shown in the figure obtained a final $\chi^2$ of 0.0279 compared to 0.0043 for the nodal GA. The nodal GA strategy has therefore been adopted for future NNPDF determinations.

\begin{figure}[!]
\centering
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/NodalGA/singlet.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/NodalGA/triplet.pdf}
\caption[Comparison of a conventional NNPDF GA fit with a Nodal GA fit in a closure test to MSTW2008]{Comparison of a conventional NNPDF GA fit (red bands) with a Nodal GA fit (green bands) in a closure test to MSTW08. PDFs are given as a ratio to the generating PDF set for the singlet (left) and triplet (right) distributions.}
\label{fig:nodalvsnonnodal}
\end{figure}

\subsection{Dynamical stopping}
The cross-validation dynamical stopping procedure utilised in previous {\tt FORTRAN} based NNPDF fits was triggered by a slope-detection algorithm applied to the fit quality profiles of each replica to the validation dataset. While providing a reasonable stopping criteria and preventing excessive overfitting, the relative balance between the degree of under- and over-learning was governed by the parameters of the slope-detection algorithm. Such sensitivity to the stopping parameters meant that a re-tune was often necessary upon large modifications to the dataset or minimisation algorithm.

The modular nature of the stopping criteria implemented in the {\tt nnpdf++} framework means that alternative stopping procedures may be quickly and safely implemented to investigate their impact. One such stopping criterion that has demonstrated greater stability than the previous slope-detection based procedure is that of \emph{look-back} cross-validation.

In this procedure all replicas are run for the maximum number of generations $N_{\text{gen}}^{\text{max}}$, all the while storing the GA generation that best described the validation dataset. At the end of the fit, the GA generation that minimised the $\chi^2$ to the validation set is selected as the best-fit stopping point, and that replica is used as a member of the Monte Carlo ensemble. This method yields an extremely clean stopping criterion, having no tuneable parameters aside from the maximum number of generations, and offers a very faithful implementation of the cross-validation method. Furthermore, the look-back procedure is not practically more time-consuming to implement despite running each replica to the maximum number of generations, as even in the previous dynamical stopping procedure the time taken to run a fit is typically given by the time taken by the slowest replica. In Figure~\ref{fig:LBCVchi2prof} the fit quality profile for a single PDF replica can be seen for the training and validation sets alongside the look-back stopping point. In this case, the look-back method can clearly discern an overlearning signal, as the fit quality to the validation set worsens while the training set $\chi^2$ improves.

\begin{figure}[!]
\centering
\includegraphics[width=0.7\textwidth]{7-PostLHC/figs/lookbackchi2prof.pdf}
\caption[Fit quality profiles for the training and validation sets in look-back cross validation]{Fit quality profiles for the training and validation sets in look-back cross validation. The red curve shows the fit quality to the training set, and the blue curve to the validation set as the number of fit generations goes on. The green line indicates the stopping point selected by the look-back criterion, generation 12813 having the minimum validation $\chi^2$.}
\label{fig:LBCVchi2prof}
\end{figure}

In Figure~\ref{fig:30kLBvsDYN} we compare the results for the singlet and gluon PDFs in the case of a look-back fit with $N_{\text{gen}}^\text{max}=$ 30,000 generations, and a fit with the NNPDF2.3 standard dynamical stopping. In both instances, the fit performed was a level two closure test using MSTW2008 as the underlying law. While differences are small the look-back fit demonstrates slightly smaller uncertainties, implying a marginal underlearning present in the NNPDF2.3 dynamical stopping procedure. The fits yield essentially equivalent results, although the optimal point determined in the look-back method is typically somewhat later than in the dynamical stopping as can be seen in the comparison of training length histograms in Figure~\ref{fig:30kLBvsDYNtl}. In this figure it is clear also that several PDF replicas in the look-back method stop close to the maximum number of generations available, implying that no significant overlearning can been resolved in their cases over the given GA interval.

In order to examine the effect of increasing the length of the look-back period, we compare the 30,000 generation fit to an extended 60,000 generation fit in Figure~\ref{fig:30kLBvs60kLB} where we use the PDF distance definition in Appendix~\ref{app:distances}. Distances of effectively zero throughout the PDF combinations and $x$-range mean that no change is observed between the two fits, demonstrating the stability of the method once a sufficiently large look-back length is used. The look-back cross-validation method as discussed here will therefore be implemented as the default stopping criterion for the NNPDF3.0 family of fits.

\begin{figure}[h!]
\centering
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/LB30kvsDYN/pdf_xg_log.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/LB30kvsDYN/pdf_xg.pdf}\\
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/LB30kvsDYN/pdf_xsigma_log.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/LB30kvsDYN/pdf_xsigma.pdf}
\caption[Comparison of PDFs obtained through look-back cross validation and NNPDF2.3 standard dynamical stopping]{Comparison of PDFs obtained through look-back cross validation and NNPDF2.3 standard dynamical stopping. PDFs for the singlet and gluon are shown, with green bands representing fits using the look-back method and red demonstrating those with the slope-detection algorithm used in NNPDF2.3 and earlier.}
\label{fig:30kLBvsDYN}
\end{figure}

  \begin{figure}[!]
\centering
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/LB30kvsDYN/tl.pdf}
\includegraphics[width=0.48\textwidth]{7-PostLHC/figs/LB30kvsDYN/tl_ref.pdf}
\caption[Comparison of training lengths in look-back cross-validation and NNPDF2.3 standard dynamical stopping]{Comparison of training lengths in look-back cross-validation and NNPDF2.3 standard dynamical stopping. The left figure demonstrates the 'optimal point' determined by looking back over the while GA interval for the minimum validation $\chi^2$. The right figure shows the stopping point based upon the slope-detection algorithm.}
\label{fig:30kLBvsDYNtl}
\end{figure}

\begin{figure}[!]
\centering
\includegraphics[width=0.9\textwidth]{7-PostLHC/figs/LB30kvsLB60k/distances_evol.pdf}
\caption[Distance comparison of two closure test fits with look-back stopping enabled and different maximum training lengths]{Distance comparison of two closure test fits with look-back stopping enabled and different maximum training lengths. Distances are computed between all evolution basis PDFs at the initial scale between $N_{\text{gen}}^\text{max}=$ 30,000 and $N_{\text{gen}}^\text{max}=$ 60,000 generation look-back fits.}
\label{fig:30kLBvs60kLB}
\end{figure}

\section{Methodology for NNPDF3.0}
We have performed an overview of the methodological developments made since the release of the NNPDF2.3 PDF set, with an aim to outline the procedure to be used in the forthcoming NNPDF3.0 set. To provide a stringent verification of the combined procedure, we shall now examine a set of closure test fits performed at various levels to differing generating PDF sets. In this section we present fits based upon a nodal genetic algorithm minimisation with look-back cross-validation stopping as detailed previously, with the iterative preprocessing procedure and new PDF fitting basis. Therefore the fits represent preliminary closure test results for the NNPDF3.0 methodology, upon a global pseudo-dataset of hadronic and DIS data. Results in this section will be presented using NLO calculations for the observables in the fit, although the conclusions will be very similar for an identical analysis at NNLO, as the closure test procedure is relatively insensitive to theory choices.

\subsection{Closure tests for NNPDF3.0}

Firstly let's consider the results obtained when fitting to an MSTW2008 generating PDF, the closure test guiding the methodological choices made so far in this section. In Figure~\ref{fig:finalClosure_MSTW} the ratio of the resulting closure test PDFs to the generating MSTW08 distributions are shown for some of the evolution basis PDFs. Here we show results for the kinematic region most constrained by the experimental pseudo-dataset: $10^{-2} \le x \le 1$. The level zero curves in Figure~\ref{fig:finalClosure_MSTW} closely reproduce the MSTW central values, achieving a final total $\chi^2/N_{\text{dat}} = 0.00182$. The uncertainty band in the case of the level zero result corresponds directly to the functional freedom available within the fitted pseudo-dataset. The level two fit clearly demonstrates the variations introduced by the simulated experimental noise, with the expected level of deviation clearly visible in the resulting PDFs. Given the simulated noise in the pseudo-dataset, the closure test still tracks the central value to an excellent level of accuracy, achieving an almost statistically ideal fit quality of $\chi^2/N_{\text{dat}} = 1.00021$.

As the preliminary NNPDF3.0 methodology has been validated against closure test fits to the MSTW2008 set, it is important to test the procedure's ability to reproduce a generating PDF with greater functional complexity. To verify the preliminary methodology in this case we now consider a closure test fit to the NNPDF2.3 PDF set. Figure~\ref{fig:finalClosure_NNPDF} demonstrates once more the level zero and two closure test fits to NNPDF2.3. Even given the greater functional freedom present in the previous NNPDF determination, the 3.0 closure test provides an excellent reproduction of the generating functions, with fit qualities of  $\chi^2/N_{\text{dat}} = 0.00287$ and $1.01356$ respectively. Once again the uncertainty due to parametrisation flexibility is demonstrated in the level zero fit, while the level two fit provides a closer simulation of a full fledged experimental data fit. These figures therefore suggest that the preliminary NNPDF3.0 methodological choices can accurately determine complex functional forms without any modification with respect to fits to much simpler parametrisations.

For a final closure test, we shall now consider a fit using the CT10 PDF set as a set of generating functions. In this way we can verify the NNPDF3.0 method in a way that is independent of the closure PDF set guiding the methodological development (MSTW2008) and previous NNPDF determinations. The results of the test, once more at level zero and two, are shown in Figure~\ref{fig:finalClosure_CT10}. The closure test fit provides once again an excellent description of data, with $\chi^2/N_{\text{dat}} = 0.00130$ for the level zero fit and $1.01324$ for the level one. The procedure detailed here has now been validated against three different generating PDFs in a closure test and is able to convincingly reproduce the generating sets in each of them. We can therefore be confident that when applied to real experimental data the procedure will yield an accurate result up to theoretical uncertainties.

\begin{figure}[!]
\centering
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/MSTW0/gluon.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/MSTW0/singlet.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/MSTW0/valence.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/MSTW0/triplet.pdf}
\caption[NNPDF3.0 methodology closure test fit to MSTW2008 NLO]{NNPDF3.0 methodology closure test fit to MSTW2008 NLO. Curves are shown normalised to the generating PDF for the gluon, singlet, triplet and valence distributions. The green curves show the results of a level zero closure test, while the red curves show the results of a level two test.}
\label{fig:finalClosure_MSTW}
\end{figure}
\begin{figure}[!]
\centering
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/NNPDF23/gluon.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/NNPDF23/singlet.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/NNPDF23/valence.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/NNPDF23/triplet.pdf}
\caption[NNPDF3.0 methodology closure test fit to NNPDF2.3 NLO]{NNPDF3.0 methodology closure test fit to NNPDF2.3 NLO. Plots as in Figure~\ref{fig:finalClosure_MSTW}. }
\label{fig:finalClosure_NNPDF}
\end{figure}
\begin{figure}[!]
\centering
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/CT10/gluon.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/CT10/singlet.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/CT10/valence.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/finalClosure/CT10/triplet.pdf}
\caption[NNPDF3.0 methodology closure test fit to CT10 NLO]{NNPDF3.0 methodology closure test fit to CT10 NLO. Plots as in Figure~\ref{fig:finalClosure_MSTW}. }
\label{fig:finalClosure_CT10}
\end{figure}

\subsection{Improvements in data fits for NNPDF3.0}
While we have now validated much of the methodology to be used in the NNPDF3.0 determination, we shall now finally investigate some of the expected differences arising with respect to the NNPDF2.3 results in the case of a fit to experimental data. In order to directly assess the changes arising purely from the methodological differences in the two approaches, we shall perform two fits to a small common dataset, one with the full NNPDF2.3 machinery and the second with the improvements implemented in the NNPDF3.0 procedure. It should be noted that these results are of an extremely preliminary nature and as so should only be taken as roughly indicative of the final results. Furthermore, the full NNPDF3.0 set will benefit from a considerably expanded dataset with respect to the NNPDF2.3 determination.

For these test fits, we use a collider-only dataset to ensure a maximally consistent set of experimental data, including the full NNPDF2.3 LHC and Tevatron datasets, and the HERA-1 combined DIS results. Once more, the fits were run with a maximum number of generations of $N_{\text{gen}} = 30,000$. The NNPDF2.3-like fit was otherwise performed according to the settings of the central NNPDF2.3 fit. The NNPDF3.0 fits were performed with identical settings to the closure test fits described in the previous section.

Looking firstly at the gluon and singlet sectors, in Figure~\ref{fig:23vs30methodology_1} we see the results of the two methodology test fits compared as a ratio to the NNPDF2.3 methodology fit's central value. The first feature to note is that in the region where data constraints in this test fit are largest, the two methodologies remain very consistent in their results, with the most significant changes occurring in the extrapolation regions and for the large-$x$ singlet. At small-$x$ the NNPDF3.0 methodology fit is more confident in the extrapolation for both singlet and gluon PDFs, resulting in a systematically smaller uncertainty. At large-$x$ there is a moderate shift in the gluon central value in the NNPDF3.0 result, and a broadening of uncertainties. The same pattern can be found in the large-$x$ gluon, where once again uncertainties are slightly larger and there is some change in central value. However both distributions remain in agreement within their uncertainties, validating that the two methodologies remain compatible within the experimental uncertainty present in the test dataset.

To investigate the impact of the methodological changes to PDFs sensitive to the valence distributions and quark flavour separation, we plot the valence and triplet PDF combinations in Figure~\ref{fig:23vs30methodology_2}. In the valence PDF comparison, we see a similar pattern as for the singlet and gluon PDFs, where the low-$x$ result from the NNPDF3.0 methodology fit obtained a narrower distribution, and at high-$x$ the uncertainties are systematically larger. The triplet PDF shows by some way the largest differences between the two methodologies, with PDF uncertainties being significantly larger across the whole range of $x$. This effect is largely due to the much more flexible preprocessing used for the triplet PDF, where now there is no requirement that the PDF be preprocessed to zero at low-$x$, the constraint now being entirely based on experimental data. Such a treatment leads to a rather conservative determination of the low-$x$ triplet, however this effect should be at least partially offset by increased data constraints in the full NNPDF3.0 determination.

Here we have seen the two methodologies provide consistent results when applied to the same experimental dataset. However there are significant changes in the fit results due to methodological improvements, particularly important in the PDF extrapolation regions at large and small values of parton-$x$, and for PDF combinations sensitive to light flavour separation. As has been shown in the validation with closure tests, the methodological modifications, particularly in allowing for greater preprocessing flexibility, result in an improved reproduction of a test PDF distribution. The upgraded methodology should therefore provide a more reliable estimate of the parton densities of the proton.

\clearpage
\begin{figure}[!]
\centering
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/30meth/plots/glulog.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/30meth/plots/glulin.pdf}\\
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/30meth/plots/snglog.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/30meth/plots/snglin.pdf}

\caption[Comparison of NNPDF2.3 and NNPDF3.0 fitting methodologies when applied to a common experimental dataset. Gluon and singlet PDF combinations]{Comparison of NNPDF2.3 and NNPDF3.0 fitting methodologies when applied to a common experimental dataset. Here the gluon (top) and singlet (bottom) PDFs are shown, with all values normalised to the result of the NNPDF2.3 methodology fit.}
\label{fig:23vs30methodology_1}
\end{figure}

\begin{figure}[!]
\centering
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/30meth/plots/vallog.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/30meth/plots/vallin.pdf}\\
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/30meth/plots/t3log.pdf}
\includegraphics[width=0.45\textwidth]{7-PostLHC/figs/30meth/plots/t3lin.pdf}

\caption[Comparison of NNPDF2.3 and NNPDF3.0 fitting methodologies when applied to a common experimental dataset. Valence and triplet PDF combinations]{Comparison of NNPDF2.3 and NNPDF3.0 fitting methodologies when applied to a common experimental dataset, for the valence (top) and triplet (bottom) PDF combinations. Plots as in Figure~\ref{fig:23vs30methodology_1}.}
\label{fig:23vs30methodology_2}
\end{figure}