\section{Results}
\label{sec:results}
This section discusses the experimental results of this project. We present the RMSLE scores for both default and tuned models when cross-validated on each of three datasets together with the training time taken.

\subsection{Sberbank Dataset}
From our perspective, this dataset can be called the most challenging among the observed ones but at the same time the most practice-oriented as it comprises substantial amount of information incorporated in 293 features and more than 12000 features. As can be seen from \cref{tab:sberbank-results}, the performance of all three models is very similar for both default and tuned models, although XGBoost slightly outperforms the others. However, they significantly differ in the terms of training time: CatBoost is more than four times slower than two other models.

\begin{table}[htbp]
	\centering
	\resizebox{\linewidth}{!}{
		\begin{tabular}{lrrrr}
			\toprule
			\textbf{Model} & \textbf{Default} & \textbf{Tuned} & \textbf{Max\_depth, n\_estimators, learning rate} & \textbf{Training time} \\
			\midrule
			CatBoost & 0.5161 &  0.5102 & 3, 100, 0.01 & 765.91 \\
			XGBoost & \textbf{0.5098} & \textbf{0.5081} & 2, 500, 0.05 & 158.67 \\
			sklearn GB & 0.5102 & 0.5091 & 1, 1000, 0.1  & \textbf{148.41} \\
			\bottomrule
		\end{tabular}
	}
	\caption{Sberbank dataset: RMSLE,  best parameters for tuned model, training time. The best result in each column is highlighted in bold.}
	\label{tab:sberbank-results}
\end{table}

\subsection{Ames Dataset}
\cref{tab:ames-results} presents the results obtained when training the models on Ames dataset. Examining it, we can derive that sklearn GB and XGBoost do comparably well with CatBoost slightly lagging behind for both the default and tuned models. We expected CatBoost to outperform the other models on this dataset as more than a half of its features are categorical. However, special processing of categorical features does not result in the substantial performance gain.

\begin{table}[htbp]
	\centering
	\begin{tabular}{lrrrr}
		\toprule
		\textbf{Model} & \textbf{Default} & \textbf{Tuned} & \textbf{Max\_depth, n\_estimators, learning rate} & \textbf{Training time}  \\
		\midrule
		CatBoost & 0.1789 &  0.1435 & 1, 1000, 0.1  & 58.82\\
		XGBoost & 0.1315 & 0.1246 & 2, 500, 0.1 & \textbf{8.72} \\
		sklearn GB & \textbf{0.1285} & \textbf{0.1241} & 1, 1000, 0.1 & 10.10 \\
		\bottomrule
	\end{tabular}
	\caption{Ames dataset: RMSLE, best parameters for tuned model, training time. The best result in each column is highlighted in bold.}
	\label{tab:ames-results}
\end{table}

\subsection{Boston Dataset}
It can be observed from the results for Boston dataset given in \cref{tab:boston-results} that it is the only dataset on which CatBoost succeeds to outperform other GDBT implemenations. However, this slight advantage gained in performance comes at the cost of training time taken: XGBoost is approximately 75 times faster than CatBoost and sklearn GB is 52 times faster than CatBoost.

\begin{table}[htbp]
	\centering
	\begin{tabular}{lrrrr}
		\toprule
		\textbf{Model} & \textbf{Default} & \textbf{Tuned} & \textbf{Max\_depth, n\_estimators, learning rate} & \textbf{Training time}  \\
		\midrule
		CatBoost & \textbf{0.1584} &  \textbf{0.1423} & 1, 1000, 0.01  & 56.33 \\
		XGBoost & 0.1952 & 0.1753  & 2, 1000, 0.01  & \textbf{0.75} \\
		sklearn GB & 0.1993 & 0.1721 & 2, 1000, 0.01  & 1.07 \\
		\bottomrule
	\end{tabular}
    \caption{Boston dataset: RMSLE, best parameters for tuned model, training time. The best result in each column is highlighted in bold.}
	\label{tab:boston-results}
\end{table}


\subsection{Stochastics}

The lack of a generally valid failure criterion in state-based PD makes an assessment of the initial idea to use a stochastic material distribution for the assessment of failure initiation difficult. However, stochastics may be used to achieve the same entropy in structured discretization as in unstructured base meshes and to individualize failure locations. The comparison of the original hex model with $dx=\SI{0.4}{\milli\meter}$ and horizon $\delta=\SI{1.2}{\milli\meter}$ and three models with stochastic material distribution is shown in \autoref{fig:Results:Hex:Stoch}. Ten different blocks are created with a deviation of the $\SI{2}{\percent}$ of the material bulk and shear modulus.

\pgfplotstableread[col sep=comma]{../../Material/Data/Numerics/Hex_0-4_1-2_Stoch.csv}{\loadedtable}

\begin{figure}[htbp]
  \setlength{\figheight}{7cm}
  \begin{subfigure}{0.55\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
    \centering
%     \includegraphics[width=\linewidth,height=\figheight]{example-image-a}
    \tikzexternalenable
    \tikzsetnextfilename{Hex_0-5_0-5625_Stoch}
    \begin{tikzpicture}
      \begin{axis}[
        height=\figheight+\baselineskip,
        width=\linewidth,
        axis lines=middle,
        cycle list name=color list,%linestyles*,
        cycle list shift=1,
        xmin=0,
        ymin=0,
        title=\empty,
        xlabel={Displacement $[\si{\milli\meter}]$},
        ylabel={Force $[\si{\newton}]$},
        x label style={at={(axis description cs:0.5,-0.075)},anchor=north},
        y label style={at={(axis description cs:-0.105,0.5)},rotate=90,anchor=south},
        legend pos=north west,
        legend cell align={left},
        legend style={font=\footnotesize},
      ]%   each nth point={2}
        \addplot+ [thick] table[x=DxNo, y=FxNo] {\loadedtable};
        \addlegendentry{No stochastics}
        \addplot+ [] table[x=DxSto1, y=FxSto1] {\loadedtable};
        \addlegendentry{Stochastic 1}
        \addplot+ [] table[x=DxSto2, y=FxSto2] {\loadedtable};
        \addlegendentry{Stochastic 2}
        \addplot+ [] table[x=DxSto3, y=FxSto3] {\loadedtable};
        \addlegendentry{Stochastic 3}
      \end{axis}
    \end{tikzpicture}
    \tikzexternaldisable
    \end{minipage}
    \caption{Force-displacement plot until failure}
    \label{fig:Results:Tet:Stoch:FD0-5_0-5625}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.10\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
    \centering
      %\includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{../../Material/Figures/PD_Hex_Damage_0-4_1-2_3630_-z_ct.png}
      \includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{PD_Hex_Damage_0-4_1-2_3630_-z_ct}
    \end{minipage}
    \caption{No}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.10\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
      \centering
      %\includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{../../Material/Figures/PD_Hex_Stoch_1_Damage_0-4_1-2_3475_-z_ct.png}
      \includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{PD_Hex_Stoch_1_Damage_0-4_1-2_3475_-z_ct}
    \end{minipage}
    \caption{1}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.10\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
      \centering
      %\includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{../../Material/Figures/PD_Hex_Stoch_2_Damage_0-4_1-2_3540_-z_ct.png}
      \includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{PD_Hex_Stoch_2_Damage_0-4_1-2_3540_-z_ct}
    \end{minipage}
    \caption{2}
  \end{subfigure}
  \begin{subfigure}{0.10\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
      \centering
      %\includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{../../Material/Figures/PD_Hex_Stoch_3_Damage_0-4_1-2_3292_-z_ct.png}
      \includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{PD_Hex_Stoch_3_Damage_0-4_1-2_3292_-z_ct}
    \end{minipage}
    \caption{3}
  \end{subfigure}
  \caption{Failure for hex-mesh with $dx=\SI{0.4}{\milli\meter}$ and stochastics}
  \label{fig:Results:Hex:Stoch}
\end{figure}

It can be seen that the overall stiffness and failure behavior does not change significantly. However, the stochastic material distribution makes it possible to spot several possible individual failure locations. One would expect a less slanted crack propagation. This can be achieved by using a finer discretization. However, a slightly angular failure path can also be observed in tests with a little different specimen geometry of the same material, see \autoref{fig:Results:Exp:AngledCrackPath}. A contact-free displacement measurement using a video extensometer is used to avoid an influence on the crack path.

\begin{figure}
  \centering
  \includegraphics[width=0.5\linewidth,keepaspectratio]{../../Material/Figures/KrauseD_Damage_LY564_statisch}
  \caption{Angled crack path in same material specimen with different geometry \cite{KrauseD2016}}
  \label{fig:Results:Exp:AngledCrackPath}
\end{figure}

The same principal results are valid for tet meshes as shown in \autoref{fig:Results:Tet:Stoch} with a modulus range of $\SI{5}{\percent}$ around the nominal value. It can be noted that the location of failure shifts slightly away from the geometric feature bordering the two separate volumes in this region. In one case failure occurs slightly earlier as a result of the stochastic material distribution. Overall, due to the higher mesh entropy, the effect of stochastic material distribution in tet meshes is smaller than in hex meshes.

\pgfplotstableread[col sep=comma]{../../Material/Data/Numerics/Tet_0-5_0-5625_Stoch.csv}{\loadedtable}

\begin{figure}[htbp]
  \setlength{\figheight}{7cm}
  \begin{subfigure}{0.55\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
    \centering
%     \includegraphics[width=\linewidth,height=\figheight]{example-image-a}
    \tikzexternalenable
    \tikzsetnextfilename{Tet_0-4_1-2_Stoch}
    \begin{tikzpicture}
      \begin{axis}[
        height=\figheight+\baselineskip,
        width=\linewidth,
        axis lines=middle,
        cycle list name=color list,%linestyles*,
        cycle list shift=1,
        xmin=0,
        ymin=0,
        title=\empty,
        xlabel={Displacement $[\si{\milli\meter}]$},
        ylabel={Force $[\si{\newton}]$},
        x label style={at={(axis description cs:0.5,-0.075)},anchor=north},
        y label style={at={(axis description cs:-0.105,0.5)},rotate=90,anchor=south},
        legend pos=north west,
        legend cell align={left},
        legend style={font=\footnotesize},
      ]%   each nth point={2}
        \addplot+ [thick] table[x=DxNo, y=FxNo] {\loadedtable};
        \addlegendentry{No stochastics}
        \addplot+ [] table[x=DxSto1, y=FxSto1] {\loadedtable};
        \addlegendentry{Stochastic 1}
        \addplot+ [] table[x=DxSto2, y=FxSto2] {\loadedtable};
        \addlegendentry{Stochastic 2}
        \addplot+ [] table[x=DxSto3, y=FxSto3] {\loadedtable};
        \addlegendentry{Stochastic 3}
      \end{axis}
    \end{tikzpicture}
    \tikzexternaldisable
    \end{minipage}
    \caption{Force-displacement plot until failure}
    \label{fig:Results:Tet:Stoch:FD0-4_1-2}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.10\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
    \centering
      %\includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{../../Material/Figures/PD_Tet_Damage_0-5_0-5625_9600_-z_ct.png}
      \includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{PD_Tet_Damage_0-5_0-5625_9600_-z_ct}
    \end{minipage}
    \caption{No}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.10\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
      \centering
      %\includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{../../Material/Figures/PD_Tet_Stoch_1_Damage_0-5_0-5625_5645_-z_ct.png}
      \includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{PD_Tet_Stoch_1_Damage_0-5_0-5625_5645_-z_ct}
    \end{minipage}
    \caption{1}
  \end{subfigure}
  \hfill
  \begin{subfigure}{0.10\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
      \centering
      %\includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{../../Material/Figures/PD_Tet_Stoch_2_Damage_0-5_0-5625_3950_-z_ct.png}
      \includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{PD_Tet_Stoch_2_Damage_0-5_0-5625_3950_-z_ct}
    \end{minipage}
    \caption{2}
  \end{subfigure}
  \begin{subfigure}{0.10\linewidth}
    \begin{minipage}[b][\figheight]{\linewidth}
      \centering
      %\includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{../../Material/Figures/PD_Tet_Stoch_3_Damage_0-5_0-5625_3950_-z_ct.png}
      \includegraphics[angle=90,width=\linewidth,height=\figheight,keepaspectratio]{PD_Tet_Stoch_3_Damage_0-5_0-5625_3950_-z_ct}
    \end{minipage}
    \caption{3}
  \end{subfigure}
  \caption{Failure for hex-mesh with $dx=\SI{0.4}{\milli\meter}$ and stochastics}
  \label{fig:Results:Tet:Stoch}
\end{figure}


\subsection{Basis of an endomorphism}


\appendix
\section{Computation}
\label{s:appendix}

In this appendix we derive an expression for
the nonlocal resistance for finite contact resistance.
We first present the key definitions and critical boundary conditions.
We then derive the relation between the nonlocal resistance
and the spin chemical potential at the far contact, $μ_s^N (L)$.
Finally, we solve the diffusion equation inside the semiconductor to find $μ_s^N (L)$.

\subsection{Definitions}

Many of the definitions and results in this section are taken from
\cite{ActaPhysicaSlovaca.57.4_5.565-907}.
The chemical potential and spin chemical potential are defined in terms
of the spin-up and spin-down chemical potentials,
\begin{subequations}
  \label{eq:potentials}
  \begin{alignat}{2}
    & μ   && = \frac{1}{2} \left( μ_↑ + μ_↓ \right) , \\
    & μ_s && = \frac{1}{2} \left( μ_↑ - μ_↓ \right) .
  \end{alignat}
\end{subequations}
The material conductances and polarization are defined in terms
of the spin-up and spin-down conductances,
\begin{subequations}
  \label{eq:conductances}
  \begin{alignat}{2}
    & σ   && = σ_↑ + σ_↓ , \\
    & σ_s && = σ_↑ - σ_↓ , \\
    %
    \label{eq:material.polarization}
    & P_σ && = \frac{σ_s}{σ} .
  \end{alignat}
\end{subequations}
The gradient of the chemical potentials drives a current and spin current,
\begin{subequations}
  \label{eq:currents}
  \begin{alignat}{3}
    & J_{↑↓} && = σ_{↑↓} ∇μ_{↑↓} , \\
    %
    \label{eq:currents.current}
    & J      && = J_↑ + J_↓ & = σ   ∇μ + σ_s ∇μ_s , \\
    %
    \label{eq:currents.spincurrent}
    & J_s    && = J_↑ - J_↓ & = σ_s ∇μ + σ   ∇μ_s .
  \end{alignat}
\end{subequations}
To indicate the material, any of the above can have a superscript
$N$ (normal semiconductor) or $F$ (ferromagnet).

The contact conductances and polarization are defined in terms
of the spin-up and spin-down contact conductances,
\begin{subequations}
  \label{eq:contact_conductances}
  \begin{alignat}{2}
    & Σ   && = Σ_↑ + Σ_↓ , \\
    & Σ_s && = Σ_↑ - Σ_↓ , \\
    %
    \label{eq:contact.polarization}
    & P_Σ && = \frac{Σ_s}{Σ} .
  \end{alignat}
\end{subequations}
The mismatch of the chemical potentials across the contact
drives a current and spin current,
\begin{subequations}
  \label{eq:contact_currents}
  \begin{alignat}{2}
    & J_{↑↓}^C && = Σ_{↑↓} \left( μ^N_{↑↓} - μ^F_{↑↓} \right)_c , \\
    %
    & J^C      && = J_↑^C + J_↓^C , \\
    & J_s^C    && = J_↑^C - J_↓^C .
  \end{alignat}
\end{subequations}
The subscript $c$ will always denote the function evaluated at the contact.

We will use the term current to refer to $J$,
when in fact this is a particle current density.
For constant $J$, the physical charge current $I$ will be related to $J$
by a relation $I = - A J / e$ for some characteristic area $A$.

To reduce the number of subscripts and superscripts in the following,
we adopt the notation for the potentials
\begin{subequations}
  \begin{equation}
    \begin{aligned}
    & \begin{alignedat}{2}
        & u && = μ^N_s , \\
        & v && = μ^N   ,
      \end{alignedat}
    & \begin{alignedat}{2}
        & φ && = μ^F_s , \\
        & ψ && = μ^F   ,
      \end{alignedat}
    \end{aligned}
  \end{equation}
  and currents
  \begin{equation}
    \begin{alignedat}{2}
      & ȷ   && = J_s   , \\
      & J_c && = J^C   , \\
      & ȷ_c && = J_s^C .
    \end{alignedat}
  \end{equation}
\end{subequations}

We rewrite \cref{eq:contact_currents} as
\begin{subequations}
  \label{eq:contact_currents.2}
  \begin{alignat}{3}
    \label{eq:contact_currents.2.current}
    & J_c && = Σ   \left( v_c - ψ_c \right) && + Σ_s \left( u_c - φ_c \right) , \\
    %
    \label{eq:contact_currents.2.spincurrent}
    & ȷ_c && = Σ_s \left( v_c - ψ_c \right) && + Σ   \left( u_c - φ_c \right) ,
  \end{alignat}
\end{subequations}
and \cref{eq:conductances,eq:currents} as
\begin{equation}
  \label{eq:bdry_current}
  ȷ = P_σ J + 4 \frac{σ_↑ σ_↓}{σ} ∇μ_s .
\end{equation}

Using \cref{eq:contact_conductances,eq:contact_currents.2},
\begin{equation}
  \label{eq:bdry_current_contact}
  ȷ_c = P_Σ^i J_c + {R_C^i}^{-1} \left( u_c - φ_c \right) ,
\end{equation}
where the contact resistance is
\begin{equation}
  \label{eq:contact.resistance}
  R_C^i = \frac{Σ^i}{4 Σ_↑^i Σ_↓^i} .
\end{equation}
The superscript $i$ allows for contacts with difference conductances.

\subsection{Boundary conditions}

In this sections, we derive the relations between the potentials and the currents
This corresponds to the needed boundary conditions.

\subsubsection{Semiconductor}

For the semiconductor, $σ^N_↑ = σ^N_↓ = σ^N / 2$, so $P_σ^N = 0$.
Evaluating \cref{eq:bdry_current} at the contact gives
\begin{equation}
  \label{eq:bdry_current.semiconductor}
  ȷ^N_c = σ^N ( ∇u )_c .
\end{equation}

\subsubsection{Ferromagnet}

For the ferromagnet, one assumes $μ_s^F$ satisfies
the one dimensional diffusion equation.
We choose the $z'$ coordinate antiparallel to $z$ with origin at the contact.
The equation
\begin{equation}
  \label{eq:diffusion.ferromagnet}
  φ'' \left( z' \right) - k_F^2 φ \left( z' \right) = 0 ,
\end{equation}
with the boundary condition
$\lim_{z' → - ∞} φ(z') = 0$
has solution
\begin{equation}
  \label{eq:diffusion.ferromagnet.solution}
  φ(z') = φ_c e^{k_F z'} ,
\end{equation}
where $φ_c = φ(0)$ is a yet undetermined constant.
Putting this into \cref{eq:bdry_current} and evaluating it at the contact gives
\begin{equation}
  \label{eq:bdry_current.ferromagnet}
  ȷ^F_c = P_σ^F J^F_c + R_F^{-1} φ_c ,
\end{equation}
where the ferromagnet resistance is
\begin{equation}
  \label{eq:ferromagnet.resistance}
  R_F = \frac{σ^F}{ 4 σ_↑^F σ_↓^F k_F } .
\end{equation}
Here, $λ_F = 1 / k_F$ is the spin diffusion length in the ferromagnet.

\subsubsection{Continuity assumptions}

At the contact, the current and spin current are assumed continuous,
\begin{subequations}
  \label{eq:continuity.current}
  \begin{alignat}{3}
    & J_c && = J^F_c && = J^N_c , \\
    & ȷ_c && = ȷ^F_c && = ȷ^N_c .
  \end{alignat}
\end{subequations}
Using \cref{eq:bdry_current.ferromagnet,eq:bdry_current_contact,eq:continuity.current}
we find the relation
\begin{subequations}
  \label{eq:bdry_solutions}
  \begin{equation}
    \label{eq:bdry_solutions.current}
    \left( P_σ^F R_F + P_Σ^i R_C^i \right) J_c = \left( R_F + R_C^i \right) ȷ_c - u_c ,
  \end{equation}
  and that $φ_c$ is determined by
  \begin{equation}
    \label{eq:bdry_solutions.potential}
    R_F^{-1} φ_c = \frac{ \left( P_Σ^i - P_σ^F \right) R_C^i ȷ_c + P_σ^F u_c}{P_σ^F R_F + P_Σ^i R_C^i} .
  \end{equation}
\end{subequations}
In the special case of zero current at the contact ($J_c = 0$),
\cref{eq:bdry_solutions} reduces to
\begin{subequations}
  \label{eq:bdry_solutions.zero}
  \begin{alignat}{2}
    \label{eq:bdry_solutions.zero.current}
    & ȷ_c && = \frac{1}{R_F + R_C^i} u_c   , \\
    %
    \label{eq:bdry_solutions.zero.potential}
    & φ_c && = \frac{R_F}{R_F + R_C^i} u_c .
  \end{alignat}
\end{subequations}

\subsection{Nonlocal resistance}
\label{s:appendix:nonlocal_resistance}

In this section we derive the precise relation between $\rNL$ and $μ_s^N (L)$.
Note that we may write in general, for some $\bar{μ}$,
\begin{equation}
  μ = \bar{μ} + P_σ μ_s ,
\end{equation}
and, following
\cite{PhysRevB.67.052409},
define the voltage due to the difference in the chemical potentials across the contacts by
\begin{equation}
  V_c = \left( \bar{μ}_c^N - \bar{μ}_c^F \right) / e.
\end{equation}

We assume a fixed current $J_0 = \abs{J_0} > 0$
flows down through the contact at $x = 0$
and to the left in the semiconductor for $x ≤ 0$,
and no current flows for $x > 0$.
The experimentally measured quantity is the
nonlocal resistance $\rNL = V_L / I_0$,
where $I_0 = - W L J_0 / e$ is the current through the contact at $x = 0$.
It is convenient to introduce the effective nonlocal resistance $\rNLeff$
defined by
\begin{equation}
  \rNLeff = W L \rNL = - e V_L / J_0 = \frac{\bar{μ}_c^F - \bar{μ}_c^N}{J_0} .
\end{equation}
To determine $\rNL$, we must express the difference
of these chemical potentials in terms of $μ_s^N (L)$.

Since there are two ferromagnetic contacts,
we have separate functions $ψ$ and $φ$ for each contact
which we will denote by $ψ^0$, $φ^0$, and $ψ^L$, $φ^L$.
From \cref{eq:diffusion.ferromagnet.solution}, we have
\begin{subequations}
  \begin{align}
    φ^0 \left( z' \right) & = φ_0 e^{k_F z'} , \\
    φ^L \left( z' \right) & = φ_L e^{k_F z'} .
  \end{align}
\end{subequations}

The physical restriction on the current flow in the semiconductor
is imposed by noting that since $σ_s^N = 0$,
\cref{eq:currents.current} gives $J^N = σ^N ∇v$, so we must have
\begin{equation}
  v_x (x) =
    \begin{cases}
      v_x (0) - \left( J_0 / σ^N \right) x & \text{ for } x ≤ 0 , \\
      v_x (0)                              & \text{ for } x > 0 ,
    \end{cases}
\end{equation}
$v_y (x) = v_y (0)$, and $v_z (x) = v_z (0)$.

Using \cref{eq:currents.current},
the restriction on the current flow in each ferromagnet gives
\begin{subequations}
  \begin{align}
    ∇ψ^0 & = \left( J_0 / σ^F \right) - P_σ^F ∇φ^0 , \\
    ∇ψ^L & = - P_σ^F ∇φ^L .
  \end{align}
\end{subequations}
Integrating and enforcing
$e V_c = v_x (0) - \left( ψ_c - P_σ^F φ_c \right)$,
\begin{subequations}
  \begin{align}
    & \begin{aligned}
        ψ^0 \left( z' \right) & = - e V_0 + P_σ^F φ_0 \left( 2 - e^{k_F z'} \right) \\
                              & \qquad + v_x (0) + \left( J_0 / σ^F \right) z' ,
      \end{aligned} \\
    & \begin{aligned}
        ψ^L \left( z' \right) & = - e V_L + P_σ^F φ_L \left( 2 - e^{k_F z'} \right) \\
                              & \qquad + v_x (0) .
      \end{aligned}
  \end{align}
\end{subequations}

There is no current at the contact at $x = 0$,
thus \cref{eq:contact_currents.2.current} gives
\begin{equation}
  ψ_L - v_L = P_Σ^L \left( u_L - φ_L \right) ,
\end{equation}
and with
\cref{eq:bdry_solutions.zero.potential},
we find
\begin{equation}
  \label{eq:rnl.full}
  \begin{aligned}
    \rNLeff & = \left( ψ_L - v_L \right) - P_σ^F φ_L \\
            & = \left[ P_Σ^L \left( 1 - \frac{R_F}{R_F + R_C^L} \right) - \frac{P_σ^F R_F}{R_F + R_C^L} \right] \frac{u_x (L)}{J_0} .
  \end{aligned}
\end{equation}

\subsection{Diffusion equation}

In this section we show how to solve for $μ_s^N (L)$.
This method is based on the one described in
\cite{PhysRevB.80.214427}.
Inside the semiconductor, $u$ satisfies the diffusion equation
\begin{equation}
  \label{eq:diffusion}
  D ∇^2 u - \frac{u}{τ} + ω × u = 0 .
\end{equation}
Here, $D$ is the diffusion constant, $τ$ the spin lifetime,
and $ω = \left( g μ_B / ℏ \right) B$ is proportional to the applied magnetic field
(with $g$ the gyromagnetic ratio and $μ_B$ the Bohr magneton).
The spin diffusion length in the semiconductor is $λ = 1 / k = \sqrt{D τ}$.

The function $u = u(x)$ only varies along $x$,
and we introduce the notation
\begin{equation}
  u_x (x) =
    \begin{cases}
      u_{x-} (x) \text{ for } x < 0     , \\
      u_{x0} (x) \text{ for } 0 ≤ x ≤ L , \\
      u_{x+} (x) \text{ for }     L < x ,
    \end{cases}
\end{equation}
with similar expressions for $u_y$ and $u_z$.
The most general solution to \cref{eq:diffusion} decouples $u_z$ from $u_x$ and $u_y$.
The requirement $\lim_{x → ± ∞} u(x) = 0$ yields
\begin{subequations}
  \begin{alignat}{3}
    & u_{z±} (x) && {}={} && A^∓ e^{∓ k x}                     , \\
    & u_{z0} (x) && {}={} && A_0^+ e^{k x} {}+{} A_0^- e^{-k x} ,
  \end{alignat}
\end{subequations}
and
\begin{subequations}
  \begin{alignat}{6}
    & u_{x±} (x) && =   && B^∓ e^{∓ κ x} && {}+{} &&   && C^∓ e^{∓ \bar{κ} x} , \\
    & u_{y±} (x) && = i && B^∓ e^{∓ κ x} && {}-{} && i && C^∓ e^{∓ \bar{κ} x} ,
  \end{alignat}
  \begin{alignat}{12}
    & u_{x0} (x) && =   && B_0^+ e^{κ x} && {}+{} &&   && B_0^- e^{- κ x} && {}+{} &&   && C_0^+ e^{ \bar{κ} x } && {}+{} &&   && C_0^- e^{ - \bar{κ} x } , \\
    & u_{y0} (x) && = i && B_0^+ e^{κ x} && {}+{} && i && B_0^- e^{- κ x} && {}-{} && i && C_0^+ e^{ \bar{κ} x } && {}-{} && i && C_0^- e^{ - \bar{κ} x } ,
  \end{alignat}
\end{subequations}
where $κ = k \sqrt{1 + i ω τ}$.
The twelve constants $A$, $B$ and $C$ (with their various subscripts and superscripts) must be determined by imposing the appropriate boundary conditions.

We first require $u$ be continuous at $x = 0$ and $x = L$; this gives six equations.
We now require a boundary condition on $∇u$, but $∇u$ cannot be assumed continuous at the contact.
We make the assumption that the total spin current at the contact
is the sum of the spin currents on either side, i.e.,
\begin{subequations}
  \label{eq:current.sum}
  \begin{alignat}{2}
    & ȷ_0 && = σ^N \left[ - u_-'(0) + u_0'(0) \right] , \\
    & ȷ_L && = σ^N \left[ - u_0'(L) + u_+'(L) \right] .
  \end{alignat}
\end{subequations}
The signs have been chosen to be consistent with the physical geometry.
The only nonzero component of the current at the contacts inside the semiconductor
is the $x$ component at $x = 0$, so we use \cref{eq:bdry_solutions.current}.
For all other components there is zero current at the contact, and we use
\cref{eq:bdry_solutions.zero.current}.
Together with \cref{eq:current.sum}, this gives the other six equations,
\begin{subequations}
  \begin{alignat}{8}
    & - & {} & u_{z-}'(0) & {}+{} & u_{z0}'(0) & {}+{} & η_0 u_z(0) && = 0 , \\
    &   & {} & u_{z+}'(L) & {}-{} & u_{z0}'(L) & {}+{} & η_L u_z(L) && = 0 , \\
    & - & {} & u_{x-}'(0) & {}+{} & u_{x0}'(0) & {}+{} & η_0 u_x(0) && = Δ , \\
    &   & {} & u_{x+}'(L) & {}-{} & u_{x0}'(L) & {}+{} & η_L u_x(L) && = 0 , \\
    & - & {} & u_{y-}'(0) & {}+{} & u_{y0}'(0) & {}+{} & η_0 u_y(0) && = 0 , \\
    &   & {} & u_{y+}'(L) & {}-{} & u_{y0}'(L) & {}+{} & η_L u_y(L) && = 0 ,
  \end{alignat}
\end{subequations}
where
\begin{subequations}
  \begin{alignat}{2}
  & η_i^{-1} && = - σ^N \left( R_F + R_C^i \right) , \\
  & Δ && = - (- J_0) \left( P_σ^F R_F + P_Σ^0 R_C^0 \right) η_0 .
  \end{alignat}
\end{subequations}
We define the $r$-parameter, $r_i = - η_i^{-1}$, introduced in \cref{eq:r-parameter}.

These equations can be organized into a matrix equation and solved algebraically.
A solution for $u_z$ corresponds to a condition of vanishing determinant,
\begin{equation}
  e^{-2 L / λ} = \left( 1 + \frac{2 r_0}{λ} \right) \left( 1 + \frac{2 r_L}{λ} \right) ,
\end{equation}
which can never be satisfied
\footnote{
  Except at the nonphysical point $L / λ = r_i / λ = 0$.
},
thus $u_z = 0$ is the only allowed solution.
The other two components form an eight dimensional linear system.
Solving this gives the remaining constants, and thus
$u_x (L) = e^{- κ L} B^- + e^{- \bar{κ} L} C^-$.

Finally, by using $p_1 = - σ^N Δ / J_0$ along with \cref{eq:rnl.full},
we can introduce $\rSQ$ from \cref{eq:square_resistance}
and the polarizations
\begin{subequations}
  \label{eq:polarizations}
  \begin{align}
    p_1 & = \frac{P_σ^F R_F + P_Σ^L R_C^L}{R_F + R_C^L} , \\
    p_2 / p_1 & = \left. \left( 1 - \frac{P_σ^F R_F}{P_Σ^L R_C^L} \right) \middle/
                  \left(1 + \frac{P_σ^F R_F}{P_Σ^L R_C^L} \right) \right. ,
  \end{align}
\end{subequations}
to write
\begin{equation}
  \frac{\rNLeff}{\rSQ} = \frac{p_1 p_2}{W / λ} \left[ - \frac{k u_x (L)}{Δ} \right] .
\end{equation}
The factor in brackets is the function $f$ given in \cref{eq:f}.


\documentclass[a4paper,11pt,final]{article}
\usepackage[american]{babel}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage[svgnames]{xcolor}
\usepackage{amsmath}
\usepackage[utf8]{inputenc}
\usepackage[pdfborder={0 0 0}]{hyperref}
\usepackage{mathptmx}
\hypersetup{
    pdftitle={CamSim Physically-Based Rendering Model},
    pdfauthor={Martin Lambers}
}

\begin{document}

\section*{CamSim illumination model}

This describes the illumination model (direct and single-bounce indirect) used
in CamSim.

It is based on this paper:
  M. Lambers, S. Hoberg, A. Kolb:
  Simulation of Time-of-Flight Sensors for Evaluation of Chip Layout Variants.
  In IEEE Sensors Journal, 15(7), 2015, pages 4019-4026.

The extension for single-bounce indirect illumination is loosely based on this paper:
  D. Bulczak, M. Lambers, A. Kolb:
  Quantified, Interactive Simulation of AMCW ToF Camera Including Multipath
  Effects.
  In MDPI Sensors, 18(1), 2018, pages 1424-8220.

\section{Symbols}

\subsection{Radiometric}

$Q$: Radiant energy $[J]$\\
$P$: Radiant flux (``power'') $[W]$\\
$I$: Radiant intensity $[W/\mathrm{sr}]$\\
$L$: Radiance (``radiant flux per unit solid angle per unit projected area'') $[W/\mathrm{sr}/m^2]$\\
$E$: Irradiance $[W/m^2]$

\subsection{Geometric}

$L$: Light source position\\
$C$: Camera center position\\
$P$: Point on object surface seen by camera pixel\\
$\widehat{n_P}$: Surface normal at $P$\\
$\theta_{P\rightarrow L}$: Angle between $\widehat{n_P}$ and $\widehat{L-P}$\\
$Q$: Another point (on another surface) that may act as a virtual point light\\
$\widehat{n_Q}$: Surface normal at $Q$\\
$\theta_{Q\rightarrow L}$: Angle between $\widehat{n_Q}$ and $\widehat{L-Q}$

\subsection{Other}

$f_P(\widehat{L-P}, \widehat{n_P}, \widehat{C-P})$: BRDF for $L\rightarrow P\rightarrow C$\\
$f_Q(\widehat{L-Q}, \widehat{n_Q}, \widehat{P-Q})$: BRDF for $L\rightarrow Q\rightarrow P$\\

\section{Direct Illumination}

$I_{L\rightarrow P} = \frac{P_L}{4\pi}$ (if $L$ is point light; there are alternatives)\\
$L_{L\rightarrow P} = \frac{I_{L\rightarrow P}}{d^2_{L\rightarrow P}}$\\
$L_{P\rightarrow C} = f_P(\widehat{L-P}, \widehat{n_P}, \widehat{C-P}) L_{L\rightarrow P} \cos(\theta_{P\rightarrow L})$\\
$E_C = L_{P\rightarrow C}$ ~~~(Note: no factor $\cos(\theta_{C\rightarrow P})$ here)\\
$P_C = E_C \cdot \mathrm{SensorPixelArea}$\\
$Q_C = P_C \cdot \mathrm{SignalDutyCycle} \cdot \mathrm{ExposureTime}$

\section{Indirect Illumination via Virtual Point Lights}

Start with the Rendering Equation:\\
$\displaystyle L_{P\rightarrow C} = \int_\Omega f_P(\omega_i, \widehat{n_P}, \widehat{C-P}) L_{i\rightarrow P} \cos(\theta_i) d\omega_i$

\noindent
Approximate this by splitting into direct and indirect parts, and approximating
the indirect part with single-bounce RSM VPLs:\\
$\displaystyle L_{P\rightarrow C} = f_P(\widehat{L-P}, \widehat{n_P}, \widehat{C-P}) L_{L\rightarrow P} \cos(\theta_{P\rightarrow L})
 + \frac{1}{|\mathrm{RSM}|}\sum_{Q\in \mathrm{RSM}} f_P(\widehat{Q-P}, \widehat{n_P}, \widehat{C-P}) L_{Q\rightarrow P} \cos(\theta_{P\rightarrow Q})$

\noindent
The direct part is already computed in the direct step as $L_{P\rightarrow C}$. Add the indirect part to it:\\
$\displaystyle L^*_{P\rightarrow C} = L_{P\rightarrow C}
 + \frac{1}{|\mathrm{RSM}|}\sum_{Q\in \mathrm{RSM}} f_P(\widehat{Q-P}, \widehat{n_P}, \widehat{C-P}) L_{Q\rightarrow P} \cos(\theta_{P\rightarrow Q})$

\noindent
using
$L_{Q\rightarrow P} = f_Q(\widehat{L-Q}, \widehat{n_Q}, \widehat{P-Q}) L_{L\rightarrow Q} \cos(\theta_{Q\rightarrow L})$\\
and
$L_{L\rightarrow Q} = \frac{I_{L\rightarrow Q}}{d^2_{L\rightarrow Q}}, \quad I_{L\rightarrow Q} = \frac{P_L}{4\pi}$

\section{Values to Store in RSM}

\begin{itemize}
\item Parameters that allow BRDF sampling at $Q$, e.g. $k_d, k_s, s$ for modified Phong
\item Position of $Q$ (to perform shadow test and to compute $\widehat{Q-P}$,
$\theta_{P\rightarrow Q}$, $\widehat{L-Q}$, $\widehat{P-Q}$,$ \theta_{Q\rightarrow L}$)
\item $\widehat{n_Q}$
\item $L_{L\rightarrow Q}$
\end{itemize}

%\section{Notes}
%
%\begin{itemize}
%\item Visibility tests:
%    \begin{itemize}
%    \item $P\rightarrow C$: rasterization does this for us
%    \item $L\rightarrow P$: per shadow map test
%    \item $L\rightarrow Q$: rasterization of the RSM ensures that only lighted points are in the RSM; this visibility test is implicit
%    \item $Q\rightarrow P$: not included in original RSM approach, BUT:
%        \begin{itemize}
%        \item Project $P$ into RSM of $Q$
%        \item Cast ray from $Q$ to $P$ and traverse it \textbf{in Q's RSM space}
%        \item Find obstacles based on depth values
%        \end{itemize}
%    \end{itemize}
%\end{itemize}

\end{document}


\chapter{Preprocessing of the data}
\label{ch:prepro}

Before the analysis steps done in this work can be performed, the raw data taken by the LST-1 has to be processed up until what is called Data Level 1 (DL1).
For the data used in this work such preprocessing was done using \texttt{lstchain}, a low-level data processing pipeline for the LST telescopes.
\texttt{lstchain} is an open-source project which is still in development on github \cite{lstchain} and is based on \texttt{ctapipe}.
\texttt{ctapipe} will be the main low-level data processing pipeline for CTA and is itself in development as an open-source project.
This work uses the versions \texttt{v0.5.1} and \texttt{v0.5.2} of \texttt{lstchain} which are both based on version \texttt{v0.7.0} of \texttt{ctapipe} \cite{ctapipe}.

In the first step the waveforms recorded by each pixel of the camera are cleaned to reduce the influence of electronic noise in the signal.
After that the waveforms are integrated to obtain the total photon charge for each pixel and a mean arrival time for the photons is defined for each pixel
as the point in time at which half of all the photons within the pixel have arrived.
This results in two 2-dimensional images for each event recorded by the camera, as can be seen in \autoref{fig:eas_images}.
\begin{figure}
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[width=0.8\textwidth]{images/eas_image1.png}
        \caption{Number of photons per pixel.}
        \label{fig:eas_image1}
    \end{subfigure}
    \hfill
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[width=0.8\textwidth]{images/eas_image2.png}
        \caption{Arrival time per pixel relative to the mean arrival time for the whole image.}
        \label{fig:eas_image2}
    \end{subfigure}
    \caption{The integrated photon count in \subref{fig:eas_image1} shows the air shower clearly but a lot of noise is visible in the non-shower pixels.
        The arrival times in \subref{fig:eas_image2} show a clear correlation for the pixels that recorded the air shower \cite{lukas}.
    }
    \label{fig:eas_images}
\end{figure}

In the next step these images are cleaned by discarding pixels that did not record the air shower.
This is done in multiple steps using two thresholds for the photon charge of each pixel and one for the number of neighboring pixels.
First all pixels above the higher photon charge threshold with enough neighboring pixels above the lower photon charge threshold are selected as part of the shower.
After that all pixels above the lower photon charge threshold with enough neighboring pixels above the higher photon charge threshold are selected as well \cite{lukas}.
For rather soft thresholds the results of this cleaning can be seen in \autoref{fig:eas_images_cleaned}.
\begin{figure}
    \centering
    \includegraphics[width=0.8\textwidth]{images/eas_images_cleaned.png}
    \caption{The pixels discarded during the image cleaning are marked as grey. The soft thresholds chosen for this example result in two smaller pixel-islands remaining \cite{lukas}.}
    \label{fig:eas_images_cleaned}
\end{figure}

Based on these cleaned images, a multitude of image parameters can be calculated.
The most important ones are the total number of photons in the shower, called \texttt{intensity}, and the so called Hillas parameters which are illustrated in \autoref{fig:image_parameters}.
The Hillas parameters are based on a principal component analysis (PCA) of the photon charge distribution and describe the extension and orientation of the shower.
The parameters \texttt{length} and \texttt{width} correspond with the standard deviations along the principal components of the distribution and are therefore a
description of the shower extension.
The orientation on the other hand is described by the angle of the main shower axis relative to the $x$-axis of the image.
The performance of the algorithms used in this work in regards to reconstructing this angle can be seen in \autoref{fig:delta_comparison}.
\begin{figure}
    \centering
    \includegraphics[width=0.5\textwidth]{images/image_parameters.png}
    \caption{The PCA-based Hillas parameters calculated for the cleaned image of the integrated photon count.
        $\psi$ describes the angle of the main shower axis relative to the x-axis of the image and $x$ and $y$ are the coordinates of the center of gravity of the shower.
        \texttt{length} and \texttt{width} can be depicted as the semi-major and semi-minor axis of an ellipse \cite{lukas}.
    }
    \label{fig:image_parameters}
\end{figure}

Higher moments of the distribution like \texttt{skewness} and \texttt{kurtosis} are also calculated.
The center of gravity of the shower image is calculated by weighting the coordinates of each pixel with the photon charge of this pixel.
Additional image featueres include multiple \texttt{leakage} parameters that describe how much of the light was recorded at the edge of the camera and
thus indicate how much light from the shower might have missed the camera.

The image of the arrival times allows for timing parameters to be calculated by fitting a linear function along the main shower axis.
This yields the parameters \texttt{time\_gradient} and \texttt{intercept} which help to determine the direction of the shower, together with \texttt{skewness}.


\SecDef{intro}{Introduction}

Lightweight cryptography is a modern direction in the design of symmetric-key primitives. It aims to provide cryptographic security with constrained resources. Lightweight ciphers usually have a low security margin against unknown attacks and rely on the cryptanalysis done in the design phase.

My colleagues developed a framework for benchmarking lightweight ciphers, called FELICS~\cite{Felics,Felics1,Felics2}. A large amount of implementations for 3 target platforms - AVR, MSP, ARM - was collected and benchmarked. The leading block ciphers were Chaskey~\cite{Chaskey}, Simon and Speck~\cite{Simon}, RECTANGLE~\cite{RECTANGLE}, LEA\cite{LEA}, HIGHT~\cite{HIGHT}, AES~\cite{AES}. Chaskey is an Even-Mansour block cipher and has a data-security trade-off; it does not have a security proof against linear/differential attacks. Simon and Speck were designed by the NSA and do not have a proof too.

As the top designs are ARX-based, i.e. they are composed from Addition, Rotation and XOR operations, we decided to design an ARX-based block cipher. However, the current \emph{wide-trail strategy} for proving security against linear/differential cryptanalysis does not apply well to ARX-based block ciphers. For this reason, we developed a novel \emph{long-trail strategy}. As a result, the block cipher \sparx{} is the first ARX-based block cipher with provable security against single-trail linear and differential cryptanalysis.

In this chapter, I describe briefly the long-trail strategy.
Afterward, I describe my contributions to the design. I developed an algorithm for efficient long-trail evaluation of a large class of SPN structures. We used this algorithm and the division property~\cite{division} to evaluate a large class of potential linear layers. Interestingly, a Feistel-like linear layer turned out to provide an optimal balance between the linear/differential and integral attacks resistance, lightweightness of the primitive and simplicity. A few alternative linear layers seem to be a good choice as well.

\subsection{Outline}
I describe briefly the long-trail strategy and my algorithms in \SecRef{longtrail}. In \SecRef{linear}, I describe the procedure that we used to choose an optimal linear layer for the cipher.
I omit the specification of \sparx{}, because it is not required for the contents of this chapter; it can be found in~\cite{OurSPARX}.


\documentclass[review]{elsarticle}

\usepackage{amsmath}
\usepackage{lineno,hyperref}
\modulolinenumbers[5]
\usepackage{caption}
\usepackage{subcaption}
\captionsetup{font=small}
\captionsetup[sub]{font=footnotesize}
\usepackage{nicefrac}
\usepackage{booktabs}
\usepackage[usenames]{xcolor}
\usepackage{units}
\usepackage[acronyms]{glossaries}
\usepackage[ruled,vlined]{algorithm2e}

\journal{Elsevier}

%% `Elsevier LaTeX' style
%\bibliographystyle{elsarticle-num}
\bibliographystyle{elsarticle-num-names}
\makeatletter
\providecommand{\doi}[1]{%
  \begingroup
  \let\bibinfo\@secondoftwo
  \urlstyle{rm}%
  \href{http://dx.doi.org/#1}{%
    doi:\discretionary{}{}{}%
    \nolinkurl{#1}%
  }%
  \endgroup
}
\makeatother

%=================================================================================
% Useful commands
\newcommand{\ud}{\,\mathrm{d}}
\newcommand{\pfrac}[2]{\frac{\partial#1}{\partial#2}}
\newcommand{\dpfrac}[2]{\dfrac{\partial#1}{\partial#2}}
\newcommand{\ufrac}[2]{\frac{\ud{}#1}{\ud{}#2}}
\newcommand{\dufrac}[2]{\dfrac{\ud{}#1}{\ud{}#2}}
\newcommand{\wt}[1]{\widetilde{#1}}
\newcommand{\ol}[1]{\overline{#1}}

%=================================================================================
% Abbreviations
\newacronym{dns}{DNS}{direct numerical simulations}
\newacronym{les}{LES}{large eddy simulations}
\newacronym{pdf}{PDF}{probability density function}
\newacronym{pmf}{PMF}{probability mass function}
\newacronym{ml}{ML}{machine learning}
\newacronym{dnn}{DNN}{deep neural network}
\newacronym{dof}{DoFs}{degrees of freedom}
\newacronym{rmse}{RMSE}{root mean square error}
\newacronym{sgs}{SGS}{subgrid-scale}
\newacronym{wale}{WALE}{Wall Adapting Local Eddy-Viscosity}
\newacronym{sgd}{SGD}{stochastic gradient descent}
\makeglossaries
\glsdisablehyper

%=================================================================================
% Document
\begin{document}

\begin{frontmatter}
\title{A block-random algorithm for learning on distributed, heterogeneous data}
\author[ut]{Prakash Mohan}
\ead{prak@utexas.edu}
\author[nrel_hpacf]{Marc T. Henry de Frahan}
\ead{Marc.HenrydeFrahan@nrel.gov}
\author[nrel_cssog]{Ryan King}
\ead{Ryan.King@nrel.gov}
\author[nrel_hpacf]{Ray W. Grout}
\ead{Ray.Grout@nrel.gov}
\address[ut]{Institute for Computational Engineering and Sciences, The University of Texas at Austin, 201 E. 24th Street, POB 4.102, Austin, Texas 78712, USA}
% \address[nrel]{Computational Science Center, National Renewable Energy Laboratory, 15013 Denver W Pkwy, ESIF301, Golden, CO 80401, USA}
\address[nrel_hpacf]{High Performance Algorithms and Complex Fluids, Computational Science Center, National Renewable Energy Laboratory, 15013 Denver W Pkwy, ESIF301, Golden, CO 80401, USA}
\address[nrel_cssog]{Complex Systems Simulation and Optimization Group, Computational Science Center, National Renewable Energy Laboratory, 15013 Denver W Pkwy, ESIF301, Golden, CO 80401, USA}

\begin{abstract}

  Most deep learning models are based on deep neural networks with
  multiple layers between input and output. The parameters defining
  these layers are initialized using random values and are ``learned''
  from data, typically using stochastic gradient descent based
  algorithms. These algorithms rely on data being randomly shuffled
  before optimization. The randomization of the data prior to
  processing in batches that is formally required for stochastic
  gradient descent algorithm to effectively derive a useful deep
  learning model is expected to be prohibitively expensive for in situ
  model training because of the resulting data communications across
  the processor nodes. We show that the \gls{sgd} algorithm can still
  make useful progress if the batches are defined on a per-processor
  basis and processed in random order even though (i) the batches are
  constructed from data samples from a single class or specific flow
  region, and (ii) the overall data samples are heterogeneous. We
  present block-random gradient descent, a new algorithm that works on
  distributed, heterogeneous data without having to pre-shuffle. This
  algorithm enables in situ learning for exascale simulations. The
  performance of this algorithm is demonstrated on a set of benchmark
  classification models and the construction of a subgrid scale
  \gls{les} model for turbulent channel flow using a data model
  similar to that which will be encountered in exascale simulation.

\end{abstract}

\begin{keyword}
  stochastic gradient descent \sep distributed \sep block-random \sep channel flow
\end{keyword}

\end{frontmatter}

\glsresetall

\section{Introduction}\label{sec:intro}

Simulating complex physics problems while resolving all the relevant
length scales is computationally expensive, requiring millions of core
hours to compute a single realization. Combining \gls{dns} with an
optimization or design cycle is infeasible, creating a need for
reduced-order models. Deep learning is an increasingly popular and
effective modeling technique that use many data to train a neural
network for a variety of tasks~\cite{Lecun2015, Schmidhuber2015,
  Prieto2016, Goodfellow2016, Liu2017}. These tasks range from visual
object recognition and speech recognition to analyzing particle
accelerator data and drug design. Recently, deep learning has been
explored as a tool for creating reduced-order closure models in
turbulent fluid flows~\cite{ling2016reynolds, duraisamy2015new,
  duraisamy2019turbulence}.

For physics simulation, the advent of exascale computing will enable
unprecedentedly high-fidelity simulations. The expectation is to
derive reduced-order models for engineering and design applications
from the many data generated by these simulations. Because it will be
increasingly difficult to save the large amounts of data generated
during the simulations for offline training, this will drive the need
to change existing approaches for training deep learning
models. Online or in situ training, where the model is trained during
the simulation to avoid data storage, has the potential to alleviate
this problem. A data parallel~\cite{zinkevich2010parallelized}
paradigm for deep learning is a practical approach for online
training. In this setting, there are two distinct computational
clusters: one for the physics computations and the other for deep
learning. Data will be transferred from the physics cluster to the
deep learning cluster as needed by the learning algorithms.

Most deep learning models use artificial neural networks with multiple
layers to capture nonlinearities. The parameters defining these layers
are ``learned'' from data, typically using algorithms that approximate
gradient descent. With the increase in the amount of available data,
deterministic learning algorithms are often expensive and rarely used
in practice.  \Gls{sgd}~\cite{bottou2010large, Goodfellow2016} and
variants using ``mini-batches'' are commonly used algorithms for practical
learning problems. The stochastic algorithms require the data to be
randomly shuffled~\cite{bottou2010large, Goodfellow2016} for
optimization; however, because fully shuffling the data will be
infeasible for exascale simulations because of the communication costs
of moving the data between processor nodes, the data shuffling
strategy necessary for \gls{sgd} will need to adapt to ensure the
adequate representation of the vastly differing physical processes
occurring in the simulation domain. Shuffling data extracted from a
single computational node will not provide sufficient randomness
because correlations tend to be spatially localized. The randomization
of the data prior to processing in batches that is formally required
for \gls{sgd} to make progress is expected to be prohibitively
expensive for in situ model training. We illustrate the memory
patterns in Figure\,\ref{fig:access_patterns} for the simulation of a
passively advected scalar using adaptive mesh refinement (setup
defined in the AMReX
tutorial\footnote{\url{https://github.com/AMReX-Codes/amrex}}). The
blocks of data are distributed among the different processors and are
heterogeneous, with the mesh adaptivity refining areas of
interest. The memory access pattern for fully shuffling the data for
\gls{sgd} is shown in Figure\,\ref{fig:shuffled_access}. This type of
memory access is detrimental to the simulation performance because the
memory access is uncoalesced, disregards data locality, and requires
many global communications. A single global communication to transfer a large
contiguous chunk of data from one processor is more efficient than many
communications transferring smaller chunks of data from multiple processors. We
show that the \gls{sgd} algorithm can still make useful progress if the batches
are defined on a per-processor basis and processed in random order even though
(i) the batches are constructed from data samples from a single class or
specific flow region, and (ii) the overall data samples are heterogeneous. In
this work, we present a new block-random algorithm that works on distributed,
heterogeneous data without having to pre-shuffle.

\begin{figure}[!tbp]%
  \centering%
  \begin{subfigure}[t]{0.32\textwidth}%
    \includegraphics[width=\textwidth]{./figs/singlevortex.png}%
    \caption{Advection of a passive scalar. Red: level 0; green: level 1; blue: level 2.}\label{fig:singlevortex}%
  \end{subfigure}\hfill%
  \begin{subfigure}[t]{0.32\textwidth}%
    \includegraphics[width=\textwidth]{./figs/proc_access.png}%
    \caption{Memory layout of the simulation data on the different processor ranks. Colors denote the processor ranks.}\label{fig:proc_access}%
  \end{subfigure}\hfill%
  \begin{subfigure}[t]{0.32\textwidth}%
    \includegraphics[width=\textwidth]{./figs/shuffled_access.png}%
    \caption{Fully shuffled memory access for training with \gls{sgd}. Colors denote the different batches.}\label{fig:shuffled_access}%
  \end{subfigure}%
  \caption{Illustration of memory access patterns for the simulation of a vortex using adaptive mesh refinement from the AMReX tutorial.}\label{fig:access_patterns}%
\end{figure}%

This paper is organized as follows. In Section\,\ref{sec:methods}, we
present the problem formulation and describe the proposed methodology
for deep learning of distributed, heterogeneous data. In
Section\,\ref{sec:results}, we detail the architecture of the deep
convolutional neural network used to perform the data recovery process
for fluid flows. In Section\,\ref{sec:benchmarks}, we evaluate our
proposed method on the EMNIST data sets, a standard set of benchmark
problems commonly used in deep learning. In
Section\,\ref{sec:channel}, we apply our methodology to a challenge
problem representative of those encountered for in situ deep learning
in large scale simulations. We construct \gls{sgs} stress models for
large eddy simulations using our proposed methodology and compare it
to standard approaches. Finally, we present conclusions and future work
in Section\,\ref{sec:ccl}.

\section{Methods}\label{sec:methods}
Gradient descent-based algorithms are typical optimization methods used for
training \glspl{dnn}.  Gradient descent is an algorithm to find the set of parameters
$\theta$ that minimize a cost function $J(\theta)$. In the case of \glspl{dnn}, the cost
function is usually a normed distance between the predictions and data in a
training set. The simplest form of gradient descent is:
\begin{equation}
\theta_{k+1} = \theta_k - \eta \nabla_{\theta_k} J(\theta_k)
\label{eq:GD}
\end{equation}
where $\eta$ is the learning rate, $k$ denotes subsequent iterations
of the gradient descent algorithm, and $J$ is computed on the entire
training data set. For very large data sets, this algorithm is very
slow because it computes the cost function on the entire data set for
a single update of the parameters. \Gls{sgd}, in contrast, performs a
parameter update for each sample in the training data set. \gls{sgd}
and \gls{sgd} variants --- such as Adam~\cite{kingma2014adam},
RMSprop~\cite{tieleman2012lecture} and
Adagrad~\cite{duchi2011adaptive} --- have proved to be effective ways
of training \glspl{dnn}. A common addition to \gls{sgd} is to add
``mini-batching'', where the training data are partitioned into
batches of size $n_b$. The batching procedure is used to provide
sequences of approximations of the gradient of the cost function with
respect to the parameters by computing:
\begin{equation}
  \nabla_{\theta_k} J(\theta_k|x \in b, y \in b)
\end{equation}
where $b$ is a batch of training data. Mini-batching provides a better
estimate of $\nabla_\theta J(\theta)$ by using several samples from
the training set instead of only one sample. As a result, the
parameter updates tend to be less noisy. It is also computationally
more efficient by using vectorized computations and parallelism
provided by modern architectures. In practice, the shuffled training
data is divided into batches, the batches are then randomly shuffled
before each pass through the training data, and each batch is used to
provide gradient approximations to update the neural network model
parameters. It has been shown that the \gls{sgd} gradient
approximations converge to the true gradient in
expectation~\cite{bottou2010large}.

Being stochastic, however, these algorithms require the data to be randomly
shuffled to converge to a minimum of the cost function. Results from
Section \ref{sec:benchmarks} show how these algorithms fail to converge without
shuffling when the batches have inherent bias.  As described in Section
\ref{sec:intro}, this shuffling operation is infeasible for online learning on
exascale simulations. We propose a block-random algorithm for use in these
cases where the data ordering needs to remain unchanged. The algorithm operates
by swapping the order of shuffling and batching operations. This shuffling of
batches appears to be sufficient for learning the parameters even when the data
are highly ordered, resulting in batches with high bias. Although individual
batches have high bias, this shuffling operation ensures that the same bias is
not seen by the optimizer consecutively, enabling it to still get to a local
minima of the cost function. This behavior will be shown over a variety of
benchmark problems in Section \ref{sec:results}. In a distributed data setting,
the shuffling of batches will be achieved by picking a random block of data and
getting a batch of size $n_b$ from it.

\begin{algorithm}[h]
\DontPrintSemicolon
    \textbf{Parameters:} learning rate $\eta$, batch size $n_b$, number of epochs $n_e$\;
    \KwIn{training data with N samples}
    \SetAlgoLined
    \While{$i \leq n_e$}{
        randomly shuffle data\;
        partition data into mini batches $B_k$ of size $n_b$\;
        \While{$k \leq \nicefrac{N}{n_b}$}{
          $\theta_{k+1} = \theta_k - \eta \nabla_{\theta_k} J(\theta_k|B_k)$\;
          $k = k +1$\;
        }
        $i = i +1$\;
    }
    \caption{Stochastic gradient descent with mini-batching}\label{algo:mini-batch}
\end{algorithm}

\begin{algorithm}[H]
    \DontPrintSemicolon
    \textbf{Parameters:} learning rate $\eta$, batch size $n_b$, number of epochs $n_e$\;
    \KwIn{training data with N samples}
    \SetAlgoLined
    partition data into batches $B_k$ of size $n_b$\;
    \While{$i \leq n_e$}{
        \While{$k \leq \nicefrac{N}{n_b}$}{
          $\theta_{k+1} = \theta_k - \eta \nabla_{\theta_k} J(\theta_k|B_k)$\;
          $k = k +1$\;
        }
        $i = i +1$\;
    }
    \caption{Block-unshuffled gradient descent}\label{algo:block-unshuffled}
\end{algorithm}

\begin{algorithm}[H]
    \DontPrintSemicolon
    \textbf{Parameters:} learning rate $\eta$, batch size $n_b$, number of epochs $n_e$\;
    \KwIn{training data with N samples}
    \SetAlgoLined
    partition data into mini batches $B_k$ of size $n_b$\;
    \While{$i \leq n_e$}{
        randomly shuffle ordering of batches $B_k$\;
        \While{$k \leq \nicefrac{N}{n_b}$}{
            $\theta_{k+1} = \theta_k - \eta \nabla_{\theta_k} J(\theta_k|B_k)$\;
            $k = k +1$\;
        }
        $i = i +1$\;
    }
    \caption{Block-random gradient descent}\label{algo:block-random}
\end{algorithm}

Throughout this work, a batch denotes the data set that is used by the
\gls{sgd} algorithm to evaluate the model and perform the
back-propagation of the neural network weights. A block, or a class
for the image classification benchmark problems, denotes a homogeneous
data set that is distributed among the multiple processors. An epoch
consists of training the model on the entire training data set.

The relative performance of the algorithms will be tested on a suite
of data sets with the results shown in Section \ref{sec:results}. Each
data set will be tested in three different scenarios: (i)
\textit{shuffled} --- fully shuffling all the data and then creating
the batches (Algorithm\,\ref{algo:mini-batch}); (ii)
\textit{block-unshuffled} --- arranging the data in blocks to
emphasize bias and running it without shuffling
(Algorithm\,\ref{algo:block-unshuffled}); (iii) \textit{block-random}
--- accessing the arranged blocks in a block-random fashion
(Algorithm\,\ref{algo:block-random}). For the
\textit{block-unshuffled} and \textit{block-random} cases, we pick the
worst-case scenario for the bias. For instance, in the image
classification case, each block (and consequently each batch) will
contain only one class as shown in Table\,\ref{table:illustration}.

\begin{table}[!htb]
    \begin{subtable}{.3\linewidth}
      \centering
        \begin{tabular}{ | c | c | c | c | c | }
        \hline
            4 & 0 & 2 & \ldots & 1 \\
            3 & 2 & 9 & \ldots & 6 \\
            8 & 7 & 1 & \ldots & 8 \\
            9 & 8 & 4 & \ldots & 5 \\
            0 & 5 & 5 & \ldots & 7 \\
            6 & 1 & 3 & \ldots & 2 \\
            5 & 3 & 6 & \ldots & 9 \\
          \vdots & \vdots& \vdots & \ldots & \vdots  \\
            1 & 3 & 5 & \ldots & 7 \\
        \hline
        \end{tabular}
        \caption{\it shuffled}
    \end{subtable}%
    \begin{subtable}{.4\linewidth}
      \centering
        \begin{tabular}{ | c | c | c | c | c | c | }
        \hline
            0 & 0 & \ldots & 1 & \ldots & 9 \\
            0 & 0 & \ldots & 1 & \ldots & 9 \\
            0 & 0 & \ldots & 1 & \ldots & 9 \\
            0 & 0 & \ldots & 1 & \ldots & 9 \\
            0 & 0 & \ldots & 1 & \ldots & 9 \\
            0 & 0 & \ldots & 1 & \ldots & 9 \\
            0 & 0 & \ldots & 1 & \ldots & 9 \\
            \vdots & \vdots& \vdots & \vdots &  \vdots & \vdots  \\
            0 & 0 & \ldots & 1 & \ldots & 9 \\
        \hline
        \end{tabular}
        \caption{\it block-unshuffled}
    \end{subtable}%
    \begin{subtable}{.4\linewidth}
      \centering
        \begin{tabular}{ | c | c | c | c | c | }
        \hline
            4 & 3 & 8 & 0 & \ldots \\
            4 & 3 & 8 & 0 & \ldots \\
            4 & 3 & 8 & 0 & \ldots \\
            4 & 3 & 8 & 0 & \ldots \\
            4 & 3 & 8 & 0 & \ldots \\
            4 & 3 & 8 & 0 & \ldots \\
            4 & 3 & 8 & 0 & \ldots \\
            \vdots & \vdots& \vdots & \vdots &  \vdots \\
            4 & 3 & 8 & 0 & \ldots \\
        \hline
        \end{tabular}
        \caption{\it block-random}
    \end{subtable}
    \caption{
    Illustration of batches from the three scenarios for training on EMNIST digits
    data set. Each column represents a single batch, in the order processed by the
    algorithms. The \textit{shuffled} case has no bias across the batches, but both
    \textit{block-unshuffled} and \textit{block-random} just have one class in each
    batch. The main difference between \textit{block-unshuffled} and
    \textit{block-random} is that the consecutive batches are not from the same
    class in the latter.}
    \label{table:illustration}
\end{table}

\section{Results}\label{sec:results}

\subsection{Benchmark results on EMNIST data sets}\label{sec:benchmarks}

To benchmark the different learning algorithms, we use the EMNIST data
sets~\cite{Cohen2017}. This is a commonly used data set of
$28 \times 28$ pixel images of handwritten character letters and
digits. We trained models using the different training algorithms
presented in Section\,\ref{sec:methods} for the seven different data
sets: ``fashion'', ``digits'', ``letters'', ``byclass'', ``balanced'',
and ``mnist''.

\subsubsection{Neural network architecture}
The neural network architecture is two
fully connected hidden layers each comprising 512 nodes, a rectified
linear unit activation function, and a dropout layer with a dropout
rate of $0.2$. The final layer includes a softmax activation function
for the category probabilities:
\begin{align}
  y = S(x) = \frac{\exp{(x)}}{\sum^n_{i=1} \exp{(x_i)}},
\end{align}
where $x$ is the layer input vector of size $n$, and $y$ is the layer
output vector of size $n$, on the output layer to ensure that
$\sum^n_{i=1} y_i = 1$ and $y_i \in [0,1]~\forall i = 1, \dots,
n$. The loss function is the categorical cross-entropy loss, and the
\gls{sgd} algorithm for this work is the Adam
optimizer~\cite{Kingma2014}. The learning procedure occurred over 50
epochs, where one epoch consists of training the model on the entire
training data set. The deep learning framework was implemented through
Keras~\cite{Chollet2015} with the TensorFlow
backend~\cite{tensorflow2015-whitepaper}.

\subsubsection{Assessments of learning algorithm performance}

Figure\,\ref{fig:sum0} shows the model accuracy using the three
different learning strategies. The results indicate that the block-random
algorithm performs as well as the shuffled algorithm with little
difference in the model accuracy. The block-unshuffled case performs poorly
for all benchmark cases. Additionally, we investigated the effect of
the ratio $\nicefrac{n_b}{n_c}$, where $n_b$ is the batch size, and
$n_c$ is the number of samples in each class, i.e., a block of
homogeneous data, Figure\,\ref{fig:sum1}. The model accuracy for the
block-random algorithm decreases as a function of the ratio
$\nicefrac{n_b}{n_c}$. This is because as this ratio increases, the
\gls{sgd} algorithm operates on batches with little
class variation.

\begin{figure}[!tbp]%
  \centering%
  \begin{subfigure}[t]{0.48\textwidth}%
    \includegraphics[page=2, width=\textwidth]{./figs/benchmark_summary.pdf}%
    \caption{Model accuracy ($n_b=64$) for the three learning algorithms. Red: shuffled; green: sorted by class; blue: block-random.}\label{fig:sum0}%
  \end{subfigure}\hfill%
  \begin{subfigure}[t]{0.48\textwidth}%
    \includegraphics[page=1, width=\textwidth]{./figs/benchmark_summary.pdf}%
    \caption{Model accuracy of block-random algorithm as a function of $\nicefrac{n_b}{n_c}$. Red squares: ``fashion''; green diamonds: ``digits''; blue circles: ``letters''; orange pentagon: ``byclass''; purple hexagon: ``balanced''; burgundy crosses: ``mnist''.}\label{fig:sum1}%
  \end{subfigure}%
  \caption{Model accuracy, $\alpha = \nicefrac{\text{number of correctly classified images}}{\text{total number of images}}$, on validation data set for different benchmark cases.}\label{fig:predictions}%
\end{figure}%

\subsection{Channel flow}\label{sec:channel}

To evaluate the performance of the algorithm in an exascale-like setting, we
developed a \gls{dnn} model for the ``closure problem'' for
\gls{les} in computational fluid dynamics. \Gls{dns} of turbulent
flows, in which all the physical length scales are resolved
explicitly, require large computational resources and are often
unfeasible for engineering and design applications. \Gls{les}
alleviate the computational requirements by resolving the large-scale
motions and modeling the \gls{sgs}, i.e., the length scales that
are not resolved by the discretization grid. In computational fluid
dynamics, \gls{les} solve the filtered Navier-Stokes equations,
presented here in their incompressible form:
\begin{align}
  \label{eq:fns}
  \pfrac{\ol{u}_i}{t} + \pfrac{}{x_j} \left( \ol{u}_i \ol{u}_i \right) &= \pfrac{}{x_j} \left( \nu \pfrac{\ol{u}_i}{x_j} \right) - \frac{1}{\rho} \pfrac{\ol{p}}{x_i} - \pfrac{\tau_{ij}}{x_j}\\
  \pfrac{\ol{u}_i}{x_i} &= 0
\end{align}
where $i$ and $j = 1,2,3$; $x_i$ is the coordinate; $u_i$ is the
velocity in the $x_i$ direction; $p$ is the pressure; $\nu$ is the
kinematic viscosity; $\ol{\cdot}$ is the filtering operation, defined
for an variable $\phi$ as
$\ol{\phi} = \int_{\mathcal{D}} \phi(x) G(x,x') \ud{} x'$, where
$G(x,x')$ is a filter function and $\mathcal{D}$ is the domain; and
$\tau_{ij}$ is the \gls{sgs} stress defined as
$\tau_{ij} = \ol{u_i u_j} - \ol{u}_i\ol{u}_j$. The \gls{les} system of
equations is unclosed because of the \gls{sgs} stress term and
requires a model for the \gls{sgs}. Extensive work has been done to
determine appropriate models for the \gls{sgs}
stress~\cite{Rogallo1984,Lesieur1996,Piomelli1999,Meneveau2000}. For
example, an early approach~\cite{Smagorinsky1963} uses an eddy
viscosity closure that relates resolved velocity gradients to the
\gls{sgs} stress according to:
\begin{align}
  \label{eq:smag}
  \tau_{ij} = -2 (C_s \ol{\Delta})^2 |\ol{S}| \ol{S}_{ij}
\end{align}
where
$\ol{S}_{ij} = \nicefrac{1}{2} \left( \pfrac{\ol{u}_i}{x_j} +
  \pfrac{\ol{u}_j}{x_i} \right)$,
$|\ol{S}| = \sqrt{2 \ol{S}_{ij} \ol{S}_{ij}}$, $\ol{\Delta}$ is the
filter length scale, and $C_s$ is a constant determined through the
\gls{dns} of turbulent flows. Though \gls{sgs} models have received
much attention, because they are often tuned to simple configurations,
the accuracy of these models continues to be problematic in a wide
range of flows. In this section, we will use deep learning to
construct a \gls{sgs} stress model for $\tau_{12}$.

We emphasize that the objective of this work is not to derive the most
accurate \gls{sgs} model for turbulence, which has been a focus of
recent investigations using deep learning~\cite{Ling2016, Maulik2017};
rather, it is to illustrate how to use \gls{dns} data from exascale-like
simulations to develop an accurate model in the context of
distributed, heterogeneous data. As such, we will use the \gls{dns} of
an incompressible channel flow at a friction Reynolds number ($Re_\tau$) of 5186
by \citet{lee2015direct}. The simulations were performed using the code
PoongBack~\cite{lee2013petascale,lee2014experiences}, with 242 billion degrees
of freedom (10240 in $x$, 1536 in $y$, and 7680 in $z$), and was run on 52488 cores,
using approximately 400 million core hours of computation. The incompressible
channel flow exhibits high inhomogeneity and anisotropy, as shown in
Figure\,\ref{fig:dns_channel}, from the presence of the walls.  This makes it a
challenging test case for the block-random algorithm because there will be a high
bias between data from each spatial block.

\begin{figure}[!tbp]%
  \centering%
  \reflectbox{\includegraphics[width=0.5\textwidth]{./figs/5200_u_XY_u_YZ_omega_z_XZ.pdf}}%
  \caption{Streamwise velocity and wall-shear stress (inset) in the turbulent channel flow \gls{dns}. Figure generated by Dr.~Myoungkyu Lee, using data from~\cite{lee2015direct}}%
  \label{fig:dns_channel}%
\end{figure}%

\subsubsection{Data generation process}

To get data for constructing a \gls{sgs} model, first the nonlinear $u_i u_j$
terms are computed from the \gls{dns} velocity fields. A circular Fourier
cutoff filter is then applied to the resulting fields, and $\tau_{ij}$ is
computed from the filtered fields.  The cutoff wavelength is chosen as
$\lambda^+\approx1500$ in the wall-parallel directions using insights from
\citet{lee_moser_2019}. This results in a grid of dimensions $(200,1536,128)$ in $(x,y,z)$
with more than 39 million data points. The input model variables are the three
filtered velocities and nine filtered velocity gradients. The model is
therefore learning a pointwise functional form:
\begin{align}
  \label{eq:nn_function_model}
  \tau_{ij}|_k = f\left( \ol{u}_i|_k, \left.\pfrac{\ol{u}_i}{x_j}\right|_k \right)
\end{align}
where $k$ denotes a point in the domain. ,The resulting data are then
arranged into spatial blocks of size $(16,16,16)$ in $(x,y,z)$, with
each block representing a computational node. In this setting, there
will be a total of 9600 nodes containing data, which mimics a
distributed large-scale computation.

\subsubsection{Neural network architecture}

The neural network architecture is a feed-forward, fully connected
\gls{dnn}. The hidden layers each comprise fully connected
nodes. The first hidden layer contains a leaky rectified linear unit
activation function:
\begin{align}
  \label{eq:relu}
  y = R(x) =
  \begin{cases}
    x, & \text{ if } x \geq 0, \\
    \gamma x, & \text{ otherwise, }
  \end{cases}
\end{align}
where $x$ is the layer input vector, $y$ is the layer output vector,
and $\gamma=10^{-2}$ is a small slope. The other hidden layers contain
a hyperbolic tangent activation function. The final network layer does
not contain an activation function. The loss used to train the network
is a mean squared error loss function. The specific \gls{sgd} algorithm for this work is the Adam
optimizer~\cite{Kingma2014} because it presents many more advantages
than traditional \gls{sgd} by maintaining a
per-parameter learning rate, which is adapted during training based on
exponential moving averages of the first and second moments of the
gradients. The deep learning framework was implemented through
Keras~\cite{Chollet2015} with the TensorFlow
backend~\cite{tensorflow2015-whitepaper}. To find a reasonably
accurate model for this study, a sweep of the model's hyperparameters
was performed, exploring the following combinations: the initial
learning rate was varied from $10^{-2}$ to $10^{-5}$; the number of
layers, $L$, was varied from 2 to 16; and the number of nodes in
each layer was varied from 8 to 512. The sweeps were performed on the
1 million samples from the fully shuffled training data set. A
neural network comprising an initial learning rate of $10^{-4}$,
four hidden layers, and 128 nodes in each layer (for a total of 68000
trainable parameters) led to a model that was accurate without
necessitating more than eight hours of training on an Intel Skylake
workstation. This set of model hyperparameters is used in all
subsequent results.

\subsubsection{Assessments of learning algorithm performance}

In this section, we present the results of the three different
learning algorithms presented in this work: (i) fully shuffling all
the data; (ii) using the data as they are in the high performance
computing simulation (block-unshuffled), (iii) accessing the data in
block-random fashion, as discussed in Section\,\ref{sec:methods}. Model
quantities are denoted by superscript $\cdot^m$, and quantities
computed with respect to the training and validation data sets are
subscripted with $t$ and $v$, respectively. The mean squared error is defined as
$\epsilon = \frac{1}{|\mathcal{D}|} \sum_{i\in \mathcal{D}} (\tau_{12}
- \tau_{12}^m)^2$, where $\mathcal{D}$ denotes the data set, and
$|\cdot|$ is the cardinality of the set. The physics model used for
comparisons is the \gls{wale} model~\cite{Ducros1998}, a model
specifically designed for \gls{les} of wall-bounded flows.

The training and validation mean squared error, $\epsilon_t$ and
$\epsilon_v$, respectively, are shown in
Figure\,\ref{fig:accuracies_batchsize} for the three different
learning algorithms as a function of the batch size, $n_b$. The
training error for the shuffled case reaches a minimum at $n_b=256$
and increases for higher $n_b$. The validation error, however, remains
constant and smaller than the other two algorithms. The training error
for the block-unshuffled algorithm is less than that for the block-random
algorithm, though the validation error is three times larger than the
other algorithms. This indicates that the model is capturing the
batches of data toward the end of the training iteration but fails to
adequately represent the full range of data. The training error
decreases as a function of batch size because it is able to get a more
representative data batch. This results, however, in an increasing
validation error because it overfits the data available at the end of the
training iteration. The block-random algorithms exhibits a validation
error that is approximately $10\%$ higher than the shuffled algorithm,
and it remains small as the batch size increases. For the remaining
results presented in this section, the batch size is fixed at 256.

\begin{figure}[!tbp]%
  \centering%
  \begin{subfigure}[t]{0.48\textwidth}%
    \includegraphics[page=1, width=\textwidth]{./figs/accuracies_batchsize.pdf}%
    \caption{Training error, $\epsilon_t$.}%
  \end{subfigure}\hfill%
  \begin{subfigure}[t]{0.48\textwidth}%
    \includegraphics[page=2, width=\textwidth]{./figs/accuracies_batchsize.pdf}%
    \caption{Validation error, $\epsilon_v$.}%
  \end{subfigure}%
  \caption{Mean squared error for the three different learning algorithms as a function of the batch size. Red squares: shuffled; green diamonds: block-unshuffled; blue circles: block-random.}\label{fig:accuracies_batchsize}%
\end{figure}%

The normalized \gls{pdf} of the error and the conditional means of the
predictions are shown in Figure\,\ref{fig:predictions}. The shuffled
algorithm exhibits a sharp \gls{pdf} of the error and a conditional
means of the predictions identical to the filtered \gls{dns}
throughout the channel domain. The block-random algorithm has a
similar error \gls{pdf} but underpredicts the peak $\tau_{12}$ by
approximately $8\%$. The block-unshuffled algorithm fails to capture the
conditional means throughout the channel. The \gls{wale} physics model
overpredicts the peak $\tau_{12}$ by a factor of two and predicts that
the peak occurs closer to the channel centerline. The \gls{wale} error
\gls{pdf} has a higher variance than the shuffled and block-random
algorithms.

\begin{figure}[!tbp]%
  \centering%
  \begin{subfigure}[t]{0.48\textwidth}%
    \includegraphics[page=4, width=\textwidth]{./figs/predictions.pdf}%
    \caption{Probability density function of the error normalized by the standard deviation of $\tau_{12}$.}%
  \end{subfigure}\hfill%
  \begin{subfigure}[t]{0.48\textwidth}%
    \includegraphics[page=17, width=\textwidth]{./figs/predictions.pdf}%
    \caption{Conditional means as a function of wall distance, $y^+ = Re_\tau y$.}%
  \end{subfigure}%
  \caption{Model accuracy on validation data set. Red squares and solid: shuffled; green diamonds and dashed: block-unshuffled; blue circles and dash-dotted: block-random; purple hexagons and solid: \gls{wale} physics model; black solid: filtered \gls{dns}}\label{fig:predictions}%
\end{figure}%

As evidenced by these results, the key factor determining the
performance of the model is the order of the blocks used by the
\gls{sgd} algorithm to adjust the model
parameters. To quantify the difference between the learning
algorithms, we use the Jensen-Shannon divergence~\cite{Endres2003,
  Osterreicher2003}, a measure of the similarity between two
\glspl{pdf}. It is a symmetric version of the Kullback-Leibler
divergence~\cite{Kullback1987}, and it is defined as:
\begin{align}
  \label{eq:jsd}
  J(Q,R) = \frac{1}{2} \left( D(Q , M) + D(R , M)\right)
\end{align}
where
$D(Q,R) = \sum_{i=1}^n R(i) \ln{\left( \frac{R(i)}{Q(i)} \right)}$;
$M = \nicefrac{1}{2} \left( Q+R \right)$; $Q$ and $R$ are \glspl{pdf}
of length $n$; and $0\leq J(Q,R) \leq \ln{(2)}$. Low values indicate
more similarity between $Q$ and $R$. The Jensen-Shannon divergence has
several advantages compared to the Kullback-Leibler divergence:
symmetry, i.e., $J(Q,R) = J(R,Q)$, bounded, and variable
support. This metric is used to quantify the difference between the
different \glspl{pdf}: the \gls{pdf} of $\tau_{12}$ in the validation
data set, $Y = P(\tau_{12} \in \mathcal{D}_v)$, and the \gls{pdf} of a
given batch, $Y_b = P(\tau_{12} \in \mathcal{D}_b)$, where
$\mathcal{D}_b$ is the set of data in batch $b$. We compare (i) how
the data in each batch are representative of the data in the entire
domain by computing $J(Y, Y_b)$ for all the batches in an epoch and
(ii) how the data in each batch vary compared to the previous batch
by computing $J(Y_{b-1}, Y_b)$.

Figure\,\ref{fig:jsds} illustrates the different metrics for the three
different learning algorithms. For the fully shuffled case, each batch
exhibits a \gls{pdf} similar to that of the entire data set. The
difference between $Y$ and $Y_b$ for the block-random case is higher
but remains constant during the training epoch. For the block-unshuffled
case, there is a clear structure to $J(Y,Y_b)$ because of the
heterogeneity of the data near the channel walls (beginning and ending
of the training epoch). For all three algorithms, the difference
between each subsequent batch is negligible. These results indicate
that ensuring that $J(Y,Y_b)$ remains less than 0.2 and constant
throughout the training epoch is a criteria for achieving high model
accuracy.

\begin{figure}[!tbp]%
  \centering%
  \begin{subfigure}[t]{0.48\textwidth}%
    \includegraphics[page=2, width=\textwidth]{./figs/jsds.pdf}%
    \caption{Comparison between each batch and the entire data set.}%
  \end{subfigure}\hfill%
  \begin{subfigure}[t]{0.48\textwidth}%
    \includegraphics[page=5, width=\textwidth]{./figs/jsds.pdf}%
    \caption{Comparison between subsequent batches.}%
  \end{subfigure}%
  \caption{Characterizing the differences between batches. Solid red: shuffled; dashed green: block-unshuffled; dash-dotted blue: block-random.}\label{fig:jsds}%
\end{figure}%

\section{Conclusions}\label{sec:ccl}

Effectively training \gls{dnn} models often assumes that the \gls{sgd}
algorithm processes data in batches of data that have been randomized
prior to model training. This is expected to be prohibitively
expensive for in situ model training from the perspective of
communication between computing nodes, as illustrated in
Figure\,\ref{fig:access_patterns}. We showed that the \gls{sgd}
algorithm can still train an effective model if the batches are
processed in random order even if the batches are each comprised of
similar data. In this work, we demonstrated a block-random learning
algorithm for training \glspl{dnn} in the context of distributed
heterogeneous data for data parallelism learning, a situation that
will be increasingly common as the exascale era approaches and models
are trained in situ. The block-random learning algorithm was tested on
several different cases. For the benchmark EMNIST data sets, the
block-random algorithm achieves accuracy similar to the traditional
fully shuffled learning algorithm. The performance decreases slightly
as the ratio of the batch size to the number of classes increases. To
demonstrate the efficacy of the block-random algorithm for
exascale-type simulations, we used a \gls{dns} simulation of turbulent
channel flow to construct a \gls{les} \gls{sgs} model. The model
constructed using the block-random algorithm performed significantly
better than the block-unshuffled learning model and is within $8\%$ of
the fully shuffled model. Using the Jensen-Shannon divergence metric,
we analyzed the characteristics of the batches used by the \gls{sgd}
to inform a criteria for successfully constructing a \gls{dnn} model
for distributed heterogeneous data.

This work --- including neural network models, analysis scripts,
Jupyter notebooks, and figures --- can be publicly accessed at the
project's GitHub
page.\footnote{\url{https://github.com/NREL/block-random}} Traditional
machine learning algorithms were implemented through
scikit-learn~\cite{Pedregosa2011} and the deep learning algorithms
through Keras~\cite{Chollet2015} with the TensorFlow
backend~\cite{tensorflow2015-whitepaper}.

\section*{Acknowledgments}
This work was authored in part by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided by U.S. Department of Energy Office of Science and National Nuclear Security Administration. The views expressed in the article do not necessarily represent the views of the DOE or the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for U.S. Government purposes.

This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations -- the Office of Science and the National Nuclear Security Administration -- responsible for the planning and preparation of a capable exascale ecosystem -- including software, applications, hardware, advanced system engineering, and early testbed platforms -- to support the nation's exascale computing imperative.

\section*{References}

\bibliography{library}

\end{document}


\subsection{Baselines}
\label{sec:baselines}

\boldparagraph{Data-Driven Approaches}
%
{We consider the works by \cite{Engelmann2016GCPR} and \cite{Gupta2015CVPR} as data-driven baselines. Additionally, we consider regular maximum likelihood (\ML). \cite{Engelmann2016GCPR} -- referred to as \Engelmann\xspace-- use a principal component analysis shape prior trained on a manually selected set of car models\footnote{\url{https://github.com/VisualComputingInstitute/ShapePriors_GCPR16}}. Shape completion is posed as optimization problem considering both shape and pose. The pre-trained shape prior provided by Engelmann \etal assumes a ground plane which is, according to KITTI's LiDAR data, fixed at $1m$ height. Thus, we don't need to optimize pose on KITTI as we use the ground truth bounding boxes; on ShapeNet, in contrast, we need to optimize both pose and shape to deal with the random rotations in \clean and \noisy.}

\input{fig_results_latent_space}
\input{tab_results_shapenet_kitti}

Inspired by the work by \cite{Gupta2015CVPR} we also consider a shape retrieval and fitting baseline. Specifically, we perform iterative closest point (\ICP) \citep{Besl1992PAMI} fitting on all training shapes and subsequently select the best-fitting one. To this end, we uniformly sample $1\text{Mio}$ points on the training shapes, and perform point-to-point \ICP\footnote{\url{http://www.cvlibs.net/software/libicp/}.} for a maximum of $100$ iterations using $\left[\begin{matrix}R & t\end{matrix}\right] = \left[\begin{matrix}I_3 & 0\end{matrix}\right]$ as initialization. On the training set, we verified that this approach is always able to retrieve the perfect shape.

Finally, we consider a simple \ML baseline iteratively minimizing \eqnref{eq:ml} using stochastic gradient descent (SGD). This baseline is similar to the work by Engelmann \etal, however, like ours it is bound to the voxel grid. Per example, we allow a maximum of $5000$ iterations, starting with latent code $z = 0$, learning rate $0.05$ and momentum $0.5$ (decayed every $50$ iterations at rate $0.85$ and $1.0$ until $10^{-5}$ and $0.9$ have been reached).

\boldparagraph{Learning-Based Approaches}
%
Learning-based approaches usually employ an encoder-decoder architecture to directly learn a mapping from observations $x_n$ to ground truth shapes $y_n^*$ in a fully supervised setting \citep{Wang2017ICCV,Varley2017IROS,Yang2018ARXIVb,Yang2017ARXIV,Dai2017CVPRa}. While existing architectures differ slightly, they usually rely on a U-net architecture \citep{Ronneberger2015MICCAI,Cicek2016ARXIV}. In this paper, we use the approach of \cite{Dai2017CVPRa}\footnote{
    We use \url{https://github.com/angeladai/cnncomplete}. On ModelNet we added one convolutional stage in the en- and decoder for larger resolutions; on ShapeNet and KITTI, we needed to adapt the convolutional strides to fit the corresponding resolutions.
} -- referred to as \Dai\xspace --
as a representative baseline for this class of approaches. In addition, we consider a custom learning-based baseline which uses the architecture of our \DVAE shape prior, \cf \figref{fig:architectures}. In contrast to \citep{Dai2017CVPRa}, this baseline is also limited by the low-dimensional ($Q = 10$) bottleneck as it does not use skip connections.