\chapter{Background}\label{chap:background}
Some background knowledge is necessary for the reader to have an understanding of the principles behind semantic segmentation and convolutional neural networks.
This chapter on the background of this work will discuss semantic segmentation, machine learning with artificial neural networks and convolutional neural networks, curb segmentation, loss functions, network optimizers, and binary dilation.

\section{Semantic Segmentation}\label{section:background-segmentation}
Unlike image classification, which classifies the contents of an image as a whole, semantic segmentation is the use of some algorithm to process an image and assign class labels to each individual pixel~\cite{segmentation-medium}.
This adds the ability to locate and classify multiple objects in a given scene.
For example, the ground truth segmentation of the street-level image in \figref{fig:background-raw} can be seen in \figref{fig:background-segmented}.
Each pixel of the image has been assigned a class, which is represented by the different colors in the segmented image.
This allows a computer or program to understand what objects are in the image it is shown.

\input{figures/background/segmentation}

By segmenting an image in this way, the program can interpret the scene semantically.
For example, by receiving the segmented image, a program can identify that there are line markings on the road and that there is a vehicle in front of it.
The segmentation of images in this way is essential in many robotic applications as it allows further higher level processing of the scene.

Towards the goal of this work, the segmentation of traversability classes in a scene allows Obelix to more accurately find a path allowing for the safe traversal from sidewalk to street level via curb cuts.

\input{chapters/3_1-neural-networks}

\section{Curbs and Curb Cuts}\label{section:background-curbs}
Curbs are the concrete or stone edging of a road. Curbs usually separate the road from some other area with a different elevation, usually a pedestrian sidewalk.
An example of a sidewalk with a curb can be seen in Figure \ref{fig:background-curb}.
Curb cuts are "cuts" in the curb that allow a pedestrian sidewalk to have a gentle slope down to street level.
An example of a sidewalk with a curb cut can be seen in Figure \ref{fig:background-curbcut}.
Originally, these cuts were made to allow accessibility access, especially for those requiring wheelchairs.
To a wheelchair user, curbs represent a significant barrier in terms of traversability, as was discussed in the article "Curb Cuts" by Cynthia Gorney and Delaney Hall \cite{99pi}.
Curb cuts first started appearing fifty years ago from the efforts of activist Ed Roberts and his push to make city streets more accessible.

Wheeled robots are also unable to traverse curbs.
They therefore require curb cuts to traverse urban environments where curbs are present.

\input{figures/background/curbs}

\section{Loss Functions}\label{section:background-loss}
The loss function $\ell$ maps the output of a neural network $\hat{y}$ and the target $y$ onto a real number and represents the "cost" of an output.
The goal of learning is to minimize this cost value.
Effectively, the loss function measures the difference between the predicted value and the target.
In our case, the target value is the ground truth labeling of an image and the output is the softmax class predictions from the network.
There are many different loss functions that are commonly used. For the purposes of image segmentation, the most commonly used function is cross entropy loss.

\subsection{Cross Entropy Loss}\label{section:background-crossentropy}
The cross entropy loss is formally defined as:
\begin{align}
	\ell(y, \hat{y}) &=\sum_{m}-y_m\log(\hat{y}_m)
\end{align}
where $m$ is the class, $y_m$ the ground truth value for a class $m$, and $\hat{y}_m$ the softmax prediction of a class $m$ by the model.
As such, cross entropy loss calculates the error of the model to classify a certain value correctly.
The loss of two predictions could be the same, as long as the prediction for the true class, i.e. the class that has value 1 in $y$, remains constant.
The loss is thus independent of how the probability is split between the remaining classes.

For example, let the ground truth classification be class 0, i.e. $y = \{1, 0, 0\}$ with one-hot encoding.
Let the current model prediction be $\hat{y} = \{0.5, 0.2, 0.3\}$, also with one-hot encoding.
The loss would then be:
\begin{equation}
	\begin{split}
		\ell(y, \hat{y}) 	&= \sum_{m}-y_m \log(\hat{y}_m) \\
		&= -(1)\log(0.5) + (-0)\log(0.2) + (-0)\log(0.3)\\
		&= -\log(0.5) \\
		&= 0.301...
	\end{split}
\end{equation}
Alternatively, if model prediction is exactly the same as the ground truth, then we would have:
\begin{equation}
	\begin{split}
		y					&= \hat{y} = \{1, 0, 0\}\\
		\ell(y, \hat{y}) 	&= \sum_{m}-y_m \log(\hat{y}_m) \\
		&= -(1)\log(1) + (-0)\log(0) + (-0)\log(0)\\
		&= -\log(1) \\
		&= 0
	\end{split}
\end{equation}
Applying this on the full image, this becomes:
\begin{align}
	\ell(y, \hat{y}) &=\sum_u \sum_v \sum_{m}-y_{m, u, v}\log(\hat{y}_{m,u,v})
\end{align}
where $u$ and $v$ are the pixel coordinates.

The weighted variant, known simply as weighted cross entropy loss, is defined as:
\begin{align}
	\ell(y, \hat{y}) &=\sum_{m}-y_m\log(\hat{y}_m) \cdot d_m
\end{align}
where $d_m$ is the weight for a given class $m$.

Using the weighted version allows changes to how much each class contributes to the loss.
This is done to counteract class imbalance that could be caused by having an imbalance in the distribution or frequency of each class in the dataset.

\section{Optimizers}\label{section:background-optimizers}
During training, network parameters must be updated to minimize the loss function at each iteration.
This is done by the optimizer.
The optimizer updates the network parameters in such a way as to minimize the loss function~\cite{optimizer-intro}.

Gradient Descent is one of the earliest optimizers.
It works by calculating what a small change in each network parameter would do to the loss function, i.e. determines the gradient of the loss for the current iteration.
It then adjusts the network parameters according to this gradient.
This process is repeated at each iteration to further minimize the loss function.
This process can be time-consuming, especially with larger datasets.
Stochastic Gradient Descent (SGD) is a more commonly used variant of gradient descent and works similarly, but working only on randomly selected batches of the training data at each iteration~\cite{optimizer-intro}.
Due to the large size of datasets, training is usually done in batches.
By using SGD, the optimization step can occur after processing every batch, rather than after the entire dataset is processed.
This speeds up with respect to wall-clock time network training.

Adaptive Moment Estimation (Adam) is another optimizer that is commonly used and first proposed in the paper "Adam: A method for stochastic optimization" by Diederik Kingma and Jimmy Ba~\cite{adam}.
It combines concepts from Adaptive Gradient Algorithm (AdaGrad), which has per-parameter learning rates, and Root Mean Square Propagation (RMSProp), whose learning rates are adapted based on the average of the magnitude of recent gradients~\cite{adam}.
Adam utilizes the concept of momentum, which incorporates previous gradients into the current one.
It does this by calculating an exponential moving average and the square of previous gradients and using these to determine the next gradient.

\section{Backpropagation}\label{section:background-backpropagation}
Backpropagation is a learning procedure used in artificial neural networks to adjust network weights and produce internal representations of "important features in the task domain," first described in 1986 by David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams in their paper "Learning representations by back-propagating errors"~\cite{backprop}.
Backpropagation efficiently and repeatedly adjusts the network weights to minimize the network error, which is calculated by the loss function.

Calculating the network weights is done layer by layer, starting with the final output layer.
The partial derivative of $\ell$ with respect to $y_j$, intuitively the effect the output unit has on the loss, is defined as:
\begin{align}\label{eq:loss-1}
	\frac{\partial \ell}{\partial y_j}
\end{align}
where $y_j$ is the output of a single unit in the final layer.
The contribution of the input of the unit $j$ on the error can then be calculated using the value previously calculated in \eqref{eq:loss-1} using the chain rule for differentiation as
\begin{align}\label{eq:loss-2}
	\frac{\partial\ell}{\partial x_j} &= \frac{\partial \ell}{\partial y_j} \cdot \frac{\partial y_j}{\partial x_j}
\end{align}
We now have a description of how changing the input of unit $j$ will affect the loss.
As such, we can then also represent how changing the weight $w_{ji}$ of a connection between the unit $j$ and a unit in the previous layer $i$ will effect the error as
\begin{align}\label{eq:loss-3}
	\frac{\partial \ell}{\partial w_{ji}} &= \frac{\partial \ell}{\partial x_j} \cdot \frac{\partial x_j}{\partial w_{ji}}
\end{align}
In this case, the previous layer means the layer which during forwards-propagation would be calculated immediately before the layer in which unit $j$ is.
Again, due to the use of the chain rule, the previous value calculated in \eqref{eq:loss-2} can be used here.
This gives us the contribution of the weight $w_{ji}$ on the error and, by multiplying by the learning rate, the amount by which the gradient must be changed for the next training iteration.

The error contribution of the output of unit $i$ via a single successor unit $j$ can also similarly be calculated using the value from \eqref{eq:loss-2}.
\begin{align}\label{eq:loss-4}
	\frac{\partial \ell}{\partial y_i} &= \frac{\partial \ell}{\partial x_j} \cdot w_{ji}
\end{align}
Since unit $i$ is connected to multiple successor units, its value is a summation of its output contribution via all units $j \in J$, where $J$ is the set of all units in successive layers that $i$ is connected to.
\begin{align}
	\frac{\partial \ell}{\partial y_i} &= \sum_{j \in J} \frac{\partial \ell}{\partial x_j} \cdot w_{ji}
\end{align}
This calculation can be done in parallel, thanks to parallel computing architecture, completing the gradient calculations for the penultimate layer.
This procedure is then repeated for all layers, each time using the previously computed values to allow for efficient computation.

\input{chapters/3_2-hyperparameters}

\section{Binary Dilation}\label{section:background-dilation}
Binary dilation is a basic morphological operation and is usually represented by the operator $\oplus$.
With regards to this work, binary dilation is used in our loss function.
For a given binary image viewed as an integer grid $\mathbb{Z}^d$ for some dimension $d$, let $E$ be an integer grid, $A \in E$ a binary image, and $B \in \{0,1\}^d$ a structuring element.
The binary dilation of $A$ by $B$ is then defined as:
\begin{align}
	A \oplus B &= \bigcup_{b \in B}A_b
\end{align}
where $A_b$ is the translation of $A$ by $b$.
This can be seen as extending the area of the binary image $A$ by locus of the points covered by $B$ given that $B$ has a center on the origin~\cite{morphology}.


\vssub
\subsection{~Depth variations in time} \label{sub:num_depth}
\vssub

Temporal depth variations result in a change of the local wavenumber
grid. Because the wavenumber spectrum is invariant with respect to temporal
changes of the depth, this corresponds to a simple interpolation of the
spectrum from the old grid to the new grid, without changes in the spectral
shape. As discussed above, the new grid simply follows from the globally
invariant frequency grid, the new water depth $d$ and the dispersion relation
Eq. (\ref{eq:disp}). The time step of updating the water level is generally
dictated by physical time scales of water level variations, but not by
numerical considerations \citep{tol:GAOS98b}.

The interpolation to the new wavenumber grid is performed with a simple
conservative interpolation method. In this interpolation the old spectrum is
first converted to discrete action densities by multiplication with the
spectral bin widths. This discrete action then is redistributed over the new
grid cf.\ a regular linear interpolation. The new discrete actions then are
converted into a spectrum by division by the (new) spectral bin widths. The
conversion requires a parametric extension of the original spectrum at high
and low frequencies because the old grid generally will not completely cover
the new grid. Energy/action in the old spectrum at low wavenumbers that are
not resolved by the new grid is simply removed. At low wavenumbers in the new
grid that are not resolved by the old grid zero energy/action is assumed. At
high wavenumbers in the new grid the usual parametric tail is applied if
necessary. The latter correction is rare, as the highest wavenumbers usually
correspond to deep water.

In practical applications the grid modification is usually relevant for a
small fraction of the grid points only. To avoid unnecessary calculations, the
grid is transformed only if the smallest relative depth $kd$ in the discrete
spectrum is smaller than 4. Furthermore, the spectrum is interpolated only if
the spatial grid point is not covered by ice, and if the largest change of
wavenumber is at least $0.05 \Delta k$.


\documentclass{beamer}
\usetheme{Warsaw}
\usepackage{nhtvslides}
\usepackage{graphicx}
\usepackage{listings}
\lstset{language=CAML,
basicstyle=\ttfamily\footnotesize,
frame=shadowbox,
breaklines=true}
\usepackage[utf8]{inputenc}

\title{Building a physics engine - part 3: collision response}

\author{Dr. Giuseppe Maggiore}

\institute{NHTV University of Applied Sciences \\
Breda, Netherlands}

\date{}

\begin{document}
\maketitle

\begin{frame}{Table of contents}
\tableofcontents
\end{frame}

\section{Collision response}
\begin{slide}{Collision response}{Collision response system}{
\item Solving physical constraints in addition to the equations of motion
\item Constraints are mostly contact constraints, but also
\begin{itemize}
\item Friction constraints
\item Distance constraints
\item Joint angle constraints
\item ...
\end{itemize}
\item The ideal collision response system deals with all of these
}\end{slide}

\begin{slide}{Collision response}{Naïve take}{
\item Apply the constraints to the objects in pairs
\item Use the laws of \textit{conservation of motion} for each collision; $P_0$ is the point of collision, $x_A$ and $x_B$ are the centres of mass of the objects, $v^{-1}$ is the pre-impact velocity of the objects, $v^+$ is the post-impact velocity of the objects
\begin{itemize}
\item $f = \frac{-(1 + \epsilon)(N_0 \cdot (v_A^{-1} - v_B^{-1})) + (\omega_A^- \cdot (r_A \times N_0) - \omega_B^- \cdot (r_B \times N_0)))}{1/m_A + 1/m_B + (r_A \times N_0)^T J_A^{-1} (r_A \times N_0) + (r_B \times N_0)^T J_B^{-1} (r_B \times N_0)}$
\item $r_A = P_0 - x_A$, $r_B = P_0 - x_B$
\item $v_A^+ = v_A^- + \frac{f N_0}{m_A}$
\item $\omega_A^+ = \omega_A^- + J_A^- (r_A \times (f N_0))$
\end{itemize}
\item Push away from interpenetration as long as interpenetration exists
}\end{slide}

\begin{slide}{Collision response}{Naïve take}{
\item Jitters a lot, and does not support stacking
\item May be acceptable in very sparse scenarios (space/flight simulator)
}\end{slide}

\begin{slide}{Collision response}{Naïve take number 2}{
\item Apply the constraints to the objects in pairs
\item Apply again during the same tick
\item Average/combine the various impulses
\item Push apart objects so they do not penetrate
\pause
\item Constraints are still broken
\item \textit{Hack-y method}, gives no guarantees
\item Still does not support stacking
}\end{slide}

\section{Constrained dynamics}
\begin{slide}{Constrained dynamics}{Unconstrained kinematics}{
\item A rigid body is characterised by
\item $\dot x = v$
\item $\dot q = \frac{1}{2}wq$
}\end{slide}

\begin{slide}{Constrained dynamics}{Unconstrained kinematics}{
\item For a system of $N$ bodies, we can define the system derivative as
$$V = \left[ \begin{matrix}
v_1 \\ \omega_1 \\ \vdots \\ v_N \\ \omega_N
\end{matrix} \right]$$
}\end{slide}

\begin{slide}{Constrained dynamics}{Constraints}{
\item Our system allows \textit{pairwise} constraints between bodies
\item The $k$-th constraint, between bodies $i$ and $j$, has the form $C_k(x_i,q_i,x_j,q_j) = 0$
\item The vector $C$ holds all the constraints. $C = 0$, or $C(x) = 0$, is a function of the state vector, so by the chain rule $\dot C = J V = 0$
}\end{slide}

\begin{slide}{Constrained dynamics}{Constraint forces}{
\item Each constraint causes a reaction force $f_c$ and a reaction torque $\tau_c$
\item The vector of all reaction forces is $$F_c = \left[ \begin{matrix}
f_{c1} \\ \tau_{c1} \\ \vdots \\ f_{cN} \\ \tau_{cN}
\end{matrix} \right]$$
}\end{slide}

\begin{slide}{Constrained dynamics}{Constraint forces}{
\item We know that $\dot C = J V = 0$
\item $J V = \left[ \begin{matrix}
J_1 \cdot V \\ \vdots \\ J_M \cdot V \\
\end{matrix} \right] = 0$
\item This means that $V$ is orthogonal to each row of $J$
}\end{slide}

\begin{slide}{Constrained dynamics}{Constraint forces}{
\item Constraint forces perform no work, so $F_c \cdot V = 0$
}\end{slide}

\begin{frame}{Principle of virtual work}
\center
\includegraphics[width=7cm]{Pics/VirtualWork.png}
\end{frame}

\begin{slide}{Constrained dynamics}{Constraint forces}{
\item We can use $F_c = J^T \lambda$ for some vector $\lambda$ of undetermined force multipliers
\item $F_c \cdot V = J^T \lambda \cdot V = (\sum_i J_i \lambda_i) \cdot V = \sum_i J_i \cdot V \lambda_i = \sum_i 0 \lambda_i = 0$
}\end{slide}

\begin{slide}{Constrained dynamics}{Constraint forces}{
\item We will thus compute the matrix $J$ of constraints from the collision system
\item We then solve for $\lambda$, compute $F_c$, and finally obtain $V$
}\end{slide}

\section{Setting up the constraints}
\begin{slide}{Constrained dynamics}{Distance constraints}{
\item The simplest constraint is a distance constraint
\item Two points of two bodies must remain at a given distance
\item $C(x_i,q_i,x_j,q_j) = \frac{1}{2}(|p_j-p_i|^2 - L^2) = 0$
\item If we derive this, we get $\dot C(x_i,q_i,x_j,q_j) = \underbrace{(p_j-p_i)}_{d}(v_j + \omega_j \times r_j - v_i - \omega_i \times r_i)$
\item We split this into a row for $J$ and a part of $V$:
\item $\dot C(x_i,q_i,x_j,q_j) = \underbrace{\left[ -d^T\ -(r_i \times d)^T\ d^T\ (r_j \times d)^T \right]}_{\text{a row of }J} \underbrace{\left[ v_i\ \omega_i\ v_j\ \omega_j \right]^T}_{\text{some columns of } V}$
}\end{slide}

\begin{frame}{Distance constraint}
\center
\includegraphics[width=7cm]{Pics/DistanceConstraint.png}
\end{frame}

\begin{slide}{Constrained dynamics}{Distance constraints}{
\item We are abusing the notation; the ``row of $J$'' also contains many zeroes $(6 \times (N_{\text{bodies}} - 2))$
\item The only columns that are not zeroed are those corresponding to the bodies $i$ and $j$
\item $\dot C(x_i,q_i,x_j,q_j) = \underbrace{\left[ \dots\ 0\ -d^T\ -(r_i \times d)^T\ 0\ \dots\ 0\ d^T\ (r_j \times d)^T\ \dots\ 0\right]}_{\text{a row of }J} V$
}\end{slide}

\begin{slide}{Constrained dynamics}{Contact constraints}{
\item We may also model contact constraints
\item The contact constraint measures the object separation; it is negative in case of overlap
\item $C(x_i,q_i,x_j,q_j) = (x_j + r_j - x_i - r_i) \cdot n_i = 0$
\item $\dot C(x_i,q_i,x_j,q_j) = (v_j + \omega_j \times r_j - v_i - \omega_i \times r_i) \cdot n_i + (x_j + r_j - x_i - r_i) \cdot \omega_i \times n_i$
\item We assume that both penetration and angular velocity are small, so we ignore the second term
\item $\dot C(x_i,q_i,x_j,q_j) \approx (v_j + \omega_j \times r_j - v_i - \omega_i \times r_i) \cdot n_i$
}\end{slide}

\begin{frame}{Contact constraint}
\center
\includegraphics[height=5cm]{Pics/ContactConstraint.png}
\end{frame}

\begin{slide}{Constrained dynamics}{Contact constraints}{
\item We can now separate $\dot C$ into $J$ and $V$:
\item $\dot C(x_i,q_i,x_j,q_j) = (v_j + \omega_j \times r_j - v_i - \omega_i \times r_i) \cdot n_i = \left[ -n_i^T\ -(r_i \times n_i)^T\ n_i^T\ (r_j \times n_i)^T\ \right] \left[ v_i\ \omega_i\ v_j\ \omega_j \right]^T$
}\end{slide}

\begin{slide}{Constrained dynamics}{Contact constraints}{
\item Notice that the force between bodies in contact can push them apart, but not pull them together
\item This means that $0 \leq \lambda_k \leq +\infty$, where $k$ is the constraint index for a contact constraint
}\end{slide}

\begin{slide}{Constrained dynamics}{Contact constraints}{
\item In some cases penetration might happen anyway
\item Numerical errors or issues with discrete steps
\item We allow the velocity to be augmented with a \textit{pushing factor} which is proportional to the penetration
\item This means that for contact constraints $J_i V = -\ \beta C_i$, for $\beta \leq \frac{1}{\Delta t}$
}\end{slide}

\begin{slide}{Constrained dynamics}{Friction constraints}{
\item Friction constraints are very similar to contact constraints
\item Friction happens along the tangent plane, so we have two constraints (one for $u_i = T$ and one for $u_j = B$)
\begin{itemize}
\item $\dot C_{u_i}(x_i,q_i,x_j,q_j) = (v_j + \omega_j \times r_j - v_i - \omega_i \times r_i) \cdot u_i = \left[ -u_i^T\ -(r_i \times u_i)^T\ u_i^T\ (r_j \times u_i)^T\ \right] \left[ v_i\ \omega_i\ v_j\ \omega_j \right]^T$
\item $\dot C_{u_j}(x_i,q_i,x_j,q_j) = (v_j + \omega_j \times r_j - v_i - \omega_i \times r_i) \cdot u_j = \left[ -u_j^T\ -(r_i \times u_j)^T\ u_j^T\ (r_j \times u_j)^T\ \right] \left[ v_i\ \omega_i\ v_j\ \omega_j \right]^T$
\end{itemize}
}\end{slide}

\begin{slide}{Constrained dynamics}{Friction constraints}{
\item We must also bound the friction value (this is an approximation) to take the friction coefficient into account
\item $-\mu m_c g \leq \lambda_{u_1} \leq \mu m_c g$ and $-\mu m_c g \leq \lambda_{u_2} \leq \mu m_c g$, where $m_c$ is the mass assigned to the contact point
}\end{slide}

\section{Equations of motion}
\begin{slide}{Equations of motion}{Equations of motion}{
\item We now integrate our constraint system with the equations of motion
\item We know the $Newton-Euler$ equations of motion are
$\begin{matrix}
m \dot v & = & F & = & f_c + f_{\text{ext}} \\
I \dot w & = & \tau & = & \tau_c + \tau_{\text{ext}} \\
\end{matrix}$
}\end{slide}

\begin{slide}{Equations of motion}{Equations of motion}{
\item We can define a single, big matrix for all the bodies
$M = \left( \begin{matrix}
m_1 E _{3 \times 3} & 0 & \dots & 0 & 0  \\
0 & I_1 & \dots & 0 & 0  \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
0 & \dots & 0 & m_n E _{3 \times 3} & 0  \\
0 & \dots & 0 & 0 & I_n \\
\end{matrix} \right)$
\item $E_{3 \times 3}$ is just the identity matrix
}\end{slide}

\begin{slide}{Equations of motion}{Equations of motion}{
\item We can easily invert this matrix
$M^{-1} = \left( \begin{matrix}
(m_1 E _{3 \times 3})^{-1} & 0 & \dots & 0 & 0  \\
0 & I_1^{-1} & \dots & 0 & 0  \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
0 & \dots & 0 & (m_n E _{3 \times 3})^{-1} & 0  \\
0 & \dots & 0 & 0 & I_n^{-1} \\
\end{matrix} \right)$
}\end{slide}

\begin{slide}{Equations of motion}{Equations of motion}{
\item We can define a single, big vector for all the external forces
$F_{\text{ext}} = \left[ \begin{matrix}
f_{\text{ext}1} \\
\tau_{\text{ext}1} \\
\vdots \\
f_{\text{ext}N} \\
\tau_{\text{ext}N} \\
\end{matrix} \right]$
}\end{slide}

\begin{slide}{Equations of motion}{Equations of motion}{
\item Since we know that $F_C = J^T \lambda$, we can rewrite the equations of motion for $n$ bodies as
$\left\{ \begin{matrix}
M \dot V & = & J^T \lambda + F_{\text{ext}} \\
JV & = & \epsilon \\
\end{matrix} \right.$
\item $\epsilon$ is the vector of force offsets which allows contact forces to perform work
\item We have too many unknowns: $V$, $\dot V$, and $\lambda$
}\end{slide}

\begin{slide}{Equations of motion}{Equations of motion}{
\item We approximate $\dot V \approx \frac{V_2 - V_1}{\Delta t}$
\item We replace $\dot V$
$\left\{\begin{matrix}
M \frac{V_2 - V_1}{\Delta t} & = & J^T \lambda + F_{\text{ext}} \\
JV_2 & = & \epsilon \\
\end{matrix} \right.$
\item We solve for $V_2$
$\left\{ \begin{matrix}
V_2 & = & \Delta t M^{-1}(J^T \lambda + F_{\text{ext}}) + V_1\\
V_2 & = & J^T \epsilon
\end{matrix} \right.$
}\end{slide}

\begin{slide}{Equations of motion}{Equations of motion}{
\item We can now finish solving for $\lambda$
\item $J^T \epsilon = \Delta t M^{-1}(J^T \lambda + F_{\text{ext}}) + V_1$
\item $J^T \epsilon - V_1 - \Delta t M^{-1}F_{\text{ext}} = \Delta t M^{-1}(J^T \lambda)$
\item $\frac{\epsilon}{\Delta t} - J V_1 - \Delta t J M^{-1}F_{\text{ext}} = J M^{-1} J^T \lambda$
}\end{slide}

\begin{slide}{Equations of motion}{Equations of motion}{
\item The equation $\frac{\epsilon}{\Delta t} - J V_1 - \Delta t J M^{-1}F_{\text{ext}} = J M^{-1} J^T \lambda$ admits infinite solutions; this is due to redundant constraints, such as a table with more than three legs
\item The force combinations that solve the system are usually infinite
}\end{slide}

\begin{slide}{Equations of motion}{Equations of motion}{
\item Once $\lambda$ is computed, we can determine $F_c$, $F$, and then $V_2$
\item A regular integration step is then performed with the new velocities $V_2$
}\end{slide}

\section{Iterative solution of a system of equations}
\begin{slide}{Iterative solution}{System of equations}{
\item The equation $\frac{\epsilon}{\Delta t} - J V_1 - \Delta t J M^{-1}F_{\text{ext}} = J M^{-1} J^T \lambda$ can be restated in simpler form
\item $\underbrace{\frac{\epsilon}{\Delta t} - J V_1 - \Delta t J M^{-1}F_{\text{ext}}}_{b} = \underbrace{J M^{-1} J^T}_{A} \underbrace{\lambda}_{x}$
\item $A x = b$ for some $A,b$
\item These systems can be solved iteratively with a method such as Projected Gauss-Seidel (PGS)
}\end{slide}

\begin{frame}[fragile]{(Projected) Gauss-Seidel}
\begin{lstlisting}[mathescape=true]
while not converged
  $\Delta x_i$ = $(b_i - \sum_j a_{ij} x_j) / A_{ii}$
  $x_i$ = clamp$(x_i + \Delta x_i, min_i, max_i)$
\end{lstlisting}
\end{frame}

\begin{slide}{Iterative solution}{Sparse matrices}{
\item Remember that $A$ is going to be very sparse
\item You may optimize the summation $\Delta x_i$ \texttt{=} $(b_i - \sum_j a_{ij} x_j) / A_{ii}$ a lot by ignoring the zero entries of $A$
}\end{slide}

\section{Contact caching}
\begin{slide}{Contact caching}{Contact caching}{
\item PGS is faster the closer the initial $x$ vector is to the solution
\item If we store previous contact points and their $\lambda_i$ values, PGS converges sooner
\item Just be aware of this
\begin{itemize}
\item Also be aware that it is rather hard to build in practice
\item If you attempt it, chances of success may be low
\end{itemize}
}\end{slide}

\section{Assignment}
\begin{slide}{Assignment}{Assignment}{
\item Before the end of next week
\item Group-work archive/video on Natschool or uploaded somewhere else and linked in your report
\item Individual report by each of you on Natschool
\item Build a collision response system that supports collisions between multiple objects
}\end{slide}

\begin{frame}{That's it}
\center
\fontsize{18pt}{7.2}\selectfont
Thank you!
\end{frame}

\end{document}

\begin{slide}{SECTION}{SLIDE}{
\item i
}\end{slide}

\begin{frame}[fragile]{SLIDE}
\begin{lstlisting}
CODE
\end{lstlisting}
\end{frame}

\begin{frame}{SLIDE}
\center
%\includegraphics[height=5cm]{Pics/recursive_multiplier.png}
\end{frame}


The GWT Model simulates three-dimensional transport of a single solute species in flowing groundwater.  The GWT Model solves the solute transport equation using numerical methods and a generalized control-volume finite-difference approach, which can be used with regular MODFLOW grids (DIS Package) or with unstructured grids (DISV and DISU Packages).  The GWT Model is designed to work with most of the new capabilities released with the GWF Model, including the Newton flow formulation, unstructured grids, advanced packages, and the movement of water between packages.  The GWF and GWT Models operate simultaneously during a \mf simulation to represent coupled groundwater flow and solute transport.  The GWT Model can also run separately from a GWF Model by reading the heads and flows saved by a previously run GWF Model.  The GWT model is also capable of working with the flows from another groundwater flow model, as long as the flows from that model can be written in the correct form to flow and head files.

The purpose of the GWT Model is to calculate changes in solute concentration in both space and time.  Solute concentrations within an aquifer can change in response to multiple solute transport processes.  These processes include (1) advective transport of solute with flowing groundwater, (2) the combined hydrodynamic dispersion processes of velocity-dependent mechanical dispersion and chemical diffusion, (3) sorption of solutes by the aquifer matrix either by adsorption to individual solid grains or by absorbtion into solid grains, (4) transfer of solute into very low permeability aquifer material (called an immobile domain) where it can be stored and later released, (5) first- or zero-order solute decay or production in response to chemical or biological reactions, (6) mixing with fluids from groundwater sources and sinks, and (7) direct addition of solute mass.

With the present implementation, there can be multiple domains and multiple phases.  There is a single mobile domain, which normally consists of flowing groundwater, and there can be one or more immobile domains.  The GWT Model simulates the dissolved phase of chemical constituents in both the mobile and immobile domains.  The dissolved phase is also referred to in this report as the aqueous phase.  If sorption is represented, then the GWT Model also simulates the solid phase of the chemical constituent in both the mobile and immobile domains.  The dissolved and solid phases of the chemical constituent are tracked in the different domains by the GWT Model and can be reported as output as requested by the user.

This section describes the data files for a \mf Groundwater Transport (GWT) Model.  A GWT Model is added to the simulation by including a GWT entry in the MODELS block of the simulation name file.  There are three types of spatial discretization approaches that can be used with the GWT Model: DIS, DISV, and DISU.  The input instructions for these three packages are not described here in this section on GWT Model input; input instructions for these three packages are described in the section on GWF Model input.

The GWT Model is designed to permit input to be gathered, as it is needed, from many different files.  Likewise, results from the model calculations can be written to a number of output files. The GWT Model Listing File is a key file to which the GWT model output is written.  As \mf runs, information about the GWT Model is written to the GWT Model Listing File, including much of the input data (as a record of the simulation) and calculated results.  Details about the files used by each package are provided in this section on the GWT Model Instructions.

The GWT Model reads a file called the Name File, which specifies most of the files that will be used in a simulation. Several files are always required whereas other files are optional depending on the simulation. The Output Control Package receives instructions from the user to control the amount and frequency of output.  Details about the Name File and the Output Control Package are described in this section.

For the GWT Model, ``flows'' (unless stated otherwise) represent solute mass ``flow'' in mass per time, rather than groundwater flow.

\subsection{Information for Existing Solute Transport Modelers}
The \mf GWT Model contains most of the functionality of MODFLOW-GWT, MT3DMS, MT3D-USGS and MODFLOW-USG.  The following list summarizes major differences between the GWT Model in \mf and previous MODFLOW-based solute transport programs.

\begin{enumerate}

\item The GWT Model simulates transport of a single chemical species; however, because \mf allows for multiple models of the same type to be included in a single simulation, multiple species can be represented by using multiple GWT Models.

\item There is no specialized flow and transport link file \citep{zheng2001modflow} used to pass the simulated groundwater flows to the transport model.  Instead, simulated flows from the GWF Model are passed in memory to the GWT Model while the program is running.  Alternatively, the GWT Model can read binary flow and head files saved by the GWF Model while it is running.  If the user intends to simulate transport through the advanced stress packages and Water Mover Package, then flows from these advanced packages must also be saved to binary files.  Names for these binary files are provided as input to the FMI Package.

\item The GWT Model is based on a generalized control-volume finite-difference method, which means that solute transport can be simulated using regular MODFLOW grids consisting of layers, rows, and columns, or solute transport can be simulated using unstructured grids.

\item Advection can be simulated using central-in-space weighting, upstream weighting, or an implicit second-order TVD scheme.  The GWT model does not have the Method of Characteristics (particle-based approaches) or an explicit TVD scheme.  Consequently, the GWT Model may require a higher level of spatial discretization than other transport models that use higher order terms for advection dominated systems.  This can be an important limitation for some problems, which require the preservation of sharp solute fronts.

\item Variable-density flow and transport can be simulated by including a GWF Model and a GWT Model in the same \mf simulation.  The Buoyancy Package should be activated for the GWF Model so that fluid density is calculated as a function of simulated concentration.  If more than one chemical species is represented then the Buoyancy Package allows the simulated concentration for each of them to be used in the density equation of state.   \cite{langevin2020hydraulic} describe the hydraulic-head formation that is implemented in the Buoyancy Package for variable-density groundwater flow and present the results from \mf variable-density simulations.  The variable-density capabilities available in \mf replicate and extend the capabilities available in SEAWAT to include the Newton flow formulation and unstructured grids, for example.

\item The GWT Model has a Source and Sink Mixing (SSM) Package for representing the effects of GWF stress package inflows and outflows on simulated concentrations.  There are two ways in which users can assign concentrations to the individual features in these stress package.  The first way is to activate a concentration auxiliary variable in the GWF stress package.  In the SSM input file, the user provides the name of the auxiliary variable to be used for concentration.  The second way is to create a special SSMI file, which contains user-assigned time-varying concentrations for stress package boundaries.

\item The GWT model includes the MST and IST Packages.  These two package collectively comprise the capabilities of the MT3DMS Reactions Package.

\item The MST Package contains the linear, Freundlich, and Langmuir isotherms for representing sorption.  The IST Packages contains only the linear isotherm for representation of sorption.

\item The GWT model was designed so that the user can specify as many immobile domains and necessary to represent observed contaminant transport patterns and solute breakthrough curves.  The effects of an immobile domain are represented using the Immobile Storage and Transfer (IST) Package, and the user can specify as many IST Packages as necessary.

\item Although there is GWF-GWF Exchange, a GWT-GWT Exchange has not yet been developed to connect multiple transport models, as might be done in a nested grid configuration.

\item There is no option to automatically run the GWT Model to steady state using a single time step.  This is an option available in MT3DMS \citep{zheng2010supplemental}.  Steady state conditions must be determined by running the transport model under transient conditions until concentrations stabilize.

\item The GWT Model described in this report is capable of simulating solute transport in the advanced stress packages of \mf, including the Lake, Streamflow Routing, Multi-Aquifer Well and Unsaturated Zone Transport Packages.  The present implementation simulates solute advection between package features, such as between two stream reaches, but dispersive transport is not represented.  Likewise, solute transport between the advanced packages and the aquifer occurs only through advection.

\item The GWT Model has not yet been programmed to work with the Skeletal Storage, Compaction, and Subsidence (CSUB) Package for the GWF Model.

\item There are many other differences between the \mf GWT Model and other solute transport models that work with MODFLOW, especially with regards to program design and input and output.  Descriptions for the GWT input and output are described here.

\end{enumerate}

\subsection{Units of Length and Time}
The GWF Model formulates the groundwater flow equation without using prescribed length and time units. Any consistent units of length and time can be used when specifying the input data for a simulation. This capability gives a certain amount of freedom to the user, but care must be exercised to avoid mixing units.  The program cannot detect the use of inconsistent units.

\subsection{Solute Mass Budget}
A summary of all inflow (sources) and outflow (sinks) of solute mass is called a mass budget.  \mf calculates a mass budget for the overall model as a check on the acceptability of the solution, and to provide a summary of the sources and sinks of mass to the flow system.  The solute mass budget is printed to the GWT Model Listing File for selected time steps.

\subsection{Time Stepping}

For the present implementation of the GWT Model, all terms in the solute transport equation are solved implicitly.  With the implicit approach applied to the transport equation, it is possible to take relatively large time steps and efficiently obtain a stable solution.  If the time steps are too large, however, accuracy of the model results will suffer, so there is usually some compromise required between the desired level of accuracy and length of the time step.  An assessment of accuracy can be performed by simply running simulations with shorter time steps and comparing results.

In \mf time step lengths are controlled by the user and specified in the Temporal Discretization (TDIS) input file.  When the flow model and transport model are included in the same simulation, then the length of the time step specified in TDIS is used for both models.  If the GWT Model runs in a separate simulation from the GWT Model, then the time steps used for the transport model can be different, and likely shorter, than the time steps used for the flow solution.  Instructions for specifying time steps are described in the TDIS section of this user guide; additional information on GWF and GWT configurations are in the Flow Model Interface section.

\newpage
\subsection{GWT Model Name File}
\input{gwt/namefile.tex}

%\newpage
%\subsection{Structured Discretization (DIS) Input File}
%\input{gwf/dis}

%\newpage
%\subsection{Discretization with Vertices (DISV) Input File}
%\input{gwf/disv}

%\newpage
%\subsection{Unstructured Discretization (DISU) Input File}
%\input{gwf/disu}

\newpage
\subsection{Initial Conditions (IC) Package}
\input{gwt/ic}

\newpage
\subsection{Output Control (OC) Option}
\input{gwt/oc}

\newpage
\subsection{Observation (OBS) Utility for a GWT Model}
\input{gwt/gwt-obs}

\newpage
\subsection{Advection (ADV) Package}
\input{gwt/adv}

\newpage
\subsection{Dispersion (DSP) Package}
\input{gwt/dsp}

\newpage
\subsection{Source and Sink Mixing (SSM) Package}
\input{gwt/ssm}

\newpage
\subsection{Mobile Storage and Transfer (MST) Package}
\input{gwt/mst}

\newpage
\subsection{Immobile Storage and Transfer (IST) Package}
\input{gwt/ist}

\newpage
\subsection{Constant Concentration (CNC) Package}
\input{gwt/cnc}

\newpage
\subsection{Mass Source Loading (SRC) Package}
\input{gwt/src}

\newpage
\subsection{Streamflow Transport (SFT) Package}
\input{gwt/sft}

\newpage
\subsection{Lake Transport (LKT) Package}
\input{gwt/lkt}

\newpage
\subsection{Multi-Aquifer Well Transport (MWT) Package}
\input{gwt/mwt}

\newpage
\subsection{Unsaturated Zone Transport (UZT) Package}
\input{gwt/uzt}

\newpage
\subsection{Flow Model Interface (FMI) Package}
\input{gwt/fmi}

\newpage
\subsection{Mover Transport (MVT) Package}
\input{gwt/mvt}


\subsection{The Golden Rule savings rate}

The Golden Rule savings rate is the rate which maximises long term consumption per capita.

If the savings rate is \(0\) there is no capital and no income. If the savings rate is \(1\) then then there is no consumption.


\chapter{Bounded Refinement Types}\label{boundedrefinements}
\input{text/boundedrefinements/newcommands}

\makequote
{Problems are hidden opportunities, and constraints can actually boost creativity.}
{Martin Villeneuve}

\begin{comment}
\makequote
{The more constraints one imposes, the more one frees one's self. \\
And the arbitrariness of the constraint serves only to obtain precision of execution.}
{Igor Stravinsky}
\end{comment}

\input{text/boundedrefinements/intro}
\input{text/boundedrefinements/overview}
\input{text/boundedrefinements/formalism}
\input{text/boundedrefinements/database}
\input{text/boundedrefinements/state}
\input{text/boundedrefinements/files}
\input{text/boundedrefinements/conclusion}

\mypara{Acknowledgments}
The material of this chapter are adapted from the following publication:
\noindent N. Vazou, A. Bakst, and R. Jhala,
``Bounded Refinement Types'',
ICFP, 2015.


\section{Good code generation}
\subsection*{Usage and liveness}
\textbf{Usage:} Let i < j. The value of x computed at i is used at j. \textbf{Liveness:} Let $i < k$. If value of x computed at i is used at j then x is live at all $i \leqslant j < k$.\\
\textbf{Algorithm:} Instructions block from last to first. $\forall$ instructions $i: x = y op z$: 1. Attach to instruction i the usage and liveness information of x, y, z. 2. Set x to not live and no next use. 3. Set y, z to live and the next uses of y, z to i.\\
\textbf{Datastructure:} Contains additional information in the symbol table, for the algorithm. Or table per basic block. Usage and Liveness per instruction.
\subsection*{Minimizing load and store}
\textbf{Register descriptors:} Variable names whose current value is in a register (per register). \textbf{Address descriptors:} All locations where the current value of a variable is stored.\\
\textbf{Algorithm:} $\forall$ instruction $i: x = y op z$ we do the following. 1. Select registers $R_x$, $R_y$, $R_z$ for the var using $getReg(x = y op z)$. 2. If y is not loaded in $R_y$ then issue instruction. 3. If z is not loaded in $R_z$ then issue instruction. 3. Issue instruction $R_x = R_y op R_z$ at end of block all marked live -> store instruction. Copy no machine-instruction output if $R_x == R_y$\\
\textbf{Algorithm reg for operands:} $R_y$ can be chosen: 1. If y in reg r then $R_y = r$ 2. If y is not in reg but r is empty then $R_y = r$. 3. r is candidate: If all var descriptor says their value is in r also have another location then return r; If only var whose descriptor says value is in r is x and $x \notin \{y,z\}$ then return r; If all var not used ofeter instruction return r; Spill the vars into memory. \textbf{result reg:} reg holding r or those of x,y if not live hereafter.
\subsection*{Peephole}
Sliding window, replaces instructions. Optimizes: redundant-instructions; Flow-of-control; Algebraic; Machine idioms (auto-increments for addresses).
\subsection*{Systematic spilling}
Infinite registers generate its code; Construct a register-interference graph (=k-colouring of graph). Where nodes are registers, vertices are nodes alive together.
\subsection*{Sethi-Ullmann}
\textbf{equal-children:} recursively gen code right child base=b+1, result in $R_{b+k-1}$; do the same for left child with base b result in register one lower; result in $R_{b+k-1}$.\\
\textbf{different-children (m<k):} big child same; result small child in $R_{b+m-1}$; Answer in big child register.\\
\textbf{With spilling:} Big child chosen; Recursive, use b=1, result in $R_r$ then in memory; generating for small child; If label >r use base 1 else b=r-j. Recursive, result in $R_r$; Big child mem in $R_{r-1}$; compute in $R_r$
\subsection*{Pipelining}
1. Instruction fetch 2. Instruction decode and register fetch 3. Execute 4. Memory access 5. Register write back


%%%%%% command used for TensorFlow-Keras Deep neural networks %%%%%%%%%%%%%%%%%
\newcommand{\layerNameAttr}[0]
{
    This node require the following attribute:
    \begin{itemize}
      \item \xmlAttr{name}, \xmlDesc{string, required field}, name of this layer. The value will be
        used in \xmlNode{layer\_layout} to construct the fully connected neural network.
    \end{itemize}
}
%%% Arguments for Dense Layer %%%
\newcommand{\activation}[0]
{
      \item \xmlNode{activation}, \xmlDesc{string, optional field}, including
        {`relu', `tanh', `elu', `selu', `softplus', `softsign', `sigmoid', `hard\_sigmoid', `linear', `softmax'}.
        (see~\ref{activationsDNN})
        \default{linear}
}
\newcommand{\dimOut}[0]
{
      \item \xmlNode{dim\_out}, \xmlDesc{positive integer, required except if this layer is used as the last output layer},
        dimensionality of the output space of this layer
}
\newcommand{\useBias}[0]
{
  \item \xmlNode{use\_bias}, \xmlDesc{boolean, optional field}, whether the layer uses a bias vector.
    \default{True}
}
\newcommand{\kernelInitializer}[0]
{
  \item \xmlNode{kernel\_initializer}, \xmlDesc{string, optional field}, initializer for the kernel weights matrix
    (see~\ref{initializersDNN}).
    \default{glorot\_uniform}
}
\newcommand{\biasInitializer}[0]
{
  \item \xmlNode{bias\_initializer}, \xmlDesc{string, optional field}, intializer for the bias vector
    (see ~\ref{initializersDNN}).
    \default{zeros}
}
\newcommand{\kernelRegularizer}[0]
{
  \item \xmlNode{kernel\_regularizer}, \xmlDesc{string, optional field}, regularizer function applied to
    the kernel weights matrix (see ~\ref{regularizersDNN}).
    \default{None}
}
\newcommand{\biasRegularizer}[0]
{
  \item \xmlNode{bias\_regularizer}, \xmlDesc{string, optional field}, regularizer function applied to the bias vector
    (see~\ref{regularizersDNN}).
    \default{None}
}
\newcommand{\activityRegularizer}[0]
{
  \item \xmlNode{activity\_regularizer}, \xmlDesc{string, optional field}, regularizer function applied to the output
    of the layer (its "activation"). (see~\ref{regularizersDNN})
    \default{None}
}
\newcommand{\kernelConstraint}[0]
{
  \item \xmlNode{kernel\_constraint}, \xmlDesc{string, optional field}, constraint function applied to the kernel weights
    matrix (see~\ref{constraintsDNN}).
    \default{None}
}
\newcommand{\biasConstraint}[0]
{
  \item \xmlNode{bias\_constraint}, \xmlDesc{string, optional field}, constraint function applied to the bias vector
    (see ~\ref{constraintsDNN})
    \default{None}
}
%%% Arguments for Dropout Layer %%%
\newcommand{\rate}[0]
{
      \item \xmlNode{rate}, \xmlDesc{float between 0 and 1, optional field}, fraction of the input units to drop.
        \default{0}
}
\newcommand{\noiseShape}[0]
{
      \item \xmlNode{noise\_shape}, \xmlDesc{list of integers, optional field}, 1D integer tensor representing the shape
        of the binary dropout mask that will be multiplied with the input.
        \default{None}
}
\newcommand{\seed}[0]
{
      \item \xmlNode{seed}, \xmlDesc{integer, optional field}, a integer to use as random seed.
        \default{None}
}
%%% Arguments for LSTM Layer %%%
\newcommand{\recurrentActivation}[0]
{
      \item \xmlNode{recurrent\_activation}, \xmlDesc{string, optional field}, activation function to use for the recurrent
        step, including {`relu', `tanh', `elu', `selu', `softplus', `softsign', `sigmoid', `hard\_sigmoid', `linear', `softmax'}.
        \default{hard\_sigmoid}
}
\newcommand{\recurrentInitializer}[0]
{
      \item \xmlNode{recurrent\_initializer}, \xmlDesc{string, optional field}, used for the linear transformation of
        the recurrent state (see ~\ref{initializersDNN}).
        \default{orthogonal}
}
\newcommand{\unitForgetBias}[0]
{
      \item \xmlNode{unit\_forget\_bias}, \xmlDesc{boolean, optional field}, add 1 to the bias of the forget gate at
        initialization if True.
        \default{True}
}
\newcommand{\recurrentRegularizer}[0]
{
      \item \xmlNode{recurrent\_regularizer}, \xmlDesc{string, optional field}, regularizer function applied to the
        \textit{recurrent\_kernel} weights matrix (see ~\ref{regularizersDNN}).
        \default{None}
}
\newcommand{\recurrentConstraint}[0]
{
      \item \xmlNode{recurrent\_constraint}, \xmlDesc{string, optional field}, constraint function applied to the
        \textit{recurrent\_kernel} weights matrix (see ~\ref{constraints}).
        \default{None}
}
\newcommand{\dropout}[0]
{
      \item \xmlNode{dropout}, \xmlDesc{float between 0 and 1, optional field}, fraction of the units to drop for the linear
        transformation of the inputs
        \default{0}
}
\newcommand{\recurrentDropout}[0]
{
      \item \xmlNode{recurrent\_dropout}, \xmlDesc{float between 0 and 1, optional field}, fraction of the units to drop for the linear
        transformation of the recurrent state.
        \default{0}
}
\newcommand{\returnSequence}[0]
{
      \item \xmlNode{return\_sequence}, \xmlDesc{boolean, optional field}, whether to return the last output in the output sequence, or
        full sequence.
        \default{False}
}
\newcommand{\implementation}[0]
{
      \item \xmlNode{implementation}, \xmlDesc{integer, optional field},
        implementation mode, either 1 or 2. Mode 1 will structure its operations as a larger number of smaller dot products and additions,
        whereas mode 2 will batch them into fewer, larger operations. These modes will have different performance profiles on different
        hardware and for different applications.
        \default{1}
}
\newcommand{\returnState}[0]
{
      \item \xmlNode{return\_state}, \xmlDesc{boolean, optional field},
        whether to return the last output in the output sequence, or the full sequence.
        \default{False}
}
\newcommand{\goBackwards}[0]
{
      \item \xmlNode{go\_backwards}, \xmlDesc{boolean, optional field},
        if True, process the input sequence backwards and return the reversed sequence.
        \default{False}
}
\newcommand{\stateful}[0]
{
      \item \xmlNode{stateful}, \xmlDesc{boolean, optional field},
        if True, the last state for each sample at index i in a batch will be used as initial state for the sample
        of index i in the following batch.
        \default{False}
}
\newcommand{\unroll}[0]
{
      \item \xmlNode{unroll}, \xmlDesc{boolean, optional field},
        if True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN,
        although it tends to be more memory-intensive. Unrolling is only suitable for short sequences.
        \default{False}
}

%%% Arguments for Conv1D Layer %%%

\newcommand{\kernelSize}[0]
{
      \item \xmlNode{kernel\_size}, \xmlDesc{integer or list of integers, required field}, specifying the length
        of the 1D convolution window.
}
\newcommand{\strides}[0]
{
      \item \xmlNode{strides}, \xmlDesc{integer or list of integers, optional field}, pecifying the stride
        length of the convolution. Specifying any stride value not equal 1 is incompatible with specifying any
        dilation\_rate value not equal 1.
        \default{1}
}
\newcommand{\padding}[0]
{
      \item \xmlNode{padding}, \xmlDesc{string, optional field},
        one of "valid", "causal" or "same" (case-insensitive).  "valid" means "no padding".
        "same" results in padding the input such that the output has the same length as the original input.
        "causal" results in causal (dilated) convolutions, e.g. output[t] does not depend on input[t + 1:].
        A zero padding is used such that the output has the same length as the original input. Useful when
        modeling temporal data where the model should not violate the temporal order.
        \default{valid}
}
\newcommand{\dataFormat}[0]
{
      \item \xmlNode{data\_format}, \xmlDesc{string, optional field},
        A string, one of "channels\_last" (default) or "channels\_first". The ordering of the dimensions in the inputs.
        "channels\_last" corresponds to inputs with shape  (batch, steps, channels) (default format for temporal data
        in Keras) while "channels\_first" corresponds to inputs with shape (batch, channels, steps).
        \default{channels\_last}
}
\newcommand{\dilationRate}[0]
{
      \item \xmlNode{dilation\_rate}, \xmlDesc{integer or list of integers, optional field},
        specifying the dilation rate to use for dilated convolution. Currently, specifying any dilation\_rate value
        not equal 1 is incompatible with specifying any strides value not equal 1.
        \default{1}
}

%%% Arguments for Pooling Layer %%%

\newcommand{\poolSize}[0]
{
      \item \xmlNode{pool\_size}, \xmlDesc{integer, required field}, size of the max pooling windows.
        \default{2}
}

\newcommand{\DenseLayer}[0]
{
  \item \xmlNode{Dense}, \xmlDesc{required field}, regular densely-connected neural network layer.
    \layerNameAttr
    In addition, this node also accepts the following subnodes
    \begin{itemize}
        \activation
        \dimOut
        \useBias
        \kernelInitializer
        \biasInitializer
        \kernelRegularizer
        \biasRegularizer
        \activityRegularizer
        \kernelConstraint
        \biasConstraint
    \end{itemize}
}
\newcommand{\DropoutLayer}[0]
{
  \item \xmlNode{Dropout}, \xmlDesc{optional field}, applies Dropout to the input. Dropout consists in
    randomly setting a fraction \xmlNode{rate} of input units to 0 at each update during training time,
    which helps prevent overfitting.
    \layerNameAttr
    In addition, this node also accepts the following subnode
    \begin{itemize}
        \rate
        \noiseShape
        \seed
    \end{itemize}
}

\newcommand{\PoolingLayer}[1]
{

  \item \xmlNode{#1}, \xmlDesc{optional field},
    In addition, this node also accepts the following subnodes
    \begin{itemize}
        \poolSize
        \strides
        \padding
        \dataFormat
    \end{itemize}
}
\newcommand{\GlobalPoolingLayer}[1]
{

  \item \xmlNode{#1}, \xmlDesc{optional field},
    In addition, this node also accepts the following subnodes
    \begin{itemize}
        \dataFormat
    \end{itemize}
}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%% ROM Model - TensorFlow-Keras Interface  %%%%%%%
\subsubsection{TensorFlow-Keras Deep Neural Networks}
\label{subsubsec:TFK_DNNs}

\textcolor{red}{\\It is important to NOTE that Python3 is required in order to use these deep neural networks.
If python2 is installed, these ROMs will not be imported by RAVEN, and an error will be raised if the user tries
to use these capabilities.}

\textbf{TensorFlow} is an open source software library for high performance numerical computation. Its flexible architecture
allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters
of servers to mobile and edge devices. Originally developed by researchers and engineers from the Google Brain team
within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible
numerical computation core is used across many other scientific domains.

\textbf{Keras} is a high-level API to build and train deep learning models. It's used for fast prototyping, advanced research,
and production, with three key advantages:
\begin{itemize}
  \item \textit{User friendly}: Keras has a simple, consistent interface optimized for common use cases.
    It provides clear and actionable feedback for user errors.
  \item \textit{Modular and composable}: Keras models are made by connecting configurable building blocks together,
    with few restrictions.
  \item \textit{Easy to extend}: Write custom building blocks to express new ideas for research. Create new layers,
    loss functions, and develop state-of-the-art models.
\end{itemize}

\textbf{tf.keras} is TensorFlow's implementation of the Keras API specification. This is a high-level API to build and train
models that include first-class support for TensorFlow-specific functionality, such as eager \textit{execution},
\textit{tf.data} pipelines, and \textit{Estimators}. \textbf{tf.keras} makes TensorFlow easier to use without sacrificing
flexibility and performance. RAVEN will utilize this high-level API to build and train deep neural networks (DNNs) as ROMs, and
these ROMs can be employed by other RAVEN entities to perform uncertainty quantification, model opimization and data analysis.

Before analyzing each classifier in detail, it is important to mention that each type has a similar syntax. In the
example below, the subnodes that can be included in the main XML node \xmlNode{ROM} are reported:
\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={name,subType}]
<Simulation>
  ...
  <Models>
    ...
    <ROM name='aUserDefinedName' subType='whatever'>
      <Features>X,Y</Features>
      <Target>Z</Target>
      <loss>mean_squared_error</loss>
      <metrics>accuracy</metrics>
      <batch_size>4</batch_size>
      <epochs>4</epochs>
      <num_classes>2</num_classes>
      <validation_split>0.25</validation_split>
      <optimizerSetting>
        <optimizer>Adam</optimizer>
        ...
      </optimizerSetting>
      <WhateverLayer1 name="layerName1">
        ...
      </WhateverLayer1>
      ...
      <WhateverLayerN name="layerNameN">
        ...
      </WhateverLayerN>
      <layer_layout>layerName1, ..., layerNameN</layer_layout>
    </ROM>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

As shown in above example, in addition to the common subnodes \xmlNode{Target} and \xmlNode{Features}, the \xmlNode{ROM} of DNNs
can be initialized with the following children:
\begin{itemize}
  \item \xmlNode{loss}, \xmlDesc{string or comma separated string, optional field}, if the model has multiple outputs, you can use a different
    loss metric on each output by passing a list of loss metrics. The value that will be minimized by the model will then
    be the sum of all individual value from each loss metric. Available loss functions include \textit{mean\_squared\_error},
    \textit{mean\_absolute\_error}, \textit{mean\_absolute\_percentage\_error}, \textit{mean\_squared\_logarithmic\_error},
    \textit{squared\_hinge}, \textit{hinge}, \textit{categorical\_hinge}, \textit{logcosh}, \textit{categorical\_crossentropy},
    \textit{sparse\_categorical\_crossentropy}, \textit{binary\_crossentropy}, \textit{kullback\_leibler\_divergence},
    \textit{poisson}, \textit{cosine\_proximity}.
  \default{mean\_squared\_error}
  \item \xmlNode{metrics}, \xmlDesc{string or comma separated string, optional field}, list of metrics to be evaluated by
    the model during training and testing. available metrics include
    \textit{binary\_accuracy}, \textit{categorical\_accuracy}, \textit{sparse\_categorical\_accuracy},
    \textit{top\_k\_categorical\_accuracy}, \textit{sparse\_top\_k\_categorical\_accuracy}.
  \default{accuracy}
  \item \xmlNode{batch\_size}, \xmlDesc{integer, optional field}, number of samples per gradient update.
  \default{20}
  \item \xmlNode{epochs}, \xmlDesc{integer, optional field}, number of epochs to train the model. An epoch
    is an iteration over the entire training data.
  \default{20}
  \item \xmlNode{num\_classes}, \xmlDesc{positive integer, optional field}, dimensionality of the output space of given classifier.
  \default{1}
  \item \xmlNode{validation\_split}, \xmlDesc{float between 0 and 1, optional field}, fraction of the training data to
    be used as validation data.
  \default{0.25}
  \item \xmlNode{plot\_model}, \xmlDesc{boolean, optional field}, if true the DNN model constructed by RAVEN will be
    plotted and stored in the working directory. The file name will be \textit{"ROM name" + "\_" + "model.png"}.
    \nb This capability requires the following libraries, i.e. pydot-ng and graphviz to be installed.
  \default{False}
  \item \xmlNode{optimizerSetting}, \xmlDesc{optional field}, including several subnode depending on the type of
    optimizers.
    \begin{itemize}
      \item \xmlNode{optimizer}, \xmlDesc{string, optional field}, name of optimizer.
    \end{itemize}
    \default{Adam}
    \nb The users can also choose different optimizers to train the ROM. The default algorithm is \textit{Adam}.
    Other available optimizers include:
    \textit{SGD}, \textit{RMSprop}, \textit{Adagrad}, \textit{Adadelta}, \textit{Adamx}, \textit{Nadam}.
    For the detailed information, i.e. the parameters for each optimization, the user can refer to
    \url{https://keras.io/optimizers/}. In raven, the user can use \xmlNode{optimizerSetting} to set the
    parameters of the above optimizer as follows:
    \begin{itemize}
      \item \textbf{Adam}, adam optimizer
        \begin{itemize}
          \item \xmlNode{beta\_1}, \xmlDesc{float, optional field}, $0 < beta < 1$. Generally close to 1.
          \default{0.9}
          \item \xmlNode{beta\_2}, \xmlDesc{float, optional field}, $0 < beta < 1$. Generally close to 1.
          \default{0.999}
          \item \xmlNode{epsilon}, \xmlDesc{float, optional field}, fuzz factor.
          \default{None}
          \item \xmlNode{decay}, \xmlDesc{float, optional field}, learning rate decay over each update.
          \default{0.0}
          \item \xmlNode{lr}, \xmlDesc{float, optional field}, learning rate.
          \default{0.001}
        \end{itemize}
    %
      \item \textbf{SGD}, stochastic gradient descent optimizer.
        \begin{itemize}
          \item \xmlNode{momentum}, \xmlDesc{float, optional field}, $> 0$. Parameter that accelerates SGD in
            the relevant direction and dampens oscillations.
          \default{0.0}
          \item \xmlNode{nesterov}, \xmlDesc{boolean, optional field}, whether to apply Nesterov momentum
          \default{False}
          \item \xmlNode{decay}, \xmlDesc{float, optional field}, learning rate decay over each update.
          \default{0.0}
          \item \xmlNode{lr}, \xmlDesc{float, optional field}, learning rate.
          \default{0.001}
        \end{itemize}
    %
      \item \textbf{RMSprop}, RMSProp optimizer.
        \begin{itemize}
          \item \xmlNode{rho}, \xmlDesc{float, optional field}, $> 0$.
          \default{0.9}
          \item \xmlNode{decay}, \xmlDesc{float, optional field}, learning rate decay over each update.
          \default{0.0}
          \item \xmlNode{lr}, \xmlDesc{float, optional field}, learning rate.
          \default{0.001}
          \item \xmlNode{epsilon}, \xmlDesc{float, optional field}, fuzz factor.
          \default{None}
        \end{itemize}
    %
      \item \textbf{Adagrad}, Adagrad optimizer.
        \begin{itemize}
          \item \xmlNode{decay}, \xmlDesc{float, optional field}, learning rate decay over each update.
          \default{0.0}
          \item \xmlNode{lr}, \xmlDesc{float, optional field}, learning rate.
          \default{0.01}
          \item \xmlNode{epsilon}, \xmlDesc{float, optional field}, fuzz factor.
          \default{None}
        \end{itemize}
    %
      \item \textbf{Adadelta}, Adadelta optimizer.
        \begin{itemize}
          \item \xmlNode{decay}, \xmlDesc{float, optional field}, learning rate decay over each update.
          \default{0.0}
          \item \xmlNode{lr}, \xmlDesc{float, optional field}, learning rate.
          \default{1.0}
          \item \xmlNode{epsilon}, \xmlDesc{float, optional field}, fuzz factor.
          \default{None}
          \item \xmlNode{rho}, \xmlDesc{float, optional field}, $> 0$.
          \default{0.95}
        \end{itemize}
    %
      \item \textbf{Adamax}, Adamax optimizer
        \begin{itemize}
          \item \xmlNode{beta\_1}, \xmlDesc{float, optional field}, $0 < beta < 1$. Generally close to 1.
          \default{0.9}
          \item \xmlNode{beta\_2}, \xmlDesc{float, optional field}, $0 < beta < 1$. Generally close to 1.
          \default{0.999}
          \item \xmlNode{epsilon}, \xmlDesc{float, optional field}, fuzz factor.
          \default{None}
          \item \xmlNode{decay}, \xmlDesc{float, optional field}, learning rate decay over each update.
          \default{0.0}
          \item \xmlNode{lr}, \xmlDesc{float, optional field}, learning rate.
          \default{0.002}
        \end{itemize}
    %
      \item \textbf{Nadam},
        \begin{itemize}
          \item \xmlNode{beta\_1}, \xmlDesc{float, optional field}, $0 < beta < 1$. Generally close to 1.
          \default{0.9}
          \item \xmlNode{beta\_2}, \xmlDesc{float, optional field}, $0 < beta < 1$. Generally close to 1.
          \default{0.999}
          \item \xmlNode{epsilon}, \xmlDesc{float, optional field}, fuzz factor.
          \default{None}
          \item \xmlNode{lr}, \xmlDesc{float, optional field}, learning rate.
          \default{0.002}
        \end{itemize}
  \end{itemize}
  \item \xmlNode{layer\_layout}, \xmlDesc{comma seperated string, required}, the layout/order of layers in the
    deep neural networks. The values in the subnode should be the name of layers defined in layer node, such as
    \xmlNode{Dense}, \xmlNode{Dropout}, and \xmlNode{Conv1D}.
\end{itemize}

\nb The descriptions regarding the \xmlNode{WhateverLayer} node will be introduced in following subsections.
Basically, different classifiers will require different layers.
In addition, most core layers will accept the \xmlNode{activation} subnode (see ~\ref{activationsDNN}).

%%%%% Activation Functions  %%%%%%%
\paragraph{Activation Functions}
\label{activationsDNN}
Activations can either be used through an \xmlNode{Activation} layer, or through the
\xmlNode{activation} argument supported by all forward layers.
Available activations include:
\begin{itemize}
  \item \textit{relu}, the rectified linear unit function, returns $f(x) = max(0, x)$.
  \item \textit{tanh}, the hyperbolic tan function, returns $f(x) = tanh(x)$.
  \item \textit{elu}, exponential linear units try to make the mean activations closer to zero which speeds
    up learning. $f(x) = x$ if $x \ge 0$, otherwise $(exp(x) - 1.)$.
  \item \textit{selu}, scaled exponential linear unit, i.e. $scale * elu(x, alpha)$, where $scale, alpha$
    are pre-defined constants.
  \item \textit{softplus}, a smooth approximation to the rectifier linear unit function, return
    $f(x) = log(1. + exp(x))$.
  \item \textit{softsign}, return $f(x) = \frac{x}{1. + |x|}$.
  \item \textit{sigmoid},return $f(x) = \frac{1.}{1. + exp(-x)}$.
  \item \textit{hard\_sigmoid}, hard sigmoid activation function.
  \item \textit{linear}, i.e. identity.
  \item \textit{softmax}, softmax activation function, return $f(x) = \frac{exp(x_i)}{\sum_i{exp(x_i)}}$
\end{itemize}

%%%%% Initializer Functions  %%%%%%%
\paragraph{Initializer Functions}
\label{initializersDNN}
Initializations define the way to set the initial random weights of TensorFlow-Keras layers. The keyword
arguments used to passing initializers to layers will depend on the layer. Usually it is simply
\xmlNode{kernel\_initializer} and \xmlNode{bias\_initializer}.
Available initializers include:
\begin{itemize}
  \item \textit{Zeros}, generates tensors initialized to 0.
  \item \textit{Ones}, generates tensors initialized to 1.
  \item \textit{Constant}, generates tensors initialized to a constant value.
  \item \textit{RandomNormal}, generates tensors with a normal distribution.
  \item \textit{RandomUniform}, generates tensors with a uniform distribution.
  \item \textit{TruncatedNormal}, generates a truncated normal distribution.
  \item \textit{VarianceScaling}, initializer capable of adapting its scale to the shape of weights.
  \item \textit{Orthogonal}, generates a random orthogonal matrix.
  \item \textit{Identity}, generates the identity matrix.
  \item \textit{lecun\_uniform}, LeCun uniform initializer.
    It draws samples from a uniform distribution within
    $[-limit, limit]$ where \textit{limit} is $sqrt(3/fanIn)$ where \textit{fanIn} is the number of input dimensions
    in the weight tensor.
  \item \textit{glorot\_normal}, Glorot normal initializer.
    It draws samples from a truncated normal distribution
    centered on 0 with $stddev = sqrt(2/(fanIn + fanOut))$ where \textit{fanIn} is the number of input dimensions
    in the weight tensor and \textit{fanOut} is the number of output dimensions in the weight tensors.
  \item \textit{glorot\_uniform}, Glorot uniform initializer.
    It draws samples from a uniform distribution within
    $[-limit, limit]$ where \textit{limit} is $sqrt(6/(fanIn+fanOut))$.
  \item \textit{he\_normal}, He normal initializer.
    It draws samples from a truncated normal distribution
    centered on 0 with $stddev = sqrt(2/fanIn)$.
  \item \textit{lecun\_normal}, LeCun normal initializer.
    It draws samples from a truncated normal distribution
    centered on 0 with $stddev = sqrt(1/fanIn)$.
  \item \textit{he\_uniform}, He uniform variance scaling initializer.
    It draws samples from a uniform distribution within
    $[-limit, limit]$ where \textit{limit} is $sqrt(6/fanIn)$ where \textit{fanIn} is the number of input dimensions
    in the weight tensor.
\end{itemize}

%%%%% Rgularizer Functions  %%%%%%%
\paragraph{Regularizer Functions}
\label{regularizersDNN}
Regularizers allow to apply penalities on layer parameters or layer activity during optimization.
These penalties are incorporated in the loss function that the network optimizes. The exact API
will depend on the layer, but the layers \xmlNode{Dense, Conv1D, Conv2D, and Conv3D} have a
unified API.
Available regularizers include:
\begin{itemize}
  \item \textit{l1}, l1 regularization
  \item \textit{l2}, l2 regularization
  \item \textit{l1\_l2}, l1 and l2 regularization
\end{itemize}

%%%%% Constraint Functions  %%%%%%%
\paragraph{Constraint Functions}
\label{constraintsDNN}
Functions from the \textit{constraint} module allow setting constraints on network parameters during optimization.
Available constraints include:
\begin{itemize}
  \item \textit{MaxNorm}, constrains the weights incident to each hidden unit to have a norm less than or equal to
    a desired value.
  \item \textit{NonNeg}, constrains the weights to be non-negative
  \item \textit{UnitNorm}, constrains the weights incident to each hidden unit to have unit norm.
  \item \textit{MinMaxNorm}, constrains the weights incident to each hidden unit to have the norm between a lower bound
    and an upper bound.
\end{itemize}

%%%%% ROM Model - KerasMLPClassifier  %%%%%%%
\paragraph{KerasMLPClassifier}
\label{KerasMLPClassifier}

Multi-Layer Perceptron (MLP) (or Artificial Neural Network - ANN), a class of feedforward
ANN, can be viewed as a logistic regression classifier where input is first transformed
using a non-linear transformation. This transformation probjects the input data into a
space where it becomes linearly separable. This intermediate layer is referred to as a
\textbf{hidden layer}. An MLP consists of at least three layers of nodes. Except for the
input nodes, each node is a neuron that uses a nonlinear \textbf{activation function}. MLP
utilizes a suppervised learning technique called \textbf{Backpropagation} for training.
Generally, a single hidden layer is sufficient to make MLPs a universal approximator.
However, many hidden layers, i.e. deep learning, can be used to model more complex nonlinear
relationships. The extra layers enable composition of features from lower layers, potentially
modeling complex data with fewer units than a similarly performing shallow network.

\zNormalizationPerformed{KerasMLPClassifier}

In order to use this ROM, the \xmlNode{ROM} attribute \xmlAttr{subType} needs to
be \xmlString{KerasMLPClassifier} (see the example below). This model can be initialized with
the following layers:

\begin{itemize}
  \DenseLayer
  \DropoutLayer
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={name,subType}]
<Simulation>
  ...
  <Models>
    ...
    <ROM name='aUserDefinedName' subType='KerasMLPClassifier'>
      <Features>X,Y</Features>
      <Target>Z</Target>
      <loss>mean_squared_error</loss>
      <metrics>accuracy</metrics>
      <batch_size>4</batch_size>
      <epochs>4</epochs>
      <optimizerSetting>
        <beta_1>0.9</beta_1>
        <optimizer>Adam</optimizer>
        <beta_2>0.999</beta_2>
        <epsilon>1e-8</epsilon>
        <decay>0.0</decay>
        <lr>0.001</lr>
      </optimizerSetting>
      <Dense name="layer1">
          <activation>relu</activation>
          <dim_out>15</dim_out>
      </Dense>
      <Dropout name="dropout1">
          <rate>0.2</rate>
      </Dropout>
      <Dense name="layer2">
          <activation>tanh</activation>
          <dim_out>8</dim_out>
      </Dense>
      <Dropout name="dropout2">
          <rate>0.2</rate>
      </Dropout>
      <Dense name="outLayer">
          <activation>sigmoid</activation>
      </Dense>
      <layer_layout>layer1, dropout1, layer2, dropout2, outLayer</layer_layout>
    </ROM>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

%%%%% ROM Model - KerasConvNetClassifier  %%%%%%%
\paragraph{KerasConvNetClassifier}
\label{KerasClassifier}

Convolutional Neural Network (CNN) is a deep learning algorithm which can take in an input image, assign
importance to various objects in the image and be able to differentiate one from the other. The
architecture of a CNN is analogous to that of the connectivity pattern of Neurons in the Human Brain
and was inspired by the organization of the Visual Cortex. Individual neurons respond to stimuli only
in a restricted region of the visual field known as the Receptive Field. A collection of such fields
overlap to cover the entire visual area. CNN is able to successfully capture the spatial and temporal
dependencies in an image through the applicaiton of relevant filters. The architecture performs
a better fitting to the image dataset due to the reduction in the number of parameters involved
and reusability of weights. In other words, the network can be trained to understand the sophistication
of the image better.

\zNormalizationPerformed{KerasConvNetClassifier}

In order to use this ROM, the \xmlNode{ROM} attribute \xmlAttr{subType} needs to
be \xmlString{KerasConvNetClassifier} (see the example below). This model can be initialized with
the following layers:

\begin{itemize}
  \DenseLayer
  \DropoutLayer
  \item \xmlNode{Conv1D}, \xmlDesc{optional field},
    \layerNameAttr
    In addition, this node also accepts the following subnodes
    \begin{itemize}
        \activation
        \dimOut
        \useBias
        \kernelSize
        \strides
        \padding
        \dataFormat
        \dilationRate
        \kernelInitializer
        \biasInitializer
        \kernelRegularizer
        \biasRegularizer
        \activityRegularizer
        \kernelConstraint
        \biasConstraint
    \end{itemize}

  \item \xmlNode{Conv2D}, \xmlDesc{optional field},
    In addition, this node also accepts the following subnodes
    \begin{itemize}
        \activation
        \dimOut
        \useBias
        \kernelSize
        \strides
        \padding
        \dataFormat
        \dilationRate
        \kernelInitializer
        \biasInitializer
        \kernelRegularizer
        \biasRegularizer
        \activityRegularizer
        \kernelConstraint
        \biasConstraint
    \end{itemize}

  \item \xmlNode{Conv3D}, \xmlDesc{optional field},
    In addition, this node also accepts the following subnodes
    \begin{itemize}
        \activation
        \dimOut
        \useBias
        \kernelSize
        \strides
        \padding
        \dataFormat
        \dilationRate
        \kernelInitializer
        \biasInitializer
        \kernelRegularizer
        \biasRegularizer
        \activityRegularizer
        \kernelConstraint
        \biasConstraint
    \end{itemize}

  \item \xmlNode{Flatten}, \xmlDesc{optional field},
    In addition, this node also accepts the following subnodes
    \begin{itemize}
        \dataFormat
    \end{itemize}

  \PoolingLayer{MaxPooling1D}
  \PoolingLayer{MaxPooling2D}
  \PoolingLayer{MaxPooling3D}
  \PoolingLayer{AveragePooling1D}
  \PoolingLayer{AveragePooling2D}
  \PoolingLayer{AveragePooling3D}
  \GlobalPoolingLayer{GlobalMaxPooling1D}
  \GlobalPoolingLayer{GlobalMaxPooling2D}
  \GlobalPoolingLayer{GlobalMaxPooling3D}
  \GlobalPoolingLayer{GlobalAveragePooling1D}
  \GlobalPoolingLayer{GlobalAveragePooling2D}
  \GlobalPoolingLayer{GlobalAveragePooling3D}
\end{itemize}

\textbf{Example:}
\begin{lstlisting}[style=XML,morekeywords={name,subType}]
<Simulation>
  ...
  <Models>
    ...
    <ROM name='aUserDefinedName' subType='KerasConvNetClassifier'>
      <Features>x1,x2</Features>
      <Target>labels</Target>
      <loss>mean_squared_error</loss>
      <metrics>accuracy</metrics>
      <batch_size>1</batch_size>
      <epochs>2</epochs>
      <plot_model>True</plot_model>
      <validation_split>0.25</validation_split>
      <num_classes>1</num_classes>
      <optimizerSetting>
        <beta_1>0.9</beta_1>
        <optimizer>Adam</optimizer>
        <beta_2>0.999</beta_2>
        <epsilon>1e-8</epsilon>
        <decay>0.0</decay>
        <lr>0.001</lr>
      </optimizerSetting>
      <Conv1D name="firstConv1D">
          <activation>relu</activation>
          <strides>1</strides>
          <kernel_size>2</kernel_size>
          <padding>valid</padding>
          <dim_out>32</dim_out>
      </Conv1D>
      <MaxPooling1D name="pooling1">
          <strides>2</strides>
          <pool_size>2</pool_size>
      </MaxPooling1D>
      <Conv1D name="SecondConv1D">
          <activation>relu</activation>
          <strides>1</strides>
          <kernel_size>2</kernel_size>
          <padding>valid</padding>
          <dim_out>32</dim_out>
      </Conv1D>
      <MaxPooling1D name="pooling2">
          <strides>2</strides>
          <pool_size>2</pool_size>
      </MaxPooling1D>
      <Flatten name="flatten">
      </Flatten>
      <Dense name="dense1">
          <activation>relu</activation>
          <dim_out>10</dim_out>
      </Dense>
      <Dropout name="dropout1">
          <rate>0.25</rate>
      </Dropout>
      <Dropout name="dropout2">
          <rate>0.25</rate>
      </Dropout>
      <Dense name="dense2">
          <activation>softmax</activation>
      </Dense>
      <layer_layout>firstConv1D, pooling1, dropout1, SecondConv1D, pooling2, dropout2, flatten, dense1, dense2</layer_layout>
    </ROM>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

%%%%% ROM Model - KerasLSTMClassifier  %%%%%%%
\paragraph{KerasLSTMClassifier and KerasLSTMRegression}
\label{KerasClassifier}

Long Short Term Memory networks (LSTM) are a special kind of recurrent neural network, capable
of learning long-term dependencies. They work tremendously well on a large variety of problems, and
are now widely used. LSTMs are explicity designed to avoid the long-term dependency problem. Remembering
information for long periods of time is practically their default behavior, not something that they
struggle to learn.

LSTM's can be used for either classification (with
\xmlString{KerasLSTMClassifier}) or prediction of values (with
\xmlString{KerasLSTMRegression}).

\zNormalizationPerformed{KerasLSTMClassifier \textup{and} KerasLSTMRegression}

In order to use this ROM, the \xmlNode{ROM} attribute \xmlAttr{subType} needs to
be \xmlString{KerasLSTMClassifier} or \xmlString{KerasLSTMRegression} (see the examples below). This model can be initialized with
the following layers:

\begin{itemize}
  \DenseLayer
  \DropoutLayer
  \item \xmlNode{LSTM}, \xmlDesc{required field}, long short-term memory layer.
    \layerNameAttr
    In addition, this node also accepts the following subnodes
    \begin{itemize}
        \activation
        \dimOut
        \recurrentActivation
        \dropout
        \recurrentDropout
        \returnSequence
        \useBias
        \kernelInitializer
        \recurrentInitializer
        \biasInitializer
        \unitForgetBias
        \kernelRegularizer
        \recurrentRegularizer
        \biasRegularizer
        \activityRegularizer
        \kernelConstraint
        \recurrentConstraint
        \biasConstraint
        \implementation
        \returnState
        \goBackwards
        \stateful
        \unroll
    \end{itemize}
\end{itemize}

\textbf{KerasLSTMClassifier Example:}
\begin{lstlisting}[style=XML,morekeywords={name,subType}]
<Simulation>
  ...
  <Models>
    ...
    <ROM name='aUserDefinedName' subType='KerasLSTMClassifier'>
      <Features>x</Features>
      <Target>y</Target>
      <loss>categorical_crossentropy</loss>
      <metrics>accuracy</metrics>
      <batch_size>1</batch_size>
      <epochs>10</epochs>
      <validation_split>0.25</validation_split>
      <num_classes>26</num_classes>
      <optimizerSetting>
        <beta_1>0.9</beta_1>
        <optimizer>Adam</optimizer>
        <beta_2>0.999</beta_2>
        <epsilon>1e-8</epsilon>
        <decay>0.0</decay>
        <lr>0.001</lr>
      </optimizerSetting>
      <LSTM name="lstm1">
          <activation>tanh</activation>
          <dim_out>32</dim_out>
      </LSTM>
      <LSTM name="lstm2">
          <activation>tanh</activation>
          <dim_out>16</dim_out>
      </LSTM>
      <Dropout name="dropout">
          <rate>0.25</rate>
      </Dropout>
      <Dense name="dense">
          <activation>softmax</activation>
      </Dense>
      <layer_layout>lstm1,lstm2,dropout,dense</layer_layout>
    </ROM>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}

\textbf{KerasLSTMRegression Example:}
\begin{lstlisting}[style=XML,morekeywords={name,subType}]
<Simulation>
  ...
  <Models>
    ...
    <ROM name="lstmROM" subType="KerasLSTMRegression">
      <Features>prev_sum, prev_square, prev_square_sum</Features>
      <Target>sum, square</Target>
      <pivotParameter>index</pivotParameter>
      <loss>mean_squared_error</loss>
      <LSTM name="lstm1">
        <dim_out>32</dim_out>
      </LSTM>
      <LSTM name="lstm2">
        <dim_out>16</dim_out>
      </LSTM>
      <Dense name="dense">
      </Dense>
      <layer_layout>lstm1, lstm2, dense</layer_layout>

    </ROM>
    ...
  </Models>
  ...
</Simulation>
\end{lstlisting}


Train-dev is Baobab data. Validation in Rung 1 data.

\subsection{Improving the source position prediction}

Establish sensitivity to bias in the source position as demonstrated in \cite{birrer2019astrometric}. See Table \ref{table:source_pos}. For each method, we can show the image positions inferred from the predicted source positions on the same lens system, for a visualization of which model does best on recovering the true image positions.

\begin{table*}
\label{table:source_pos}
\caption{Summary of experiments on source position recovery\footnotesize }
\centering
\begin{tabular}{l|l|SSSSSSSSS}
    \toprule
    \multirow{2}{*}{Experiment} &
    \multirow{2}{*}{Method} &
      \multicolumn{2}{c}{Train} &
      \multicolumn{2}{c}{Validation} &
      \multicolumn{3}{c}{Performance} \\
      &
      &
      {N} & {Time (hr)} &
      {N} & {Time (hr)} &
      {Precision} &
      {Accuracy} &
      {$\chi^2$} \\
      \midrule
    A1 & raw BNN
    & {N} & t
    & {-} & t
   \\
    A2 & transformed input
    & {N} & t
    & {-} & t
   \\
    A3 & regression with image positions
    & {N} & t
    & {-} & t
    \\
    A4 & inbuilt magnification matrix
    & {N} & t
    & {-} & t
     \\
    A5 & hybrid forward modeling
    & {N} & t
    & {-} & t
     \\
    \bottomrule
  \end{tabular}
  %\begin{flushleft}
  %Experiments varied in key features of the dataset: the proportion of %positives, generation procedure (emulated or fully simulated), sky %coverage, and cadence (WFD or DDF).
  %\end{flushleft}
\end{table*}%

\subsection{Point spread function}
Implicit in the input images of the training set is the shape of the point spread function (PSF). We can also experiment with explicitly parameterizing drizzled HST-like point spread functions, and have the network predict the parameters.

\subsection{Dataset size}
Establish that, as the number of predicted macromodel parameters increases, the parameter space grows exponentially (curse of dimensionality). Demonstrating that a training set that reasonably spans the space of the assumed conditional PDF is key to demonstrating the usefulness of any ML method.

Increase the dataset size with the same data configuration (maybe the best combination of cross-validated hyperparameters) and observe the convergence with increasing number of images. Do this for both diagonal and empirical conditional PDFs.

More important than convergence is the target precision.

\subsection{Sensitivity tests}
\subsubsection{Priors on $\kappa_{ext}$ and $r_{ani}$}
Show a 3-by-3 cornerplot of the three metrics (like the one from the TDLMC) for the following combinations of priors on $\kappa_{ext}$ and $r_{ani}$. Each experiment takes a different shape or color.
\begin{enumerate}
    \item $\kappa_{ext} \sim N(0, 0.025^2)$ as used in \cite{ding2018time}
    \item $\kappa_{ext} \sim N(0, 0.05^2)$
    \item $\kappa_{ext} \sim N(0, 0.1^2)$
\end{enumerate}
\begin{enumerate}
    \item $r_{ani} = r_{eff}$ as used in \cite{ding2018time}
    \item $r_{ani} \sim U(0.5, 2)*r_{eff}$ as used in \cite{birrer2016mass}
\end{enumerate}

\subsubsection{Noise level}
Some effective noise level increasing from 0 to the level of (some benchmark real data noise). Per-pixel SNR

\subsubsection{Effect of image configuration}

\subsubsection{Comparison to traditional forward-modeling}
Quote precision
How BNN performance scales with
3-4 pixels
Moffat, with PSF FWHM

increase exposure time


\documentclass[aspectratio=169]{beamer}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{listings}
\title[How to use \LaTeX for GUADEC 2015]{How to use \LaTeX for creating awesome slides for GUADEC 2015}
\author{Adrien Plazas \texttt{kekun.plazas@laposte.net}}
\date{August 7, 2015}

\usetheme{guadec}

\AtBeginSection{\frame{\sectionpage}}

\begin{document}

\begin{frame}
\titlepage
\end{frame}

\section{Title Slide}

\begin{frame}[containsverbatim]
\frametitle{Content slide}
\framesubtitle{With a subtitle}

    Do math simply!

    \begin{center}
        $\sqrt{5} \neq 1$
    \end{center}

    \texttt{Use} the $^{power}$ of \LaTeX!

    \begin{block}{With Lower Level Headers}
        \begin{lstlisting}[language=Python]
def some_code(is_easy_to):
    integrate()

# With syntax highlighting, of course!
        \end{lstlisting}
    \end{block}
\end{frame}

\begin{frame}
\frametitle{Incrementally Showing Lists}

    \begin{itemize}
        \item<1-> Is
        \item<2-| alert@2> Dead
        \item<3-> Easy!
        \item<1-> Yet flexible
    \end{itemize}
\end{frame}

\section{Compiling}

\begin{frame}
\frametitle{Compiling}

    Simply use \texttt{make tex} in this directory!
\end{frame}

\end{document}


\section{F-test}


\paragraph{}
The IGES\citep{IGES1983} files introduced in Sec.~\ref{lr_sec:IGES} are used as the bridge between the engineering design and the numerical analysis in the proposed method.
As a standard file format in engineering design industry, it is supported by almost all design softwares all over the world.
Consequently, it offers a possibility to minimize the human efforts spent on geometric modeling.

\subsection{Parse geometry in IGES file}
\paragraph{}
The IGES file provides all information that describe the geometric input.
Abstract structure of the IGES file that describes the 2D geometry can be regarded as a simple curve-surface structure.
In other words, it defines the geometric input as certain number of surfaces with their boundaries in different colors, which represents different materials in engineering practice.
Each surface contains a list of curve indices which are corresponding to the curve information described in IGES file.

\paragraph{}
When the IGES file is feed into the programme, each line in the directory entry will be parsed entity by entity.
Parameters in directory entry describe the type and reference to other useful values for the entity.
An entity may refer to one or many entities in directory entry.
Detail of the specification of each entity is explained in \cite{Nasr2007}.

%=====================================================================================================================%
\subsection{Output to mesh generator}
\label{qdt_section:iges_output}
\paragraph{}
The output file that the mesh generator read in will be a shorter summary of the geometric input.
Boundary representation will be kept as the data structure to describe the geometric input.
However, some fields apart from the curves or the surfaces will be added.
In the output file, the geometry will be organized in key points, polylines and surfaces.
The key points define the coordinates of all points and polylines are used to represent all curves including NURBS curves for simplification.
NURBS curves introduced in in \ref{lr_sec:NURBS} can be used directly by the mesh generator as well but it is found that the computational cost in the calculation of the distance between points to NURBS curves surpasses the merits of using it directly.
The exact geometry can be retained by projecting the nodes on the boundary back to the origin NURBS curves after the mesh is generated.

\paragraph{}
Representing a straight line with polyline can be natural, first and the last points will be enough to achieve an exact representation.
However, that is not the case for curves whose curvature is not always zero.
Although adding more key points can increase the quality of polyline representation, computational cost in calculating the distance increases at the same time.
Yet, having only few key points may result in a bad polyline representation which leads to a poor mesh quality.
Elements may be twisted after the nodes on the boundary are projected to the origin curves.
As a consequence, a quantified indicator that is able to control the number and the position of the vertices on the polyline can be necessary.

%=====================================================================================================================%
\subsection{Discretization of the circular arc}
\paragraph{}
The chord ratio can be a good indicator for circular arc.
The arc length to chord length ratio $\frac{a}{L}$ illustrated in Fig.~\ref{qt_fig:iges_chord_ratio} can be expressed in angle $\alpha$ as:
    \begin{equation}
        \frac{a}{L} = \frac{
            \sqrt{2-2\cos\alpha}
        }{\alpha}
        = \frac{
            2\sin\frac{\alpha}{2}
        }{\alpha}
    \end{equation}
%
The maximum angle $\alpha$ satisfy arc length to chord length ratio $\frac{a}{L} > 1-\epsilon$ with $
    \sin\frac{\alpha}{2} = 1 - \frac{x^3}{6} + O(x^7)
$ can be derived as:
    \begin{equation}
        \alpha < \sqrt{24 \epsilon}
    \end{equation}

    \begin{figure}[h!]
        \centering
        \scalebox{1}{
            \includegraphics{quadtree/images/iges_chord_ratio.eps}
        }
        \caption{Chord length for circular arc}
        \label{qt_fig:iges_chord_ratio}
    \end{figure}
%
%=====================================================================================================================%
\subsection{Discretization of the NURBS curve}
\paragraph{}
For NURBS curves who have no closed form chord ratio, similar idea can be applied numerically.
The NURBS curve is first divided into serval smaller ones based on the knot vector as described in Sec.~\ref{lr_sec:nurbs_knot_ins}.  % may be explained in detail
Since the order of each subdivided NURBS curve used in engineering softwares are predominantly lower or equal to three, the sub-curves then can be divided into two classes, convex curves or concave curves with an inflection point as shown in Fig.~\ref{qt_fig:iges_chord_ratio_nurbs}.
    \begin{figure}
        \begin{subfigure}[b]{0.5\linewidth}
            \centering
            \scalebox{0.5}{
                \includegraphics{quadtree/images/iges_chord_ratio_nurbs_convex.eps}
            }
            \caption{Convex}
        \end{subfigure}
        \begin{subfigure}[b]{0.5\linewidth}
            \centering
            \scalebox{0.5}{
                \includegraphics{quadtree/images/iges_chord_ratio_nurbs_concave.eps}
            }
            \caption{Concave with an inflection point}
        \end{subfigure}
    \caption{Type of sub-devided NURBS curves: convex and concave with an inflection point}
    \label{qt_fig:iges_chord_ratio_nurbs}
    \end{figure}
%
In order to determine the target NURBS curve is convex one or concave one, a cross product will be conducted.
by assuming the subdivided NURBS curve is cubic, there will be four control points $P_1,P_2,P_3$ and $P_4$.
If the signs of $cross(\overrightarrow{P_1P_2},\overrightarrow{P_2P_3})$ and $cross(\overrightarrow{P_1P_2},\overrightarrow{P_2P_3})$ are the same, then the curve is convex.
Otherwise it will be concave.

%=====================================================================================================================%
\subsubsection{Convex curves}
\label{qt_ssc:convex_curves}
\paragraph{}
Start with the simple case, in the situation illustrated in Fig.~\ref{qt_fig:iges_chord_split_convex_sum} where line $C(u_0)C(u_n)$ and the NURBS curve form a convex set, we are looking for a point $C(u_m)$ on the curve so that $C'(u_m)$ is parallel to $\overrightarrow{C(u_0)C(u_n)}$
    \begin{figure}[h!]
        \centering
        \scalebox{0.6}{
            \includegraphics{quadtree/images/iges_chord_split_convex_sum.eps}
        }
        \caption{Discretization for convex NURBS curve}
        \label{qt_fig:iges_chord_split_convex_sum}
    \end{figure}
%
The target is to split one NURBS curve segment into two.
The splitting will be processed until the arc length to chord length ratio of any splitted curves are smaller than the tolerance.
Based on the property of the convex set, there can be one and only one parameter $u_m$ that satisfies the condition.
As a consequence, numerical methods such as Newton's method can be adopted to determine it.
For a given $u_m$, the next iteration will be:
    \begin{equation}
        u_{m_{new}} =  u_n - \frac{f(u_m)}{f'(u_m)}
    \end{equation}
%
where
    \begin{equation}
        f(u) = C'(u) \begin{bmatrix}
            - C_y(u_n) + C_y(u_0) \\
            C_x(u_n) - C_x(u_0)
        \end{bmatrix}
    \end{equation}
%
The procedure to find $u_m$ can be concluded in Algorithm.~\ref{qdt_alg:split_convex_nurbs} and Algorithm.~\ref{qdt_alg:discrete_convex_nurbs} describes the procedure to find all knots corresponding to the vertexes of the polylines recursively.
\begin{algorithm}
    function splitConvexCurve(curve,u\_0, u\_n) \\
    \Input{
        curve, the input NURBS curve \\
        u\_0,u\_n, two end knots of the NURBS curve
    }
    \Output{
        u\_m, in Fig.~\ref{qt_fig:iges_chord_split_convex_sum}
    }
    u\_m = (u\_n + u\_0) * 0.5 \\
    pt\_0, pt\_n = getCurvePts(u\_0, u\_n) \\
    vector\_0n = Vector(pt\_0,pt\_n) \\
    angle = vector\_0n.atan2() \\
    deri1\_m = getCurveDeri(u\_m,1) \\
    angle\_m = deri1\_m.atan2() \\
    \While{abs(angle\_m-anlge)$<10^{-4}$}
      {
        deri1\_m, deri2\_m = getCurveDeri(u\_m,2) \\
        angle\_m = atan2(deri1\_m.y, deri1\_m.x)  \\
        fu = deri1\_m * vector\_0n.normalVector() \\
        fu\_deri = deri2\_m * vector\_0n.normalVector() \\
        u\_m = u\_m - fu / fu\_deri \\
        deri1\_m = getCurveDeri(u\_m,1) \\
        angle\_m = deri1\_m.atan2()
      }
    \caption{Split a convex NURBS curve into two}
    \label{qdt_alg:split_convex_nurbs}
\end{algorithm}
%
%
\begin{algorithm}
    function discreteConvexCurve(curve,eps,u\_0,u\_n,u)
    \Input{
        curve, the input NURBS curve \\
        eps, the tolerance of the chord to arc-length ratio \\
        u\_0,u\_n, two end knots of the NURBS curve
    }
    \Output{
        u, the vector of the knot corresponding to vertexes of the polylines
    }
    arcLength = curve.arcLength(u\_0,u\_n) \\
    chordLength = curve.getPt(u\_0).distanceTo(curve.getPt(u\_n)) \\
    \eIf{1-chordLength/arcLength $<$ eps}{
        return
    }{
        u\_m = (splitConvexCurve(curve,u\_0,u\_n))
        u.add(u\_m)
        discreteConvexCurve(curve,eps,u\_0,u\_m)
        discreteConvexCurve(curve,eps,u\_m,u\_n)
        return
    }
\caption{Discrete a convex NURBS curve recursively}
\label{qdt_alg:discrete_convex_nurbs}
\end{algorithm}
%=====================================================================================================================%
\subsubsection{Concave curves}
\paragraph{}
As can be seen in Fig.~\ref{qt_fig:iges_chord_ratio_nurbs}, the extracted cubic NURBS curve will have no more than one inflection point.
The reason for that is because the target function is cubic and hence its second derivative will be in first order.
Consequently, numerical methods such as Newton's method can be used to find this unique point.
After that, the curve can be divided into two convex ones and the algorithms introduced in \ref{qt_ssc:convex_curves} can be used separately.
The Newton's iteration can be written as
    \begin{equation}
        u_{n_{new}} = u_n - \frac{f(u_n)}{f'(u_n)}
    \end{equation}

where
    \begin{equation}
        f(u) = C''(u)
    \end{equation}
%
%=====================================================================================================================%
\subsubsection{Calculation of the arc length}
\paragraph{}
The arc length of the NURBS curve defined on $u \in [u_0, u_n]$ can be expressed as
    \begin{equation}
        L = \int_{u_0} ^{u_n} \sqrt{C_x^2(u) + C_y^2(u)} du
    \end{equation}
%
The integration can be solved by the help of the numerical integration quadrature described in \ref{iso_section:numerical_integration} as:
    \begin{equation}
        L = \sum_{i=0}^n a_i \sqrt{C_x^2(u_i) + C_y^2(u_i)}
    \end{equation}