% Full title as you would like it to appear on the page
\chapter{Training a Convolutional Neural Network To Predict The Local Wavefront}
\label{chap:cnn}
\chaptermark{Training a CNN}

\epigraph{Uncrumpling paper balls is what machine learning is about: finding neat representations for complex, highly folded data manifolds.}{Fran\c{c}ois Chollet}

In the first step, we use a convolutional neural network (CNN) to predict the relationship between donuts and their corresponding local wavefronts. There are three inputs to the network: the $256\ \times\ 256$ pixel donut image $D$, the position $r$ of the sensor. This position $r = (x,y,d)$ is comprised of the $(x,y)$ field position in radians, and the offset $d$ along the z-axis indicating whether the defocus is intra-focal or extra-focal. The output is the vector of Zernike coefficients $\alpha \in \mathbb{R}^{18}$. The sub-problem is to find a function approximator $\phi$,

\begin{equation*}
\phi(D, r) \to \alpha(x,y) \text{ such that } W(u,v|x,y) \approx \sum_{j=4}^{21}\alpha_j(x,y)Z_j(u,v)
\end{equation*}

Predicting the local wavefront $W(u,v|x,y)$ from donuts and their metadata is an intrinsically challenging problem because the optics contribution to the image is sometimes degenerate with, and typically subdominant to, the atmospheric contribution. Thus, even for very bright donuts with no effective sources of background noise, the model will have a systematic ceiling. In this section, we demonstrate that a CNN can produce accurate estimates despite this challenge.

\section{Network Architecture}

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/cnn/arch.png}
\end{tabular}
\end{center}
\caption[Network Architecture]{The network architecture and example inputs and output. The image network is in blue and the position network is in red.\label{fig:arch}}
\end{figure}

Modern CNN architectures are governed by an astounding number of design decisions and hyper-parameter options. Here we show that a simple network can make highly accurate wavefront estimates without many additional design or hyper-parameter optimizations. This shows that this step can also serve as a network in other optics algorithms.

The network, shown in Figure \ref{fig:arch}, has two sub-networks: an image network and a position network. e will describe the architecture through the convolutions that are applied and how they change the data tensors. Each 2-dimensional convolution or linear layer, except the final linear layer, is followed by a 2-dimensional batch normalization layer (batchnorm) and a rectified linear activation (ReLU) \cite{726791, batchnorm, relu}. The batch normalization protects against covariate-shift and provides weak regularization and the ReLU activation introduces non-linearity.

The image network reduces the donut image to a 1024 dimensional vector. The first component is a convolution that increases the depth of the input from 1 to 8, and retains the 256 by 256 height and width. This is followed by a sequence of 8 Down-Blocks. Each Down-Block has an initial convolution that does not change the tensor dimensions, followed by a convolution that increases the depth by a factor of 2 and reduces the height and width by a factor of 2. There is a skip connection between these two convolutions for better gradient flow \cite{resnet}. In the 8th and final Down-Block, the depth is not increased. Thus by the end of these steps we have a depth of $\text{depth} = 8 \cdot 2^7 = 1024$ and $\text{height} = \text{width} = 256 \cdot 2^{-8} = 1$. This $1 \times 1 \times 1024$ output is a learned representation of the input image. In total, the image network has 28,326,360 trainable parameters.

The position network combines the $1 \times 1 \times 1024$ output from the image network vector with the 3-dimensional position input $r=(x,y,d)$ and estimates the 18 local wavefront coefficients. It first concatenates and flattens these tensors into a combined 1027-dimensional tensor. Then three linear layers reduce the dimensionality to 266, 69, and 18 respectively (the sequential dimensions are log-linear). After the first linear layer we insert a dropout layer with a 90\% keep probability \cite{10.5555/2999134.2999257} to improve generalization. In total, the position network has 293,801 trainable parameters.

We developed the model in PyTorch \cite{pytorch}. We determined this balance of parameters between sub-networks through testing. In the next Section we describe how the network is trained.

\section{Training, Validation, and Testing}

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/cnn/training_curve.png}
\end{tabular}
\end{center}
\caption[Training Curve]{The training loss (train), validation loss (dev), and learning rate versus the number of samples used for training the neural network. The MSE loss corresponds to the y-axis on the left; the learning rate corresponds to the y-axis on the right.\label{fig:train}}
\end{figure}

In addition to the large number of decisions that go into the network architecture, there are also a lot of decisions on the configuration of the training. For the loss function, we use the mean-squared-error (MSE) between the estimated and true wavefront coefficients and found it to be superior to the mean-absolute-error. We evaluate batches of 64 samples at a time and use the Adam optimizer to compute the parameter updates for the backward pass after computing the loss function in the forward pass \cite{adam}. We train the model for 8 epochs over the donut dataset. After each epoch we multiply the learning rate by a factor of 0.65. Every 200 batches, we evaluate the MSE of the model on the donut validation set, and keep the best model. Figure \ref{fig:train} shows the training and validation (dev) errors, and the learning rate, versus the number of training samples.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/cnn/train_vs_test.png}
\end{tabular}
\end{center}
\caption[Train Versus Test MSE Loss]{The distributions of the training and test MSE loss on stars (donuts with no overlaps) and blends (donuts with overlaps).\label{fig:train_vs_test}}
\end{figure}

We went through multiple iterations with different training configurations to arrive at this final configuration. To assess the models we used the validation set. Once we finalized the training configuration, we used the held out test set for a final assessment of the network. We compared the MSE loss on the training and test sets to check for overfitting. Figure \ref{fig:train_vs_test} shows the comparison. The training and test set MSE losses are $4.5\pm 3.2$ and $4.4\pm 3.5$ thousandths of waves on stars respectively, and $9.5\pm 20.0$ and $9.6\pm 22.0$ thousandths of waves on blends respectively. The performance on the training and test sets is almost identical, which suggests our model is not overfitting.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/cnn/donut_panel_test.png}
\end{tabular}
\end{center}
\caption[Panel of Donut Results]{Top: six donut images; two from the bottom 10\% of the MSE distribution, two from the median, and two from the top 10\%. Bottom: the corresponding true (blue) and neural network estimated (red) local wavefronts. The MSEs in the bottom row are in units of thousandths of waves.\label{fig:donut_panel}}
\end{figure}

We also inspected the performance of the network on many samples. Figure \ref{fig:donut_panel} shows the performance on six donuts from across the MSE loss distribution. At the bottom decile, the network achieves a MSE of 9.5 thousandths of a wave. While benchmarks do not exist for this task, based on our experience with other approaches and previous generation telescopes, this already seems like a competitive result. The median MSE is 4.0 thousandths of a wave. Here a few differences between the true and estimated coefficients can be seen by eye. At the top decile, the network achieves a MSE of 1.4 thousandths of a wave. The match between the true and estimated coefficients is astounding. It is perhaps more astounding to consider that the network is able to achieve this when the optics contribution that we are estimating is sub-dominant to the atmospheric contribution.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/cnn/cov_cor.png}
\end{tabular}
\end{center}
\caption[Wavefront Coefficient Error Covariance and Correlation]{The covariance (left) and correlation (right) of the error between different local wavefront coefficients.\label{fig:cov_cor}}
\end{figure}

It is also important to examine the correlation between the errors on different wavefront coefficients to see if there are systematic errors and biases. Figure \ref{fig:cov_cor} shows the covariance and correlation matrices for the error on the different wavefront coefficients. The covariance diagonal matches with the coefficients that are most excited in our donut dataset. For example, the $Z_5, Z_6$ astigmatism coefficients have the largest error, but also have the largest values in the true wavefront. The correlation matrix shows that the errors are fairly independent across coefficients. The largest correlations are 0.57 between $Z_{11}, Z_{16}$; 0.41 between $Z_{11},Z_{17}$; 0.40 between $Z_{16},Z_{17}$; and -0.34 between $Z_{16}, Z_{21}$.

There are a lot of parameters that determine a donut image. Another natural question is whether any of these are correlated with the error. Figure \ref{fig:err_sources} shows 8, out of the many, error sources we examined. We found that the error generally increases when the background noise increases or foreground number of photons in the donut decreases, as expected. The ratio of these contributions serves as a proxy for the signal-to-noise ratio.

Interestingly, the seeing does not have much impact, suggesting that the network is really able to differentiate between PSF contributions due to the atmosphere and due to the optics. Before training the network, we had also expected the radial distance, a proxy for the amount of vignetting, to be correlated with the error. However, in practice, it seems like the network is able to utilize the $(x,y)$ coordinates in the position to account for this effect and process vignetted donuts accurately.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/cnn/error_analysis.png}
\end{tabular}
\end{center}
\caption[Panel of Potential Sources of Error]{The MSE, in waves, versus the feature distributions for 8 different potential sources of error. The stars are highlighted in yellow; the blends are in green. We also show the best fit line for the 6 non-categorical features. Units: Background Noise is the standard deviation of photons in the background, Foreground Photons is the total number of photons in the donut, Seeing is in arcseconds, Radial Distance is in meters, Temperature is in Kelvin. \label{fig:err_sources}}
\end{figure}

The biggest surprise was the impact of chromatic effects. The temperature of the source is reasonably correlated with the error. The fact that there is also some error correlation with airmass suggests some of this stems from differential chromatic refraction and some from the variations in wavelength across Rubin's r-passband and how this contributes to PSF structure. In future versions of this model, one could imagine training the network to also predict the temperature, or peak wavelength, of the source. This would better encourage it to differentiate the chromatic effects during training.

Finally, there is a clear difference between the stars (sources with no overlaps) and blends (sources with overlaps). The network performs much worse on the blends. This has important implications for which sources, and even fields, to do wavefront sensing in. As the number of overlapping neighbors increases, the error increases. However, it is important to note that there are fewer samples with large numbers of overlapping neighbors so these distributions are less reliable. In Section \ref{chap:interp}, we explore whether ignoring blends improves the performance of the global wavefront interpolation.

\section{Neural Network Enabled Insights}

At a high level, neural networks are differentiable and parameterized function approximators. The same properties that make them trainable, make them effective tools for exploring tasks in new ways. Here our network $\varphi$ has been trained to solve the task $\varphi(D, r) = \alpha$. In the next three subsections, we perform mathematical manipulations on $\varphi$ to gain insight into the behavior of our network and more general properties of this problem.

\subsection{Saliency Maps}

What aspects of the donut images is the neural network using to make its predictions? Following in the footsteps of \cite{saliency}, we define a metric that shows the pixels the predictions of the network are most sensitive to. We call this \textit{attention} and compute it by taking the magnitude of the gradient of the norm of the local wavefront prediction with respect to \textit{the pixels}. The explicit formula is

\begin{equation*}
\text{Attention}\ = \left|\nabla_D || \varphi(D, r)||_2 \right|
\end{equation*}

\noindent This gradient with respect to pixels is similar to the mechanism behind style transfer \cite{style_transfer}.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/cnn/attention.png}
\end{tabular}
\end{center}
\caption[Neural Network Attention]{The neural network attention values for six samples.\label{fig:attention}}
\end{figure}

Figure \ref{fig:attention} shows the attention for six samples. The first pattern that emerges is that the edges dominate, particularly when there is significant vignetting. The network focuses on the outline of a donut, in tandem with its position, to determine the wavefront. This suggests it may be possible to develop a successful network that takes only the outline of the donuts as input. It is not clear how we would have picked up this insight from the other approaches. This is an example of how the neural network can teach us new things about a problem and help us discover new approaches.

The second pattern is that, in the case of blends, the network either utilizes or ignore the neighboring donuts. It is curious that this behavior is so bimodal. For example, where in the network is this decision being made? We suspect answering this question could unlock ways to further improve the performance of the network on blends.

\subsection{Memorizing Versus Learning}

A common grievance of opponents to machine learning being applied in physics is that the network is memorizing, rather than truly learning, the relationships between the input and output. Here, we test this. We examine the intensity changes that would most quickly move the predictions on a base donut towards the Zernike of interest, which we deem the label-shift. We take the gradient of the norm of the difference of the network output and the target Zernike unit vector, or explicitly

\begin{equation*}
\text{Label-Shift}\ = \nabla_D || \varphi(D, r) - Z_i||_2
\end{equation*}

Figure \ref{fig:label-shift} shows the results for the Zernike unit vectors $Z_4$ through $Z_{21}$ under the respective annular Zernike polynomial. The similarity between the intensity changes and the Zernike polynomials is striking, especially given that the network has not been explicitly trained to learn these patterns. For example, the network even learns the 5 cycle periodicity of $Z_{20}$ and $Z_{21}$. It also learns the complicated radial dependence of $Z_{18}$. This demonstrates that the network is engaging in higher order learning, where it is learning general features of the problem space.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/cnn/morph.png}
\end{tabular}
\end{center}
\caption[Neural Network Label-Shifts]{The odd rows show the annular Zernike polynomials $Z_4$ through $Z_{21}$. The even rows show the neural network label-shifts for the corresponding Zernike unit vectors. \label{fig:label-shift}}
\end{figure}

\subsection{Adversarial Tradeoffs}

The input space for our problem, the image and the position, has $256 \times 256 + 3 = 65539$ dimensions. This is an extremely high dimensional space. It is not hard to find subtle changes in this space that are hardly discernable \textit{by eye}, or at a global level, but which can dramatically change the output of the network. In \cite{2014arXiv1412.6572G}, the authors show how to use these adversarial examples to improve training. Here, we examine a few adversarial examples to understand how our network could be hacked.

An attacker has two competing objectives. The first is to find a modified donut image $D^\prime$ that is similar to $D$; the second is to find a modified donut image $D^\prime$ such that $\varphi(D^\prime, r)$ is close to some target $t^\prime \in \mathbb{R}^{18}$. The Adversarial-Objective is,

\begin{equation*}
\text{Adversarial-Objective}\ = ||\varphi(D^\prime, r) - t^\prime||_2^2 + ||D - D^\prime||_2^2
\end{equation*}

\noindent In order to find such a $D^\prime$ we can minimize this objective function with respect to $D^\prime$. We use the Adam optimizer to perform gradient descent. Two example results are shown in Figure \ref{fig:adversarial}. In both cases we only change one coefficient in the target $t^\prime$.

\begin{figure}[!htbp]
\begin{center}
\begin{tabular}{c}
\includegraphics[width=\textwidth]{figs/cnn/adversarial.png}
\end{tabular}
\end{center}
\caption[Adversarial Attack]{Two example adversarial attacks. For each of the attacks we show three donut images, from left to right: the original donut image $D$, the new donut image $D^\prime$, and the difference $D^\prime - D$. Below we show the original prediction $\varphi(D,r)$ in black, the target $t^\prime$ as a red ``x'', and the new prediction $\varphi(D^\prime, r)$ in gray.\label{fig:adversarial}}
\end{figure}

In practice, one must scale the ratio of the two terms in the Adversarial-Objective to get good convergence. This ratio determines the relative priority of similarity between $D$ and $D^\prime$ versus $\varphi(D^\prime, r)$ and $t^\prime$. In the example on the left, we prioritized the similarity of the donuts. Here, the two donuts appear the same by eye. However, the coefficient for $Z_8$ is increased significantly, although not all the way to the target - while keeping most of the other coefficients close to their original values. This highlights the power of this kind of attack.

In the second example, the difference between the donut images is clearly discernible. Here the $Z_4$ coefficient changes dramatically, and largely hits the target, and even changes signs. This shows how the prediction can be manipulated to the target with sufficient changes to the donut.

In the context of the Rubin Observatory, a closed environment, adverse attacks need not be feared. However, the algorithm we develop here serves a general building block for other wavefront sensing methods. These algorithms are also used for defense purposes. In this context, the sensitivity to adversarial attacks is an important characteristic to be aware of, and potentially protect against.


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{ifsym}
\usepackage{url}
\usepackage{enumitem}
\usepackage{tikz}
\usetikzlibrary{arrows}

\setlength\parindent{0pt}

\title{Query Optimization Homework 5}
\date{\today}

\usepackage{natbib}
\usepackage{graphicx}

\begin{document}

\maketitle

\section{Exercise 1}
\subsection{Perform the IKKBZ algorithm}

1. Call $IKKBZ(G,C_{out})$ with G:

\begin{tikzpicture}
  \node (A) at (0,0) {A};
  \node (B) [right of=A] {B};
  \node (C) [below of=A] {C};
  \node (D) [right of=C] {D};
  \draw (A) -- (C);
  \draw (A) -- (B);
  \draw (B) -- (D);
\end{tikzpicture}

\newpage
2. Call $IKKBZ-Sub(G_A,C_{out})$ with $G_A:$

\begin{tikzpicture}
\tikzset{
    edge/.style={->,> = latex'}
}
  \node (A) at (0,0) {A};
  \node (B) at (1, -1) {B};
  \node (C) at (-1, -1) {C};
  \node (D) at (1, -2) {D};
  \draw[edge] (A) -- (C);
  \draw[edge] (A) -- (B);
  \draw[edge] (B) -- (D);
\end{tikzpicture}

\begin{table}[!hbtp]
\begin{tabular}{|l|l|l|l|l|l|}
\hline
Relation & n   & s   & C  & T  & rank           \\ \hline
B        & 10  & 0,9 & 9  & 9  & $\frac{8}{9}$  \\ \hline
C        & 100 & 0,1 & 10 & 10 & $\frac{9}{10}$ \\ \hline
D        & 100 & 0,1 & 10 & 10 & $\frac{9}{10}$ \\ \hline
\end{tabular}
\end{table}

2.1 Normalize(A) doesn't change anything\\

2.2 Merge the chains under A:

\begin{tikzpicture}
\tikzset{
    edge/.style={->,> = latex'}
}
  \node (A) at (0, 0) {A};
  \node (B) at (0, -1) {B};
  \node (C) at (0, -3) {C};
  \node (D) at (0, -2) {D};
  \draw[edge] (A) -- (B);
  \draw[edge] (B) -- (D);
  \draw[edge] (D) -- (C);
\end{tikzpicture}

\vspace*{1cm}
2.3 Denormalize$(G_A)$ doesn't change anything

2.4 Compute $C_{out}$:
\begin{align}
& C_{out}(A \bowtie B) = 10 \cdot 10 \cdot 0.9 = 90 \nonumber \\
& C_{out}((A \bowtie B) \bowtie D) = 90 \cdot 100 \cdot 0.1 + 90 = 990 \nonumber \\
& C_{out}(((A \bowtie B) \bowtie D) \bowtie C) = 900 \cdot 100 \cdot 0.1 + 990 = 9990 \nonumber
\end{align}

\newpage
3. Call $IKKBZ-Sub(G_B,C_{out})$ with $G_B:$

\begin{tikzpicture}
\tikzset{
    edge/.style={->,> = latex'}
}
  \node (A) at (-1, -1) {A};
  \node (B) at (0, 0) {B};
  \node (C) at (-1, -2) {C};
  \node (D) at (1, -1) {D};
  \draw[edge] (A) -- (C);
  \draw[edge] (B) -- (A);
  \draw[edge] (B) -- (D);
\end{tikzpicture}

\begin{table}[!hbtp]
\begin{tabular}{|l|l|l|l|l|l|}
\hline
Relation & n   & s   & C  & T  & rank           \\ \hline
A        & 10  & 0,9 & 9  & 9  & $\frac{8}{9}$  \\ \hline
C        & 100 & 0,1 & 10 & 10 & $\frac{9}{10}$ \\ \hline
D        & 100 & 0,1 & 10 & 10 & $\frac{9}{10}$ \\ \hline
\end{tabular}
\end{table}

3.1 Normalize(B) doesn't change anything\\

3.2 Merge the chains under B:

\begin{tikzpicture}
\tikzset{
    edge/.style={->,> = latex'}
}
  \node (A) at (0, -1) {A};
  \node (B) at (0, 0) {B};
  \node (C) at (0, -2) {C};
  \node (D) at (0, -3) {D};
  \draw[edge] (B) -- (A);
  \draw[edge] (A) -- (C);
  \draw[edge] (C) -- (D);
\end{tikzpicture}

\vspace*{1cm}
3.3 Denormalize$(G_B)$ doesn't change anything

3.4 Compute $C_{out}$:
\begin{align}
& C_{out}(B \bowtie A) = 10 \cdot 10 \cdot 0.9 = 90 \nonumber \\
& C_{out}((B \bowtie A) \bowtie C) = 90 \cdot 100 \cdot 0.1 + 90 = 990 \nonumber \\
& C_{out}(((B \bowtie A) \bowtie C) \bowtie D) = 900 \cdot 100 \cdot 0.1 + 990 = 9990 \nonumber
\end{align}

\newpage
4. Call $IKKBZ-Sub(G_C,C_{out})$ with $G_C:$

\begin{tikzpicture}
\tikzset{
    edge/.style={->,> = latex'}
}
  \node (A) at (0, -1) {A};
  \node (B) at (0, -2) {B};
  \node (C) at (0, 0) {C};
  \node (D) at (0, -3) {D};
  \draw[edge] (C) -- (A);
  \draw[edge] (A) -- (B);
  \draw[edge] (B) -- (D);
\end{tikzpicture}

\begin{table}[!hbtp]
\begin{tabular}{|l|l|l|l|l|l|}
\hline
Relation & n   & s   & C  & T  & rank           \\ \hline
A        & 10  & 0,1 & 1  & 1  & $0$  \\ \hline
B        & 10  & 0,9 & 9  & 9  & $\frac{8}{9}$  \\ \hline
D        & 100 & 0,1 & 10 & 10 & $\frac{9}{10}$ \\ \hline
\end{tabular}
\end{table}

\vspace*{1cm}
4.1 Denormalize$(G_C)$ doesn't change anything

4.2 Compute $C_{out}$:
\begin{align}
& C_{out}(C \bowtie A) = 100 \cdot 10 \cdot 0.1 = 100 \nonumber \\
& C_{out}((C \bowtie A) \bowtie B) = 100 \cdot 10 \cdot 0.9 + 100 = 1000 \nonumber \\
& C_{out}(((C \bowtie A) \bowtie B) \bowtie D) = 900 \cdot 100 \cdot 0.1 + 1000 = 10000 \nonumber
\end{align}

\newpage
5. Call $IKKBZ-Sub(G_D,C_{out})$ with $G_D:$

\begin{tikzpicture}
\tikzset{
    edge/.style={->,> = latex'}
}
  \node (A) at (0, -2) {A};
  \node (B) at (0, -1) {B};
  \node (C) at (0, -3) {C};
  \node (D) at (0, 0) {D};
  \draw[edge] (A) -- (C);
  \draw[edge] (B) -- (A);
  \draw[edge] (D) -- (B);
\end{tikzpicture}

\begin{table}[!hbtp]
\begin{tabular}{|l|l|l|l|l|l|}
\hline
Relation & n   & s   & C  & T  & rank           \\ \hline
A        & 10  & 0,9 & 9  & 9  & $\frac{8}{9}$  \\ \hline
B        & 10  & 0,1 & 1  & 1  & $0$  \\ \hline
C        & 100 & 0,1 & 10 & 10 & $\frac{9}{10}$ \\ \hline
\end{tabular}
\end{table}

\vspace*{1cm}
5.1 Denormalize$(G_D)$ doesn't change anything

5.2 Compute $C_{out}$:
\begin{align}
& C_{out}(D \bowtie B) = 100 \cdot 10 \cdot 0.1 = 100 \nonumber \\
& C_{out}((D \bowtie B) \bowtie A) = 100 \cdot 10 \cdot 0.9 + 100 = 1000 \nonumber \\
& C_{out}(((D \bowtie B) \bowtie A) \bowtie C) = 900 \cdot 100 \cdot 0.1 + 1000 = 10000 \nonumber
\end{align}

\newpage
6. Choose the precedence graph with the minimal cost, both $G_A$ and $G_B$ are minimal, so we choose $G_A$ and construct the following join tree:

\begin{tikzpicture}[sibling distance=1.5cm]
  \node (j1) {$\bowtie$} {
    child {
      node (j2) {$\bowtie$}
      child {
        node (j3) {$\bowtie$}
        child {
          node (j4) {$A$}
        }
        child {
          node (j5) {$B$}
        }
      }
      child {
        node (j6) {$D$}
      }
    }
    child {
      node (j7) {$C$}
    }
  };
\end{tikzpicture}

\end{document}


\section{Introduction}\label{sec:intro}
%<Introduction of the Introduction>
% NJU:
% I'd emphasise the Belief Propagation, because
% 1) it was the initial motivation to work with this topic
% 2) most of our improvements are applicable to general BP and not just our example

Whether you are planing to segment an image \cite{1544822}, recognize speech \cite{5373446} or analyze medical datasets \cite{bailly2011finding}, \textit{Belief Propagation} (BP) plays an important role in all these research areas. This is due to the very general nature of the BP algorithm which allows to approximate the probabilities for certain events. In recent years, BP has also been applied successfully in the domain of recommender systems \cite{Ha:2012:TRT:2396761.2398636}. Popularized by the Netflix challenge and E-commerce shops such as Amazon, recommender systems try to predict which items a user wants to see based on their previous ratings as well as the ratings of other, often similar, users. BP can be used to compute the probability of a user to like a particular item. Given these probabilities we are able to rank items and output the Top-N items for the given user.

%The work at hand presents a run-time optimized implementation of loopy Belief Propagation, tuned for a Recommender System. In the following, we quickly motivate our work and present the structure of this document.

%Do not start the introduction with the abstract or a slightly modified
%version. It follows a possible structure of the introduction.
%Note that the structure can be modified, but the
%content should be the same. Introduction and abstract should fill at most the first page, better less.

% Elias: We should keep that in mind because the introduction to the introduction sounds quite similiar to an abstract.

\mypar{Motivation}
While the results of Jiwoon Ha et al.\cite{Ha:2012:TRT:2396761.2398636} look very promising, one problem is that BP takes a long time to converge. Because we need to run BP once for every user (to predict their ratings), the system seems to be unusable for a large number of users. In our initial tests a baseline implementation took about 21 hours to predict the ratings on a 100k Movielens dataset \cite{riedl1998movielens}. In order to make this approach feasible, a  run-time efficient implementation of the BP algorithm is needed.

\mypar{Related work}
We evaluated two popular open-source C++ frameworks for graph-based inference that implement the BP algorithm in a generic way: \textit{libDAI} \cite{Mooij_libDAI_10} on the one hand is a clean and accessible piece of software, whereas \textit{OpenGM} \cite{andres2012opengm} on the other hand is written with a higher degree of abstraction. We chose libDAI as our reference, because it offers a native implementation of \textit{residual} BP.

In the following, we briefly sketch the \textit{Top-N Recommender System based on Residual Belief Propagation} in section \ref{sec:background} before we inform the reader in section \ref{sec:method} about the optimization techniques that have been applied. The results are presented in section \ref{sec:results}.


\section*{\huge Abstract}

Programming languages let us communicate with machines. Yet to date, nearly every language makes this a laborious process. Every instruction, with fine detail, must be specified to the machine. We aren't doomed to program this way. Instead, an upgrowth of works in programming languages aim to design programming languages which allow for more informal specifications of programs. Programs can be specified by providing examples of expected behavior---or by providing an informative type. Think of how one might communicate the rules of game to a friend, both by formal specification and by examples of play. I describe these languages as having machinery for "learning". You specify the program you want, then the language uses its machinery to learn the program you specified.

In this work I show that System F, a classic and storied programming language, can "learn" programs. Unlike previous works which introduce machinery explicitly for learning, I show System F needs nothing new. Moreover, System F can learn programs no previous language has been shown to learn---higher-order programs from examples. These strengths give System F the capacity to learn not only programs, but programming languages.

In Chapter 1, I introduce the problem of learning. These include musings over learning as discussed in cognitive science, followed by learning in programming languages.

In Chapter 2, I introduce System F. Here I tersely cover the language and state important theorems which hold in System F and which make learning possible.

In Chapter 3, I begin the technical treatment of learning, when data are types. A key theorem is proved. Namely, that any program in System F can be learned from types.

In Chapter 4, I extend learning from types, when data are examples. Another key theorem is proved. Namely, that any program in System F can be learned from examples.

In Chapter 5, I muse over the significance of learning in System F---discussing the prospect of learning programming languages in System F. Considerations for future work, including barriers to implementation and extending to new type systems, are discussed.

With the exception of Chapter 2, all content and interpretatons are original contributions.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\chapter{Results and Discussions}
\label{chap:Results}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Experimental Settings}
\label{experimental_settings}
The settings for all experiments were conducted using test trains split defined by \cite{Talebi2018}, and data augmentation outlined in chapter \ref{chap:Methodology}, with input of raw images resized to $3 \times 224 \times 224$. To enable effective side-by-side comparison with baseline, state of the art pre-processing steps are kept the same. We also used transfer learning from the same source dataset, ImageNet1k. Transformers and CNNS are, however, inherently different - for example, learning rate alongside approaches such as warming-up\cite{Zhang2021} are necessary for transformers to converge, therefore some aspects - such as learning rate - change during training of baseline and transformer models.

%%%%%%%%%%%%%%%%
% FLOW CHART %%%
%%%%%%%%%%%%%%%%

\subsection{ResNets and ConViTS}

We conduct experiments to obtain baselines using ResNets $\in\{18,50,152\}$ and ConViT models $\in\{tiny,small,base$\}$ $, which are pre-trained on ImageNet-1k, and added an additional, fully connected layer to correspond to binary classes. Hyper parameter selection is a crucial part of tuning and adapting models to the target domain.

%%%%batch_size=50
%color_jitter=0.4,
%cooldown_epochs=10,
%lr=0.0005,
%local_up_to_layer=10,
%locality_strength=4.0,
%lr_noise_pct=0.67,
%opt='adamw',
%opt_eps=1e-08,
%sched='cosine',
%warmup_epochs=5,
%warmup_lr=1e-06,
%weight_decay=0.05,

\begin{table}[ht!]
\small
\centering
\caption{Hyper Params}
\begin{tabular}{c?ccc}
\toprule
Hyper Param &  \multicolumn{1}{c}{ResNet innitial} & \multicolumn{1}{c}{ConViT} & Equal Conditions\\
\midrule
Learning Rate     &  0.0001        & 0.0005          & 0.0003         \\
Colour Jitter     &  0.5           & 0.5             & 0.5            \\
Optimiser         &  $Adam$        &  $AdamW$        & $AdamW$        \\
Batch size        & 50-250         & 10-200          & 10-200         \\
Loss Function     & $cross entropy$& $cross entropy$ & $cross entropy$\\
Cool down epoch   & -              & 5               & 0              \\
Scheduler         & $StepLR$       & $cosine$        & $cosine$       \\
Opt. $\epsilon$    & -             & 1e-08           &                \\
Local up to layer & -              & 10              & -              \\
Locality strength & -              & 4.0             & -              \\
lr noise pct.     & -              & 0.67            & 0              \\
Warm up epochs    & -              & 5               & 3              \\
Warm up lr        & -              & 1e-06           & 1e-06          \\
\bottomrule
\end{tabular}
\end{table}

\subsection{Optimiser and Scheduler}
These differ slightly between ConViT and baseline models.

\begin{description}
\item[For ConViTs] We use AdamW with a Cosine\footnote{\href{https://github.com/rwightman/pytorch-image-models/blob/master/timm/scheduler/cosine_lr.py}{https://github.com/rwightman/pytorch-image-models/blob/master/timm/scheduler/cosine\_lr.py}}  learning rate Scheduler;
\item[for ResNets] we use Adam and AdamW with Stepped learning rate scheduler and cosine scheduler;
\item[for ViT and CvT] we use Adam with Cosine learning rate scheduler;
\end{description}

We use an Adam optimiser\cite{Kingma2015} as gradient base optimisation for simplicity of implementation and computational efficiency. We use a Cosine Scheduler by \cite{DAscoli2021} for ConViTs, as there is clearly a trade-off in keeping domain parameters the same. Both schedulers are a form of simulated annealing, which allowed for a change in learning rate over time - mitigating against the challenges of gradient based approaches, such as saddle points.

We train networks in separate code environments\footnote{\href{https://github.com/facebookresearch/convit}{https://github.com/facebookresearch/convit}}, using code supplied by \cite{DAscoli2021} for ConViTs and CvT. We train all models under equal conditions, in the same code environment, with identical training and data augmentation pipelines. Various aspects, such as data augmentation, are inherent features of training pipeline.

\subsection{Loss Function Vs Data Sampler}

Loss function is necessary to minimise the difference between GT labels, and where minimisation of loss or error between GT and predicted result is required.  While formative experiments were conducted with focal loss, we adopt a cross entropy loss function. While \cite{Lin2020} have shown that focal loss, combined with cross entropy, have been shown to be effective techniques in addressing class imbalance.

We use cross entropy with a sampler, where random sampling from the minority class with a weighted probability is used to address this. This is effective, as it allows repeated augmentation of the minority class during training. Further, this is the technique used in code for training on ImageNet-1k by \cite{Touvron2020a, DAscoli2021}, and therefore provides methodological consistency across baseline and vision transformer models .\lstinline[language=Python]{torch.utils.data.sampler.WeightedRandomSampler}

Where samples are computed for every batch given by:
\begin{equation}
    Sc = \frac{\sum_{i=1}^{Bn}}{\sum_{i=1}^{Cn}}
    \label{eq:class weights}
\end{equation}

Where $Sc$ s sample ratio $Bn$ is the number of batch samples, and $Cn$ n is the class count of each batch.

\subsection{Software and Hardware}
The experiments were conducted using a single Nvidia Tesla P100 12Gb GPU, with CUDA 11.2. The software used for training was PyTorch open source framework, with data ammunition using both Torchvision's native fictions alongside Albumentations\cite{Buslaev2020a}. ConViT models were built using timm\footnote{\href{https://github.com/rwightman/pytorch-image-models/tree/master/timm}{https://github.com/rwightman/pytorch-image-models/tree/master/timm}}. All software is implemented using Python 3.7.

\begin{table}[]
    \centering
    \begin{tabular}{c?cc}
    \toprule
     Software & Version & Purpose \\
     \midrule
     torch &   1.7.0 & Training ConViTs\\
     timm & 0.3.2 & Building ConViTs\\
     albumentations & 0.4.6 & Data augmentation\\
     torchvision & 0.8.1 & Pre-trained models\\
     \bottomrule
    \end{tabular}
    \caption{Software}
    \label{tab:software}
\end{table}

\section{Quantitative Results}
\label{Quantitative Results}
\subsection{Training}
\subsubsection{ResNets}
Training ResNets for 10 epochs shows differences in overall accuracy. These are shown in figure \ref{fig:resnet_train}. In training and validation accuracy, loss, and balanced accuracy, all layers were unfrozen to allow pre-trained models to be fully adapted during training.

\begin{figure}[ht!]
    \begin{center}
    \includegraphics[height=0.3\textwidth]{figures/results_and_discussion/resnets.png}
    \caption{ Side by Side Comparison of ResNets}
    \label{fig:resnet_train}
    \end{center}
\end{figure}

Note that only ResNet18 converges after pre-training, which importantly shows that ResNet18's depth may not have enough trainable parameters or be deep enough to produce the best results. Further, note that after convergence there is very little improvement of training or validation accuracy.

Balanced accuracy shows more instability, indicating that accuracy in itself may be `masking' model fitting. However, both appear similar where equal numbers of positive and negative classes are sampled in each training batch.

ResNet152 continues improvement in accuracy and balanced accuracy (there is also less difference). This indicates that when a model has sufficiently trainable parameters and is deep enough that there is a high degree of similarity to features that are trained on ImageNet21k.

\begin{table}[ht!]
\centering
\small
\begin{tabular}{lcccccccc}
\toprule
{} & \multicolumn{2}{c}{Resnet 18} & {}& \multicolumn{2}{c}{Resnet 50} & {}& \multicolumn{2}{c}{Resnet 152} \\
 \midrule
{} & Validation & Training &{} & Validation & Training &{} & Validation & Training \\
\cline{2-3} \cline{5-6} \cline{8-9}
Accuracy &      0.706 &    0.726 & &      0.710 &    0.906 & &      0.741 &    0.980 \\
Balanced Acc &      0.747 &    0.815 & &      0.854 &    0.944 & &    0.964 &    0.977 \\
\bottomrule
\end{tabular}
\caption{ArgMax Resnet Training Metrics}
\label{tab:resnet train}
\end{table}

Table \ref{tab:resnet train} shows the differences in training accuracy and balanced accuracy - there is a clear improvement in increasing model size over the same number of epochs.

\subsubsection{CvT and ConVits}

We train both CvT\cite{Wu2021} and ConViTs to enable side by side comparison. Results from training CvT\cite{Wu2021} show that a model will converge after 5 of epochs (see figure \ref{fig:CvT} right), however, the model quickly plateaus. This can be seen in figure \ref{fig:CvT} left.
\begin{figure}[htp!]
    \centering
    \includegraphics[width=0.9\textwidth]{figures/results_and_discussion/CVT.png}
    \caption{Hybrid Convolutional Transformer Training}
    \label{fig:CvT}
\end{figure}

Further, it can be seen that training inference is unstable (both training and validation (balanced accuracy) vary significantly), which is consistent with observations elsewhere - often requiring a warm-up (tapering of learning rate) during initialisation. Further, given the inclusion of a data sampler, balanced accuracy does not provide a clear prediction of model accuracy and therefore subsequently trained models do not report this. \ref{fig:CvT} left.

Comparing results between CvTs and ConViTs, both without pre-training on ImageNet1k, shows erratic training performance on validation sets.

\begin{figure}[htp!]
    \centering
    \includegraphics[width=\textwidth]{figures/results_and_discussion/ConViT Ti,S,B Without Pre-Training.png}
    \caption{ConViT Without Pre-Trainign on ImageNet1k}
    \label{fig:ConViT_non_pre_trained}
\end{figure}

A marked improvement can be seen in the use of pre-trained models with a 5 epoch warm up, shown in \ref{fig:ConViT_non_pre_trained}, where it is also clear the model size has a bearing on validation accuracy (although this appears to be slight).
\begin{figure}[htp!]
    \centering
    \includegraphics[width=\textwidth]{figures/results_and_discussion/ConViT Ti,S,B Compared.png}
    \caption{ConViTs With Pre-training }
    \label{fig:ConViT_pre_trained}
\end{figure}

\subsubsection{ Hyper-Parameter Tuning and Data Augmentation}

Conducting a grid-search over learning rate and locality strength, it is clear that learn locality strength has a slight effect on model performance. Training was conducted with no warm start epochs. This shows that decreasing learning rate improves overall performance. This is shown in figure \ref{fig:ConViT_grid} left. The newly introduced hyper-parameter has some effect on overall performance.

\begin{figure}[htp!]
    \centering
    \includegraphics[width=\textwidth]{figures/results_and_discussion/hyper-params_compared.pdf}
    \caption{ConViT Ti Grid Search Between Learning Rate and Locality Strength (With Control)}
    \label{fig:ConViT_grid}
\end{figure}

Control training, including warm up epochs, was introduced to demonstrate difference between training with and without warm starting. This is shown in black on figure \ref{fig:ConViT_grid}. This also shows that using pre-training that not using a warm start results in relatively high accuracy, with almost no training on the AVA  dataset.

In order to compare spatial data augmentation, we performed training of salient crops where $5 \times (224 \times 224)$ sub patches of each image were taken using the attention algorithm provided by\cite{Ma2017}. This was in order to gauge how a self attention model might train on data, and whether higher resolution patches might enhance training. Training on all five sub patches separately, alongside training on all patches pooled, is shown in figure \ref{fig:sub_patching}.

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{figures/results_and_discussion/A-Lamp Salient Crops.png}
    \caption{Comparison of ConViT Ti Trained on Salient Patches from A-Lamp Model \cite{Ma2017}}
    \label{fig:sub_patching}
\end{figure}
\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{figures/results_and_discussion/Compared Augmentation.png}
    \caption{Comparison of Augmentation on ConViT B and S}
    \label{fig:my_label}
\end{figure}

Combining all demonstrates best performance, however all perform less well than a global/non cropped image, which indicates that the ConViTs are learning composition information.

We also compared training on zero-padding to square and reflection padding. The logic of this is that reflection padding maintains composition data, while reflecting the shortest edge effectively also encodes an image border.

This shows some improvement during training by reflection padding.

\begin{table}[]
\small
    \centering
\begin{tabular}{lcc?ccc?c}
 \toprule
{} & \multicolumn{2}{c}{Starting State} & \multicolumn{3}{c}{Augmentation} & Search         \\
 \toprule
{} & Pre-train  & No Pre-train                  & Reflection Pad & Zero Pad & Salient Crop  &  Grid\\
ConViT B        &                0.766          &   0.708        &       0.756   &  0.758  &   -  &  -      \\
ConViT S        &                0.767          &   0.711        &       0.769   &   0.769 &  -   &   -      \\
ConViT Ti       &                0.765          &   0.705        &       0.749   &   0.714 & 0.747& 0.776    \\
\bottomrule
\end{tabular}
    \caption{ArgMax Validation Results after 10 Epochs}
    \label{tab:results_all}
\end{table}

\subsection{Test Metrics}

Table \ref{tab:reflection} shows all results evaluated on AVA test data: 19k images on reflection padded and normalised images. Note that accuracy does not improve monotonically with model size for ConViTs, but does for ResNet models - however, F1 score does appear to obey a monotonic relationship with size of ConViT Model.

We also compared this with zero-padded images to square, as this might reflect how images are processed in the wild:

Here, each ConViT out performs in accuracy, however it is interesting to note ResNet152 out performance on balanced accuracy and F1 Score ConViT B.
\begin{table}[ht!]
\small
    \centering
\begin{tabular}{lccccl}
\toprule
{}             &  Accuracy$\downarrow$ &  Balanced Acc.& F1     & N epochs & Weight Save Arg \\
\midrule
ViT B         &     39.96 &           26.66 &  39.93 & 10       & Last Epoch          \\
CvT           &     63.90 &           76.68 &  52.47 & 25       & Max Acc. Epoch      \\
ResNet 18 R   &     70.23 &           58.08 &  65.66 & 10       & Last Epoch          \\
ResNet 50     &     70.85 &           99.54 &  41.70 & 10       & Last Epoch          \\
ResNet 18     &     71.31 &           86.84 &  54.74 & 10       & Last Epoch          \\
ResNet 152    &     72.69 &           87.66 &  55.84 & 10       & Last Epoch          \\
ConViT Ti     &     73.17 &           95.52 &  49.58 & 10       & Last Epoch          \\
ResNet 50 R   &     74.13 &           72.06 &  65.69 & 10       & Last Epoch          \\
ConViT Ti     &     74.33 &           90.90 &  55.42 & 10       & Max Val. Acc. Epoch \\
ConViT B      &     74.58 &           85.53 &  59.65 & 10       & Last Epoch          \\
ConViT S      &     75.02 &           87.58 &  58.85 & 10       & Last Epoch          \\
ConViT S      &     75.06 &           90.22 &  56.94 & 10       & Max Val. Acc. Epoch \\
ResNet 152 R  &     75.63 &           60.32 &  70.82 & 10       & Max Val. Acc. Epoch \\
ConViT Ti R   &     76.29 &           72.33 &  68.07 & 10       & Max Val. Acc. Epoch \\
ConViT S R    &     76.29 &           72.33 &  68.07 & 10       & Max Val. Acc. Epoch \\
ConViT B      &     76.47       &         81.00 &  64.59 & 10       & Max Val. Acc. Epoch \\
ConViT B R    &     76.82       &         70.38 &  69.35 & 10       & Max Val. Acc. Epoch \\
NIMA          &     77.25       &         69.51 &  70.56 & 0        & Pre-Trained         \\
ConViT B R    & \textbf{78.11}  &\textbf{73.17} &  69.92 & 25       & Max Val. Acc. Epoch \\
MLSP          &     81.65       &         66.68 &  76.08 & 0        & Pre-Trained         \\
\bottomrule
\end{tabular}
\caption{Evaluation on Refection Padded Normalised Images Square Padded on 19k AVA Test Set}
    \label{tab:reflection}
\end{table}

We show that, when trained on best hyper parameters (from the grid search conducted [learning rate <0.0015 and locality strength 1.5]), ConViT B outperforms NIMA\cite{Talebi2018}, whose results we reproduce on their test set with a pre-trained model made available alongside their publication. Both the F1 score and balanced accuracy of MLSP\cite{Hosu2019} are lower than ours. This is in spite of the potential that training an end to end binary classifier is more challenging, with a smaller, fully connected final layer output of 2 rather than 10 (one for each MOS probability produced by normalised MOS score over 10 bins).

One further important feature to consider is that \cite{Talebi2018, Hosu2019} both sample from much larger initial images, and we only train on $224 \times 224$ resized images. We also show a confusion matrix (each column represents the entirety of the 19k AVA Test set sample space). Here, we see that ConViT B outperforms in maintaining a high number of $TP$s while not sacrificing the the negative minority class $TN$.

\begin{table}[ht]
\tiny
    \centering
   \begin{tabular}{lrrrrrrrrr}
\toprule
{} &  ResNet 50 R &  ResNet 152 R &  ResNet 18 10 &  ConViT Ti R &  ConViT B R &  ConViT S M &  ViT B &  MLSP &  NIMA \\
\midrule
tp &         0.62 &          0.58 &          0.66 &         0.64 &        0.63 &        0.70 &   0.52 &  0.65 &  0.62 \\
tn &         0.12 &          0.18 &          0.05 &         0.13 &        0.14 &        0.05 &   0.08 &  0.17 &  0.15 \\
fp &         0.09 &          0.13 &          0.05 &         0.08 &        0.08 &        0.01 &   0.19 &  0.06 &  0.08 \\
fn &         0.16 &          0.11 &          0.23 &         0.16 &        0.15 &        0.24 &   0.20 &  0.12 &  0.15 \\
\bottomrule
\end{tabular}

    \caption{Confusion Metrics on 19k AVA Test-Set 19k }
    \label{tab:Cofusion Metrics}
\end{table}

Table \ref{tab:Model Types} shows nomenclature used in tables (\ref{tab:reflection}, \ref{tab:Cofusion Metrics}), as models performed differently according to whether zero-padding or reflection padding were used during data augmentation. We also show type number of layers and dimension of resized images.

\begin{table}[ht!]
\small
    \centering
   \begin{tabular}{llllc}
\toprule
{}         & Augment             & Layers                    & Type            & Dimensions      \\
\midrule
ResNet 18  Z & Zero Pad      & 18 Residual    &  CNN                           & $224 \times 244$\\
ResNet 50  Z & Zero Pad      & 50 Residual    &  CNN                           & $224 \times 244$\\
ResNet 152 Z & Zero Pad      & 152 Residual   &  CNN                           & $224 \times 244$\\
ConViT Ti  Z & Zero Pad      & $12 \times 3$  & Conv. Vision Transformer       & $224 \times 244$\\
ConViT S   Z & Zero Pad      & $12 \times 6$  & Conv. Vision Transformer       & $224 \times 244$\\
ConViT B   Z & Zero Pad      & $12\times 12$  & Conv. Vision Transformer       & $224 \times 244$\\
ResNet 18  R & Reflect Pad   & 18 Residual    &  CNN                           & $224 \times 244$\\
ResNet 50  R & Reflect Pad   & 50 Residual    &  CNN                           & $224 \times 244$\\
ResNet 152 R & Reflect Pad   & 152 Residual   &  CNN                           & $224 \times 244$\\
ConViT Ti  R & Reflect Pad   & $12 \times 3$  & Conv. Vision Transformer       & $224 \times 244$\\
ConViT S   R & Reflect Pad   & $12 \times 6$  & Conv. Vision Transformer       & $224 \times 244$\\
ConViT B   R & Reflect Pad   & $12\times 12$  & Conv. Vision Transformer       & $224 \times 244$\\
ViT B      R & Reflect Pad   & $12\times 12$  & Vision Transformer             & $224 \times 244$\\
CvT        R & Reflect Pad   & $3\times 10$   & Conv. Vision Transformer       & $224 \times 244$\\

\bottomrule
\end{tabular}
    \caption{Number Parameters and Training Time Per Epoch 230k Images on 1 GPU- batch size 10 Inference on 19k Images on AVA dataset}
    \label{tab:Model Types}
\end{table}

\section{Exclusive Set Analysis}
\label{sec:exclusive set}

This section shows images from the subsets shown in Table \ref{tab:unique}. These images are a complement set of the union of all other model inferences on the 19k image AVA test set, $M= \overline{\cup}_{i=1}^{n}F_{i}$ where $F$ is all other models.  This is performed for each model and each inference type $\in \{tp,tn,fp,fn\}$. The results are shown in table \ref{tab:unique}, we compute this for the best performing model of each type. This shows as a discreet value, the number of images where a single model over or under performs against all the others in the entirety of the 19k.

\begin{table}[ht!]
\small
    \centering
\begin{tabular}{lrrrr}
\toprule
{} &    tn &    fp &    fn &     tp \\
\midrule
ResNet 50 R  &   131 &    17 &   408 &      7 \\
ResNet 152 R &   343 &     2 &   599 &      1 \\
ResNet 18 10 &    52 &   122 &   286 &     13 \\
ConViT Ti R  &    86 &     3 &   170 &      1 \\
ConViT B R   &   129 &     9 &   201 &      4 \\
ConViT S R   &     5 &    76 &     9 &     24 \\
ViT B        &   341 &   157 &  2422 &     18 \\
\bottomrule
\end{tabular}
    \caption{Number of Unique Images by Confusion Metric Categories}
    \label{tab:unique}
\end{table}

An interesting observation is that ViT, despite performing poorly overall, still correctly identifies a significant number of images that are challenging for other models. Overall, ResNets have far more false negatives where they alone under-performed. This is an important observation, as this is for the minority class. Here, one might consider whether this gap would grow with the availability of more data.

\section{Qualitative Results}
\label{sec:Qualitative Results}
We show images from the test 19k test set - which are identified by each network in $\in \{tp,tn,fp,fn\}$ - and are members of the exclusive sets shown in table \ref{tab:unique}. This may provide insight into the types of images that each network is performing best and worst (that is unique to that each networks inference). Images are chosen at random from each exclusive set. This is to provide insight into how each network is selecting attributes that may be visible to the human eye.  It is clear from the MOS score that these images are generally from the ambiguous range of MOS.

\subsection{ResNets and ConViTs True Negative}

\begin{figure}[ht!]

    \subfloat[True Negatives Only Identified By ConViT]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/ConViT B TN.png} \label{fig:ConViT_B_TN}}
     \subfloat[ConViT S True Negative Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/ConViT S TN.png}
     \label{fig:ConViT_S_TN}}
     \subfloat[ConViT Ti True Negative Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/ConViT Ti TN.png}
     \label{fig:ConViT_Ti_TN}}
    \vfill
    \subfloat[ResNet 152 True Negative Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/resnet 152 TN.png}
    \label{fig:ResNet_152_TN}}
    \subfloat[ResNet 50 True Negative Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/Resnet 50 TN.png}\label{fig:ResNet_50_TN}}
    \subfloat[ResNet 18 True Negative Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/ResNet 18 TN}, \label{fig:ResNet_18_TN}}
    \caption{Examples of Exclusive True Negatives Predictions by Each Network (Exclusive Set) on AVA Test Set}
    \label{fig:true_negative}
\end{figure}

ConViTs (\ref{fig:ConViT_B_TN}, \ref{fig:ConViT_S_TN}, \ref{fig:ConViT_Ti_TN}) appear to pay more attention to composition information, where as ResNets - shown in figures ( \ref{fig:ResNet_152_TN}, \ref{fig:ResNet_50_TN}, \ref{fig:ResNet_18_TN})- appear to predict better where attributes such as good lighting and colour harmony are distinguishing features for images in the ambiguous range.

\subsection{ResNets and ConViTs True Positive Images}
\label{qualitative positives}

ConViT models appear to be less able to resolve positive class images that are not identified elsewhere. Again, it seems clear that interplay between composition and salient objects/distinguishing features, where figures (\ref{fig:ConViT_B_TP}, \ref{fig:ConViT_S_TP},\ref{fig:ConViT_Ti_TP}) appear to show some semantic class ambiguity \ref{fig:ConViT_Ti_TP}. Left shows an animal made of grass/leaves and \ref{fig:ConViT_B_TP} shows an image of parking meters that have eyes.

\begin{figure}[ht!]

    \subfloat[True Positives Only Identified By ConViT]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/ConViT B TP.png} \label{fig:ConViT_B_TP}}
     \subfloat[ConViT S True Positive Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/ConViT S TP.png}
     \label{fig:ConViT_S_TP}}
     \subfloat[ConViT Ti True Positive Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/ConViT Ti TP.png}
     \label{fig:ConViT_Ti_TP}}
    \vfill
    \subfloat[ResNet 152 True Positive Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/resnet_152 TP.png}
    \label{fig:ResNet_152_TP}}
    \subfloat[ResNet 50 True Positive Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/resnet_50 TP.png}\label{fig:ResNet_50_TP}}
    \subfloat[ResNet 18 True Negative Imgaes]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/ResNet 18 TP.png}, \label{fig:ResNet_18_TP}}
    \caption{Examples of Exclusive True Positive Predictions by Each Network (Exclusive Set) on AVA Test Set}
    \label{fig:true_positive}

\end{figure}
\subsection{ResNets and ConViTs False Negative Images}
\label{qualitative false negative}

ConViT $FN$s are shown in figures (\ref{fig:ConViT_B_FN}, \ref{fig:ConViT_S_FN},\ref{fig:ConViT_Ti_FN}) and appear to falsely rate images as negative with a bias again towards high level semantic information, in contrast with ResNets' $FN$s shown in  figures(\ref{fig:ResNet_152_FN}, \ref{fig:ResNet_50_FN}, \ref{fig:ResNet_18_FN}).

\begin{figure}[ht!]

    \subfloat[ConVit B Only False Negative ]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_negative/ConVit B FN.png} \label{fig:ConViT_B_FN}}
     \subfloat[ConViT S Only False Negative Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_negative/ConVit S FN.png}
     \label{fig:ConViT_S_FN}}
     \subfloat[ConViT Ti Only False Negative Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_negative/ConVit Ti FN.png}
     \label{fig:ConViT_Ti_FN}}
    \vfill
    \subfloat[ResNet 152 Only False Negative Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_negative/ResNet 152 FN.png}
    \label{fig:ResNet_152_FN}}
    \subfloat[ResNet 50 Only False Negative Images]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_negative/ResNet 50 FN.png}\label{fig:ResNet_50_FN}}
    \subfloat[ResNet 18 Only False Negative Imgaes]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_negative/ResNet 18 FN .png}, \label{fig:ResNet_18_FN}}
    \caption{Examples of Exclusive False Negative Predictions by Each Network (Exclusive Set) on AVA Test Set }
    \label{fig:true_positive}

\end{figure}
\subsection{ResNets and ConViTs False Positive Images}
\label{qualitative false positive}
\begin{figure}[ht!]
    \subfloat[ConViT B Only False Positive]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_positive/ConVit B FP.png} \label{fig:ConViT_B_FP}}
     \subfloat[ConViT S Only False Positive]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_positive/ConVit S FP.png}
     \label{fig:ConViT_S_FP}}
     \subfloat[ConViT Ti Only False Positive]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_positive/ConVit Ti FP.png}
     \label{fig:ConViT_Ti_FP}}
    \vfill
    \subfloat[ResNet 152 Only False Positives]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_positive/ResNet 152 FP.png}
    \label{fig:ResNet_152_FP}}
    \subfloat[ResNet 50 Only False Positive]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_positive/ResNet 50 FP.png}\label{fig:ResNet_50_FP}}
    \subfloat[ResNet 18 Only False Positive]{\includegraphics[width=0.333\textwidth]{figures/results_and_discussion/qualitative_results/false_positive/ResNet 18 FP .png}, \label{fig:ResNet_18_FP}}
    \caption{Examples of Exclusive False Positive Predictions by Each Network (Exclusive Set) on AVA Test Set}
    \label{fig:true_positive}
\end{figure}

ConViT $FP$ images appear to show a bias for global form, shown in figures (\ref{fig:ConViT_B_FP}, \ref{fig:ConViT_S_FP},\ref{fig:ConViT_Ti_FP}), however, not the MOS of figure \ref{fig:ConViT_Ti_FP}, which is exacly 5.0 where negative class is $\leq 5$. Figures (\ref{fig:ResNet_152_FP}, \ref{fig:ResNet_50_FP}, \ref{fig:ResNet_18_FP}) show $FPs$ exclusive to respective ResNets.

\subsection{Training Metrics}

When shown in the same plot (figure \ref{fig:all models}), it is clear that ResNet50 and 152 overfit during training and that ConViTs (as well as CvT) consistently improve with accuracy in a more linear fashion. Further, it is noteworthy that ViT (pure transformer) shows a dip in initial training accuracy during training, and validation loss steadily increases.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.99\textwidth]{figures/results_and_discussion/All Models Compared ViT,ConViT,CNN .png}
    \caption{All Models Compared}
    \label{fig:all models}
\end{figure}

High maximum validation loss for all of the ResNet models is shown in table \ref{tab:summary of training}. It is also noteworthy that ViT B performs well on the validation set (outstripping all other models which declined after first epoch).
\begin{table}[ht!]
\small
    \centering
    \begin{tabular}{lrrrr}
\toprule
{} &  Train. Acc. &  Train. Loss &  Val. Acc. &  Val. Loss \\
\midrule
ViT B      &      0.778 &       0.604 &    0.786 &     0.621 \\
ConViT B   &      0.766 &       0.629 &    0.781 &     0.549 \\
ConViT S   &      0.767 &       0.638 &    0.746 &     0.554 \\
ConViT Ti  &      0.765 &       0.663 &    0.762 &     0.578 \\
ResNet 152 &      0.980 &       0.535 &    0.741 &     1.074 \\
ResNet 50  &      0.906 &       0.604 &    0.710 &     0.819 \\
ResNet 18  &      0.726 &       0.665 &    0.706 &     1.232 \\
CvT        &      0.700 &       0.681 &    0.625 &     0.711 \\

\bottomrule
\end{tabular}
\caption{Overall Maximum Training Results Compared}
    \label{tab:summary of training}
\end{table}

\newpage

We also show inference times and training times per epoch to give additional compute resources, and show how models might perform when embedded in a real world application (table \ref{tab:compute time}). ConVit B is clearly resource intensive.

\begin{table}[ht!]
    \centering
    \tiny
   \begin{tabular}{lccc}
\toprule
{} &  \# Million Trainable Params. & Train Time Per Epoch 230k Images & Inference Time 19k Images\\
\midrule
ResNet 18  &        11.2 & 56 mins.  & 3 mins.\\
ResNet 50  &        23.5 & 73 mins.  & 4 mins.\\
ResNet 152 &        58.1 & 121 mins. & 5 mins.\\
ConViT Ti  &         5.5 & 78 mins.  & 3 mins.\\
ConViT S   &        27.3 & 92 mins.  & 4 mins.\\
ConViT B   &        85.8 & 144 mins. & 5 mins.\\
ViT B      &        85.8 & 121 mins. & 5 mins.\\
CvT        &        17.6 & 94 mins.  & 4 mins.\\
\bottomrule
\end{tabular}
    \caption{Number Parameters and Training Time Per Epoch 230k Images on 1 GPU- batch size 10 Inference on 19k Images on AVA dataset}
    \label{tab:compute time}
\end{table}


\documentclass[a4paper]{article}

%% Language and font encodings
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}

%% Sets page size and margins
\usepackage[a4paper,top=3cm,bottom=3cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}

%% Useful packages
\usepackage{graphicx}
\usepackage{url}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{physics}
\usepackage{amssymb}
\usepackage{mathtools}

\numberwithin{equation}{subsection}
\newcommand{\mb}[1]{\mathbf{#1}}
\title{Floquet-Drude Conductivity in Dressed Quantum Hall Systems}
\author{Kosala Sananthana Herath}
\numberwithin{equation}{section}

\begin{document}

\maketitle

\section*{Abstract}

Interactions between light and matter have dragged research attention in the fields of optoelectronics, sensing, energy harvesting, quantum computing, bio-information, and in many branches of recent technologies. For many years, the foremost aims for examing the characteristics of dressed fermion systems were focused on the different types of atomic and molecular arrangements. These researches of extreme electron-light engagements introduced an astonishing scope of twentieth-century physics namely quantum optic physics.

On the other hand, in nanostructures that are applicable in electronic devices, the investigations with the help of quantum optic were centered on polaritonic and exciton influences on nanostructures and material characteristics of dressed electrons in two-dimensional(2D) materials and quantum wires. When considering the transport characteristics of dressed nanostructures, they are still expecting extensive analysis.

Therefore, transport properties of nanostructures exposed to a high intensity periodic electromagnetic fields have been explored theoretically in this study. The dressing field is analyzed non perturbatively using the Floquet theory whilst the probing field is examined perturbatively by applying the linear response method using the Kubo formula. The general Floquet-Drude conductivity has been derived in a fully closed analytical form in most recent research [1,2], introducing a novel type of Green’s functions namely four-times Green’s functions.
As a consequence, the established formalism introduces a novel approach to manipulate the transport characteristics of nanostructures by an intense dressing field. From an empirical sense, this study applies directly to various nanostructures illuminated by a high-intensity electromagnetic field.
In this research we have developed a robust mathematical model for dressed two-dimensional electron gas(2DEG) exposed to another stationary magnetic field and that will enable efficient manipulation of transport characteristics in nanoscale electronic devices.

\section*{Purpose of the Study}

When a stationary magentic field applied perpendicularly across the surface of 2DEG systems, the orbital motion of electrons becomes completely quantized and the energy spectrium becomes discrete by creating Landau levels. Such a singular system known as a quantum Hall system and in this study we explictly calculate the diagonal ($\sigma_{xx},\sigma_{yy}$) components of the conductivity tensor in the periodically driven quantum Hall systems.

Although there are already exist a number of adavanced theories devoted to the calculation of conductivity tensor elements in a quantum Hall systems [3-5], they have not been applied to the optically manipulation the magneto-electric properties of the quantum Hall systems. However K. Dini et al. [6] have recently investigated the one directional conductivity behaviour of dressed quantum Hall systems, they have not used the state of art model to describe the conductivity in a quantum Hall system. In their study they used the conductivity models from T. Ando et al. [3,4] and as mentioned in A. Endo et al. investigation [5] those models are far less accurate representation of the experimantally observed Landau levels because they present a semi-elliptical broadening.

In this study we develop a genralized mathematical model to describe transport properties of dressed quantum Hall systems using Floquet-Drude conductivity [1,2]. In addition, we demonstrate that our generalized model is agreed with the state of art conductivity model [5] for specalized quantum Hall system which has been considered without the external dressing field.

\section*{Research Method}

In this analysis we consider 2DEG which has been distrubuted in confined $(x,y)$ plane in configuration space.
\begin{figure}[ht!]
  \centering
  \includegraphics[scale=0.9]{figures/fig0.pdf}
  \caption{Confined 2DEG in configuration space with the size of $A=L_xL_y$.}
  \label{fig:1}
\end{figure}

We examined the properties of 2DEG with stationary magnetic field which directed on $z$ axis and a linearly $y$-polarized strong electomagnetic wave (dressing field) with electric field which also propagate in $z$ direction.
\begin{figure}[ht!]
  \centering
  \includegraphics[scale=0.9]{figures/fig1.pdf}
  \caption{Stationary magnetic filed (blue color) and Strong EM wave (red color) applied to the 2DEG.}
  \label{fig:2}
\end{figure}

In the state of art conductivity model [5] which only describes trasport properies quantum Hall system without external dressing field has used an approximation equivalent to asssumin that the broadening of a Landau level due to disorder is represented by a constant broadening. However, from the Floquet-Drude conductivity model we have foud that we can manipulate the impurity broadening using intense high-frequency electromagnetic waves and this study we have developed a mathematical formula to calculate this Landau energy level broadening and using those values we were able to evaluate diagonal elements of conductivity tensor in dressed quantum Hall systems. Ultimately we can express the normalized conductivity for a quantum Hall system with linearly polarized drerssing field analytically as follows
\begin{equation*}
    \sigma^{dd}(X_F,\tilde{I}) =
    \sum_{n}
    \frac{\qty(n+1)}{\gamma_{n}\gamma_{n+1}}
    \qty[
      \frac{1}
      {
        1 + \qty(\frac{X_F - n -1}{\gamma_{n+1}})^2
      }
    ]
    \qty[
      \frac{1}
      {
        1 + \qty(\frac{X_F - n}{\gamma_{n}})^2
      }
    ]
\end{equation*}
where $X_F$ is normalized Fermi level and $\gamma_n$ represnts normalized energy band broadening of $n$-th Landau level. This $\gamma_n$ only depends on the normalized intensity value of dressing field $\tilde{I}$. We can choose the direction $d$ of the diagonal conductivity tensor element from $(x,y)$.

\section*{Outcomes and Discussion}

First we have identified that we can control the Landau level broadening using the dressing field intensity as given in the Figure \ref{fig:3}.
\begin{figure}[ht!]
  \centering
  \includegraphics[scale=0.6]{figures/fig3.pdf}
  \caption{The dependence of the normalized Landau level broadening on the dressing field intensity for different Landau levels ($N=0,1,2,3,4$).}
  \label{fig:3}
\end{figure}

As mentioned in above derived analytical expression the changes can be done to the transverse conductivity using external dressing field. As given in Figure \ref{fig:4} and \ref{fig:5} we can manipulate the conductivity using external dressing field.

\begin{figure}[ht!]
  \centering
  \includegraphics[scale=1]{figures/fig4.pdf}
  \caption{Normalized transverse conductivity against Fermi level ($X_F$) with different intensities ($\tilde{I}$) of dressing field.}
  \label{fig:4}
\end{figure}

When the dressing field's intensity increase the broadening of energy bands of Landau levels get reduced and the conductivity also get decrease in all the regions except the peak points of each level. Using this manipulation we can filter the conductivity which is change with the Fermi level. Since Fermi level can be change with the applied gate voltage of the material this can be used as a 2D switch for optoelectonic applications. However with the existance of  external dressing field we can fine tune the switching mechanism and it is an exceptional advantage in optoelectonic applications.

\begin{figure}[ht!]
  \centering
  \includegraphics[scale=0.9]{figures/fig5.pdf}
  \caption{$3$rd Landau level's normalized transverse conductivity against Fermi level ($X_F$) with different intensities ($\tilde{I}$) of dressing field.}
  \label{fig:5}
\end{figure}

Now we can compare our result with previous studies to the check the validity of the outcomes. In the research done by K. Dini et al. [6] has demonstrated that the diagonal elements of conductivity tensor in a quantum Hall system can manipute with apllying high intense light as given in Figure \ref{fig:6}. However the behaviour of the conductivity with the Fermi level is not accurate as given in the state of art study [5] which has been represent in Figure \ref{fig:7}. When we consider the our outcome which derived using novel Floquet-Drude conductivity model, we can see that it agree with the results given in the much accurate study [5]. However, our study present a more generalized mathematical model rather than the specific scenario studied in the study [6].

\begin{figure}[ht!]
  \centering
  \includegraphics[scale=0.3]{figures/fig6.pdf}
  \caption{The dependence of the longitudinal conductivity $\sigma_{xx}$ , on
the Fermi energy $\varepsilon_F$. The solid line describes the conductivity
of unirradiated 2DEG, whereas the dotted line corresponds to the
conductivity at the irradiation intensity $I = 600 \; \text{W/cm2}$. The inset
shows the difference between these two conductivities. (from K.Dini at el. [6])}
  \label{fig:6}
\end{figure}

\newpage
\begin{figure}[ht!]
  \centering
  \includegraphics[scale=0.3]{figures/fig7.pdf}
  \caption{The diagonal and the off-diagonal components
of the conductivity tensor modified to account for the long-range potential. The
horizontal axis is the inverse magnetic field. (from A. Endo at el. [5])}
  \label{fig:7}
\end{figure}

We can conclude that the using external dressing field we can manipulate the magneto-electric properties of a quantum Hall system and in this study we were able to demonstrate that using Floquet-Drude conductivity method we can derive a more accurate generalized mathematical model that describe the trasport properties of quantum Hall syatems. Therefore this theory describes that the  dressing field can be used as a tool to utilize transport properties in various 2D nanostructures which serve as a basis for nano-optoelectonic devices.

\section*{Reference}

[1] M. Wackerl, P. Wenk, and J. Schliemann, Floquet-Drude Conductivity, Phys. Rev. B 101, 184204 (2020).

\noindent
[2] M. Wackerl, Transport in Periodically Driven Systems, University of Regensburg, (2020).

\noindent
[3] T. Ando and Y. Uemura, Theory of Quantum Transport in a Two-Dimensional Electron System under Magnetic Fields. I. Characteristics of Level Broadening and Transport under Strong Fields, J. Phys. Soc. Jpn. 36, 959 (1974).

\noindent
[4] T. Ando, A. B. Fowler, and F. Stern, Electronic Properties of Two-Dimensional Systems, Rev. Mod. Phys. 54, 437 (1982).

\noindent
[5] A. Endo, N. Hatano, H. Nakamura, and R. Shirasaki, Fundamental Relation between Longitudinal and Transverse Conductivities in the Quantum Hall System, J. Phys.: Condens. Matter 21, 345803 (2009).

\noindent
[6] K. Dini, O. V. Kibis, and I. A. Shelykh, Magnetic Properties of a Two-Dimensional Electron Gas Strongly Coupled to Light, Phys. Rev. B 93, 235411 (2016).

\end{document}


\subsubsection*{Manipulation Operations \hspace*{\fill}\hyperlink{ElementwiseOperations}{(up)}\hypertarget{manipulationOperations}{}} \addcontentsline{toc}{subsubsection}{Manipulation Operations}
Manipulation operations are functions which copy \ttbf{view}s, or parts of \ttbf{view}s, from one location to another while doing some manipulation operation to convert the data. For instance the \ttbf{cmplx} function takes two real \ttbf{view}s and copies one \ttbf{view} to the imaginary part of a complex vector and the other \ttbf{view} to the real part of a complex vector.
\begin{table}[H]
\caption{Manipulation Operations}
\label{tab:manipulationOperations}
\begin{center}
\begin{tabular}{|l|l|}
\hline
\hlnkFunc{cmplx} & Complex\\
\hlnkFunc{gather} & Data Gather\\
\hlnkFunc{imag} & Imaginary Part\\
\hlnkFunc{polar} & Hypotenuse\\
\hlnkFunc{real} & Real Part\\
\hlnkFunc{rect} & Rectangular\\
\hlnkFunc{scatter} & Data Scatter\\
\hlnkFunc{swap} & Swap\\
\hline
\end{tabular}
\end{center}
%\label{default}
\end{table}%


\subsection{Decision Trees}
\label{sec:decision_trees}

\noindent{\bf Description}
\smallskip

Decision tree (for classification) is a classifier that is considered
more interpretable than other statistical classifiers. This implementation
is well-suited to handle large-scale data and builds a (binary) decision
tree in parallel.\\

\smallskip
\noindent{\bf Usage}
\smallskip

{\hangindent=\parindent\noindent\it%
	{\tt{}-f }path/\/{\tt{}decision-tree.dml}
	{\tt{} -nvargs}
	{\tt{} X=}path/file
	{\tt{} Y=}path/file
	{\tt{} R=}path/file
	{\tt{} bins=}integer
	{\tt{} depth=}integer
	{\tt{} num\_leaf=}integer
	{\tt{} num\_samples=}integer
	{\tt{} impurity=}Gini$\mid$entropy
	{\tt{} M=}path/file
	{\tt{} O=}path/file
	{\tt{} S\_map=}path/file
	{\tt{} C\_map=}path/file
	{\tt{} fmt=}format

}

 \smallskip
 \noindent{\bf Usage: Prediction}
 \smallskip

 {\hangindent=\parindent\noindent\it%
 	{\tt{}-f }path/\/{\tt{}decision-tree-predict.dml}
 	{\tt{} -nvargs}
 	{\tt{} X=}path/file
 	{\tt{} Y=}path/file
 	{\tt{} R=}path/file
 	{\tt{} M=}path/file
 	{\tt{} P=}path/file
 	{\tt{} A=}path/file
 	{\tt{} CM=}path/file
 	{\tt{} fmt=}format

 }\smallskip

\noindent{\bf Arguments}
\begin{Description}
	\item[{\tt X}:]
	Location (on HDFS) to read the matrix of feature vectors;
	each row constitutes one feature vector. Note that categorical features in $X$ need to be both recoded and dummy coded.
	\item[{\tt Y}:]
	Location (on HDFS) to read the matrix of (categorical)
	labels that correspond to feature vectors in $X$. Note that class labels are assumed to be both recoded and dummy coded.
	This argument is optional for prediction.
	\item[{\tt R}:] (default:\mbox{ }{\tt " "})
	Location (on HDFS) to read matrix $R$ which for each feature in $X$ contains column-ids (first column), start indices (second column), and end indices (third column).
	If $R$ is not provided by default all features are assumed to be continuous-valued.
	\item[{\tt bins}:] (default:\mbox{ }{\tt 20})
	Number of thresholds to choose for each continuous-valued feature (determined by equi-height binning).
	\item[{\tt depth}:] (default:\mbox{ }{\tt 25})
	Maximum depth of the learned tree
	\item[{\tt num\_leaf}:] (default:\mbox{ }{\tt 10})
	Parameter that controls pruning. The tree
	is not expanded if a node receives less than {\tt num\_leaf} training examples.
	\item[{\tt num\_samples}:] (default:\mbox{ }{\tt 3000})
	Parameter that decides when to switch to in-memory building of subtrees. If a node $v$ receives less than {\tt num\_samples}
	training examples then this implementation switches to an in-memory subtree
	building procedure to build the subtree under $v$ in its entirety.
	\item[{\tt impurity}:] (default:\mbox{ }{\tt "Gini"})
	Impurity measure used at internal nodes of the tree for selecting which features to split on. Possible value are entropy or Gini.
	\item[{\tt M}:]
	Location (on HDFS) to write matrix $M$ containing the learned decision tree (see below for the schema)
	\item[{\tt O}:] (default:\mbox{ }{\tt " "})
	Location (on HDFS) to store the training accuracy (\%). Note that this argument is optional.
	\item[{\tt A}:] (default:\mbox{ }{\tt " "})
	Location (on HDFS) to store the testing accuracy (\%) from a
	held-out test set during prediction. Note that this argument is optional.
	\item[{\tt P}:]
	Location (on HDFS) to store predictions for a held-out test set
	\item[{\tt CM}:] (default:\mbox{ }{\tt " "})
	Location (on HDFS) to store the confusion matrix computed using a held-out test set. Note that this argument is optional.
	\item[{\tt S\_map}:] (default:\mbox{ }{\tt " "})
	Location (on HDFS) to write the mappings from the continuous-valued feature-ids to the global feature-ids in $X$ (see below for details). Note that this argument is optional.
	\item[{\tt C\_map}:] (default:\mbox{ }{\tt " "})
	Location (on HDFS) to write the mappings from the categorical feature-ids to the global feature-ids in $X$ (see below for details). Note that this argument is optional.
	\item[{\tt fmt}:] (default:\mbox{ }{\tt "text"})
	Matrix file output format, such as {\tt text}, {\tt mm}, or {\tt csv};
	see read/write functions in SystemML Language Reference for details.
\end{Description}

 \noindent{\bf Details}
 \smallskip

Decision trees~\cite{BreimanFOS84:dtree} are simple models of
classification that,  due to their structure,  are easy to
interpret. Given an example feature vector, each node in the learned
tree runs a simple test on it. Based on the result of the test, the
example is either diverted to the left subtree or to the right
subtree. Once the example reaches a leaf, then the label stored at the
leaf is returned as the prediction for the example.

Building a decision tree from a fully labeled training set entails
choosing appropriate splitting tests for each internal node in the tree and this is usually performed in a top-down manner.
The splitting test (denoted by $s$) requires
first choosing a feature $j$ and depending on the type of $j$, either
a threshold $\sigma$, in case $j$ is continuous-valued, or a subset of
values $S \subseteq \text{Dom}(j)$ where $\text{Dom}(j)$ denotes
domain of $j$, in case it is categorical. For continuous-valued
features the test is thus of form $x_j < \sigma$ and for categorical
features it is of form $x_j \in S$, where $x_j$ denotes the $j$th
feature value of feature vector $x$. One way to determine which test
to include, is to compare impurities of the tree nodes induced by the test.
The {\it node impurity} measures the homogeneity of the labels at the node. This implementation supports two commonly used impurity measures (denoted by $\mathcal{I}$): {\it Entropy} $\mathcal{E}=\sum_{i=1}^{C}-f_i \log f_i$, as well as {\it Gini impurity} $\mathcal{G}=\sum_{i=1}^{C}f_i (1-f_i)$, where $C$ denotes the number of unique labels and $f_i$ is the frequency of label $i$.
Once the impurity at the tree nodes has been obtained, the {\it best split} is chosen from a set of possible splits that maximizes the {\it information gain} at the node, i.e., $\arg\max_{s}\mathcal{IG}(X,s)$, where $\mathcal{IG}(X,s)$ denotes the information gain when the splitting test $s$ partitions the feature matrix $X$.
Assuming that $s$ partitions $X$ that contains $N$ feature vectors into $X_\text{left}$ and $X_\text{right}$ each including $N_\text{left}$ and $N_\text{right}$ feature vectors, respectively, $\mathcal{IG}(X,s)$ is given by
\begin{equation*}
\mathcal{IG}(X,s)=\mathcal{I}(X)-\frac{N_\text{left}}{N}\mathcal{I}(X_\text{left})-\frac{N_\text{right}}{N}\mathcal{I}(X_\text{right}),
\end{equation*}
where $\mathcal{I}\in\{\mathcal{E},\mathcal{G}\}$.
In the following we discuss the implementation details specific to {\tt decision-tree.dml}.

\textbf{Input format.}
In general implementations of the decision tree algorithm do not require categorical features to be dummy coded. For improved efficiency and reducing the training time, our implementation however assumes dummy coded categorical features and dummy coded class labels.

\textbf{Tree construction.}
Learning a decision tree on large-scale data has received some
attention in the literature. The current implementation includes logic
for choosing tests for multiple nodes that belong to the same level in
the decision tree in parallel (breadth-first expansion) and for
building entire subtrees under multiple nodes in parallel (depth-first
subtree building). Empirically it has been demonstrated that it is
advantageous to perform breadth-first expansion for the nodes
belonging to the top levels of the tree and to perform depth-first
subtree building for nodes belonging to the lower levels of the tree~\cite{PandaHBB09:dtree}. The parameter {\tt num\_samples} controls when we
switch to  depth-first subtree building. Any node in the decision tree
that receives $\leq$ {\tt num\_samples} training examples, the subtree
under it is built in its entirety in one shot.

\textbf{Stopping rule and pruning.}
The splitting of data at the internal nodes stops when at least one the following criteria is satisfied:
\begin{itemize}
	\item the depth of the internal node reaches the input parameter {\tt depth} controlling the maximum depth of the learned tree, or
	\item no candidate split achieves information gain.
\end{itemize}
This implementation also allows for some automated pruning via the argument {\tt num\_leaf}. If
a node receives $\leq$ {\tt num\_leaf} training examples, then a leaf
is built in its place.

\textbf{Continuous-valued features.}
For a continuous-valued feature
$j$ the number of candidate thresholds $\sigma$ to choose from is of
the order of the number of examples present in the training set. Since
for large-scale data this can result in a large number of candidate
thresholds, the user can limit this number via the arguments {\tt bins} which controls the number of candidate thresholds considered
for each continuous-valued feature. For each continuous-valued
feature, the implementation computes an equi-height histogram to
generate one candidate threshold per equi-height bin.

\textbf{Categorical features.}
In order to determine the best value subset to split on in the case of categorical features, this implementation greedily includes values from the feature's domain until the information gain stops improving.
In particular, for a categorical feature $j$ the $|Dom(j)|$ feature values are sorted by impurity and the resulting split candidates $|Dom(j)|-1$ are examined; the sequence of feature values which results in the maximum information gain is then selected.

\textbf{Description of the model.}
The learned decision tree is represented in a matrix $M$ that
contains at least 6 rows. Each column in the matrix contains the parameters relevant to a single node in the tree.
Note that for building the tree model, our implementation splits the feature matrix $X$ into $X_\text{cont}$ containing continuous-valued features and $X_\text{cat}$ containing categorical features. In the following, the continuous-valued (resp. categorical) feature-ids correspond to the indices of the features in $X_\text{cont}$ (resp. $X_\text{cat}$).
Moreover, we refer to an internal node as a continuous-valued (categorical) node if the feature that this nodes looks at is continuous-valued (categorical).
Below is a description of what each row in the matrix contains.
\begin{itemize}
\item Row 1: stores the node-ids. These ids correspond to the node-ids in a complete binary tree.
\item Row 2: for internal nodes stores the offsets (the number of columns) in $M$ to the left child, and otherwise 0.
\item Row 3: stores the feature index of the feature (id of a continuous-valued feature in $X_\text{cont}$ if the feature is continuous-valued or id of a categorical feature in $X_\text{cat}$ if the feature is categorical) that this node looks at if the node is an internal node, otherwise 0.
\item Row 4: store the type of the feature that this node looks at if the node is an internal node: 1 for continuous-valued and 2 for categorical features,
otherwise the label this leaf node is supposed to predict.
\item Row 5: for the internal nodes contains 1 if the feature chosen for the node is continuous-valued, or the size of the subset of values used for splitting at the node stored in rows 6,7,$\ldots$ if the feature chosen for the node is categorical. For the leaf nodes, Row 5 contains the number of misclassified training examples reaching at this node.
\item Row 6,7,$\ldots$: for the internal nodes, row 6 stores the threshold to which the example's feature value is compared if the feature chosen for this node is continuous-valued, otherwise if the feature chosen for this node is categorical rows 6,7,$\ldots$ store the value subset chosen for the node.
For the leaf nodes, row 6 contains 1 if the node is impure and the number of training examples at the node is greater than {\tt num\_leaf}, otherwise 0.
\end{itemize}
As an example, Figure~\ref{dtree} shows a decision tree with $5$ nodes and its matrix
representation.

\begin{figure}
\begin{minipage}{0.3\linewidth}
\begin{center}
\begin{tikzpicture}
\node (labelleft) [draw,shape=circle,minimum size=16pt] at (2,0) {$2$};
\node (labelright) [draw,shape=circle,minimum size=16pt] at (3.25,0) {$1$};

\node (rootleft) [draw,shape=rectangle,minimum size=16pt] at (2.5,1) {$x_5 \in \{2,3\}$};
\node (rootlabel) [draw,shape=circle,minimum size=16pt] at (0.9,1) {$1$};
\node (root) [draw,shape=rectangle,minimum size=16pt] at (1.75,2) {$x_3 < 0.45$};

\draw[-latex] (root) -- (rootleft);
\draw[-latex] (root) -- (rootlabel);
\draw[-latex] (rootleft) -- (labelleft);
\draw[-latex] (rootleft) -- (labelright);

\end{tikzpicture}
\end{center}
\begin{center}
(a)
\end{center}
\end{minipage}
\hfill
\begin{minipage}{0.65\linewidth}
\begin{center}
\begin{tabular}{c|c|c|c|c|c|}
& Col 1 & Col 2 & Col 3 & Col 4 & Col 5\\
\hline
Row 1 & 1 & 2 & 3 & 6 & 7 \\
\hline
Row 2 & 1 & 0 & 1 & 0 & 0 \\
\hline
Row 3 & 3 & 5 & 0 & 0 & 0 \\
\hline
Row 4 & 1 & 1 & 2 & 2 & 1 \\
\hline
Row 5 & 1 & 0 & 2 & 0 & 0 \\
\hline
Row 6 & 0.45 & 0 & 2 & 0 & 0 \\
\hline
Row 7 &  &  & 3 &  & \\
\hline
\end{tabular}
\end{center}
\begin{center}
(b)
\end{center}
\end{minipage}
\caption{(a) An example tree and its (b) matrix representation. $x$ denotes an example and $x_j$ is the value of the $j$th continuous-valued (resp. categorical) feature in $X_\text{cont}$ (resp. $X_\text{cat}$). In this example all leaf nodes are pure and no training example is misclassified.}
\label{dtree}
\end{figure}

\smallskip
\noindent{\bf Returns}
\smallskip

The matrix corresponding to the learned model as well as the training accuracy (if requested) is written to a file in the format specified. See
details where the structure of the model matrix is described.
Recall that in our implementation $X$ is split into $X_\text{cont}$ and $X_\text{cat}$. If requested, the mappings of the continuous-valued feature-ids in $X_\text{cont}$ (stored at {\tt S\_map}) and the categorical feature-ids in $X_\text{cat}$ (stored at {\tt C\_map}) to the global feature-ids in $X$ will be provided.
Depending on what arguments are provided during
invocation, the {\tt decision-tree-predict.dml} script may compute one or more of predictions, accuracy and confusion matrix in the requested output format.

\smallskip
\noindent{\bf Examples}
\smallskip

{\hangindent=\parindent\noindent\tt
	\hml -f decision-tree.dml -nvargs X=/user/biadmin/X.mtx Y=/user/biadmin/Y.mtx
	R=/user/biadmin/R.csv M=/user/biadmin/model.csv
	bins=20 depth=25 num\_leaf=10 num\_samples=3000 impurity=Gini fmt=csv

}\smallskip

\noindent To compute predictions:

{\hangindent=\parindent\noindent\tt
	\hml -f decision-tree-predict.dml -nvargs X=/user/biadmin/X.mtx Y=/user/biadmin/Y.mtx R=/user/biadmin/R.csv
	M=/user/biadmin/model.csv  P=/user/biadmin/predictions.csv
	A=/user/biadmin/accuracy.csv CM=/user/biadmin/confusion.csv fmt=csv

}\smallskip

%\noindent{\bf References}
%
%\begin{itemize}
%\item B. Panda, J. Herbach, S. Basu, and R. Bayardo. \newblock{PLANET: massively parallel learning of tree ensembles with MapReduce}. In Proceedings of the VLDB Endowment, 2009.
%\item L. Breiman, J. Friedman, R. Olshen, and C. Stone. \newblock{Classification and Regression Trees}. Wadsworth and Brooks, 1984.
%\end{itemize}


\chapter{Statics}

% https://en.wikipedia.org/wiki/Timeline_of_fundamental_physics_discoveries

\emph{Thermodynamics} began as a theory of steam engines.

\emph{Volume} is how much space something occupies.

\emph{Density} is weight per volume.

% https://en.wikipedia.org/wiki/Work_(physics)
\index{definitions!work}%
\index{work!definition}%
\emph{Work}: If one lifts a weight \(F\) so that its height increases by \(h\),
then he does a \emph{work} of \( W = F \cdot h \).
Coriolis defined this in 1826 \cite{coriolis1829calcul}
when steam engines lifted buckets of water out of flooded ore mines.
We shall generalize this definition later,
if the force and the displacement make an angle.

% https://en.wikipedia.org/wiki/Power_(physics)
\index{definitions!power}%
\index{power!definition}%
\emph{Power} is work done per unit time: \( P = W / t \).
This means that a steam engine with twice the power
will clean the same mine in half the time.

% https://hsm.stackexchange.com/questions/414/when-were-the-modern-notions-of-work-and-energy-created
% Helmholtz 1847?

\section{Archimedes's principle of buoyancy}

% https://en.wikipedia.org/wiki/Archimedes%27_principle
% https://en.wikipedia.org/wiki/On_Floating_Bodies

Put a solid into a container full of liquid.

The volume of the spilled part of the liquid is equal to
the volume of the submerged part of the solid.

\index{Archimedes!principle of buoyancy}%
\index{laws named after people!Archimedes's principle of buoyancy}%
\index{laws!buoyancy}%
\paragraph{Archimedes's principle of buoyancy}
Equal are the weight of the object and the buoyant force on the object.
(???)

\section{Pascal's law of fluid pressure transmission}

Blaise Pascal 1647

Pascal's law: Incompressible fluid spreads pressure evenly.

\index{Pascal!law of fluid pressure transmission}%
\index{laws named after people!Pascal's law of fluid pressure transmission}%
\index{laws!fluid pressure transmission}%
\index{statics!Pascal's law of fluid pressure transmission}%
\( P = \rho g h \)

\paragraph{Appreciating Pascal's barrel demonstration}

Counterintuitive: The hydrostatic pressure
does not depend on \emph{how much} fluid.
It depends on \emph{how deep}.
\footnote{\url{https://www.youtube.com/watch?v=EJHrr21UvY8}}

\section{Understanding the zeroth law of thermodynamics}

Put hot iron into cold water.
Eventually both become equally warm.

\index{laws!thermodynamics, zeroth}%
\emph{Zeroth law of thermodynamics}:
Heat never spontaneously flows from cold to hot.

\section{Unstructured content}

TODO Pendulum

\index{definitions!pendulum}%
\index{pendulum!definition}%
A pendulum is a bob hung on a string.

\emph{Conservation of mechanical energy}:
A released pendulum comes back to the same height.

TODO
Interplay between potential and kinetic energy:
Galileo's interrupted pendulum

TODO Vacuum

Boyle showed that objects of different masses fall with the same acceleration.

TODO Toricelli manometer

TODO von Guericke, Magdeburg

TODO Boyle

TODO Pascal

Boyle's experiments

\index{laws named after people!Lavoisier's law of conservation of mass}%
TODO Lavoisier's law of conservation of mass

\section{Understanding energy}

Conservation of energy

Kinetic energy

\emph{Kinetic energy} is \( \frac{1}{2} m |v|^2 \) which can also be written as \( |p|^2 / (2m) \).
This is explained by energy conservation and work by a constant force \(F\) that accelerates an initially resting mass.
\(F = ma\) and \(s = \frac{1}{2}at^2\) and \( W = Fs \) and \( v = at \) therefore \( W = E_k = \frac{1}{2} m(at)^2 = \frac{1}{2}mv^2 \).

\section{Understanding gases}

% https://en.wikipedia.org/wiki/Perfect_gas
% https://en.wikipedia.org/wiki/Gas#Historical_synthesis

A \emph{gas} is ...

\emph{Pressure} is measured by a manometer.

In statics, the \emph{volume} of a gas is the volume of its container.
Statics assumes that a gas fills its container evenly.

\emph{Temperature} is measured by a thermometer.
The unit of temperature is \emph{kelvin} (K).

% ?
Gas and piston at equilibrium:
Gas and a piston with weight \(F\).

\section{Using gas laws}

Let there be a container of gas with pressure \(P_1\) and volume \(V_1\).
Let this gas expand or shrink without changing its temperature
so that its pressure becomes \(P_2\) and its volume becomes \(V_2\).

\index{laws!gas pressure and volume}%
\index{laws named after people!Boyle's law of gas pressure and volume}%
\index{Boyle!Boyle's law of gas pressure and volume}%
\emph{Boyle's law}: \( P_1 V_1 = P_2 V_2 \).

Other gas laws

\emph{Charles's law}?
\emph{Dalton's law}?

% https://en.wikipedia.org/wiki/Dalton%27s_law
% https://en.wikipedia.org/wiki/Combined_gas_law
% https://en.wikipedia.org/wiki/Gay-Lussac%27s_law#Pressure-temperature_law
% https://en.wikipedia.org/wiki/Avogadro%27s_law

\index{laws!ideal gas}
\emph{Ideal gas law}: \( PV = nRT \).

Kinetic energy of one mole of gas is \( \frac{3}{2} RT \).

Statistical thermodynamics: kinetic theory of gases?

\section{Understanding Boltzmann's constant}

% https://en.wikipedia.org/wiki/Boltzmann_constant
\emph{Boltzmann's constant} relates the average kinetic energy of particles in a gas and the temperature of the gas?

% https://en.wikipedia.org/wiki/Gas_constant
The \emph{gas constant} (molar gas constant, universal gas constant, ideal gas constant)?

\section{Understanding Avogadro's number}

\emph{Avogadro's number} is?

Terms?

System and environment

Thermodynamic equilibrium

\section{Understanding heat}

Heat capacity

\emph{Black's principle}:
When two liquids are mixed, the heat released by one equals the heat absorbed by the other.
???

???
If \(m_1\) amount of water at temperature \(T_1\) is mixed with \(m_2\) amount of water at temperature \(T_2\),
then the result, after equilibrium, is \(m_1+m_2\) amount of water at temperature \(\frac{m_1 T_1 + m_2 T_2}{m_1+m_2}\).

Specific heat

Latent heat

\section{Understanding thermodynamic process and cycle?}

Isobaric?
Isochoric?
Adiabatic?
Expansion of gas?
Work done by a gas?

Carnot engine?

Thermodynamic efficiency?

\section{Understanding the laws of thermodynamics}

% https://en.wikipedia.org/wiki/Laws_of_thermodynamics
% https://en.wikipedia.org/wiki/History_of_entropy

\section{Working with simple machines}

% https://en.wikipedia.org/wiki/Simple_machine

\UnorderedList{
\item Lever
\item Wheel and axle
\item{Pulley}
\item{Tilted plane}
\item{Wedge}
\item{Screw}
}

TODO:
Modern machine theory: Kinematic chains

\section{On ignorance}

In the 18th century, occasionally, steam boilers and coal mines exploded, killing tens of people.

Then nuclear power plants exploded.

What if a Dyson sphere exploded...

% Chemistry
% Thermostatics
% Heat and fluids