\section{The Jigsaw Puzzle Problem}
In the first part of this chapter we will discuss about the theory that has been used to set the problem. In the second part we will explain the implementation details that we adopted to implement and improve the performance of the paper.\newline
An immediate approach to solve the Jigsaw puzzle is to stack the tiles of the puzzles along the axis of the color (so \(9\times3=27\) channels). However, with this approach the net is caused to learn only low-level texture similarities instead of high-level primitives. We as humans use kind of analogies between the tiles - like similar patterns or contiguous borders between pieces - as cues to solve Jigsaw puzzles. However, this learning does not require any understanding of the global object. Thus, we need a network that delays the computation of statistics across different tiles and that can find the parts arrangement using these features. The objective is to force the network to understand representative features that can discriminate the relative location of each part of an object.

% ------------------------------------------------------------------------------------------------ %

\subsection{The Context-Free Architecture}\label{ss:CFN}
The solution found in \cite{Noroozi_2016} is to build a siamese convolutional network (see fig. \ref{fig:siamese_network}), where AlexNet (until layer \textit{fc6}) is used for each crop of the puzzle, with shared weights. The outputs of all \textit{fc6} layers are concatenated and given as input to the fully connected layer \textit{fc7}. In this way each patch is given to a different AlexNet separately, until the very last fully connected layer: the \textit{fc7} is responsible for the understanding of the rearrangement of the tiles - the context of the objet. Therefore this network architecture has been called \textit{context-free}: each patch remains separated from the others until the last layer that handles the context of the object. \newline
The performance of this architecture is similar to the AlexNet one in the classification task on the ImageNet 2012 dataset \cite{Imagenet_large_scale_hierarchical_image_database}. However, the CFN is more compact than AlexNet: the first depends on \(27.5M\) parameters, the latter on \(61M\). In addition, the network can be used interchangeably for different tasks including detection and classification.
\begin{figure}[!ht]
    \centering
    \includegraphics[scale=0.26]{images/Siamese_diagram.png}
    \caption{The Siamese network. The layers inside the AlexNet block have shared weights. The AlexNet block is composed by the repeated layers.}
    \label{fig:siamese_network}
\end{figure}
In the next section we show how to train the CFN for the Jigsaw Puzzle reassembly.

% ------------------------------------------------------------------------------------------------ %

\subsection{The Jigsaw Puzzle Task}
To train the CFN we define a set of Jigsaw puzzle permutation, \textit{e.g.}, a tile configuration \(S=(4, 2, 3, 5, 8, 7, 1, 0, 6)\). We randomly pick one of those permutations, rearrange the 9 input patches that has been retrieved from an image accordingly to that permutation, and ask the CFN to return a vector with the probability value for each index. Given \(9!=362,880\) possible permutations, it is clear that this is an important factor on the performance of the representation that the network learns. \newline
The permutation set controls the ambiguity of the task. If the permutations are too close to each other, the Jigsaw puzzle task is more challenging and ambiguous. For example, if the difference between two different permutations lies only in the position of two tiles and two of them are equal, the prediction of the right solution is impossible. To show this issue quantitatively, as specified in the ablation study of \cite{Noroozi_2016}[4.2], the learned representation on the PASCAL VOC 2007 detection task has been compared using several permutation sets based on the following two criteria:
\paragraph{Cardinality}
they trained the network with a different number of permutations to see what impact this had on the learned features. They found that as the total number of permutation increases, the training on the Jigsaw becomes more difficult. The performance of the detection task increases, though. As regard the experiments that we have made, we decided to use the first 40 permutation for the PC task and 100 for the GCP one, as the paper results were promising and the training has shown to be easier than the 1000 task. More information about those two tasks can be found at \ref{sss:select_crop_image} and \ref{s:experiments}.
\paragraph{Average Hamming Distance}
it has been seen \cite{Noroozi_2016}[p.11] that the average Hamming distance between permutations controls the difficulty of the Jigsaw puzzle reassembly task and is correlated with the object detection performance. As the average Hamming distance increases, the CFN gains higher accuracy and so does the fine-tuning tasks. For this reason we choose the higher possible Hamming distance. The algorithm that has been used is better described in Algorithm \ref{a:hm}: it begins with an empty permutation set and at each iteration it selects the one that has the maximum Hamming distance to the current permutation set.

\begin{algorithm}\label{a:hm}
    \caption{Generation of the \textit{maximal} Hamming distance permutation set}
    \SetKwInOut{Input}{Input}\SetKwInOut{Output}{Output}
    \SetKwData{P}{P}
    \SetKwFunction{repeat}{repeat}
    \SetKwFunction{until}{until}
    \SetKwFunction{hamming}{Hamming}
    \Input{N number of permutations}
    \Output{P maximal permutation set}
    \BlankLine
    \emph{// $\bar{P}$ is a $9 \times 9!$ matrix\;}
    $\bar{P} \leftarrow all \ permutations \ [\bar{P_1}, ..., \bar{P_9!}]$\;
    $P \leftarrow \emptyset$\;
    \emph{// uniform sample out of 9! permutations}\;
    $j \sim U[1, 9!]$\;
    $i \leftarrow 1 $\;
    \While{$i \leq{} N$}{
        \emph{// add permutation $\bar{P}_j $ to $P$}\;
        $P \leftarrow [P \ \bar{P_j}]$\;
        \emph{// remove $\bar{P}_j$ from $\bar{P}$}\;
        $\bar{P} \leftarrow [\bar{P}_1, ..., \bar{P}_{j-1}, \bar{P}_{j+1}, ...]$\;
        \emph{// D is a $i \times (9! - i)$ matrix)}\;
        $D \leftarrow \hamming{P, P'}$\;
        \emph{// $\bar{D}$ is a $1 \times (9! - i)$ row vector}\;
        $\bar{D} \leftarrow 1^T D$\;
        \emph{// $\bar{D}_k$ denotes the k-th entry of $\bar{D}$}\;
        $j \leftarrow \arg \max{}_k \bar{D}_k$\;
        $i \leftarrow i + 1$\;
    }
\end{algorithm}

% ------------------------------------------------------------------------------------------------ %

\subsection{Training the CFN}\label{s:training_CFN}
The output of the CFN can be seen as the conditional probability density function (pdf) of the spatial arrangement of object parts (or scene parts) in part-based model, \emph{i.e.},
\[p(S|A_1, ..., A_9)=p(S|F_1, ..., F_9)\prod_{i=1}^{9}p(F_i|A_i)\]
where $S$ is the configuration of the tiles, $A_i$ is the $i$-th part appearance of the object, and \{${F_i}$\}$_{i=1,...,9}$ form the intermediate feature representation. Our objective is to train the CFN so that the features $F_i$ have semantic attributes that can identify relative position between parts. As the pdf is very high-dimensional, close attention must be paid to the training strategy. In general, a self-supervised learning system might lead to representation that are suitable to solve the pre-text task, but not the target task, \emph{e.g.}, object classification, detection and segmentation. As regard this, an important factor to learn better representations is to prevent our model from taking undesiderable solutions, called solutions \textit{shortcuts}. In \cite{Noroozi_2016} they have used multiple techniques that apply to different problems:

\paragraph{Absolute position}
the CFN could learn to associate each appearance of $A_i$ to an absolute position. In this case, the feature $F_i$ carries no semantic meaning but just information about an arbitrary 2D position. To avoid this kind of learning we slowed down the training of the network in order to feed as much as possible Jigsaw puzzle per image; in addition, the maximum Hamming distance and the random selection of those patch reassembles assure us that different Jigsaw puzzle are chosen per each image with high probability. In this way the same tile should be assigned to multiple positions (ideally all 9) this making the mapping features $F_i$ to any absolute position equally likely.

\paragraph{Edge continuity}\label{p:edge_continuity}
to avoid shortcuts due to edge continuity and pixel intensity distribution we leave random gap between the tiles. In this way we discourage the network to learn context representation based on edges. During training the images are resized to a square of 256 pixels, preserving the original aspect ratio. Then they are cropped with a box of $225 \times 225$ randomly positioned and afterwards cropped into a $3 \times 3$ grid of $75 \times 75$ pixel tiles. From each of those tiles a $64 \times 64$ region is extracted randomly and then fed into the network. Thus, the average gap between each tile may range from a minimum of 0 pixels to a maximum of 22 pixels.

\paragraph{Chromatic aberration}\label{p:chromatic_aberration}
chromatic aberration is a relative spatial shift between color channels that increases from the image center to the borders. This type of distortion helps the network to estimate the tile position. To avoid this kind of shortcut three techniques were applied:
\begin{itemize}
    \item crop central square of original image randomly, getting a $225 \times 225$ pixels image;
    \item at training time about the 30\% of the dataset is converted to grayscale
    \item spatial jitter over the color channels is applied randomly differently for each tile by $\pm0$, $\pm1$, $\pm2$ pixels.
\end{itemize}
As shown in \cite{Noroozi_2016}[Tab.5], the application of all of these techniques makes the Jigsaw task more difficult - the accuracy is lower avoiding shortcuts - but the performance of the target task (detection in that case) is higher. So we decided to implement all of them.

% ------------------------------------------------------------------------------------------------ %

% \textcolor{red}{"FOD DATASET: describe the data you are working with for your project. What type of data is it? Where did it come from? How much data are you working with? Did you have to do any preprocessing, filtering, or other special treatment to use this data in your project?"\newline
% FOR METHOD: discuss your approach for solving the problems that you set up in the introduction. Why is your approach the right thing to do? Did you consider alternative approaches? It may be helpful to include figures, diagrams, or tables to describe your method or compare it with others.}

\subsection{The Dataset}
The dataset for the Jigsaw puzzle solver task is composed by squared images. Every images must be cropped into pieces, then fed into the Siamese network. No labels required. This is why we decided to use the Object Localization dataset \cite{ILSVRC15} of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC2017). This dataset contains ~1.281M images for the training set, 50K validation images and 100K for the test. There are about 1K categories with a range from 732 to 1300 images each. All images are in JPEG format.

\subsubsection{Select and crop images}\label{sss:select_crop_image}
once the images have been downloaded, we decided to build our own dataset on the basis of what we needed for the pre-task. So, we prepared two datasets, both divided into 70\% training, 25\% validation and 5\% test:
\begin{itemize}
    \item one made of a total of 500K random images, that served the model trained with a personal computer;
    \item one made of a total of 1.4M random images, that served the model trained with a virtual machine hosted in Google Cloud Platform.
\end{itemize}
More information about the computational power is provided in section \ref{ss:machines}. For every image of both datasets we decided to:
\begin{itemize}
    \item select only RGB images;
    \item take the shortest dimension to crop the image to a squared proportion. The crop position was random;
    \item resize the cropped image to ($256 \times 256$) pixels with the Lancsoz algorithm;
    \item save the cropped resized image as new ones for further access.
\end{itemize}
Moreover we used the Welford's online algorithm \cite{Welford_online_algorithm} to calculate the mean and standard deviation of the training set, that was used later to make any scaled feature value the number of standard deviation away from the mean. This information, together with train, validation and test set dimensions has been stored inside a specific \emph{h5} file.

\subsubsection{The Hamming Set}
to create the Hamming set the algorithm \ref{a:hm} has been implemented. In particular we used the maximum Euclidean distance between $P$ and $\bar{P}$. As regard the network on the personal computer, we decided to generate 40 permutations; for the GCP one 100 as this was the best compromise \cite{Noroozi_2016}[Tab.4] between the difficulty of the training of the pre-task and the results of the target task.

\subsubsection{The Data Pipeline}
for the implementation of the dataset we decided to use TensorFlow in Python. This was the best choice since we decided to use the same framework also for the design of the Siamese network (see \ref{ss:siamese}). We implemented the data stream trying to:
\begin{itemize}
    \item reduce the memory usage of the dataset both for RAM and GPU;
    \item speed up the data flow from the persistency to the GPU.
\end{itemize}
We decided to use the TensorFlow \emph{tf.data.Dataset} API \cite{tf_data_dataset}: this API provides many powerful tools to read images, manage the data stream and serve the dataset to the network.
In a first moment we decided to store the whole dataset inside a \emph{h5} format file, in which the images were store as \emph{numpy} arrays. In this way the dataset was a big blob file, easy to manage and access. With this kind of persistency, we created a generator that read the images one at a time and serve to the \emph{Dataset from generator} API of TensorFlow. This approach was successfull because there was a minimum memory usage since every image was loaded in a binary mode. While in the personal computer all the resources were used, once we moved to the Google Cloud Platform we realized that the online GPU was barely exploited. This was because the GPU was much more powerful, and it processed much more images than the generator could provide.
So we decided to change approach: first we saved the images produced in \ref{sss:select_crop_image} as new images on disk. Then we collected the all paths to the images, and let the \emph{tf.io} API to read them. In this way the shuffling was applied to very little list of string object, and the TensorFlow API was responsible for reading and parsing every image in a concurrent way.\newline
As regard the data transformations, we implemented every step in \ref{s:training_CFN}. We implemented every transformation as a \emph{lambda} python callable function, that we included in the \emph{tf.data.Dataset} pipeline with the specific \emph{dataset.map} method. This let TensorFlow to manage the reading and preprocessing of the images within the computational graph, in a concurrent (and distributed, if necessary) manner, mainly with the GPU cores. The steps are the following:
\begin{enumerate}
    \item we created the list of paths to train/val/test images and a random list of permutation indexes of the same length of the list of paths; in addition, a pointer to the \emph{h5} information file described above was provided. This file is useful to determine the dimension of the sets together with the mean and standard deviation of the training set at compile time, sparing computational power. The list of permutation indexes is generated every repetition of the dataset randomly, so the labels are always different for each epoch;
    \item we provided the two lists to TensorFlow \emph{tf.io} API, which is responsible for taking one element at a time. It also gives an "imperative" way-of-programming to implement the data pipeline;
    \item now the image is a tensor of float array of numbers. From every image is subtract the training mean and it is divided for the training variance as specified in \ref{sss:select_crop_image};
    \item with a probability of 30\% the image is then converted to grayscale within the training set;
    \item the image is cropped as described in the section \ref{p:edge_continuity}: every operation (multiplication, addition, slicing, etc.) is made with the TensorFlow methods and types to keep the computation efficient;
    \item every crop is jittered spatially as described in \ref{p:chromatic_aberration};
    \item the label is converted in the OneHot format;
    \item the dataset is now served.
\end{enumerate}

% ------------------------------------------------------------------------------------------------ %

\subsection{The Siamese Neural Network}\label{ss:siamese}
The architecture of the Siamese Neural Network, within the AlexNet one, has been well defined in \ref{ss:CFN} and in \cite{alexnet_paper}. Although the number of layers and the dimension of the kernels were strictly defined, we were quite free on the definition of the optimizers, the regularizes and the parameters. In addition, we also tried to insert some other layers - mainly batch normalization ones. In the next sections we will discuss about the framework that we used and the two planned networks

\subsubsection{TensorFlow and the \emph{eager execution}}
TensorFlow's eager execution \cite{tf_eager} is an imperative programming environment that evaluates operations immediately, without building graphs: operations return concrete values instead of constructing a computational graph to run later. This makes it easy to get started with TensorFlow and debug models, and it reduces boilerplate as well. Eager execution is a flexible machine learning platform for research and experimentation and provides:
\begin{itemize}
    \item \textit{an intuitive interface}: the programmer can structure the code naturally and use Python data structures;
    \item \textit{easier debugging}: the operations can be called directly and be inspected by standard Python debugging tools;
    \item \textit{natural control flow}: it can be used the Python control flow instead of the graph one, simplifying the specification of dynamic models.
\end{itemize}
We made the first implementation of the Jigsaw puzzle solver method with the computational graph. The debugging was painful; in addition, since the architecture is not a standard one, we encountered many troubles with the operations to be attached to the computational graph, such as the loss and accuracy ops. This is why we decided to move to this new framework and to implement the Jigsaw puzzle solver network from scratch.\newline
Since we needed a Siamese network with shared weights, we could not take benefits from the \emph{models} that \emph{tf.keras} offers. So we implemented the pre-task neural network as a subclass of the \emph{tf.keras.Model} API. This \emph{Model} class exposes built-in training, evaluation and prediction loops; in addition, it gives the list of its inner layers and the saving and serialization APIs. In this way we wrote all the layers that we needed in the initialization of the class (the AlexNet's layers and the following ones). Then we set the behavior of the \emph{call} function so:
\begin{enumerate}
    \item the \emph{AlexNet} block is called many times as the number of crops: the inputs are the different tiles, one per AlexNet;
    \item the outputs (the \textit{fc6} layers, one per each AlexNet of the block) are collected and concatenated;
    \item the concatenation is then provided to the dense and then the output layer is computed.
\end{enumerate}
This configuration let us save and restore the weights easily for the target task, since the architecture remains mostly the same, apart from the \emph{call} method.

\subsubsection{The PC architecture}\label{sss:PC_arch}
We calls the architecture that we trained in the personal computer the "PC" architecture, in contrast with the "GCP" one that we trained in Google Cloud Platform.\newline
The PC network is well represented in fig. \ref{fig:PC_net}.
\begin{figure}[!ht]
    \centering
    \includegraphics[scale=0.45]{images/PC_net.png}
    \caption{Architecture of the PC neural network.}
    \label{fig:PC_net}
\end{figure}
In this architecture we implemented all the features that were present in the original paper \cite{Noroozi_2016} with no additions. All the activations are \emph{relu}s, the dimensions of the kernels are already listed in fig. \ref{fig:siamese_network}. We used the Nesterov Stochastic Gradient Descent optimizer with momentum as precised in the paper. More information about the parameters and the training tactics are presented in \ref{sss:PC_training}.

\subsubsection{The GCP architecture}\label{sss:GCP_arch}
The GCP architecture is showed in fig. \ref{fig:GCP_net}. The main two changes are the batch normalization layers and the optimizer as we used Adam.
\begin{figure}[!ht]
    \centering
    \includegraphics[scale=0.45]{images/GCP_net.png}
    \caption{Architecture of the GCP neural network.}
    \label{fig:GCP_net}
\end{figure}
The training task we asked the network was the same as in the original paper, with slightly the same number of images (1.4M instead of 1.3M). So we decided to try to test different architecture to see if we were able to gain performances.\newline
We decided to give batch normalization a chance since it has been demonstrated \cite{batch_norm_paper} that the reduction of the covariate shift between internal layers can speed up the training. The pre-task training was already faster than some other previous works \cite{Noroozi_2016}[tab.1]; however, they decided not to use batch normalization layers. We were curious to see if this kind of normalization between convolutions with shared weights would have helped the training task. \newline
As regard the optimizer, Adam is an adaptive learning rate optimization algorithm that has been designed specifically for training deep neural networks \cite{adam_trend}. The paper contained some very promising diagrams, showing huge performance gains in term of speed of training. However, it has been studied that adaptive methods (such as Adam or Adadelta) do not generalize as well as SGD with momentum \cite{adam_generalization} when tested on a diverse set of deep learning tasks. Despite this we wanted to try and see the performance of this optimizer applied to our pre-task problem. More information about the parameters and the training tactics and results are presented in \ref{sss:GCP_training}.


\documentclass[main.tex]{subfiles}
\begin{document}

\section*{Thu Jan 09 2020}

In the last four hours we will finish the discussion on GWs.

We solved the vacuum equations \(R_{\mu \nu }=0\) for a perturbed Minkowski spacetime \(g_{\mu \nu } = \eta_{\mu \nu } + h_{\mu \nu } \) for small \(h_{\mu \nu }\).

We found that these are equivalent to
%
\begin{align}
R_{\mu \nu }  = - \frac{1}{2} \square h_{\mu \nu } + \partial_{(\mu } \qty(h^{\lambda }_{\nu) , \lambda } - \frac{1}{2} \partial_{\nu) }h )  =0
\,,
\end{align}
%
which can become the wave equation \(\square h_{\mu \nu } = 0\) in the harmonic gauge
%
\begin{align}
\partial_{\lambda } h^{\lambda }_{\nu } - \frac{1}{2} \partial_{\nu } h =0
\,.
\end{align}

Our gauge transformations are
%
\begin{align}
h_{\mu \nu } \rightarrow h_{\mu \nu } - \partial_{(\nu  } \epsilon_{\mu )}
\,,
\end{align}
%
where \(x^{\mu } \rightarrow x^{\mu } + \epsilon^{\mu }(x)\)

The solutions to the wave equation are
%
\begin{align}
h_{\mu \nu } = C_{\mu \nu } \exp(i k \cdot x)
\,,
\end{align}
%
where \(k^2=0\), since \(p^{\mu } = \hbar k^{\mu }\) and the wave travels at the speed of light \(p^2=0\).

\begin{claim}
If \(h_{\mu \nu }\) satisfies the harmonic gauge and \(\epsilon_{\mu }\) satisfies \(\square \epsilon_{\mu } = 0\) then \(\widetilde{h}_{\mu \nu } =  h_{\mu \nu } - \partial_{(\mu } \epsilon_{\nu )}\) also satisfies the harmonic gauge.
\end{claim}

\begin{proof}
The equation we want to show is true is
%
\begin{align}
0 = \partial_{\lambda } \widetilde{h}^{\lambda }_{\mu } - \frac{1}{2} \partial_{\mu } \widetilde{h}
\,,
\end{align}
%
where \(\widetilde{h} = \widetilde{h}^{\mu }_{\mu }\).
We can expand it into
%
\begin{subequations}
\begin{align}
0 &= \partial_{\lambda } \qty(\cancelto{}{h^{\lambda }_{\mu }} - \partial^{\lambda } \epsilon_{\mu } - \partial_{\mu } \epsilon^{\lambda }) - \frac{1}{2} \partial_{\mu } \qty(\cancelto{}{h} - 2 \partial^{\nu } \epsilon_{\nu } )  \marginnote{Harmonic gauge for \(h_{\mu \nu }\)}\\
&= - \square \epsilon_{\mu } - \partial_{\mu } \qty(\partial_{\nu } \epsilon^{\nu }) + \partial_{\mu } \qty(\partial^{\nu } \epsilon_{\nu }) = 0 \,. \marginnote{\(\square \epsilon_{\mu } = 0\) and commuting derivatives}
\end{align}
\end{subequations}
\end{proof}

Now, we want to impose 4 additional conditions that remove the residual freedom \(x^{\mu } \rightarrow x^{\mu } + \epsilon^{\mu }\) with \(\square \epsilon^{\mu } = 0\).

This is a wave equation for \(\epsilon^{\mu }\), so the solutions will look like
%
\begin{align}
\epsilon_{\mu } = \gamma_{\mu } \exp( i k \cdot
 x )
\,,
\end{align}
%
with \(\gamma_{\mu }\) constant and \(k^2=0\).

Under this transformation, the coefficient \(C_{\mu \nu }\) changes into
%
\begin{align}
\widetilde{C}_{\mu \nu } = C_{\mu \nu } - 2i k_{(\mu } \gamma_{\nu )}
\,,
\end{align}
%
as can be seen by substitutin in our expression for \(\gamma_{\mu } \) into the transformation law for \(h_{\mu \nu }\).
Do note that these correspond to the same physical fields.

We further require \(\widetilde{C}_{00 } = \widetilde{C}_{0i}= 0 \). This fixes the residual gauge freedom completely.

We need to show that from a generic \(C_{\mu \nu }\) we can find a \(\gamma_{\mu }\) such that this condition holds.

We write out the conditions:
%
\begin{align}
\widetilde{C}_{00} = C_{00} - 2 i k_0 \gamma_0
\,,
\end{align}
%
so setting \(\widetilde{C}_{00} =0 \) fixes
%
\begin{align}
\gamma_0 = \frac{C_{00} }{2ik_0 }
\,,
\end{align}
%
while  in order to set
%
\begin{subequations}
\begin{align}
0=\widetilde{C}_{0i} &= C_{0i} - ik_0 \gamma_{i} - i k_{i} \gamma_0  \\
&= C_{0i} - i k_0 \gamma_{i} - i k_{i} \frac{C_{00} }{2 i k_0 }  \\
\gamma_{i } &= \frac{1}{i k_0 } \qty(C_{0i} - \frac{k_{i} C_{00} }{2 k_0 })
\,.
\end{align}
\end{subequations}

With these choices for \(\gamma_{\mu }\) we can impose the desired condition.

There is no issue if \(k_0 = 0\), since that would correspond to a GW of zero frequency, which can be shown to be equivalent to flat spacetime.

In the end, we need to solve the system
%
\boxalign{
\begin{subequations}
\begin{align}
\square h_{\mu \nu } &=0  \label{eq:gw-wave-equation} \\
\partial_{\lambda } h^{\lambda }_{\mu } - \frac{1}{2} \partial_{\mu } h&=0  \label{eq:gw-harmonic-gauge}\\
h_{0\mu } &=0 \label{eq:gw-residual-gauge}
\,,
\end{align}
\end{subequations}}

%
since \(C_{\mu \nu }= 0\) for some \(\mu \nu \) implies \(h_{\mu \nu }=0\) for those indices.

\emph{A priori}, we have 10 degrees of freedom for our metric \(h_{\mu \nu }\) since it is symmetric. Equation \eqref{eq:gw-harmonic-gauge} eliminates 4 of them, and equation \eqref{eq:gw-residual-gauge} eliminates 4 more.

So, in the end we have 2 independent degrees of freedom.

Let us write the harmonic gauge condition explicitly: for \(\mu =0\) we get
%
\begin{subequations}
\begin{align}
\partial_{\lambda } h^{\lambda }_{0} - \frac{1}{2} \partial_{0} h &= 0  \\
- \partial_{0} h_{00} + \partial_{i} h_{i0} - \frac{1}{2} \partial_{0} h &= 0
\,,
\end{align}
\end{subequations}
%
but the first two vanish if \(h_{0\mu } =0\): so we get \(\partial_{0} h = 0\): this, when it is written explicitly, is
%
\begin{align}
\partial_{0} \qty(C^{\mu }_{\mu } \exp(i k \cdot x)) = 0
\,,
\end{align}
%
which means that \(C^{ \mu }_{\mu } = C = 0\): so the gravitational wave is traceless, this can be written as \(h_{ii}=0\) since \(h_{00} =0  \).

So, for \(\mu = j\) we can write the harmonic gauge condition as
%
\begin{subequations}
\begin{align}
\partial_{\lambda } h^{\lambda }_{j} &= 0 \marginnote{\(h = 0\)}  \\
\partial_{i} h_{ij} &=0 \marginnote{\(\eta_{ij} = \delta_{ij}\) and \(h_{0j} = 0\)}
\,.
\end{align}
\end{subequations}
%

This implies that the GW is \emph{transverse}: it oscillates only in the directions perpendicular to the motion.

Then, our system becomes
%
\boxalign{
\begin{subequations}
\begin{align}
\square h_{\mu \nu } &= 0  \\
h_{0\mu } &= 0  \\
\partial_{i} h_{ij} &= 0  \\
h_{ii} &= 0
\,.
\end{align}
\end{subequations}}

%

In terms of the matrix \(C_{\mu \nu }\) these conditions mean:
%
\begin{align}
C_{00} = C_{0i} = C_{ii} = k_{i} C_{ij} = 0
\,,
\end{align}
%
since the condition \(\partial_{i} h_{ij} =0\) means \(i k_{i} C_{ij} \exp(i k \cdot x ) = 0\).

Without  loss of generality we can write a coordinate system in which our gravitational wave is travelling along  the \(z\) direction: therefore \(\vec{k} = [0,0,k]^{\top}\).
Then, we get the conditions: \(C_{i 3}=0\), \(C_{11}+ C_{22} = 0\) and \(C_{0 \mu } = 0\).

So we have two conditions: \(C_{12} = C_{21} \neq 0\) and the rest of the matrix is zero, and \(C_{11} = - C_{22} \neq 0\) and the rest of the matrix is zero. Then, the basis tensors are
%
\begin{subequations}
\begin{align}
e_{ij,+} = \left[\begin{array}{ccc}
1 & 0 & 0 \\
0 & -1 & 0 \\
0 & 0 & 0
\end{array}\right]
\,,
\end{align}
\end{subequations}
%
and
%
\begin{subequations}
\begin{align}
e_{ij, \times } = \left[\begin{array}{ccc}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0
\end{array}\right]
\,.
\end{align}
\end{subequations}
%

The Frobenius norm of these matrices is \(e_{ij,+} e_{ij, +} = e_{ij, \times} e_{ij, \times } = 2\), which can also be written as \(\tr (e_{+} e_{+}) = \tr (e_{ \times  } e_{ \times }) = 2\). Also, in this sense they are orthogonal: \(\tr (e_{+} e_{ \times }) = 0\).

So, we can write \(\tr (e_{r} e_{s}) = 2 \delta_{rs}\), where  \(r\) and \(s\) are either \(+\) or \( \times \).

Then, any GW \(h_{ij}\) can be written as
%
\begin{align}
h_{ij} (x) = \qty(h_{+} e_{ij, +} + h_{ \times } e_{ij, \times }) \exp(i k \cdot x)
\,,
\end{align}
%
with \(k^2 = 0\). Do note that \(h_{+} \) and \(h_{ \times }\) are two independent complex numbers.

The solutions we want have real \(h_{\mu \nu }\), so if \(h_{\mu \nu } \in \mathbb{C}\) is a solution then \(h_{\mu \nu }^{*}\) also is a solution. So, we are allowed to take the real part with \((h_{\mu \nu } + h_{\mu \nu }^{*}) / 2\). This will then give us
%
\begin{align}
\Re (C_{ij} \exp(i k \cdot
 x ) ) = C_{ij}  \cos (i k \cdot x  + \varphi )
\,,
\end{align}
%
for some real \(C_{ij}\). So, in general we will have
%
\begin{align}
h_{ij}^{k} (x) = \sum_{r = +, \times } h_{r} e_{ij, r} \cos(kt - kz + \varphi_{r} )
\,,
\end{align}
%
do note that we changed the sign of the argument of the cosine, which we can do since the cosine is even.

Do also note that the phase can in principle depend on the polarization.

We can then recover any general solution travelling along the \(z\) direction by having \(h_r\) depend on \(k\): then  we will have
%
\begin{align}
h_{ij} = \int  \dd{k} \sum_{r = +, \times } h_{r} (k) e_{ij, r} \cos(kt - kz + \varphi_{r}(k) )
\,.
\end{align}

Also, note that the frequency of the gravitational waves can be found by \(f = k / 2\pi \); this can be seen from the periodicity of the waves.

What do these polarizations physically entail?

Recall that the distance along a certain direction, say \(x\), is given by the integral \(\int \dd{x} \sqrt{g_{xx}}\).

So, if \(g_{xx}\) changes than the physical distance also changes.

Let us consider the point \(z=0\), at \(t=0\), with zero phase \(\varphi \) for the polarization \(e_{ij, +}\): then the component \(g_{xx}\) has increased, since the cosine is equal to one, while the component \(g_{yy}\) has decreased

After half a wavelength, when the cosine is equal to \(-1\), this effect reverses. Then, \(g_{xx}\) decreases while \(g_{yy} \) increases.

The wave is travelling along the \(z\) axis, but because of the transversality condition it stretches the \(x\) and \(y\) directions.

What happens if we perform a \SI{45}{\degree} rotation? the spatial rotation matrix is
%
\begin{subequations}
\begin{align}
R = \frac{1}{\sqrt{2}} \left[\begin{array}{ccc}
1 & 1 & 0 \\
-1 & 1 & 0 \\
0 & 0 & \sqrt{2}
\end{array}\right]
\,,
\end{align}
\end{subequations}
%
so, for example, \([1,1,0]^{\top} \rightarrow [\sqrt{2},0,0]^{\top}\).

The transformation law with this matrix is \(\widetilde{x}^{i} = R_{j}^{i} x^{j}\).

It is known that a rotation matrix satisfies \(R^{\top} R = \mathbb{1}\).

The line element changes as
%
\begin{align}
\dd{\ell^2} = \dd{x^{i}} g_{ij} \dd{x^{j}} = \dd{\widetilde{x}^{i} } \widetilde{g}_{ij} \dd{\widetilde{x}^{j}}
\,,
\end{align}
%
which means that \(g = R^{\top} \widetilde{g} R\), since \(\dd{\widetilde{x}} = R \dd{x}\). This is coherent with the tensor transformation law. Inverting it, we find \(\widetilde{g} = R g R^{\top}\).

How does the perturbed metric change?
%
\begin{subequations}
\begin{align}
\mathbb{1} + \widetilde{h} &= R (\mathbb{1} + h) R^{\top}  \\
&= R \mathbb{1} R^{\top} + R h R^{\top}  \\
&= \mathbb{1} + R h R^{\top}
\,,
\end{align}
\end{subequations}
%
so \(\widetilde{h} = R h R^{\top}\).

Let us suppose that \(h = e_{ \times }\): what is then \(\widetilde{h}\)? It can be shown straightforwardly that
%
\begin{align}
\widetilde{h} = R e_{ \times } R^{\top} = e_{+}
\,,
\end{align}
%
so the cross-polarization is equivalent to the plus-polarization, rotated by \SI{45}{\degree}.
This is somewhat of an informal line of reasoning, since the basis polarization ``tensors'' are not actually tensors under generic transformation. However, the result is correct and the reasoning is illustrative so we stick with it.

\end{document}


\chapter{Background}\label{ch:background}

\begin{remark}{Outline}
In this chapter, we introduce supervised learning and fundamental concepts on
which this work builds upon. In Section~\ref{sec:2:learning-from-data}, we
first describe classification and regression learning tasks and then formally
define the notion of model. In Section~\ref{sec:2:performance-evaluation}, we
proceed with a discussion on performance evaluation and then describe, in
Section~\ref{sec:2:model-selection}, procedures for selecting the best possible
model. Finally, we conclude in Section~\ref{sec:2:classes-of-algorithms} with a
brief overview of some of the main methods in supervised learning.
\end{remark}

\section{Learning from data}
\label{sec:2:learning-from-data}

In the examples introduced in Chapter~\ref{ch:introduction}, the objective
which is sought is to find a systematic way of predicting a phenomenon given a
set of measurements. In machine learning terms, this goal is formulated as the
{\it supervised learning} task of inferring from collected data a model that
predicts the value of an output variable based on the observed values of input
variables. As such, finding an appropriate model is based on the assumption
that the output variable does not take its value at random and that there
exists a relation between the inputs and the output. In medicine for instance,
the goal is to find a decision rule (i.e., a model) from a set of past cases
(i.e., the collected data) for predicting the condition of an incoming patient
(i.e., the output value) given a set of measurements such as age, sex, blood
pressure or history (i.e., the input values).

To give a more precise formulation, let us assume as set of \textit{cases} or
\textit{objects} taken from a universe $\Omega$\label{ntn:omega}. Let us arrange the set of
measurements on a case in a pre-assigned order, i.e., take the input values to
be $x_1, x_2, ..., x_p$, where $x_j \in {\cal X}_j$\label{ntn:value-x_j}\label{ntn:space-X_j} (for $j = 1, ..., p$\label{ntn:p})
corresponds to the value of the input variable $X_j$\label{ntn:var-X_j}. Together, the input
values $(x_1, x_2, ..., x_p)$ form a $p$-dimensional input vector $\mathbf{x}$\label{ntn:sample-x}
taking its values in ${\cal X}_1 \times ... \times {\cal X}_p = {\cal X}$\label{ntn:space-X},
where ${\cal X}$ is defined as the input space. Similarly, let us define as $y
\in {\cal Y}$\label{ntn:value-y} the value of the output variable $Y$\label{ntn:var-Y}, where ${\cal Y}$\label{ntn:space-Y} is defined
as the output space\footnote{Unless stated otherwise, supervised learning is
reduced to the prediction of a \textit{single} output variable. More generally
however, this framework can be defined as the prediction of one or several
output variables.}. By definition, both the input and the output spaces are
assumed to respectively contain all possible input vectors and all possible
output values. Note that input variables are sometimes known as {\it features},
input vectors as {\it instances} or {\it samples} and the output variable as
{\it target}.

Among variables that define the problem, we distinguish between two general
types. The former correspond to quantitative variables whose values are integer
or real numbers, such as age or blood pressure. The latter correspond to
qualitative variables whose values are symbolic, such as gender or condition.
Formally, we define them as follows:

\begin{definition}
A variable $X_j$\label{ntn:var-X_j2} is \emph{ordered} if ${\cal X}_j$ is a
totally ordered set. In particular, $X_j$ is said to be \emph{numerical} if
${\cal X}_j = \mathbb{R}$.
\end{definition}

\begin{definition}
A variable $X_j$ is \emph{categorical} if ${\cal X}_j$ is a finite set of values,
without any natural order.
\end{definition}

In a typical supervised learning task, past observations are summarized by a
dataset called {\it learning set}. It consists in a set of observed input
vectors together with their actual output value and formally defined as
follows:

\begin{definition}
A \emph{learning set} ${\cal L}$\label{ntn:learning-set} is a set of $N$\label{ntn:N}
pairs of input vectors and output values $(\mathbf{x}_1, y_1), ...,
(\mathbf{x}_N, y_N)$, where $\mathbf{x}_i \in {\cal X}$ and $y_i \in {\cal Y}$.
\end{definition}

Equivalently, a set of $p$-input vectors $\mathbf{x}_i$\label{ntn:sample-x_i}
(for $i=1, ..., N$) can be denoted by a $N\times p$ matrix
$\mathbf{X}$\label{ntn:matrix-X}, whose rows $i=1, ..., N$ correspond to input
vectors $\mathbf{x}_i$ and columns $j=1, ..., p$ to input variables $X_j$.
Similarly, the corresponding output values can be written as a vector
$\mathbf{y}=(y_1, ..., y_N)$\label{ntn:vector-y}.

\begin{remark}{Data representation}
For optimal implementations of machine learning algorithms, data needs to be
represented using structures which allow for high-performance numerical
computation. In this work, code snippets are described under  the assumption
that data is represented using a data structure similar to a NumPy
array~\citep{vanderwalt:2011}.

A NumPy array is basically a multidimensional uniform collection of values, all
of the same type and organized in a given shape. For instance, a matrix
$\mathbf{X}$ can be represented as a 2-dimensional NumPy array of shape $N
\times p$ that contains numbers (e.g., floating point values or integers). This
structure allows for random access in constant time, vectorized high-level
operations and efficient memory usage.

Additionally, using a data representation which is close to the matrix
formulation, like NumPy arrays, makes it possible to write implementations that
are close to their original textbook formulation, thereby making them easier to code,
understand and maintain. \end{remark}

In this framework, the supervised learning task can be stated as learning a
function $\varphi: {\cal X} \mapsto {\cal Y}$\label{ntn:varphi} from a learning
set ${\cal L}=(\mathbf{X}, \mathbf{y})$. The objective is to find a model such
that its predictions $\varphi(\mathbf{x})$\label{ntn:varphi-x}, also denoted by
the variable $\hat{Y}$, are as good as possible. If $Y$ is a categorical
variable then the learning task is a classification problem. If $Y$ is
numerical variable, then learning task is a regression problem. Without loss of
generality, the resulting models can be defined as follows:

\begin{definition}
A \emph{classifier} or \emph{classification rule} is a function $\varphi: {\cal X}
\mapsto {\cal Y}$, where ${\cal Y}$ is a finite set of classes (or labels) denoted $\{c_1, c_2, ..., c_J\}$\label{ntn:J}\label{ntn:c_k}.
\end{definition}

\begin{definition}
A \emph{regressor} is a function $\varphi: {\cal X} \mapsto {\cal Y}$, where ${\cal Y}=\mathbb{R}$.
\end{definition}

\begin{remark}{Estimator interface}
We follow in this work the API conventions proposed by~\citet{buitinck:2013}.
Learning algorithms are described as \textit{estimator} objects implementing the
following interface:

\begin{itemize}
\item Hyper-parameters of an algorithm are all passed to the constructor of the
estimator. The constructor does not see any actual data. All it does
is to attach hyper-parameters values as public attributes to the estimator object.
\item Learning is performed in a \texttt{fit} method. This method is called with
a learning set (e.g., supplied as two arrays \texttt{X\_train} and \texttt{y\_train}). Its
task is to run a learning algorithm and to determine model-specific parameters
from the data. The \texttt{fit} method always returns the estimator object
it was called on, which now serves as a model and can be used to make predictions.
\item Predictions are performed through a \texttt{predict} method, taking
as input an array \texttt{X\_test} and producing as output the predictions for
\texttt{X\_test} based on the learned parameters. In the case of classification,
this method returns labels from ${\cal Y} = \{c_1, c_2, ..., c_J\}$. In the case of regression,
it returns numerical values from ${\cal Y} = \mathbb{R}$.
\end{itemize}

With this API, a typical supervised learning task is performed as follows:

\vskip0.3cm
\begin{pythoncode}
# Instantiate and set hyper-parameters
clf = DecisionTreeClassifier(max_depth=5)
# Learn a model from data
clf.fit(X_train, y_train)
# Make predictions on new data
y_pred = clf.predict(X_test)
\end{pythoncode}
\end{remark}

\section{Performance evaluation}
\label{sec:2:performance-evaluation}

In the statistical sense, input and output variables $X_1, ..., X_p$ and $Y$
are \textit{random variables} taking jointly their values from ${\cal X} \times
{\cal Y}$ with respect to the joint probability distribution $P(X, Y)$\label{ntn:P_XY}, where
$X$\label{ntn:vector-X} denotes the random vector $(X_1, ..., X_p)$. That is, $P(X=\mathbf{x},
Y=y)$ is the probability that random variables $X$ and $Y$ take values
$\mathbf{x}$ and $y$ from ${\cal X}$ and ${\cal Y}$ when drawing an object
uniformly at random from the universe $\Omega$.

Accordingly, using an algorithm ${\cal A}$\label{ntn:A} for learning a model\footnote{Unless
it is clear from the context, models are now denoted $\varphi_{\cal L}$\label{ntn:varphi-L} to
emphasize that they are built from the learning set ${\cal L}$.} $\varphi_{\cal
L}$ whose predictions are as good as possible can be stated as finding a model
which minimizes its expected prediction error, defined as follows:

\begin{definition}
The \emph{expected prediction error}, also known as \emph{generalization
error} or \emph{test error}, of the model $\varphi_{\cal L}$ is\footnote{$\mathbb{E}_X \{f(X)\}$ denotes the expected value of $f(x)$
(for $x \in {\cal X}$) with respect to the probability distribution of the
random variable $X$. It is defined as $$\mathbb{E}_X \{f(X)\} = \sum_{x \in {\cal
X}} P(X=x) f(x).$$}
\begin{equation}\label{eqn:generalization-error}
Err(\varphi_{\cal L}) = \mathbb{E}_{X, Y}\{ L(Y, \varphi_{\cal L}(X)) \},
\end{equation}
where ${\cal L}$ is the learning set used to build $\varphi_{\cal L}$ and $L$\label{ntn:L} is a loss
function measuring the discrepancy between its two
arguments~\citep{geurts:2002}.
\end{definition}

Equation~\ref{eqn:generalization-error} basically measures the prediction error
of $\varphi_{\cal L}$ over all possible objects in $\Omega$ (each represented by a couple
$(\mathbf{x}, y)\in {\cal X}\times{\cal Y}$), including the observed couples
from the learning set ${\cal L}$ but also all the \textit{unseen} ones from
${\cal X}\times{\cal Y} \setminus {\cal L}$. Indeed, the goal is not in fact to
make the very-most accurate predictions over the subset ${\cal L}$ of known
data, but rather to learn a model which is correct and reliable on all possible
data.

For classification, the most common loss function is the \textit{zero-one} loss
function\footnote{$1(\text{\textit{condition}})$ denotes the unit function. It
is defined as
$$1(\text{\textit{condition}}) =
\begin{cases}
1 & \text{if \textit{condition} is true}\\
0 & \text{if \textit{condition} is false}
\end{cases}.
$$} $L(Y, \varphi_{\cal L}(X)) = 1(Y \neq \varphi_{\cal L}(X))$, where all
misclassifications are equally penalized. In this case, the generalization
error of $\varphi_{\cal L}$ becomes the probability of misclassification of the model:
\begin{equation}
Err(\varphi_{\cal L}) = \mathbb{E}_{X, Y}\{ 1(Y \neq \varphi_{\cal L}(X)) \} = P(Y \neq \varphi_{\cal L}(X))
\end{equation}

Similarly, for regression, the most used loss function is the \textit{squared
error} loss $L(Y, \varphi_{\cal L}(X)) = (Y - \varphi_{\cal L}(X))^2$, where large differences
between the true values and the predicted values are penalized more heavily
than small ones. With this loss, the generalization error of the model becomes:
\begin{equation}
Err(\varphi_{\cal L}) = \mathbb{E}_{X, Y}\{ (Y - \varphi_{\cal L}(X))^2  \}
\end{equation}

\subsection{Estimating $Err(\varphi_{\cal L})$}
\label{sec:2:estimating-generalization-error}

In practice, the probability distribution $P(X, Y)$ is usually unknown, making
the direct evaluation of $Err(\varphi_{\cal L})$ infeasible. Equivalently, it
is often not possible to draw additional data, thereby making infeasible the
empirical estimation of $Err(\varphi_{\cal L})$ on a (virtually infinite) set
${\cal L}'$ drawn independently from ${\cal L}$. In most problems, ${\cal L}$
constitutes the only data available, on which both the model needs to be
learned and its generalization error estimated. As reviewed and described on
multiple occasions by several
authors~\citep{toussaint:1974,stone:1978,breiman:1984,kohavi:1995,nadeau:2003,hastie:2005,arlot:2010},
the generalization error in Equation~\ref{eqn:generalization-error} can however
be estimated in several ways.

To make notations clearer, let us first define $\overline{E}(\varphi_{\cal L}, {\cal
L}^\prime)$\label{ntn:E_bar} as the average prediction error of the model $\varphi_{\cal L}$ over the set
${\cal L}^\prime$ (possibly different from the learning set ${\cal L}$ used to produce
$\varphi_{\cal L}$), that is:
\begin{equation}
\overline{E}(\varphi_{\cal L}, {\cal L}^\prime) = \frac{1}{N^\prime} \sum_{(\mathbf{x_i}, y_i) \in {\cal L}^\prime} L(y_i, \varphi_{\cal L}(\mathbf{x_i}))
\end{equation}
where $N^\prime$ is the size of the set ${\cal L}^\prime$.

The first and simplest estimate of the generalization error is the
\textit{resubstitution estimate} or \textit{training sample estimate}. It
consists in empirically estimating $Err(\varphi_{\cal L})$ on the same data as
the learning set ${\cal L}$ used to build $\varphi_{\cal L}$, that is:
\begin{equation}\label{eqn:training-error}
\widehat{Err}^\text{train}(\varphi_{\cal L}) = \overline{E}(\varphi_{\cal L}, {\cal L})
\end{equation}
In general, the resubstitution error is a poor estimate of $Err(\varphi_{\cal L})$.
In particular, since most machine learning algorithms aim at
precisely minimizing Equation~\ref{eqn:training-error} (either directly or
indirectly), it typically results in an overly optimistic estimate of the
generalization error, which accounts for all couples $(\mathbf{x}, y)$, i.e.,
not only those from ${\cal L}$.

The second approach is the \textit{test sample estimate}. It consists in
dividing the learning set ${\cal L}$ into two disjoint sets ${\cal
L}_\text{train}$ and ${\cal L}_\text{test}$, called \textit{training set} and
\textit{test set}, and then to use each part respectively for learning a model
and estimating its generalization error. The test sample estimate of the
generalization error of the model $\varphi_{\cal L}$ that would be obtained from ${\cal
L}$ is then given as the average prediction error over ${\cal L}_\text{test}$
of the model $\varphi_{{\cal L}_\text{train}}$ built on ${\cal L}_\text{train}$:
\begin{equation}\label{eqn:test-error}
\widehat{Err}^\text{test}(\varphi_{\cal L}) = \overline{E}(\varphi_{{\cal L}_\text{train}}, {\cal L}_\text{test})
\end{equation}
As a rule-of-thumb,
${\cal L}_\text{train}$ is usually taken as $70\%$ of the samples in
${\cal L}$ and ${\cal L}_\text{test}$ as the remaining $30\%$, though
theoretical work~\citep{guyon:1997} suggests to progressively reduce the size of test set as
the size of ${\cal L}$ increases. In any case, care must be taken when splitting ${\cal
L}$ into two subsets, so that samples from ${\cal L}_\text{train}$ can be
considered independent from those in ${\cal L}_\text{test}$ and drawn from the
same distribution. This is however usually guaranteed by drawing ${\cal
L}_\text{train}$ and ${\cal L}_\text{test}$ simply at random from ${\cal L}$.
While being an unbiased estimate of $Err(\varphi_{\cal L})$, the test sample
estimate has the drawback that it reduces the
effective sample size on which the model $\varphi_{{\cal L}_\text{train}}$ is learned. If ${\cal L}$ is large,
then this is usually not an issue, but if ${\cal L}$ only contains a few dozens
of samples, then this strategy might not correctly approximate the true
generalization error of the model that would have been learned on the entire learning set.

When ${\cal L}$ is small, the \textit{$K$-fold cross-validation estimate}\label{ntn:K-cv} is
usually preferred over the test sample estimate. It consists in randomly
dividing the learning set ${\cal L}$ into $K$ disjoint subsets, ${\cal
L}_1, ..., {\cal L}_K$\label{ntn:L_k}, and then to estimate the generalization error as the average prediction error over the folds ${\cal
L}_k$ of the models $\varphi_{{\cal L}\setminus {\cal L}_k}$  learned on the
remaining data:
\begin{equation}\label{eqn:cv-error}
\widehat{Err}^{CV}(\varphi_{\cal L}) = \frac{1}{K} \sum_{k=1}^K  \overline{E}(\varphi_{{\cal L}\setminus{\cal L}_k}, {\cal L}_k)
\end{equation}
The assumption behind this approach is that since each model $\varphi_{{\cal L}\setminus{\cal L}_k}$ is
built using almost all ${\cal L}$, they should all be close to the model
$\varphi_{\cal L}$ learned on the entire set. As a result the unbiased
estimates $\overline{E}(\varphi_{{\cal L}\setminus{\cal L}_k}, {\cal L}_k)$ should also all be close to
$Err(\varphi_{\cal L})$. While more computationally intensive, the $K$-fold
cross-validation estimate has the advantage that every couple $(\mathbf{x}, y)
\in {\cal L}$ is used for estimating the generalization error of $\varphi_{\cal L}$. In
a typical setting, $K$ is usually fixed to $10$, a value often yielding stable
and reliable estimates~\citep{kohavi:1995}.

\begin{remark}{Expected generalization error}
As we have shown, our goal is to estimate the generalization error
$Err(\varphi_{\cal L})$ conditional on the learning set ${\cal L}$. A related
quantity is the \textit{expected} generalization error
\begin{equation}\label{eqn:expected-generalization-error}
\mathbb{E}_{\cal L} \{ Err(\varphi_{\cal L}) \},
\end{equation}
averaging over everything which is random, including the randomness in the
learning set ${\cal L}$ used to produce $\varphi_{\cal L}$. As discussed by
\citet{hastie:2005}, this quantity is close, yet different from
$Err(\varphi_{\cal L})$. The authors point out that most estimates, including $K$-fold
cross-validation, effectively estimate
Equation~\ref{eqn:expected-generalization-error} rather than
Equation~\ref{eqn:generalization-error}.
\end{remark}

\subsection{Bayes model and residual error}
\label{sec:2:bayes-model}

In theory, when the probability distribution $P(X, Y)$ is known, the best possible model,
i.e., the model $\varphi_B$ which minimizes the generalization error of
Equation~\ref{eqn:generalization-error}, can be derived analytically and
independently of any learning set ${\cal L}$.
By conditioning on $X$, the generalization error of this model can be rewritten as:
\begin{equation}\label{eqn:generalization-error-conditionned}
\mathbb{E}_{X, Y}\{ L(Y, \varphi_B(X)) \} = \mathbb{E}_{X}\{ \mathbb{E}_{Y|X} \{ L(Y, \varphi_B(X)) \} \}
\end{equation}
In this latter form, the model which minimizes
Equation~\ref{eqn:generalization-error-conditionned} is a model which
minimizes the inner expectation point-wise, that is:
\begin{equation}
\varphi_B(\mathbf{x}) = \argmin_{y \in {\cal Y}} \mathbb{E}_{Y|X=\mathbf{x}} \{ L(Y, y) \}
\end{equation}
In the literature, $\varphi_B$ is known as the \textit{Bayes model} and its
generalization error $Err(\varphi_B)$ as the \textit{residual error}. It represents the minimal
error that any supervised learning algorithm can possibly attain, that is
the irreducible error purely due to random deviations in the data.

\begin{definition}
A model $\varphi_B$\label{ntn:varphi-B} is a \emph{Bayes model} if, for any model $\varphi$ built from any
learning set ${\cal L}$, $Err(\varphi_B) \leq Err(\varphi_{\cal L})$.
\end{definition}

In classification, when $L$ is the zero-one loss, the Bayes model is:
\begin{align}
\varphi_B(\mathbf{x}) &= \argmin_{y \in {\cal Y}} \mathbb{E}_{Y|X=\mathbf{x}} \{ 1(Y, y) \} \nonumber \\
                      &= \argmin_{y \in {\cal Y}} P(Y \neq y|X=\mathbf{x}) \nonumber \\
                      &= \argmax_{y \in {\cal Y}} P(Y = y|X=\mathbf{x}) \label{eqn:bayes-model-classification}
\end{align}
Put otherwise, the best possible classifier
consists in systematically predicting the most likely class $y \in \{c_1, c_2, ..., c_J\}$
given $X=\mathbf{x}$.

Similarly, for regression with the squared error loss, we have:
\begin{align}
\varphi_B(\mathbf{x}) &= \argmin_{y \in {\cal Y}} \mathbb{E}_{Y|X=\mathbf{x}} \{ (Y - y)^2 \} \nonumber \\
                      &= \mathbb{E}_{Y|X=\mathbf{x}} \{ Y \} \label{eqn:bayes-model-regression}
\end{align}
In other words, the best possible regressor consists in systematically predicting
the average value of $Y$ at $X=\mathbf{x}$.

For practical problems, $P(X, Y)$ is unknown and the Bayes model cannot be
derived analytically. In this context, the effectiveness of a model $\varphi_{\cal L}$
may be difficult to evaluate since (estimates of) $Err(\varphi_{\cal L})$ may
not be very indicative of the goodness of $\varphi_{\cal L}$ if the lowest attainable
error $Err(\varphi_B)$ is unknown. On simulated data however, where the
distribution is known, deriving $\varphi_B$ and $Err(\varphi_B)$ is feasible.
This is beneficial as $Err(\varphi_B)$ can now be used for comparing the test
set error of $\varphi_{\cal L}$ and thereby evaluate the actual effectiveness of
$\varphi_{\cal L}$ with respect to the best model one can possibly build.

From a theoretical point of view, the concepts of Bayes model and residual
error are also useful to study the learning capabilities of an algorithm. In
particular, as ${\cal L}$ gets arbitrarily large, a fundamental question is to
know whether it is possible to produce a model $\varphi_{\cal L}$ which is
\textit{consistent}, that is such that its generalization error gets
arbitrarily close to the lowest possible generalization error $Err(\varphi_B)$.
Formally, consistency is defined as follows~\citep{devroye:1996}:

\begin{definition}\label{def:consistency}
A learning algorithm ${\cal A}$ is said to be \emph{weakly consistent} for a certain distribution
$P(X, Y)$ if $\mathbb{E}_{\cal L} \{ Err(\varphi_{\cal L}) \} \to Err(\varphi_B)$
as the size $N$ of the learning set ${\cal L}$ used to build $\varphi_{\cal L}$ using ${\cal A}$ tends
to infinity.
\end{definition}

\begin{definition}\label{def:consistency-strong}
A learning algorithm ${\cal A}$ is said to be \emph{strongly consistent} for a certain distribution
$P(X, Y)$ if $Err(\varphi_{\cal L}) \to Err(\varphi_B)$ almost surely
as the size $N$ of the learning set ${\cal L}$ used to build $\varphi_{\cal L}$ using ${\cal A}$ tends
to infinity.
\end{definition}

Note that the definition of consistency depends on the distribution $P(X, Y)$.
In general, a learning algorithm ${\cal A}$ can be proven to be consistent for some
classes of distributions, but not for others. If consistency can be proven
for any distribution $P(X, Y)$, then ${\cal A}$ is said to be \textit{universally}
(strongly) consistent.

\section{Model selection}
\label{sec:2:model-selection}

From the previous discussion in Section~\ref{sec:2:bayes-model}, it appears that
to solve the supervised learning problem it would be sufficient to estimate
$P(Y|X)$ from the learning sample ${\cal L}$ and then to define a model
accordingly using either Equation~\ref{eqn:bayes-model-classification} or
Equation~\ref{eqn:bayes-model-regression}. Unfortunately, this approach is
infeasible in practice because it requires ${\cal L}$ to grow exponentially
with the number $p$ of input variables in order to compute accurate estimates of
$P(Y|X)$~\citep{geurts:2002}.

\subsection{Selecting the (approximately) best model}

To make supervised learning work in high-dimensional input spaces with learning
sets of moderate sizes, simplifying assumptions must be made on
the structure of the best model $\varphi_B$. More specifically, a supervised
learning algorithm assumes that $\varphi_B$ -- or at least a good enough approximation
-- lives in a family ${\cal H}$\label{ntn:H} of candidate models, also known as
\textit{hypotheses} in statistical learning theory, of restricted structure. In
this setting, the \textit{model selection} problem is then defined as finding
the best model among ${\cal H}$ on the basis of the learning set ${\cal L}$.

\begin{remark}{Approximation error}
Depending on restrictions made on the structure of the problem, the
Bayes model usually does not belong to ${\cal H}$, but there may be models
$\varphi \in {\cal H}$ that are sufficiently close to it. As such, the
\textit{approximation error}~\citep{bottou:2011} measures how closely the
models in ${\cal H}$ can approximate the optimal model $\varphi_B$:
\begin{equation}
Err({\cal H}) = \min_{\varphi \in {\cal H}} \{ Err(\varphi) \} - Err(\varphi_B)
\end{equation}
\end{remark}

To be more specific, let $\theta$\label{ntn:theta} be the vector of hyper-parameters values
controlling the execution of a learning algorithm ${\cal A}$. The application
of ${\cal A}$ with hyper-parameters $\theta$ on the learning set ${\cal L}$ is a
deterministic\footnote{If ${\cal A}$ makes use of a pseudo-random generator to
mimic a stochastic process, then we assume that the corresponding random seed is
part of $\theta$.} process yielding a model ${\cal A}(\theta, {\cal L}) =
\varphi_{\cal L} \in {\cal H}$\label{ntn:A-func}. As such, our goal is to find the vector of hyper-parameters values
yielding to the best model possibly learnable in ${\cal H}$ from ${\cal L}$:
\begin{equation}
\theta^* = \argmin_{\theta} Err({\cal A}(\theta, {\cal L}))
\end{equation}\label{ntn:theta-star}
Again, this problem cannot (usually) be solved exactly in practice since
it requires the true generalization error of a model to be computable. However
approximations $\widehat{\theta}^*$\label{ntn:theta-star-approx} can be obtained in several ways.

When ${\cal L}$ is large, the easiest way to find $\widehat{\theta}^*$ is to
use test sample estimates (as defined by Equation~\ref{eqn:test-error}) to
guide the search of the hyper-parameter values, that is:
\begin{align}
\widehat{\theta}^* &= \argmin_{\theta} \widehat{Err}^\text{test}({\cal A}(\theta, {\cal L})) \\
                   &= \argmin_{\theta} \overline{E}({\cal A}(\theta, {\cal L}_\text{train}), {\cal L}_\text{test})
\end{align}
In practice, solving this later equation is also a difficult task but
approximations can be obtained in several ways, e.g., using either manual
tuning of $\theta$, exhaustive exploration of the parameter space using grid
search, or dedicated optimization procedures (e.g., using random
search~\citep{bergstra:2012}). Similarly, when ${\cal L}$ is scarce, the same
procedure can be carried out in the exact same way but using $K$-fold
cross-validation estimates (as defined by Equation~\ref{eqn:cv-error}) instead of test sample
estimates. In any case, once $\widehat{\theta}^*$ is identified, the learning
algorithm is run once again on the entire learning set ${\cal L}$, finally
yielding the approximately optimal model ${\cal A}(\widehat{\theta}^*, {\cal
L})$.

When optimizing $\theta$, special care must be taken so that the resulting model
is neither too simple nor too complex. In the former case, the model is indeed
said to \textit{underfit} the data, i.e., to be not flexible enough the capture
the structure between $X$ and $Y$. In the later case, the model is said to
\textit{overfit} the data, i.e., to be too flexible and to capture isolated
structures (i.e., noise) that are specific to the learning set.

\begin{figure}
    \centering
    \includegraphics[width=0.9\textwidth]{figures/ch2_train_test_error.pdf}
    \caption{Training and test error with respect to the complexity
             of a model. The light blue curves show the training error over
             ${\cal L}_\text{train}$ while the light red curves show the test
             error estimated over ${\cal L}_\text{test}$ for $100$ pairs of training
             and test sets ${\cal L}_\text{train}$ and ${\cal L}_\text{test}$
             drawn at random from a known distribution. The thick blue curve
             is the average training error while the thick red curve is the
             average test error. (Figure inspired from \citep{hastie:2005}.) }
    \label{fig:train-test-error}
\end{figure}

As Figure~\ref{fig:train-test-error} illustrates, this phenomenon can be
observed by examining the respective training and test estimates of the model
with respect to its complexity\footnote{Unless mentioned otherwise, {\it
complexity} here refers to the complexity of the model in function space. {\it
Computational complexity} is studied later in
Section~\ref{sec:5:complexity-fit}}. When the model is too simple, both the training and test estimates are
large because of underfitting. As complexity increases, the model gets more
accurate and both the training and test estimates decrease. However, when the
model becomes too complex, specific elements from the training set get
captured, reducing the corresponding training estimates down to $0$, as if the
model were perfect. At the same time, the test estimates become worse because
the structure learned from the training set is actually too specific and does
not generalize. The model is overfitting. The best hyper-parameter value
$\theta$ is therefore the one making the appropriate trade-off and producing a
model which is neither too simple nor to complex, as shown by the gray line on
the figure.

As we will see later in Chapter~\ref{ch:forest},
overfitting can also be explained by decomposing the generalization error
in terms of bias and variance. A model which is too simple usually has
high bias but low variance, while a model which is too complex usually has
low bias but high variance. In those terms, finding the best model amounts
to make the appropriate \textit{bias-variance trade-off}.

\subsection{Selecting and evaluating simultaneously}
On real applications, and in particular when reporting results, it usually
happens that one wants to do both model selection and model assessment. That
is, having chosen a final model ${\cal A}(\widehat{\theta}^*, {\cal L})$, one
wants to also estimate its generalization error.

A naive assessment of the generalization error of the selected model might be
to simply use the test sample estimate $\smash{\widehat{Err}^\text{test}
({\cal A}(\widehat{\theta}^*, {\cal L}))}$ (i.e., $\overline{E}({\cal
A}(\widehat{\theta}^*, {\cal L}_\text{train}), {\cal L}_\text{test})$) that was
minimized during model selection. The issue with this estimate is that the
learned model is not independent from ${\cal L}_\text{test}$ since its repeated
construction was precisely guided by the minimization of the prediction error
over ${\cal L}_\text{test}$. As a result, the minimized test sample error is in
fact a biased, optimistic, estimate of the true generalization error, sometimes
leading to substantial underestimations. For the same reasons, using the $K$-fold
cross-validation estimate does not provide a better estimate since model
selection was similarly guided by the minimization of this quantity.

To guarantee an unbiased estimate, the test set on which the generalization
error is evaluated should ideally be kept out of the entire model selection
procedure and only be used once the final model is selected.
Algorithm~\ref{algo:train-valid-test} details such a protocol. Similarly,
per-fold estimates of the generalization error should be kept out of the model
selection procedure, for example using nested cross-validation within each
fold, as explicited in Algorithm~\ref{algo:cv}.

\begin{algorithm}\label{algo:train-valid-test}
Train-Valid-Test set protocol for both model selection and evaluation.

\begin{enumerate}
\item Divide the learning set ${\cal L}$ into three parts
      ${\cal L}_\text{train}$, ${\cal L}_\text{valid}$ and ${\cal L}_\text{test}$;
\item Perform model selection on ${\cal L}_\text{train} \cup {\cal L}_\text{valid}$
      using test sample estimates, i.e., find:
      \begin{align}
      \widehat{\theta}^* &= \argmin_{\theta} \widehat{Err}^\text{test}({\cal A}(\theta, {\cal L}_\text{train} \cup {\cal L}_\text{valid})) \\
                         &= \argmin_{\theta} \overline{E}({\cal A}(\theta, {\cal L}_\text{train}), {\cal L}_\text{valid});
      \end{align}
\item Evaluate the (unbiased) generalization error of the final model as
      \begin{equation}
      \overline{E}({\cal A}(\widehat{\theta}^*, {\cal L}_\text{train} \cup {\cal L}_\text{valid}), {\cal L}_\text{test});
      \end{equation}
\item Learn the final model ${\cal A}(\widehat{\theta}^*, {\cal L})$ on the entire learning set.
\end{enumerate}
\end{algorithm}

\begin{algorithm}\label{algo:cv}
Nested $K$-fold cross-validation protocol for both model selection and evaluation.

\begin{enumerate}
\item Divide the learning set ${\cal L}$ into $K$ folds ${\cal L}_1, ..., {\cal L}_K$;
\item For each fold $k=1, ..., K$:
    \begin{enumerate}
        \item Divide ${\cal L}\setminus{\cal L}_k = {\cal L}^{-k}$ into $K$ folds ${\cal L}_1^{-k}, ..., {\cal L}_K^{-k}$;
        \item Perform model selection on the subset ${\cal L}^{-k}$ using nested $K$-fold estimates, i.e., find:
        \begin{align}
            \widehat{\theta}_k^* &= \argmin_{\theta} \widehat{Err}^{CV}({\cal A}(\theta, {\cal L}^{-k}))\\
                                 &= \argmin_{\theta} \frac{1}{K} \sum_{l=1}^K \overline{E}({\cal A}(\theta, {\cal L}^{-k} \setminus {\cal L}_l^{-k}), {\cal L}_l^{-k});
        \end{align}
        \item Evaluate the generalization error of the selected sub-model as
        \begin{equation}
            \overline{E}({\cal A}(\widehat{\theta}^*_k, {\cal L}\setminus{\cal L}_k), {\cal L}_k);
        \end{equation}
    \end{enumerate}
\item Evaluate the (unbiased) generalization error of the selected model as the average generalization estimate of the sub-models selected over the folds:
\begin{equation}
\frac{1}{K} \sum_{k=1}^K \overline{E}({\cal A}(\widehat{\theta}^*_k, {\cal L}\setminus{\cal L}_k), {\cal L}_k);
\end{equation}
\item Perform model selection on the entire learning set ${\cal L}$ using $K$-fold cross-validation estimates, i.e., find:
\begin{align}
    \widehat{\theta}^* &= \argmin_{\theta} \widehat{Err}^{CV}({\cal A}(\theta, {\cal L}))\\
                       &= \argmin_{\theta} \frac{1}{K} \sum_{k=1}^K \overline{E}({\cal A}(\theta, {\cal L} \setminus {\cal L}_k), {\cal L}_k);
\end{align}
\item Learn the final model ${\cal A}(\widehat{\theta}^*, {\cal L})$ on the entire learning set.
\end{enumerate}
\end{algorithm}

\section{Classes of learning algorithms}
\label{sec:2:classes-of-algorithms}

Before proceeding in the next chapters, and for the rest of this work, to an
in-depth analysis of the class of tree-based methods,  we briefly review in
this section some of the other learning algorithms that have matured in the
field, including linear methods, support vector machines, neural networks and
nearest neighbor methods.

\subsection{Linear methods}

One of the oldest class of supervised learning algorithms is the class of
\textit{linear methods} from the field of statistics. In these methods, the central
assumption made on ${\cal H}$ is that the output variable $Y$ can be described
as a linear combination of the input variables $X_1, ..., X_p$ (i.e., as an hyperplane), or at least
that the linear model is a reasonable approximation. For regression, ${\cal H}$
includes all models $\varphi$ of the form:
\begin{equation}
\varphi(\mathbf{x}) = b + \sum_{j=1}^p x_j w_j
\end{equation}
For binary classification (i.e., when ${\cal Y}=\{c_1, c_2\}$),  ${\cal H}$ includes all
models $\varphi$ of the form:
\begin{equation}\label{eqn:linear-model}
\varphi(\mathbf{x}) = \begin{cases}
c_1 & \text{if } b + \sum_{j=1}^p x_j w_j > 0\\
c_2 & \text{otherwise}
\end{cases}
\end{equation}

Linear methods come in many flavors but mostly differ from each other in the
way they estimate the coefficients $b$ and $w_j$ (for $j=1,\dots,p$), usually
using some specific optimization procedure to minimize a specific criterion.
Among all of them, the most famous is the method of \textit{least squares}. It
consists in finding the $b$ and $w_j$ coefficients that minimize the
resubstitution estimate (Equation~\ref{eqn:training-error}) using the squared
error loss.

Despite an apparent simplicity, linear methods often provide reliable
predictions and an interpretable description of how the input variables affect
the output. Contrary to their name, linear methods can also be used to model
non-linear relations between $X$ and $Y$, for example by applying these methods
on (non-linear) transformations of the input variables. For more details on
(generalized) linear methods, see the reviews of \citet{maccullagh:1989},
\citet{hastie:2005}, \citet{bishop:2006} or \citet{duda:2012}.

\subsection{Support vector machines}

When data points from the learning set are linearly separable, there exist
several hyperplanes (i.e., several linear models) that are in fact equally as
good when evaluated in resubstitution. In generalization however, these
hyperplanes are usually not equivalent. As illustrated in
Figure~\ref{fig:hyperplane}, a good separation is intuitively achieved when the
distance (also known as the \textit{margin}) to the nearest training data
points is as large as possible, since in general the larger the margin the
lower the generalization error of the model.

\begin{figure}
    \centering
    \includegraphics[width=0.9\textwidth]{figures/ch2_hyperplane.pdf}
    \caption{A good separating hyperplane is an hyperplane that maximizes the distance to the nearest training data points.}
    \label{fig:hyperplane}
\end{figure}

Mathematically, support vector machines~\citep{boser:1992,cortes:1995} are
maximum-margin linear models of the form of Equation~\ref{eqn:linear-model}.
Assuming without loss of generality that ${\cal Y}=\{-1,1\}$ and that $b=0$,
support vector machines are learned by solving the following primal
optimization problem:
\begin{equation}
\min_{\mathbf{w},\xi } \left\{\frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{i=1}^N \xi_i \right\}
\end{equation}
subject to
\begin{equation}
y_i(\mathbf{w}\cdot\mathbf{x_i}) \ge 1 - \xi_i, \quad \xi_i \ge 0.
\end{equation}
In its dual the form, the optimization problem is
\begin{equation}
\max_{\alpha} \left\{ \sum_{i=1}^N \alpha_i - \frac{1}{2}\sum_{i, j} \alpha_i \alpha_j y_i y_j \mathbf{x}_i\cdot\mathbf{x}_j \right\}
\end{equation}
subject to
\begin{equation}
0 \leq \alpha_i \leq C,
\end{equation}
where $C$ is an hyper-parameter that controls
the degree of misclassification of the model, in case classes are not linearly
separable. From the solution of dual problem, we have
\begin{align}
\textbf{w} = \sum_{i=1}^N \alpha_i y_i \mathbf{x}_i,
\end{align}
from which the final linear model can finally be expressed.

Support vector machines extend to non-linear classification by projecting the
original input space into a high-dimensional space (the so-called
\textit{kernel trick}), where a separating hyperplane can hopefully be found.
Interestingly, the dual optimization problem is exactly the same, except that
the dot product $\mathbf{x}_i\cdot\mathbf{x}_j$ is replaced by a
\textit{kernel} $K(\mathbf{x}_i, \mathbf{x}_j)$\label{ntn:kernel}, which
corresponds the dot product of $\mathbf{x}_i$ and $\mathbf{x}_j$ in the new
space.

\subsection{Neural networks}

The family of \textit{neural networks} methods finds its origins in attempts to
identify mathematical representations of information processing in biological
systems. While this objective is still far from being reached, (artificial)
neural networks have nonetheless proven all their worth from a statistical point of
view, and have actually grown into one of the most effective methods in machine
learning.

A neural network is usually made of several units, also known as \textit{neurons}, of the form
\begin{equation}
h_j(\mathbf{x}) = \sigma(w_j + \sum_{i=1}^n w_{ij} x_i),
\end{equation}
where $\sigma$ is a non-linear activation function, such as the sign function,
the sigmoid function or the softmax activation. In most cases, these units are
structured into successive layers, where the outputs of a layer are directed
through weighted connections, also known as \textit{synapses}, to the inputs of
the next layer. As an example, Figure~\ref{fig:mlp} illustrates a three layered
neural network. The first layer is the input layer, which transmits the input
values $\mathbf{x} = (x_1, ..., x_p)$ to the second layer. The second layer is
made of activation units $h_j$, taking as inputs the weighted values of the
input layer and producing non-linear transformations as outputs. The third
layer is made of a single activation unit, taking as inputs the weighted
outputs of the second layer and producing the predicted value $\hat{y}$.
Assuming that this structure is fixed and that all units from the network make
use of the same activation function $\sigma$, the hypothesis space ${\cal H}$
therefore includes all models $\varphi$ of the form
\begin{equation}
\varphi(\mathbf{x}) = \sigma(w_5 + \sum_{j=1}^4 w_{j5} \sigma(w_j + \sum_{i=1}^p w_{ij} x_i)).
\end{equation}

\begin{figure}
    \centering
    \includegraphics[scale=1.0]{figures/ch2_mlp.pdf}
    \caption{An artificial neural network.}
    \label{fig:mlp}
\end{figure}

As for linear methods, learning a neural network  amounts to estimate the
weights $w_{ij}$ that minimize some specific loss function, using some specific
optimization procedure. Among all of them, the most famous is the
\textit{backpropagation algorithm}~\citep{bryson:1975}.

Recent advances on neural networks, now themed as \textit{deep learning}, have
recently shown the capability of these models to autonomously learn high-level
and very effective representations of data. On various difficult tasks, such as
image classification or speech recognition, neural networks have indeed
achieved outstanding performance, outperforming both human operators and
state-of-the-art methods. From a theoretical point of view however, little is
still known about what makes them truly work. In particular, picking the
appropriate combination of number of units, layers and types of activation
functions remains a delicate task, which unfortunately makes (deep) neural
networks still difficult to use for non-experts. Reviews on recent developments
include the works of \citet{hinton:2007}, \citet{arel:2010} or \citet{bengio:2013}.

\subsection{Nearest neighbor methods}

\textit{Nearest neighbor methods} belong to a class of non-parametric algorithms
known as \textit{prototype methods}~\citep{hastie:2005}. They distinguish from
other learning algorithms in the sense that they are memory-based and require
no model to be fit.

The principle behind nearest neighbor methods is to find a number of
training samples closest in distance to the new sample, and then infer from
these the value of the output variable. For regression, the $k$-nearest
neighbor algorithm~\citep{fix:1951} averages the output values from the $k$
closest training samples, that is:
\begin{equation}
\varphi(\mathbf{x}) = \frac{1}{k} \sum_{(\mathbf{x_i}, y_i) \in NN(\mathbf{x}, {\cal L}, k)} y_i,
\end{equation}
where $NN(\mathbf{x}, {\cal L}, k)$ denotes the $k$ nearest neighbors of ${\mathbf{x}}$ in ${\cal L}$.
For classification, the procedure is the same except that the predicted
output value is computed as the majority class among the $k$ nearest neighbors:
\begin{equation}
\varphi(\mathbf{x}) = \argmax_{c \in {\cal Y}} \sum_{(\mathbf{x_i}, y_i) \in NN(\mathbf{x}, {\cal L}, k)} 1(y_i = c).
\end{equation}
In general, the distance function used to identify the $k$ nearest neighbors
can be any metric, but the standard Euclidean distance is the most common
choice. A notable variant is the
\textit{radius-based neighbor algorithm}, in which the predicted output value
is computed from the training samples within a radius $r$ of the new sample.
In cases where data is not uniformly sampled, this latter algorithm can be
a better choice than the $k$-nearest neighbor method.

Despite their simplicity, nearest neighbor methods usually yield decent
results in practice. They are are often successful in classification situations
where the decision boundary is very irregular. From a theoretical point of
view, an important result due to \citet{cover:1967} is the proof of consistency
of the method. When the size of ${\cal L}$ tends to infinity and $k$ is
appropriately adjusted to ${\cal L}$, the generalization error of the model
produced by the $k$-nearest neighbor method is shown to converge towards the
generalization error of the Bayes model.


\chapter*{Abstract}

Dependent typing is a double-edged sword: One the one hand, it allows for more precise types, hence more guarantees of the program's correctness at compile time while also giving more hints about the program's structure and properties when reading the code. On the other hand, the programmer needs to provide these types in the first place, thus more work is required, especially in volatile code bases. A combination of label dependent types with gradual typing---allowing dynamic typing in an otherwise statically typed language---is proposed by \cite{fu2021}. Building upon the proposal, this thesis provides an implementation of the \emph{Cast Calculus Label Dependent Lambda Calculus} (CCLDLC) in the Haskell programming language. The \emph{Cast Calculus} introduces dynamic type $\star$ as well as an explicit type cast notation $(M : A \Rightarrow B)$. By applying various rules for casts between arbitrary types and the dynamic type these additional constructs integrate into the basic \emph{Label Dependent Lambda Calculus} (LDLC). Extending an existing interpreter, this builds the foundation for full gradual typting.
\\
The accompanying code resides at \url{https://github.com/leyhline/ldgv/}.