\chapter{Machine Learning and Random Forests}
\label{ch:ML}
%
The analysis tasks described in \autoref{ch:iact} make strong use of machine
learning tools to be performed in the most efficient way. There are a lot of
different machine learning techniques that find increasingly many and
successfull applications in modern physics. The following overview is based on
the works \cite{bagging, goodfellow, mitchell}.

\section{Machine Learning}
%
Machine learning describes the field of applied statistics that uses computers
to learn to solve certain problems. The \textit{learning} is defined by Tom
Mitchell~\cite{mitchell} as: \enquote{A computer program is said to learn from
experience $E$ with respect to some class of tasks $T$ and performance measure
$P$, if its performance at tasks in $T$, as measured by $P$, improves with
experience $E$.} There is a wide variety of different tasks this can be applied
to and many performance measures. The most important preposition for well
suited problems is the available experience or in this case the amount of data
to learn from.

Machine learning aims at solving problems that profit from the computing power
of modern technology but are not the kind of problem to be solved by a typical
program written by a human \cite{goodfellow}.

\subsection{The Experience}
%
To make a generic machine learning algorithm work for a specific task it has to
be \textit{trained} on data. Just like a human, it needs to be given
information to base a decision on and to be shown how that decision based on
the information is supposed to look. The kind of data suitable for learning
depends on the kind of algorithm to be used and vice versa. Generally, machine
learning algorithms can be divided into \textbf{supervised} and
\textbf{unsupervised} algorithms, both of which will be used in this work.

Data to derive experience from for supervised algorithms consists of data points
with a certain feature set and a specific label or true value. The data sets for
the \textit{separation} of two classes for example contain a certain feature
set for every data point and a label for the corresponding class each of these
points belongs to. Supervised learning therefore aims at learning to predict
a certain target value (or multiple) from a given feature set including this
value. The used learning data in this analysis is provided by simulations, so
that the true values for the machine learning tasks are known. Unsupervised
learning, however, works on unlabelled data and is used in this work in form of
a clustering algorithm.

\subsection{The Task}
%
The task a machine learning algorithm is supposed to do is the solution of a
specific problem in the best possible way. To reach this solution the process
of learning is used. A machine learning algorithm uses data to adapt to a specific task upon the given data. The three desired tasks
within the analysis of Cherenkov telescope images are the \textit{separation} of
gamma-rays from hadronic cosmic rays, as well as the \textit{estimation} of the
energy and source position of cosmic gamma-rays. The kind of task already
determines what machine learning techniques are best to be used or which ones
do not suit the problem and how working architectures have to look. A
classification like the separation of gamma-rays from hadronic cosmic rays
requires a different output and therefore network structure than a regression
task.

\subsection{The Performance Measure}
%
As described above, machine learning is about improving on certain tasks.
Therefore, it is essential to quantify the performance during, but also after the
learning process. This already implicates that performance measures are very
specific to the task. During the training of a machine learning algorithm a
loss is being minimalized. This loss is dependant on the task at hand and improved during the training epochs until a saturation is reached. The performance of a classification task is naturally
measured by the \textbf{accuracy} of the model, because it simply describes how
many of the model's outputs are correct. When validating continuous outputs
rather than discrete classifications an accuracy is not appropriate. The
estimation of the energy, e.g., requires a continuous-valued metric.

The models are trained on specific, simulated data but only to be applied on
data sets they have \textit{not} been trained on. Thus, the interesting metrics
are those calculated on data sets complementary to the training data sets,
because they resemble the real use case. To do so, the whole data set is divided
into a fraction determined for training and a test set on which the metrics can
be calculated. A frequently used method for this is the
\textbf{cross-validation}. The data set is divided into $n$ equaly sized, random
subsamples; $n-1$ samples are used for training whereas the single excluded
sample is used for validation. This is done $n$ times for each one of the
subsamples and the metrics are calculated as the mean value of all the single
validations.

\section{Random Forests}
%
One of the most frequently used machine learning algorithms and the one used
for the tasks in this analysis is the so called random forest. This is a
supervised learning algorithm based on the so called \textit{decision tree}.

\subsection{Decision Trees}
%
Decision trees classify data points based on consecutive binary decisions. The
decisions are based on the single features of the data point and result in a
point specific result that is being returned. The number of single binary
decisions (also called \textit{node}) preceding the final classification is
called \textit{depth} of the tree. Decision trees are trained on labelled data
and determine thresholds for every feature to classify the data point.
%
\begin{figure}[H]
  \centering
  \begin{tikzpicture}[node distance = 3cm, auto]
    \node (f1) [feature] {\texttt{feature\_1}};
    \node (f2) [feature, below of=f1, xshift=-2.4cm] {\texttt{feature\_2}};
    \node (f3) [feature, below of=f1, xshift=2.4cm] {\texttt{feature\_3}};

    \node (c1) [class1, below of=f2, xshift=-1.2cm] {\texttt{class 1}};
    \node (c2) [class2, below of=f2, xshift=1.2cm] {\texttt{class 2}};

    \node (c3) [class1, below of=f3, xshift=-1.2cm] {\texttt{class 1}};
    \node (c4) [class2, below of=f3, xshift=1.2cm] {\texttt{class 2}};

    \draw [pil] (f1) -- node[anchor=east] {$\mathbf{>15}$\;} (f2);
    \draw [pil] (f1) -- node[anchor=west] {\;$\mathbf{\leq15}$} (f3);
    \draw [pil] (f2) -- node[anchor=east] {$\mathbf{<100}$\;} (c1);
    \draw [pil] (f2) -- node[anchor=west] {\;$\mathbf{\geq100}$} (c2);
    \draw [pil] (f3) -- node[anchor=east] {$\symbf{<-\sfrac{\pi}{2}}$\;} (c3);
    \draw [pil] (f3) -- node[anchor=west] {\;$\symbf{\geq-\sfrac{\pi}{2}}$} (c4);
  \end{tikzpicture}
  \caption{Example sketch of a decision tree. The tree decides whether an input data point is of class 1 or class 2. The available features are \texttt{feature\_1}, \texttt{feature\_2} and \texttt{feature\_3}. This decision tree has a depth of 2 and solves the task of a binary separation by performing binary decisions at each node (green boxes). The decision thresholds are written next to the respective connecting lines.}
  \label{fig:tree}
\end{figure}
%
To determine the best decision threshold for each node, a specific loss function
to be minimalized is necessary. For a classification task the required
loss would be the information gain. The thresholds are thus optimized to get the best gain in information at the respective node. This way the decision tree is build from the
top node downwards until a perfect classification is achieved or until a set
maximum depth is reached.

Decision trees can also be used for regression tasks. The only difference when
solving such tasks is the metric to find the best thresholds at each node.
While for classification tasks the information gain is the natural choice, regression
tasks require a metric describing the error of the single result. Thus, the
thresholds are determined by minimizing the variance of the target variable.

\subsection{Random forests}
%
A decision tree that is fitted to the data too extensively will reach
very high accuracies, but will suffer from a very bad generalization. At some
point the tree starts to adapt to the training data's specifics too much and
will not work on other data that does not have these specific characteristics.
This phenomenon is called \textit{overfitting} and generally is represented by
a large gap between training and test errors \cite{goodfellow}. A way to
prevent this from happening is to constrain the complexity of the decision tree
while using a large number of different trees. This way the trees are not
overfitted but the complete model is still complex enough.

Random forests are generated by a process called
\textit{bagging}~\cite{bagging}. Every tree within the forest is trained on a
subset of the data randomly sampled with replacement. This way, a large number
of slightly different trees is generated. While every single decision tree is
still prone to overfitting to fluctuations in its respective data set, averaging over
all of them is not, as long as the trees are not correlated. To prevent such
correlations and further generalize the model, additionally every decision node
inside a single tree is only given a random subset of all available features.
Random forests therefore have two additional parameters: the number of trees
$n$ and the number $k$ of available features at each node.

The output of the model consisting of such a forest is then generated
by counting trees with a specific decision. A confidence for a multi-class
separation, e.g., can be generated by counting the fraction of trees that
decided for the specific class. The output of a random forest therefore is generated by averaging over all decision trees. Thus, the output score for a classification lies between \num{0} and \num{1}, representing the average class decision of the trees. By setting a threshold on this confidence
classification, decisions can be performed. In case of regression tasks the
output of the forest is the mean of the regression results of the single trees,
as well.

\section{Performance Measures}
%
As mentioned earlier, the right performance measure depends on the task to be
validated. Since there are several different tasks to be performed during this
analysis a number of performance measures is needed for the evaluation.

\textbf{Accuracy.} The accuracy of a model describes the proportion of input
examples for which the model computes the right output. This performance
measure can be used for models with discrete outputs, such as classifications
because it only takes perfect matches as the right output.

\textbf{Receiver Operating Characteristic.} For every classification task solved via the above mentioned forest output the
so called \textit{receiver operating characteristic} curve (ROC) can be
determined. For a given class this represents the rate of correctly classified
examples dependant on the rate of falsely classified ones. The area under this
curve (AUC) can be used to quantify the performance of a classification model.
When randomly deciding on an equally distributed data set of two classes, the
AUC is expected to be about \num{0.5}, representing the worst possible
classification performance. A perfect classification yields an AUC of \num{1}.

\textbf{Confusion Matrix.} The confusion matrix shows the results of a machine
learning algorithm against the true values. It therefore shows how well the
algorithm performs by showing where mismatches occur and with what effect. For
classifications this matrix shows what classes are most frequently confused
with each other, while for a regression task it shows how big the spread of
reconstructed values around the true value is.

% \textbf{Precision.} For a classification the precision describes the fraction
% of events correctly classified as signal events over all events labelled as
% signal events.
% %
% \begin{equation}
%   \text{precision} = \frac{T_{\text{p}}}{T_{\text{p}} + F_{\text{p}}}
% \end{equation}
% %
% \textbf{Recall.} The recall of a classification model describes the fraction
% of events correctly classified as signal events over all true signal events.
% %
% \begin{equation}
%   \text{recall} = \frac{T_{\text{p}}}{T_{\text{p}} + F_{\text{n}}}
% \end{equation}
% %
% \textbf{$F_\beta$ score.} The $F_\beta$ score of a classification model describes the harmonic mean of the precision and recall.
% %
% \begin{equation}
%   F_\beta = (1 + \beta^2)\frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}
% \end{equation}
%
\textbf{$\symbf{R^2}$ Score.} This metric quantifies the goodness of fit for a
chosen model. For a regression task, the $\symbf{R^2}$ score or sometimes
called \textit{coefficient of determination} measures how well the input data
points are approximated by the model. A regression perfectly describing the
input data is indicated by an $\symbf{R^2}$ score of 1, whereas an
$\symbf{R^2}$ score closer to \num{0} characterizes a decreasing quality of the
model. Models with scores below zero can be interpreted as not suitable to the
data.