\documentclass[10pt,leqno]{amsart}

% Les packages
\usepackage{amssymb,amsbsy}
\usepackage{mathrsfs}
\usepackage{color}
\usepackage{soul}
\usepackage{graphicx}
\usepackage{booktabs}
%\setlength{\heavyrulewidth}{1.5pt}
%\setlength{\abovetopsep}{4pt}
\usepackage[pdfborder={0 0 0}]{hyperref}
\usepackage[left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}
\setlength{\parskip}{3pt} % 1ex plus 0.5ex minus 0.2ex}
%\setlength{\parindent}{15pt}

% La profondeur de la table des matieres
\setcounter{tocdepth}{1}

% Les principaux ensembles
\newcommand{\Abs}[1]{\left\lvert#1\right\rvert}
\newcommand\N{{\mathbb N}}
\newcommand\R{{\mathbb R}}
\newcommand\T{{\mathbb T}}
\newcommand\C{{\mathbb C}}
\newcommand\Q{{\mathbb Q}}
\newcommand\Z{{\mathbb Z}}
\newcommand\Pp{{\mathbb P}}
\newcommand\Ee{{\mathbb E}}
\def\x{{\mathbf x}}
\def\w{{\mathbf w}}
\def\xxi{{\pmb \xi}}

% Quelques definitions raccourcis pour les calculs
\DeclareMathOperator{\diver}{div}
\DeclareMathOperator{\Supp}{Supp}
\newcommand{\dbm}{\mathbf{d}}
\def\esssup{\mathop{\rm ess\,sup\,}}
\def\essinf{\mathop{\rm ess\,inf\,}}
\newcommand{\indiq}{{\bf 1}}

% Les defs, theo, et equations
\newtheorem{theo}{Theorem}
\newtheorem{prop}[theo]{Proposition}
\newtheorem{lem}[theo]{Lemma}
\newtheorem{cor}[theo]{Corollary}
\newtheorem{rem}[theo]{Remark}
\newtheorem{rems}[theo]{Remarks}
\newtheorem{defin}[theo]{Definition}
\newtheorem{ex}[theo]{Example}
\newtheorem{notation}[theo]{Notations}
\renewcommand{\theequation}{\thesection.\arabic{equation}}
\renewcommand{\thetheo}{\thesection.\arabic{theo}}

\newcommand{\PM}[1]{\textcolor{red}{#1}}
\newcommand{\CQ}[1]{\textcolor{blue}{#1}}

%%%%%%%%%%      Document     %%%%%%%%%%%%%%%

\begin{document}

\title[]{Machine learning for Social Ideology Recognition: An Experimental Case Study.}

\author{H. Bahamondes \and C. Quininao}

\begin{abstract}

\end{abstract}

\maketitle

\begin{center} {\bf Preliminary version of \today}
\end{center}

\bigskip
\textbf{Keywords}: Vote Selling - Machine Learning

\vspace{0.3cm}
\bigskip

\tableofcontents

\section{Materials and methods}

The overall model training and validation process consists in a three phases. First we clean the database by eliminating repeated observations, and we normalize using a MinMax scaler. We perform two different experiments, first we use machine learning algorithms to classify participants as vary liberals or not, and later we do the same for very conservative. Since our resulting training pairs datasets are strongly imbalanced, prior to the construction of learning models, we use the SMOTE~\cite{chawla2002smote} oversampling procedure with five nearest neighbors for each vote-selling sample~\cite{artetxe2020balanced}. The resulting training data pairs is then used for validation in a repeated cross-validation (RCV) process. This process consists in construct several machine learning models by using the same dataset but under the idea of generate folds and repetition. In particular, each cross-validation repetition consists in: partitioning the dataset in a particular number of folds, using each fold as a test dataset while using all remaining data for model training. Finally, we report average performance measures across all repetitions. In this manuscript, the reported results are the average of the 30 repetitions of the CV models.

\subsection{Classification Methods}

Several machine learning approaches can be used for analyze the present data, in a predictive model framework~\cite{vapnik2013nature,witten2002data,maimon2005data}. Due to the nature of the experiment and the data, and since we do not have a large dataset, we discard the application of deep learning algorithms~\cite{balas2019handbook}. Therefore, we focus in the following classical well-known methods: \textit{Support Vector Machines, Multilayer Perceptron and Na\"ive Bayes method.}

\subsubsection{Support Vector Machines (SVM)} As stated in~\cite[Chapter 12]{maimon2005data}, support vector machines (SVMs) are a set of related methods for supervised learning, which has applications to classification (the output data is a sequence of tags) and regression (the output data is a continuous variable) problems. When using SVM for the classifying problem, we look for linear functions that can separate the data based on the support vectors or boundaries of the classes. The idea of the algorithm is to find the optimal hyperplane, i.e., the hyperplane that maximizes the distance to the boundaries of each class. This can be done by stating a quadratic programming problem~\cite{vapnik2013nature}. When the classes are not linearly separable, then it is possible place the data into a space of superior dimensionality using a kernel trick~\cite{vapnik2013nature,maimon2005data}, so that the transformed dataset might become linearly separable. We used the SVM module from Python Sklearn library for training and estimation. We try linear and Radial Bases Function (RBF).

\subsubsection{Multilayer Perceptron} Multilayer perceptron (MLP) is the classical feed-forward artificial neural networks (ANN) composed of multiple densely interconnected layers of computational units, aka artificial neurons~\cite{wolff2019machine}. It corresponds to a supervised learning algorithm that finds a function that maps our multidimensional samples (socio demographic data) into the vote-selling response of the experimental participants. Any MLP is constructed as follows: (i) we fix a number of hidden layers, (ii) the architecture starts with $m$ artificial neurons corresponding to the features of the input data corresponding to the input layer, (iii) any artificial neuron in the first hidden layer integrates the inputs from the input layer and combines the input values with a weighted linear summation (the weights become part of the parameters to be tuned in the learning phase), (iv) the result of this summation is nonlinearly transformed through an activation function (for instance an hyperbolic $\tanh$ function). The procedure is repeated for each hidden layer, until the output layer is reached. The connection weights can be learned from data applying the back-propagation algorithm~\cite{haykin2004comprehensive}. Concerning the implementation of the MLP, we used the MLPClassifier module from Python Sklearn library for training and estimation. The MLPClassifier implements a MLP algorithm that trains using back-propagation through a stochastic gradient descent. We used the rectified linear unit function for the activation of the hidden layers. The MLP model constructed has two hidden layer, each of 50 neurons.

\subsubsection{Na\"ive Bayes Method} Na\"ive Bayes methods are a set of supervised learning algorithms based on applying Bayes’ theorem with the ``naive'' assumption that all features of an individual are independent. Bayes’ theorem provides a rule to calculate the conditional probability of an event given some knowledge, in terms of the conditional probability of the knowledge given the event. This idea along with the naive independent assumption allows the model to estimate the joint probability distribution of a  feature vector as the product of the unidimensional distribution probabilities of each feature~\cite{wolff2019machine}. In our study we use the GaussianNB module from Sklearn library, which implements the Gaussian Naive Bayes algorithm for classification. The likelihood of the features is assumed to be Gaussian.

\subsection{Classification Performance Metrics} We illustrate the chosen metrics with only liberal classification machines, the conservative situation is straightforward. At the end of each cross-validation fold we have two different vectors: $y_{test}$ corresponding to the responses of the participants to the very liberal question restricted to the fold used for testing, and $y_{pred}$ corresponding to the predicted answers given by the model for the input features. With these two vectors at hand we compute the
\begin{itemize}
\item True positive counts (tp): number of participants such that the model predicts as very liberal and they actually belong to the very liberal class
\item True negative counts (tn): number of participants such that the model predicts as not very liberal and they do not belong to the very liberal class
\item False positive counts (fp): number of participants such that the model predicts as very liberal but they do not belong to the very liberal class
\item False negative counts (fn): number of participants such that the model predicts as not very liberal but they actually belong to the very liberal class
\end{itemize}
with this numbers, we compute the Recall or sensitivity $R$\footnote{The fraction of examples classified as very liberal, among the total number of very liberal examples.}, the positive predictive value $PPV$ or precision\footnote{The fraction of true very liberal examples among the examples that the model classified as very liberal.} and the $f$-score\footnote{A perfect model has an $f$-score of 1.} with the relationships
$$
R=\frac{tp}{tp+fn},\quad PPV=\frac{tp}{tp+fp},\quad F=\frac{2}{1/R+1/PPV},
$$
and finally report the average and the standard deviation across all cross-validation folds. Notice one could use the accuracy metric defined as
$$
A = \frac{tp+tn}{tp+tn+fp+fn}=\frac{tp+tn}{\text{size of the fold}},
$$
but considering that the dataset is strongly imbalanced, previous measures are more informative. Another metric widely used to compare binary classifiers~\cite{wolff2019machine,artetxe2020balanced} is the Receiver Operating Curve or ROC, which corresponds to a plot of sensitivity $R$ versus the false positive rate
$
FPR = \frac{fp}{fp+tn}.
$
ROC is computed as a probability curve and, and the area under ROC curve of AUC represents the degree of separability. In other words, it quantifies in a number how much the model is capable of distinguishing between classes. In our setting, the higher the AUC, the better the model is at predicting liberals as liberals and non liberals as non liberals. Remark that (i) a perfect model has AUC equals to 1, (ii) a model with AUC approximately 0, means that the model is actually reciprocating the classes, therefore a simple relabelling is enough to get good results; and (iii) the worst case scenario is when the AUC equals 0.5, meaning that the model has no class separation capacity.

The methodology we use to compute previously explained metrics is as follows: at each fold test, we use the remaining folds as training dataset. Since the resulting training data set is imbalanced, we use the oversampling technique SMOTE with five neighbors on the minority class. Remark that the test set remains imbalanced which has some consequences. In particular, even one misclassified sample translates into large reductions of the performance measures. Moreover, the number of remaining liberal samples in each fold depends on the number of folds, thus having consequences in the imbalance ratio. We report results with decreasing number of RVC folds in order to test the stability of the results.

\section{Results}

Tables~\ref{tab:r_ppv} and~\ref{tab:f_auc} show the results for  metrics mentioned above of the machine learning techniques after 40 repetitions (3 fold), 10 repetitions (4 fold), 8 repetitions (5 fold) and 4 repetitions (10 fold) of the RCV experiments with SMOTE class imbalance correction. There is no statistical evidence to argument that the number of folds has an effect on the performance of each method. An F- test over the number of folds shows that there is no statistically significant difference $p\gg0.1$.

If we test the performance of each methods we find some differences, specially on the PPV, f-Score and AUC metrics. SVM with RBF kernel is above SVM Linear, MLP, and NB independently of the number of folds. On the other hand, the recall metric (R), shown in Table~\ref{tab:r_ppv} (left), shows no sufficient evidence to affirm that methods perform differently. Indeed, applying an F-test over the different type of machines for the each folds number we find no strong statistical evidence, and it would depend on the confidence level whether or not we reject the null hypothesis  (for instance, the p-value es approx $0.026$ for 10 folds). However, the result changes dramatically, if we discard the MLP model which performs below any other model ($p\approx0.77$ for 10 folds). We conclude that according to the recall metric, all methods except for MLP have similar performance. Figure 2 shows the ROC curves for all approaches in the case of RCV with 5 folders.

The f-scores shown in Table~\ref{tab:f_auc} confirm that SVM with RBF kernel improves over SVM Linear, MLP and NB regardless of RCV number of folders. An F-test carried out over these results confirms that the performance differences between predictive models are statistically significant  ($p\approx0.002$) . Specific one-sided t-tests comparing each pair of modeling approaches confirms that SVM with RBF kernel perform better than SVM Linear, MLP and NB. However, the superiority of SVM with RBF kernel relative to MLP is less pronounced ($p\approx0.05$). On the contrary, the effect is more pronounced for the AUC metric. The F-test carried out over these results confirms ($p\ll0.001$), t-test pairwise tests ($p<0.001$).
\begin{table}[!htbp]
\centering
\footnotesize
\caption{Average $\pm$ standard deviation Recall R (left) and positive predictive value PPV (right) performance of SVM (linear), SVM (RBF), MLP and NB for decreasing number of folders in the repeated cross validation process. All results are calculated with SMOTE oversampling correction of class imbalance.}
\label{tab:r_ppv}
\begin{tabular}{*5c}
\toprule
  \multicolumn{5}{c}{SMOTE} \\
nfolds & SVM (linear) & SVM (RBF) & MLP & NB
\\
\midrule
10 & 0.68$\pm$0.07 & 0.67$\pm$0.08 & 0.65$\pm$0.07 & 0.68$\pm$0.08
\\
5 & 0.68$\pm$0.05 & 0.67$\pm$0.04 & 0.65$\pm$0.06 & 0.66$\pm$0.06
\\
4 & 0.67$\pm$0.03 & 0.66$\pm$0.04 & 0.64$\pm$0.04 & 0.67$\pm$0.04
\\
3 & 0.68$\pm$0.03 & 0.67$\pm$0.05 & 0.67$\pm$0.05 & 0.67$\pm$0.05
\\
\bottomrule
\end{tabular}
\quad
\begin{tabular}{*5c}
\toprule
  \multicolumn{5}{c}{SMOTE} \\
nfolds & SVM (linear) & SVM (RBF) & MLP & NB
\\
\midrule
10 & 0.45$\pm$0.06 & 0.52$\pm$0.08 & 0.47$\pm$0.07 & 0.44$\pm$0.07
\\
5 & 0.46$\pm$0.05 & 0.52$\pm$0.07 & 0.48$\pm$0.05 & 0.44$\pm$0.05
\\
4 & 0.45$\pm$0.04 & 0.52$\pm$0.06 & 0.47$\pm$0.04 & 0.43$\pm$0.05
\\
3 & 0.45$\pm$0.03 & 0.52$\pm$0.04 & 0.45$\pm$0.03 & 0.44$\pm$0.04
\\
\bottomrule
\end{tabular}
\end{table}

\begin{table}[!htbp]
\centering
\footnotesize
\caption{Average $\pm$ standard deviation f-score (left) and AUC (right) performance of SVM (linear), SVM (RBF), MLP and NB for decreasing number of folders in the repeated cross validation process. All results are calculated with SMOTE oversampling correction of class imbalance.}
\label{tab:f_auc}
\begin{tabular}{*5c}
\toprule
  \multicolumn{5}{c}{SMOTE} \\
nfolds & SVM (linear) & SVM (RBF) & MLP & NB
\\
\midrule
10 & 0.54$\pm$0.06 & 0.58$\pm$0.07 & 0.54$\pm$0.06 & 0.53$\pm$0.06
\\
5 & 0.55$\pm$0.05 & 0.59$\pm$0.05 & 0.55$\pm$0.05 & 0.53$\pm$0.05
\\
4 & 0.54$\pm$0.03 & 0.58$\pm$0.04 & 0.54$\pm$0.03 & 0.52$\pm$0.04
\\
3 & 0.54$\pm$0.02 & 0.58$\pm$0.03 & 0.54$\pm$0.03 & 0.53$\pm$0.04
\\
\bottomrule
\end{tabular}
\quad
\begin{tabular}{*5c}
\toprule
  \multicolumn{5}{c}{SMOTE} \\
nfolds & SVM (linear) & SVM (RBF) & MLP & NB
\\
\midrule
10 & 0.68$\pm$0.04 & 0.72$\pm$0.05 & 0.68$\pm$0.04 & 0.67$\pm$0.04
\\
5 & 0.67$\pm$0.03 & 0.72$\pm$0.03 & 0.68$\pm$0.03 & 0.67$\pm$0.03
\\
4 & 0.67$\pm$0.02 & 0.72$\pm$0.04 & 0.67$\pm$0.03 & 0.67$\pm$0.03
\\
3 & 0.68$\pm$0.02 & 0.71$\pm$0.02 & 0.68$\pm$0.02 & 0.67$\pm$0.02
\\
\bottomrule
\end{tabular}

\end{table}
%$$
%0.000001
%Ttest_indResult(statistic=4.705128336546723, pvalue=1.0954486268281921e-05)
%Ttest_indResult(statistic=4.044118570892123, pvalue=0.00012315327310524272)
%Ttest_indResult(statistic=4.82778883801976, pvalue=6.762417354421785e-06)
%$$
\bibliographystyle{abbrv}
\bibliography{biblio_conjoint.bib}

\end{document}


\section{Estimators}

\subsection{Batch Least Squares}

\subsection{Extended Kalman Filter}


%%
%% Template chap2.tex
%%

\chapter{Photometric Classification}
\label{cha:ml}

Let us now bring machine learning to the realm of astronomy. We start with a motivation for why
astronomers might find machine learning helpful (Section \ref{sec:machine}) and an overview of
three families of classifiers (Section \ref{sec:classifiers}). We then discuss active learning and
six heuristics that can be used to rank unlabelled examples (Section \ref{sec:active} and
\ref{sec:heuristics}). Our novel contributions include the application of Thompson sampling to the
heuristic selection setting  (Section \ref{sec:bandit} and \ref{sec:thompson}) and a derivation of
the multi-class posterior balanced accuracy (Section \ref{sec:measures}) which can be used to
measure the performance of our algorithms.

\section{Maching Learning in Astronomy} \index{machine learning}
\label{sec:machine}

The two most common types of celestial objects are stars and galaxies. There are also some other
interesting objects such as quasars and white dwarfs. Quasars \index{quasar} are thought to be
supermassive black holes surrounded by an accretion disc. They are very luminous and, unlike
galaxies, appear as single-source objects. White dwarfs \index{white dwarf} are low to intermediate
mass stars that are in their final evolutionary stage. They are very dense and have a faint
luminosity.

One way to classify objects into these various groups is to manually inspect their spectra.
\index{spectrum} There have even been attempts to make the process more automatic. For example,
\citeN{hala14} achieved a 95\% accuracy rate by training a convolutional neural network
\index{neural network} on one-dimensional spectra to classify objects into stars, quasars, and
galaxies. Even so, it is currently not possible to obtain a spectrum of every object, especially
faint ones. This means that only a small number of objects (e.g. 0.35\% in the SDSS dataset) can be
classified this way. For the rest, we only have photometric measurements.

Fortunately, the field of machine learning came about to solve this kind of problem. In the most
basic set-up, we have a collection of objects, each with a vector of photometric measurements
$\bm{x} \in \X$. A subset of them has been spectroscopically classified into some class $y \in \Y$
and they form the labelled set $\Labelled \subset \X \times \Y$. We call $\X$ the feature space and
$\Y$ the label space. Let us now partition $\Labelled$ into two disjoint subsets, a training set
$\Labelled_T$ and a test set $\Labelled_S$. During the training phase, we feed  $\Labelled_T$ to a
classifier, and the classifier will then learn a hypothesis $h: \X \rightarrow \Y$. The labelling
process might not be perfect, so the goal in machine learning is to capture as much of the
underlying trend in the data as possible, while avoiding fitting the random noise. To see how well
the hypothesis generalises to unknown data, in the testing phase, we ask the classifier to predict
labels of objects in  $\Labelled_S$. These predictions are then be compared to the true labels,
and an accuracy rate can be calculated.

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Classifiers}
\label{sec:classifiers}

Three families of classifiers are used in our experiments. They are random forests, logistic
regression, and support vector machines. Below we give a quick overview of how each of them works.
We do not implement these classifiers ourselves. Rather we shall use scikit-learn
\cite{pedregosa11}, the most well-known machine learning package in Python.

\subsection{Random Forest} \index{random forest}
\label{sub:forest}

To understand the motivation behind random forests, we first need to know how to construct
individual decision trees\index{decision tree}. Building each of these trees is like playing a game
of Twenty Questions. We start with the whole training set and at each step, we slice the feature
space along the axis of one feature. After many steps, we end up with a set of hyper-dimensional
cuboids which form a partition of the feature space. The algorithm stops when each of these cuboids
contains data from only one class. There are many criteria that we can use to decide on which
feature and where we slice along the axis. In this thesis, we use the Gini impurity \index{Gini
    impurity} which intuitively measures the potential misclassification rate. In particular, let $k$
be the number of classes and $q_D(i)$ be the frequency of objects belonging to class $i$ in set
$D$. Then the Gini impurity of $D$ is the probability that a randomly selected object from $D$ is
misclassified if it were labelled according to its frequency in $D$:
	\begin{IEEEeqnarray*}{lCl}
		\iota_G(D) &=& \sum_{i=1}^{k} q_D(i) (1 - q_D(i)) \\
		           &=& \sum_{i=1}^{k} q_D(i)  - \sum_{i=1}^{k} q_D(i)^2 \\
		           &=& 1 - \sum_{i=1}^{k} q_D(i)^2
	\end{IEEEeqnarray*}
When we partition $D$ into subsets $\{D_1, D_2, ..., D_d\}$, the Gini impurity of $D$ is now the sum
of the individual Gini impurities, weighted by the size of the subsets:
	\begin{IEEEeqnarray*}{lCl}
		\iota_G(D) &=& \sum_{i=1}^{d} \frac{|D_i|}{|D|} \iota_G(D_i)
	\end{IEEEeqnarray*}
Observe that if a subset contains objects from only one class, then its Gini impurity will be zero.
This gives us the following splitting criterion: at each step, slice the feature space along the
axis that will result in the greatest drop in the Gini impurity.

One problem with decision trees is that they tend to overfit \index{overfitting} the data and thus
do not generalise well. To solve this, \citeN{breiman01} proposes that we build many decision
trees, thus creating a random forest. The random forest makes its prediction by simply counting up
the predictions of all the individual trees and then choosing the most popular choice. By taking an
average of the predictions, we avoid the problem of overfitting. Furthermore, for each tree, we
only give it a small bootstrap sample and at each split, we only consider a small number of
features. This bootstrapping and random subspace selection have been shown empirically to improve
the accuracy \cite{breiman96, ho98, louppe12}. Another nice feature of random forests is that
they are extremely fast and hence scale well with large datasets. Although they do not provide
class probability estimates, we can use proportions of the votes as a proxy for the
probabilities. However, in practice, we find these probability estimates to be a bit unstable.

\subsection{Logistic Regression} \index{logistic regression}
\label{sub:logistic}

If we want the hypothesis to model actual class probabilities, then an alternative approach is to use
logistic regression. Developed by \citeN{cox58}, the algorithm tries to directly model the
probability of being in a class. Let $\bm{x}$ be the feature vector and $\bm{\theta}$ be the vector
of coefficients. The linear predictor $\eta$ is defined as
	\begin{IEEEeqnarray*}{lCl}
		\eta(\bm{x}) &=& \bm{\theta}^T \bm{x}
	\end{IEEEeqnarray*}
Since probabilities must lie between 0 and 1, we want our predictor to have the same range. This can
be achieved by wrapping $\eta$ around the logistic function:
	\begin{IEEEeqnarray*}{lCl}
		p(y=1 | \bm{x}; \bm{\theta}) &=& \sigma(\eta(\bm{x}))  \\
                                    &=& \frac{1}{1 + e^{-\eta(\bm{x})}}
	\end{IEEEeqnarray*}
We can now interpret $p(y=1 | \bm{x}; \bm{\theta})$ as the probability that an object with feature vector
$\bm{x}$ belongs to the positive class. The goal of the algorithm is then to use the training data
to estimate the coefficient vector $\bm{\theta}$. This can be done by finding $\hat{\bm{\theta}}$ that
maximises the log-likelihood function:
    \begin{IEEEeqnarray*}{lCl}
        \hat{\bm{\theta}} &=& \argmax_{\bm{\theta}} \sum_{i = 1}^{n} \log p(y_i| \bm{x}_i; \bm{\theta})
                         - \frac{1}{C} R(\bm{\theta})
    \end{IEEEeqnarray*}
where $n$ is the size of the training set, $R(\bm{\theta})$ is the regularisation term, and $C$ is
the inverse of the regularisation strength. \index{regularisation} A low value of $C$ forces the
values of the parameters to be small, thus avoiding overfitting. However if $C$ is too low, we
might have a hypothesis that is too simple. For the regularisation term, we can either sum up the
absolute values of the coefficients (L1) or the squares of the coefficients (L2):
    \begin{IEEEeqnarray*}{lClllCl}
        R_{L1}(\bm{\theta}) &=& \sum_{i = 1}^{m} \abs{\bm{\theta}_i } &\qquad\qquad
        R_{L2}(\bm{\theta}) &=& \sum_{i = 1}^{m} \bm{\theta}_i^2
    \end{IEEEeqnarray*}
where $m$ is the number of features. One advantage of the L1 regularisation is that it leads to
sparse solutions, where a lot of coefficients become zero \cite{tibshirani96}. This is useful if we
have many features, which for example is the case after we do a polynomial transformation (see
Section \ref{sub:complex}).

There are a few ways to extend the above model to the multi-class setting. One option is
multinomial logistic regression, where we would need to jointly solve a set of $(k-1)$ binary
regressions if we have $k$ classes. In practice, when running the multinomial option in
scikit-learn, \index{scikit-learn} the probability estimates are not very reliable, especially when
we have many classes. The cause of this is unknown, but it is more likely due to flaws in the
scikit-learn implementation than in the actual theory. A more empirically stable alternative is to
use the one-vs-rest \index{one-vs-rest} strategy, where we run $k$ independent binary regressions.
In particular, for class $i$, we pretend that the dataset contains only objects from class $i$ and
class `not $i$'. We then train the binary logistic model on this simplified dataset and we do this
$k$ times, one for each class. At the end, we end up with $k$ probabilities. These can be
interpreted like normal class probabilities after normalisation.

\subsection{Support Vector Machines} \index{support vector machine}
\label{sub:svm}

Support vector machines (SVMs), first introduced by \citeN{boser92}, are another popular family of
algorithms. They have been used in astronomy, for example by \citeN{elting08}, to find non-linear
decision boundaries in the colour space of the SDSS dataset. The idea here is to find a decision
boundary that can maximise the distance between the boundary and the closest data points, which we
call support vectors. This involves finding the weights $\hat{\bm{w}}$ and the bias $\hat{b}$ that
minimise the objective function
    \begin{IEEEeqnarray*}{lCl}
        \argmin_{\bm{w}, b} R(\bm{w}) + C \sum_{i = 1}^{n} \ell(\bm{x}_i)
    \end{IEEEeqnarray*}
where $R(\bm{w})$ can either be L1 or L2 regularisation \index{regularisation} like in logistic
regression, and $\ell(\bm{x}_i)$ is the loss function. \index{loss function} Two common loss functions
are the hinge loss
    \begin{IEEEeqnarray*}{lCl}
        \ell(\bm{x}_i) &=& \max (0, 1 - y_i (\bm{w}^T \bm{x}_i + b))
    \end{IEEEeqnarray*}
and the square of the hinge loss
    \begin{IEEEeqnarray*}{lCl}
        \ell(\bm{x}_i) &=& \max (0, 1 - y_i (\bm{w}^T \bm{x}_i + b))^2
    \end{IEEEeqnarray*}
As usual, the penalty parameter $C$ controls the trade-off between the misclassification of
training examples and the simplicity of the decision surface, with a low value of $C$ resulting in
a simple hypothesis. SVMs, in their original formulation, are inherently binary classifiers.
However we can still use the one-vs-rest \index{one-vs-rest} strategy to extend it to the
multi-class setting. There has even been an attempt to derive an inherently multi-class SVM
\cite{crammer02}.

\subsection{Learning Complex Hypotheses}
\label{sub:complex}

Both SVMs and logistic regression are linear classifiers. When dealing with real-world data like
those in astronomy, we should not expect to be able to separate classes with a hyper-dimensional
plane. If we want them to learn more complex hypotheses, one option is do an explicit polynomial
transformation \index{polynomial transformation} of the original photometric measurements. When we
give the classifier the transformed features, it would still find a linear boundary in the
transformed space. However, the boundary would most of the time be non-linear in the original space.

A second option is to use the kernel trick which does an implicit map into a high-dimensional
feature space. For example, a popular kernel that is often used with SVMs is the radial basis
function \index{RBF kernel} (RBF), which actually maps the inputs into an infinite-dimensional
space. The RBF kernel has the parameter $\gamma$ which is inversely proportional to the radius of
influence of the support vectors. This means that a low value of $\gamma$ corresponds to a smoother
model.

In random forests, we do not have to worry about any transformation. Although in each round, we
slice the data along only one axis, there is no limit on how many slices we can take and how small
the resulting cuboids can be. This allows us to learn arbitrarily complex hypotheses.

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Overview of Active Learning} \index{active learning}
\label{sec:active}

We now turn our attention to the construction of the training set. Getting spectroscopic labels is
expensive. Until now, astronomers do not have a quantitative method to help them choose objects
whose spectroscopic labels would provide them with the most amount of new information. Often, they
simply take a random sample of the sky. This, however, might not always be optimal. To see why,
imagine that there is a group of objects with very similar photometric measurements. We can obtain
spectra from all of them and conclude, for example, that they are all stars. However, a smarter way
is to get only one spectrum from this group for labelling and let the classifier generalise to
other similar objects. Keeping the size of the training set as small as possible while not
sacrificing the classifier accuracy is the goal of active learning.

There are three main types of active learning: membership query synthesis, stream-based selective
sampling, and pool-based active learning. In membership query synthesis\index{membership query
    synthesis}, we are allowed to request labels for any unlabelled instance in the feature space
\cite{angluin88}. Equivalently, we may request the astronomer to find an object with a certain
combination of colours and magnitudes. This is not very realistic since such objects might not even
exist. In stream-based selective sampling\index{stream-based selective sampling}, we sample objects
from the source one at a time, and as objects are streaming in, we must decide to either label or
discard each of them \cite{cohn94}. The assumption here is that it is free to obtain unlabelled
examples, which again is not applicable to astronomy. Thus we shall not discuss membership query
synthesis and stream-based selective sampling further in this thesis.

Instead, we shall focus on pool-based active learning \index{active learning!pool-based}
\cite{lewis94}, the most relevant type of active learning for astronomy. In this setting, we keep
track of two sets. As usual, we have a labelled set $\Labelled$, which can be further partitioned
into a training set $\Labelled_T$ and a test set $\Labelled_S$. There is also an unlabelled set
$\Unlabelled \subset \X$, which contains all the remaining unlabelled examples. The question now is
how to select the next example from $\Unlabelled$ for labelling. In practical terms, where should
we next point the telescope to, in order to obtain a spectrum? To answer this question, we need a
rule $r(\bm{x}; h)$ that can assign a score to each object in $\Unlabelled$, based solely on their
photometric features $\bm{x}$ and the current hypothesis $h$. This score should reflect the amount
of new information we would gain if we were to label the object. Once we have computed $r(\bm{x};
h)$ for all candidates, we can then pick the example with the highest score and obtain its
spectrum. The object's feature vector and its label are then added to the training set and the
classifier is retrained to obtain a new $h$.

Finding an algorithm to compute $r(\bm{x}; h)$ exactly is still an open problem. For now, the best
that we can do is to come up with heuristics that can approximate $r(\bm{x}; h)$. Another problem
is that in practice, the unlabelled pool can be arbitrarily large. For example, there are 800
million unlabelled objects in the SDSS. Thus if we only have a limited computing power, in each
round, we might only be able to assign scores to a subset $\Ecal \subseteq \Unlabelled$ of size
$E$. A formal description of the active learning routine is given in Algorithm \ref{alg:active}. As
we shall see in Section \ref{sec:heuristics}, for some active learning heuristics, we need to
substitute $\argmax$ with $\argmin$ in line 4 of Algorithm \ref{alg:active}.

\algblock[Name]{Start}{End}

\algblockdefx[Forall]{Foreach}{Endforeach}%
			[1]{\textbf{for each} #1 \textbf{do}}%
			{\textbf{end for}}

\begin{algorithm}[h]
	\caption{The general pool-based active learning algorithm}
	\label{alg:active}
	\begin{algorithmic}[1]
		\Procedure {ActiveLearner}{$\Unlabelled$, $\Labelled_T$, $h$, $r$, $n$, $E$}
			\While {$|\Labelled_T| < n$}
				\State $\Ecal$ $\leftarrow$ random sample of size $E$ from $\Unlabelled$
				\State $\bm{x}_* \leftarrow \argmax_{\bm{x} \in \Ecal} r(\bm{x}; h)$
				\State $y_* \leftarrow$ ask the expert to label $\bm{x}_*$
				\State $\Labelled_T \leftarrow \Labelled_T  \cup (\bm{x}_*, y_*)$
				\State $\Unlabelled \leftarrow \Unlabelled \setminus \bm{x}_*$
				\State $h(\bm{x}) \leftarrow$ retrain the classifier
			\EndWhile
			\EndProcedure
	\end{algorithmic}
\end{algorithm}

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Active Learning Heuristics} \index{heuristic}
\label{sec:heuristics}

Many methods have been proposed to rank the informativeness of unlabelled objects. The four
prominent families of heuristics are uncertainty sampling, version space reduction, loss function
minimisation, and classifier certainty. All of these heuristics require the class probabilities
estimated by the current hypothesis. We now discuss each of them in turn, starting with the least
computationally expensive one.

\subsection{Uncertainty Sampling} \index{uncertainty sampling}
\label{sub:uncertainty}

\citeN{lewis94} introduce the idea of uncertainty sampling, where we select the example whose class
membership the classifier is least certain about. These tend to be points that are near the
decision boundary of the classifier. One way to quantify the uncertainty is to calculate the
entropy \cite{shannon48}, which measures the amount of information needed to encode a distribution.
Intuitively, the closer class probabilities of an object are to random guessing, the higher its
entropy will be. This gives us the heuristic of picking the candidate with the highest entropy:
\index{entropy}
	\begin{IEEEeqnarray*}{lCl}
        \bm{x}_*
        &=&  \argmax_{x \in \Ecal} r_S(\bm{x}; h) \\
        &=&  \argmax_{x \in \Ecal} \left\{-\sum_{y \in \Y} p(y | \bm{x}; h)
        \log \big[ p(y | \bm{x}; h) \big] \right\}
    \end{IEEEeqnarray*}
In fact, if we care about how close the class probabilities are to random guessing, there is an
even simpler measure. \shortciteN{scheffer01} define the margin as the difference between the two
highest class probabilities. Since the sum of all probabilities must be 1, the smaller the margin
is, the more uncertain we are about the object's class membership. Thus another heuristic is to pick
the candidate with the smallest margin: \index{margin}
	\begin{IEEEeqnarray*}{lCl}
        \bm{x}_*
        &=& \argmin_{x \in \Ecal} r_M(\bm{x}; h)  \\
		&=& \argmin_{x \in \Ecal} \left\{ \max_{y \in \Y} p(y | \bm{x}; h) -
            \max_{z \in \Y \setminus \{y\}} p(z | \bm{x}; h)  \right\}
	\end{IEEEeqnarray*}

%--------------------------------------------------------------------------------------------------
\subsection{Version Space Reduction} \index{version space reduction}
\label{sub:qbb}

Let us define the version space as the set of all possible hypotheses that are consistent with the
current training set. Instead of focussing on the uncertainty of individual predictions, we could
instead try to constrain the size of the version space, thus allowing the search for the optimal
hypothesis to be more precise. To quantify the size the version space, we can train a committee of
classifiers, $\B = \{h_1, h_2, ..., h_B\}$, and measure the disagreement among the members about an
object's class membership. Each committee member needs to have a hypothesis that is as different
from the others as possible but that is still in the version space \cite{melville04}. In order to
have this diversity, we give each member only a subset of the training examples. Since there might
not be enough training data (for example, in our experiments, we have 11 members but only a maximum
of 300 labelled points), we need to use bootstrapping and select samples with replacement. Hence
this method is often called Query by Bagging (QBB). \index{query by bagging}

One way to measure the level of disagreement is to calculate the margin using the class
probabilities estimated by the committee \cite{melville04}. This looks similar to one of the
uncertainty sampling heuristics, except now we first average out the probabilities of the members
before minimising the margin: \index{margin}
    \begin{IEEEeqnarray*}{lCl}
        \bm{x}_*
        &=& \argmin_{x \in \Ecal} r_{QM}(\bm{x}; \B)  \\
        &=& \argmin_{x \in \Ecal} \left\{ \max_{y \in \Y} p(y | \bm{x}; \B) -
        \max_{z \in \Y \setminus \{y\}} p(z | \bm{x}; \B)  \right\}
    \end{IEEEeqnarray*}
where
	\begin{IEEEeqnarray*}{lCl}
		p(y | \bm{x}; \B) &=& \dfrac{1}{B} \sum_{b=1}^{B} p(y | \bm{x}, h_b)
	\end{IEEEeqnarray*}
In addition to the margin, \citeN{mccallum98} offer an alternative disagreement measure which
involves picking the candidate with the largest expected Kullback--Leibler (KL) divergence
\index{KL divergence} from the average:
	\begin{IEEEeqnarray*}{lCl}
        \bm{x}_*
        &=& \argmax_{x \in \Ecal} r_{QK}(\bm{x}; \B)  \\
		&=& \argmax_{x \in \Ecal} \left\{ \dfrac{1}{B} \sum_{b=1}^B D_{\mathrm{KL}}(p_b\|p_\B) \right\}
	\end{IEEEeqnarray*}
where
	\begin{IEEEeqnarray*}{lCl}
		D_{\mathrm{KL}}(p_b\|p_\B) = \sum_{y \in \Y} p(y | \bm{x}; h_b) \,
		                             \ln\frac{p(y | \bm{x}; h_b)}{p(y | \bm{x}; \B)}
	\end{IEEEeqnarray*}
The KL divergence measures the amount of information lost when $p_\B$ is used to approximate $p_b$.
Intuitively, the larger the KL divergence is, the more disagreement there is between $p_\B$ and
$p_b$. In the active learning context, $p_\B$ is the average prediction probability distribution of
the committee, while $p_b$ is the prediction of a particular committee member $h_b$.

%--------------------------------------------------------------------------------------------------
\subsection{Loss Function Minimisation} \index{loss function}
\label{sub:variance}

The third approach involves minimising a loss function directly, which in turn will minimise the
future generalisation error. A commonly used loss function is the squared loss that has the
following decomposition:
	\begin{IEEEeqnarray*}{lCl}
		\E{\text{Squared Loss}} &=& \text{Bias}^2 + \text{Variance} + \text{Noise}
	\end{IEEEeqnarray*}
Since the noise is intrinsic to the data and represents the expected loss under the optimal
hypothesis, there is nothing we can do about it. The squared bias reflects the error due to the model
class itself. For example, there will be bias if we use a linear hypothesis to learn a non-linear
function. Thus the bias is fixed under the same classifier. However, under certain assumptions
like the consistency of parameter estimates, the variance will vanish as the training set size
approaches infinity. This gives us the heuristic of picking the candidate that would cause the
greatest drop in the variance if we knew its label. Unfortunately, this is a chicken-and-egg problem
since we need to know the labelling information before we can calculate the drop in the variance,
which defeats the purpose of the approach. The next best thing we can do is to pick the candidate
that will result in the lowest expected variance: \index{variance minimisation}
    \begin{IEEEeqnarray*}{lCl}
        \bm{x}_*
        &=& \argmin_{x \in \Ecal} r_V(\bm{x}; h)  \\
        &=& \argmin_{x \in \Ecal}  \E{V(\Labelled_T \cup (\bm{x}, y); h)} \\
        &=& \argmin_{x \in \Ecal} \left\{ \sum_{y \in \Y} p(y | \bm{x}; h)
            V(\Labelled_T \cup (\bm{x}, y); h)  \right\}
    \end{IEEEeqnarray*}
where the expectation is over the class probability distribution under the current hypothesis and
$V(\Labelled_T \cup (\bm{x}, y); h)$ is the variance of the model after $(\bm{x}, y)$ has been
added to the label set $\Labelled_T$. Note that this is quite an expensive computation, since for
us to assign a score to each candidate, we first need to give it each of the possible labels, add
it to the training set to get an updated hypothesis, and calculate the new variance.

In addition, estimating $V(\Labelled_T; h)$ requires a bit of work. In multinomial logistic
regression, we can take the first two terms of the Taylor series expansion of the probability
	\begin{IEEEeqnarray*}{lCl}
		p(y | \bm{x}, \bm{\hat{\theta}}, h)
		&\approx& p(y | \bm{x}; \bm{\theta}, h) + \nabla p(y | \bm{x}; \bm{\theta}, h)^T(\bm{\hat{\theta}} - \bm{\theta})
	\end{IEEEeqnarray*}
where $\bm{\theta}$ and $\bm{\hat{\theta}}$ are the expected and the current estimates of the model
parameters, respectively, and $\nabla p(y | \bm{x}, \bm{\theta}, h)$ is called the gradient vector. Let
$\mathcal{I}$ be the Fisher information matrix \index{Fisher information matrix} and
	\begin{IEEEeqnarray*}{lCl}
		O &=& \sum_{\bm{x} \in \Unlabelled}
		      \sum_{y \in \Y} \nabla p(y | \bm{x}; \bm{\theta}, h) ~ \nabla p(y | \bm{x}; \bm{\theta}, h)^T
	\end{IEEEeqnarray*}
\citeN{schein07} show that
	\begin{IEEEeqnarray*}{lCl}
		V(\Labelled_T; h) &=& \tr(O\mathcal{I}^{-1})
	\end{IEEEeqnarray*}
where the trace \index{trace} function $\tr(X)$ is the sum of the elements along the main diagonal
of the square matrix $X$. Note that the above expression is specific to multinomial logistic
regression. If we use the one-vs-rest strategy with binary logistic regression or another entirely
different classifier like SVMs in our experiments, the same approximation might not hold and we
should not expect to get good results. We leave the variance estimation for other learning
algorithms to future work.

%--------------------------------------------------------------------------------------------------
\subsection{Classifier Certainty} \index{classifier certainty}
\label{sub:cc}

Finally, instead of minimising the variance of $\Unlabelled$, \citeN{mackay91} proposes
minimising the entropy \index{entropy} of the classifier's predictions on $\Unlabelled$:
    \begin{IEEEeqnarray*}{lCl}
		H(\Labelled_T; h) &=& - \sum_{\bm{x} \in \Unlabelled} \sum_{y \in \Y}
							 p(y | \bm{x}; h) \log \big[ p(y | \bm{x}; h)  \big]
	\end{IEEEeqnarray*}
Although this sounds similar to one of the uncertainty sampling heuristics, here instead of picking
the most uncertain candidate, we pick the candidate that is expected to increase the classifier's
prediction certainty by the the greatest amount:
	\begin{IEEEeqnarray*}{lCl}
        \bm{x}_*
        &=& \argmin_{x \in \Ecal} r_{H}(\bm{x}; h) \\
        &=& \argmin_{x \in \Ecal}  \E{H(\Labelled_T \cup (\bm{x}, y); h)} \\
		&=& \argmin_{x \in \Ecal} \left\{ \sum_{y\in \Y} p(y|\bm{x}; h)
             H(\Labelled_T \cup (\bm{x}, y); h) \right\}
	\end{IEEEeqnarray*}
Like the variance minimisation technique, we need to calculate the expectation over the possible classes.
Thus to get the score of just one candidate, we need to retrain the classifier $k$ times, where
$k$ is the number of labels. In practice, both the variance and the classifier certainty
heuristics are too computationally expensive to run.

\subsection{Summary of Heuristics}

\begin{table}[h]
	\caption {Summary of active learning heuristics used in our experiments} \label{tab:heuristics}
	\centering
	\begin{tabular}{lll}
		\toprule
		{Name}  & Notation &  Objective  \\
		\midrule
		Entropy & $r_S(\bm{x}; h)$
			& $\argmax_{x \in \Ecal} \left\{-\sum_{y \in \Y} p(y | \bm{x}; h)
            \log \big[ p(y | \bm{x}; h) \big] \right\}$
			\\[2ex]
		Margin & $r_M(\bm{x}; h)$
			& $\argmin_{x \in \Ecal} \left\{ \max_{y \in \Y} p(y | \bm{x}; h) -
            \max_{z \in \Y \setminus \{y\}} p(z | \bm{x}; h)  \right\}$
			\\[2ex]
		QBB Margin & $r_{QM}(\bm{x}; h)$
			& $\argmin_{x \in \Ecal} \left\{ \max_{y \in \Y} p(y | \bm{x}; \B) -
            \max_{z \in \Y \setminus \{y\}} p(z | \bm{x}; \B)  \right\}$
			\\[2ex]
		QBB KL & $r_{QK}(\bm{x}; h)$
			& $\argmax_{x \in \Ecal} \left\{ \dfrac{1}{B}
               \sum_{b=1}^B D_{\mathrm{KL}}(p_b\|p_\B) \right\}$
			\\[2ex]
		Pool Variance & $r_V(\bm{x}; h)$
			& $\argmin_{x \in \Ecal} \left\{ \sum_{y \in \Y} p(y | \bm{x}; h)
            V(\Labelled_T \cup (\bm{x}, y); h)  \right\}$
			\\[2ex]
		Pool Entropy & $r_{H}(\bm{x}; h)$
			& $\argmin_{x \in \Ecal} \left\{ \sum_{y\in \Y} p(y|\bm{x}; h)
               H(\Labelled_T \cup (\bm{x}, y); h) \right\}  $
			\\
		\bottomrule
	\end{tabular}
\end{table}

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Multi-arm Bandit} \index{bandit}
\label{sec:bandit}

Out of the six heuristics discussed, how do we know which one is the optimal, anyway? There have
been some attempts in the literature to do a theoretical analysis of them. However proofs are
scarce, and when there is one available, they normally only work under simplifying assumptions. For
example, \shortciteN{freund97} show that the query by committee algorithm (a slight variant of our
QBB heuristics) guarantees an exponential decrease in the prediction error with the training size,
but only under certain restrictions such as there is no noise. Thus whether any of these heuristics
is guaranteed to beat random sampling is still an open question. We shall not worry too much about
the theoretical analysis in this thesis. Instead we shall focus on an empirical analysis in the
astronomical domain.

To help us automatically choose the optimal heuristic, we now turn our attention to the multi-armed
bandit problem in probability theory. The colourful name originates from the situation where a
gambler stands in front of a slot machine with $s$ levers. When pulled, each lever gives out a random
reward according to some unknown distribution. The goal of the game is to come up with a strategy
that can maximise the gambler's lifetime rewards while minimising the number of pulls.

Our key novel contribution is the application of this theory to the problem of heuristic selection.
Suppose we have a set of $s$ heuristics $ \R = \{r_1, ..., r_s \}$. Each heuristic has a different
ability to identify the candidate whose labelling information is most valuable. An appropriate
reward is then the incremental increase in the accuracy rate after the candidate is added to the
training set. We assume that the heuristic rewards are independent of each other. This is
reasonable since the theories with which we use to derive the heuristics are mostly unrelated.

Let $\rho_i$ be the reward \index{reward} of heuristic $r_i \in \R$. Observe that even with the
optimal heuristic, there are still two sources of error. First, there could be error during the
labelling process that causes the accuracy rate to decrease. In addition, even without label noise,
the classifier trained on finite data might not be the right one, so we still cannot score
perfectly due to having a poor $h$. Conversely, a bad heuristic might be able to pick an
informative candidate due to pure luck. Thus there is always a certain level of randomness in the
reward received. These errors are probably normally distributed, so
	\begin{IEEEeqnarray*}{lCl}
		(\rho_i \mid \nu_i) \sim \Normal(\nu_i, \tau_i^2)
	\end{IEEEeqnarray*}
and the probably density function (PDF) of the reward is
    \begin{IEEEeqnarray}{lCl}
        f(\rho_i \mid \nu_i)
        &=& \frac{1}{\tau_i \sqrt{2\pi}} \exp\left[ -\frac{(\rho_i - \nu_i)^2}{2 \tau_i^2} \right]
         \label{eqn:rlike}
    \end{IEEEeqnarray}
If we knew both the mean $\nu_i$ and the variance $\tau_i^2$ for all heuristics, the problem would
become trivially easy since we just need to always use the heuristic that has the highest mean
reward. In practice, we do not know $\nu_i$, so let us assume that it follows a normal
distribution
	\begin{IEEEeqnarray*}{lCl}
        \nu_i \sim \Normal(\mu_i, \sigma_i^2)
    \end{IEEEeqnarray*}
Thus the PDF of $\nu_i$ is
    \begin{IEEEeqnarray}{lCl}
        f(\nu_i)
        &=& \frac{1}{\sigma_i \sqrt{2\pi}} \exp\left[ -\frac{(\nu_i - \mu_i)^2}{2 \sigma_i^2} \right]
         \label{eqn:rprior}
    \end{IEEEeqnarray}
To make the problem tractable, assume that $\tau_i^2$ is a known constant. The goal now is to find a
good algorithm that can estimate $\mu_i$ and $\sigma_i^2$.

One problem in multi-arm bandits is the trade-off between exploration and exploitation.
\index{exploration vs exploitation} Suppose we have managed to estimate $\mu_i$ and $\sigma_i^2$
for all $i$, then by always selecting the heuristic with the highest possible mean, or the greedy
heuristic, we would be exploiting our current knowledge. If however we select one of the other
non-greedy heuristics, we would then be exploring with the intention of improving our estimates of
$\mu_i$ and $\sigma_i^2$. There are many instances in which we find our previously held beliefs to
be completely wrong. Thus by always exploiting, we could miss out on the optimal heuristic. On the
other hand, if we explore too much, it might take a long time to reach the desired accuracy and the
strategy ends up being no different from random sampling.

\section{Thompson Sampling} \index{Thompson sampling}
\label{sec:thompson}

There are two main methods in the literature that address this exploration vs exploitation problem.
The algorithm with a strong theoretical guarantee is Upper Confidence Bound \cite{auer02}. We
shall, however, focus on a simpler and much older algorithm called Thompson sampling. First
introduced by \citeN{thompson33}, this algorithm solves the trade-off from a Bayesian perspective.
It has been shown to achieve results that are comparable and sometimes even better than Upper
Confidence Bound \cite{chapelle11}.

Under the heuristic selection setting, Thompson sampling works as follows. We start with a prior
knowledge of $\mu_i$ and $\sigma_i^2$ for all $i$. So long as we do not choose anything stupid,
e.g. a zero variance, our choice of prior should not matter too much in the long run. Since
initially we do not have any information about the performance of each heuristic, the appropriate
prior value for $\mu_i$ is $0$, i.e. there is no evidence (yet) that any of the heuristics offers an
improvement to the accuracy.

In each round, we draw a random sample $\nu_i'$ from the distribution $\Normal(\mu_i, \sigma_i^2)$ for
each $i$ and select heuristic $r_*$ that has the highest sampled value of the mean reward:
    \begin{IEEEeqnarray*}{lCl}
        r_* = \argmax_{i} \nu_i'
    \end{IEEEeqnarray*}
We then use this heuristic to assign scores to the candidates. The object that is deemed to be the
most informative is then added to the training set $\Labelled_T$ and the classifier is retrained.
Next we use the updated hypothesis to predict the labels of objects in the test set $\Labelled_S$.
Let $\delta$ be the reward observed, which is the incremental increase in the accuracy rate on
$\Labelled_S$. We now have a new piece of information that we can use to update our prior belief
about the mean $\mu_*$ and the variance $\sigma_*^2$ of $r_*$. In the Bayesian setting, \index{Bayesian}
(\ref{eqn:rprior}) is called the prior and (\ref{eqn:rlike}) is the likelihood. From Bayes'
theorem, the posterior distribution is proportional to the product of the prior and the likelihood:
    \begin{IEEEeqnarray*}{lCl}
        f(\nu_* \mid \rho_* = \delta)
        &\propto& f(\nu_*) f(\delta \mid \nu_*) \\
        &\propto& \exp \left[ -\frac{1}{2 \sigma_*^2} (\nu_* - \mu_*)^2 \right]
                  \exp \left[ -\frac{1}{2 \tau_*^2} (\delta - \nu_*)^2 \right] \\
        &=& \exp \left[ -\frac{1}{2 \sigma_*^2} (\nu_* - \mu_*)^2
                        -\frac{1}{2 \tau_*^2} (\delta - \nu_*)^2  \right] \\
        &=& \exp \left[ -\frac{\nu_*^2}{2} \left( \frac{1}{\sigma_*^2} + \frac{1}{\tau_*^2} \right)
                        +\nu_* \left( \frac{\mu_*}{\sigma_*^2} + \frac{\delta}{\tau_*^2} \right)
                        - \left( \frac{\mu_*^2}{2\sigma_*^2} + \frac{\delta^2}{2 \tau_*^2} \right)
                        \right] \\
        &\propto& \exp \left[ -\frac{\nu_*^2}{2} \left( \frac{1}{\sigma_*^2} + \frac{1}{\tau_*^2} \right)
                        +\nu_* \left( \frac{\mu_*}{\sigma_*^2} + \frac{\delta}{\tau_*^2} \right)
                        \right] \\
        &=& \exp \left[ -\frac{\nu_*^2}{2} \left( \frac{\sigma_*^2 + \tau_*^2}{\sigma_*^2 \tau_*^2}
                        \right)
                        +\nu_* \left( \frac{\mu_*\tau_*^2 + \delta\sigma_*^2}{\sigma_*^2 \tau_*^2}
                         \right) \right] \\
        &=& \exp \left[ -\frac{1}{2 {\sigma'_*}^2} (\nu^2_* - 2\nu_*\mu_*' ) \right] \\
        &\propto& \exp \left[ -\frac{1}{2 {\sigma'_*}^2} (\nu^2_* - 2\nu_*\mu_*'  + {\mu'_*}^2)
                   \right] \\
        &=& \exp \left[ -\frac{1}{2 {\sigma'_*}^2} (\nu_* - \mu'_*)^2 \right]
    \end{IEEEeqnarray*}
where we have defined
    \begin{IEEEeqnarray*}{lClllCl}
        \mu_*' &=& \frac{\mu_* \tau^2_* + \delta \sigma^2_*}{\sigma^2_* + \tau^2_*} &\qquad\qquad
        {\sigma'_*}^2 &=& \frac{\sigma^2_* \tau^2_*}{\sigma^2_* + \tau^2_*}
    \end{IEEEeqnarray*}
Thus the posterior distribution of the mean reward of $r_*$ remains normal:
    \begin{IEEEeqnarray*}{lCl}
        (\nu_* \mid \rho_* = \delta) \sim \Normal (\mu_*', {\sigma'_*}^2)
    \end{IEEEeqnarray*}
For compactness, let:
    \begin{IEEEeqnarray*}{lClllCl}
        \bm{\nu} &=& (\nu_1, \nu_2, ..., \nu_s) &\qquad\qquad
        \bm{\mu} &=& (\mu_1, \mu_2, ..., \mu_s) \\
        \bm{\sigma}^2 &=& (\sigma^2_1, \sigma^2_2, ..., \sigma^2_s) &\qquad\qquad
        \bm{\tau^2} &=& (\tau^2_1, \tau^2_2, ..., \tau^2_s)
    \end{IEEEeqnarray*}
Algorithm \ref{alg:thompson} on page \pageref{alg:thompson} shows the formal specification of the
Thompson sampling procedure with normally distributed rewards. If we then combine this with the
general pool-based active learning algorithm, we end up with a multi-arm bandit version of active
learning where each arm is a heuristic (Algorithm \ref{alg:bandit}). Note again that in line 8 of
Algorithm \ref{alg:bandit}, some heuristics have $\argmin$ in place of $\argmax$.

\begin{algorithm}[p]
	\caption{Thompson sapmling with normally distributed rewards} \index{Thompson sampling}
	\label{alg:thompson}
	\begin{algorithmic}[1]
		\Procedure {ThompsonSampling}{$\R$, $\bm{\mu}$, $\bm{\sigma}^2$, $\bm{\tau}^2$, n}
    		\Foreach {$t \in \{1, 2, ..., n\}$}
        		\Foreach {$i \in \{1, 2, ..., |\R|\}$}
            		\State $\nu_i' \leftarrow$ draw a sample from $\Normal(\mu_i, \sigma^2_i)$
        		\Endforeach
        		\State $r_* \leftarrow \argmax_{i} \nu_i'$
        		\State Observe reward $\delta$
        		\State $\mu_* \leftarrow \dfrac{\mu_* \tau^2_* + \delta \sigma^2_*}{\sigma^2_* + \tau^2_*}$
        		\State $\sigma_*^2 \leftarrow \dfrac{\sigma^2_* \tau^2_*}{\sigma^2_* + \tau^2_*}$
    		\Endforeach
		\EndProcedure
	\end{algorithmic}
\end{algorithm}

\begin{algorithm}[p]
	\caption{The multi-arm bandit active learning algorithm} \index{bandit}
	\label{alg:bandit}
	\begin{algorithmic}[1]
		\Procedure {ActiveBandit}{$\Unlabelled$, $\Labelled_T$, $h$, $n$, $E$,
			                      $\R$, $\bm{\mu}$, $\bm{\sigma}^2$, $\bm{\tau}^2$}
    		\While {$|\Labelled_T| < n$}
        		\Foreach  {$i \in \{1, 2, ..., |\R|\}$}
        			\State $\nu_i' \leftarrow$ draw a sample from $\Normal(\mu_i, \sigma^2_i)$
        		\Endforeach
        		\State $r_* \leftarrow \argmax_{i} \nu_i'$
        		\State $\Ecal$ $\leftarrow$ random sample of size $E$ from $\Unlabelled$
        		\State $\bm{x}_* \leftarrow \argmax_{\bm{x} \in \Ecal} r_*(\bm{x})$
        		\State $y_* \leftarrow$ ask the expert to label $\bm{x}_*$
        		\State $\Labelled_T \leftarrow \Labelled_T  \cup (\bm{x}_*, y_*)$
        		\State $\Unlabelled \leftarrow \Unlabelled \setminus \bm{x}_*$
        		\State $h(\bm{x}) \leftarrow$ retrain the classifier
        		\State $\delta$ $\leftarrow$ incremental increase in the accuracy
        		\State $\mu_* \leftarrow \dfrac{\mu_* \tau^2_* + \delta \sigma^2_*}{\sigma^2_* + \tau^2_*}$
                \State $\sigma_*^2 \leftarrow \dfrac{\sigma^2_* \tau^2_*}{\sigma^2_* + \tau^2_*}$
    		\EndWhile
		\EndProcedure
	\end{algorithmic}
\end{algorithm}

\pagebreak
How does Thompson sampling balance between exploration and exploitation? If we only wanted to
exploit our current knowledge, then we would naturally select the heuristic that maximises the
immediate reward:
    \begin{IEEEeqnarray*}{lCl}
        r_*
        &=& \argmax_{i} \E{\rho_i} \\
        &=& \argmax_{i} \mathbb{E} \{ \E{\rho_i \given \bm{\nu}} \} \\
        &=& \argmax_{i} \mathbb{E} \{ \E{\rho_i \given \nu_i} \} \\
        &=& \argmax_{i} \int_{-\infty}^{+\infty} \E{\rho_i \given \nu_i} f(\nu_i) \, d\nu_i \\
        &=& \argmax_{i} \int_{-\infty}^{+\infty} \nu_i f(\nu_i) \, d\nu_i \\
        &=& \argmax_{i} \E{\nu_i} \\
        &=& \argmax_{i} \mu_i
    \end{IEEEeqnarray*}
where we have used the law of total expectation and the fact that both $(\rho_i \mid \nu_i)$ and
$\nu_i$ are normally distributed. Unsurprisingly, the heuristic that maximises the immediate reward
is simply the one with the highest $\mu_i$. However, there is a chance that we could be wrong about
our estimate of $\bm{\mu}$. Thus a better option is to only select a heuristic at the same
frequency as the probability that it is optimal. For a given heuristic $r_i$, this probability is
    \begin{IEEEeqnarray*}{lCl}
        \Prob{ \E{\rho_i} = \max_j \E{\rho_j} }
        &=& \E{ \Prob{ \E{\rho_i} = \max_j \E{\rho_j} | \bm{\nu}} } \\
        &=& \int_{\bm{\nu}} \Prob{ \E{\rho_i} = \max_j \E{\rho_j}  \given \bm{\nu} }
             f(\bm{\nu}) \, d\bm{\nu} \\
        &=& \int_{\bm{\nu}} \Prob{ \E{\rho_i \given \bm{\nu}} = \max_j \E{\rho_j \given \bm{\nu}} }
            f(\bm{\nu}) \, d\bm{\nu} \\
        &=& \int_{\bm{\nu}} \Prob{ \nu_i = \max_j \nu_j }
            f(\bm{\nu}) \, d\bm{\nu} \\
        &=& \int_{\bm{\nu}} \mathbb{I}( \nu_i = \max_j \nu_j )
              f(\bm{\nu}) \, d\bm{\nu}
    \end{IEEEeqnarray*}
where $\mathbb{I}$ is the indicator function. In fact we do not have to evaluate this integral
directly. If in each round, we draw a random sample of $\bm{\nu}$ and act optimally according to
the sample values, then over the long run, the frequency that we select each heuristic will
approach its probability of being optimal. This is exactly what Thompson sampling does.

Finally, as we shall see in Chapter \ref{cha:expt2}, the reward function \index{reward!drifting}
dynamically evolves as the training size increases. Intuitively, the accuracy rate can never go
beyond 100\%, so we would expect the incremental change in the accuracy to become smaller over
time. Attempts to address this problem have been made in the literature. For example
\citeN{gupta11} introduce the Dynamic Thompson Sampling method that can adapt to the evolving
parameters faster than Algorithm \ref{alg:thompson} and \ref{alg:bandit}. However, we shall leave
the investigation of this method to future work.

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Posterior Balanced Accuracy} \index{performance measure} \index{MPBA} \index{accuracy}
\label{sec:measures}

Certain astronomical objects are either rarer or more difficult to detect than others. In the SDSS
labelled set, there are 4.5 times as many galaxies as quasars. The problem of class imbalance is
even more severe in the VST ATLAS set, with 43 times more stars than white dwarfs. An easy fix is
to undersample the dominant class when creating training and test sets. This, of course, means that
the size of these sets are limited by the size of the minority class.

When we do not want to alter the underlying class distributions or when larger training and test
sets are desired, we need a performance measure that can correct for the class imbalance.
\shortciteN{brodersen10} show that the posterior balanced accuracy distribution can overcome the
bias in the binary case. We now extend this idea to the multi-class setting.

Suppose we have $k$ classes. For each class $i$ between $1$ and $k$, there are $N_i$ objects in the
universe. Given a hypothesis, we can predict the label of every object and compare our prediction
to the true label. Let $G_i$ be the number of objects in class $i$ that are correctly predicted.
Then we define the recall $A_i$ of class $i$ as \index{recall}
	\begin{IEEEeqnarray*}{lCl}
		A_i &=& \frac{G_i}{N_i}
	\end{IEEEeqnarray*}
The problem is that it is not feasible to get the actual values of $G_i$ and $N_i$ since that would
require us to obtain the true label of every object. Thus we need a method to estimate these
quantities when we only have a sample. Initially we have no information about $G_i$ and
$N_i$, so we can assume that each $A_i$ follows a uniform prior from 0 to 1. This is the same as a
Beta distribution with shape parameters $\alpha = \beta = 1$:
	\begin{IEEEeqnarray*}{lCl}
		A_i &\sim& \Beta(1,1)
	\end{IEEEeqnarray*}
The PDF of $A_i$ is then
    \begin{IEEEeqnarray}{lCl}
        f_{A_i}(a) &=& \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\,
        a^{\alpha-1}(1-a)^{\beta-1} \label{eqn:prior} \\
        &\propto&   a^{1-1}(1-a)^{1-1}  \notag
    \end{IEEEeqnarray}
where $\Gamma(\alpha)$ is the gamma function.

After we have trained the classifier, suppose we have a test set containing $n_i$ objects in class
$i$. Running the classifier on this test set is the same as conducting $k$ binomial experiments,
where, in the $i$th experiment, the sample size is $n_i$ and the probability of success is simply
$A_i$. Let $g_i$ be the number of correctly labelled objects belonging to class $i$ in the test
set. Then, conditional on the accuracy rate, $g_i$ follows a binomial distribution:
	\begin{IEEEeqnarray*}{lCl}
		(g_i \mid A_i) &\sim& \Bin(n_i, A_i)
	\end{IEEEeqnarray*}
The probability mass function of $(g_i \mid A_i = a)$ is thus
    \begin{IEEEeqnarray}{lCl}
        p_{g_i \mid A_i}(g_i) &=& \binom{n_i}{g_i} a^{g_i} (1 - a)^{n_i - g_i} \label{eqn:likelihood} \\
                              &\propto& a^{g_i} (1 - a)^{n_i - g_i} \notag
    \end{IEEEeqnarray}
In the Bayesian \index{Bayesian} setting, \eqref{eqn:prior} is the prior and \eqref{eqn:likelihood}
is the likelihood. To get the posterior PDF, we simply multiply the prior with the likelihood:
	\begin{IEEEeqnarray*}{lCl}
		f_{A_i \mid \bm{g}}(a)
		&\propto& f_{A_i}(a) \times f_{g_i \mid A_i}(g_i) \\
		&\propto& a^{1-1}(1-a)^{1-1} \times a^{g_i} (1 - a)^{n_i - g_i} \\
		&=& a^{1 + g_i - 1}(1-a)^{1 + n_i - g_i - 1}
	\end{IEEEeqnarray*}
Thus, with respect to the binomial likelihood function,
the Beta distribution is conjugate to itself. The posterior recall rate $A_i$
also follows a Beta distribution, now with parameters
	\begin{IEEEeqnarray*}{lCl}
		(A_i \mid g_i) &\sim& \Beta(1 + g_i, 1 + n_i - g_i)
	\end{IEEEeqnarray*}
Our goal is to have a balanced accuracy rate, $A$, that puts an equal weight in each class. One way
to achieve this is to take the average of the individual recalls:
	\begin{IEEEeqnarray*}{lCl}
		A &=& \frac{1}{k} \sum_{i=1}^k A_i \\
		&=& \frac{1}{k} A_T
	\end{IEEEeqnarray*}
Here we have defined $A_T$ to be the sum of the individual recalls. We call  $(A \mid \bm{g})$ the
posterior balanced accuracy (PBA), where $\bm{g} =(g_1,...,g_k)$. Most of the time, we simply want
to calculate its expected value:
	\begin{IEEEeqnarray*}{lCl}
		\E{A \given \bm{g}} &=& \frac{1}{k} \, \E{A_T \given \bm{g}} \\
		&=& \frac{1}{k} \int a \cdot f_{A_T \mid \bm{g}}(a) \, da
	\end{IEEEeqnarray*}
Let us call this the mean posterior balanced accuracy (MPBA). Note that there is no closed
form solution for the PDF $f_{A_T \mid \bm{g}}(a)$. However assuming that $A_T$ is a sum of $k$
independent Beta random variables, $f_{A_T \mid \bm{g}}(a)$ can be approximated by numerically
convolving $k$ Beta distributions. The independence assumption is reasonable here, since there
should be little to no correlation between the individual class accuracy rates. Knowing that a
classifier is really good at recognising stars does not tell us much about how well that classifier
can recognise galaxies.

Having the knowledge of $f_{A \mid \bm{g}}(a)$ will allow us to make violin plots,
construct confidence intervals and do hypothesis tests. To get an expression for this,
let us first rewrite the cumulative distribution function (CDF) as
	\begin{IEEEeqnarray*}{lCl}
		F_{A\mid \bm{g}}(a) &=& \Prob{A \leq a \mid \bm{g}} \\
		&=& \Prob[\Big]{\frac{1}{k} A_T \leq a \given \bm{g}} \\
		&=& \Prob{A_T \leq ka \given \bm{g}} \\
		&=& F_{A_T \mid \bm{g}}(ka) \IEEEyesnumber \label{eqn:CDF}
	\end{IEEEeqnarray*}
Differentiating \eqref{eqn:CDF} with respect to $a$, we obtain the PDF of $(A \mid \bm{g})$:
	\begin{IEEEeqnarray*}{lCl}
		f_{A \mid \bm{g}}(a) &=& \frac{\partial}{\partial a} F_{A \mid \bm{g}}(ka) \\
		&=& \frac{\partial}{\partial a} (ka) \cdot \frac{\partial}{\partial ka} F_{A_T \mid \bm{g}}(ka) \\
		&=& k \cdot f_{A_T \mid \bm{g}}(ka)
	\end{IEEEeqnarray*}
We shall use the posterior balanced accuracy rate throughout the experiments to report the overall
performance of a classifier on a test set.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "thesis"
%%% End:


\begin{algorithm}[H]
    \caption{Feature Kernel}\label{algo:feature-kernel}
    \begin{algorithmic}[1]
        \REQUIRE $inst: \text{MIP instance},\ config: \text{Configuration}$
        \STATE {$solutions = build\_solutions^{\ref{algo:build-solutions}}(inst,\ config);$} \label{fk:line:init}
        \STATE {$dataset = generate\_data\_set(solutions);$}
        \STATE {$importance = generate\_feature\_importance(dataset); $} \label{fk:line:feature-importance}
        \STATE {$kernel,\ bucket\_vars = build\_kernel(solutions,\ importance,\ config);$} \label{fk:line:build-kernel}
        \RETURN {$ (kernl,\ bucket\_vars); $}
    \end{algorithmic}
\end{algorithm}

\paragraph{Description} \Cref{algo:feature-kernel} shows an high-level description of the method. On line \ref{fk:line:init}
\Cref{algo:feature-kernel} builds the initial solution set, see \Cref{algo:build-solutions} for the details. What meters here is
that there is a list of sub models, with relative variable values and feasibility status if the model was at least continuous feasible. This
set can be used to generate the dataset, train the Random Forrest and build the feature importance vector (on line \ref{fk:line:feature-importance}).
The feature importance vector is then used to split variables into  kernel variables and bucket variables. Function $build\_kernel$, on line \ref{fk:line:build-kernel},
builds the initial kernel but does not build the initial buckets, this is done in a separate section of the program. \Cref{algo:feature-kernel} simply implements a
new Kernel builder, so it can be used in substitution of the default Kernel builder.


\section{\namee by Examples}
\label{nested:sec:overview}

This section illustrates \namee with an encoding of a family polymorphism
solution to the expression problem, and informally presents its salient
features.

%-------------------------------------------------------------------------------
\subsection{The Expression Problem, \namee Style}

The \namee calculus allows us to solve the expression problem in a way that is
very similar to \citeauthor{Ernst_2001}'s \textsf{gbeta} solution in \cref{sec:ernst}.
However, the underlying mechanisms of \namee are quite different from those of
\textsf{gbeta}. In particular, \namee features a structural type system in which we can
model objects with records, and object types with record types. For instance, we
model the interface of \lstinline{Lang.Exp} with the singleton record type
\lstinline${ print : String }$. For the sake of conciseness, we use \lstinline{type} aliases
to abbreviate types.
\lstinputlisting[linerange=4-4]{./examples/overview.sl}% APPLY:linerange=PRINT_INTERFACE
Similarly, we capture the interface of the \lstinline{Lang} family in a record,
with one field for each case's constructor.
\lstinputlisting[linerange=8-11]{./examples/overview.sl}% APPLY:linerange=LANG_FAMILY
Here is the implementation of \lstinline{Lang}.
\lstinputlisting[linerange=20-27]{./examples/overview.sl}% APPLY:linerange=LANG_IMPL
We assume several primitive types: fixed width integers \lstinline{Int},
\lstinline{Double} for numeric operations and \lstinline{String} for text
manipulation. A \namee program consists of a collection of definitions and
declarations, separated by semicolon \lstinline{;}.

% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
\paragraph{Adding evaluation.}
We obtain \lstinline{IPrint & IEval}, which is the corresponding type for \lstinline{LangEval.Exp}, by
intersecting \lstinline{IPrint} with \lstinline{IEval} where
\lstinputlisting[linerange=32-32]{./examples/overview.sl}% APPLY:linerange=EVAL_INTERFACE
The type for \lstinline{LangEval} is then
\lstinputlisting[linerange=37-40]{./examples/overview.sl}% APPLY:linerange=EVAL_PRINT_INTERFACE
We obtain an implementation for \lstinline{LangEval} by merging the existing
\lstinline{Lang} implementation \lstinline{implLang} with the new evaluation
functionality \lstinline{implEval} using the merge operator \lstinline{,,}.
\lstinputlisting[linerange=48-56]{./examples/overview.sl}% APPLY:linerange=EVAL_PRINT_IMPL

% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
\paragraph{Adding negation.}
Adding negation to \lstinline{Lang} works similarly.
\lstinputlisting[linerange=60-68]{./examples/overview.sl}% APPLY:linerange=LANG_NEG
% \begin{Verbatim}[xleftmargin=10mm,fontsize=\relscale{.80}]
% type LangNeg = Lang & { neg : IPrint -> IPrint }

% implLangNeg : LangNeg
% implLangNeg = implLang ,, implNeg

% implNeg = { neg = \a.{print = "-" ++ a.print } }
% \end{Verbatim}

% - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
\paragraph{Putting everything together.}
Finally, we can combine the two extensions and provide the missing
implementation of evaluation for the negation case.
\lstinputlisting[linerange=73-83]{./examples/overview.sl}% APPLY:linerange=LANG_FINAL
We can test \lstinline{implLangNegEval} by creating an object that represents $-2 + 3$, which is able to print and evaluate at the same time.
\lstinputlisting[linerange=101-103]{./examples/overview.sl}% APPLY:linerange=TEST

%- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
\paragraph{Multi-field records.}
Recall that in \cref{bg:sec:intersection}, we show how to model multi-field records by
single-field records. Thus \namee does not have multi-field record types built in.
They are merely syntactic sugar for intersections of single-field record types.
Hence, the following is an equivalent definition of \lstinline{Lang}:
\lstinputlisting[linerange=16-16]{./examples/overview.sl}% APPLY:linerange=LANG_FAMILY2
Similarly, the multi-field record expression in the definition of
\lstinline{implLang} is syntactic sugar for the explicit merge of two
single-field records.
\begin{lstlisting}
implLang : Lang = { lit = ... } ,, { add = ... };
\end{lstlisting}

%- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
\paragraph{Subtyping.}
A distinctive difference compared to \textsf{gbeta} is that many more \namee types are related through
subtyping. Indeed, \textsf{gbeta} is unnecessarily conservative~\citep{ernst_hoh}: none of the families is related
through subtyping, nor is any of the class members of one family related to any
of the class members in another family. For instance, \lstinline{LangEval} is
not a subtype of \lstinline{Lang}, nor is \lstinline{LangNeg.Lit} a subtype
of \lstinline{Lang.Lit}.

In contrast, subtyping in \namee is much more nuanced and depends entirely on the
structure of types. The primary source of subtyping are intersection types:
any intersection type is a subtype of its components. For instance,
\lstinline{IPrint & IEval} is a subtype of both \lstinline{IPrint} and
\lstinline{IEval}. Similarly \lstinline{LangNeg = Lang & NegPrint} is a subtype
of \lstinline{Lang}. Compare this to \textsf{gbeta} where \lstinline{LangEval.Expr} is
not a subtype of \lstinline{Lang.Expr}, nor is the family \lstinline{LangNeg} a
subtype of the family \lstinline{Lang}.

However, \textsf{gbeta} and \namee agree that \lstinline{LangEval} is not a subtype of
\lstinline{Lang}. The \namee-side of this may seem contradictory at first, as we
have seen that intersection types arise from the use of the merge operator.
We have created an implementation for \lstinline{LangEval} with
\lstinline{implLang ,, implEval} where \lstinline{implLang} has type \lstinline{Lang}, which
suggests that \lstinline{LangEval} is a subtype of \lstinline{Lang}.
Yet, there is a flaw in our reasoning:
strictly speaking, \lstinline{implLang ,, implEval} is not of
type \lstinline{LangEval} but instead of type \lstinline{Lang & EvalExt}, where
\lstinline{EvalExt} is the type of \lstinline{implEval}:
\lstinputlisting[linerange=44-44]{./examples/overview.sl}% APPLY:linerange=EVAL_INTERFACE2

Nevertheless, the definition of \lstinline{implLangEval} is valid because
\lstinline{Lang & EvalExt} is a subtype of \lstinline{LangEval}.
Indeed, if we consider for the sake of simplicity only the \lstinline{lit}
field, we have that \lstinline{(Int -> IPrint) & (Int -> IEval)} is a
subtype of \lstinline{Int -> IPrint & IEval}. This follows from a standard
subtyping axiom for distributivity of functions and intersections in the BCD system inherited by \namee.
In conclusion, \lstinline{Lang & EvalExt} is a subtype of both \lstinline{Lang}
and of \lstinline{LangEval}. However, neither of the latter two types is a subtype of the other.
Indeed, \lstinline{LangEval} is not a subtype of \lstinline{Lang} as the type
of \lstinline{add} is not covariantly refined and thus admitting the subtyping
is unsound. For the same reason \lstinline{Lang} is not a subtype of \lstinline{LangEval}.

A summary of the various relationships between the language components is shown
in \cref{fig:diagram}. Admittedly, the figure looks quite complex because our
calculus has a structural type system (as often more foundational calculi
do) where more types are related through subtyping, whereas mainstream object-oriented
languages have nominal type systems.

\begin{figure}[t]
  \centering
\includegraphics[scale=0.75]{figures/diagram.eps}
\caption{Summary of the relationships between language components}
\label{fig:diagram}
\end{figure}

\paragraph{Stand-alone extensions.}
Unlike in \textsf{gbeta} and other class-based inheritance systems, in \namee
the extension \lstinline{implEval} is not tied to \lstinline{LangEval}. In that
sense, it resembles trait and mixin systems that can apply the same extension
to different classes. However, unlike those systems, \lstinline{implEval} can also
exist as a value on its own, i.e., it is not an extension per se.

%-------------------------------------------------------------------------------
% \subsection{Disjoint Intersection Types and Ambiguity}

% The above example shows that intersection types and the merge operator
% are closely related to multiple
% inheritance. Indeed, they share a major concern with multiple inheritance,
% namely ambiguity. When a subclass inherits an implementation of the same
% method from two different parent classes, it is unclear which of the two
% methods is to be adopted by the subclass. In the case where the two parent classes
% have a common superclass, this is known as the \emph{diamond problem}.
% The ambiguity problem also appears in \namee,
% e.g., if we merge two numbers to obtain $\mer{1}{2}$ of type
% $\inter{\mathsf{Int}}{\mathsf{Int}}$. Is the result of $\mer{1}{2} + 3$
% either $4$ or $5$?

% Disjoint intersection types offer to statically detect potential ambiguity and
% to ask the programmer to explicitly resolve the ambiguity by rejecting the
% program in its ambiguous form. In the previous work on \oname, ambiguity is
% avoided by dictating that all intersection types have to be disjoint, i.e.,
% $\inter{\mathsf{Int}}{\mathsf{Int}}$ is ill-formed because the first component
% has the same type as the second.

% Disjoint intersection types ensure unambiguity and conflicts are
% statically detected and manually resolved by programmers. This
% is similar to the trait model.

% Local Variables:
% TeX-master: "../../Thesis"
% org-ref-default-bibliography: ../../Thesis.bib
% End:


\problemname{Crafting Mines}
\illustration{.5}{anvil}{Wikimedia, cc-by-sa}

\noindent You have been contacted by your server's admin, who informs you that he's running out of resources and would
like some assistance in gathering more Stone to be able to build Westeros in Minecraft or something. You didn't really
listen to what he said - you were too excited because you love mining! It's your dream to be deep in a mine, mining in
the dark for hours on end.\\

\noindent Just like in our own world, ore is quite difficult to obtain. You have already picked a few spots where you
dug into the ground - these are the mines you want to expand! However, expanding mines isn't easy. Currently the only
way to access your mining pits is to walk above ground and enter the mining pit from above. As we all know the night
is dark and full of creepers, so it will be easier to traverse them underground on future expeditions. To connect mines
together, you need to dig a tunnel between them. This is easier said than done because in this world there are three
types of rocks: \textbf{Stone}, \textbf{Obsidian} and \textbf{Bedrock}. Bedrock is impossible to mine through and should
be ignored, and Obsidian is more difficult to mine through than Stone so you would like minimize the number of Obsidian
patches you mine through, making Stone the most optimal choice.\\

\noindent Your objective on this expedition is to maximize the rock mined on your expedition but above all else you will
also want to minimize the number of Obsidian paths you must choose - these paths severely damage your equipment. You may
assume that there is always a way to connect the mines, and that any paths between mines that are not specified are pure
Bedrock.

\section*{Input}
Input will consit of $1 \leq N \leq 100$, the number of mining pits, $1 \leq M \leq 10000$, the number of possible paths
between your mining pits. Following this will be $M$ lines, each with four integers $a,b,c,d$. Each line signifies a
possible (bidirectonal) link between mining pit a and b, containing a total of c rocks. d will be 1 if the path is made
out of Obsidian, and 0 otherwise.

\section*{Output}
Your output should be two integers, one specifying the minimum amount of Obsidian paths required to connect these mines,
and two being the amount of Stone mined on this expedition.