  \documentclass{mynotes}

%\geometry{showframe}% for debugging purposes -- displays the margins

\newcommand{\E}{\mbox{E}}

\usepackage{amsmath}
%\usepackage[garamond]{mathdesign}
\usepackage{url}

% Set up the images/graphics package
\usepackage{graphicx}
\setkeys{Gin}{width=\linewidth,totalheight=\textheight,keepaspectratio}
\graphicspath{{graphics/}}

\title[Lesson 1 $\cdot$ SDS 383D]{Exercises 2: Generalized linear models}
%\author[ ]{ }
\date{}  % if the \date{} command is left out, the current date will be used

% The following package mwakes prettier tables.  We're all about the bling!
\usepackage{booktabs}

% The units package provides nice, non-stacked fractions and better spacing
% for units.
\usepackage{units}

% The fancyvrb package lets us customize the formatting of verbatim
% environments.  We use a slightly smaller font.
\usepackage{fancyvrb}
\fvset{fontsize=\normalsize}

% Small sections of multiple columns
\usepackage{multicol}

% Provides paragraphs of dummy text
\usepackage{lipsum}

% These commands are used to pretty-print LaTeX commands
\newcommand{\doccmd}[1]{\texttt{\textbackslash#1}}% command name -- adds backslash automatically
\newcommand{\docopt}[1]{\ensuremath{\langle}\textrm{\textit{#1}}\ensuremath{\rangle}}% optional command argument
\newcommand{\docarg}[1]{\textrm{\textit{#1}}}% (required) command argument
\newenvironment{docspec}{\begin{quote}\noindent}{\end{quote}}% command specification environment
\newcommand{\docenv}[1]{\textsf{#1}}% environment name
\newcommand{\docpkg}[1]{\texttt{#1}}% package name
\newcommand{\doccls}[1]{\texttt{#1}}% document class name
\newcommand{\docclsopt}[1]{\texttt{#1}}% document class option name

\newcommand{\N}{\mbox{N}}
\newcommand{\thetahat}{\hat{\theta}}
\newcommand{\sigmahat}{\hat{\sigma}}
\newcommand{\betahat}{\hat{\beta}}

\begin{document}

\maketitle% this prints the handout title, author, and date

\section{Exponential families}

We say that a distribution $f(y; \theta, \phi)$ is in an exponential family if we can write its PDF or PMF in the form
$$
f(y; \theta, \phi) = \exp \left\{ \frac{y \theta - b(\theta)}{a(\phi)} + c(y; \phi)   \right \}
$$
for some known functions $a$, $b$ and $c$.  We refer to $\theta$ as the canonical parameter of the family, and  (for reasons that will become clear) to $\phi$ as the dispersion parameter.

\begin{enumerate}[(A)]

\item Starting from the ``standard'' form of each PDF/PMF, show that the following distributions are in an exponential family, and find the corresponding $b$, $c$, $\theta$, and $a(\phi)$.

\begin{itemize}
\item $Y \sim N(\mu, \sigma^2)$ for known $\sigma^2$.
\item $Y = Z/N$ where $Z \sim \mbox{Binom}(N, P)$ for known $N$.
\item $Y \sim \mbox{Poisson}(\lambda)$
\end{itemize}

\item We want to characterize the mean and variance of a distribution in the exponential family.  To do this, we'll take an unfamiliar route, involving a preliminary lemma (that holds much more generally than just the exponential family).  Define the \emph{score} $s(\theta)$ as the gradient of the log likelihood with respect to the parameter of interest:
$$
s(\theta) = \frac{\partial}{\partial \theta} \log L(\theta) \, , \quad L(\theta) = \sum_{i=1}^n f(y_i; \theta) \, .
$$
We've written this in multivariate form for the sake of generality, but of course it just involves an ordinary partial derivative (w.r.t.~$\theta$) in the case where $\theta$ is one-dimensional.
%Let's also define $H(\theta)$ as the Hessian matrix, i.e.~the matrix of second partial derivatives of the log likelihood:
%$$
%H(\theta) = \frac{\partial}{\partial \theta^T} s(\theta) =   \frac{\partial^2}{ \partial \theta \partial \theta^T} \log L(\theta)
%$$

While we think of the score as a function of $\theta$, clearly (just like the likelihood) the score also depends on the data.  So a natural question is: what can we say about the \emph{distribution} of the score over different random realizations of the data under the true data-generating process, i.e.~at the true $\theta$?  It turns out we can say the following, sometimes referred to as the score equations:
$$
\begin{aligned}
E\{ s(\theta) \} &= 0 \\
\mathcal{I}(\theta) \equiv \mbox{var} \{ s(\theta) \}  &= - E \left\{ H(\theta) \right\}
\end{aligned}
$$
where the mean and variance are taken under the true $\theta$.  \textbf{Prove the score equations.}  Hints: prove the first equation first.  You can assume that it's OK to switch the order of differentiation and integration (i.e.~that any necessary technical conditions are met).  To prove the second equation, differentiate both sides of the first equation with respect to $\theta^T$ and switch the order of differentiation and integration again.  Expand out and simplify.

\item Use the score equations you just proved to show that, if $Y \sim f(y; \theta, \phi)$ is in an exponential family, then
$$
\begin{aligned}
E(Y) &=  b'(\theta) \\
\mbox{var} ( Y ) &= a(\phi) b''(\theta)
\end{aligned}
$$

Thus the variance of $Y$ is a product of two terms.  One of these terms, $b''(\theta)$, depends only on the canonical parameter $\theta$, and hence on the mean, since you showed that $E(Y) =  b'(\theta)$.  The other, $a(\phi)$, is independent of $\theta$.  Note that the most common form of $a$ is $a(\phi) = \phi/w$, where $\phi$ is called a dispersion parameter and where $w$ is a known prior weight that can vary from one observation to another; we'll see this below.

\item To convince yourself that your result in $(C)$ is correct, use these results to compute the mean and variance of the $N(\mu, \sigma^2)$ distribution.

\end{enumerate}

\section{Generalized linear models}

Suppose we observe data like in the typical regression setting: that is, pairs $\{y_i, x_i\}$ where $y_i$ is a scalar response for case $i$, and $x_i$ is a $p$-vector of predictors or features for that same case $i$.  We say that the $y_i$'s follow a \textit{generalized linear model} (GLM) if two conditions are met.  First, the PDF (or PMF, if discrete) can be written as:
$$
f(y_i; \theta_i,  \phi)  = \exp \left\{ \frac{y_i \theta_i - b(\theta_i)}{\phi/w_i} + c(y_i; \phi/w_i)   \right \}
$$
where the weights $w_i$ are all known.  This is referred to as the stochastic or random component of the model.  Second, for some known invertible function $g$ we have
$$
g(\mu_i) = x_i^T \beta
$$
where $\mu_i = E(Y_i; \theta_i, \phi)$.  This is the systematic component of the model, and $g$ is referred to as a link function, since it links the mean of the response $\mu_i$ with the \emph{linear predictor} $\eta_i = x_i^T \beta$.

\begin{enumerate}[(A)]

\item Deduce from your results above that, in a GLM,
 $$
\begin{aligned}
\theta_i &= (b')^{-1} \Big( g^{-1}(x_i^T \beta) \Big) \\
\mbox{var} (Y_i ) &= \frac{\phi}{w_i} V(\mu_i)
\end{aligned}
$$
for some function $V$ that you should specify in terms of the building blocks of the exponential family model.  $V$ is often referred to as a the \emph{variance function}, since it explicitly relates the mean and the variance in a GLM.

\item Take two special cases.
\begin{enumerate}[(1)]
\item Suppose that $Y$ is a Poisson GLM, i.e. that the stochastic component of the model is a Poisson distribution.  Show that $V(\mu) = \mu$.
\item Suppose that $Y = Z/N$ is a Binomial GLM, i.e. that the stochastic component of the model is a Binomial distribution $Z \sim \mbox{Binom}(N, P)$ and that $Y$ is the fraction of yes outcomes (1's).  Show that $V(\mu) = \mu(1-\mu)$.
\end{enumerate}

\item To specify a GLM we must choose the link function $g(\mu_i)$.  Recall that $g$ links the predictors with the mean of the response: $g(\mu_i) = x_i^T \beta$.  Since you've shown that
$$
\theta_i = (b')^{-1} \left\{g^{-1}(x_i^T \beta) \right\} \, ,
$$
a ``simple'' choice of link function is one where $g^{-1} = b'$, or equivalently $g(\mu) = (b')^{-1}(\mu)$.  This is known as the \textit{canonical link}, in which case the canonical parameter simplifies to
$$
\theta_i = (b')^{-1} \left\{b'(x_i^T \beta) \right\} = x_i^T \beta \, .
$$
So under the canonical link $g(\mu) = b'^{-1}(\mu)$, we have the model
$$
f(y_i; \beta, \phi) \exp \left\{ \frac{y_i x_i^T \beta - b(x_i^T \beta)}{\phi/w_i} + c(y_i; \phi/w_i)   \right \}
$$

Now return to the two special cases from the previous problem.
\begin{enumerate}[(1)]
\item Suppose that $Y$ is a Poisson GLM, i.e. that the stochastic component of the model is a Poisson distribution. Show that the canonical link is the log link, $g(\mu) = \log \mu$.
\item Suppose that $Y = Z/N$ is a Binomial GLM, i.e. that the stochastic component of the model is a Binomial distribution $Z \sim \mbox{Binom}(N, P)$.   Show that the canonical link is the logistic link $g(\mu) = \log \left\{ \mu/(1-\mu) \right\}$.
\end{enumerate}

\end{enumerate}

\section{Fitting GLMs}

 The regression coefficients $\beta$ in a GLM are typically fit using some variation on likelihood-based inference.  To this end, define the likelihood function for a given GLM as
$$
L(\beta, \phi) = \prod_{i=1}^n \exp \left\{ \frac{y_i \theta_i - b(\theta_i)}{\phi/w_i} + c(y_i; \phi/w_i)   \right \} \, ,
$$
where based on results you proved above, we define $\theta_i = (b')^{-1}(\mu_i)$ and $\mu_i = g^{-1}(x_i^T \beta)$.  This allows us to define the score function $s(\beta, \phi)$ as the gradient of the log likelihood with respect to $\beta$:
$$
s(\beta, \phi) = \nabla_\beta \  \log L(\beta, \phi) = \frac{\partial}{\partial \beta} \log L(\beta, \phi) \, .
$$

\begin{enumerate}[(A)]

\item Using the chain rule
$$
\frac{\partial}{\partial \beta} = \frac{\partial}{\partial \theta} \times \frac{\partial \theta}{\partial \mu } \times \frac{\partial \mu}{\partial \beta} \, ,
$$
show that
$$
s(\beta, \phi) \equiv \nabla_\beta \  \log L(\beta, \phi)  = \sum_{i=1}^n \frac{w_i(Y_i - \mu_i) x_i}{ \phi V(\mu_i) g'(\mu_i)}
$$
where $x_i$ is the vector of predictors for case $i$ (i.e.~row $i$ of the predictor matrix $X$, transposed to be a column vector).

\item Show that under the canonical link, $g'(\mu) = 1/V(\mu)$, so that the score function simplifies to:
$$
s(\beta, \phi) = \sum_{i=1}^n \frac{w_i(Y_i - \mu_i) x_i}{ \phi} \, .
$$

Hint: remember from calculus that
$$
(g^{-1})'(x) = \frac{1}{g'\left\{ g^{-1}(x) \right\}}
$$

\item Let's take the specific case of a GLM for a binomial outcome, where $Y_i \sim \mbox{Binom}(N_i, \mu_i)$ for known sample size $N_i$, $Y_i = Z_i/N_i$ is the observed success fraction, and where $\mu_i$ is related to the predictors $x_i \in \mathcal{R}^p$ via the (canonical) logistic link.  This is called the logistic regression model.  But how should we fit the parameters?

Read up on the method of steepest descent, i.e.~gradient descent\footnote{if you want a textbook reference, see \textit{Numerical optimization}, by Nocedal and Wright. This should be available in electronic form through from the UT Library website.}  Write your own function that will fit a logistic regression model by gradient descent.  For extra coding brownie points, try to maintain some level of generality to your code, i.e. so that it could also work with different GLMs, assuming your wrote different sub-routines.

Grab the data ``wdbc.csv'' from the course website, or obtain some other real data that interests you, and test out your fitter.  The WDBC file has information on 569 breast-cancer patients from a study done in Wisconsin.  The first column is a patient ID, the second column is a classification of a breast cell (Malignant or Benign), and the next 30 columns are measurements computed from a digitized image of the cell nucleus.  These are things like radius, smoothness, etc.  For this problem, use the first 10 features for $X$, i.e.~columns 3-12 of the file.  (If you use all 30 features you'll run into trouble.)

Some notes:
\begin{itemize}
\item We're trying to maximize the log likelihood function, but the convention in the optimization literature is to minimize things.  No big deal; what we're doing is the same as \textit{minimizing} the negative of the log likelihood.
\item You need to add an intercept term, and the simplest way is to add a column of 1's as the first column of the feature matrix $X$.  (If you've never seen this trick before, convince yourself why it makes sense.)
\item Make sure that, at every iteration of gradient descent, you compute and store the current value of the log likelihood, so that you can track and plot the convergence of the algorithm.
\item Be sensitive to the numerical consequences of an estimated success probability that is either very near 0, or very near 1.
\item Finally, you can be as clever as you want about the gradient-descent step size.  Small step sizes will be more robust but slower; larger step sizes can be faster but may overshoot and diverge; step sizes based on line search (Chapter 3 of Nocedal and Wright) are cool but involve some extra, optional work.
\end{itemize}

\item Consider the Hessian matrix, i.e. the matrix of partial second derivatives of the log likelihood:
$$
H(\beta, \phi) = \frac{\partial^2}{\partial \beta \partial \beta^T} \log L(\beta, \phi)
$$
Give an expression for the Hessian matrix $H(\beta, \phi)$ of a GLM that is as simple as possible, ideally in matrix form.   Note: to keep things a little more streamlined, please assume the canonical link function here; it's the same idea, just with hairier algebra, under an arbitrary link function.

\item Now consider a point $\beta_0 \in \mathcal{R}^P$, which serves as an intermediate guess for our vector of regression coefficients.  Show that, for any GLM, the second-order Taylor approximation of $\log L(\beta, \phi)$, around the point $\beta_0$, can be expressed in the form
$$
q(\beta; \beta_0) = -\frac{1}{2}(\tilde{y} - X \beta)^T W (\tilde{y} - X \beta) + c\, ,
$$
where $\tilde{y}$ is a vector of ``working responses'' and $W$ is a diagonal matrix of ``working weights,'' and $c$ is a constant that doesn't involve $\beta$.  Give explicit expressions for the diagonal elements $W_{ii}$ and for $\tilde{y}$ (which will necessarily involve the point $\beta_0$, around which you're doing the expansion).\footnote{Remember the trick of completing the square, e.g.~\url{https://davidrosenberg.github.io/mlcourse/Notes/completing-the-square.pdf}.} Again, we're assuming the canonical link to make the algebra a bit simpler.

\item  Read up on Newton's method for optimizing smooth functions (e.g. in Nocedal and Wright, Chapter 2).  Implement it for the logistic regression model model and test it out on the same data set you just used to test out gradient descent.\footnote{Hey, cool!  You should be able to use your own solver for weighted least squares that you wrote for the first set of problems.}  Note: while you could do line search, there is a ``natural'' step size of 1 in Newton's method.  Verify that your solution replicates the $\beta$ estimate you get when using a package solver, e.g. the glm function in R, up to minor numerical differences.

\item Standard asymptotic theory, which we won't go into here, implies that the maximum likelihood estimator is consistent and asymptotically normal around the true value $\beta_0$:
$$
\hat{\beta} \sim N(\beta_0, I(\beta_0, \phi)^{-1}) \, ,
$$
where $\mathcal{I}(\beta_0, \phi)$, called the \textit{Fisher information matrix}, is the same $\mathcal{I}$ you met all the way back when you proved the score equations:
$$
\mathcal{I}(\beta_0, \phi) \equiv \mbox{var} \{ s(\beta_0, \phi) \}  = - E \left\{ H(\beta_0, \phi) \right\} \, .
$$
The fact that Fisher information is the negative of the expected Hessian motivates the following idea: use the inverse of the negative Hessian matrix at the MLE to approximate the inverse Fisher information, i.e.~the covariance matrix of the estimator.  Happily, you get this Hessian matrix for free when fitting by Newton's method.

For your logistic regression on the WDBC data fit via Newton's method, compute the square root of each diagonal element of the inverse Hessian matrix, evaluated at the MLE.\footnote{These are your standard errors for each coefficient, i.e.~the square root of the variance of each coefficient's (approximate) sampling distribution.}  Compare these to the standard errors you get when using a package solver, e.g. the glm function in R.

\end{enumerate}

\end{document}


\chapter{Ensembles on Random Patches}\label{ch:random-patches}

% sampling samples: an easy way to reduce time complexity
% sampling features: an easy way to decorrelate predictions

\begin{remark}{Outline}
In this chapter, we consider supervised learning under the assumption that the
available memory is small compared to the size of the dataset. This general framework
is relevant in the context of big data, distributed databases and embedded
systems. In Section~\ref{sec:9:rp}, we propose a very simple, yet effective,
ensemble framework that builds each individual model of the ensemble from a
random patch of data obtained by drawing random subsets of \textit{both}
samples and input variables from the whole dataset. In
sections~\ref{sec:9:accuracy} and \ref{sec:9:memory}, we carry out an extensive
and systematic evaluation of this method on 29 datasets, using decision trees
as base models. With respect to popular ensemble methods, these experiments
show that the proposed method provides on par  performance  in terms of
accuracy while simultaneously lowering the memory needs, and attains
significantly better performance when memory is severely constrained. We
conclude and discuss future work directions in Section~\ref{sec:9:conclusions}.
\textit{This chapter is based on previous work published in \citep{louppe:2012}.}
\end{remark}

Within the past few years, big data has become a popular trend among many
scientific fields. In life sciences, computer vision, Internet search or
finance, to cite a few, quantities of data have grown so large that it is
increasingly difficult to process, analyze or visualize. In many cases, single
computers are no longer fit for big data and distributed environments need to
be considered to handle it. Although research is very active in this area,
machine learning is no exception to this new paradigm. Much still needs to be
done and methods and algorithms have to be reinvented to take this constraint
into account.

In this context, we consider supervised learning problems for which the dataset
is so large that it cannot be loaded into memory. \citet{breiman:1999} proposed
the Pasting method to tackle this problem by learning an ensemble of models
individually built on random subsets of the training examples, hence
alleviating the memory requirements since the base models would be built on
only small parts of the whole dataset. Earlier, \citet{ho:1998} proposed to
learn an ensemble of models individually built on random subspaces, i.e., on
random subsets of the input variables (or \textit{features}). While the first motivation of the Random
Subspace method was to increase the diversity within the models of the
ensemble, it can actually also be seen as way to reduce the memory requirements
of building individual models. In this work, we propose to combine and leverage
both approaches at the same time: learn an ensemble of models on \textit{random
patches}, i.e., on random subsets of the samples \textit{and} of the input
variables. Through an extensive empirical study, we show that this approach (1)
improves or preserves comparable accuracy with respect to other ensemble
approaches which build base models on the whole dataset while (2) drastically
lowering the memory requirements and hence allowing an equivalent reduction of
the global computing time. In other words, this analysis shows that there is
usually no need to build individual models on the whole dataset. For the same
accuracy, models can be learned independently on small portions of the data,
within significantly lower computational requirements.

\section{Random Patches}
\label{sec:9:rp}

The Random Patches algorithm proposed in this work (further referred to as RP)
is a wrapper ensemble method that can be described in the following terms. Let
${\cal R}(\alpha_s, \alpha_f, {\cal L})$ be the set of all random patches of
size $\alpha_s N \times \alpha_f p$ than can be drawn from the dataset ${\cal
L}$, where $N$ (resp., $p$) is the number of samples in ${\cal L}$ (resp.,  the
number of input variables in ${\cal L}$) and where $\alpha_s \in [0, 1]$\label{ntn:alpha_s} (resp.
$\alpha_f$\label{ntn:alpha_f}) is an hyper-parameter that controls the number of samples in a
patch (resp., the number of variables). That is, ${\cal R}(\alpha_s, \alpha_f,
{\cal L})$ is the set of all possible subsets containing $\alpha_s N$ samples
(among $N$) with $\alpha_f p$ variables (among $p$). The method then works as
follows:

\begin{algorithm}\label{algo:rp}
Random Patches algorithm.
\textnormal{
\begin{algorithmic}[1]
    \For{$m=1, \dots, M$}
        \State Draw a patch $r \sim U({\cal R}(\alpha_s, \alpha_f, {\cal L}))$ uniformly at random
        \State Build a model on the selected patch $r$
    \EndFor
    \State Aggregate the predictions of the $M$ models in an ensemble
\end{algorithmic}
}
\end{algorithm}

While the RP algorithm can exploit any kind of base estimators, we consider in
this work only tree-based estimators. In particular, we evaluate the RP
algorithm using  standard classification trees (as described in
Chapter~\ref{ch:cart}) and (single) extremely randomized
trees~\citep{geurts:2006}. Unless stated otherwise, trees are unpruned and
grown using Gini index as impurity criterion for node splitting. The
parameter $K$ of extremely randomized trees within RP is set to its maximum
value $K=\alpha_f p$ (i.e., corresponding to no further random selection of
variables).

The first benefit of RP is that it generalizes both the Pasting Rvotes (P)
method~\citep{breiman:1999} (and its extensions \citep{chawla:2004,basilico:2011})
and the Random Subspace (RS) algorithm~\citep{ho:1998}. Both are indeed merely
particular cases of RP: setting $\alpha_s=1.0$ yields RS while setting
$\alpha_f=1.0$ yields P. As such, it is expected that when both
hyper-parameters $\alpha_s$ and $\alpha_f$ are tuned simultaneously, RP should be at
least as good as the best of the two methods, provided there is no overfitting
associated with this tuning.

When the base estimators are standard decision trees (resp., extremely
randomized trees with $K=\alpha_f p$), interesting parallels can also be
drawn between RP and the RF algorithm (resp., ET). For $\alpha_s=1.0$, the
value of $\alpha_f p$ is indeed nearly equivalent to the number $K$ of
features randomly considered when splitting a node. A major difference
remains though. In RP, the subset of features is selected globally
once and for all, prior to the construction of each tree. By contrast,
in RF (resp., in ET) subsets of features are drawn locally at each
node. Clearly, the former approach already appears to be more
attractive when dealing with large databases. Non-selected features
indeed do not need to be considered at all, hence lowering the memory
requirements for building a single tree. Another interesting parallel
can be made when bootstrap samples are used like in RF: it nearly
amounts to set $\alpha_s=0.632$, i.e. the average proportion of unique
samples in a bootstrap sample. Differences are that in a bootstrap
sample, the number of unique training samples varies from one to
another (while it would be fixed to $0.632 N$ in RP), and that
samples are not all equally weighted.

In addition, RP also closely relates to the SubBag algorithm
which combines Bagging and RS for constructing ensembles. Using $N$
bootstrapped samples (i.e., nearly equivalent to $\alpha_s=0.632$) and setting
$\alpha_f=0.75$, \citet{panov:2007} showed that SubBag has comparable performance to that of
RF. An added advantage of SubBag, and hence of RP, is that it is applicable to
any base estimator without the need to randomize the latter.

\section{On Accuracy}
\label{sec:9:accuracy}

Our validation of the RP algorithm is carried out in two
steps. In this section, we first investigate how RP compares with
other popular tree-based ensemble methods in terms of accuracy. In the
next section, we then focus on its memory requirements for
achieving optimal accuracy and its capability to handle strong memory
constraints, again in comparison with other ensemble methods.

Considering accuracy only, our main objective is to investigate whe\-ther the
additional degrees of freedom brought by $\alpha_s$ and $\alpha_f$ significantly improve,
or degrade, the performance of RP. At first glance, one might indeed think that
since the base estimators are (intentionally) built on parts of the data, the
accuracy of the ensemble will be lower than if they were all built on the whole
set. Additionally, our goal is also to see whether sampling features once
globally, instead of locally at each node, impairs performance, as this is the
main difference between RP and state-of-the-art methods such as RF or ET.

\subsection{Protocol}

We compare our method with P and RS, as well as with RF and ET. For RP, P and
RS, two variants have been considered, one using standard decision trees
(suffixed below with '-DT') as base estimators, and the other using extremely
randomized trees (suffixed below with '-ET') as base estimators.  Overall, 8
methods are compared: RP-DT, RP-ET, P-DT, P-ET, RS-DT, RS-ET, RF and ET.

We evaluate the accuracy of the methods on an extensive list of both
artificial and real classification problems.  For each dataset, three
random partitions were drawn: the first and larger (50\% of the
original dataset) to be used as the training set, the second (25\%) as
validation set and the third (25\%) as test set. For all methods, the
hyper-parameters $\alpha_s$ and $\alpha_f$ were tuned on the validation set with
a grid-search procedure, using the grid \{0.01, 0.1, ..., 0.9, 1.0\}
for both $\alpha_s$ and $\alpha_f$. All other hyper-parameters were set to
default values. In RF and ET, the number $K$ of features randomly
selected at each node was tuned using the grid $\alpha_f p$. For all
ensembles, 250 fully developed trees were generated and the
generalization accuracy was estimated on the test set. Unless
otherwise mentioned, for all methods and for all datasets, that
procedure was repeated 50 times, using the same 50 random partitions
between all methods, and all scores reported below are averages over
those 50 runs. All algorithms and experiments have been implemented in
Python, using Scikit-Learn~\citep{pedregosa:2011} as base framework.

\subsection{Small datasets}

Before diving into heavily computational experiments, we first wanted to
validate our approach on small to medium datasets. To that end, experiments
were carried out on a sample of 16 well-known and publicly
available datasets (see Table~\ref{table:rp:small-data}) from the UCI machine
learning repository~\citep{frank:2010}, all chosen a priori and
independently of the results obtained. Overall, these datasets cover a wide
range of conditions, with the sample sizes ranging from 208 to 20000 and the
number of features varying from 6 to 168. Detailed average performances of the
8 methods for all 16 datasets using the protocol described above are reported
in Table~\ref{table:rp:accuracy}. Below, we analyze general trends by performing
various statistical tests.

\begin{table}
    \centering
    \footnotesize
    \begin{tabular}{| l | c c |}
    \hline
    \textbf{Dataset} & $N$ & $p$ \\
    \hline
    \hline
        \textsc{diabetes}    & 768       & 8         \\
        \textsc{dig44}       & 18000     & 16        \\
        \textsc{ionosphere}  & 351       & 34        \\
        \textsc{pendigits}   & 10992     & 16        \\
        \textsc{letter}      & 20000     & 16        \\
        \textsc{liver}       & 345       & 6         \\
        \textsc{musk2}       & 6598      & 168       \\
        \textsc{ring-norm}   & 10000     & 20        \\
        \textsc{satellite}   & 6435      & 36        \\
        \textsc{segment}     & 2310      & 19        \\
        \textsc{sonar}       & 208       & 60        \\
        \textsc{spambase}    & 4601      & 57        \\
        \textsc{two-norm}    & 9999      & 20        \\
        \textsc{vehicle}     & 1692      & 18        \\
        \textsc{vowel}       & 990       & 10        \\
        \textsc{waveform}    & 5000      & 21        \\
    \hline
    \end{tabular}
    \caption{Small datasets.}
    \label{table:rp:small-data}
\end{table}

\begin{table}
    \centering
    \footnotesize
    \hspace{-3.2cm}
\begin{tabular}{|l|cccccccc|}
\hline
\textbf{Validation} & RF & ET & P-DT & P-ET & RS-DT & RS-ET & RP-DT & RP-ET \\
\hline
\hline
    \textsc{diabetes  }  & 77.12 (6)    & 77.25 (5)    & 77.67 (4)    & 78.01 (3)    & 75.11 (8)    & 76.77 (7)    & 78.82 (2)   &  \textbf{79.07} (1) \\
    \textsc{dig44     }  & 94.99 (7)    & \textbf{95.78} (1)    & 91.86 (8)    & 95.46 (4)    & 95.07 (6)    & 95.69 (3)    & 95.13 (5)   &  95.72 (2) \\
    \textsc{ionosphere}  & 94.40 (6)    & 95.15 (3)    & 93.86 (8)    & 94.75 (5)    & 94.11 (7)    & 94.90 (4)    & 95.20 (2)   &  \textbf{95.36} (1) \\
    \textsc{pendigits }  & 98.94 (7)    & \textbf{99.33} (1)    & 98.09 (8)    & 99.28 (4)    & 99.02 (6)    & 99.31 (3)    & 99.07 (5)   &  99.32 (2) \\
    \textsc{letter    }  & 95.36 (7)    & \textbf{96.38} (1)    & 92.72 (8)    & 95.87 (4)    & 95.68 (6)    & 96.08 (3)    & 95.74 (5)   &  96.10 (2) \\
    \textsc{liver     }  & 72.37 (5)    & 71.90 (6)    & 72.55 (4)    & 72.88 (3)    & 68.06 (8)    & 70.88 (7)    & \textbf{74.53} (1)   &  74.37 (2) \\
    \textsc{musk2     }  & 97.18 (7)    & \textbf{97.73} (1)    & 96.89 (8)    & 97.60 (4)    & 97.58 (6)    & 97.72 (3)    & 97.60 (5)   &  97.73 (2) \\
    \textsc{ring-norm }  & 97.44 (6)    & 98.10 (5)    & 96.41 (8)    & 97.28 (7)    & 98.25 (4)    & 98.41 (3)    & 98.50 (2)   &  \textbf{98.54} (1) \\
    \textsc{satellite }  & 90.97 (7)    & \textbf{91.56} (1)    & 90.01 (8)    & 91.40 (5)    & 91.31 (6)    & 91.50 (3)    & 91.41 (4)   &  91.54 (2) \\
    \textsc{segment   }  & 97.46 (6)    & 98.17 (2)    & 96.78 (8)    & 98.10 (4)    & 97.33 (7)    & 98.14 (3)    & 97.52 (5)   &  \textbf{98.21} (1) \\
    \textsc{sonar     }  & 82.92 (7)    & 86.92 (3)    & 80.03 (8)    & 84.73 (5)    & 83.07 (6)    & 87.07 (2)    & 85.42 (4)   &  \textbf{88.15} (1) \\
    \textsc{spambase  }  & 94.80 (7)    & 95.36 (3)    & 93.69 (8)    & 95.01 (6)    & 95.01 (5)    & 95.50 (2)    & 95.11 (4)   &  \textbf{95.57} (1) \\
    \textsc{two-norm  }  & 97.54 (6)    & 97.77 (2)    & 97.52 (7)    & 97.59 (5)    & 97.46 (8)    & 97.63 (4)    & 97.76 (3)   &  \textbf{97.82} (1) \\
    \textsc{vehicle   }  & 88.67 (5)    & 88.68 (4)    & 88.26 (8)    & 88.74 (3)    & 88.41 (7)    & 88.60 (6)    & \textbf{89.22} (1)   &  89.21 (2) \\
    \textsc{vowel     }  & 92.04 (5)    & \textbf{95.12} (1)    & 85.19 (8)    & 93.49 (4)    & 89.76 (7)    & 94.34 (3)    & 91.10 (6)   &  94.48 (2) \\
    \textsc{waveform  }  & 85.45 (6)    & 85.96 (2)    & 84.89 (8)    & 85.68 (5)    & 84.91 (7)    & 85.69 (4)    & 85.85 (3)   &  \textbf{86.21} (1) \\
\hline
\textit{Average rank} & 6.25 & 2.5625 & 7.4375 & 4.4375 & 6.5 & 3.75 & 3.5626 & 1.5 \\
\hline
\hline
\textbf{Test} & RF & ET & P-DT & P-ET & RS-DT & RS-ET & RP-DT & RP-ET \\
\hline
\hline
    \textsc{diabetes  }  & 75.62 (4)    & 75.38 (5)    & 75.67 (3)    & \textbf{76.34} (1)    & 73.03 (8)    & 74.63 (7)    & 75.32 (6)   &  75.82 (2) \\
    \textsc{dig44     }  & 94.96 (6)    & \textbf{95.67} (1)    & 91.79 (8)    & 95.39 (4)    & 94.98 (5)    & 95.58 (2)    & 94.95 (7)   &  95.55 (3) \\
    \textsc{ionosphere}  & 92.20 (6)    & \textbf{93.22} (1)    & 92.09 (7)    & 92.40 (4)    & 92.02 (8)    & 93.22 (2)    & 92.34 (5)   &  92.68 (3) \\
    \textsc{pendigits }  & 98.84 (7)    & \textbf{99.23} (1)    & 97.97 (8)    & 99.21 (3)    & 98.95 (5)    & 99.21 (2)    & 98.93 (6)   &  99.20 (4) \\
    \textsc{letter    }  & 95.27 (7)    & \textbf{96.29} (1)    & 92.57 (8)    & 95.89 (4)    & 95.61 (5)    & 96.03 (2)    & 95.61 (6)   &  95.99 (3) \\
    \textsc{liver     }  & 69.95 (3)    & 68.22 (6)    & \textbf{70.43} (1)    & 69.58 (5)    & 63.17 (8)    & 67.35 (7)    & 70.20 (2)   &  69.67 (4) \\
    \textsc{musk2     }  & 97.08 (7)    & \textbf{97.61} (1)    & 96.69 (8)    & 97.54 (4)    & 97.47 (5)    & 97.58 (2)    & 97.42 (6)   &  97.56 (3) \\
    \textsc{ring-norm }  & 97.48 (6)    & 98.07 (5)    & 96.42 (8)    & 97.25 (7)    & 98.16 (4)    & \textbf{98.31} (1)    & 98.22 (3)   &  98.30 (2) \\
    \textsc{satellite }  & 90.67 (7)    & 91.22 (2)    & 89.66 (8)    & 91.20 (3)    & 91.15 (5)    & \textbf{91.28} (1)    & 91.04 (6)   &  91.20 (4) \\
    \textsc{segment   }  & 97.02 (5)    & \textbf{97.93} (1)    & 96.44 (8)    & 97.86 (3)    & 96.86 (7)    & 97.90 (2)    & 96.88 (6)   &  97.84 (4) \\
    \textsc{sonar     }  & 79.53 (5)    & \textbf{82.76} (1)    & 75.15 (8)    & 80.07 (4)    & 78.50 (6)    & 82.19 (2)    & 78.26 (7)   &  81.92 (3) \\
    \textsc{spambase  }  & 94.76 (7)    & 95.17 (3)    & 93.51 (8)    & 94.84 (5)    & 94.88 (4)    & 95.22 (2)    & 94.80 (6)   &  \textbf{95.22} (1) \\
    \textsc{two-norm  }  & 97.22 (7)    & \textbf{97.50} (1)    & 97.26 (6)    & 97.29 (4)    & 97.20 (8)    & 97.33 (2)    & 97.33 (3)   &  97.28 (5) \\
    \textsc{vehicle   }  & 87.47 (7)    & 87.85 (3)    & 87.01 (8)    & 87.98 (2)    & 87.50 (6)    & 87.68 (5)    & 87.73 (4)   &  \textbf{88.08} (1) \\
    \textsc{vowel     }  & 91.51 (5)    & \textbf{93.95} (1)    & 84.17 (8)    & 92.93 (4)    & 89.51 (7)    & 93.60 (2)    & 89.89 (6)   &  93.09 (3) \\
    \textsc{waveform  }  & 85.23 (5)    & \textbf{85.77} (1)    & 84.68 (8)    & 85.40 (3)    & 84.74 (7)    & 85.38 (4)    & 85.16 (6)   &  85.56 (2) \\
\hline
\textit{Average rank} & 5.875 & 2.125 & 7.0625 & 3.75 & 6.125 & 2.8125 & 5.3125 & 2.9375 \\
\hline
\end{tabular}
    \caption{Accuracy on small datasets (in \%).}
    \label{table:rp:accuracy}
\end{table}

Following recommendations in \citep{demsar:2006}, we first performed a
Friedman test that rejected the hypothesis that all algorithms are
equivalent at a significance level $\alpha=0.05$.
We then proceeded with a post-hoc Nemenyi test for a
pairwise comparison of the average ranks of all 8 methods. According to
this test, the performance of two classifiers is significantly
different (at $\alpha=0.05$) if their average ranks differ by at least
the critical difference $CD = 2.6249$
(See~\citep{demsar:2006} for further details). The diagram of
Figure~\ref{fig:8:small-cmp} summarizes these comparisons. The top line
in the diagram is the axis along which the average rank $R_m$ of each method $m$ is plotted,
from the highest ranks (worst methods) on the left to the lowest ranks
(best methods) on the right.  Groups of methods that are not
statistically different from each other are connected. The critical
difference CD is shown above the graph.  To further support these rank
comparisons, we also compare the 50 accuracy values obtained
over each dataset split for each pair of methods by using a paired
t-test (with $\alpha=0.01$). The results of these comparisons are
summarized in Table \ref{table:rp:small-ttest} in terms of ``Win-Draw-Loss''
statuses of all pairs of methods; the three values at the intersection
of row $i$ and column $j$ of this table respectively indicate on how
many datasets method $i$ is significantly better/not significantly
different/significantly worse than method $j$.

\begin{figure}
    \centering
    \includegraphics[width=0.9\textwidth]{figures/ch8_rank_small.pdf}
    \caption{Average ranks of all methods on small datasets.}
    \label{fig:8:small-cmp}
\end{figure}

\begin{table}
    \centering
    \footnotesize
  \begin{tabular}{|l|cccccccc|}
    \hline
         & RF & ET & P-DT & P-ET & RS-DT & RS-ET & RP-DT & RP-ET \\
    \hline
    \hline
        RF     &  ---   & 1/2/13 & 12/4/0  & 1/7/8   & 4/7/5   & 2/2/12 & 1/10/5  & 0/4/12 \\
        ET     &  13/2/1 & ---   & 14/1/1  & 10/5/1  & 13/3/0  & 4/11/1 & 12/2/2  & 5/10/1 \\
        P-DT   &  0/4/12 & 1/1/14 & ---    & 0/4/12  & 2/3/11  & 2/1/13 & 0/4/12  & 0/4/12 \\
        P-ET   &  8/7/1  & 1/5/10 & 12/4/0  & ---    & 9/6/1   & 2/6/8  & 9/6/1   & 0/11/5 \\
        RS-DT  &  5/7/4  & 0/3/13 & 11/3/2  & 1/6/9   & ---    & 0/2/14 & 1/11/4  & 0/4/12 \\
        RS-ET  &  12/2/2 & 1/11/4 & 13/1/2  & 8/6/2   & 14/2/0  & ---   & 11/4/1  & 1/13/2 \\
        RP-DT  &  5/10/1 & 2/2/12 & 12/4/0  & 1/6/9   & 4/11/1  & 1/4/11 & ---    & 0/6/10 \\
        RP-ET  &  12/4/0 & 1/10/5 & 12/4/0  & 5/11/0  & 12/4/0  & 2/13/1 & 10/6/0  & ---\\
    \hline
    \end{tabular}
    \caption{Pairwise t-test comparisons on small datasets.}
    \label{table:rp:small-ttest}
\end{table}

% ET versus DT

Since all methods are variants of ensembles of decision trees, average
accuracies are not strikingly different from one method to another (see Table 1
of the supplementary materials). Yet, significant trends appear when looking at
Figure \ref{fig:8:small-cmp} and Table \ref{table:rp:small-ttest}.  First, all ET-based
methods are ranked before DT-based methods, including the popular Random
Forest algorithm. Overall, the original ET algorithm is ranked first ($R_{ET} =
2.125$), then come RS-ET and RP-ET at close positions ($R_{RS-ET} = 2.8125$ and
$R_{RP-ET} = 2.9375$) while P-ET is a bit behind ($R_{P-ET} = 3.75$). According
to Figure \ref{fig:8:small-cmp}, only ET is ranked significantly higher than all
DT-based method but looking at Table \ref{table:rp:small-ttest}, the worse ET-based
variant (P-ET) is still 9 times significantly better (w.r.t. the 50
runs over each set) and only 1 time significantly worse than the best DT-based
variant (RP-DT). The separation between these two families of algorithm
thus appears quite significant. This observation clearly suggests that using
random split thresholds, instead of optimized ones like in decision trees, pays
off in terms of generalization.

% RP versus RS et P (ET)

Among ET-based methods, RP-ET is better than P-ET but it is superseded by ET and
RS-ET in terms of average rank. Since RS-ET is a particular case of RP-ET, this
suggests that we are slightly overfitting when tuning the additional parameter
$\alpha_s$. And indeed RP-ET is better ranked than RS-ET in average on the validation
set (results not shown). Table~\ref{table:rp:small-ttest} however indicates
otherwise and makes RP-ET appear as slightly better than RS-ET (2/13/1).
Regarding ET over RP-ET, the better performance of the former (5/10/1) is
probably due to the fact that in ET subsets of features are redrawn locally at
each node when building trees and not once and for all prior to their
construction. This gives less chances to generate improper trees because of a
bad initial choice of features and thus leads to a lower bias and a better
accuracy.

% RP versus RS et P (DT)

Among DT-based methods, RP-DT now comes first (mean rank of 5.3125),
then RF ($R_{RF} = 5.875$), RS-DT ($R_{RS-DT} = 6.125$) and then P-DT
in last ($R_{P-DT} = 7.0625$). RP is only significantly worse than
another DT-based variant on 1 dataset. The extra-randomization brought
by the random choices of both samples and features seems to be
beneficial with decision trees that do not benefit from the
randomization of discretization thresholds. The fact that RF samples
features locally does not appear here anymore as an advantage over RP
(RF is significantly worse on 5 problems and better on only one),
probably because the decrease of bias that it provides does not exceed
the increase of variance with respect to global feature selection.

\subsection{Larger datasets}

While the former experiments revealed promising results, it is fair to ask
whether the conclusions that have been drawn would hold on and generalize to
larger problems, for example when dealing with a few relevant features buried
into hundreds or thousands of not important features (e.g., in genomic data),
or when dealing with many correlated features (e.g., in images). To investigate
this question, a second bench of experiments was carried out on 13 larger
datasets (see Table~\ref{table:rp:large-data}). All but
\textsc{madelon} are real data. In terms of dimensions, these datasets are far bigger,
ranging from a few hundreds of samples and thousands of features, to thousands of
samples but hundreds of features. As such, the complexity of the problems is
expected to be greater.  We adopted the exact same protocol as for smaller
datasets. However, to lower computing times, for datasets marked with $*$, the
methods were run using 100 trees instead of 250 and the minimum number of
samples required in an internal node was set to 10 in order to control
complexity. Detailed results are provided in Table~\ref{table:rp:accuracy-large}
and are summarized in Figure \ref{fig:8:large-cmp} and
Table \ref{table:rp:large-ttest}, respectively in terms of average rank (the
critical difference at $\alpha=0.05$ is now 2.9120) and Win/Draw/Loss statuses
obtained with paired t-tests. A Friedman test (at $\alpha=0.05$) still
indicates that some methods are significantly different from the others.

\begin{table}
    \centering
    \footnotesize
    \begin{tabular}{| l | c c |}
    \hline
    \textbf{Dataset} & $N$ & $p$ \\
    \hline
    \hline
                \textsc{cifar10}*   & 60000   & 3072   \\
                \textsc{mnist3vs8}  & 13966   & 784    \\
                \textsc{mnist4vs9}  & 13782   & 784    \\
                \textsc{mnist}*     & 70000   & 784    \\
                \textsc{isolet}     & 7797    & 617    \\
                \textsc{arcene}     & 900     & 10000  \\
                \textsc{breast2}    & 295     & 24496  \\
                \textsc{madelon}    & 4400    & 500    \\
                \textsc{marti0}     & 500     & 1024   \\
                \textsc{reged0}     & 500     & 999    \\
                \textsc{secom}      & 1567    & 591    \\
                \textsc{tis}        & 13375   & 927    \\
                \textsc{sido0}*     & 12678   & 4932   \\
    \hline
    \end{tabular}
    \caption{Large datasets.}
    \label{table:rp:large-data}
\end{table}

\begin{table}
    \centering
    \footnotesize
    \hspace{-3.2cm}
\begin{tabular}{|l|cccccccc|}
\hline
\textbf{Validation} & RF & ET & P-DT & P-ET & RS-DT & RS-ET & RP-DT & RP-ET \\
\hline
\hline
    \textsc{cifar10 }  &  45.16 (6)  &  \textbf{46.17} (1)  &  44.92 (7)  &  44.88 (8)  &  45.83 (5)  &  46.09 (3)  &  45.89 (4)  &  46.13 (2) \\
    \textsc{mnist3v8}  &  98.42 (7)  &  98.77 (5)  &  97.63 (8)  &  98.57 (6)  &  98.79 (4)  &  98.86 (2)  &  98.81 (3)  &  \textbf{98.87} (1) \\
    \textsc{mnist4v9}  &  98.47 (7)  &  98.82 (3)  &  97.39 (8)  &  98.47 (6)  &  98.77 (5)  &  98.89 (2)  &  98.80 (4)  &  \textbf{98.91} (1) \\
    \textsc{mnist   }  &  96.14 (7)  &  96.56 (3)  &  94.35 (8)  &  96.19 (6)  &  96.52 (5)  &  96.57 (2)  &  96.53 (4)  &  \textbf{96.57} (1) \\
    \textsc{isolet  }  &  93.96 (7)  &  95.07 (3)  &  90.90 (8)  &  94.23 (6)  &  94.38 (5)  &  95.15 (2)  &  94.47 (4)  &  \textbf{95.20} (1) \\
    \textsc{arcene  }  &  73.84 (7)  &  77.76 (3)  &  73.76 (8)  &  75.12 (6)  &  76.40 (5)  &  77.60 (4)  &  79.44 (2)  &  \textbf{80.00} (1) \\
    \textsc{breast2 }  &  69.64 (6)  &  69.91 (4)  &  70.54 (3)  &  69.83 (5)  &  69.59 (7)  &  69.54 (8)  &  \textbf{72.59} (1)  &  71.62 (2) \\
    \textsc{madelon }  &  76.12 (8)  &  \textbf{81.41} (1)  &  78.48 (7)  &  81.06 (5)  &  80.68 (6)  &  81.29 (3)  &  81.18 (4)  &  81.36 (2) \\
    \textsc{marti   }  &  87.66 (8)  &  87.92 (6)  &  88.08 (4)  &  88.09 (3)  &  87.77 (7)  &  88.00 (5)  &  88.19 (2)  &  \textbf{88.20} (1) \\
    \textsc{reged   }  &  98.00 (6)  &  98.46 (2)  &  97.08 (8)  &  98.24 (5)  &  98.00 (7)  &  98.41 (3)  &  98.40 (4)  &  \textbf{98.57} (1) \\
    \textsc{secom   }  &  93.37 (5)  &  93.36 (6)  &  93.34 (8)  &  93.42 (2)  &  93.35 (7)  &  93.40 (4)  &  93.41 (3)  &  \textbf{93.45} (1) \\
    \textsc{tis     }  &  91.81 (5)  &  91.53 (7)  &  91.59 (6)  &  91.50 (8)  &  92.04 (3)  &  91.97 (4)  &  \textbf{92.26} (1)  &  92.06 (2) \\
    \textsc{sido    }  &  97.46 (5)  &  97.44 (6)  &  97.36 (8)  &  97.36 (7)  &  97.47 (3)  &  97.46 (4)  &  \textbf{97.52} (1)  &  97.52 (2) \\
\hline
\textit{Avg. rank} & 6.4615 & 3.8461 & 7.0   & 5.6153 & 5.3076 & 3.5384 &  2.8461 & 1.3846 \\
\hline
\hline
\textbf{Test} & RF & ET & P-DT & P-ET & RS-DT & RS-ET & RP-DT & RP-ET \\
\hline
\hline
    \textsc{cifar10 }  &  44.87 (7)  &  45.95 (2)  &  44.75 (8)  &  44.95 (6)  &  45.86 (4)  &  \textbf{46.02} (1)  &  45.74 (5)  &  45.93 (3) \\
    \textsc{mnist3v8}  &  98.31 (7)  &  98.64 (5)  &  97.47 (8)  &  98.48 (6)  &  98.70 (2)  &  \textbf{98.71} (1)  &  98.68 (4)  &  98.68 (3) \\
    \textsc{mnist4v9}  &  98.33 (6)  &  98.68 (2)  &  97.16 (8)  &  98.32 (7)  &  98.59 (4)  &  \textbf{98.69} (1)  &  98.59 (5)  &  98.67 (3) \\
    \textsc{mnist   }  &  96.10 (7)  &  96.47 (3)  &  94.30 (8)  &  96.18 (6)  &  96.47 (4)  &  \textbf{96.49} (1)  &  96.46 (5)  &  96.48 (2) \\
    \textsc{isolet  }  &  93.87 (7)  &  \textbf{94.95} (1)  &  90.71 (8)  &  94.15 (6)  &  94.33 (4)  &  94.94 (2)  &  94.29 (5)  &  94.88 (3) \\
    \textsc{arcene  }  &  71.04 (4)  &  \textbf{72.56} (1)  &  66.16 (8)  &  68.56 (7)  &  71.52 (3)  &  72.24 (2)  &  69.44 (6)  &  70.00 (5) \\
    \textsc{breast2 }  &  65.86 (6)  &  \textbf{66.16} (1)  &  65.54 (8)  &  65.59 (7)  &  66.13 (2)  &  65.91 (5)  &  65.94 (4)  &  66.08 (3) \\
    \textsc{madelon }  &  75.33 (8)  &  \textbf{80.83} (1)  &  77.86 (7)  &  80.78 (2)  &  80.06 (6)  &  80.59 (3)  &  80.06 (5)  &  80.42 (4) \\
    \textsc{marti   }  &  87.74 (8)  &  87.85 (6)  &  \textbf{88.24} (1)  &  88.17 (2)  &  87.82 (7)  &  88.06 (4)  &  88.08 (3)  &  88.01 (5) \\
    \textsc{reged   }  &  97.60 (5)  &  \textbf{98.20} (1)  &  96.40 (8)  &  97.92 (3)  &  97.39 (6)  &  98.00 (2)  &  97.32 (7)  &  97.82 (4) \\
    \textsc{secom   }  &  93.30 (7)  &  93.22 (8)  &  \textbf{93.38} (1)  &  93.30 (6)  &  93.37 (2)  &  93.31 (4)  &  93.33 (3)  &  93.31 (5) \\
    \textsc{tis     }  &  91.68 (5)  &  91.42 (7)  &  91.49 (6)  &  91.40 (8)  &  \textbf{92.05} (1)  &  91.91 (3)  &  92.05 (2)  &  91.90 (4) \\
    \textsc{sido    }  &  97.33 (4)  &  97.33 (3)  &  97.26 (7)  &  97.26 (8)  &  \textbf{97.35} (1)  &  97.34 (2)  &  97.33 (5)  &  97.32 (6) \\
\hline
\textit{Avg. rank} & 6.2307 & 3.1538 & 6.6153 & 5.6923 & 3.5384 & 2.3846 & 4.5384 & 3.8461 \\
\hline
\end{tabular}
    \caption{Accuracy on larger datasets (in \%).}
    \label{table:rp:accuracy-large}
\end{table}

As it may be observed from Figure~\ref{fig:8:large-cmp}, the average ranks of the
methods are closer to each other than in the previous experiments, now ranging
from 2.38 to 6.61, while they were previously ranging from 2.12 to 7. Methods
are more connected by critical difference bars. This suggests that overall they
behave more similarly to each other than before. General trends are nevertheless
comparable to what we observed earlier. ET-based methods still seem to be the
front-runners.  From Figure~\ref{fig:8:large-cmp}, RS-ET, ET and RP-ET are in the
top 4, while P-DT, RF and RP-DT remain in the second half of the ranking.
Surprisingly however, RS-DT now comes right after RS-ET and ET and just before
RP-ET whereas it ranked penultimate on the smaller datasets. Table~\ref{table:rp:large-ttest}
however suggests that RS-DT performs actually a little worse
against RP-ET (1/10/2).  All in all, it thus still seems beneficial to randomize
split thresholds on the larger datasets.

Comparing ET-based variants, ET is no longer the best method on average, but RS-ET
is (with 4/9/0 for RS-ET versus ET). This suggests than on larger datasets,
picking features globally at random prior to the construction of the trees is as
good, or even beat picking them locally at each node. Due to the quantitatively
larger number of samples in a patch, and also to the larger number of redundant
features expected in some large datasets (e.g., in \textsc{cifar10} or \textsc{mnist}), it is
indeed less likely to build improper trees with strong biases. As a result,
variance can be further decreased by sampling globally. In support of this
claim, on a few problems such as \textsc{arcene}, \textsc{breast2}, or \textsc{madelon} that contain many
irrelevant features, ET remains the best method. In that case, it is indeed more
likely to sample globally improper random patches, and hence to build improper
trees. The average rank of RP-ET suggests that it performs worse than RS-ET and
thus that there is some potential overfitting when tuning $\alpha_s$ in addition to
$\alpha_f$. This difference is however not confirmed in Table~\ref{table:rp:large-ttest}
where the accuracies of these two methods are shown to be never significantly
different (0/13/0). RP-ET is also on a perfect par with ET (1/11/1). Among DT-based variants,
RP-DT, which was the best performer on small datasets, is still
ranked above RF and P-DT, but it is now ranked below RS-DT with a win/draw/loss
of 0/11/2. This is again due to some overfitting.

While less conclusive than before, the results on larger datasets are
consistent with what we observed earlier. In particular, they indicate that
the Random Patches method (with ET) remains competitive with the best
performers.

\begin{figure}
    \centering
    \includegraphics[width=0.9\textwidth]{figures/ch8_rank_large.pdf}
    \caption{Average ranks of all methods on larger datasets.}
    \label{fig:8:large-cmp}
\end{figure}

\begin{table}
    \centering
    \footnotesize
  \begin{tabular}{|l|cccccccc|}
    \hline
         & RF & ET & P-DT & P-ET & RS-DT & RS-ET & RP-DT & RP-ET \\
    \hline
    \hline
                RF     & ---    & 1/5/7   & 8/3/2  & 2/6/5  & 0/6/7  & 0/5/8  & 0/6/7  & 0/6/7 \\
                ET     & 7/5/1   & ---    & 9/2/2  & 7/6/0  & 3/7/3  & 0/9/4  & 5/6/2  & 1/11/1 \\
                P-DT   & 2/3/8   & 2/2/9   & ---   & 1/5/7  & 0/3/10 & 0/3/10 & 1/3/9  & 0/4/9 \\
                P-ET   & 5/6/2   & 0/6/7   & 7/5/1  & ---   & 0/6/7  & 0/5/8  & 2/5/6  & 1/5/7 \\
                RS-DT  & 7/6/0   & 3/7/3   & 10/3/0 & 7/6/0  & ---   & 1/8/4  & 2/11/0 & 1/10/2 \\
                RS-ET  & 8/5/0   & 4/9/0   & 10/3/0 & 8/5/0  & 4/8/1  & ---   & 4/8/1  & 0/13/0 \\
                RP-DT  & 7/6/0   & 2/6/5   & 9/3/1  & 6/5/2  & 0/11/2 & 1/8/4  & ---   & 1/9/3 \\
                RP-ET  & 7/6/0   & 1/11/1  & 9/4/0  & 7/5/1  & 2/10/1 & 0/13/0 & 3/9/1  & ---\\
    \hline
    \end{tabular}
    \caption{Pairwise t-test comparisons on larger datasets.}
    \label{table:rp:large-ttest}
\end{table}

\subsection{Conclusions}

Overall, this extensive experimental study reveals many interesting results. The
first and foremost result is that ensembles of randomized trees nearly always
beat ensembles of standard decision trees. As off-the-shelf methods, we advocate
that ensembles of such trees should be preferred to ensembles of decision trees.
In particular, these results show that the well-known Random Forest algorithm
does not compete with the best performers.  Far more important to our concern
though, this study validates our RP approach. Building ensembles (of ET) on
random patches of data is competitive in terms of accuracy. Overall, there is no
strong statistical evidence that the method performs less well, but there is
also no conclusive evidence that it significantly improves performance. Yet,
results show that RP is often as good as the very best methods. Regarding the
shape of the random patches, the strategy behind Pasting (i.e., $\alpha_s$ free and
$\alpha_f=1.0$) proved to be (very) ineffective on many datasets while the Random
Subspace algorithm (i.e., $\alpha_s=1.0$ and $\alpha_f$ free) always ranked among the very
best performers. On average, RS indeed came in second on the small datasets and
in first on the larger datasets, which tends to indicate that sampling features
is crucial in terms of accuracy. As for patches of freely adjustable size (i.e.,
using RP), they showed to be slightly sensitive to overfitting but proved to
remain closely competitive with the very best methods. In addition, these
results also suggest that sampling features globally, once and for all, prior to
the construction of a (randomized) decision tree, does not actually impair
performance. For instance, RS-ET or RP-ET are indeed not strongly worse, nor
better, than ET, in which candidates features are re-sampled locally at each
node.

\section{On Memory}
\label{sec:9:memory}

Section~\ref{sec:9:accuracy} reveals that building an ensemble of base
estimators on random patches, instead of the whole data, is a competitive
strategy. In the context of big data, that is when the size of the dataset is
far bigger than the available memory, this suggests that using  random
parts of the data of the appropriate size to build each base estimator would
likely result in an ensemble which is actually as good as if the whole data
could have been loaded and used.

Formally, we assume a general framework where the number of data units that can
be loaded at once into memory is constrained to be lower than a given threshold
$\mu_{\text{max}}$. Not considering on-line algorithms within the scope of this study,
$\mu_{\text{max}}$ can hence be viewed as the total units of data allowed to be used to
build a single base estimator. In the context of our sampling methods, the
amount of memory required for a patch is given by $(\alpha_s N)(\alpha_f p)$ and thus
constraining memory by $\mu_{\text{max}}$ is equivalent to constraining the relative
patch size $\alpha_s\alpha_f$ to be lower than $\mu'_{\text{max}}=\mu_{\text{max}}/(Np)$. While
simplistic\footnote{e.g., the quantity of memory used by the estimator itself
  is not taken into account.}, this framework has the advantage of clearly
addressing one of the main difficulties behind big data, that is the lack of
fast memory. Yet, it is also relevant in other contexts, for example when data
is costly to access (e.g., on remote locations) or when algorithms are run on embedded systems
with strong memory constraints.

In Section \ref{sec:9:memory:sensitivity}, we first study the effects of $\alpha_s$
and $\alpha_f$ on the accuracy of the resulting ensemble and show that it is problem
and base estimator dependent. Second, we show that the memory requirements,
i.e., the relative size $\alpha_s \alpha_f$ of the random patches, can often be
drastically reduced without significantly degrading the performance of the
ensemble (Section \ref{sec:9:memory:noloss}). Third, because the sensitivity of
the ensemble to $\alpha_s$ and $\alpha_f$ is problem and base estimator specific, we show
that under very strong memory constraints adjusting both parameters at the same
time, as RP does, is no longer merely as good but actually significantly better
than other ensemble methods (Section \ref{sec:9:memory:loss}).

\subsection{Sensitivity to $\alpha_s$ and $\alpha_f$}
\label{sec:9:memory:sensitivity}

Let us first consider and analyze the triplets
\begin{equation}
\{(\alpha_s, \alpha_f, \text{Acc}_{\cal L}(\alpha_s,\alpha_f)) | \forall \alpha_s, \alpha_f\}
\end{equation} for various problems, where $\text{Acc}_{\cal L}(\alpha_s,\alpha_f)$ is the average
test accuracy of an ensemble built on random patches of size $\alpha_s \alpha_f$ (using
the same protocol as previously) on the dataset ${\cal L}$.

As Figure~\ref{fig:9:surfaces} illustrates for six datasets, the
surfaces defined by these points vary significantly from one problem to
another. We observed four main trends. In Figures \ref{fig:surfaces:arcene},
and \ref{fig:surfaces:cifar10} (resp., \ref{fig:surfaces:tis}), accuracy
increases with $\alpha_s$ (resp., $\alpha_f$) while adjusting $\alpha_f$ (resp., $\alpha_s$) has no
or limited impact. In Figure \ref{fig:surfaces:madelon}, the best strategy is
to increase both $\alpha_s$ and $\alpha_f$. Finally, in Figures \ref{fig:surfaces:isolet}
and \ref{fig:surfaces:mnist}, the surface features plateaus, which means that
beyond some threshold, increasing $\alpha_s$ or $\alpha_f$ does not yield any significant
improvement. Interestingly, in most of the cases, the optimum corresponds to a
value $\alpha_s\alpha_f$ much smaller than 1.

\begin{figure}
\begin{center}
        \subfloat[\textsc{arcene} (RP-ET)]{
            \label{fig:surfaces:arcene}
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-a-arcene-rp-et.pdf}
        }
        \subfloat[\textsc{cifar10} (RP-ET)]{
            \label{fig:surfaces:cifar10}
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-b-cifar10-rp-et.pdf}
        }
        \subfloat[\textsc{tis} (RP-ET)]{
            \label{fig:surfaces:tis}
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-c-tis-rp-et.pdf}
        }

        \setcounter{subfigure}{0}
        \subfloat[\textsc{arcene} (RP-DT)]{
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-a-arcene-rp-dt.pdf}
        }
        \subfloat[\textsc{cifar10} (RP-DT)]{
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-b-cifar10-rp-dt.pdf}
        }
        \subfloat[\textsc{tis} (RP-DT)]{
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-c-tis-rp-dt.pdf}
        }

        \subfloat[\textsc{madelon} (RP-ET)]{
            \label{fig:surfaces:madelon}
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-d-madelon-rp-et.pdf}
        }
        \subfloat[\textsc{isolet} (RP-ET)]{
            \label{fig:surfaces:isolet}
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-e-isolet-rp-et.pdf}
        }
        \subfloat[\textsc{mnist3vs8} (RP-ET)]{
            \label{fig:surfaces:mnist}
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-f-mnist-rp-et.pdf}
        }

        \setcounter{subfigure}{3}
        \subfloat[\textsc{madelon} (RP-DT)]{
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-d-madelon-rp-dt.pdf}
        }
        \subfloat[\textsc{isolet} (RP-DT)]{
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-e-isolet-rp-dt.pdf}
        }
        \subfloat[\textsc{mnist3vs8} (RP-DT)]{
            \includegraphics[width=0.31\textwidth]{figures/ch8/figure3-f-mnist-rp-dt.pdf}
        }
\caption{Learning surfaces.}
\label{fig:9:surfaces}
\end{center}
\end{figure}

The choice of the base estimators does not have a strong impact on the aspect
of the curves (compare the 1$^{st}$ and 3$^{rd}$ rows of sub-figures in
Figure~\ref{fig:9:surfaces} with those in the 2$^{nd}$ and 4$^{th}$ rows). The only
difference is the decrease of the accuracy of RP-DT when $\alpha_s$ and
$\alpha_f$ grow towards $1.0$. Indeed, since the only source of randomization in RP-DT
is patch selection, it yields in this case ensembles of identical
trees and therefore amounts to building a single tree on the whole
dataset. By contrast, because of the extra-randomization of the split
thresholds in ET, there is typically no drop of accuracy for RP-ET when $\alpha_s$
and $\alpha_f$ grow to $1.0$.

Overall, this analysis suggests that not only the best pair $\alpha_s \alpha_f$ depends
on the problem, but also that the sensitivity of the ensemble to changes to the
size of a random patch is both problem and base estimator specific. As a
result, these observations advocate for a method that could
favor $\alpha_s$, $\alpha_f$ or both, and do so
appropriately given the base estimator.

\subsection{Memory reduction, without significant loss}
\label{sec:9:memory:noloss}

We proceed to study in this section the actual size of the random patches when
the values of $\alpha_s$ and $\alpha_f$ are tuned using an independent validation
set. Our results are summarized in Figure~\ref{fig:9:memory:original}. Each
ellipse corresponds to one of the 29 datasets of our benchmark, whose center is
located at $(\overline{\alpha_s}, \overline{\alpha_f})$ (i.e., the average parameter
values over the 50 runs) and whose semi-axes correspond to the standard
deviations of $\alpha_s$ and $\alpha_f$. Any point in the plot corresponds to a pair
$(\alpha_s, \alpha_f)$ and thus to a relative consumption $\mu' = \alpha_s \alpha_f$ of memory. To
ease readability, level curves are plotted for $\mu' = 0.01, 0.1, ..., 0.9$. In
the right part of the figure, the histogram counts the number of datasets such
that $\overline{\alpha_s}\cdot\overline{\alpha_f}$ falls in the corresponding level set.

Figure~\ref{fig:9:memory:original} corroborates our previous discussion. On some
datasets, it is better to favor $\alpha_s$ while on some other increasing $\alpha_f$ is
a better strategy. The various sizes of the ellipses also confirm that the
sensitivity to variations of $\alpha_s$ and $\alpha_f$ is indeed problem-specific. The
figure also clearly highlights the fact that, even under no memory constraint, the
optimal patches rarely consume the whole memory. A majority of ellipses indeed
lie below the level set $\mu'=0.5$ and only a couple of them are above
$\mu'=0.75$. With respect to ET or RF for which the base estimators are all built
on the whole dataset, this means that ensembles of patches are not only as
competitive but also less memory greedy. In addition, the figure also
points out the difference between RP-ET and RP-DT as discussed in the previous
section. To ensure diversity, RP-DT is constrained to use smaller patches than
RP-ET, hence explaining why the ellipses in red are on average below those in
blue. While RP-DT proved to be a bit less competitive in terms of accuracy,
this indicates on the other hand that RP-DT may actually be more interesting
from a memory consumption point of view.

\begin{figure}
\begin{center}
        \subfloat[Original patches]{
            \label{fig:9:memory:original}
            \includegraphics[width=0.5\textwidth]{figures/ch8/figure4-none.pdf}
        }
        \subfloat[Reduced patches]{
            \label{fig:9:memory:opt}
            \includegraphics[width=0.5\textwidth]{figures/ch8/figure4-mem.pdf}
        }
\caption{Optimal sizes of the random patches on our benchmark.}
\label{fig:9:memory}
\end{center}
\end{figure}

In Section~\ref{sec:9:memory:sensitivity}, we observed plateaus or very
gentle slopes around the optimal pair $(\alpha_s, \alpha_f)$.  From a memory point of
view, this suggests that the random patches are likely to be reducible without
actually degrading the accuracy of the resulting ensemble. Put otherwise, our
interest is to find the smallest size $\alpha_s \alpha_f$ such that the accuracy of the
resulting ensemble is not significantly worse than an ensemble built without
such constraint. To that end, we study the extent at which the constraint $\alpha_s
\alpha_f < \mu'_{\text{max}}$ can be strengthened without any significant drop in
accuracy. If $\mu'_{\text{max}}$ can be reduced significantly then it would indeed mean that
even when only small parts of the data are actually used to build single base
estimators, competitive performance can still be achieved.

Figure~\ref{fig:9:memory:opt} summarizes our results. For all datasets, $\mu'_{\text{max}}$
was set to the lowest value such that it cannot be statistically detected that
the average accuracy of the resulting ensemble is different from the average
accuracy of an ensemble built with no memory constraint (at $\alpha=0.05$). With
regard to Figure~\ref{fig:9:memory:original}, the shift of most ellipses to lower
memory level sets confirm our first intuition. In many cases, the size of the
random patches can indeed be reduced, often drastically, without significant
decrease of accuracy. For more than half of the datasets, memory can indeed be
decreased to $\mu'=0.1$ or $\mu'=0.2$. In other words, building trees on small parts
of the data (i.e., $10\%$ or $20\%$ of the original dataset) is, for more than
half of the datasets, enough to reach competitive accuracy. Also, the
sensitivity to $\alpha_s$ and $\alpha_f$ is now even more patent. Some ensembles use very
few samples ($\alpha_s < 0.1$) but with many features, while other uses many samples
with few features ($\alpha_f < 0.1$). Again, from a memory point of view, RP-DT
appears to be more interesting than RP-ET. The memory reduction is larger, as
the histogram indicates. Optimized splits in the decision trees may indeed lead
to a better exploitation of the data, hence to a potentially larger reduction of
memory. In conclusion, while not detecting significant differences in accuracy
does not allow to conclude that the performances are truly similar, these
figures at least illustrate that memory requirements can be drastically reduced
without apparent loss in accuracy.

\subsection{Memory reduction, with loss}
\label{sec:9:memory:loss}

The previous section has shown that the memory consumption can be reduced up to
some threshold $\mu'_{\text{max}}$ with no significant loss in accuracy. In this section we
now look at the accuracy of the resulting ensemble when $\mu'_{\text{max}}$ is further
decreased. We argue that with severe constraints, and because datasets have all a
different sensitivity, it is even more crucial to better exploit data and thus
to find the right trade-off between both $\alpha_s$ and $\alpha_f$, as only RP can.

\begin{figure}
\begin{center}
        \subfloat[\textsc{arcene}]{
        \label{fig:memc:arcene}
            \includegraphics[width=0.5\textwidth]{figures/ch8/figure5-a-arcene.pdf}
        }
        \subfloat[\textsc{cifar10}]{
        \label{fig:memc:cifar10}
            \includegraphics[width=0.5\textwidth]{figures/ch8/figure5-b-cifar10.pdf}
        }

        \subfloat[\textsc{tis}]{
        \label{fig:memc:tis}
            \includegraphics[width=0.5\textwidth]{figures/ch8/figure5-c-tis.pdf}
        }
        \subfloat[\textsc{madelon}]{
        \label{fig:memc:madelon}
            \includegraphics[width=0.5\textwidth]{figures/ch8/figure5-d-madelon.pdf}
        }

        \subfloat[\textsc{isolet}]{
        \label{fig:memc:isolet}
            \includegraphics[width=0.5\textwidth]{figures/ch8/figure5-e-isolet.pdf}
        }
        \subfloat[\textsc{mnist3vs8}]{
        \label{fig:memc:mnist}
            \includegraphics[width=0.5\textwidth]{figures/ch8/figure5-f-mnist.pdf}
        }
\caption{Accuracy under memory constraint.}
\label{fig:9:memc}
\end{center}
\end{figure}

To illustrate our point, Figure~\ref{fig:9:memc} compares for 6 representative
datasets the accuracy of the methods with respect to the memory constraint $\alpha_s
\alpha_f < \mu'_{\text{max}}$.  A plain line indicates that the generalization error of the
best resulting ensemble under memory constraint $\mu'_{\text{max}}$ is significantly (at $\alpha=0.05$)
worse on the test sets than when there is no constraint
(i.e., $\mu'_{\text{max}}=1$). A dotted line indicates that on average, on the test set, the
ensemble is not significantly less accurate.

As the figure shows, when $\mu'_{\text{max}}$ is low, RP-based ensembles often achieve
the best accuracy. Only on \textsc{arcene} (Figure~\ref{fig:memc:arcene}), RS seems to be a
better strategy, suggesting some overfitting in setting $\alpha_s$ in RP. On all 5
other example datasets, RP is equivalent or better than RS and P for low values
of $\mu'_{\text{max}}$, with the largest gaps appearing on \textsc{isolet}
(Figure~\ref{fig:memc:isolet}) and \textsc{mnist3vs8} (Figure~\ref{fig:memc:mnist}). As
already observed in the previous section, although RP-DT is not the best
strategy when memory is unconstrained, its curve dominates the curve of RP-ET
for small values of $\mu'_{\text{max}}$ in Figures \ref{fig:memc:cifar10},
\ref{fig:memc:tis}, and \ref{fig:memc:madelon}. Because split thresholds are
not randomized in RP-DT, this method is more resistant than RP-ET to the strong
randomization induced by a very low $\mu'_{\text{max}}$ threshold.

For comparison, Figure~\ref{fig:9:memc} also features the learning curves of both
ET and RF (with $K$ optimized on the validation set), in which the trees have
all been built on the {\it same} training sample of $\mu'_{\text{max}}N$ instances,
with all features. These results are representative of the use of a
straightforward sub-sampling of the instances to handle the memory constraint.
On all datasets, this setting yields very poor performance when $\mu'_{\text{max}}$ is
low. Building base estimators on re-sampled random patches thus brings a clear advantage to RP, RS and P and hence confirms the conclusions of \citet{basilico:2011} who showed that using more data indeed produces
more accurate models than learning from a single sub-sample.
This
latter experiment furthermore shows that the good performances of RP
cannot be trivially attributed to the fact that our datasets contain so many
instances that only processing a sub-sample of them would be enough. On most
problems, the slopes of the learning curves of RF and ET indeed suggest that
convergence has not yet been reached on these datasets. Yet, important
improvement are gained by sub-sampling random patches.
Contrary to intuition, it
also does not appear that the bigger the datasets, the lower $\mu'_{\text{max}}$ can be
reduced without loss of accuracy. Indeed, we found no conclusive correlation
between the size $Np$ of the dataset and the minimum value of $\mu'_{\text{max}}$ to
reach good performance\footnote{Spearman's rank correlation coefficient is
-0.0354 ($p-\mathrm{value}=0.8550$).}.
Overall, these results thus indicate that
building an ensemble on random patches is not only a good strategy when data is
abundant and redundant but also that it works even for scarce datasets with
limited information regarding the problem.

\subsection{Conclusions}

We have shown in this section that the memory requirements of sampling based
ensembles are intrinsically low. Better, we have shown that they can often be
drastically decreased without significant loss in accuracy. When the size of the
dataset is far bigger than the available memory, we have also demonstrated that
sampling data along both samples and features, as RP does, not only competes
with other ensemble algorithms but also significantly improves the accuracy of
the resulting ensemble. It also brings a significant improvement over a
straightforward sub-sampling of the instances.

\section{Overall conclusions}
\label{sec:9:conclusions}

The main contribution of this work is to explore a new framework for supervised
learning in the context of very strong memory constraints or, equivalently, very
large datasets. To address such problems, we proposed the Random Patches
ensemble method that builds each individual model of the ensemble from a random
patch of the dataset obtained by drawing random subsets of both samples and
features from the whole dataset. Through extensive experiments with tree-based
estimators, we have shown that this strategy works as well as other popular
randomization schemes in terms of accuracy (Section~\ref{sec:9:accuracy}), at the
same time reduces very significantly the memory requirements to build each
individual model (Section~\ref{sec:9:memory:noloss}), and, given its flexibility,
attains significantly better accuracy than other methods when memory is severely
constrained (Section~\ref{sec:9:memory:loss}). Since all models are built
independently of each other, the approach is furthermore trivial to parallelize.
All in all, we believe that the paradigm of our method highlights a very promising
direction of research to address supervised learning on big data.

There remain several open questions and limitations to our approach that we
would like to address in the future. First, this study motivates our interest in
experimenting with truly large-scale problems (of giga-scale and higher). Since
RP already appears advantageous for small to medium datasets, the potential benefits on
very large-scale data indeed look very promising.

Second, the conclusions drawn in sections~\ref{sec:9:accuracy} and
\ref{sec:9:memory} are all based on the optimal values of the parameters $\alpha_s$ and
$\alpha_f$ tuned through an exhaustive grid search on the validation set. Our
analysis did not account for the memory and time required for tuning these two
parameters. In practice, hyper-parameter tuning can not be avoided as we have
shown that the optimal trade-off between $\alpha_f$ and $\alpha_s$ was problem dependent.
It would therefore be interesting to design an efficient strategy to
automatically find and adjust the values of $\alpha_s$ and $\alpha_f$, taking into account
the global memory constraint. Our simplistic framework also only accounts for
the memory required to store the training set in memory and not for the total
memory required to actually build the ensemble.

We have only explored uniform sampling of patches of fixed size in our
experiments. In the context of the Pasting approach, Breiman proposed
an iterative instance weighting scheme that proved to be more
efficient than uniform sampling \citep{breiman:1999}. It would be
interesting to extend this approach when sampling both instances and
features. Yet, parallelization would not be trivial anymore, although
probably still possible in the line of the work in \citep{chawla:2004}.

Finally, our analysis of RP is
mostly empirical. In the future, we would like to strengthen these
results with a more theoretical analysis. A starting point could be
the work in \citep{zinkevich:2010} that studies a scheme similar to the
Pasting method applied to linear models trained through parallel
stochastic gradient descent. The extension of this work to non
parametric tree-based estimators does not appear trivial however,
since these latter are not well characterized
theoretically. In retrospect, latest results~\citep{scornet:2014} regarding the consistency
of random forests could certainly also be used to motivate the use of random subsets of samples.
Similarly, theoretical results from Chapter~\ref{ch:importances} (Section~\ref{sec:6:rs})
also indicate that the use of random subsets of features does not systematically
degrade performance.


\section{Application to field data}

We applied our method to airborne gravity data from Caraj{\'a}s, north of Brazil, which were provided by
the Geological Survey of Brazil (CPRM). The data were collected along $131$ north-south flight lines separated
by $3$ km and $29$ east-west tie lines separated by $12$ km.
This data set was divided in two different areas, collected in different times, having samples spacing of
$7.65$ m and $15.21$ m along the lines, totalizing  $5,492,551$ observation points at a fixed height
of $900$ m ($z_{1} = -900$ m).
The gravity data were interpolated (Figure \ref{fig:carajas_real_data}) into a regularly spaced grid of
$500 \times 500$ observation points ($N = 250,000$) with a grid spacing of
$\approx 717$ m north-south and $\approx 782$ m east-west.

To apply our method, we set an equivalent layer at $z_{0} = 300$ m.
Figure \ref{fig:carajas_gz_predito_val}a shows the predicted data obtained with our method after
$N^{it} = 50$ iterations.
The residuals (Figure \ref{fig:carajas_gz_predito_val}b), defined as the difference between the observed
(Figure \ref{fig:carajas_real_data}) and predicted (Figure \ref{fig:carajas_gz_predito_val}a) data, show a
very good data fit with mean close to zero ($0.0003$ mGal) and small standard deviation ($0.1160$ mGal),
which corresponds to approximately $0.1$ \% of the maximum amplitude of the gravity data.
By using the estimated mass distribution (not shown), we performed an upward-continuation of the
observed gravity data to a horizontal plane located $5,000$ m above. Figure \ref{fig:up2000_carajas_500x500}
shows a very consistent upward-continued gravity data, with a clear attenuation of the short
wavelengths. By using our approach, the processing of the $250,000$ observations took only
$0.216$ s.


\documentclass[../report/main.tex]{subfiles}

\begin{document}

% The asterix after \subsection disables section numbering
\subsection*{Algorithm Description}

We learned about the minimum spanning tree heuristic from a discussion in ~\cite{skiena2008} .  The concept is to find the minimum spanning tree (MST).  Then the tour order is set by the discovery order while performing a depth first search (DFS) on the minimum spanning tree.

\subsection*{Algorithm Discussion}

We implemented this algorithm because it seemed relatively simple.  For this implementation we used Python 3.5.  At first we implemented Prim's algorithm to find the MST but we found it to be too slow.  Therefore, we switched to using Kruskal's algorithm to find the MST which provided a better running time of O(n lg n).  Finally, according to ~\cite{skiena2008}, this heuristic is typically 15\% to 20\ over optimal in practice.  Unfortunately, while it was decently fast for smaller input sizes, it was inefficent for very large input sizes (n $\geq$ 5000).  Also, for the three example cases it didn't provide the expected results relative to optimal.  Depending on the case the route found was 31\% to 88\% worse than optimal.

\subsection*{Algorithm Pseudo-code}

\begin{verbatim}
DFSofMST()
    cities = []
    read data from file into cities[] // each city has id, x, and y attributes

    adjMatrix=[len(cities)][len(cities)]
    for i = 0 to len(cities) - 1
        for j = i + 1 to len(cities)
            ij = dist(cities[i][j])
            adjMatrix[i][j] = ij
            adjMatrix[i][j] = ji

    mst = kruskalsAlg(adjMatrix)
    adjList = mstToAdjList(adjMatrix, mst)
    disc = dfs(adjList)

    totalDist = 0
    prevCity = cities[disc[0]]
    for i = 1 to len(disc)
        eachCity = cities[disc[i]]
        addDist = dist(eachCity, prevCity)
        totalDist += addDist
        prevCity = eachCity
    addDist = dist(prevCity, cities[disc[0]])
    totalDist += addDist

    write totalDist to file
    write contents of disc[] to file // 1 item per line

dist(cityOne, cityTwo)
    dx = cityOne['x'] - cityTwo['x']
    dy = cityOne['y'] - cityTwo['y']
    dxSq = dx ^ 2
    dySq = dy ^ 2
    return ((dxSq + dySq) ^ 0.5)

dfs(adjList)
    vstd = []
    stack = []
    stack.append(0)

    while(len(stack) > 0)
        u = stack[len(stack) - 1]
        stack.remove(stack[len(stack) - 1])
        if not (u in vstd)
            vstd.append(u)
            auxStack = []
            for eachAdj in reverse_sorted(adjList[u].items()):
                v = eachAdj[0]
                if not (v in vstd):
                    auxStack.append(v)
            while len(auxStack) > 0
                v = auxStack[0]
                stack.append(v)
                auxStack.remove(v)

    return vstd

mstToAdjList(adjMatrix, mst)
    adjList = {}
    for i = 0 to len(adjMatrix)
        adjV = {}
        for j = 0 to len(mst)
            if i == mst[j]
                adjV[j] = adjMatrix[i][j]
        adjList[i] = adjV
    return adjList

popMin(prqu)
    minU = prqu[0]
    prqu.remove(prqu[0])
    return minU

decreaseKey(prqu, cost)
    for i = 0 to len(prqu)
        for j = 0 to len(prqu)
            if cost[prqu[i]] < cost[prqu[j]]
                prqu[i] = prqu[i] + prqu[j]
                prqu[j] = prqu[i] - prqu[j]
                prqu[i] = prqu[i] - prqu[j]

def kruskalsAlg(adjMatrix)
    edges = []
    for i = 0 to len(adjMatrix) - 1
        for j = i + 1 to len(adjMatrix)
            e = edge(i, j, adjMatrix[i][j])
            edges.append(e)
    edges.sort(key = lambda x: x.d)

    prev = [None for x in range(len(adjMatrix))]
    vstd = []
    for e in edges:
        if (not(e.v in vstd))
            vstd.append(e.v)
            prev[e.v] = e.u
    return prev
\end{verbatim}
\end{document}


\documentclass[a4paper,11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{algorithmic}
\usepackage{algorithm}
\usepackage{pst-plot}
\usepackage{graphicx}
\usepackage{endnotes}
\usepackage{graphics}
\usepackage{floatflt}
\usepackage{wrapfig}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{verbatim}
\usepackage{hyperref}
\usepackage{multirow}
\usepackage{pdflscape}
 \usepackage{enumitem}

\usepackage{hyperref}
\hypersetup{pdfborder={0 0 0 0}}

\pdfpagewidth 210mm
\pdfpageheight 297mm
\setlength\topmargin{0mm}
\setlength\headheight{0mm}
\setlength\headsep{0mm}
\setlength\textheight{250mm}
\setlength\textwidth{159.2mm}
\setlength\oddsidemargin{0mm}
\setlength\evensidemargin{0mm}
\setlength\parindent{7mm}
\setlength\parskip{0mm}

\newenvironment{exercise}[3]{\paragraph{Exercise #1: #2 (#3pt)}\ \\}{
\medskip}
\newcommand{\question}[2]{\setlength\parindent{0mm}\ \\$\mathbf{Q_#1:}$ #2\ \\}

\author{\large{Ardi Tampuu, Ilya Kuzovkin, Raul Vicente}}
\title{\huge{Introduction to Computational Neuroscience}\\\LARGE{Practice on Single Neuron Models}}

\begin{document}
\maketitle

\textbf{A request:} Please track how long it will take to complete this set of exercises. Add this time to your final report.
\ \\
\textbf{NOTE:} In this homework you are required to also submit you code.\\
\textbf{NOTE:} If you are not using Matlab/Octave, you will unfortunately need to rewrite the code in your preferred language. In the beginning of the course I warned this will happen eventually. But consider also doing this session in Matlab, as not much knowledge of the language is required for these tasks - just replace some values here and there.\\
\ \\
\ \\
%
% Intro
%
In this session we will have a brief look on three different computational models of a neuron: McCulloch-Pitts, Intergrate-and-Fire and Hodgkin-Huxley.\

\ \\

%
% Logic gates
%
\begin{exercise}{1}{Logic gates}{1}
On the lecture we have seen how to construct \texttt{AND}, \texttt{OR} and \texttt{NOT} logic gates using the the McCulloch-Pitts model of a neuron. Remember that:\\
 \texttt{if sum(w.*x)<0, output is 0}\\
 \texttt{if sum(w.*x)>=0, output is 1} (notice that 0 is included)\\
 Your task is to construct more logical operations using MP neurons. Please construct the following two gates:
\begin{enumerate}
\itemsep 0em
	\item \texttt{NAND}
	\item \texttt{XOR}
\end{enumerate}
\textbf{HINT 1} For the \texttt{XOR} gate you will need more than one neuron.\\
\textbf{HINT 2} Same input can go simultaneously to several neurons (and with different weights).
\end{exercise}

%
% Integrate and Fire
%
\begin{exercise}{2}{Integrate and Fire neuron model}{2.5}

Integrate-and-Fire neuron (IAF neuron) accumulates voltage until it reaches the \emph{threshold}. After that it outputs/emits a spike and resets the voltage back to the reset value. In this exercise we will model the behaviour of IAF neurons and study their properties. We will add features and behaviours to our model step by step. \\

The file \texttt{code/integratefire.m} contains first of all some theoretical explanation of how we are going to simulate a neuron's behaviour (read it!) by numerically solving the differential equation governing integrate-and-fire neuron.\\ \textbf{Your task is} to complete the series of \texttt{TODO}-s in the \texttt{integratefire.m}. You need to complete these TODO-s one by one in the given order and report \textbf{ALL} figures, answers, interpretations and conclusions you will make during the work.\\

\textbf{You will need to:}\\
\begin{enumerate}
\item \texttt{TODO 1}: write how to calculate V(2), given V(1), I and C
\item \texttt{TODO 2}: write how to calculate V(3), given V(2), I and C
\item \texttt{TODO 3}: write how to calculate V(t+1), given V(t), I and C. Plot what happens and describe (describe in comparison with TODO 4)
\item \texttt{TODO 4}: replace the 0 with the conditional sentence to check if V(t) is above the threshold level Vth. Plot what happens. Describe, compare.
\item \texttt{TODO 5}: replace the 0 with an expression that would add 1 to $number\_of\_spikes$. Run the for loop again, report the count with default values.
\item \texttt{TODO 6}: increase the current to 10.0. Plot the behaviour and report the count.
\item \texttt{TODO 7}: Try also other currents, observe the spike count (no need to plot Vm(t) for each current). Plot it in your head and/or on a graph. What is the relationship - exponential? logarithmic? linear? less than linear?
\item \texttt{TODO 8}: add the term $noise*sqrt(dt)*randn $ to the V(t+1) equation. This input reflects he fact that neurons always receive random inputs from their neighbours. Observe the noise on Vm plot.
\item \texttt{TODO 9}: This \texttt{TODO} has its separate part of code below the for loop used so far. Your task is to change the noise level and plot 3 different raster plots (10 trials on each) with different noise levels. Instructions are at the end of the file.
\end{enumerate}

\ \\
One of the outputs should look something like this
\begin{figure}[H]
   \centering
   \includegraphics[width=0.8\textwidth]{raster_plot.png}
   \caption{10 trials of data generated using Integrate-and-Fire neuron model.}
   \label{fig:rasterplot}
\end{figure}
\end{exercise}

% Integrate and Fire with synapses
%
\begin{exercise}{3}{Integrate and fire advanced}{1.5}

In this exercise we will play with somewhat more realistic versions of integrate-and-fire model.\\

In first stage we add a leak current to the neuron. This current is caused by the ion pumps and it drives the membrane potential slowly towards its resting state. In the previous exercise there was no such current - when the input current stops, the potential remains fixed at the current value. With leak current added, the Vm now slowly returns to its resting state at -70 mV.\\
Another difference is that beforehand any current, no matter how small, would eventually drive the neuron to spike. In the current exercise however, weak inputs are balanced out by the leak current.\\

The code you have to read, understand and modify is in \texttt{leakyintegratefire.m}\\

Your task is to:
\begin{itemize}
\item find the lowest $I_{injected}$ (with 0.1 precision) that leads to a spike. Plot the behaviour at this \textbf{I} value and at the value 0.1 below (last one with no spike). Describe what happens. Describe also what happens after the stimulus is turned off.
\item How does firing rate (spikes per second) differ at I = 5.0, 10.0, 25.0, 50.0 with and without (results in TODO 7 in previous exercise) leak current?
\end{itemize}

In the second stage we explore the neuron's behaviour if it receives input from incoming (\emph{presynaptic}) spikes rather than constant injected current. Even though one incoming signal is not enough to activate our neuron, \emph{temporal summation} of inputs happening in a short period of time can lead to the voltage reaching the threshold. The code you need to understand and modify is provided in \texttt{integratefiresynapses.m}.\\

For this task you have to remember that if a neuron gets enough excitatory input in a short period of time, it will activate. The same amount of inputs dispersed over a longer period will not lead to firing. Your task is again to modify code near the \texttt{TODO} signs and to answer some questions:
\begin{enumerate}
	\item \texttt{TODO 1} The neuron only receives one incoming spike at t=300ms. Plot the membrane potential and the strength of the synaptic current in time. i) Describe how membrane potential and strength of current change over time (peak values, shape). ii) How long does it take for the current to decrease by half (half-life)? How is that related to the "taus" parameter? iii) Modify the strength of the synaptic current up to a point where one presynaptic spike is enough to produce a postsynaptic spike.
	\item \texttt{TODO 2} Uncomment the line and the neuron will receive a spike every 50ms. Nevertheless it never fires. Why?
	\item \texttt{TODO 3} Uncomment the line and replace t3 (time of third spike) with a number bigger than 101. What is the lowest value for t3 so that the receiving neuron does not emit a spike?
\end{enumerate}
\end{exercise}

%
% Hodgkin-Huxley
%
\begin{exercise}{4}{Hodgkin-Huxley neuron model}{1.5}

Hodgkin-Huxley model is considered to be the most important computational neuronal model in the neuroscience today. We have the model already implemented in the file \texttt{HH0.m}, study it. Follow the instructions in the \texttt{hodgkinhuxley.m} file and report all figures, thoughts, interpretations and conclusions you will have during the work.\\

The \texttt{TODO} marker will indicate places where you have to do something: complete the code, plot and report a figure, give an interpretation, etc.
\begin{enumerate}
\item Explain the nature of m, h and n in terms of Na+ and K+ ion channels being open or closed (discussed in lecture).
\item Add the figure at I0=5.0 to the report. Modify I0 and report the lowest I0 that still causes a spike in the beginning of simulation. And the lowest I0 that causes continuous spiking till the end of simulation. (Precision of your answer should be at least 0.1)
\item Count the number of spikes at different I0 values (code provided). \textbf{Note that} this series of simulations can take up to 5-10 min, do not panic if it takes time. Describe what you see. Why this \textit{input-output curve} it more biologically realistic than the input-output relation of IAF? How is the non-linearity of the relationship related to modelling the ion channels and not simply resetting the membrane potential instantaneously as soon as threshold is reached?
\end{enumerate}
\end{exercise}

\begin{exercise}{5*}{Bonus}{0.5}
I take the freedom to award bonus points for the appearance of the report. Some reports are really badly formatted and hard to read. So I want to reward people who make their reports more readable and my life more pleasant.
\end{exercise}

\ \\
\ \\
\ \\
\ \\

Please submit a \texttt{pdf} report with answers to the questions and comments about your solutions. Include figures, explanations. \textbf{Include the completed code files and zip the files together.} The code files are not the place to answer questions, answers must also be contained in the pdf. Upload it on the practice session page on the course website.

\end{document}

2016/Practices/04 - Single Neuron Models/text/cns-singleneuron.tex


\subsubsection{\stid {3.01} xSDK Sub-project: batched sparse linear algebra}

\paragraph{Overview}

Over the course of the development of the xSDK libraries and
interactions with the ECP applications teams, the need for batched
sparse linear algebra (LA) functions has emerged in order to make more
efficient use of the GPUs for many small and sparse linear algebra
problems. Currently there is almost no support from the vendors for
batched sparse linear algebra routines, except
for a few functions from NVIDIA (batched SpMV and batched sparse QR).
The broad appeal of such a functionality that could be
integrated across xSDK libraries represents a unique opportunity in
terms of avoiding critical performance bottlenecks and integrating
algorithmic advances across low-level software libraries used in ECP.
We have identified six ECP applications that will benefit from new batched sparse solvers.
These include chemical and nuclear kinetics, device modeling  power grid planning and additive manufacturing applications.
We will work with vendors to develop batched sparse linear algebra software, starting with the design of an interface,
followed by the implementation of the software and their integration into application codes and libraries.
We will evaluate the performance, pinpoint any issues and implement solutions to improve performance.

\paragraph{Key Challenges}

The applications we identified need to solve many small linear systems
with varying sizes and sparsity patterns simultaneously.  System sizes
range between 30 and 200. The insight into the challenges of this effort
then are the combination of dealing with sparsity, batched interface,
iterative solvers, and matrix properties that depend on the elements'
values.
One of the main challenges facing this effort is how recent the
discovery of the need for better handling of batched sparse LA is. This
will require bridging the gap between what is possible on the upcoming
Exascale hardware platform and what the eventual need of ECP
applications will be.
There are no such solvers available yet to draw the design potential and
limitations from or try to conduct early feasibility and performance
studies. This implies that a lot of the effort will face the lack of
easy starting points and more careful analysis is needed on how to proceed.
We envision challenges due to the structural sparsity that is already an
issue on the accelerators for the existing software that deals with
sparsity such as direct or iterative solvers and graph processing
kernels.
Additionally, potential solutions would be limited to what the application
data allows. In particular, the matrix elements will determine numerical
properties such as conditioning that could vary even if other parameters
remain the same across the batch.
All the variable factors mentioned above will require exploration and
pose a challenge to limit the potential splintering of work with little
eventual return. Minimizing this will require tight coordination between
participants.

\paragraph{Solution Strategy}

We will try to draw on the existing libraries and their solvers and
either extend their capability to our new context or limit their
features that are an unlikely fit for the specialized function of
batched sparse LA for small batch sizes, such as multi-GPU support.
With the abundance of iterative solvers, we will
start exploring their functionality and potential extensions for our
effort.
We are collaborating with ECP industry partners, including NVIDIA, AMD,
and Intel, to develop a set of new APIs to support required batched
sparse linear algebra operations for various GPUs. This is essential to
receive an early and frequent feedback on the direction and the
potential benefit of the solvers and data formats we propose and
evaluate.
We will then develop batched sparse linear solvers, both direct and
iterative, and preconditioners for the GPUs. We will also develop new
interoperability layers for integration across the applications,
solvers, and lower-level libraries. The planned work will involve the
following math libraries: Ginkgo, hypre, Kokkos Kernels, MAGMA, PETSc,
PLASMA, STRUMPACK, SUNDIALS, SuperLU-DIST, and Trilinos.
We envision that the functionality
should be for on-device compute with careful use of shared/local/private
memory (e.g., only 64KB on NVIDIA GPU). The interface will support
multiple sparse matrix formats. For language interoperability, C++ and C
will be the core languages, and SWIG will be used to build the Fortran
wrappers.

\paragraph{Recent Progress}

In initial meetings with team members and vendors, we started reviewing existing batched
interface options in vendor and xSDK libraries.
We extracted application data in the form of small sparse matrices that
occur in batches in the Chemical kinetics ECP application Pele. They are now available to the
project members in an easily readable form for analysis and testing with
the existing and future batched kernels.
We started reviewing the options available for host-side and device-side
interface design.

\paragraph{Next Steps}

Our next efforts include
\begin{itemize}
\item Finalization of vendor batched interface review,
\item Organization of batched sparse matrix examples,
\item Design of batched sparse linear algebra interface for the relevant application use cases,
\item Initial implementation of batched sparse linear algebra software,
\item Integration of batched sparse LA software into applications and libraries and performance assessment of effort.
\end{itemize}


\subsection{Demand Forecasting with Machine Learning Methods}
\label{ml_methods}

ML methods have been employed in all kinds of prediction tasks in recent
    years.
In this section, we restrict ourselves to the models that performed well in
    our study: Random Forest (RF) and Support Vector Regression (SVR).
RFs are in general well-suited for datasets without a priori knowledge about
    the patterns, while SVR is known to perform well on time series data, as
    shown by \cite{hansen2006} in general and \cite{bao2004} specifically for
    intermittent demand.
Gradient Boosting, another popular ML method, was consistently outperformed by
    RFs, and artificial neural networks require an amount of data
    exceeding what our industry partner has by far.